As a disclaimer, i realize that Debian in any version isn’t a supported OS on Hyper-V R2 – i just want to tell of my experiences with this unsupported configuration.

The hardware, an aging IBM xSeries 306m with a Pentium 4 CPU wasn’t getting any younger and after a drive failure about half a year ago that lead to a system crash (No data loss though – it just crashed the machine, that’s Software RAID for you), it was finally time to modernize this.

The plan is to consolidate all our DMZ workloads (ISA, OCS Edge, XMPP Gateway, Exchange Edge) on Hyper-V 2008 R2 and doing the trickiest part first seemed like a good idea.

So i created a new VM using SCVMM 2008 R2, selected Other Linux 32bit as the guest OS, inserted a Debian 5.0 netboot CD and that’s where the problems already started. While the installation worked well in general, the Framebuffer used by the Debian installed is awfully slow. So it took me about half an hour just to get the install done (on a 5GB partition of the 80GB VHD).

After finishing the installation, i formatted the rest of the disk appropiately and then used rsync to transfer the machine contents over. A short bit after reconfiguring Grub, i could choose to boot either the transferred OS with it’s kernel, or the Debian 5 rescue system i installed alongside.

Booting the transferred system worked well enough, but the tulip driver wasn’t compiled into that (custom) kernel and building the module failed. So i read up a bit, and realized that the newest kernel (2.6.32.8) shipped with experimental Hyper-V VMbus drivers, that allowed synthetic NICs to be used.

I tried to compile the kernel after chrooting into the old installation, but it failed because gcc was too old. Not to worry, i compiled it in the rescue system, but couldn’t install the dpkg that make-kpkg created. So i installed it manually, which worked pretty well.

One reboot later, i was back in business with the extremely verbose Hyper-V drivers cluttering up dmesg, but the Synthetic NICs showed up as seth0 – seth2. After quickly changing all the necessary configuration files, everything was working.

After a bit of more testing, i disconnected the physical machine from the network and plugged the VM into the production VLANs.

I tested everything thoroughly and didn’t find any issues. Sent out an information mail and continued on my merry way.

Half an hour later, i decided to do a quick systems check again – and i realized that the external interface (seth2 in this case) wasn’t working anymore. tcpdump showed no packets being received and other machines in the same VLANs didn’t see any answers to their ARP requests either. So i rebooted the VM, and everything was working again. No error messages of any kind, neither in dmesg nor in the system logs or on the Hyper-V host.

Hoping this was just a fluke, i waited until it happened again – which it did, roughly 10 minutes later. So i decided to skip on the synthetic devices and go with emulated NICs and the tulip driver.

Everything came back up, but i couldn’t ping any devices on the eth0 VLAN from the start, but the other two interfaces worked.

After a few more tries, i arrived at a configuration that has now been stable for 4 hours and 26 minutes, which sounds good so far. For this, i configured a single synthetic NIC that i used as a replacement for the non-working eth0 and three tulip NICs (of which the first was unused).

There are other things that also worry me:

Every reboot of the Linux machine created the following event log entry on the Hyper-V host:

'LINUX' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID [])

Loading the synthetic NIC drivers logs the following in the event log on the Hyper-V host:

Networking driver on 'LINUX' loaded but has a different version from the server. Server version 3.2 Client version 0.2 (Virtual machine ID []). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.

Loading the synthetic NIC drivers also logs all this on the Linux side of things:

VMBUS_DRV: Vmbus initializing.... current log level 0x1f1f0006 (1f1f,6)
VMBUS: +++++++ Build Date=Feb 17 2010 12:37:00 +++++++
VMBUS: +++++++ Build Description=Version 2.0 +++++++
VMBUS: +++++++ Vmbus supported version = 13 +++++++
VMBUS: +++++++ Vmbus using SINT 2 +++++++
VMBUS: Windows hypervisor detected! Retrieving more info...
VMBUS: Vendor ID: Microsoft Hv
VMBUS: Interface ID: Hv#1
VMBUS: OS Build:7600-6.1-16-0.16485
VMBUS: Hypercall page VA=f80c9000, PA=0x36afe000
VMBUS_DRV: irq 0x5 vector 0x35
VMBUS: SynIC version: 1
VMBUS: Vmbus connected!!
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={c5295816-f63a-4d5f-8d1a4daf999ca185}
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
hv_netvsc: module is from the staging directory, the quality is unknown, you have been warned.
NETVSC_DRV: Netvsc initializing....
VMBUS_DRV: child driver (f80dc570) registering - name netvsc
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={32412632-86cb-44a2-9b5c50d1417354f5}
VMBUS_DRV: child device (f73a8634) registered
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a}
VMBUS_DRV: child device (f73a5a34) registered
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={f8615163-df3e-46c5-913ff2d2f965ed0e}
VMBUS_DRV: device object (f73a5ee4) set to driver object (f80dc5c0)
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS: channel f73aac00 open success!!
NETVSC: *** NetVSC channel opened successfully! ***
NETVSC: Sending NvspMessageTypeInit...
NETVSC: NvspMessageTypeInit status(1) max mdl chain (34)
NETVSC: Sending NvspMessage1TypeSendNdisVersion...
NETVSC: Establishing receive buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendReceiveBuffer...
NETVSC: Receive sections info (count 1, offset 0, endoffset 1048000, suballoc size 1600, num suballocs 655)
NETVSC: Establishing send buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendSendBuffer...
NETVSC: *** NetVSC channel handshake result - 0 ***
NETVSC: Device 0xf6552e80 mac addr 00155d031a09
NETVSC: Device 0xf6552e80 link state up
VMBUS_DRV: child device (f73a5e34) registered

So, it works. But not without troubles. I’ve still got the physical machine to fall back on, but i sure hope Microsoft will get this to work better.

These issues are the reason why i decided to deploy my private server using ESXi instead of Hyper-V – because i need both Linux and Windows guests.

Sure, there are loads of MVPs and TAP-Members that have migrated to Exchange 2010 a long time ago, but i’m still proud of this.

At a starting point, i had a Exchange 2007 SP2 machine, with one Mailbox database, no public folders and 35 Mailboxes that used up 25GB of space. Moving this is simple enough, but the issue is that our Exchange isn’t virtualized, and i couldn’t get my hands on new hardware since the current box was only a year old.

Since in-place upgrades are not supported, i needed a temporary server for the migration. I used an HP ML110 from the Lab, which offered enough space to migrate.

Another issue was BackupExec 12.5, which did not support Exchange 2010 yet. Fortunately, Exchange 2010 (and 2007 SP2) can be backed up by using Windows Server Backup. So my goal was to just let WSB backup to a file server, and have BackupExec pickup the files from there. This way, i will get a reliable, clean and supported Exchange backup, and still have it on tape.

To Migration itself was straightforward and easy. There’s already _lots_ of content on the web about Exchange 2010, most of it from the RCs or Beta of course.

I followed the Migration Guide from TechNet, which worked out well enough. Unfortunately, the iPhone does not support Exchange 2010/2007 coexistence, which made it necessary for several people to manually reconfigure their phone.

Removing Exchange 2007 worked without issues, but after moving all the Exchange 2010 data back to the real hardware and removing the temporary server i ran into the issue of moving arbitration mailboxes, which fortunately was already documented widely on the web.

In the end, upgrading from Exchange 2007 to 2010 while keeping the same hardware is not difficult, it just needs a bit more time.

The error message in the IIS Log looks like this:

RdirTo:https%3a%2f%2flegacy.contoso.com%2fMicrosoft-Server-ActiveSync_LdapC2_LdapL15_Error:MisconfiguredDevice_Budget

During this time, i’ve gained valuable experience, which i’ll try to share here so that others can profit from it. Take all this with a grain of salt, as some observations may simply be my fault. Also, as times changes these things might change too.

Software

• Make sure to install Windows Server 2008 SP2 after installing SBS 2008. Some media may come with SP2 already preloaded. You can use the normal SP2 package that’s also used for Vista and the normal Server 2008
• Do not install SBS rollup updates before completing the configuration wizard. This is extremely counter-intuitive, but is described on the Official SBS blog
• Installing Exchange 2007 SP2 requires you to follow special considerations Here
• Installing WSUS 3.0 SP2, which is needed to support Windows 7, is currently not recommended. I was able to do this without issues on my lab machines, but others have reported issues doing this on machines that were in production. If you’re deploying a new SBS server, this should probably be safe to go. But make sure to test functionality afterward.
• Always use the answer file to deploy SBS 2008. This will make it possible to choose a custom domain name. Read my post about choosing your AD DNS namespace
• Do whatever tasks you can do using the SBS console. Resist of using the normal administration tools as much as possible, as you can break SBS with them easily.
• Ensure that the AV software you install is compatible with WS08 x64. Symantec Endpoint Protection Manager works well – Forefront Client Security on the other hand requires a seperate server running 32bit Windows for management. You may consider deploying FCS unmanaged in smaller environments, and configure FCS using the FCS ADM File

Hardware

• Use servers with the new Xeon 5500 CPUs. Read my x3650 M2 tips to find more about them. Consider using an E5530 or faster CPU. Using two CPUs (for a total of 16 virtual and 8 physical cores) makes little sense.
• Buy enough memory. Lots of it. Really. I mean it. You’ll need lots and lots of memory. I would consider 12GB to bare minimum. In a 3x4GB configuration which makes the most sense for the Xeon 5500 setups, this is quite cheap. Consider more memory if you intend to run SQL Server as, consider bumping the memory to 24GB. Remember that you can only use the first 8 slots in a single socket machine.
• Buy enough disks. A good starting layout is 8x147GB 2.5″ disks. Use a RAID 1 for the OS, another RAID1 for Exchange and Sharepoint, and a RAID10 for Data and WSUS. This is all up for debate of course, and it might make sense to consider other disk layouts.

If you have any additions, think i’m wrong somewhere just send in a comment.

Security updates are important. And as we’re currently an evaluation setup for OCS 2007 R2, i’ve decided to install todays batch of security updates on these lesser important machines first. And after a reboot, OCS 2007 R2 was broken.

A quick view into the event log revealed that OCS 2007 R2′s evaluation license has expired. Now, this seemed very strange as i’ve installed from volume license media. I’ve the checked the media again, but they weren’t evaluation media.

Here’s the message in all it’s glory:

Event source: OCS Server
Event id: 12290
Event text: The evaluation period for Microsoft Office Communications Server 2007 R2 has expired. Please upgrade from the evaluation version to the full released version of the product.

Maybe i really did use other media to install it? I doubted myself, because that’s usually the most reasonable approach to take. The error is usually behind the keyboard.

Luckily, Microsoft has published documentation on how to upgrade an evaluation version to a full version. Unfortunately, this didn’t work, because as it appears i was running a Volume license version of OCS.

EVALTOFULL parameter cannot be used with currently installed license type Volume

At this point, i was pretty sure that this wasn’t my fault. There has been an issue with the OCS 2007 R2 Evaluation Media expiring at the wrong point in time, but apparently this has been sorted out and did never affect the full versions of OCS 2007 R2.

So i was bummed. A quick view using process monitor revealed that the licensing information was most likely to be stored here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RtcSrv\InstallInfo\ValidationData

I created a backup of that part of the registry, and then renamed the key. I got a file not found error, and created a new key of the same type and wrote binary data of the same length into it. This yielded the following error:

The service is shutting down due to an internal error.

Error Code: 80093102 (ASN1 unexpected end of data.)

At that point, i was pretty sure what might’ve caused this – the MS Crypto API security update KB974571.

I removed the update, rebooted the machine, and OCS 2007 R2 was up and running again, without any issues.

I’ve already opened a case with Microsoft to get this sorted out.

Update:
Appears that this is an official issue: See here

Update: The license is not entirely clear – it does not exclude commercial use, only SaaS use. But in the beginning of the license it says that only home-based small businesses are allowed to use it. So take this with a grain of salt – the license is certainly more permissive than Avira, but it’s not as easy as i thought.

I’ve been participating in the MSE beta test with my Windows 7 machines at home, and my impression has been very good. Performance is excellent, and the GUI is simple and straightforward.

After a few negative experiences with McAfee 8.7 at work, and my very good experiences with MSE at home, i tried to have another look at FCS.

Well, FCS is rather old right now, with the new release just on the horizon. Still, the current release is supported on Windows 7 x64 with the latest patches, and so far my impressions have been very good. The management server only runs on 32bit Windows, which also means it won’t run on WS08R2.

But my impression has been good so far – the package installed on the client is far more lightweight than McAfee, and even the managment software leaves a much better impression.

We’ve also been using Symantec Endpoint Protection at a few customers, but my impression of that product was even worse than McAfee.

We’ll see how FCS will fare, and the test deployment is currently running. If you have any good tips or websites for me, i’d be delighted to read them.

I’m using a Diva BRI-2 2 Channel PCI-E Card, which already has support for Windows Server 2008 R2, and installing the Diva Software went without any issues.

Installing the Fax service was also easy, but there was no Fax printer to be seen anywhere.

I’ve followed the TechNet documentation for creating Fax printer on Windows Server 2008 R2, but it didn’t work – at first i received a “Permission denied” error message, after which i started Windows Fax & Scan using Administrator privileges.

This didn’t help that much – i could now go through the wizard, but no Fax account and no printer was created. This seemed strange.

Now, this really seemed like a permission issue. So i disabled UAC, rebooted the server, and tried it again. Everything worked – i was able to create the Fax printer, and after sharing it faxing worked as it should.

So, what now? Why doesn’t this work with UAC? I’ve been running our WS08 servers with UAC disabled (our Vista client were UAC enabled, and so are our Windows 7 clients), and thought WS08R2 should also work well with UAC enabled. But apparently, that wasn’t a good idea.

I have to admit that Failover Clustering isn’t exactly the field i have a lot of experience in (in other words, i have never used it in producation). But after seeing that i wouldn’t be drowning in work this friday afternoon, i decided to give it a whirl.

So, in order to get started i needed two machines that were able to support running Hyper-V Server 2008 R2. One of them was HP ML110 G5, about which i wrote a few months back. Unfortunately, i could use only one of them. So my next choice was an old HP desktop, which fortunately had a VT compatible CPU.

Next, i needed a storage backend. Of course i had to use a software solution, but having no experience and only a very old PIV era IBM SFF PC, i just picked the first Google search result which supported SCSI-3 Reservations, which are required for WS08 clusters. I’ve downloaded and installed Open-E DSS.

For networking, all i was able to find was a 100mbit 3com 24 port hub. Yes, this looked like one of the most ghetto environments i put together yet, but interestingly i got it all to work.

Open-E DSS installs to an USB stick, formatted with FAT32. You just unzip the installation file, run an .exe on the stick to make it bootable, and then you can run the system directly from USB. In my case, using rather outdated hardware, everything was recognized by the Linux kernel. Of course, the machine only having a slow 40GB 5400RPM hard disk wasn’t exactly the fastest on the block, but configuration was surprisingly simple. Unfortunately, installing and activating the Lite license required two reboots, after which it lost all it’s iSCSI settings (but no data!)

Installing Hyper-V Server 2008 R2 on the ML110 was a breeze. Using sconfig, the machine was quickly joined to the domain, remote administration enabled, failover clustering enabled and using the graphical iscsicpl iSCSI was configured, the volumes formatted and attached.

Next was the HP desktop machine. Installing was fine, worked perfectly, all the necessary hardware was recognized. Unfortunately, the machine only had 1GB of RAM, which meant that i couldn’t do all that much fancy stuff with it. I was in for a nasty surprise here later, because i didn’t enable Intel VT in the BIOS (which is hidden in the “Security” Options). I think the Hyper-V Server setup should give you a warning here if the feature isn’t enabled.

Next i created the cluster. I’ve used this blogpost and TechNet to get a basic overview on what i needed to do. In just a few steps through the cluster configuration wizards, my cluster was configured and ready. I was able to bring my VM online on the first node (the ML110) and decided to install Windows XP, since i only had 1GB of RAM on the second node. I gave the VM 256MB of RAM and ran through the setup (which took ages – iSCSI over a 100mbit Hub to an old PIV with a 5400rpm hard drive isn’t a good idea anymore).

Next, i decided to setup VM networking, created the appropriate VM interfaces on both machines, restarted my XP VM and tried to do a live migration. Which failed. “Insufficient system resources”. Turns out i needed to adjust the amount of memory reserved for the root partition using PowerShell – all described in this Clustering and High Availability blog post.

After running (get-cluster HV01).RootMemoryReserved=128, it failed again. This time with these event log entries:

‘Test-VM’ The switch port connection for “Network Adapter” (BE62B93F-1490-4F7E-8229-FA18D50DC974) is invalid.

‘Test-VM’ Microsoft Synthetic Ethernet Port (Instance ID {BE62B93F-1490-4F7E-8229-FA18D50DC974}): Failed to Power on with Error ‘The system cannot find the path specified.’ (0×80070003).

Failed to connect NIC ’9144ED30-35D9-4E5F-8012-70AC436EC603–BE62B93F-1490-4F7E-8229-FA18D50DC974′ to port ” on switch ’0734959D-3′, status = C000003A.

I disabled networking in the VM altogether, and tried Live Migration again. It worked! The next was spent with searching the internet for information about my issue, about which i found nothing. Obviously the network interfaces should be named the same in all cluster hosts, but that was the case. Yet, no matter what i did it didn’t work!

I was starting to doubt my hardware, added a second pair of NICs since the configuration of using the same NIC for everything wasn’t really recommended, but when reading the error message it really didn’t sound like that was my issue. Of course adding the second pair of NICs didn’t help.

So i did what i always did: i started guessing, and after quite a bit of time i got it rights. Turns out you must not use the Hyper-V MMC to manage the VM configuration, and instead the “Settings” button in the failover cluster manager. Only issue is that the failover cluster manager has a much more prominent button labeled “manage virtual machine”, which opened the Hyper-V MMC.

After that, everything worked. I was able to live migrate my machine including the network from host to host. I tested running a Top Gear clips through RDP, while live migrating the machine.

Migrating from the slow HP desktop to the ML110 gave about 2 seconds of video outage, but migrating from the ML110 to the HP desktop just resulted in a slow hiccup. My assumption was that this would probably be completely invisible on more modern hardware.

So what does this mean? Microsoft has made Live Migration and Clustering a feature available to everyone, at (almost) no cost. Administrating such a cluster requires Active Directory, and either a WS08R2 server or a Windows 7 machine with RSAT installed.

This means we can finally have decent virtualization features without paying thousands of francs in licensing fees. I hope this makes it possible to create a few virtualization projects for our customers, which are mostly in the small business range.

Hyper-V Server R2 should be available around mid-August, at which i’ll need to rebuild my Ghetto setup here. I’m of course hoping to get some more cash in order to move or internal virtualization setup from a single-host to a SAN-hosted cluster, but somehow i doubt that will happen quickly.

Update:

I’ve played around with Expression Encoder a bit, and created a Video of a Live Migration. I’ve put the probably most boring video on Youtube – Live Migration of Pinball.

I’m currently running Windows 7 Build 7100 x64 on my laptop, so i decided to uninstall Office 2007 and install Office 2010, the x64 version.

Unfortunately, the x64 version does not support running any x32 plugins. This is unfortunate, since one of the key features of our ERP package is a tight integration with Microsoft Office, which even i used from time to time. We call it the Office Solution Pack, which has been completely rewritten as a native .NET Office 2007 Plugin. The screenshot to the right is slightly outdated, but it clearly shows the integration we had.

I’m already pushing our developers for a x64 version of the plugin, but i suspect it will take a lot of time until all applications we are using (e.G. System i Access) are running on Office 2010 x64.

Also, its important to know that having 32bit plugins around after installing Office x64 will lead to several nasty error messages. You will need to uninstall the 32bit plugins before installing Office x64. Interestingly, the Primary Interop Assemblies come now shipped with Office by default. This alleviates installing them, streamlining the plugin installation process.

This also means that Office 2010 x64 will be a very niche product, and most people will opt to deploy the 32bit version. This isn’t bad – now that an x64 version is around, plugin developers can start adopting and we may see larger deployments of Office 2010.Next x64.

I’m now running my productive mailbox against Office 2010. Yep, might be risky, but it also gives me a very easy way to learn a new product before we’re actually pushing it to customers. I’ve found that i’ve never learned all that well in a separate environment.

So, whats new in Office 2010? Well, the only application i use daily is Outlook. And we will see how it plays out. Outlook 2010 now also comes with the new ribbon interface, and so far i like it. We will see how this plays out.

We’re looking for a developer with knowledge on .NET (C#) and Java for our office in Horgen/ZH.

I’m not that much involved with our development team, but i can tell you that you’ll have current infrastructure to work on – Windows Vista/7, Visual Studio 2008, a current PC with two highres screens, a virtualization environment for testing and of course free coffee and soda.

Official text and contact on the official Website.

One of these runs our TS Gateway Server and our NPS Server for Wireless LAN authentication.

Unfortunately, since the SP2 installation, the NPS service started crashing, taking several other services with him.

Error message is as follows:

Faulting application svchost.exe_IAS, version 6.0.6001.18000, time stamp 0×47919291,
faulting module msvcrt.dll, version 7.0.6002.18005, time stamp 0x49e04189, exception code 0xc0000005,
fault offset 0×0000000000001467, process id 0×1444, application start time 0x01c9d570f76f56bc.

I’ve found one other reference to this issue on the on the TechNet Forums.

I’ve uninstalled SP2 and delayed SP2 deployment until this has been resolved.

But today i experienced the most grave issue with their equipment that critically impacted a customers business.

The customer has two sites – an HQ with an SBS 2008 and a branch office with two Lenovo SFF machines running Windows Vista Business. Both sites are using 20/2 VDSL lines from Swisscom, with ZyXEL P-2802HWL routers.

There is an IPsec VPN configured between these two sites. This has been working fine since January.

Now, about a month ago a telecom service company installed VoIP telephones in the branch office, and enabled QoS on both ZyXEL routers.

Since then, Outlook was unable to synchronize correctly with the SBS server. Unfortunately, the customers personnel isn’t that technically savy, so they weren’t able to tell that they had a problem – because smaller e-mails were able to successfully synchronize, but larger ones failed. This led to very inconsistent states of the OST files, with some mails there and some mails not there.

When i arrived at the branch office i didn’t have a single clue what the issue was or may be. At first i suspected an Outlook problem, so i deleted the OST file. But from there on, nothing happened – Outlook wasn’t able to download anything.

Next, i tried to copy a 50kbyte Excel file from a share to the local computer. This worked. So i tried a 2 megabyte Word file. This failed about halfway through, with Explorer just hanging there and doing nothing. From that point on, i suspected a network issue, but the fact that copying a 50kbyte file worked and a 2 megabyte file didn’t was very odd.

Using Outlook with Outlook Anywhere also worked (when the VPN tunnel was downed).

Whenever i’m confronted with strange network problems, i suspect MTU issues (which was my first “real” network problem i solved back on my first ADSL line – took me weeks for a simple fix). ping -l 5000 CUSTSBS01 worked. ping -l 15000 CUSTSBS01 worked, too. So thought it wasn’t an MTU issue.

Disabling QoS on the ZyXEL router fixed the issue, but made the phones unusuable while Outlook was filling it’s OST files.

So i ran through the usual check points – tcp checksum offloading, chimney, receive window autotuning, reboots, etc. Nothing helped. At the end i was just changing network settings at will. But nothing helped.

Out of any reasonable ideas, i changed the MTU to 1300. That fixed it – with QoS enabled and the NIC MTU of the two machines, everything was working as it should. File transfers worked, Outlook worked, Phones worked.

Don’t buy ZyXEL.

At work, we’ve decided to move several people with a strong technical background over to Windows 7 x64 (if they want, of course). In order to drive internal testing, usage data and generally bring awareness to the whole personnel at the company and also our customers.

By now, i’ve migrated 8 laptops to Windows 7 RC – with which people are working in production and using for their everyday work. Of course in case we run in real troubles with Windows 7, we still have a few spare laptops that run Windows Vista SP2 x32.

The migration has been without any major issues moving from Windows Vista to Windows 7 than when moving from XP to Windows 7, most of this can probably be attributed to the fact that all the applications we use internally are compatible with Windows Vista and we also got a lot of experience with the new deployment model and tools available since Windows Vista.

Still, we ran into a few smaller problems that are mostly un-resolved as of yet, but do not majorly impact anything.

We use Lenovo T60, R61, T61, T500, W500 and R500 laptops. All of these have been running Windows Vista SP1 x32 with BitLocker enabled in TPM+PIN Mode. We installed Windows 7 using Clean (Custom), without formatting the hard drive first – this required us to suspend Bitlocker protection in Windows Vista before running setup. Two devices were reformatted – at the wish of the person using them.

I also upgraded all laptops to 4GB of RAM – which now can actually be used. For example, my W500 with Vista x32 only saw 2.25GB of the 4GB RAM (not a typo – only 2GB).

My biggest issue was that Bitlocker on Windows 7 didn’t properly backup it’s Bitlocker Key and TPM to Active Directory. This is a major issue, as i now had to manually backup the Bitlocker Keys to a secure network share. I didn’t find much about this on the Web, i suspect that not many people used this functionality, and there’s almost no documentation available about Windows 7 Bitlocker. As the workaround of saving the key works just as well, i can live with this.

The fingerprint reader installed on all those Thinkpads has a driver available, but the different drivers have different issues (most of them just crash when using them). I didn’t try installing the Lenovo tools. We don’t use the fingerprint readers, so that’s a non issue for me, but if you do this might require some investigation.

Switchable graphics on the W500 and T500 doesn’t work. Also, the Intel GMA adapter seems to be a lot slower than it was under Windows Vista – so i switched these devices to the internal ATI graphics card. No issues with that, except higher power usage.

WSUS does not contain Windows 7 updates – which makes perfect sense. I created a new WMI filter and a GPO to ensure that Windows 7 got updates directly from Microsoft.

After installing Windows 7 on the devices, all hardware including UMTS modems worked perfectly. Intel AMT doesn’t have Windows 7 drivers yet, but we don’t use that either.

I migrated user data using USMT Hardlink Migration, for which i created a nice batch file using the idea from this feature walkthrough.

I’ll keep you up to date – there’s one more machine considered for migration next week, and after a weeks i’ll have proper feedback from the power users at my office. I’ll even try to persuade our head sales and CEO to try Windows 7, just for the heck of it.

I already did 70-270 (Windows XP) and 70-620 (Windows Vista) two years ago, and the Vista exam was far too easy for my taste. It took me about 20 minutes, and i walked out with a score about 900. That’s not good – too easy questions will just devalue the certification.

With this in mind, i expected 70-680 to get Microsoft back on track, and they did. The exam has much better and much more difficult questions than 70-620. Not questions which require you to memorize stuff, but questions which require you to understand the subject matter.

As usual for beta exams, there were no simulations, VM tasks or anything else except multiple choice questions. I can understand why that’s the case (they probably want to use the final version for that), but i’m still not entirely with this as it is.

One thing that was new in this exam is that you get a questionary that asks you to judge your knowledge levels on Windows 7 for yourself. Several fields are presented, in which you have to choose between very high, high, mediocre, low and very low skills – another questions asks how much experience you already had with Windows 7 (with options such as “Over a year”).

I think that’s a good idea – most exam betas are open now, which means that many less-skilled people will also attend them. As long as those are truthful, this can actually help to improve the exam.

Unfortunately, i had very much difficulty finding what’s my personal baseline. I opted to choose either High or Mediocre for most answers, but was that correct? What does high mean? What does mediocre mean? What’s my knowledge level?

It might make sense to ask questions which are more task oriented – if you already did a task X and if you think if you’re proficient at doing task X.

The exam content was pretty much what was in the official docs – there’s a lot more focus on using group policies (local ones in this case), and also a few more detailed networking questions regarding Subnetting, in both IPv4 and IPv6.

General list of things i’ve seen:

• New features: BranchCache, DirectAccess and VPN (not overly technical – if you got it to work once, you can answer these)
• Bitlocker – not overly many questions
• Setup – the USB stick install gets featured more
• USMT gets a lot more focus and also Windows EasyTransfer
• Imaging, Deployment, VHDs

I’ll see if i passed the exam in officially 8 weeks, so probably in about 4 real moths ;)

So today i decided the install the German language pack on my home PC and on my laptop – on the home PC, this worked as expected. On my laptop, which has it’s hard drive encrypted and protected by BitLocker in TPM mode.

After the obligatory reboot, i changed the system language. The machine rebooted and then asked for my Bitlocker recovery password – in German. It was obvious what happened: On German Vista machines with Bitlocker enabled, the Windows Boot Manager was still in English, but on Windows 7 the boot manager was also translated – which means that it now failed the integrity check because it was modified.

Luckily i could use our Terminal Services Gateway to log onto my administrative terminal server, where i had the BitLocker Recovery Password Viewer installed, so viewing my recovery key was quick and easy.

After booting into my now (mostly) German Windows 7, i temporarily halted Bitlocker protection, and immediately reenabled. This caused Windows 7 to reverify the state of the Boot Manager, and after i another reboot i was sure that everything was fine.

Oh, and this is one of the rather funny translation episodes: The window is not resizeable and the text doesn’t fit.

First of all, i had Bitlocker enabled on my ThinkPad W500, which was running Windows Vista x32 and i intended to install Windows 7 x64. So a direct inplace upgrade was out of the question. I created a backup of the machine, disabled Bitlocker, upgraded my laptops BIOS to the latest version, and booted Windows 7 setup from an USB stick.

Next, i pressed Shift-F10 on the setup screen, deleted all the Windows and Program Files folders, and then started an installation directly on the Bitlocker-enabled drive (this way, i didn’t have to restore all the files i already had on the drive, saving me valuable time).

Windows 7 was done after about 25 minutes, and greeted me with Aero enabled and the 1920×1200 15″ screen already set to a scaling factor of 125%. This is were i also noticed that DPI settings are now user dependant, instead of affecting the whole system. An extremely nice feature, that probably needed quite a bit of work. I set the scaling factor to 115%, which is the best factor between readability and remaining screen real estate for me.

Unfortunately, the switchable graphics driver available from Lenovo did not support WDDM 1.1. I went into the BIOS and configured the machine to always use the Intel graphics. However, i noticed that unlike in Vista, the Intel graphic card did not produce 100% smooth Aero animations. Since i have the power supply connected most of the time anyway, i configured the system to always use the ATI card. This produced better results.

The fingerprint reader does not work yet, but i didn’t invest time in that since i don’t use it anyway. Also, there are issues with Intel AMT, which i don’t use either.

So the base OS worked flawlessly after install. Even switching the graphics card around didn’t phase it, Aero was automatically enabled and the correct resolution configured. WLAN, Audio, everything you would need worked out of the Box.

I joined the machine to the domain, where it sucked down all the GPOs for our corporate network. I unplugged the network cable, and it automatically connected to the corporate wireless network, authenticated by EAP-TLS.

Since our printserver is WS08 x64 box, corporate printing also worked automatically, without any additional work. Of course, all the other group policy settings applied as they should, and i didn’t find any issues yet regarding policy settings.

But an OS alone doesn’t serve a purpose, you need applications. I’ve installed the following applications:

• Adobe Reader 9.1 Works perfectly.
• DIAS-iS Network Client 3.2 Works perfectly.
• DIAS-iS OSP Version 3 for Office 2007 Works perfectly.
• Office 2007 SP1 Enterprise, Visio and PDF/XPS plugin Works perfectly.
• Office 2007 Primary Interop Assemblies Works perfectly.
• Office 2007 VSTO 3.0 Works perfectly.
• Office 2007 Communicator R1 with latest Hotfix Works perfectly.
• Solitas InfoStore Windows Retrieval Works perfectly.
• IBM System i Access V6R1M0 x64 Works perfectly.
• IrfanView Works perfectly.
• Mozilla Firefox 3.1b3 Works perfectly.
• PuTTY 0.60 Works perfectly.
• SonicWALL Global VPN Client x64 Sometimes loses it’s IPsec driver – repairing the program helps.
• Windows Live Messenger Works perfectly.
• Virtual CloneDrive Works perfectly.
• WinRAR Works perfectly.
• tn5250 Works perfectly.

So far, so good. The SonicWALL issue may be annoying, but it’s not a dealbreaker. Judging from my experience, it’s a SonicWALL issue. Opening a bug there won’t help, as they don’t support Windows 7 yet. I can live with that.

Perfomance on Windows 7 on this machine is even better than Vista. I can now fully use the 4GB RAM installed in my laptop. Never used Windows XP on this machine, i can’t compare performance. All the business apps i need to do my job work flawlessly. Printing works flawlessly.

Windows 7 is even better than Vista. But for those that didn’t spend the last three years using Windows Vista, it may be rather hard to get used to all the new stuff. For example, the deployment options between 7 and Vista are both based on WIM imaging, with a few improvements here and there. If you know how to do it on Vista, you can also do it in Windows 7.

As a bonus, the score to the right from my desktop PC.

First, lets talk about the hardware. It’s important to know that small businesses handle their infrastructure completely differently than large businesses, and in my opinion there are some things that require “unusual” thinking.

Reliability: an SBS server is extremely critical for operation of a small business and they usually do not replace servers after three years

Maintainability: small businesses do not have dedicated IT personnel. Usually, most “heavy” tasks are done by an IT service provider, and the daily IT tasks are done inhouse by someone as a secondary job

Functionality: small businesses are sometimes just as demanding as larger companies – the small size requires setups that maximize the productivity of each employee

So, there are three main aspects one should focus on when deploying an SBS server.

Reliablity is a key aspect. An SBS server is critical for the business and this requires hardware that is highly capable and reliable. After all, an SBS server can serve a Business for up to five years without replacing the hardware. This is why it makes sense to buy really good hardware that lasts that long, combined with appropriate maintenance contracts to get it back up in case you run into problems.

In this case, we decided to use the following hardware:

IBM System x3500
Intel Quadcore 2.66 Ghz 12M (leaving 1 slot available)
10 GB Memory (leaving 6 slots available)
8 2.5″ 147GB 10kRPM SAS Disks (leaving 4 slots available)
ServeRAID 8k with 256MB BBWC (for the first 8 disks)
ServeRAID 8s with 256MB BBWC (for the other 4 disks)
Disk configuration:
RAID 1 consisting of two 147GB Disks
RAID 5 consisting of five 147GB Disks
Global Hotspare
IBM SAS HBA (for tapedrive)
LTO4 SAS attached HH internal tapedrive
Redundant fans & power
IBM Remote Supervisor Adapter II
5Y of IBM ServicePac with committed service option

As you can see, the system has lots of storage and redundancy. It’s also important to know that the SBS server does not run any third party applications (except those necessary for operation), the ERP runs on an IBM POWER machine. With SBS 2008, i would not recommend running any third party applications on the SBS itself – if necessary to run third party apps on a server, purchase SBS premium and run the third party apps on the second server.

The second aspect is ease of use, for which we should use software that can be automated as well as possible. SBS handles lots of things on his own, but we opted for a third party backup application because we still see tape backups as the best way to fulfill most of a customers needs. Especially since LTO drives have WORM media, that can help to comply with certain local laws.

As for the software, i’ve installed BackupExec 12.5 to handle the backups to tape. Tape backups are easier to handle for customers, offer superior performance, and make archival and external storage of data easy. Unfortunately, BackupExec 12.5 does not integrate with the SBS Console (yet?).

For virus scan, we’ve opted for McAfee VirusScan Enterprise. A central management application was unfortunately not yet available, so we deployed McAfee manually on each client, and on the server.

Exchange is protected using ForeFront for Exchange, which has served me well in the past.

SBS 2008 has a nice reporting function, but there are other important notifications: the RSA Adapter notifies for all hardware failures like power supplies, fans, etc. independently through e-mail (which can contact external adresses and even works if the failure killed the server), and allows remote troubleshooting in case the machine does not boot. ServeRAID manager and BackupExec also send daily reports to be viewed by the customer.

Last, but not least, is functionality. After all, customers aren’t like me that want an SBS because they like technology – no, they want an SBS to fulfill certain needs their business has.

In this case, there were several unique requirements regarding mailflow – thanks to the included Exchange 2007 server, which offers a very flexible transport rule system i was able to implement these requirements without having to purchase third party software or even program event sinks on our own.

So far, i’ve had zero issues with SBS 2008 – it worked without any problems and hardware support wasn’t a problem either. Looks like IBM got all the kinks figured out since the release of Windows Server 2008 at the beginning of the year.

The System x3500 is also very nice hardware – it looks like a tower variant of the x3650, which i also like very much. The only criticism i have for the machine is that installing the redundant fan kit is total pain in the ass, mostly because the documentation covers both the x3400 and x3500, and some parts don’t apply for the x3500 (but aren’t marked as such).

It shows that IBM can still deliver top notch hardware at affordable prices.

In case you couldn’t figure it out from reading this far, i really like SBS 2008, and it’s ready for action in a production environment. In case you’re thinking about deploying a new SBS, go with SBS 2008!

About now, we’ve been running on this Infrastructure for 6 months. While configuring back then was very interesting (especially Exchange 2007) and finding vendors which supported their apps under WS08 wasn’t always easy, it worked out.

We’re running McAfee VirusScan Enterprise, which was supported WS08. Unfortunately, the ProtectionPilot Management App was not supported on WS08, which is why it’s running in an WS03 x32 VM. For Backup, we’ve used Symantec BackupExec 12 (since then, upgraded to 12.5).

I’ve been running six productive VMs in Hyper-V since May. The upgrades to the RTM version of Hyper-V ran flawlessly, and we’ve had zero production issues with those VMs. The VMs are a mix of WS03 x32 and WS08 x64.

Except one WS08 Core x32 Domain Controller, all WS08 machines are x64. Even the setting up an x64 print server for x32 clients was less of an issue than i initially thought.

The feature most applauded by our users is probably the TS Gateway.

We currently OCS 2007 in an (unsupported) VM, because we only use the IM functionality right now (the reason that VMs are unsupported is that voice heavily depends on timing, which can be icky in VMs). Our plan is to migrate to OCS 2007 R2 when it comes out, this time running on WS08 on native hardware, so we can start our internal VoIP rollout.

IBM has finally released Director 6.1, which supports running on WS08 x64.

For Active Directory, i run two WS 08 Core DCs, one x64 (on newer hardware) and one x32 (on rather old hardware). We also have an RODC in our branch office. BackupExec has it’s fair share of troubles running on RODCs and so do other apps that depend on SQL Server, like WSUS. So keep this in mind if you want to deploy branch offices – the single server approach worked with DCs, but it won’t with RODCs. Get two machines, one for the RODC, another one for the rest.

For branch office connectivity, we’ve always used DFS-N and DFS-R, which has continued to work flawlessly on WS08.

In our Edge environment, i’ve deployed an Exchange 2007 Edge Server, an OCS 2007 Edge Server and an ISA 2006 server. The latter two are still running on WS03, which i plan to upgrade as soon as it is possible.

I currently only have one unresolved issue, which is NTLM Authentication for Outlook Anywhere. UR4 should have resolved it, but i haven’t gotten around to test this.

As for the clients: We run three quarters Vista, one quarter XP. The XP machines only remain because i don’t have any jurisdiction over them, there are no technical reasons why they shouldn’t get upgraded.

So, after this you will probably assume that i got paid to write this. Well, i do work for a Microsoft Partner, so the Software cost associated with upgrading to WS08 was rather low, as we have Software Assurances for our Volume Licenses and we also get many internal use licenses through the MSPP.

The experience of deploying and running a production system has been a tremendous help for me to get acquainted with WS08 as a platform. I’m currently in the process of deploying my first SBS08 into production, about which i’ll write as soon as that project is done.

Still, i honestly believe that WS08 is ready to deployed. Not anywhere, mind you. Application Support is still an issue, and especially ERP vendors are slow to catch up (not us, though – we supported WS08 TS as platform from the start).

So, what do you think about WS08? Looked at it? Tried it? Running it?

The reason for this failure is that it expects the WSUS groups to have English names – until this is fixed, there is an easy workaround to install the update:

Rename the group “WSUS-Administratoren” to “WSUS Administrators”
Rename the group “WSUS-Berichterstatter” to “WSUS Reporters”

This will allow the update to install. It might make sense to rename the groups back to their original names, in case some other script depends on their names.

Another reason why running non-US versions of Windows is a bad idea in production environments – you just get additional trouble with zero advantages – but that’s for another post.

I’ve ran a test environment, mostly to play with Citrix XenApp, SBS 2008 and EBS 2008. Especially the latter, as multi server solution, could be run with multiple roles distributed on Hyper-V and ESXi hosts, which made it interesting to deal with.

Hyper-V Server management

Well, i’ll be blunt. Hyper-V Server standalone management sucks. Big time. The problem here is that you need a Vista machine (which is a problem in the testlab which mostly consists of older PCs), and then you’ll need to create same-username same-password accounts to connect the two.

Also, if you just want to delegate specific VMs, you’ll need to dive into the depths of WMI.

Many of those problems vanish when you’re using a domain setup, with Hyper-V Server joined to the domain. But that’s usually not the case in a test lab.

Add to that that Server Core is still very young, and a lot of 3rd party hardware manufacturers do not have anything Server Core ready yet. Many don’t even announce whether certain tools are supported on Server Core or not.

This doesn’t mean Server Core is a bad idea – it just means that we will need to wait for hardware manufacturers to catch up – also management agents like Backup, Anti Virus, etc. will need to get up to speed for running on Server Core.

ESXi management

ESXi works very well with standalone management. You go to the website, download the VI Client, enter user and password, and you can manage the VM host – you can also delegate permissions easily.

From what i’ve read so far, using single signon and Active Directory with ESXi is more cumbersome than on Hyper-V server. Makes sense.

ESXi integrates a lot of hardware drivers and management. However, fewer servers are supported than with Hyper-V Server. On the other hand, you can use the native ESXi tools to create teams etc., something which you can’t on Server Core because the HW manufacturers do not ship the tools for Server Core.

My opinion

ESXi works very well in a standalone lab environment. Hyper-V is lackluster at best. What does that mean for you? If you want to build a test lab, go with hardware that supports ESXi, or if that is too expensive, go with hardware that at least works with ESXi.