Lukas Beeler's IT Blog

I’m still running a Linux box to run a legacy business app that’s about to be replaced, and runs a few legacy VPNs. Setup ages ago, when i didn’t have the experience i have today, the setup on the machine was a mess – originally installed using testing of what was-to-be Debian 3.1 with several custom packages (Postfix, Apache, OpenVPN, etc.), this has been overdue for some fixup work for quite some time.

As a disclaimer, i realize that Debian in any version isn’t a supported OS on Hyper-V R2 – i just want to tell of my experiences with this unsupported configuration.

The hardware, an aging IBM xSeries 306m with a Pentium 4 CPU wasn’t getting any younger and after a drive failure about half a year ago that lead to a system crash (No data loss though – it just crashed the machine, that’s Software RAID for you), it was finally time to modernize this.

The plan is to consolidate all our DMZ workloads (ISA, OCS Edge, XMPP Gateway, Exchange Edge) on Hyper-V 2008 R2 and doing the trickiest part first seemed like a good idea.

So i created a new VM using SCVMM 2008 R2, selected Other Linux 32bit as the guest OS, inserted a Debian 5.0 netboot CD and that’s where the problems already started. While the installation worked well in general, the Framebuffer used by the Debian installed is awfully slow. So it took me about half an hour just to get the install done (on a 5GB partition of the 80GB VHD).

After finishing the installation, i formatted the rest of the disk appropiately and then used rsync to transfer the machine contents over. A short bit after reconfiguring Grub, i could choose to boot either the transferred OS with it’s kernel, or the Debian 5 rescue system i installed alongside.

Booting the transferred system worked well enough, but the tulip driver wasn’t compiled into that (custom) kernel and building the module failed. So i read up a bit, and realized that the newest kernel (2.6.32.8) shipped with experimental Hyper-V VMbus drivers, that allowed synthetic NICs to be used.

I tried to compile the kernel after chrooting into the old installation, but it failed because gcc was too old. Not to worry, i compiled it in the rescue system, but couldn’t install the dpkg that make-kpkg created. So i installed it manually, which worked pretty well.

One reboot later, i was back in business with the extremely verbose Hyper-V drivers cluttering up dmesg, but the Synthetic NICs showed up as seth0 – seth2. After quickly changing all the necessary configuration files, everything was working.

After a bit of more testing, i disconnected the physical machine from the network and plugged the VM into the production VLANs.

I tested everything thoroughly and didn’t find any issues. Sent out an information mail and continued on my merry way.

Half an hour later, i decided to do a quick systems check again – and i realized that the external interface (seth2 in this case) wasn’t working anymore. tcpdump showed no packets being received and other machines in the same VLANs didn’t see any answers to their ARP requests either. So i rebooted the VM, and everything was working again. No error messages of any kind, neither in dmesg nor in the system logs or on the Hyper-V host.

Hoping this was just a fluke, i waited until it happened again – which it did, roughly 10 minutes later. So i decided to skip on the synthetic devices and go with emulated NICs and the tulip driver.

Everything came back up, but i couldn’t ping any devices on the eth0 VLAN from the start, but the other two interfaces worked.

After a few more tries, i arrived at a configuration that has now been stable for 4 hours and 26 minutes, which sounds good so far. For this, i configured a single synthetic NIC that i used as a replacement for the non-working eth0 and three tulip NICs (of which the first was unused).

There are other things that also worry me:

Every reboot of the Linux machine created the following event log entry on the Hyper-V host:

'LINUX' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID [])

Loading the synthetic NIC drivers logs the following in the event log on the Hyper-V host:

Networking driver on 'LINUX' loaded but has a different version from the server. Server version 3.2 Client version 0.2 (Virtual machine ID []). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.

Loading the synthetic NIC drivers also logs all this on the Linux side of things:

VMBUS_DRV: Vmbus initializing.... current log level 0x1f1f0006 (1f1f,6)
VMBUS: +++++++ Build Date=Feb 17 2010 12:37:00 +++++++
VMBUS: +++++++ Build Description=Version 2.0 +++++++
VMBUS: +++++++ Vmbus supported version = 13 +++++++
VMBUS: +++++++ Vmbus using SINT 2 +++++++
VMBUS: Windows hypervisor detected! Retrieving more info...
VMBUS: Vendor ID: Microsoft Hv
VMBUS: Interface ID: Hv#1
VMBUS: OS Build:7600-6.1-16-0.16485
VMBUS: Hypercall page VA=f80c9000, PA=0x36afe000
VMBUS_DRV: irq 0x5 vector 0x35
VMBUS: SynIC version: 1
VMBUS: Vmbus connected!!
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={c5295816-f63a-4d5f-8d1a4daf999ca185}
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
hv_netvsc: module is from the staging directory, the quality is unknown, you have been warned.
NETVSC_DRV: Netvsc initializing....
VMBUS_DRV: child driver (f80dc570) registering - name netvsc
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={32412632-86cb-44a2-9b5c50d1417354f5}
VMBUS_DRV: child device (f73a8634) registered
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a}
VMBUS_DRV: child device (f73a5a34) registered
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={f8615163-df3e-46c5-913ff2d2f965ed0e}
VMBUS_DRV: device object (f73a5ee4) set to driver object (f80dc5c0)
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS: channel f73aac00 open success!!
NETVSC: *** NetVSC channel opened successfully! ***
NETVSC: Sending NvspMessageTypeInit...
NETVSC: NvspMessageTypeInit status(1) max mdl chain (34)
NETVSC: Sending NvspMessage1TypeSendNdisVersion...
NETVSC: Establishing receive buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendReceiveBuffer...
NETVSC: Receive sections info (count 1, offset 0, endoffset 1048000, suballoc size 1600, num suballocs 655)
NETVSC: Establishing send buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendSendBuffer...
NETVSC: *** NetVSC channel handshake result - 0 ***
NETVSC: Device 0xf6552e80 mac addr 00155d031a09
NETVSC: Device 0xf6552e80 link state up
VMBUS_DRV: child device (f73a5e34) registered

So, it works. But not without troubles. I’ve still got the physical machine to fall back on, but i sure hope Microsoft will get this to work better.

These issues are the reason why i decided to deploy my private server using ESXi instead of Hyper-V – because i need both Linux and Windows guests.

Exchange 2010 was released last Monday, the 9th. Today, we have Saturday the 14th – and i’m done with the Migration to Exchange 2010.

Sure, there are loads of MVPs and TAP-Members that have migrated to Exchange 2010 a long time ago, but i’m still proud of this.

At a starting point, i had a Exchange 2007 SP2 machine, with one Mailbox database, no public folders and 35 Mailboxes that used up 25GB of space. Moving this is simple enough, but the issue is that our Exchange isn’t virtualized, and i couldn’t get my hands on new hardware since the current box was only a year old.

Since in-place upgrades are not supported, i needed a temporary server for the migration. I used an HP ML110 from the Lab, which offered enough space to migrate.

Another issue was BackupExec 12.5, which did not support Exchange 2010 yet. Fortunately, Exchange 2010 (and 2007 SP2) can be backed up by using Windows Server Backup. So my goal was to just let WSB backup to a file server, and have BackupExec pickup the files from there. This way, i will get a reliable, clean and supported Exchange backup, and still have it on tape.

To Migration itself was straightforward and easy. There’s already _lots_ of content on the web about Exchange 2010, most of it from the RCs or Beta of course.

I followed the Migration Guide from TechNet, which worked out well enough. Unfortunately, the iPhone does not support Exchange 2010/2007 coexistence, which made it necessary for several people to manually reconfigure their phone.

Removing Exchange 2007 worked without issues, but after moving all the Exchange 2010 data back to the real hardware and removing the temporary server i ran into the issue of moving arbitration mailboxes, which fortunately was already documented widely on the web.

In the end, upgrading from Exchange 2007 to 2010 while keeping the same hardware is not difficult, it just needs a bit more time.

The iPhone does not properly support coexistence between Exchange 2010/Exchange 2007. See this TechNet Posting.

The error message in the IIS Log looks like this:

RdirTo:https%3a%2f%2flegacy.contoso.com%2fMicrosoft-Server-ActiveSync_LdapC2_LdapL15_Error:MisconfiguredDevice_Budget

SBS 2008 is out for roughly a year. In this time, i did four deployments of SBS 2008, each with 15-30 users.

During this time, i’ve gained valuable experience, which i’ll try to share here so that others can profit from it. Take all this with a grain of salt, as some observations may simply be my fault. Also, as times changes these things might change too.

Software

• Make sure to install Windows Server 2008 SP2 after installing SBS 2008. Some media may come with SP2 already preloaded. You can use the normal SP2 package that’s also used for Vista and the normal Server 2008
• Do not install SBS rollup updates before completing the configuration wizard. This is extremely counter-intuitive, but is described on the Official SBS blog
• Installing Exchange 2007 SP2 requires you to follow special considerations Here
• Installing WSUS 3.0 SP2, which is needed to support Windows 7, is currently not recommended. I was able to do this without issues on my lab machines, but others have reported issues doing this on machines that were in production. If you’re deploying a new SBS server, this should probably be safe to go. But make sure to test functionality afterward.
• Always use the answer file to deploy SBS 2008. This will make it possible to choose a custom domain name. Read my post about choosing your AD DNS namespace
• Do whatever tasks you can do using the SBS console. Resist of using the normal administration tools as much as possible, as you can break SBS with them easily.
• Ensure that the AV software you install is compatible with WS08 x64. Symantec Endpoint Protection Manager works well – Forefront Client Security on the other hand requires a seperate server running 32bit Windows for management. You may consider deploying FCS unmanaged in smaller environments, and configure FCS using the FCS ADM File

Hardware

• Use servers with the new Xeon 5500 CPUs. Read my x3650 M2 tips to find more about them. Consider using an E5530 or faster CPU. Using two CPUs (for a total of 16 virtual and 8 physical cores) makes little sense.
• Buy enough memory. Lots of it. Really. I mean it. You’ll need lots and lots of memory. I would consider 12GB to bare minimum. In a 3×4GB configuration which makes the most sense for the Xeon 5500 setups, this is quite cheap. Consider more memory if you intend to run SQL Server as, consider bumping the memory to 24GB. Remember that you can only use the first 8 slots in a single socket machine.
• Buy enough disks. A good starting layout is 8×147GB 2.5″ disks. Use a RAID 1 for the OS, another RAID1 for Exchange and Sharepoint, and a RAID10 for Data and WSUS. This is all up for debate of course, and it might make sense to consider other disk layouts.

If you have any additions, think i’m wrong somewhere just send in a comment.

Update: See here for Microsoft’s description of this issue KB974571

Security updates are important. And as we’re currently an evaluation setup for OCS 2007 R2, i’ve decided to install todays batch of security updates on these lesser important machines first. And after a reboot, OCS 2007 R2 was broken.

A quick view into the event log revealed that OCS 2007 R2’s evaluation license has expired. Now, this seemed very strange as i’ve installed from volume license media. I’ve the checked the media again, but they weren’t evaluation media.

Here’s the message in all it’s glory:

Event source: OCS Server
Event id: 12290
Event text: The evaluation period for Microsoft Office Communications Server 2007 R2 has expired. Please upgrade from the evaluation version to the full released version of the product.

Maybe i really did use other media to install it? I doubted myself, because that’s usually the most reasonable approach to take. The error is usually behind the keyboard.

Luckily, Microsoft has published documentation on how to upgrade an evaluation version to a full version. Unfortunately, this didn’t work, because as it appears i was running a Volume license version of OCS.

EVALTOFULL parameter cannot be used with currently installed license type Volume

At this point, i was pretty sure that this wasn’t my fault. There has been an issue with the OCS 2007 R2 Evaluation Media expiring at the wrong point in time, but apparently this has been sorted out and did never affect the full versions of OCS 2007 R2.

So i was bummed. A quick view using process monitor revealed that the licensing information was most likely to be stored here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RtcSrv\InstallInfo\ValidationData

I created a backup of that part of the registry, and then renamed the key. I got a file not found error, and created a new key of the same type and wrote binary data of the same length into it. This yielded the following error:

The service is shutting down due to an internal error.

Error Code: 80093102 (ASN1 unexpected end of data.)

At that point, i was pretty sure what might’ve caused this – the MS Crypto API security update KB974571.

I removed the update, rebooted the machine, and OCS 2007 R2 was up and running again, without any issues.

I’ve already opened a case with Microsoft to get this sorted out.

Update:
Appears that this is an official issue: See here

Microsoft recently released Microsoft Security Essentials, which is a free AV solution for anyone running genuine Windows. This is great news, because most other free AV packages exclude commercial use – like Avira or AVG. Especially in the SMB space, were you to argue for every license, this is a good way to ensure that _every_ machine is running an AV package, even without central reporting.

Update: The license is not entirely clear – it does not exclude commercial use, only SaaS use. But in the beginning of the license it says that only home-based small businesses are allowed to use it. So take this with a grain of salt – the license is certainly more permissive than Avira, but it’s not as easy as i thought.

I’ve been participating in the MSE beta test with my Windows 7 machines at home, and my impression has been very good. Performance is excellent, and the GUI is simple and straightforward.

After a few negative experiences with McAfee 8.7 at work, and my very good experiences with MSE at home, i tried to have another look at FCS.

Well, FCS is rather old right now, with the new release just on the horizon. Still, the current release is supported on Windows 7 x64 with the latest patches, and so far my impressions have been very good. The management server only runs on 32bit Windows, which also means it won’t run on WS08R2.

But my impression has been good so far – the package installed on the client is far more lightweight than McAfee, and even the managment software leaves a much better impression.

We’ve also been using Symantec Endpoint Protection at a few customers, but my impression of that product was even worse than McAfee.

We’ll see how FCS will fare, and the test deployment is currently running. If you have any good tips or websites for me, i’d be delighted to read them.

Windows Server 2008 R2 was released to MSDN today, and of course i want and installed it on a machine that did something more or less useful – a Fax server. Which is of course an internal system and not really in production.

I’m using a Diva BRI-2 2 Channel PCI-E Card, which already has support for Windows Server 2008 R2, and installing the Diva Software went without any issues.

Installing the Fax service was also easy, but there was no Fax printer to be seen anywhere.

I’ve followed the TechNet documentation for creating Fax printer on Windows Server 2008 R2, but it didn’t work – at first i received a “Permission denied” error message, after which i started Windows Fax & Scan using Administrator privileges.

This didn’t help that much – i could now go through the wizard, but no Fax account and no printer was created. This seemed strange.

Now, this really seemed like a permission issue. So i disabled UAC, rebooted the server, and tried it again. Everything worked – i was able to create the Fax printer, and after sharing it faxing worked as it should.

So, what now? Why doesn’t this work with UAC? I’ve been running our WS08 servers with UAC disabled (our Vista client were UAC enabled, and so are our Windows 7 clients), and thought WS08R2 should also work well with UAC enabled. But apparently, that wasn’t a good idea.

In the light of the recent announcement how Hyper-V Server 2008 R2 will be licensed, i thought about familiarizing myself with the Clustering & Live Migration capabilities, using the RC release of Hyper-V Server 2008 R2.

I have to admit that Failover Clustering isn’t exactly the field i have a lot of experience in (in other words, i have never used it in producation). But after seeing that i wouldn’t be drowning in work this friday afternoon, i decided to give it a whirl.

So, in order to get started i needed two machines that were able to support running Hyper-V Server 2008 R2. One of them was HP ML110 G5, about which i wrote a few months back. Unfortunately, i could use only one of them. So my next choice was an old HP desktop, which fortunately had a VT compatible CPU.

Next, i needed a storage backend. Of course i had to use a software solution, but having no experience and only a very old PIV era IBM SFF PC, i just picked the first Google search result which supported SCSI-3 Reservations, which are required for WS08 clusters. I’ve downloaded and installed Open-E DSS.

For networking, all i was able to find was a 100mbit 3com 24 port hub. Yes, this looked like one of the most ghetto environments i put together yet, but interestingly i got it all to work.

Open-E DSS installs to an USB stick, formatted with FAT32. You just unzip the installation file, run an .exe on the stick to make it bootable, and then you can run the system directly from USB. In my case, using rather outdated hardware, everything was recognized by the Linux kernel. Of course, the machine only having a slow 40GB 5400RPM hard disk wasn’t exactly the fastest on the block, but configuration was surprisingly simple. Unfortunately, installing and activating the Lite license required two reboots, after which it lost all it’s iSCSI settings (but no data!)

Installing Hyper-V Server 2008 R2 on the ML110 was a breeze. Using sconfig, the machine was quickly joined to the domain, remote administration enabled, failover clustering enabled and using the graphical iscsicpl iSCSI was configured, the volumes formatted and attached.

Next was the HP desktop machine. Installing was fine, worked perfectly, all the necessary hardware was recognized. Unfortunately, the machine only had 1GB of RAM, which meant that i couldn’t do all that much fancy stuff with it. I was in for a nasty surprise here later, because i didn’t enable Intel VT in the BIOS (which is hidden in the “Security” Options). I think the Hyper-V Server setup should give you a warning here if the feature isn’t enabled.

Next i created the cluster. I’ve used this blogpost and TechNet to get a basic overview on what i needed to do. In just a few steps through the cluster configuration wizards, my cluster was configured and ready. I was able to bring my VM online on the first node (the ML110) and decided to install Windows XP, since i only had 1GB of RAM on the second node. I gave the VM 256MB of RAM and ran through the setup (which took ages – iSCSI over a 100mbit Hub to an old PIV with a 5400rpm hard drive isn’t a good idea anymore).

Next, i decided to setup VM networking, created the appropriate VM interfaces on both machines, restarted my XP VM and tried to do a live migration. Which failed. “Insufficient system resources”. Turns out i needed to adjust the amount of memory reserved for the root partition using PowerShell – all described in this Clustering and High Availability blog post.

After running (get-cluster HV01).RootMemoryReserved=128, it failed again. This time with these event log entries:

‘Test-VM’ The switch port connection for “Network Adapter” (BE62B93F-1490-4F7E-8229-FA18D50DC974) is invalid.

‘Test-VM’ Microsoft Synthetic Ethernet Port (Instance ID {BE62B93F-1490-4F7E-8229-FA18D50DC974}): Failed to Power on with Error ‘The system cannot find the path specified.’ (0×80070003).

Failed to connect NIC ‘9144ED30-35D9-4E5F-8012-70AC436EC603–BE62B93F-1490-4F7E-8229-FA18D50DC974′ to port ” on switch ‘0734959D-3′, status = C000003A.

I disabled networking in the VM altogether, and tried Live Migration again. It worked! The next was spent with searching the internet for information about my issue, about which i found nothing. Obviously the network interfaces should be named the same in all cluster hosts, but that was the case. Yet, no matter what i did it didn’t work!

I was starting to doubt my hardware, added a second pair of NICs since the configuration of using the same NIC for everything wasn’t really recommended, but when reading the error message it really didn’t sound like that was my issue. Of course adding the second pair of NICs didn’t help.

So i did what i always did: i started guessing, and after quite a bit of time i got it rights. Turns out you must not use the Hyper-V MMC to manage the VM configuration, and instead the “Settings” button in the failover cluster manager. Only issue is that the failover cluster manager has a much more prominent button labeled “manage virtual machine”, which opened the Hyper-V MMC.

After that, everything worked. I was able to live migrate my machine including the network from host to host. I tested running a Top Gear clips through RDP, while live migrating the machine.

Migrating from the slow HP desktop to the ML110 gave about 2 seconds of video outage, but migrating from the ML110 to the HP desktop just resulted in a slow hiccup. My assumption was that this would probably be completely invisible on more modern hardware.

So what does this mean? Microsoft has made Live Migration and Clustering a feature available to everyone, at (almost) no cost. Administrating such a cluster requires Active Directory, and either a WS08R2 server or a Windows 7 machine with RSAT installed.

This means we can finally have decent virtualization features without paying thousands of francs in licensing fees. I hope this makes it possible to create a few virtualization projects for our customers, which are mostly in the small business range.

Hyper-V Server R2 should be available around mid-August, at which i’ll need to rebuild my Ghetto setup here. I’m of course hoping to get some more cash in order to move or internal virtualization setup from a single-host to a SAN-hosted cluster, but somehow i doubt that will happen quickly.

Update:

I’ve played around with Expression Encoder a bit, and created a Video of a Live Migration. I’ve put the probably most boring video on Youtube – Live Migration of Pinball.

Finally, Office 2010 is now available officially.

I’m currently running Windows 7 Build 7100 x64 on my laptop, so i decided to uninstall Office 2007 and install Office 2010, the x64 version.

Unfortunately, the x64 version does not support running any x32 plugins. This is unfortunate, since one of the key features of our ERP package is a tight integration with Microsoft Office, which even i used from time to time. We call it the Office Solution Pack, which has been completely rewritten as a native .NET Office 2007 Plugin. The screenshot to the right is slightly outdated, but it clearly shows the integration we had.

I’m already pushing our developers for a x64 version of the plugin, but i suspect it will take a lot of time until all applications we are using (e.G. System i Access) are running on Office 2010 x64.

Also, its important to know that having 32bit plugins around after installing Office x64 will lead to several nasty error messages. You will need to uninstall the 32bit plugins before installing Office x64. Interestingly, the Primary Interop Assemblies come now shipped with Office by default. This alleviates installing them, streamlining the plugin installation process.

This also means that Office 2010 x64 will be a very niche product, and most people will opt to deploy the 32bit version. This isn’t bad – now that an x64 version is around, plugin developers can start adopting and we may see larger deployments of Office 2010.Next x64.

I’m now running my productive mailbox against Office 2010. Yep, might be risky, but it also gives me a very easy way to learn a new product before we’re actually pushing it to customers. I’ve found that i’ve never learned all that well in a separate environment.

So, whats new in Office 2010? Well, the only application i use daily is Outlook. And we will see how it plays out. Outlook 2010 now also comes with the new ribbon interface, and so far i like it. We will see how this plays out.

My current employer Acommit AG is hiring.

We’re looking for a developer with knowledge on .NET (C#) and Java for our office in Horgen/ZH.

I’m not that much involved with our development team, but i can tell you that you’ll have current infrastructure to work on – Windows Vista/7, Visual Studio 2008, a current PC with two highres screens, a virtualization environment for testing and of course free coffee and soda.

Official text and contact on the official Website.