Lukas Beeler's IT Blog

In the light of the recent announcement how Hyper-V Server 2008 R2 will be licensed, i thought about familiarizing myself with the Clustering & Live Migration capabilities, using the RC release of Hyper-V Server 2008 R2.

I have to admit that Failover Clustering isn’t exactly the field i have a lot of experience in (in other words, i have never used it in producation). But after seeing that i wouldn’t be drowning in work this friday afternoon, i decided to give it a whirl.

So, in order to get started i needed two machines that were able to support running Hyper-V Server 2008 R2. One of them was HP ML110 G5, about which i wrote a few months back. Unfortunately, i could use only one of them. So my next choice was an old HP desktop, which fortunately had a VT compatible CPU.

Next, i needed a storage backend. Of course i had to use a software solution, but having no experience and only a very old PIV era IBM SFF PC, i just picked the first Google search result which supported SCSI-3 Reservations, which are required for WS08 clusters. I’ve downloaded and installed Open-E DSS.

For networking, all i was able to find was a 100mbit 3com 24 port hub. Yes, this looked like one of the most ghetto environments i put together yet, but interestingly i got it all to work.

Open-E DSS installs to an USB stick, formatted with FAT32. You just unzip the installation file, run an .exe on the stick to make it bootable, and then you can run the system directly from USB. In my case, using rather outdated hardware, everything was recognized by the Linux kernel. Of course, the machine only having a slow 40GB 5400RPM hard disk wasn’t exactly the fastest on the block, but configuration was surprisingly simple. Unfortunately, installing and activating the Lite license required two reboots, after which it lost all it’s iSCSI settings (but no data!)

Installing Hyper-V Server 2008 R2 on the ML110 was a breeze. Using sconfig, the machine was quickly joined to the domain, remote administration enabled, failover clustering enabled and using the graphical iscsicpl iSCSI was configured, the volumes formatted and attached.

Next was the HP desktop machine. Installing was fine, worked perfectly, all the necessary hardware was recognized. Unfortunately, the machine only had 1GB of RAM, which meant that i couldn’t do all that much fancy stuff with it. I was in for a nasty surprise here later, because i didn’t enable Intel VT in the BIOS (which is hidden in the “Security” Options). I think the Hyper-V Server setup should give you a warning here if the feature isn’t enabled.

Next i created the cluster. I’ve used this blogpost and TechNet to get a basic overview on what i needed to do. In just a few steps through the cluster configuration wizards, my cluster was configured and ready. I was able to bring my VM online on the first node (the ML110) and decided to install Windows XP, since i only had 1GB of RAM on the second node. I gave the VM 256MB of RAM and ran through the setup (which took ages – iSCSI over a 100mbit Hub to an old PIV with a 5400rpm hard drive isn’t a good idea anymore).

Next, i decided to setup VM networking, created the appropriate VM interfaces on both machines, restarted my XP VM and tried to do a live migration. Which failed. “Insufficient system resources”. Turns out i needed to adjust the amount of memory reserved for the root partition using PowerShell – all described in this Clustering and High Availability blog post.

After running (get-cluster HV01).RootMemoryReserved=128, it failed again. This time with these event log entries:

‘Test-VM’ The switch port connection for “Network Adapter” (BE62B93F-1490-4F7E-8229-FA18D50DC974) is invalid.

‘Test-VM’ Microsoft Synthetic Ethernet Port (Instance ID {BE62B93F-1490-4F7E-8229-FA18D50DC974}): Failed to Power on with Error ‘The system cannot find the path specified.’ (0×80070003).

Failed to connect NIC ’9144ED30-35D9-4E5F-8012-70AC436EC603–BE62B93F-1490-4F7E-8229-FA18D50DC974′ to port ” on switch ’0734959D-3′, status = C000003A.

I disabled networking in the VM altogether, and tried Live Migration again. It worked! The next was spent with searching the internet for information about my issue, about which i found nothing. Obviously the network interfaces should be named the same in all cluster hosts, but that was the case. Yet, no matter what i did it didn’t work!

I was starting to doubt my hardware, added a second pair of NICs since the configuration of using the same NIC for everything wasn’t really recommended, but when reading the error message it really didn’t sound like that was my issue. Of course adding the second pair of NICs didn’t help.

So i did what i always did: i started guessing, and after quite a bit of time i got it rights. Turns out you must not use the Hyper-V MMC to manage the VM configuration, and instead the “Settings” button in the failover cluster manager. Only issue is that the failover cluster manager has a much more prominent button labeled “manage virtual machine”, which opened the Hyper-V MMC.

After that, everything worked. I was able to live migrate my machine including the network from host to host. I tested running a Top Gear clips through RDP, while live migrating the machine.

Migrating from the slow HP desktop to the ML110 gave about 2 seconds of video outage, but migrating from the ML110 to the HP desktop just resulted in a slow hiccup. My assumption was that this would probably be completely invisible on more modern hardware.

So what does this mean? Microsoft has made Live Migration and Clustering a feature available to everyone, at (almost) no cost. Administrating such a cluster requires Active Directory, and either a WS08R2 server or a Windows 7 machine with RSAT installed.

This means we can finally have decent virtualization features without paying thousands of francs in licensing fees. I hope this makes it possible to create a few virtualization projects for our customers, which are mostly in the small business range.

Hyper-V Server R2 should be available around mid-August, at which i’ll need to rebuild my Ghetto setup here. I’m of course hoping to get some more cash in order to move or internal virtualization setup from a single-host to a SAN-hosted cluster, but somehow i doubt that will happen quickly.

Update:

I’ve played around with Expression Encoder a bit, and created a Video of a Live Migration. I’ve put the probably most boring video on Youtube – Live Migration of Pinball.

So SSDs have been out for quite some time and with the upcoming release of Windows 7 (August 6th on TechNet), i decided to get myself one at home. This coinceded nicely with Intels announcement of the X25-M G2, the 34nm SSD.

I ordered mine last Friday and it arrived yesterday. It looks like i was rather lucky with this, because Intel has since withdrawn them because of an issue with using a HDD password. Since i don’t use that, i didn’t care and installed Windows 7 RC.

And, well, the performance is absolutely astonishing. I’ve upgraded from an old AMD X2 running Windows 7 Beta earlier this year to an i7 also running Windows 7 Beta – more RAM, new hard disk, new graphics. The UI wasn’t noticably faster, sometimes Explorer decided to take ages to work things out, the event viewer was still rather slow, but of course video recoding and games were a lot better.

But now, with an SSD, the UI is extremely responsive – event viewer opens instantly, switches instantly between categories. It just feels like a new machine, with just one part replaced.

So, yeah. I’m quite pleased with my purchase.

Finally, Office 2010 is now available officially.

I’m currently running Windows 7 Build 7100 x64 on my laptop, so i decided to uninstall Office 2007 and install Office 2010, the x64 version.

Unfortunately, the x64 version does not support running any x32 plugins. This is unfortunate, since one of the key features of our ERP package is a tight integration with Microsoft Office, which even i used from time to time. We call it the Office Solution Pack, which has been completely rewritten as a native .NET Office 2007 Plugin. The screenshot to the right is slightly outdated, but it clearly shows the integration we had.

I’m already pushing our developers for a x64 version of the plugin, but i suspect it will take a lot of time until all applications we are using (e.G. System i Access) are running on Office 2010 x64.

Also, its important to know that having 32bit plugins around after installing Office x64 will lead to several nasty error messages. You will need to uninstall the 32bit plugins before installing Office x64. Interestingly, the Primary Interop Assemblies come now shipped with Office by default. This alleviates installing them, streamlining the plugin installation process.

This also means that Office 2010 x64 will be a very niche product, and most people will opt to deploy the 32bit version. This isn’t bad – now that an x64 version is around, plugin developers can start adopting and we may see larger deployments of Office 2010.Next x64.

I’m now running my productive mailbox against Office 2010. Yep, might be risky, but it also gives me a very easy way to learn a new product before we’re actually pushing it to customers. I’ve found that i’ve never learned all that well in a separate environment.

So, whats new in Office 2010? Well, the only application i use daily is Outlook. And we will see how it plays out. Outlook 2010 now also comes with the new ribbon interface, and so far i like it. We will see how this plays out.

My current employer Acommit AG is hiring.

We’re looking for a developer with knowledge on .NET (C#) and Java for our office in Horgen/ZH.

I’m not that much involved with our development team, but i can tell you that you’ll have current infrastructure to work on – Windows Vista/7, Visual Studio 2008, a current PC with two highres screens, a virtualization environment for testing and of course free coffee and soda.

Official text and contact on the official Website.

So, IBM’s x3650 M2 have been out for some time and in the meantime i’ve deployed three of them – two with SBS 2008, and one with Windows Server 2008. No pictures, since the camera in the office is broken.

The x3650 M2 comes with a new IMM (Integrated Management Module) that replaces both the BMC and the RSA II Slimline. In order to get remote KVM capability, a physical licensing key must be installed into the server. Standard features like remote power on/off are available without the licensing key – which is more expensive than the physical RSA II slimline adapter. With the IMM also comes UEFI, as a replacement to the aging BIOS.

The x3650 M2 also gets rid of the Adaptec ServeRAID 8k controllers, and introduces us to the ServeRAID 10 series manufactured by LSI. The ServeRAID 8k series have been plagued by several extremely heavy bugs that never caused me loss of data (but several other people), but nevertheless cost me a lot of my nerves.

The two standard RAID controllers in the x3650 M2 are the ServeRAID BR10i, which is the baseline controller without BBWC and without support for RAID5. The ServeRAID MR10i is the better model, which comes with 256MB BBWC and support for RAID5/RAID6. Unlike the ServeRAID 8k/8k-l story, these are entirely standalone controllers, that are located in a special daughterboard position with a standard PCI-e x8 interface.

Both controllers support only 8 drives – in order to get 12 drives, you need a special enabler kit that comes with a SAS expander hard and several of other stuff that doesn’t look all that trivial. I’ve used such a kit, and so i can’t comment on how it works exactly.

The power supplies have gotten a lot smaller, the server seems to look a lot more organized, the 2.5″ SAS HDDs are no longer as finicky as they were in the x3650 and now fit very well into their slots, the Lightpath diagnostics panel now looks like it belongs to an expensive server and locks into place securely.

Of course, all the usual changes that come with the new Nehalem based Xeons, triple channel DDR-3 memory, both processors needed for using more than 8 slots, using to many memory modules will downgrade the speed, etc.

So much for the general rundown – now for my assorted observations:

• The ServeRAID BR10i seems to be a slightly newer variant than the SAS RAID Controllers found in the x3250 and x3250 M2. The configuration interface is simple, but it works reasonably well.
• The ServeRAID MR10i is a controller i haven’t dealt with before. It does not offer a standard character based interface for configuration, only a graphical interface called WebBIOS. It’s completely awful and half done – half of the buttons have no text on them, every button press requires several seconds until something happens. Configuring a controller with this interface requires you to guess actions based on the manual, since they aren’t labeled onscreen.
• As usual, both LSI based controllers use the MegaRAID management software. Compared to the old Adaptec software, it’s really awful. It runs extremely slow – even on these new servers, is much more complicated than the old ServeRAID software and offers less options in terms of notification.
• The IMM webinterface has gotten even slower than the already slow RSA II interface. Web2.0 style “loading” icons have been added, but viewing the status screen can take up to a minute now. This is retarded, and clearly a step back. At least IMM standard now comes with every server.
• The IMM’s KVM capability have gotten a lot better. Instead of a java applet running in the browser, a java application is launched using java webstart. While IMM itself is slow, the remote KVM capability is actually very fast, and even works with decent speed through a VPN connection.
• While the ServeRAID 8k with newer firmware usually spent 2-3 minutes looking for the drives, the new LSI based ServeRAID 10 series now only takes a few seconds. This is compensated by UEFI which now takes roughly 2 minutes instead of the 10 seconds the old BIOS took. With this, IBM is successfully keeping the server at roughly 4 minutes until OS boot
• The IMM connects to the OS using an USB LAN interface. This is a real problem on Windows, since it confuses the Windows Firewall (switching it into “Public network” mode) and the Windows DNS client. Install the driver and disable it in Windows. Ensure you never enable it on DCs! Run the IMM firmware updates from CD. Clearly a step back.
• The UEFI configuration screens act a lot slower than the old BIOSes. But the options available are decent.
• You can install Windows Server 2008 Standard in UEFI mode. I did that on our internal x3650 M2 which is going to run our WebFOCUS deployment
• You cannot install SBS 2008 in UEFI mode. It will work fine with legacy BIOS emulation, though.
• The IMM can’t send alert messages to email addresses with a – (Dash) in them. Retarded.
• The machine is extremely silent. Compared to the Power 520, which will kill your ears within minutes, they’re a blessing.
• Just like the RSA II interface, the IMM web interface has a tendency to lock up randomly and stop working. Requires a physical power cycle on the server to fix.

That’s it for now. Lot’s of negative stuff in here, but the machines are actually extremely good performers. I hope that IBM will fix those outstanding bugs soon.