Interestingly, the Hotline wasn’t reachable either – busy signal, Swisscom text “Leitung gestört” or simply “Call Failed”.

Lasted from 19:33 to 20:30, but it looks like everything is back online now.

As a disclaimer, i realize that Debian in any version isn’t a supported OS on Hyper-V R2 – i just want to tell of my experiences with this unsupported configuration.

The hardware, an aging IBM xSeries 306m with a Pentium 4 CPU wasn’t getting any younger and after a drive failure about half a year ago that lead to a system crash (No data loss though – it just crashed the machine, that’s Software RAID for you), it was finally time to modernize this.

The plan is to consolidate all our DMZ workloads (ISA, OCS Edge, XMPP Gateway, Exchange Edge) on Hyper-V 2008 R2 and doing the trickiest part first seemed like a good idea.

So i created a new VM using SCVMM 2008 R2, selected Other Linux 32bit as the guest OS, inserted a Debian 5.0 netboot CD and that’s where the problems already started. While the installation worked well in general, the Framebuffer used by the Debian installed is awfully slow. So it took me about half an hour just to get the install done (on a 5GB partition of the 80GB VHD).

After finishing the installation, i formatted the rest of the disk appropiately and then used rsync to transfer the machine contents over. A short bit after reconfiguring Grub, i could choose to boot either the transferred OS with it’s kernel, or the Debian 5 rescue system i installed alongside.

Booting the transferred system worked well enough, but the tulip driver wasn’t compiled into that (custom) kernel and building the module failed. So i read up a bit, and realized that the newest kernel (2.6.32.8) shipped with experimental Hyper-V VMbus drivers, that allowed synthetic NICs to be used.

I tried to compile the kernel after chrooting into the old installation, but it failed because gcc was too old. Not to worry, i compiled it in the rescue system, but couldn’t install the dpkg that make-kpkg created. So i installed it manually, which worked pretty well.

One reboot later, i was back in business with the extremely verbose Hyper-V drivers cluttering up dmesg, but the Synthetic NICs showed up as seth0 – seth2. After quickly changing all the necessary configuration files, everything was working.

After a bit of more testing, i disconnected the physical machine from the network and plugged the VM into the production VLANs.

I tested everything thoroughly and didn’t find any issues. Sent out an information mail and continued on my merry way.

Half an hour later, i decided to do a quick systems check again – and i realized that the external interface (seth2 in this case) wasn’t working anymore. tcpdump showed no packets being received and other machines in the same VLANs didn’t see any answers to their ARP requests either. So i rebooted the VM, and everything was working again. No error messages of any kind, neither in dmesg nor in the system logs or on the Hyper-V host.

Hoping this was just a fluke, i waited until it happened again – which it did, roughly 10 minutes later. So i decided to skip on the synthetic devices and go with emulated NICs and the tulip driver.

Everything came back up, but i couldn’t ping any devices on the eth0 VLAN from the start, but the other two interfaces worked.

After a few more tries, i arrived at a configuration that has now been stable for 4 hours and 26 minutes, which sounds good so far. For this, i configured a single synthetic NIC that i used as a replacement for the non-working eth0 and three tulip NICs (of which the first was unused).

There are other things that also worry me:

Every reboot of the Linux machine created the following event log entry on the Hyper-V host:

'LINUX' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID [])

Loading the synthetic NIC drivers logs the following in the event log on the Hyper-V host:

Networking driver on 'LINUX' loaded but has a different version from the server. Server version 3.2 Client version 0.2 (Virtual machine ID []). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.

Loading the synthetic NIC drivers also logs all this on the Linux side of things:

VMBUS_DRV: Vmbus initializing.... current log level 0x1f1f0006 (1f1f,6)
VMBUS: +++++++ Build Date=Feb 17 2010 12:37:00 +++++++
VMBUS: +++++++ Build Description=Version 2.0 +++++++
VMBUS: +++++++ Vmbus supported version = 13 +++++++
VMBUS: +++++++ Vmbus using SINT 2 +++++++
VMBUS: Windows hypervisor detected! Retrieving more info...
VMBUS: Vendor ID: Microsoft Hv
VMBUS: Interface ID: Hv#1
VMBUS: OS Build:7600-6.1-16-0.16485
VMBUS: Hypercall page VA=f80c9000, PA=0x36afe000
VMBUS_DRV: irq 0x5 vector 0x35
VMBUS: SynIC version: 1
VMBUS: Vmbus connected!!
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={c5295816-f63a-4d5f-8d1a4daf999ca185}
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
hv_netvsc: module is from the staging directory, the quality is unknown, you have been warned.
NETVSC_DRV: Netvsc initializing....
VMBUS_DRV: child driver (f80dc570) registering - name netvsc
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={32412632-86cb-44a2-9b5c50d1417354f5}
VMBUS_DRV: child device (f73a8634) registered
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a}
VMBUS_DRV: child device (f73a5a34) registered
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={f8615163-df3e-46c5-913ff2d2f965ed0e}
VMBUS_DRV: device object (f73a5ee4) set to driver object (f80dc5c0)
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS: channel f73aac00 open success!!
NETVSC: *** NetVSC channel opened successfully! ***
NETVSC: Sending NvspMessageTypeInit...
NETVSC: NvspMessageTypeInit status(1) max mdl chain (34)
NETVSC: Sending NvspMessage1TypeSendNdisVersion...
NETVSC: Establishing receive buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendReceiveBuffer...
NETVSC: Receive sections info (count 1, offset 0, endoffset 1048000, suballoc size 1600, num suballocs 655)
NETVSC: Establishing send buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendSendBuffer...
NETVSC: *** NetVSC channel handshake result - 0 ***
NETVSC: Device 0xf6552e80 mac addr 00155d031a09
NETVSC: Device 0xf6552e80 link state up
VMBUS_DRV: child device (f73a5e34) registered

So, it works. But not without troubles. I’ve still got the physical machine to fall back on, but i sure hope Microsoft will get this to work better.

These issues are the reason why i decided to deploy my private server using ESXi instead of Hyper-V – because i need both Linux and Windows guests.

I’ve setup a new Hyper-V server on a x3650 M2 (using server core) – i’ve also installed the latest Broadcom NetXtreme II drivers, all the firmware updates, all the best practices you do.

Setting up the machine, transferring VMs from another host (using BITS) worked well and fast, no issues.

And then i installed the DPM agent, started a backup. Two hours later, it was still stuck at “Replica creation in progress”.

I tried reading through the DPM agent logs, through the DPM server logs, looked if DPM created shadow copies (using vssadmin list shadows).

After two hours of fruitless searching (which included restarting everything), i wasn’t any further to a solution.

Well, backup wasn’t working right, but this was just a testing environment, so i decided to do other stuff.

A while later, i ran netstat -t to lookup connections – and also realized that TCP Chimney Offloading was still active. So i disabled it using netsh int tcp set global chimney=disabled. Just a few seconds later, the utilization of the management network adapter jumped to 100% and 5 minutes later, all the VMs were replicated to the DPM server.

So, if you’re having issues with DPM backups being stuck, check the status of your network offloading.

If you’re reading this, this blog is now hosted on Windows Server 2008 R2 Web Edition (Yay NFR promotions!). There may still be some kinks that have to be worked out, because this was quite a rush job. Leave a comment if you find any issues.

Our current cablecom hispeed business line is ADSL2+ with 20/2 megabits. While the upstream is too low for my taste, i haven’t really seen better offers.

I talked with a sales on the phone – for about 200 CHF more, we could get 20/2 SDSL (which sounded strange) and a 20/2 DOCSIS backup line, together with a “Bronze” level SLA. This sounded very attractive to me and i told the sales to send me the offer.

In the written offer, the ominous 20/2 SDSL was downgraded to 4/4 SDSL (which made much more sense). Of course, downgrading our internet connection from 20/2 to 4/4 seemed like a rather bad idea. We have about 30 people working here everyday, and almost all of them really use the internet to do their job. We’ve upgraded from 6/.6 ADSL to the current cablecom connection, because 6 megabit downstream wasn’t fast enough.

So i asked what else they could offer us – for 500 CHF more than we pay today, we could get 8/8 SDSL with a 20/2 DOCSIS backup. That still didn’t sound interesting to me.

I, personally, think 1000 CHF per month would be okay for a redundant 20/20 connection or something in this direction. My current connection at home is 25/2.5 – for 75 CHF a month. It works well enough, and the last failure i had was fixed in three days. Just like the failure we had on our 500 CHF per month 20/2 connection. This should be a telltale sign that something is very wrong with either the pricing or the service level.

The next question i asked if they could do a 20/2 ADSL with a 20/2 DOCSIS backup. Apparently, that’s not technically possible right now, but they might introduce this later this year. That sounds attractive to me.

All in all, i still think that cablecom hispeed business sucks. They can’t be bothered to do a 5 minute fix in a 2 hour time window on Friday evening. Then, they make one ludicrous offer that noone can take serious after the other.

I’m pretty sure that cablecom doesn’t really understand what small businesses need.

As a side note, if you work for an ISP and think you can make us a better offer than cablecom, i’d be very much interested. Send your stuff to l dot beeler at acommit dot ch. We will be moving to Horgen/ZH at Seestrasse 202 in March 2010 and need 32 static IP addresses.

Shortly after installing the line back in 2008, we’ve ran into an issue where cablecom hispeed business blocks GRE packets. After almost three days and speaking with a variety of technicians, they were finally able to resolve the issue.

Now, we’ve run into another, much more grave problem. Since about 15:45, a variety of hosts on the Internet aren’t reachable and of course several other hosts can’t reach us.

Of course this isn’t a clear-cut “my DSL modem has no link” issue – so cablecom currently isn’t even trying to fix the problem. I’ve been on the phone twice, never get any callbacks and don’t get any updates on the state of the problem resolution.

Fact is, some hosts can reach our OWA 2010 and some can’t. Nasty thing is, Swisscom’s GPRS/UMTS IP addresses can’t – this means no push-email for all 35 of our employees. Since we’re working for a rather important project (ERP and POS implementation) this weekend, this is a big issue for us.

It also looks interesting in a tcpdump – some packets just get lost – and from other hosts it works without any issues.

The 77. addresses are cablecom hispeed business, the 217. are my cablecom residential connection. In the first part, we see a TCP connection to port 80. In the second part, we see a ping -t. As you can see, there are a lot of dropped packets.

23:12:12.629457 IP 217.162.252.98.18417 > 77.59.216.227.80: S 4006182815:4006182815(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
23:12:12.629479 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:15.826736 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:22.026734 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:34.026733 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>

08:51:49.642995 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 65
08:51:49.643024 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 65
08:52:00.641330 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 68
08:52:00.641345 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 68
08:53:16.641813 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 84
08:53:16.641829 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 84

Cablecom gets 180 CHF per month for 24/7 support. The case has now been open for 7 hours, with no resolution in sight. There’s no escalation path and there are no workarounds – we don’t have redundant connections.

Interestingly, one of our customers who also uses cablecom hispeed business had a similar issue, that lasted for roughly three weeks – one of their IP addresses wasn’t reachable externally, from one minute to the other. Unfortunately for us, all of our public IP addresses are affected by this issue, so we don’t have an easy workaround.

Of course, for some part we’re also to blame. Luckily i’m not one of the higher ups who gambled with non-redundant internet connections and lost.

Have you made negative experiences with cablecom hispeed business? Positive ones? Was support able to fix your issues quickly?

Update: I’ve called cablecom again on Saturday at 09:00. Apparently, these sort of issues are supported on a best-effort base from 9 to 5, and not covered by our 24/7 support contract. We will have to wait until monday – they will not look at this issue further during the weekend.

Update: Monday morning, 11:00. Problem is still unsolved.

--- hor-fw-01.acommit.ch ping statistics ---
20 packets transmitted, 3 received, 85% packet loss, time 19012ms
rtt min/avg/max/mdev = 20.490/21.360/22.585/0.891 ms

Update: Monday morning, 11:36. Problem is now solved. According to the Tech i talked to, the he fixed the issue in 5 minutes. He could’ve done that on Friday, but apparently noone at cablecom felt like doing so. The issue was that cablecom configured our new line for the planned office move and configured load sharing between the new line for the new office and the old line. Since the new line didn’t physically exist yet, half of the packets were dropped.

Thanks to the Tech who fixed the issue, no thanks to cablecom in general for wasting an entire weekend on what could’ve been a five minute fix on Friday evening.

Currently, we’re looking for:

Project Manager (PDF)
Sales (PDF)

Sure, there are loads of MVPs and TAP-Members that have migrated to Exchange 2010 a long time ago, but i’m still proud of this.

At a starting point, i had a Exchange 2007 SP2 machine, with one Mailbox database, no public folders and 35 Mailboxes that used up 25GB of space. Moving this is simple enough, but the issue is that our Exchange isn’t virtualized, and i couldn’t get my hands on new hardware since the current box was only a year old.

Since in-place upgrades are not supported, i needed a temporary server for the migration. I used an HP ML110 from the Lab, which offered enough space to migrate.

Another issue was BackupExec 12.5, which did not support Exchange 2010 yet. Fortunately, Exchange 2010 (and 2007 SP2) can be backed up by using Windows Server Backup. So my goal was to just let WSB backup to a file server, and have BackupExec pickup the files from there. This way, i will get a reliable, clean and supported Exchange backup, and still have it on tape.

To Migration itself was straightforward and easy. There’s already _lots_ of content on the web about Exchange 2010, most of it from the RCs or Beta of course.

I followed the Migration Guide from TechNet, which worked out well enough. Unfortunately, the iPhone does not support Exchange 2010/2007 coexistence, which made it necessary for several people to manually reconfigure their phone.

Removing Exchange 2007 worked without issues, but after moving all the Exchange 2010 data back to the real hardware and removing the temporary server i ran into the issue of moving arbitration mailboxes, which fortunately was already documented widely on the web.

In the end, upgrading from Exchange 2007 to 2010 while keeping the same hardware is not difficult, it just needs a bit more time.

The error message in the IIS Log looks like this:

RdirTo:https%3a%2f%2flegacy.contoso.com%2fMicrosoft-Server-ActiveSync_LdapC2_LdapL15_Error:MisconfiguredDevice_Budget

For the benchmarks, i’ve used the free version of HDtune. I’ve benchmarked four systems, and five different disk configurations. Note that the free version only does benchmarks for disk reads, and it’s a not a very pervasive test. None of these benchmarks are scientific. They should serve as a general indicator of performance, not as a final world on this topic. I don’t have that much clue about benchmarking.

The first system is my computer at home: It has an i7-920 CPU at stock speed, with 3×2GB RAM at 1333 Mhz (which is a slight overclock, but within the spec of the memory i purchased). Attached to it’s ICH10R controller are an Intel X25-M G2 160GB (Firmware 02HA) and a WD1001FALS (1TB, 7×24), running Windows 7 x64.

The next system is my work laptop, which is a ThinkPad W500 with a 2.53 Ghz T9400 C2D CPU, with 4GB of RAM. Attached to it’s onboard controller is an OCZ Vertex 120GB (Firmware 1.40), running Windows 7 x64.

The third system is our Exchange Edge server, on which i dared to install a benchmark utility. It’s an IBM x3250 with two 70GB 15kRPM 2.5″ SAS drives installed, attached to an onboard LSI1064E SAS controller. The system has a Xeon 3040 2.4Ghz Dualcore CPU and 5 GB RAM. It is running Windows Server 2008 x64 SP2.

And the final system is a HP ML110 G5 with a 2.33 Ghz Xeon 3065 CPU, 8GB of RAM and a E200 with the latest firmware (1.78). Attached to that are 4 WD1001FALS drives in a RAID10 configuration. The E200 has a backup battery and 128MB of cache installed. The system is running Windows Server 2008 R2.

Please note that none of these benchmarks are scientific. They were done on real systems, with workload minimized as much as possible, but virus scanners and other mandatory background applications active. Both the laptop and the desktop have not been formatted since Windows 7 RC was installed (i migrated to Windows 7 RTM using Windows.old), but the ML110 was freshly setup and the only application that’s been installed so far is the HP ACU and Forefront Client Security. The Exchange Edge server has been in use since May 2008. As such, the ML110 is the “cleanest” machine out of these four.

Intel’s X25-M G2 160GB on an ICH10R (AHCI Mode)

This is how a graph should look. It’s nice, it’s clean, it’s fast. Intel’s X25-M G2 shows how a modern SSD and storage subsystem should behave. Clean, predictable performance.

OCZ’s Vertex 160GB on an ICH7 (AHCI Mode)

Here’s the OCZ Vertex. It’s running on a machine that’s a lot slower than the one the X25-M is attached to, and it’s storage controller is also quite a bit older. It still shows remarkably good performance. It should also be considered that this Vertex is quite a bit older – it was bought in May 09. It’s still very fast and responsive and a good SSD.

2x IBM’s 73GB 15kRPM 2.5″ SAS Disks on an LSI Logic 1064E SAS Controller

As you can see, this is the performance you get from the server hard disks on an entry-level controller in an entry-level system. It’s not astonishing, but the performance is very well acceptable.

Western Digital’s 1001FALS 1TB on an ICH10R (AHCI Mode)

Here’s how the Western Digital disk behaves on a proper controller. Please note that this is a single disk, not part of a RAID array. The performance is quite good.

4x WD’s 1001FALS 1TB on an HP E200 in RAID10

And here’s how it shouldn’t look. Compare this to the stand-alone disks above, which exhibits better performance. HP fucked up bad on this one, and there’s no fix in sight. Stay away from the E200.

And as a final word: I really don’t have much of a clue about benchmarking. If you see an obvious error here, please state what you think. If possible, i will try to correct it.

Update: As requested in the comments, i upgraded the E200 to Firmware 1.84 and redid the benchmark. It looks roughly the same.

If you’re following the recommended deployment method for Exchange 2007, you’ll already be using a SAN certificate in order to publish AutoDiscovery and OWA. For coexistence of Exchange 2007 and Exchange 2010, an additional name will need to be added to your SAN certificate.

With most CAs, this is a pretty straightforward process that can be done using their web interface, since the private key doesn’t need to be touched. After modifying this, you will get a new .crt file containing the certificate, but no private key (which is correct).

However, importing this into Exchange 2007 using Import-ExchangeCertificate doesn’t work – Windows won’t know which private key is associated with the newly imported certificate. When you try to use Enable-ExchangeCertificate, you will receive the following error message:

Enable-ExchangeCertificate : The certificate with thumbprint 1234 was found but is
not valid for use with Exchange Server (reason: PrivateKeyMissing).

I searched high and low on how to replace a certificate without touching the private key, but i didn’t find anything. So i turned to the community for support – MCSEBoard.de is an excellent Windows community for those who speak German.

Unfortunately, noone knew an easy way either – the suggestion was to use OpenSSL to create a new keystore.

This was rather easy, but i didn’t find any guides on the net on how to do this, so i’m publishing this here in the hope that it will help others with the same issue.

• First, you need to export the key including the private key using the Windows certificate manager. Open an elevated MMC, add the Certificate snap-in and focus on the Computer certificate. Click “Personal”, and then export the certificate with the private key.
• Download and Install OpenSSL for Windows
• Issue the following command: openssl pkcs12 -in mykey.pfx > out.txt
• Open out.txt using an LF-aware text editor, such as Notepad++. Save the PRIVATE KEY part to a textfile called key.pem.
• Save the certificate to a file called cert.crt
• Issue to the following command: openssl pkcs12 -export -in cert.crt -inkey key.pem -out newcert.p12
• Copy the newly created newcert.p12 to the Exchange server.
• Open PowerShell and run the following command: $secureString = ConvertTo-SecureString "blubb" -AsPlainText -Force – Replace “blubb” with the Passphrase you used in the step before
• Run Import-ExchangeCertificate -path newcert.p12 -pass $secureString to import the certificate back into Exchange
• The rest is as usual – use Enable-ExchangeCertificate to enable the certificate.

And that’s it. It might be a bit cumbersome – and i really hope that there is an easier way to to this. If you know, let me know so i can update this page.

See here for the fix and it’s description KB974571

Click here to download the ocsasnfix.exe directly, which will fix the incorrect ASN License data – something which i already guessed about in my previous post about this issue.

During this time, i’ve gained valuable experience, which i’ll try to share here so that others can profit from it. Take all this with a grain of salt, as some observations may simply be my fault. Also, as times changes these things might change too.

Software

• Make sure to install Windows Server 2008 SP2 after installing SBS 2008. Some media may come with SP2 already preloaded. You can use the normal SP2 package that’s also used for Vista and the normal Server 2008
• Do not install SBS rollup updates before completing the configuration wizard. This is extremely counter-intuitive, but is described on the Official SBS blog
• Installing Exchange 2007 SP2 requires you to follow special considerations Here
• Installing WSUS 3.0 SP2, which is needed to support Windows 7, is currently not recommended. I was able to do this without issues on my lab machines, but others have reported issues doing this on machines that were in production. If you’re deploying a new SBS server, this should probably be safe to go. But make sure to test functionality afterward.
• Always use the answer file to deploy SBS 2008. This will make it possible to choose a custom domain name. Read my post about choosing your AD DNS namespace
• Do whatever tasks you can do using the SBS console. Resist of using the normal administration tools as much as possible, as you can break SBS with them easily.
• Ensure that the AV software you install is compatible with WS08 x64. Symantec Endpoint Protection Manager works well – Forefront Client Security on the other hand requires a seperate server running 32bit Windows for management. You may consider deploying FCS unmanaged in smaller environments, and configure FCS using the FCS ADM File

Hardware

• Use servers with the new Xeon 5500 CPUs. Read my x3650 M2 tips to find more about them. Consider using an E5530 or faster CPU. Using two CPUs (for a total of 16 virtual and 8 physical cores) makes little sense.
• Buy enough memory. Lots of it. Really. I mean it. You’ll need lots and lots of memory. I would consider 12GB to bare minimum. In a 3×4GB configuration which makes the most sense for the Xeon 5500 setups, this is quite cheap. Consider more memory if you intend to run SQL Server as, consider bumping the memory to 24GB. Remember that you can only use the first 8 slots in a single socket machine.
• Buy enough disks. A good starting layout is 8×147GB 2.5″ disks. Use a RAID 1 for the OS, another RAID1 for Exchange and Sharepoint, and a RAID10 for Data and WSUS. This is all up for debate of course, and it might make sense to consider other disk layouts.

If you have any additions, think i’m wrong somewhere just send in a comment.

Security updates are important. And as we’re currently an evaluation setup for OCS 2007 R2, i’ve decided to install todays batch of security updates on these lesser important machines first. And after a reboot, OCS 2007 R2 was broken.

A quick view into the event log revealed that OCS 2007 R2’s evaluation license has expired. Now, this seemed very strange as i’ve installed from volume license media. I’ve the checked the media again, but they weren’t evaluation media.

Here’s the message in all it’s glory:

Event source: OCS Server
Event id: 12290
Event text: The evaluation period for Microsoft Office Communications Server 2007 R2 has expired. Please upgrade from the evaluation version to the full released version of the product.

Maybe i really did use other media to install it? I doubted myself, because that’s usually the most reasonable approach to take. The error is usually behind the keyboard.

Luckily, Microsoft has published documentation on how to upgrade an evaluation version to a full version. Unfortunately, this didn’t work, because as it appears i was running a Volume license version of OCS.

EVALTOFULL parameter cannot be used with currently installed license type Volume

At this point, i was pretty sure that this wasn’t my fault. There has been an issue with the OCS 2007 R2 Evaluation Media expiring at the wrong point in time, but apparently this has been sorted out and did never affect the full versions of OCS 2007 R2.

So i was bummed. A quick view using process monitor revealed that the licensing information was most likely to be stored here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RtcSrv\InstallInfo\ValidationData

I created a backup of that part of the registry, and then renamed the key. I got a file not found error, and created a new key of the same type and wrote binary data of the same length into it. This yielded the following error:

The service is shutting down due to an internal error.

Error Code: 80093102 (ASN1 unexpected end of data.)

At that point, i was pretty sure what might’ve caused this – the MS Crypto API security update KB974571.

I removed the update, rebooted the machine, and OCS 2007 R2 was up and running again, without any issues.

I’ve already opened a case with Microsoft to get this sorted out.

Update:
Appears that this is an official issue: See here

Update: The license is not entirely clear – it does not exclude commercial use, only SaaS use. But in the beginning of the license it says that only home-based small businesses are allowed to use it. So take this with a grain of salt – the license is certainly more permissive than Avira, but it’s not as easy as i thought.

I’ve been participating in the MSE beta test with my Windows 7 machines at home, and my impression has been very good. Performance is excellent, and the GUI is simple and straightforward.

After a few negative experiences with McAfee 8.7 at work, and my very good experiences with MSE at home, i tried to have another look at FCS.

Well, FCS is rather old right now, with the new release just on the horizon. Still, the current release is supported on Windows 7 x64 with the latest patches, and so far my impressions have been very good. The management server only runs on 32bit Windows, which also means it won’t run on WS08R2.

But my impression has been good so far – the package installed on the client is far more lightweight than McAfee, and even the managment software leaves a much better impression.

We’ve also been using Symantec Endpoint Protection at a few customers, but my impression of that product was even worse than McAfee.

We’ll see how FCS will fare, and the test deployment is currently running. If you have any good tips or websites for me, i’d be delighted to read them.

If you work on the IBM i platform, the Midrange Mailing Lists may also be a place to visit and subscribe. Also, check out the IMHO Midrange Blog.

However, we’re also an ISV. DIAS-iS has been running on Windows Vista since the release – thanks to the efforts of our developers, who fixed everything during the beta phase of Windows Vista. As such, our software ran on Windows 7 since the beginning.

During the past few weeks, i did all the necessary administrative work to get our Software certified with the “Compatible with Windows 7″ Logo.

Doing this isn’t that hard, but it requires you to jump through quite a few hoops.

Here’s a basic rundown of steps:

• Obtain a MS Authenticode certificate from Verisign. Note that other code signing certs won’t work (e.G. Thawte)
• Create a WinQual Account here
  • You’ll need to sign a sample .exe with the code signing cert from step one
• Download the Software Logo Toolkit
• Download the Windows 7 Logo Requirements Document
• Both of these packages contain all the documentation you need – most of the requirements are easily satisfied if you have an application that behaves nicely, uninstalls correctly, works in TS environments
• Create an empty Windows 7 x64 VM. Note that it must be x64.
• Install the Software Logo Toolkit on the machine
• Start the GUI, start the Session Server in a second session on the same machine
• Run through all the phases, make sure the report says “Pass” or “Pass with warnings” (verify that the warnings are not real errors)
• Submit the .xml through the WinQual account. You’ll immediately get certified

So it’s not that hard.

The key point to delivering a good user experience is to ensure that your application uses standard installation technology like .MSI, that it doesn’t require administrative privileges, that all configuration is stored in the userprofile (Registry or %APPDATA%) and that it’s multi-session capable.

And that’s all the “Compatible with Windows 7″ logo verifies – so if you already have a well-behaving Windows application, getting that logo is easy as pie. It does not cost anything directly – the only costly requirement is the fact hat you need a Verisign Authenticode certificate. This will set you back 400$. Microsoft does not want any money from you for this Logo – and it can be great in Marketing your competitiveness and readiness as a software vendor.

I’m using a Diva BRI-2 2 Channel PCI-E Card, which already has support for Windows Server 2008 R2, and installing the Diva Software went without any issues.

Installing the Fax service was also easy, but there was no Fax printer to be seen anywhere.

I’ve followed the TechNet documentation for creating Fax printer on Windows Server 2008 R2, but it didn’t work – at first i received a “Permission denied” error message, after which i started Windows Fax & Scan using Administrator privileges.

This didn’t help that much – i could now go through the wizard, but no Fax account and no printer was created. This seemed strange.

Now, this really seemed like a permission issue. So i disabled UAC, rebooted the server, and tried it again. Everything worked – i was able to create the Fax printer, and after sharing it faxing worked as it should.

So, what now? Why doesn’t this work with UAC? I’ve been running our WS08 servers with UAC disabled (our Vista client were UAC enabled, and so are our Windows 7 clients), and thought WS08R2 should also work well with UAC enabled. But apparently, that wasn’t a good idea.

So, with a fresh newly installed laptop and the new release candidate of Windows XP mode, i gave it a whirl again. But it failed with the same sequence of completely intelligible error messages, namely “Integration features have been disabled” and the even more helpful “Parameter is incorrect”.

So i installed it on my desktop as well, where it worked without a hitch. The major difference between my desktop and my laptop is that the laptop is joined to the corporate domain and the desktop at home obviously not.

I dug a bit deeper into the event log, and drilled down to Microsoft\Windows\Virtual PC\Admin, where i found this error message:

Could not enable the Integration features for ‘Windows XP Mode’. The current mode is – 0. Last Channel start Value – 0×800700B7, Last Disconnect Reason – 0×300001B, Last Extended Disconnect Reason – 0×0, GHI State of the guest machine – 0×1

Now, this whole “disconnect” thing sounded strange until i remembered that Windows Virtual PC used RDP to deliver the screen – and at that point i thought about the RD Gateway server that’s being pushed by a GPO.

So for a quick test, i set the following key in the registry to zero:
HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows NT\Terminal Services\UseProxy

And tried starting Virtual PC again. It worked! Setting the key back to 1 predictably led to the same error message.

So next i excluded Windows 7 users from this GPO using a simple WMI filter, which will be a temporary measure to mitigate this issue.

This seems to be a bug somewhere, as those settings shouldn’t break Virtual PC. I’m not sure where i should report this to, but i’ll have a look at that. At least now people with the same issue should get this solution through Google.

I have to admit that Failover Clustering isn’t exactly the field i have a lot of experience in (in other words, i have never used it in producation). But after seeing that i wouldn’t be drowning in work this friday afternoon, i decided to give it a whirl.

So, in order to get started i needed two machines that were able to support running Hyper-V Server 2008 R2. One of them was HP ML110 G5, about which i wrote a few months back. Unfortunately, i could use only one of them. So my next choice was an old HP desktop, which fortunately had a VT compatible CPU.

Next, i needed a storage backend. Of course i had to use a software solution, but having no experience and only a very old PIV era IBM SFF PC, i just picked the first Google search result which supported SCSI-3 Reservations, which are required for WS08 clusters. I’ve downloaded and installed Open-E DSS.

For networking, all i was able to find was a 100mbit 3com 24 port hub. Yes, this looked like one of the most ghetto environments i put together yet, but interestingly i got it all to work.

Open-E DSS installs to an USB stick, formatted with FAT32. You just unzip the installation file, run an .exe on the stick to make it bootable, and then you can run the system directly from USB. In my case, using rather outdated hardware, everything was recognized by the Linux kernel. Of course, the machine only having a slow 40GB 5400RPM hard disk wasn’t exactly the fastest on the block, but configuration was surprisingly simple. Unfortunately, installing and activating the Lite license required two reboots, after which it lost all it’s iSCSI settings (but no data!)

Installing Hyper-V Server 2008 R2 on the ML110 was a breeze. Using sconfig, the machine was quickly joined to the domain, remote administration enabled, failover clustering enabled and using the graphical iscsicpl iSCSI was configured, the volumes formatted and attached.

Next was the HP desktop machine. Installing was fine, worked perfectly, all the necessary hardware was recognized. Unfortunately, the machine only had 1GB of RAM, which meant that i couldn’t do all that much fancy stuff with it. I was in for a nasty surprise here later, because i didn’t enable Intel VT in the BIOS (which is hidden in the “Security” Options). I think the Hyper-V Server setup should give you a warning here if the feature isn’t enabled.

Next i created the cluster. I’ve used this blogpost and TechNet to get a basic overview on what i needed to do. In just a few steps through the cluster configuration wizards, my cluster was configured and ready. I was able to bring my VM online on the first node (the ML110) and decided to install Windows XP, since i only had 1GB of RAM on the second node. I gave the VM 256MB of RAM and ran through the setup (which took ages – iSCSI over a 100mbit Hub to an old PIV with a 5400rpm hard drive isn’t a good idea anymore).

Next, i decided to setup VM networking, created the appropriate VM interfaces on both machines, restarted my XP VM and tried to do a live migration. Which failed. “Insufficient system resources”. Turns out i needed to adjust the amount of memory reserved for the root partition using PowerShell – all described in this Clustering and High Availability blog post.

After running (get-cluster HV01).RootMemoryReserved=128, it failed again. This time with these event log entries:

‘Test-VM’ The switch port connection for “Network Adapter” (BE62B93F-1490-4F7E-8229-FA18D50DC974) is invalid.

‘Test-VM’ Microsoft Synthetic Ethernet Port (Instance ID {BE62B93F-1490-4F7E-8229-FA18D50DC974}): Failed to Power on with Error ‘The system cannot find the path specified.’ (0×80070003).

Failed to connect NIC ‘9144ED30-35D9-4E5F-8012-70AC436EC603–BE62B93F-1490-4F7E-8229-FA18D50DC974′ to port ” on switch ‘0734959D-3′, status = C000003A.

I disabled networking in the VM altogether, and tried Live Migration again. It worked! The next was spent with searching the internet for information about my issue, about which i found nothing. Obviously the network interfaces should be named the same in all cluster hosts, but that was the case. Yet, no matter what i did it didn’t work!

I was starting to doubt my hardware, added a second pair of NICs since the configuration of using the same NIC for everything wasn’t really recommended, but when reading the error message it really didn’t sound like that was my issue. Of course adding the second pair of NICs didn’t help.

So i did what i always did: i started guessing, and after quite a bit of time i got it rights. Turns out you must not use the Hyper-V MMC to manage the VM configuration, and instead the “Settings” button in the failover cluster manager. Only issue is that the failover cluster manager has a much more prominent button labeled “manage virtual machine”, which opened the Hyper-V MMC.

After that, everything worked. I was able to live migrate my machine including the network from host to host. I tested running a Top Gear clips through RDP, while live migrating the machine.

Migrating from the slow HP desktop to the ML110 gave about 2 seconds of video outage, but migrating from the ML110 to the HP desktop just resulted in a slow hiccup. My assumption was that this would probably be completely invisible on more modern hardware.

So what does this mean? Microsoft has made Live Migration and Clustering a feature available to everyone, at (almost) no cost. Administrating such a cluster requires Active Directory, and either a WS08R2 server or a Windows 7 machine with RSAT installed.

This means we can finally have decent virtualization features without paying thousands of francs in licensing fees. I hope this makes it possible to create a few virtualization projects for our customers, which are mostly in the small business range.

Hyper-V Server R2 should be available around mid-August, at which i’ll need to rebuild my Ghetto setup here. I’m of course hoping to get some more cash in order to move or internal virtualization setup from a single-host to a SAN-hosted cluster, but somehow i doubt that will happen quickly.

Update:

I’ve played around with Expression Encoder a bit, and created a Video of a Live Migration. I’ve put the probably most boring video on Youtube – Live Migration of Pinball.