In memory of Lukas Beeler

In memory of Lukas Beeler, who passed away on September 15, 2010.

In silent mourning and loving memory, his family and friends keep Lukas’ website available.

Everything changes depending on who walks with us on our journey and who is missing.

WE LINGER ON THIS EARTH LIKE TRAVELERS WHO ARRIVE AT NIGHTFALL AT A SHELTER AND CONTINUE THEIR JOURNEY AT DAWN. WE WALK TOWARDS A DESTINATION FROM SHELTER TO SHELTER, ONE STAGE AFTER THE OTHER.

For each one, the destination holds a different meaning, and each one will use different words to describe it.

We were privileged to walk a part of our journey with Lukas. He has passed the last stage; there we will follow him one day. His gift to each one of us, which he gave so generously, was the time he shared with us. This gift has become even more precious now that he has gone from us.

We hope that all users of this website may find a piece of an answer to their questions on IT related topics, bringing them one step closer to a solution. Save journey!

UEFI continues to haunt me

We have an IBM x3650 M2, that runs a specific business application, using Windows Server 2008 R2 installed in EFI mode. Now, requirements have changed and we need to virtualize this.

Unfortunately, SCVMM 2008 R2′s P2V crashes when run on this machine. disk2vhd can produce a proper VHD from an EFI/UEFI install of Windows Server 2008 R2, but there’s not way of getting it to boot in Hyper-V (i tried a myriad of ways, including several Linux tools that can convert GPT disks to MBR-style disks, got the Windows Boot Manager installed, but it still wouldn’t boot).

So. What now? I’m out of reasonable ideas. I have opened a Microsoft support case regarding the SCVMM 2008 R2 P2V crash on an EFI machine, but i’m not sure i’ll get a quick out of this. If anyone has any ideas on how to get this fixed, i’d be thankful for any replies.

If i ever get a solution that does not include reinstalling everything from scratch, i’ll of course post it.

Update: Here’s the official statement:

There are no workarounds for moving a Windows system with an EFI partition to non-EFI architecture. EFI and Itanium are in lockstep. Classic x86 and x64 cannot boot EFI, and there and is no simple switch back to MBR boot.

Es tut mir Leid, das ich keine besseren Informationen für Sie habe aber das Feedback von unseren Development ist sehr eindeutig das keine Kombination von P2V / GPT bzw. EFI zur Zeit unterstützt wird. Mein Vorschlag wäre, unseren Service Request als “Dokumentations-Bug” für Sie kostenfrei zu schließen. Was halten Sie von meinen Vorschlag?

Bunch of idiots. Their agent shouldn’t crash in this scenario anyway and it should be documented that you can’t migrate machines installed in UEFI modes.

IBM i Access 7.1 installation hangs indefinitively with a Windows Installer Coordinator window

If you’re trying to install IBM i Access 7.1 on a Windows Server 2008 R2 based Remote Desktop Session Host (RDS), formerly known as Terminal Server, you’ll most likely encounter this issue.

A window titled “Windows Installer Coordinator” will pop up behind the IBM i Access 7.1 Installer (hidden until you click on it in the task bar). This “Windows Installer Coordinator” will run indefinitively, without ever successfully installing the application.

Thanks to a helpful guy from IBM Software Support Austria, i now have a solution to this issue. It’s caused by a new feature in WS08R2 RDS.

It’s called Windows Installer RDS Compatibility. If this feature is enabled, IBM i Access 7.1 will not install successfully, and hang at the “Windows Installer Coordinator” window.

To successfully install IBM i Access 7.1 on a Windows Server 2008 R2 Remote Desktop Session host, set the following DWORD registry key to 0:

HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows NT\Terminal Services\TSAppSrv\TSMSI\Enable

It’s possible that not all keys exist – in my case, the TSAppSrv and TSMSI keys didn’t exist yet – you have to create them manually. After creating this key, you can rerun the installation – a reboot is not necessary.

TMG 2010 seems to be still in Beta

Our apprentice is doing an a project for his final exams (IPA). For that, we’ve chosen to replace our current Exchange 2007 Edge with a Forefront TMG 2010 / Exchange 2010 Edge combination.

As the project progressed, we’ve found a few extremely irritating and hard-to-debug issues, which needed my involvement to figure out the root cause and get them resolved, without compromising the exam results.

Be aware that most of the debugging and research here was mostly done by our apprentice, not by myself.

There are several key issues with TMG, that we’ve noticed so far:

IP Blocklist Entries

If IP Blocklist Entries are present in Exchange 2010 Edge, enabling E-Mail Policy Integration will cause TMG to reject all further changes, with the following error message:

Windows Could not Start the "Microsoft Forefront TMG Managed Control" service on Local Computer
Error 0x80070057 : Parameter is incorrect

I’ve found this solution in the TechNet forums. You need to remove all IP Blocklist and Allow List Entries.

Extremely slow boot

Forefront TMG 2010 with Exchange 2010 and FPE 2010 installed will boot extremely slowly, requiring up to 30 Minutes to boot. This issue is caused by the coexistence with Exchange 2010.

Again, i’ve found a solution in the TechNet forums.

You need to set the service Microsoft Exchange Transport and Microsoft Forefront TMG Managed Control to Automatic (Delayed Start). This will reduce the boot time to about 3 minutes.

lsass.exe crashes when creating Edge subscriptions

The next issue we’ve noticed is that while the initial edge subscription worked, the second one didn’t. It crashed lsass.exe, which subsequently caused a bluescreen. Not a very nice experience.

Again, we’ve found a solution on the TechNet forums, and this is getting worse by the minute. The lsass.exe crash can be mitigated by removing all except one SSL certificate – not exactly a good approach since a TMG likely has multiple SSL certificates for publishing a variety of services. But it worked. Except that mailflow didn’t.

Outgoing Mailflow doesn’t work with TMG 2010

Of course, stuff wasn’t working yet. While incoming mailflow now worked flawlessly, outgoing mailflow didn’t – mails where stuck in the queue with “Primary Target IP Address responded with 421 Unable to establish connection”.

We’ve tried to look at this, but everything seemed alright – but we couldn’t modify any connectors on the Edge server – TMG prevented this, and thus we had no Verbose logging from the Receive Connectors. Changing the configuration in the Exchange Edge console resulted in the following error message:

Forefront TMG detected changes in Microsoft Exchange Server or Microsoft Forefront Protection configuration, and reapplied the e-mail policy configuration on server

So i’m not supposed to do that. The TMG console didn’t give me the option of enabling Verbose logging. We were stumped.

Luckily, further research showed that one could disable the integration between the Exchange Edge role and Forefront TMG – this was mentioned on this TechNet forums post.

After disabling this integration, i was able to allow Verbose logging. Which didn’t help at all, since the Exchange 2010 HT just wouldn’t show up in them, suspecting a deeper issue.

At that point, we’ve checked the receive connectors that were created by Forefront – and the internal Receive Connector didn’t allow Exchange Server Authentication. After setting that to enabled, we were finally able to send mail successfully using the Exchange Edge services.

Final words

Forefront TMG 2010 still seems to be in Beta. The integration with Exchange 2010 doesn’t work as nicely as it should. I hope these things get fixed soon with Hotfixes for TMG 2010. Until then, we’ve found workarounds for all of these issues.

I’m publishing this article as quickly as i can, because i’m most likely not the only one with these issues.

TechDays 2010 Basel will be the last that I have attended

Today marks the second and last day of TechDays 2010 in Basel.

I have attended TechDays 05, 06, 07, 08 and 2010. While i’ve always had something complain about, there was always something to gain from attending. Not this time.

When reading this, keep in mind that i’ve only attended IT Pro Sessions (with the exception of the Windows Phone 7 Developer Briefing).

There were to many things that have gone wrong.

  • The keynote was boring and it didn’t even remotely have anything to do with the job of an IT Pro or a Developer. The keynote speaker also used a MacBook with OS X/Keynote.app. Seriously.
  • The food was worse than what I got to eat at my Berufsschule (which wasn’t very good).
  • The long lines were still there – waiting 20 minutes in line for bad food isn’t my idea of spending the day. They should’ve solved this problem by now.
  • No more English talks. Why? I think we could use some experts.
  • All the talks were very basic. No In-Depth stuff. Nothing to learn.
  • I don’t want a basic talk about what OCS 2007 R2 can do. We’ve been using this for two years. It’s old news. Talk about Wave 14.
  • Giving a basic intro on what SCVMM and Hyper-V are is not an IT Pro track – these technologies have been out since years and everyone that’s interested will already know those basics
  • Make sure your stuff works. 75% of the demos did not work. Most of them because of bad internet connectivity. Yeah, i guess moving all the stuff to the “cloud” is a good idea.

The location and the whole ship theme was okay though. The evening event was also nice, and they did have good food there (different catering organization). Not sure what some danish bloke was doing there yelling “in the cloud” about 50 times. I wanted to see some chair-throwing.

Another interesting tidbit: The number of iPhones at the event. I’ve seen more iPhones than WinMo phones.

So, did you attend TechDays? What did you think about it?

My OCZ Vertex 120GB is dying

I currently have two SSDs – an OCZ Vertex 120GB bought before Intel priced it’s SSDs competitively (April 2009) and an Intel X25-M G2 160GB i bought at launch (September 2009). The OCZ Vertex is the one i use in my work laptop, and the Intel X25-M G2 is the one i use in my system at home. Both see extensive use, and both have always been used with Windows 7, which is TRIM-enabled.

The most important thing between the laptop and the desktop is that i’m using BitLocker on the laptop, which might have an influence on things. I’ve always been using BitLocker on the SSD, so it would seem strange that this is now suddenly an issue.

I’ve always been aggressive about SSD firmware updates, after a good backup. I’ve upgraded both the Vertex and the Intel drives to be TRIM capable as soon as the respective firmware was out.

Unfortunately, a few days after using the OCZ Vertex in my new laptop, it started to have serious hickups – during which no IO would take place (perfmon disk queue shooting up to 50). During this time, the HDD light on the laptop is not lit.

I’ve tried to make sure that this issue was related to the SSD, so i ran HDTune benchmark:

This looked bad. Further investigation showed that there was a new Firmware out – 1.5. I’ve upgrade to Firmware 1.5, which supposedly had a Garbage Collection and TRIM support. After upgrading to 1.5, the hickups became much worse – the laptop needed about an hour just to boot up.

After looking at and posting on the OCZ support forum, i was told that i’d need to wait for Garbage collection to kick in. I let my laptop sit for a night, during which it crashed and the subsequent reboot was stuck on a “No harddisk found” message from the BIOS. Things looked bleak.

Further replies on the OCZ support forum requested that i do a sanitary erase, which would reset the disk to pristine performance levels (and delete all the data on it).

Unfortunately, the machine was too slow to run a Windows Complete PC Backup (wasn’t finished after 4 hours in). Fortunately, all the important data on my laptop is backed up using the Client Protection of DPM 2010, meaning all i had to do was reinstall my apps and i’d be good to go.

After reinstalling Windows 7, i installed the most important apps and then reenabled BitLocker protection – during which the hickups started happening again. The laptop would sometimes hang for 20-30 seconds, and then continue on on it’s merry way.

At this point, i went to sleep and let the laptop idle at the boot selection screen, so that the garbage collection could do it’s magic.

And now here we are, 8 hours later. While the read performance using HD Tune is nowwhere near as bad as it was before the sanitary erase, the write performance is stil abysmal.

For Comparison, here’s my Intel X25-M G2:

What now? I think i will RMA the drive. It’s the only choice i have left at this point.

If anyone has a better idea, give me a whirl.

Lenovo ThinkPad T510

Since December 2008, i’ve used my ThinkPad W500 as my work laptop. We’ve bought this as part of a promotion package.

The W500 i had had a 15.4″ 1920×1200 Panel, which wasn’t too great. While the high fidelity was certainly nice, the screen was very, very dark. It could only be used indoors, and required you to darken the room on sunny days.

Today i’ve had the chance to upgrade from the W500 to a T510, which i did. So far, i’m very much impressed with the changes Lenovo has do to this device. The W500 is running Windows 7 Enterprise x64.

  • New controls for volume, microphone mute. Much easier to use than before
  • New bigger and multitouch capable touchpad. As i prefer the touchpad over the TrackPoint, this is something that helps me tremendously
  • Integrated Camera and eSATA connectivity
  • Improved connectivity layout

There’s only one thing that i don’t like very much right now – the redesigned keyboard. As part of my job i deal with IBM’s IBM i platform, which still makes use the Function keys – which have all been shifted to the right for one key. So i regularly press F3 instead of F4, but chances are i will get used to it.

There’s one thing that worked very well – moving my Windows installation from the W500 to the T510. I’ve disabled Bitlocker protection, removed the OCZ Vertex SSD from the W500, placed it into the T510, booted it up, Windows installed several new drivers. Then, i installed the Intel LAN drivers from an USB stick, rebooted once more and installed the rest of the necessary drivers from Lenovo’s driver matrix. The whole process was done in less than half an hour, and reenabling Bitlocker protection was a breeze.

Windows 7 automatically reactivated by contacting our KMS servers, and i’ve had to reactivate my Office 2010 Beta manually, which also worked flawlessly.

While this portability is great (and also existed with Vista), it’s something I was able todo with Linux back in 2004 (assuming of course that the kernel had the storage drivers you required).

I’ve been using ThinkPads exclusively since 2004 – my first ThinkPad was an R51 and my first new laptop (my first laptop was a Compaq Armada i’ve bought used for 50.- CHF). When Lenovo took over the brand, i wasn’t to sure what to think of it, but having gone through several iterations of ThinkPad devices now (R51, T60, W500 and now the T510) i can see that Lenovo is commited to provide further well built, high performance devices.

Both the T60 and the W500 are still in service, neither of them are broken. The T60 is used by my apprentice and around 3 or 4 years old. We’ve replaced the Mouse and Keyboard to mitigate the wear and tear of several 40 hour work weeks on the device, but aside from that it stills works great.

Cablecom did it again

Another Cablecom outage – this time, it was nation wide and affected both Business and end user accounts.

Interestingly, the Hotline wasn’t reachable either – busy signal, Swisscom text “Leitung gestört” or simply “Call Failed”.

Lasted from 19:33 to 20:30, but it looks like everything is back online now.

Hyper-V 2008 R2 and Linux guests

I’m still running a Linux box to run a legacy business app that’s about to be replaced, and runs a few legacy VPNs. Setup ages ago, when i didn’t have the experience i have today, the setup on the machine was a mess – originally installed using testing of what was-to-be Debian 3.1 with several custom packages (Postfix, Apache, OpenVPN, etc.), this has been overdue for some fixup work for quite some time.

As a disclaimer, i realize that Debian in any version isn’t a supported OS on Hyper-V R2 – i just want to tell of my experiences with this unsupported configuration.

The hardware, an aging IBM xSeries 306m with a Pentium 4 CPU wasn’t getting any younger and after a drive failure about half a year ago that lead to a system crash (No data loss though – it just crashed the machine, that’s Software RAID for you), it was finally time to modernize this.

The plan is to consolidate all our DMZ workloads (ISA, OCS Edge, XMPP Gateway, Exchange Edge) on Hyper-V 2008 R2 and doing the trickiest part first seemed like a good idea.

So i created a new VM using SCVMM 2008 R2, selected Other Linux 32bit as the guest OS, inserted a Debian 5.0 netboot CD and that’s where the problems already started. While the installation worked well in general, the Framebuffer used by the Debian installed is awfully slow. So it took me about half an hour just to get the install done (on a 5GB partition of the 80GB VHD).

After finishing the installation, i formatted the rest of the disk appropiately and then used rsync to transfer the machine contents over. A short bit after reconfiguring Grub, i could choose to boot either the transferred OS with it’s kernel, or the Debian 5 rescue system i installed alongside.

Booting the transferred system worked well enough, but the tulip driver wasn’t compiled into that (custom) kernel and building the module failed. So i read up a bit, and realized that the newest kernel (2.6.32.8) shipped with experimental Hyper-V VMbus drivers, that allowed synthetic NICs to be used.

I tried to compile the kernel after chrooting into the old installation, but it failed because gcc was too old. Not to worry, i compiled it in the rescue system, but couldn’t install the dpkg that make-kpkg created. So i installed it manually, which worked pretty well.

One reboot later, i was back in business with the extremely verbose Hyper-V drivers cluttering up dmesg, but the Synthetic NICs showed up as seth0 – seth2. After quickly changing all the necessary configuration files, everything was working.

After a bit of more testing, i disconnected the physical machine from the network and plugged the VM into the production VLANs.

I tested everything thoroughly and didn’t find any issues. Sent out an information mail and continued on my merry way.

Half an hour later, i decided to do a quick systems check again – and i realized that the external interface (seth2 in this case) wasn’t working anymore. tcpdump showed no packets being received and other machines in the same VLANs didn’t see any answers to their ARP requests either. So i rebooted the VM, and everything was working again. No error messages of any kind, neither in dmesg nor in the system logs or on the Hyper-V host.

Hoping this was just a fluke, i waited until it happened again – which it did, roughly 10 minutes later. So i decided to skip on the synthetic devices and go with emulated NICs and the tulip driver.

Everything came back up, but i couldn’t ping any devices on the eth0 VLAN from the start, but the other two interfaces worked.

After a few more tries, i arrived at a configuration that has now been stable for 4 hours and 26 minutes, which sounds good so far. For this, i configured a single synthetic NIC that i used as a replacement for the non-working eth0 and three tulip NICs (of which the first was unused).

There are other things that also worry me:

Every reboot of the Linux machine created the following event log entry on the Hyper-V host:


'LINUX' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID [])

Loading the synthetic NIC drivers logs the following in the event log on the Hyper-V host:


Networking driver on 'LINUX' loaded but has a different version from the server. Server version 3.2 Client version 0.2 (Virtual machine ID []). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.

Loading the synthetic NIC drivers also logs all this on the Linux side of things:


VMBUS_DRV: Vmbus initializing.... current log level 0x1f1f0006 (1f1f,6)
VMBUS: +++++++ Build Date=Feb 17 2010 12:37:00 +++++++
VMBUS: +++++++ Build Description=Version 2.0 +++++++
VMBUS: +++++++ Vmbus supported version = 13 +++++++
VMBUS: +++++++ Vmbus using SINT 2 +++++++
VMBUS: Windows hypervisor detected! Retrieving more info...
VMBUS: Vendor ID: Microsoft Hv
VMBUS: Interface ID: Hv#1
VMBUS: OS Build:7600-6.1-16-0.16485
VMBUS: Hypercall page VA=f80c9000, PA=0x36afe000
VMBUS_DRV: irq 0x5 vector 0x35
VMBUS: SynIC version: 1
VMBUS: Vmbus connected!!
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={c5295816-f63a-4d5f-8d1a4daf999ca185}
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
hv_netvsc: module is from the staging directory, the quality is unknown, you have been warned.
NETVSC_DRV: Netvsc initializing....
VMBUS_DRV: child driver (f80dc570) registering - name netvsc
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={32412632-86cb-44a2-9b5c50d1417354f5}
VMBUS_DRV: child device (f73a8634) registered
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a}
VMBUS_DRV: child device (f73a5a34) registered
VMBUS: Channel offer notification - child relid 1 monitor id 0 allocated 1, type {32412632-86cb-44a2-9b5c50d1417354f5} instance {00000000-0000-8899-0000000000000000}
VMBUS_DRV: generating uevent - VMBUS_DEVICE_CLASS_GUID={f8615163-df3e-46c5-913ff2d2f965ed0e}
VMBUS_DRV: device object (f73a5ee4) set to driver object (f80dc5c0)
VMBUS: Channel offer notification - child relid 2 monitor id 255 allocated 0, type {cfa8b69e-5b4a-4cc0-b98b8ba1a1f3f95a} instance {58f75a6d-d949-4320-99e1a2a2576d581c}
VMBUS: Channel offer notification - child relid 9 monitor id 1 allocated 1, type {f8615163-df3e-46c5-913ff2d2f965ed0e} instance {9d44a66e-4b09-41d5-80d807ae24bf537d}
VMBUS: channel f73aac00 open success!!
NETVSC: *** NetVSC channel opened successfully! ***
NETVSC: Sending NvspMessageTypeInit...
NETVSC: NvspMessageTypeInit status(1) max mdl chain (34)
NETVSC: Sending NvspMessage1TypeSendNdisVersion...
NETVSC: Establishing receive buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendReceiveBuffer...
NETVSC: Receive sections info (count 1, offset 0, endoffset 1048000, suballoc size 1600, num suballocs 655)
NETVSC: Establishing send buffer's GPADL...
NETVSC: Sending NvspMessage1TypeSendSendBuffer...
NETVSC: *** NetVSC channel handshake result - 0 ***
NETVSC: Device 0xf6552e80 mac addr 00155d031a09
NETVSC: Device 0xf6552e80 link state up
VMBUS_DRV: child device (f73a5e34) registered

So, it works. But not without troubles. I’ve still got the physical machine to fall back on, but i sure hope Microsoft will get this to work better.

These issues are the reason why i decided to deploy my private server using ESXi instead of Hyper-V – because i need both Linux and Windows guests.

DPM 2010 hangs at replica creation when backing up Hyper-V VMs

I’ve been playing with DPM 2010 and SCVMM 2008 R2, planning for our new development lab.

I’ve setup a new Hyper-V server on a x3650 M2 (using server core) – i’ve also installed the latest Broadcom NetXtreme II drivers, all the firmware updates, all the best practices you do.

Setting up the machine, transferring VMs from another host (using BITS) worked well and fast, no issues.

And then i installed the DPM agent, started a backup. Two hours later, it was still stuck at “Replica creation in progress”.

I tried reading through the DPM agent logs, through the DPM server logs, looked if DPM created shadow copies (using vssadmin list shadows).

After two hours of fruitless searching (which included restarting everything), i wasn’t any further to a solution.

Well, backup wasn’t working right, but this was just a testing environment, so i decided to do other stuff.

A while later, i ran netstat -t to lookup connections – and also realized that TCP Chimney Offloading was still active. So i disabled it using netsh int tcp set global chimney=disabled. Just a few seconds later, the utilization of the management network adapter jumped to 100% and 5 minutes later, all the VMs were replicated to the DPM server.

So, if you’re having issues with DPM backups being stuck, check the status of your network offloading.