Lukas Beeler's IT Blog

We have an IBM x3650 M2, that runs a specific business application, using Windows Server 2008 R2 installed in EFI mode. Now, requirements have changed and we need to virtualize this.

Unfortunately, SCVMM 2008 R2′s P2V crashes when run on this machine. disk2vhd can produce a proper VHD from an EFI/UEFI install of Windows Server 2008 R2, but there’s not way of getting it to boot in Hyper-V (i tried a myriad of ways, including several Linux tools that can convert GPT disks to MBR-style disks, got the Windows Boot Manager installed, but it still wouldn’t boot).

So. What now? I’m out of reasonable ideas. I have opened a Microsoft support case regarding the SCVMM 2008 R2 P2V crash on an EFI machine, but i’m not sure i’ll get a quick out of this. If anyone has any ideas on how to get this fixed, i’d be thankful for any replies.

If i ever get a solution that does not include reinstalling everything from scratch, i’ll of course post it.

Update: Here’s the official statement:

There are no workarounds for moving a Windows system with an EFI partition to non-EFI architecture. EFI and Itanium are in lockstep. Classic x86 and x64 cannot boot EFI, and there and is no simple switch back to MBR boot.

Es tut mir Leid, das ich keine besseren Informationen für Sie habe aber das Feedback von unseren Development ist sehr eindeutig das keine Kombination von P2V / GPT bzw. EFI zur Zeit unterstützt wird. Mein Vorschlag wäre, unseren Service Request als “Dokumentations-Bug” für Sie kostenfrei zu schließen. Was halten Sie von meinen Vorschlag?

Bunch of idiots. Their agent shouldn’t crash in this scenario anyway and it should be documented that you can’t migrate machines installed in UEFI modes.

If you’re trying to install IBM i Access 7.1 on a Windows Server 2008 R2 based Remote Desktop Session Host (RDS), formerly known as Terminal Server, you’ll most likely encounter this issue.

A window titled “Windows Installer Coordinator” will pop up behind the IBM i Access 7.1 Installer (hidden until you click on it in the task bar). This “Windows Installer Coordinator” will run indefinitively, without ever successfully installing the application.

Thanks to a helpful guy from IBM Software Support Austria, i now have a solution to this issue. It’s caused by a new feature in WS08R2 RDS.

It’s called Windows Installer RDS Compatibility. If this feature is enabled, IBM i Access 7.1 will not install successfully, and hang at the “Windows Installer Coordinator” window.

To successfully install IBM i Access 7.1 on a Windows Server 2008 R2 Remote Desktop Session host, set the following DWORD registry key to 0:

HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows NT\Terminal Services\TSAppSrv\TSMSI\Enable

It’s possible that not all keys exist – in my case, the TSAppSrv and TSMSI keys didn’t exist yet – you have to create them manually. After creating this key, you can rerun the installation – a reboot is not necessary.

Our apprentice is doing an a project for his final exams (IPA). For that, we’ve chosen to replace our current Exchange 2007 Edge with a Forefront TMG 2010 / Exchange 2010 Edge combination.

As the project progressed, we’ve found a few extremely irritating and hard-to-debug issues, which needed my involvement to figure out the root cause and get them resolved, without compromising the exam results.

Be aware that most of the debugging and research here was mostly done by our apprentice, not by myself.

There are several key issues with TMG, that we’ve noticed so far:

IP Blocklist Entries

If IP Blocklist Entries are present in Exchange 2010 Edge, enabling E-Mail Policy Integration will cause TMG to reject all further changes, with the following error message:

Windows Could not Start the "Microsoft Forefront TMG Managed Control" service on Local Computer
Error 0x80070057 : Parameter is incorrect

I’ve found this solution in the TechNet forums. You need to remove all IP Blocklist and Allow List Entries.

Extremely slow boot

Forefront TMG 2010 with Exchange 2010 and FPE 2010 installed will boot extremely slowly, requiring up to 30 Minutes to boot. This issue is caused by the coexistence with Exchange 2010.

Again, i’ve found a solution in the TechNet forums.

You need to set the service Microsoft Exchange Transport and Microsoft Forefront TMG Managed Control to Automatic (Delayed Start). This will reduce the boot time to about 3 minutes.

lsass.exe crashes when creating Edge subscriptions

The next issue we’ve noticed is that while the initial edge subscription worked, the second one didn’t. It crashed lsass.exe, which subsequently caused a bluescreen. Not a very nice experience.

Again, we’ve found a solution on the TechNet forums, and this is getting worse by the minute. The lsass.exe crash can be mitigated by removing all except one SSL certificate – not exactly a good approach since a TMG likely has multiple SSL certificates for publishing a variety of services. But it worked. Except that mailflow didn’t.

Outgoing Mailflow doesn’t work with TMG 2010

Of course, stuff wasn’t working yet. While incoming mailflow now worked flawlessly, outgoing mailflow didn’t – mails where stuck in the queue with “Primary Target IP Address responded with 421 Unable to establish connection”.

We’ve tried to look at this, but everything seemed alright – but we couldn’t modify any connectors on the Edge server – TMG prevented this, and thus we had no Verbose logging from the Receive Connectors. Changing the configuration in the Exchange Edge console resulted in the following error message:

Forefront TMG detected changes in Microsoft Exchange Server or Microsoft Forefront Protection configuration, and reapplied the e-mail policy configuration on server

So i’m not supposed to do that. The TMG console didn’t give me the option of enabling Verbose logging. We were stumped.

Luckily, further research showed that one could disable the integration between the Exchange Edge role and Forefront TMG – this was mentioned on this TechNet forums post.

After disabling this integration, i was able to allow Verbose logging. Which didn’t help at all, since the Exchange 2010 HT just wouldn’t show up in them, suspecting a deeper issue.

At that point, we’ve checked the receive connectors that were created by Forefront – and the internal Receive Connector didn’t allow Exchange Server Authentication. After setting that to enabled, we were finally able to send mail successfully using the Exchange Edge services.

Final words

Forefront TMG 2010 still seems to be in Beta. The integration with Exchange 2010 doesn’t work as nicely as it should. I hope these things get fixed soon with Hotfixes for TMG 2010. Until then, we’ve found workarounds for all of these issues.

I’m publishing this article as quickly as i can, because i’m most likely not the only one with these issues.

Today marks the second and last day of TechDays 2010 in Basel.

I have attended TechDays 05, 06, 07, 08 and 2010. While i’ve always had something complain about, there was always something to gain from attending. Not this time.

When reading this, keep in mind that i’ve only attended IT Pro Sessions (with the exception of the Windows Phone 7 Developer Briefing).

There were to many things that have gone wrong.

• The keynote was boring and it didn’t even remotely have anything to do with the job of an IT Pro or a Developer. The keynote speaker also used a MacBook with OS X/Keynote.app. Seriously.
• The food was worse than what I got to eat at my Berufsschule (which wasn’t very good).
• The long lines were still there – waiting 20 minutes in line for bad food isn’t my idea of spending the day. They should’ve solved this problem by now.
• No more English talks. Why? I think we could use some experts.
• All the talks were very basic. No In-Depth stuff. Nothing to learn.
• I don’t want a basic talk about what OCS 2007 R2 can do. We’ve been using this for two years. It’s old news. Talk about Wave 14.
• Giving a basic intro on what SCVMM and Hyper-V are is not an IT Pro track – these technologies have been out since years and everyone that’s interested will already know those basics
• Make sure your stuff works. 75% of the demos did not work. Most of them because of bad internet connectivity. Yeah, i guess moving all the stuff to the “cloud” is a good idea.

The location and the whole ship theme was okay though. The evening event was also nice, and they did have good food there (different catering organization). Not sure what some danish bloke was doing there yelling “in the cloud” about 50 times. I wanted to see some chair-throwing.

Another interesting tidbit: The number of iPhones at the event. I’ve seen more iPhones than WinMo phones.

So, did you attend TechDays? What did you think about it?

I currently have two SSDs – an OCZ Vertex 120GB bought before Intel priced it’s SSDs competitively (April 2009) and an Intel X25-M G2 160GB i bought at launch (September 2009). The OCZ Vertex is the one i use in my work laptop, and the Intel X25-M G2 is the one i use in my system at home. Both see extensive use, and both have always been used with Windows 7, which is TRIM-enabled.

The most important thing between the laptop and the desktop is that i’m using BitLocker on the laptop, which might have an influence on things. I’ve always been using BitLocker on the SSD, so it would seem strange that this is now suddenly an issue.

I’ve always been aggressive about SSD firmware updates, after a good backup. I’ve upgraded both the Vertex and the Intel drives to be TRIM capable as soon as the respective firmware was out.

Unfortunately, a few days after using the OCZ Vertex in my new laptop, it started to have serious hickups – during which no IO would take place (perfmon disk queue shooting up to 50). During this time, the HDD light on the laptop is not lit.

I’ve tried to make sure that this issue was related to the SSD, so i ran HDTune benchmark:

This looked bad. Further investigation showed that there was a new Firmware out – 1.5. I’ve upgrade to Firmware 1.5, which supposedly had a Garbage Collection and TRIM support. After upgrading to 1.5, the hickups became much worse – the laptop needed about an hour just to boot up.

After looking at and posting on the OCZ support forum, i was told that i’d need to wait for Garbage collection to kick in. I let my laptop sit for a night, during which it crashed and the subsequent reboot was stuck on a “No harddisk found” message from the BIOS. Things looked bleak.

Further replies on the OCZ support forum requested that i do a sanitary erase, which would reset the disk to pristine performance levels (and delete all the data on it).

Unfortunately, the machine was too slow to run a Windows Complete PC Backup (wasn’t finished after 4 hours in). Fortunately, all the important data on my laptop is backed up using the Client Protection of DPM 2010, meaning all i had to do was reinstall my apps and i’d be good to go.

After reinstalling Windows 7, i installed the most important apps and then reenabled BitLocker protection – during which the hickups started happening again. The laptop would sometimes hang for 20-30 seconds, and then continue on on it’s merry way.

At this point, i went to sleep and let the laptop idle at the boot selection screen, so that the garbage collection could do it’s magic.

And now here we are, 8 hours later. While the read performance using HD Tune is nowwhere near as bad as it was before the sanitary erase, the write performance is stil abysmal.

For Comparison, here’s my Intel X25-M G2:

What now? I think i will RMA the drive. It’s the only choice i have left at this point.

If anyone has a better idea, give me a whirl.

Another Cablecom outage – this time, it was nation wide and affected both Business and end user accounts.

Interestingly, the Hotline wasn’t reachable either – busy signal, Swisscom text “Leitung gestört” or simply “Call Failed”.

Lasted from 19:33 to 20:30, but it looks like everything is back online now.

I’ve been playing with DPM 2010 and SCVMM 2008 R2, planning for our new development lab.

I’ve setup a new Hyper-V server on a x3650 M2 (using server core) – i’ve also installed the latest Broadcom NetXtreme II drivers, all the firmware updates, all the best practices you do.

Setting up the machine, transferring VMs from another host (using BITS) worked well and fast, no issues.

And then i installed the DPM agent, started a backup. Two hours later, it was still stuck at “Replica creation in progress”.

I tried reading through the DPM agent logs, through the DPM server logs, looked if DPM created shadow copies (using vssadmin list shadows).

After two hours of fruitless searching (which included restarting everything), i wasn’t any further to a solution.

Well, backup wasn’t working right, but this was just a testing environment, so i decided to do other stuff.

A while later, i ran netstat -t to lookup connections – and also realized that TCP Chimney Offloading was still active. So i disabled it using netsh int tcp set global chimney=disabled. Just a few seconds later, the utilization of the management network adapter jumped to 100% and 5 minutes later, all the VMs were replicated to the DPM server.

So, if you’re having issues with DPM backups being stuck, check the status of your network offloading.

The old DL140 running Debian Linux finally died this Monday, due to a hard drive error which Linux software raid couldn’t deal with. Luckily, the second disk survived and i didn’t have to test my disaster recovery strategy.

If you’re reading this, this blog is now hosted on Windows Server 2008 R2 Web Edition (Yay NFR promotions!). There may still be some kinks that have to be worked out, because this was quite a rush job. Leave a comment if you find any issues.

The company i’m working for, Acommit AG, is hiring again.

Currently, we’re looking for:

Project Manager (PDF)
Sales (PDF)

Exchange 2010 will be out soon, and i’ve been preparing for the migration. One of the more important parts is that you will need to have both Exchange 2007 and Exchange 2010 client access servers accessible from the Internet.

If you’re following the recommended deployment method for Exchange 2007, you’ll already be using a SAN certificate in order to publish AutoDiscovery and OWA. For coexistence of Exchange 2007 and Exchange 2010, an additional name will need to be added to your SAN certificate.

With most CAs, this is a pretty straightforward process that can be done using their web interface, since the private key doesn’t need to be touched. After modifying this, you will get a new .crt file containing the certificate, but no private key (which is correct).

However, importing this into Exchange 2007 using Import-ExchangeCertificate doesn’t work – Windows won’t know which private key is associated with the newly imported certificate. When you try to use Enable-ExchangeCertificate, you will receive the following error message:

Enable-ExchangeCertificate : The certificate with thumbprint 1234 was found but is
not valid for use with Exchange Server (reason: PrivateKeyMissing).

I searched high and low on how to replace a certificate without touching the private key, but i didn’t find anything. So i turned to the community for support – MCSEBoard.de is an excellent Windows community for those who speak German.

Unfortunately, noone knew an easy way either – the suggestion was to use OpenSSL to create a new keystore.

This was rather easy, but i didn’t find any guides on the net on how to do this, so i’m publishing this here in the hope that it will help others with the same issue.

• First, you need to export the key including the private key using the Windows certificate manager. Open an elevated MMC, add the Certificate snap-in and focus on the Computer certificate. Click “Personal”, and then export the certificate with the private key.
• Download and Install OpenSSL for Windows
• Issue the following command: openssl pkcs12 -in mykey.pfx > out.txt
• Open out.txt using an LF-aware text editor, such as Notepad++. Save the PRIVATE KEY part to a textfile called key.pem.
• Save the certificate to a file called cert.crt
• Issue to the following command: openssl pkcs12 -export -in cert.crt -inkey key.pem -out newcert.p12
• Copy the newly created newcert.p12 to the Exchange server.
• Open PowerShell and run the following command: $secureString = ConvertTo-SecureString "blubb" -AsPlainText -Force – Replace “blubb” with the Passphrase you used in the step before
• Run Import-ExchangeCertificate -path newcert.p12 -pass $secureString to import the certificate back into Exchange
• The rest is as usual – use Enable-ExchangeCertificate to enable the certificate.

And that’s it. It might be a bit cumbersome – and i really hope that there is an easier way to to this. If you know, let me know so i can update this page.