Lukas Beeler's IT Blog

So, IBM’s x3650 M2 have been out for some time and in the meantime i’ve deployed three of them – two with SBS 2008, and one with Windows Server 2008. No pictures, since the camera in the office is broken.

The x3650 M2 comes with a new IMM (Integrated Management Module) that replaces both the BMC and the RSA II Slimline. In order to get remote KVM capability, a physical licensing key must be installed into the server. Standard features like remote power on/off are available without the licensing key – which is more expensive than the physical RSA II slimline adapter. With the IMM also comes UEFI, as a replacement to the aging BIOS.

The x3650 M2 also gets rid of the Adaptec ServeRAID 8k controllers, and introduces us to the ServeRAID 10 series manufactured by LSI. The ServeRAID 8k series have been plagued by several extremely heavy bugs that never caused me loss of data (but several other people), but nevertheless cost me a lot of my nerves.

The two standard RAID controllers in the x3650 M2 are the ServeRAID BR10i, which is the baseline controller without BBWC and without support for RAID5. The ServeRAID MR10i is the better model, which comes with 256MB BBWC and support for RAID5/RAID6. Unlike the ServeRAID 8k/8k-l story, these are entirely standalone controllers, that are located in a special daughterboard position with a standard PCI-e x8 interface.

Both controllers support only 8 drives – in order to get 12 drives, you need a special enabler kit that comes with a SAS expander hard and several of other stuff that doesn’t look all that trivial. I’ve used such a kit, and so i can’t comment on how it works exactly.

The power supplies have gotten a lot smaller, the server seems to look a lot more organized, the 2.5″ SAS HDDs are no longer as finicky as they were in the x3650 and now fit very well into their slots, the Lightpath diagnostics panel now looks like it belongs to an expensive server and locks into place securely.

Of course, all the usual changes that come with the new Nehalem based Xeons, triple channel DDR-3 memory, both processors needed for using more than 8 slots, using to many memory modules will downgrade the speed, etc.

So much for the general rundown – now for my assorted observations:

• The ServeRAID BR10i seems to be a slightly newer variant than the SAS RAID Controllers found in the x3250 and x3250 M2. The configuration interface is simple, but it works reasonably well.
• The ServeRAID MR10i is a controller i haven’t dealt with before. It does not offer a standard character based interface for configuration, only a graphical interface called WebBIOS. It’s completely awful and half done – half of the buttons have no text on them, every button press requires several seconds until something happens. Configuring a controller with this interface requires you to guess actions based on the manual, since they aren’t labeled onscreen.
• As usual, both LSI based controllers use the MegaRAID management software. Compared to the old Adaptec software, it’s really awful. It runs extremely slow – even on these new servers, is much more complicated than the old ServeRAID software and offers less options in terms of notification.
• The IMM webinterface has gotten even slower than the already slow RSA II interface. Web2.0 style “loading” icons have been added, but viewing the status screen can take up to a minute now. This is retarded, and clearly a step back. At least IMM standard now comes with every server.
• The IMM’s KVM capability have gotten a lot better. Instead of a java applet running in the browser, a java application is launched using java webstart. While IMM itself is slow, the remote KVM capability is actually very fast, and even works with decent speed through a VPN connection.
• While the ServeRAID 8k with newer firmware usually spent 2-3 minutes looking for the drives, the new LSI based ServeRAID 10 series now only takes a few seconds. This is compensated by UEFI which now takes roughly 2 minutes instead of the 10 seconds the old BIOS took. With this, IBM is successfully keeping the server at roughly 4 minutes until OS boot
• The IMM connects to the OS using an USB LAN interface. This is a real problem on Windows, since it confuses the Windows Firewall (switching it into “Public network” mode) and the Windows DNS client. Install the driver and disable it in Windows. Ensure you never enable it on DCs! Run the IMM firmware updates from CD. Clearly a step back.
• The UEFI configuration screens act a lot slower than the old BIOSes. But the options available are decent.
• You can install Windows Server 2008 Standard in UEFI mode. I did that on our internal x3650 M2 which is going to run our WebFOCUS deployment
• You cannot install SBS 2008 in UEFI mode. It will work fine with legacy BIOS emulation, though.
• The IMM can’t send alert messages to email addresses with a – (Dash) in them. Retarded.
• The machine is extremely silent. Compared to the Power 520, which will kill your ears within minutes, they’re a blessing.
• Just like the RSA II interface, the IMM web interface has a tendency to lock up randomly and stop working. Requires a physical power cycle on the server to fix.

That’s it for now. Lot’s of negative stuff in here, but the machines are actually extremely good performers. I hope that IBM will fix those outstanding bugs soon.

I’ve just migrated our internal OCS 2007 setup to OCS 2007 R2. Yeah, i’m very late at this.

Everything worked, but LiveMeeting when using the Edge server. It worked fine internally, or when a VPN connection was established. The LiveMeeting Error Log showed me exactly what failed, but it took my almost half an hour to figure out why it was failing.

[P] SEQ#16,placeware::SslSocket::connectInternal::TLSNegotiationTimer stop,112029,,
[D] [X-PSOM] SslSocket::connect end OK
[D] [X-PSOM] TunnelSocket::connect ProxyHeader sent.
[I] [X-PSOM] SSLTunnelStream: Established SSL Tunnel Stream to hor-ocsgw-01.acommit.ch
[I] [X-PSOM] Forwarded TCP probe succeeded
[P] SEQ#14,placeware::ServerInfo::ForwardedTcpProbeThread::run::ForwardedTcpProbeTimer stop,145082,,
[I] [X-PSOM] Best mode for Client RPC is : 1
[I] [X-PSOM] Best mode is fwdtls. Reusing stream in probe.
[I] [X-PSOM] PWS Handshake sent.
[E] [X-PSOM] placeware::Socket::readWSAGetOverlappedResult failed, error = 10054
[E] [X-PSOM] Socket error while reading.
[E] [X-PSOM] SslSocket::close: socket is not connected

So, looks good at first. And then it fails. No log entry on the OCS Edge, no entry on the OCS Standard.

I figured out the solution when rechecking my entire configuration – i misconfigured the external Edge server hostname on the Standard Server.

Fixing the issue is easy:

• Log on OCS Standard Server
• Right click on Pool – Properties – Web Conference -Web Conference Edge Server.
• Then, enter the correct external host name. You’ll find this name in the Edge server configuration.

The dump then reads like this:

[P] SEQ#16,placeware::SslSocket::connectInternal::TLSNegotiationTimer stop,83410,,
[D] [X-PSOM] SslSocket::connect end OK
[D] [X-PSOM] TunnelSocket::connect ProxyHeader sent.
[I] [X-PSOM] SSLTunnelStream: Established SSL Tunnel Stream to hor-ocsgw-01-1.acommit.ch
[I] [X-PSOM] Forwarded TCP probe succeeded
[P] SEQ#14,placeware::ServerInfo::ForwardedTcpProbeThread::run::ForwardedTcpProbeTimer stop,122853,,
[I] [X-PSOM] Best mode for Client RPC is : 1
[I] [X-PSOM] Best mode is fwdtls. Reusing stream in probe.
[I] [X-PSOM] PWS Handshake sent.
[I] [X-PSOM] Received PWS Handshake.

For several months, i’ve had a problem on a Hyper-V host described WS08 and the black screen of waiting. Basically, the machine boots up, hangs 50 minutes being completely unresponsive, and then goes on working perfectly for weeks.

The problem was resolved (temporarily) by deleting shadow copies, but it still exists. As i’ve had time this weekend to investigate this closely, i’m pretty sure that i found the root cause of the problem, but i have no solution yet. Remember, this is all just a theory i cooked up – i’m putting this information out there in case anyone else has a similar problem.

My theory is that this is related to Plug & Play manager running enumeration of devices left by the Hyper-V VSS writer backup.

On the affected machine, the C:\windows\system32\config\SYSTEM file is around 170 MB. Using dureg, i could boil this down to two registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

Which are about 6 megabytes each, when looking at them using dureg:

C:\Users\z-l.beeler\Desktop>dureg.exe /lm “SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk”
Size of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk: 6575468

Since this machine has been operational since about a year, with daily backups (BE12.5), it is much more pronounced here than on other machines. The Virtual Disk being part of the backup procedure is visible in the System log – it produces errors during the backup and Microsoft even has a KB article on the issue KB958669.

The eventlog on the affected machine looks like this:
18:02 The quota minifilter driver completed rescanning directories under quota management on volume “\Device\HarddiskVolume3 (G:)”. All quota information is up-to-date.
18:48 The Plug and Play service entered the running state.

Which for me further indicates that there is some kind of issue with the Plug and Play service. Unfortunately, the machine is not reachable remotely during the issue, but my guess would be that the Plug and Play service is hung in a “Starting” state, causing the lockup issue because of kernel interactions.

Unfortunately, i don’t have enough information and i’m not sure if deleting random registry keys is a good approach on this. I’ve posted on MCSEboard.de and the TechNet Forums – in the hope of getting valuable feedback from other long-term Hyper-V users.

Update: I don’t have a solution yet, but i’ve received a few insights. Thanks to zahni from MCSEBoard.de i got a link to KB959476, which doesn’t match my specific issue, but definitively goes into the right direction.

I’ve also found the Device Remover software, which gives me a clear graphic representation of the issue – over 9500 devices on the affected server. It even offers a removal function, but i don’t want to risk using this tool on a production server.

I’ve also opened a case with Microsoft PSS, in hope of getting an official solution to this issue soon.

Update 2:Removing the devices cut down the number of devices to about 300. I did this after Microsoft PSS recommended me to remove them. As i assumed, this resolved the issue during boot-up hang. Unfortunately, even after installing WS08 SP2, the machines still creates new virtual hard drives when running backup. I will try to get this resolved completely.

Since SP2 was released on April 30th, i’ve installed it on a few uncritical machines.

One of these runs our TS Gateway Server and our NPS Server for Wireless LAN authentication.

Unfortunately, since the SP2 installation, the NPS service started crashing, taking several other services with him.

Error message is as follows:

Faulting application svchost.exe_IAS, version 6.0.6001.18000, time stamp 0×47919291,
faulting module msvcrt.dll, version 7.0.6002.18005, time stamp 0x49e04189, exception code 0xc0000005,
fault offset 0×0000000000001467, process id 0×1444, application start time 0x01c9d570f76f56bc.

I’ve found one other reference to this issue on the on the TechNet Forums.

I’ve uninstalled SP2 and delayed SP2 deployment until this has been resolved.

I’ve had my share of experiences with ZyXEL equipment, like the ZyWALL vs. Exchange post i did a few years ago.

But today i experienced the most grave issue with their equipment that critically impacted a customers business.

The customer has two sites – an HQ with an SBS 2008 and a branch office with two Lenovo SFF machines running Windows Vista Business. Both sites are using 20/2 VDSL lines from Swisscom, with ZyXEL P-2802HWL routers.

There is an IPsec VPN configured between these two sites. This has been working fine since January.

Now, about a month ago a telecom service company installed VoIP telephones in the branch office, and enabled QoS on both ZyXEL routers.

Since then, Outlook was unable to synchronize correctly with the SBS server. Unfortunately, the customers personnel isn’t that technically savy, so they weren’t able to tell that they had a problem – because smaller e-mails were able to successfully synchronize, but larger ones failed. This led to very inconsistent states of the OST files, with some mails there and some mails not there.

When i arrived at the branch office i didn’t have a single clue what the issue was or may be. At first i suspected an Outlook problem, so i deleted the OST file. But from there on, nothing happened – Outlook wasn’t able to download anything.

Next, i tried to copy a 50kbyte Excel file from a share to the local computer. This worked. So i tried a 2 megabyte Word file. This failed about halfway through, with Explorer just hanging there and doing nothing. From that point on, i suspected a network issue, but the fact that copying a 50kbyte file worked and a 2 megabyte file didn’t was very odd.

Using Outlook with Outlook Anywhere also worked (when the VPN tunnel was downed).

Whenever i’m confronted with strange network problems, i suspect MTU issues (which was my first “real” network problem i solved back on my first ADSL line – took me weeks for a simple fix). ping -l 5000 CUSTSBS01 worked. ping -l 15000 CUSTSBS01 worked, too. So thought it wasn’t an MTU issue.

Disabling QoS on the ZyXEL router fixed the issue, but made the phones unusuable while Outlook was filling it’s OST files.

So i ran through the usual check points – tcp checksum offloading, chimney, receive window autotuning, reboots, etc. Nothing helped. At the end i was just changing network settings at will. But nothing helped.

Out of any reasonable ideas, i changed the MTU to 1300. That fixed it – with QoS enabled and the NIC MTU of the two machines, everything was working as it should. File transfers worked, Outlook worked, Phones worked.

Don’t buy ZyXEL.

Since the 30th of April, Windows 7 RC is available. I’ve been using Windows 7 for quite some time, but that usually doesn’t tell us much about end user experience with Windows 7.

At work, we’ve decided to move several people with a strong technical background over to Windows 7 x64 (if they want, of course). In order to drive internal testing, usage data and generally bring awareness to the whole personnel at the company and also our customers.

By now, i’ve migrated 8 laptops to Windows 7 RC – with which people are working in production and using for their everyday work. Of course in case we run in real troubles with Windows 7, we still have a few spare laptops that run Windows Vista SP2 x32.

The migration has been without any major issues moving from Windows Vista to Windows 7 than when moving from XP to Windows 7, most of this can probably be attributed to the fact that all the applications we use internally are compatible with Windows Vista and we also got a lot of experience with the new deployment model and tools available since Windows Vista.

Still, we ran into a few smaller problems that are mostly un-resolved as of yet, but do not majorly impact anything.

We use Lenovo T60, R61, T61, T500, W500 and R500 laptops. All of these have been running Windows Vista SP1 x32 with BitLocker enabled in TPM+PIN Mode. We installed Windows 7 using Clean (Custom), without formatting the hard drive first – this required us to suspend Bitlocker protection in Windows Vista before running setup. Two devices were reformatted – at the wish of the person using them.

I also upgraded all laptops to 4GB of RAM – which now can actually be used. For example, my W500 with Vista x32 only saw 2.25GB of the 4GB RAM (not a typo – only 2GB).

My biggest issue was that Bitlocker on Windows 7 didn’t properly backup it’s Bitlocker Key and TPM to Active Directory. This is a major issue, as i now had to manually backup the Bitlocker Keys to a secure network share. I didn’t find much about this on the Web, i suspect that not many people used this functionality, and there’s almost no documentation available about Windows 7 Bitlocker. As the workaround of saving the key works just as well, i can live with this.

The fingerprint reader installed on all those Thinkpads has a driver available, but the different drivers have different issues (most of them just crash when using them). I didn’t try installing the Lenovo tools. We don’t use the fingerprint readers, so that’s a non issue for me, but if you do this might require some investigation.

Switchable graphics on the W500 and T500 doesn’t work. Also, the Intel GMA adapter seems to be a lot slower than it was under Windows Vista – so i switched these devices to the internal ATI graphics card. No issues with that, except higher power usage.

WSUS does not contain Windows 7 updates – which makes perfect sense. I created a new WMI filter and a GPO to ensure that Windows 7 got updates directly from Microsoft.

After installing Windows 7 on the devices, all hardware including UMTS modems worked perfectly. Intel AMT doesn’t have Windows 7 drivers yet, but we don’t use that either.

I migrated user data using USMT Hardlink Migration, for which i created a nice batch file using the idea from this feature walkthrough.

I’ll keep you up to date – there’s one more machine considered for migration next week, and after a weeks i’ll have proper feedback from the power users at my office. I’ll even try to persuade our head sales and CEO to try Windows 7, just for the heck of it.

This morning i attended the Beta for Exam 70-680 – i was one of the lucky few that got a seat in this beta.

I already did 70-270 (Windows XP) and 70-620 (Windows Vista) two years ago, and the Vista exam was far too easy for my taste. It took me about 20 minutes, and i walked out with a score about 900. That’s not good – too easy questions will just devalue the certification.

With this in mind, i expected 70-680 to get Microsoft back on track, and they did. The exam has much better and much more difficult questions than 70-620. Not questions which require you to memorize stuff, but questions which require you to understand the subject matter.

As usual for beta exams, there were no simulations, VM tasks or anything else except multiple choice questions. I can understand why that’s the case (they probably want to use the final version for that), but i’m still not entirely with this as it is.

One thing that was new in this exam is that you get a questionary that asks you to judge your knowledge levels on Windows 7 for yourself. Several fields are presented, in which you have to choose between very high, high, mediocre, low and very low skills – another questions asks how much experience you already had with Windows 7 (with options such as “Over a year”).

I think that’s a good idea – most exam betas are open now, which means that many less-skilled people will also attend them. As long as those are truthful, this can actually help to improve the exam.

Unfortunately, i had very much difficulty finding what’s my personal baseline. I opted to choose either High or Mediocre for most answers, but was that correct? What does high mean? What does mediocre mean? What’s my knowledge level?

It might make sense to ask questions which are more task oriented – if you already did a task X and if you think if you’re proficient at doing task X.

The exam content was pretty much what was in the official docs – there’s a lot more focus on using group policies (local ones in this case), and also a few more detailed networking questions regarding Subnetting, in both IPv4 and IPv6.

General list of things i’ve seen:

• New features: BranchCache, DirectAccess and VPN (not overly technical – if you got it to work once, you can answer these)
• Bitlocker – not overly many questions
• Setup – the USB stick install gets featured more
• USMT gets a lot more focus and also Windows EasyTransfer
• Imaging, Deployment, VHDs

I’ll see if i passed the exam in officially 8 weeks, so probably in about 4 real moths ;)

I’ve installed the Windows 7 RC in English. Worked perfectly, but most of our customers run their systems in German, so’ill have to stay up-to-date on how Microsoft’s translators “creatively” translated their work into German (actually, Microsoft’s translations aren’t the worst i’ve seen).

So today i decided the install the German language pack on my home PC and on my laptop – on the home PC, this worked as expected. On my laptop, which has it’s hard drive encrypted and protected by BitLocker in TPM mode.

After the obligatory reboot, i changed the system language. The machine rebooted and then asked for my Bitlocker recovery password – in German. It was obvious what happened: On German Vista machines with Bitlocker enabled, the Windows Boot Manager was still in English, but on Windows 7 the boot manager was also translated – which means that it now failed the integrity check because it was modified.

Luckily i could use our Terminal Services Gateway to log onto my administrative terminal server, where i had the BitLocker Recovery Password Viewer installed, so viewing my recovery key was quick and easy.

After booting into my now (mostly) German Windows 7, i temporarily halted Bitlocker protection, and immediately reenabled. This caused Windows 7 to reverify the state of the Boot Manager, and after i another reboot i was sure that everything was fine.

Oh, and this is one of the rather funny translation episodes: The window is not resizeable and the text doesn’t fit.

Windows 7 is finally nearing it’s completion, and the Release Candidate is finally available. After installing the Windows 7 Beta Build 7000 back in December on my PC at home, i decided to upgrade my work Laptop to Windows 7. The score to the right is from my Laptop.

First of all, i had Bitlocker enabled on my ThinkPad W500, which was running Windows Vista x32 and i intended to install Windows 7 x64. So a direct inplace upgrade was out of the question. I created a backup of the machine, disabled Bitlocker, upgraded my laptops BIOS to the latest version, and booted Windows 7 setup from an USB stick.

Next, i pressed Shift-F10 on the setup screen, deleted all the Windows and Program Files folders, and then started an installation directly on the Bitlocker-enabled drive (this way, i didn’t have to restore all the files i already had on the drive, saving me valuable time).

Windows 7 was done after about 25 minutes, and greeted me with Aero enabled and the 1920×1200 15″ screen already set to a scaling factor of 125%. This is were i also noticed that DPI settings are now user dependant, instead of affecting the whole system. An extremely nice feature, that probably needed quite a bit of work. I set the scaling factor to 115%, which is the best factor between readability and remaining screen real estate for me.

Unfortunately, the switchable graphics driver available from Lenovo did not support WDDM 1.1. I went into the BIOS and configured the machine to always use the Intel graphics. However, i noticed that unlike in Vista, the Intel graphic card did not produce 100% smooth Aero animations. Since i have the power supply connected most of the time anyway, i configured the system to always use the ATI card. This produced better results.

The fingerprint reader does not work yet, but i didn’t invest time in that since i don’t use it anyway. Also, there are issues with Intel AMT, which i don’t use either.

So the base OS worked flawlessly after install. Even switching the graphics card around didn’t phase it, Aero was automatically enabled and the correct resolution configured. WLAN, Audio, everything you would need worked out of the Box.

I joined the machine to the domain, where it sucked down all the GPOs for our corporate network. I unplugged the network cable, and it automatically connected to the corporate wireless network, authenticated by EAP-TLS.

Since our printserver is WS08 x64 box, corporate printing also worked automatically, without any additional work. Of course, all the other group policy settings applied as they should, and i didn’t find any issues yet regarding policy settings.

But an OS alone doesn’t serve a purpose, you need applications. I’ve installed the following applications:

• Adobe Reader 9.1 Works perfectly.
• DIAS-iS Network Client 3.2 Works perfectly.
• DIAS-iS OSP Version 3 for Office 2007 Works perfectly.
• Office 2007 SP1 Enterprise, Visio and PDF/XPS plugin Works perfectly.
• Office 2007 Primary Interop Assemblies Works perfectly.
• Office 2007 VSTO 3.0 Works perfectly.
• Office 2007 Communicator R1 with latest Hotfix Works perfectly.
• Solitas InfoStore Windows Retrieval Works perfectly.
• IBM System i Access V6R1M0 x64 Works perfectly.
• IrfanView Works perfectly.
• Mozilla Firefox 3.1b3 Works perfectly.
• PuTTY 0.60 Works perfectly.
• SonicWALL Global VPN Client x64 Sometimes loses it’s IPsec driver – repairing the program helps.
• Windows Live Messenger Works perfectly.
• Virtual CloneDrive Works perfectly.
• WinRAR Works perfectly.
• tn5250 Works perfectly.

So far, so good. The SonicWALL issue may be annoying, but it’s not a dealbreaker. Judging from my experience, it’s a SonicWALL issue. Opening a bug there won’t help, as they don’t support Windows 7 yet. I can live with that.

Perfomance on Windows 7 on this machine is even better than Vista. I can now fully use the 4GB RAM installed in my laptop. Never used Windows XP on this machine, i can’t compare performance. All the business apps i need to do my job work flawlessly. Printing works flawlessly.

Windows 7 is even better than Vista. But for those that didn’t spend the last three years using Windows Vista, it may be rather hard to get used to all the new stuff. For example, the deployment options between 7 and Vista are both based on WIM imaging, with a few improvements here and there. If you know how to do it on Vista, you can also do it in Windows 7.

As a bonus, the score to the right from my desktop PC.

With the release of the new generation of System x servers, IBM also revamped it’s tool offering.

Central point of the new IBM offering is the ToolsCenter, which serves as a starting point for all important IBM tools.

The two most important tools, which every admin dealing with IBM System x servers should know are now available in new versions, which offer improved functionality.

UpdateXpress System Pack Installer

UpdateXpress is now available in version 3. Pictured to the right is the new user interface, which offers much needed improvements. The previous versions looked like a leftover from the cold war.

UpdateXpress allows you to update all your System x drives in one automatic swoop, without the need to meticulously check the IBM web site for newly released drivers.

Dynamic System Analysis

DSA is now available in version 2.20. While the handling of the tool hasn’t changed much, there is now a 64bit version available. A few bugs i’ve encountered on 64bit systems are fixed with this new release.