Lukas Beeler’s IT Blog

I’ve just migrated our internal OCS 2007 setup to OCS 2007 R2. Yeah, i’m very late at this.

Everything worked, but LiveMeeting when using the Edge server. It worked fine internally, or when a VPN connection was established. The LiveMeeting Error Log showed me exactly what failed, but it took my almost half an hour to figure out why it was failing.

[P] SEQ#16,placeware::SslSocket::connectInternal::TLSNegotiationTimer stop,112029,,
[D] [X-PSOM] SslSocket::connect end OK
[D] [X-PSOM] TunnelSocket::connect ProxyHeader sent.
[I] [X-PSOM] SSLTunnelStream: Established SSL Tunnel Stream to hor-ocsgw-01.acommit.ch
[I] [X-PSOM] Forwarded TCP probe succeeded
[P] SEQ#14,placeware::ServerInfo::ForwardedTcpProbeThread::run::ForwardedTcpProbeTimer stop,145082,,
[I] [X-PSOM] Best mode for Client RPC is : 1
[I] [X-PSOM] Best mode is fwdtls. Reusing stream in probe.
[I] [X-PSOM] PWS Handshake sent.
[E] [X-PSOM] placeware::Socket::readWSAGetOverlappedResult failed, error = 10054
[E] [X-PSOM] Socket error while reading.
[E] [X-PSOM] SslSocket::close: socket is not connected

So, looks good at first. And then it fails. No log entry on the OCS Edge, no entry on the OCS Standard.

I figured out the solution when rechecking my entire configuration – i misconfigured the external Edge server hostname on the Standard Server.

Fixing the issue is easy:

• Log on OCS Standard Server
• Right click on Pool – Properties – Web Conference -Web Conference Edge Server.
• Then, enter the correct external host name. You’ll find this name in the Edge server configuration.

The dump then reads like this:

[P] SEQ#16,placeware::SslSocket::connectInternal::TLSNegotiationTimer stop,83410,,
[D] [X-PSOM] SslSocket::connect end OK
[D] [X-PSOM] TunnelSocket::connect ProxyHeader sent.
[I] [X-PSOM] SSLTunnelStream: Established SSL Tunnel Stream to hor-ocsgw-01-1.acommit.ch
[I] [X-PSOM] Forwarded TCP probe succeeded
[P] SEQ#14,placeware::ServerInfo::ForwardedTcpProbeThread::run::ForwardedTcpProbeTimer stop,122853,,
[I] [X-PSOM] Best mode for Client RPC is : 1
[I] [X-PSOM] Best mode is fwdtls. Reusing stream in probe.
[I] [X-PSOM] PWS Handshake sent.
[I] [X-PSOM] Received PWS Handshake.

For several months, i’ve had a problem on a Hyper-V host described WS08 and the black screen of waiting. Basically, the machine boots up, hangs 50 minutes being completely unresponsive, and then goes on working perfectly for weeks.

The problem was resolved (temporarily) by deleting shadow copies, but it still exists. As i’ve had time this weekend to investigate this closely, i’m pretty sure that i found the root cause of the problem, but i have no solution yet. Remember, this is all just a theory i cooked up – i’m putting this information out there in case anyone else has a similar problem.

My theory is that this is related to Plug & Play manager running enumeration of devices left by the Hyper-V VSS writer backup.

On the affected machine, the C:\windows\system32\config\SYSTEM file is around 170 MB. Using dureg, i could boil this down to two registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

Which are about 6 megabytes each, when looking at them using dureg:

C:\Users\z-l.beeler\Desktop>dureg.exe /lm “SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk”
Size of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\SCSI\Disk&Ven_Msft&Prod_Virtual_Disk: 6575468

Since this machine has been operational since about a year, with daily backups (BE12.5), it is much more pronounced here than on other machines. The Virtual Disk being part of the backup procedure is visible in the System log – it produces errors during the backup and Microsoft even has a KB article on the issue KB958669.

The eventlog on the affected machine looks like this:
18:02 The quota minifilter driver completed rescanning directories under quota management on volume “\Device\HarddiskVolume3 (G:)”. All quota information is up-to-date.
18:48 The Plug and Play service entered the running state.

Which for me further indicates that there is some kind of issue with the Plug and Play service. Unfortunately, the machine is not reachable remotely during the issue, but my guess would be that the Plug and Play service is hung in a “Starting” state, causing the lockup issue because of kernel interactions.

Unfortunately, i don’t have enough information and i’m not sure if deleting random registry keys is a good approach on this. I’ve posted on MCSEboard.de and the TechNet Forums – in the hope of getting valuable feedback from other long-term Hyper-V users.

Update: I don’t have a solution yet, but i’ve received a few insights. Thanks to zahni from MCSEBoard.de i got a link to KB959476, which doesn’t match my specific issue, but definitively goes into the right direction.

I’ve also found the Device Remover software, which gives me a clear graphic representation of the issue – over 9500 devices on the affected server. It even offers a removal function, but i don’t want to risk using this tool on a production server.

I’ve also opened a case with Microsoft PSS, in hope of getting an official solution to this issue soon.

Update 2:Removing the devices cut down the number of devices to about 300. I did this after Microsoft PSS recommended me to remove them. As i assumed, this resolved the issue during boot-up hang. Unfortunately, even after installing WS08 SP2, the machines still creates new virtual hard drives when running backup. I will try to get this resolved completely.

Since SP2 was released on April 30th, i’ve installed it on a few uncritical machines.

One of these runs our TS Gateway Server and our NPS Server for Wireless LAN authentication.

Unfortunately, since the SP2 installation, the NPS service started crashing, taking several other services with him.

Error message is as follows:

Faulting application svchost.exe_IAS, version 6.0.6001.18000, time stamp 0×47919291,
faulting module msvcrt.dll, version 7.0.6002.18005, time stamp 0×49e04189, exception code 0xc0000005,
fault offset 0×0000000000001467, process id 0×1444, application start time 0×01c9d570f76f56bc.

I’ve found one other reference to this issue on the on the TechNet Forums.

I’ve uninstalled SP2 and delayed SP2 deployment until this has been resolved.

I’ve had my share of experiences with ZyXEL equipment, like the ZyWALL vs. Exchange post i did a few years ago.

But today i experienced the most grave issue with their equipment that critically impacted a customers business.

The customer has two sites – an HQ with an SBS 2008 and a branch office with two Lenovo SFF machines running Windows Vista Business. Both sites are using 20/2 VDSL lines from Swisscom, with ZyXEL P-2802HWL routers.

There is an IPsec VPN configured between these two sites. This has been working fine since January.

Now, about a month ago a telecom service company installed VoIP telephones in the branch office, and enabled QoS on both ZyXEL routers.

Since then, Outlook was unable to synchronize correctly with the SBS server. Unfortunately, the customers personnel isn’t that technically savy, so they weren’t able to tell that they had a problem – because smaller e-mails were able to successfully synchronize, but larger ones failed. This led to very inconsistent states of the OST files, with some mails there and some mails not there.

When i arrived at the branch office i didn’t have a single clue what the issue was or may be. At first i suspected an Outlook problem, so i deleted the OST file. But from there on, nothing happened – Outlook wasn’t able to download anything.

Next, i tried to copy a 50kbyte Excel file from a share to the local computer. This worked. So i tried a 2 megabyte Word file. This failed about halfway through, with Explorer just hanging there and doing nothing. From that point on, i suspected a network issue, but the fact that copying a 50kbyte file worked and a 2 megabyte file didn’t was very odd.

Using Outlook with Outlook Anywhere also worked (when the VPN tunnel was downed).

Whenever i’m confronted with strange network problems, i suspect MTU issues (which was my first “real” network problem i solved back on my first ADSL line – took me weeks for a simple fix). ping -l 5000 CUSTSBS01 worked. ping -l 15000 CUSTSBS01 worked, too. So thought it wasn’t an MTU issue.

Disabling QoS on the ZyXEL router fixed the issue, but made the phones unusuable while Outlook was filling it’s OST files.

So i ran through the usual check points – tcp checksum offloading, chimney, receive window autotuning, reboots, etc. Nothing helped. At the end i was just changing network settings at will. But nothing helped.

Out of any reasonable ideas, i changed the MTU to 1300. That fixed it – with QoS enabled and the NIC MTU of the two machines, everything was working as it should. File transfers worked, Outlook worked, Phones worked.

Don’t buy ZyXEL.

Since the 30th of April, Windows 7 RC is available. I’ve been using Windows 7 for quite some time, but that usually doesn’t tell us much about end user experience with Windows 7.

At work, we’ve decided to move several people with a strong technical background over to Windows 7 x64 (if they want, of course). In order to drive internal testing, usage data and generally bring awareness to the whole personnel at the company and also our customers.

By now, i’ve migrated 8 laptops to Windows 7 RC – with which people are working in production and using for their everyday work. Of course in case we run in real troubles with Windows 7, we still have a few spare laptops that run Windows Vista SP2 x32.

The migration has been without any major issues moving from Windows Vista to Windows 7 than when moving from XP to Windows 7, most of this can probably be attributed to the fact that all the applications we use internally are compatible with Windows Vista and we also got a lot of experience with the new deployment model and tools available since Windows Vista.

Still, we ran into a few smaller problems that are mostly un-resolved as of yet, but do not majorly impact anything.

We use Lenovo T60, R61, T61, T500, W500 and R500 laptops. All of these have been running Windows Vista SP1 x32 with BitLocker enabled in TPM+PIN Mode. We installed Windows 7 using Clean (Custom), without formatting the hard drive first – this required us to suspend Bitlocker protection in Windows Vista before running setup. Two devices were reformatted – at the wish of the person using them.

I also upgraded all laptops to 4GB of RAM – which now can actually be used. For example, my W500 with Vista x32 only saw 2.25GB of the 4GB RAM (not a typo – only 2GB).

My biggest issue was that Bitlocker on Windows 7 didn’t properly backup it’s Bitlocker Key and TPM to Active Directory. This is a major issue, as i now had to manually backup the Bitlocker Keys to a secure network share. I didn’t find much about this on the Web, i suspect that not many people used this functionality, and there’s almost no documentation available about Windows 7 Bitlocker. As the workaround of saving the key works just as well, i can live with this.

The fingerprint reader installed on all those Thinkpads has a driver available, but the different drivers have different issues (most of them just crash when using them). I didn’t try installing the Lenovo tools. We don’t use the fingerprint readers, so that’s a non issue for me, but if you do this might require some investigation.

Switchable graphics on the W500 and T500 doesn’t work. Also, the Intel GMA adapter seems to be a lot slower than it was under Windows Vista – so i switched these devices to the internal ATI graphics card. No issues with that, except higher power usage.

WSUS does not contain Windows 7 updates – which makes perfect sense. I created a new WMI filter and a GPO to ensure that Windows 7 got updates directly from Microsoft.

After installing Windows 7 on the devices, all hardware including UMTS modems worked perfectly. Intel AMT doesn’t have Windows 7 drivers yet, but we don’t use that either.

I migrated user data using USMT Hardlink Migration, for which i created a nice batch file using the idea from this feature walkthrough.

I’ll keep you up to date – there’s one more machine considered for migration next week, and after a weeks i’ll have proper feedback from the power users at my office. I’ll even try to persuade our head sales and CEO to try Windows 7, just for the heck of it.

This morning i attended the Beta for Exam 70-680 – i was one of the lucky few that got a seat in this beta.

I already did 70-270 (Windows XP) and 70-620 (Windows Vista) two years ago, and the Vista exam was far too easy for my taste. It took me about 20 minutes, and i walked out with a score about 900. That’s not good – too easy questions will just devalue the certification.

With this in mind, i expected 70-680 to get Microsoft back on track, and they did. The exam has much better and much more difficult questions than 70-620. Not questions which require you to memorize stuff, but questions which require you to understand the subject matter.

As usual for beta exams, there were no simulations, VM tasks or anything else except multiple choice questions. I can understand why that’s the case (they probably want to use the final version for that), but i’m still not entirely with this as it is.

One thing that was new in this exam is that you get a questionary that asks you to judge your knowledge levels on Windows 7 for yourself. Several fields are presented, in which you have to choose between very high, high, mediocre, low and very low skills – another questions asks how much experience you already had with Windows 7 (with options such as “Over a year”).

I think that’s a good idea – most exam betas are open now, which means that many less-skilled people will also attend them. As long as those are truthful, this can actually help to improve the exam.

Unfortunately, i had very much difficulty finding what’s my personal baseline. I opted to choose either High or Mediocre for most answers, but was that correct? What does high mean? What does mediocre mean? What’s my knowledge level?

It might make sense to ask questions which are more task oriented – if you already did a task X and if you think if you’re proficient at doing task X.

The exam content was pretty much what was in the official docs – there’s a lot more focus on using group policies (local ones in this case), and also a few more detailed networking questions regarding Subnetting, in both IPv4 and IPv6.

General list of things i’ve seen:

• New features: BranchCache, DirectAccess and VPN (not overly technical – if you got it to work once, you can answer these)
• Bitlocker – not overly many questions
• Setup – the USB stick install gets featured more
• USMT gets a lot more focus and also Windows EasyTransfer
• Imaging, Deployment, VHDs

I’ll see if i passed the exam in officially 8 weeks, so probably in about 4 real moths ;)

I’ve installed the Windows 7 RC in English. Worked perfectly, but most of our customers run their systems in German, so’ill have to stay up-to-date on how Microsoft’s translators “creatively” translated their work into German (actually, Microsoft’s translations aren’t the worst i’ve seen).

So today i decided the install the German language pack on my home PC and on my laptop – on the home PC, this worked as expected. On my laptop, which has it’s hard drive encrypted and protected by BitLocker in TPM mode.

After the obligatory reboot, i changed the system language. The machine rebooted and then asked for my Bitlocker recovery password – in German. It was obvious what happened: On German Vista machines with Bitlocker enabled, the Windows Boot Manager was still in English, but on Windows 7 the boot manager was also translated – which means that it now failed the integrity check because it was modified.

Luckily i could use our Terminal Services Gateway to log onto my administrative terminal server, where i had the BitLocker Recovery Password Viewer installed, so viewing my recovery key was quick and easy.

After booting into my now (mostly) German Windows 7, i temporarily halted Bitlocker protection, and immediately reenabled. This caused Windows 7 to reverify the state of the Boot Manager, and after i another reboot i was sure that everything was fine.

Oh, and this is one of the rather funny translation episodes: The window is not resizeable and the text doesn’t fit.

Windows 7 is finally nearing it’s completion, and the Release Candidate is finally available. After installing the Windows 7 Beta Build 7000 back in December on my PC at home, i decided to upgrade my work Laptop to Windows 7. The score to the right is from my Laptop.

First of all, i had Bitlocker enabled on my ThinkPad W500, which was running Windows Vista x32 and i intended to install Windows 7 x64. So a direct inplace upgrade was out of the question. I created a backup of the machine, disabled Bitlocker, upgraded my laptops BIOS to the latest version, and booted Windows 7 setup from an USB stick.

Next, i pressed Shift-F10 on the setup screen, deleted all the Windows and Program Files folders, and then started an installation directly on the Bitlocker-enabled drive (this way, i didn’t have to restore all the files i already had on the drive, saving me valuable time).

Windows 7 was done after about 25 minutes, and greeted me with Aero enabled and the 1920×1200 15″ screen already set to a scaling factor of 125%. This is were i also noticed that DPI settings are now user dependant, instead of affecting the whole system. An extremely nice feature, that probably needed quite a bit of work. I set the scaling factor to 115%, which is the best factor between readability and remaining screen real estate for me.

Unfortunately, the switchable graphics driver available from Lenovo did not support WDDM 1.1. I went into the BIOS and configured the machine to always use the Intel graphics. However, i noticed that unlike in Vista, the Intel graphic card did not produce 100% smooth Aero animations. Since i have the power supply connected most of the time anyway, i configured the system to always use the ATI card. This produced better results.

The fingerprint reader does not work yet, but i didn’t invest time in that since i don’t use it anyway. Also, there are issues with Intel AMT, which i don’t use either.

So the base OS worked flawlessly after install. Even switching the graphics card around didn’t phase it, Aero was automatically enabled and the correct resolution configured. WLAN, Audio, everything you would need worked out of the Box.

I joined the machine to the domain, where it sucked down all the GPOs for our corporate network. I unplugged the network cable, and it automatically connected to the corporate wireless network, authenticated by EAP-TLS.

Since our printserver is WS08 x64 box, corporate printing also worked automatically, without any additional work. Of course, all the other group policy settings applied as they should, and i didn’t find any issues yet regarding policy settings.

But an OS alone doesn’t serve a purpose, you need applications. I’ve installed the following applications:

• Adobe Reader 9.1 Works perfectly.
• DIAS-iS Network Client 3.2 Works perfectly.
• DIAS-iS OSP Version 3 for Office 2007 Works perfectly.
• Office 2007 SP1 Enterprise, Visio and PDF/XPS plugin Works perfectly.
• Office 2007 Primary Interop Assemblies Works perfectly.
• Office 2007 VSTO 3.0 Works perfectly.
• Office 2007 Communicator R1 with latest Hotfix Works perfectly.
• Solitas InfoStore Windows Retrieval Works perfectly.
• IBM System i Access V6R1M0 x64 Works perfectly.
• IrfanView Works perfectly.
• Mozilla Firefox 3.1b3 Works perfectly.
• PuTTY 0.60 Works perfectly.
• SonicWALL Global VPN Client x64 Sometimes loses it’s IPsec driver – repairing the program helps.
• Windows Live Messenger Works perfectly.
• Virtual CloneDrive Works perfectly.
• WinRAR Works perfectly.
• tn5250 Works perfectly.

So far, so good. The SonicWALL issue may be annoying, but it’s not a dealbreaker. Judging from my experience, it’s a SonicWALL issue. Opening a bug there won’t help, as they don’t support Windows 7 yet. I can live with that.

Perfomance on Windows 7 on this machine is even better than Vista. I can now fully use the 4GB RAM installed in my laptop. Never used Windows XP on this machine, i can’t compare performance. All the business apps i need to do my job work flawlessly. Printing works flawlessly.

Windows 7 is even better than Vista. But for those that didn’t spend the last three years using Windows Vista, it may be rather hard to get used to all the new stuff. For example, the deployment options between 7 and Vista are both based on WIM imaging, with a few improvements here and there. If you know how to do it on Vista, you can also do it in Windows 7.

As a bonus, the score to the right from my desktop PC.

With the release of the new generation of System x servers, IBM also revamped it’s tool offering.

Central point of the new IBM offering is the ToolsCenter, which serves as a starting point for all important IBM tools.

The two most important tools, which every admin dealing with IBM System x servers should know are now available in new versions, which offer improved functionality.

UpdateXpress System Pack Installer

UpdateXpress is now available in version 3. Pictured to the right is the new user interface, which offers much needed improvements. The previous versions looked like a leftover from the cold war.

UpdateXpress allows you to update all your System x drives in one automatic swoop, without the need to meticulously check the IBM web site for newly released drivers.

Dynamic System Analysis

DSA is now available in version 2.20. While the handling of the tool hasn’t changed much, there is now a 64bit version available. A few bugs i’ve encountered on 64bit systems are fixed with this new release.

Today i attended an interesting product presentation from IBM, about their Nehalem product line up. In addition to the information i’ve already gleaned from the IBM web page, i’ve learned several things that are equally important.

If you’ve never heard about the x3650 M2, i suggest you to read my introduction post first.

Planned availability, other new products

Planned general availability (GA) of the new x3650 M2 is the the 20th of April, or in pretty much 3 weeks. Orders, configuration and pricing is available – so if you want to buy a new server now and can wait for three weeks, you should order a x3650 M2.

At the end of April, the M2 versions of the x3400 and x3500 will be announced. Judging from the current timeline, this will put GA of those new products near May.

4 CPU machines are planned for Q1 2010. I don’t care much about those since we run all our heavy DB workloads on IBM POWER.

Positioning of the new Intel 5500 Xeons

Just like the Core Microarchitecture brought many changes to servers, the new 5500 Xeons bring even more changes. It’s especially important for system administrators to understand the differences, and even more important if you’re selling systems to customers.

Here are a few key differences:

• Memory speed depends on the CPU purchased and the amoung and type of memory installed into the server
• Memory slots are only usuable if the associated CPU is installed
• HyperThreading is reintroduced in 2/3rds of the CPUs – systems will show twice the amount of logical processors
• TurboBoost is a new functionality that allows the CPU to run at higher clock speeds, depending on load and cooling

To the right you can see Intel’s official spec sheet. Intel introduces a “garbage bin” of CPUs that you should never use – the E5502 / E5504 / E5506 models. These CPUs do not support HyperThreading, TurboBoost, 1066 Memory and only have 4 instead of 8 megabytes of cache. Make sure to use E5520 or faster CPUs to ensure best performance. The performance difference between an E5506 and an E5520 is 15-20%, while the price difference is much smaller! In my opinion, the E5520 is currently the sweet spot between price and performance.

Order the right memory configuration

With FB-DIMMs, memory configuration was simple, because FB-DIMMs were slow no matter which way you put it. However, with the new integrated memory controllers, memory of much higher speed is now available. Now, as a technician or sales things will get more complicated.

I wrote in an earlier post that i didn’t understand why IBM only put 16 DIMM slots into the machine, while HP installed 18 DIMM slots – the reason is that in most cases it makes little sense to populate all DIMM slots, because this will heavily reduce the bandwidth available, as the memory must run at lower speeds.

While HP has decided to offer registered and unregistered memory for their DL3xx G6 models, IBM only offers registered DIMMs. The x3650 M2 COG Guide offers a lot of in terms of possible configurations. The most important thing is keep the numbers of memory modules down – this makes it easier deploy them correctly.

Another important part is that memory is no longer ordered in pairs, but again separately as it was a few generations before. Most servers ship with two 1GB modules standard. This is not an optimal configuration, since you have three channels that could be used.

There are two ways to deal with this: either add a third 1 GB module and then add the rest of the memory you need. Or discard the two memory modules that come with the servers and just install the higher capacity modules you bought. My recommendation would be to discard the 1GB modules and install three 4GB modules – for most SMB environments, 12GB of memory suffices for almost all services.

Other part changes

The onboard RAID-Controller is gone, there is a new specially positioned but otherwise standard PCI-E slot for the RAID controller. The system ships with a non-BBWC controller called the BR10i. In most cases, it makes sense to replace the standard RAID-Controller with a ServeRAID MR10i.

Some configurations also require an enablement kit to drive all 12 disk slots. Only 8 disk slots are standard.

The RSA II Slimline is gone. IMM now offers a lot of the RSA II functionality by default, but the most important functionality, remote KVM still requires a so called “Virtual Media Key” (as it enables Remote Media as well). In general, if a customer has used RSA II Slimline up to now, also include a Virtual Media Key. This enables full IMM functionality.

SSD offerings have also been added. Currently, the pricing for the 50GB SATA SSD is 3970 CHF in SSCT. My assumption would be that this is a pricing error, but i’m not too sure about that.

Pricing

Pricing hasn’t changed much. DDR3 memory seems to be a bit more expensive than the FB-DIMMs were, but that’ll pass as soon as DDR3 volume ramps up. I’ve created a sample config that is probably valid for most SMB deployments that clocks in at around 10k CHF. This is roughly the same as it was before with the standard x3650, except that the x3650 M2 will deliver a lot more performance for the money.