Lukas Beeler's IT Blog

Since SP2 was released on April 30th, i’ve installed it on a few uncritical machines.

One of these runs our TS Gateway Server and our NPS Server for Wireless LAN authentication.

Unfortunately, since the SP2 installation, the NPS service started crashing, taking several other services with him.

Error message is as follows:

Faulting application svchost.exe_IAS, version 6.0.6001.18000, time stamp 0×47919291,
faulting module msvcrt.dll, version 7.0.6002.18005, time stamp 0x49e04189, exception code 0xc0000005,
fault offset 0×0000000000001467, process id 0×1444, application start time 0x01c9d570f76f56bc.

I’ve found one other reference to this issue on the on the TechNet Forums.

I’ve uninstalled SP2 and delayed SP2 deployment until this has been resolved.

I’ve had my share of experiences with ZyXEL equipment, like the ZyWALL vs. Exchange post i did a few years ago.

But today i experienced the most grave issue with their equipment that critically impacted a customers business.

The customer has two sites – an HQ with an SBS 2008 and a branch office with two Lenovo SFF machines running Windows Vista Business. Both sites are using 20/2 VDSL lines from Swisscom, with ZyXEL P-2802HWL routers.

There is an IPsec VPN configured between these two sites. This has been working fine since January.

Now, about a month ago a telecom service company installed VoIP telephones in the branch office, and enabled QoS on both ZyXEL routers.

Since then, Outlook was unable to synchronize correctly with the SBS server. Unfortunately, the customers personnel isn’t that technically savy, so they weren’t able to tell that they had a problem – because smaller e-mails were able to successfully synchronize, but larger ones failed. This led to very inconsistent states of the OST files, with some mails there and some mails not there.

When i arrived at the branch office i didn’t have a single clue what the issue was or may be. At first i suspected an Outlook problem, so i deleted the OST file. But from there on, nothing happened – Outlook wasn’t able to download anything.

Next, i tried to copy a 50kbyte Excel file from a share to the local computer. This worked. So i tried a 2 megabyte Word file. This failed about halfway through, with Explorer just hanging there and doing nothing. From that point on, i suspected a network issue, but the fact that copying a 50kbyte file worked and a 2 megabyte file didn’t was very odd.

Using Outlook with Outlook Anywhere also worked (when the VPN tunnel was downed).

Whenever i’m confronted with strange network problems, i suspect MTU issues (which was my first “real” network problem i solved back on my first ADSL line – took me weeks for a simple fix). ping -l 5000 CUSTSBS01 worked. ping -l 15000 CUSTSBS01 worked, too. So thought it wasn’t an MTU issue.

Disabling QoS on the ZyXEL router fixed the issue, but made the phones unusuable while Outlook was filling it’s OST files.

So i ran through the usual check points – tcp checksum offloading, chimney, receive window autotuning, reboots, etc. Nothing helped. At the end i was just changing network settings at will. But nothing helped.

Out of any reasonable ideas, i changed the MTU to 1300. That fixed it – with QoS enabled and the NIC MTU of the two machines, everything was working as it should. File transfers worked, Outlook worked, Phones worked.

Don’t buy ZyXEL.

Since the 30th of April, Windows 7 RC is available. I’ve been using Windows 7 for quite some time, but that usually doesn’t tell us much about end user experience with Windows 7.

At work, we’ve decided to move several people with a strong technical background over to Windows 7 x64 (if they want, of course). In order to drive internal testing, usage data and generally bring awareness to the whole personnel at the company and also our customers.

By now, i’ve migrated 8 laptops to Windows 7 RC – with which people are working in production and using for their everyday work. Of course in case we run in real troubles with Windows 7, we still have a few spare laptops that run Windows Vista SP2 x32.

The migration has been without any major issues moving from Windows Vista to Windows 7 than when moving from XP to Windows 7, most of this can probably be attributed to the fact that all the applications we use internally are compatible with Windows Vista and we also got a lot of experience with the new deployment model and tools available since Windows Vista.

Still, we ran into a few smaller problems that are mostly un-resolved as of yet, but do not majorly impact anything.

We use Lenovo T60, R61, T61, T500, W500 and R500 laptops. All of these have been running Windows Vista SP1 x32 with BitLocker enabled in TPM+PIN Mode. We installed Windows 7 using Clean (Custom), without formatting the hard drive first – this required us to suspend Bitlocker protection in Windows Vista before running setup. Two devices were reformatted – at the wish of the person using them.

I also upgraded all laptops to 4GB of RAM – which now can actually be used. For example, my W500 with Vista x32 only saw 2.25GB of the 4GB RAM (not a typo – only 2GB).

My biggest issue was that Bitlocker on Windows 7 didn’t properly backup it’s Bitlocker Key and TPM to Active Directory. This is a major issue, as i now had to manually backup the Bitlocker Keys to a secure network share. I didn’t find much about this on the Web, i suspect that not many people used this functionality, and there’s almost no documentation available about Windows 7 Bitlocker. As the workaround of saving the key works just as well, i can live with this.

The fingerprint reader installed on all those Thinkpads has a driver available, but the different drivers have different issues (most of them just crash when using them). I didn’t try installing the Lenovo tools. We don’t use the fingerprint readers, so that’s a non issue for me, but if you do this might require some investigation.

Switchable graphics on the W500 and T500 doesn’t work. Also, the Intel GMA adapter seems to be a lot slower than it was under Windows Vista – so i switched these devices to the internal ATI graphics card. No issues with that, except higher power usage.

WSUS does not contain Windows 7 updates – which makes perfect sense. I created a new WMI filter and a GPO to ensure that Windows 7 got updates directly from Microsoft.

After installing Windows 7 on the devices, all hardware including UMTS modems worked perfectly. Intel AMT doesn’t have Windows 7 drivers yet, but we don’t use that either.

I migrated user data using USMT Hardlink Migration, for which i created a nice batch file using the idea from this feature walkthrough.

I’ll keep you up to date – there’s one more machine considered for migration next week, and after a weeks i’ll have proper feedback from the power users at my office. I’ll even try to persuade our head sales and CEO to try Windows 7, just for the heck of it.

This morning i attended the Beta for Exam 70-680 – i was one of the lucky few that got a seat in this beta.

I already did 70-270 (Windows XP) and 70-620 (Windows Vista) two years ago, and the Vista exam was far too easy for my taste. It took me about 20 minutes, and i walked out with a score about 900. That’s not good – too easy questions will just devalue the certification.

With this in mind, i expected 70-680 to get Microsoft back on track, and they did. The exam has much better and much more difficult questions than 70-620. Not questions which require you to memorize stuff, but questions which require you to understand the subject matter.

As usual for beta exams, there were no simulations, VM tasks or anything else except multiple choice questions. I can understand why that’s the case (they probably want to use the final version for that), but i’m still not entirely with this as it is.

One thing that was new in this exam is that you get a questionary that asks you to judge your knowledge levels on Windows 7 for yourself. Several fields are presented, in which you have to choose between very high, high, mediocre, low and very low skills – another questions asks how much experience you already had with Windows 7 (with options such as “Over a year”).

I think that’s a good idea – most exam betas are open now, which means that many less-skilled people will also attend them. As long as those are truthful, this can actually help to improve the exam.

Unfortunately, i had very much difficulty finding what’s my personal baseline. I opted to choose either High or Mediocre for most answers, but was that correct? What does high mean? What does mediocre mean? What’s my knowledge level?

It might make sense to ask questions which are more task oriented – if you already did a task X and if you think if you’re proficient at doing task X.

The exam content was pretty much what was in the official docs – there’s a lot more focus on using group policies (local ones in this case), and also a few more detailed networking questions regarding Subnetting, in both IPv4 and IPv6.

General list of things i’ve seen:

• New features: BranchCache, DirectAccess and VPN (not overly technical – if you got it to work once, you can answer these)
• Bitlocker – not overly many questions
• Setup – the USB stick install gets featured more
• USMT gets a lot more focus and also Windows EasyTransfer
• Imaging, Deployment, VHDs

I’ll see if i passed the exam in officially 8 weeks, so probably in about 4 real moths ;)

I’ve installed the Windows 7 RC in English. Worked perfectly, but most of our customers run their systems in German, so’ill have to stay up-to-date on how Microsoft’s translators “creatively” translated their work into German (actually, Microsoft’s translations aren’t the worst i’ve seen).

So today i decided the install the German language pack on my home PC and on my laptop – on the home PC, this worked as expected. On my laptop, which has it’s hard drive encrypted and protected by BitLocker in TPM mode.

After the obligatory reboot, i changed the system language. The machine rebooted and then asked for my Bitlocker recovery password – in German. It was obvious what happened: On German Vista machines with Bitlocker enabled, the Windows Boot Manager was still in English, but on Windows 7 the boot manager was also translated – which means that it now failed the integrity check because it was modified.

Luckily i could use our Terminal Services Gateway to log onto my administrative terminal server, where i had the BitLocker Recovery Password Viewer installed, so viewing my recovery key was quick and easy.

After booting into my now (mostly) German Windows 7, i temporarily halted Bitlocker protection, and immediately reenabled. This caused Windows 7 to reverify the state of the Boot Manager, and after i another reboot i was sure that everything was fine.

Oh, and this is one of the rather funny translation episodes: The window is not resizeable and the text doesn’t fit.

Windows 7 is finally nearing it’s completion, and the Release Candidate is finally available. After installing the Windows 7 Beta Build 7000 back in December on my PC at home, i decided to upgrade my work Laptop to Windows 7. The score to the right is from my Laptop.

First of all, i had Bitlocker enabled on my ThinkPad W500, which was running Windows Vista x32 and i intended to install Windows 7 x64. So a direct inplace upgrade was out of the question. I created a backup of the machine, disabled Bitlocker, upgraded my laptops BIOS to the latest version, and booted Windows 7 setup from an USB stick.

Next, i pressed Shift-F10 on the setup screen, deleted all the Windows and Program Files folders, and then started an installation directly on the Bitlocker-enabled drive (this way, i didn’t have to restore all the files i already had on the drive, saving me valuable time).

Windows 7 was done after about 25 minutes, and greeted me with Aero enabled and the 1920×1200 15″ screen already set to a scaling factor of 125%. This is were i also noticed that DPI settings are now user dependant, instead of affecting the whole system. An extremely nice feature, that probably needed quite a bit of work. I set the scaling factor to 115%, which is the best factor between readability and remaining screen real estate for me.

Unfortunately, the switchable graphics driver available from Lenovo did not support WDDM 1.1. I went into the BIOS and configured the machine to always use the Intel graphics. However, i noticed that unlike in Vista, the Intel graphic card did not produce 100% smooth Aero animations. Since i have the power supply connected most of the time anyway, i configured the system to always use the ATI card. This produced better results.

The fingerprint reader does not work yet, but i didn’t invest time in that since i don’t use it anyway. Also, there are issues with Intel AMT, which i don’t use either.

So the base OS worked flawlessly after install. Even switching the graphics card around didn’t phase it, Aero was automatically enabled and the correct resolution configured. WLAN, Audio, everything you would need worked out of the Box.

I joined the machine to the domain, where it sucked down all the GPOs for our corporate network. I unplugged the network cable, and it automatically connected to the corporate wireless network, authenticated by EAP-TLS.

Since our printserver is WS08 x64 box, corporate printing also worked automatically, without any additional work. Of course, all the other group policy settings applied as they should, and i didn’t find any issues yet regarding policy settings.

But an OS alone doesn’t serve a purpose, you need applications. I’ve installed the following applications:

• Adobe Reader 9.1 Works perfectly.
• DIAS-iS Network Client 3.2 Works perfectly.
• DIAS-iS OSP Version 3 for Office 2007 Works perfectly.
• Office 2007 SP1 Enterprise, Visio and PDF/XPS plugin Works perfectly.
• Office 2007 Primary Interop Assemblies Works perfectly.
• Office 2007 VSTO 3.0 Works perfectly.
• Office 2007 Communicator R1 with latest Hotfix Works perfectly.
• Solitas InfoStore Windows Retrieval Works perfectly.
• IBM System i Access V6R1M0 x64 Works perfectly.
• IrfanView Works perfectly.
• Mozilla Firefox 3.1b3 Works perfectly.
• PuTTY 0.60 Works perfectly.
• SonicWALL Global VPN Client x64 Sometimes loses it’s IPsec driver – repairing the program helps.
• Windows Live Messenger Works perfectly.
• Virtual CloneDrive Works perfectly.
• WinRAR Works perfectly.
• tn5250 Works perfectly.

So far, so good. The SonicWALL issue may be annoying, but it’s not a dealbreaker. Judging from my experience, it’s a SonicWALL issue. Opening a bug there won’t help, as they don’t support Windows 7 yet. I can live with that.

Perfomance on Windows 7 on this machine is even better than Vista. I can now fully use the 4GB RAM installed in my laptop. Never used Windows XP on this machine, i can’t compare performance. All the business apps i need to do my job work flawlessly. Printing works flawlessly.

Windows 7 is even better than Vista. But for those that didn’t spend the last three years using Windows Vista, it may be rather hard to get used to all the new stuff. For example, the deployment options between 7 and Vista are both based on WIM imaging, with a few improvements here and there. If you know how to do it on Vista, you can also do it in Windows 7.

As a bonus, the score to the right from my desktop PC.

The past few days were rather busy – i’ve spent them revamping the current network and infrastructure of a small business – deploying new PCs and the first SBS 2008.

First, lets talk about the hardware. It’s important to know that small businesses handle their infrastructure completely differently than large businesses, and in my opinion there are some things that require “unusual” thinking.

Reliability: an SBS server is extremely critical for operation of a small business and they usually do not replace servers after three years

Maintainability: small businesses do not have dedicated IT personnel. Usually, most “heavy” tasks are done by an IT service provider, and the daily IT tasks are done inhouse by someone as a secondary job

Functionality: small businesses are sometimes just as demanding as larger companies – the small size requires setups that maximize the productivity of each employee

So, there are three main aspects one should focus on when deploying an SBS server.

Reliablity is a key aspect. An SBS server is critical for the business and this requires hardware that is highly capable and reliable. After all, an SBS server can serve a Business for up to five years without replacing the hardware. This is why it makes sense to buy really good hardware that lasts that long, combined with appropriate maintenance contracts to get it back up in case you run into problems.

In this case, we decided to use the following hardware:

IBM System x3500
Intel Quadcore 2.66 Ghz 12M (leaving 1 slot available)
10 GB Memory (leaving 6 slots available)
8 2.5″ 147GB 10kRPM SAS Disks (leaving 4 slots available)
ServeRAID 8k with 256MB BBWC (for the first 8 disks)
ServeRAID 8s with 256MB BBWC (for the other 4 disks)
Disk configuration:
RAID 1 consisting of two 147GB Disks
RAID 5 consisting of five 147GB Disks
Global Hotspare
IBM SAS HBA (for tapedrive)
LTO4 SAS attached HH internal tapedrive
Redundant fans & power
IBM Remote Supervisor Adapter II
5Y of IBM ServicePac with committed service option

As you can see, the system has lots of storage and redundancy. It’s also important to know that the SBS server does not run any third party applications (except those necessary for operation), the ERP runs on an IBM POWER machine. With SBS 2008, i would not recommend running any third party applications on the SBS itself – if necessary to run third party apps on a server, purchase SBS premium and run the third party apps on the second server.

The second aspect is ease of use, for which we should use software that can be automated as well as possible. SBS handles lots of things on his own, but we opted for a third party backup application because we still see tape backups as the best way to fulfill most of a customers needs. Especially since LTO drives have WORM media, that can help to comply with certain local laws.

As for the software, i’ve installed BackupExec 12.5 to handle the backups to tape. Tape backups are easier to handle for customers, offer superior performance, and make archival and external storage of data easy. Unfortunately, BackupExec 12.5 does not integrate with the SBS Console (yet?).

For virus scan, we’ve opted for McAfee VirusScan Enterprise. A central management application was unfortunately not yet available, so we deployed McAfee manually on each client, and on the server.

Exchange is protected using ForeFront for Exchange, which has served me well in the past.

SBS 2008 has a nice reporting function, but there are other important notifications: the RSA Adapter notifies for all hardware failures like power supplies, fans, etc. independently through e-mail (which can contact external adresses and even works if the failure killed the server), and allows remote troubleshooting in case the machine does not boot. ServeRAID manager and BackupExec also send daily reports to be viewed by the customer.

Last, but not least, is functionality. After all, customers aren’t like me that want an SBS because they like technology – no, they want an SBS to fulfill certain needs their business has.

In this case, there were several unique requirements regarding mailflow – thanks to the included Exchange 2007 server, which offers a very flexible transport rule system i was able to implement these requirements without having to purchase third party software or even program event sinks on our own.

So far, i’ve had zero issues with SBS 2008 – it worked without any problems and hardware support wasn’t a problem either. Looks like IBM got all the kinks figured out since the release of Windows Server 2008 at the beginning of the year.

The System x3500 is also very nice hardware – it looks like a tower variant of the x3650, which i also like very much. The only criticism i have for the machine is that installing the redundant fan kit is total pain in the ass, mostly because the documentation covers both the x3400 and x3500, and some parts don’t apply for the x3500 (but aren’t marked as such).

It shows that IBM can still deliver top notch hardware at affordable prices.

In case you couldn’t figure it out from reading this far, i really like SBS 2008, and it’s ready for action in a production environment. In case you’re thinking about deploying a new SBS, go with SBS 2008!

In May, we migrated from our old company name and a Windows Server 2003 infrastructure to Windows Server 2008.

About now, we’ve been running on this Infrastructure for 6 months. While configuring back then was very interesting (especially Exchange 2007) and finding vendors which supported their apps under WS08 wasn’t always easy, it worked out.

We’re running McAfee VirusScan Enterprise, which was supported WS08. Unfortunately, the ProtectionPilot Management App was not supported on WS08, which is why it’s running in an WS03 x32 VM. For Backup, we’ve used Symantec BackupExec 12 (since then, upgraded to 12.5).

I’ve been running six productive VMs in Hyper-V since May. The upgrades to the RTM version of Hyper-V ran flawlessly, and we’ve had zero production issues with those VMs. The VMs are a mix of WS03 x32 and WS08 x64.

Except one WS08 Core x32 Domain Controller, all WS08 machines are x64. Even the setting up an x64 print server for x32 clients was less of an issue than i initially thought.

The feature most applauded by our users is probably the TS Gateway.

We currently OCS 2007 in an (unsupported) VM, because we only use the IM functionality right now (the reason that VMs are unsupported is that voice heavily depends on timing, which can be icky in VMs). Our plan is to migrate to OCS 2007 R2 when it comes out, this time running on WS08 on native hardware, so we can start our internal VoIP rollout.

IBM has finally released Director 6.1, which supports running on WS08 x64.

For Active Directory, i run two WS 08 Core DCs, one x64 (on newer hardware) and one x32 (on rather old hardware). We also have an RODC in our branch office. BackupExec has it’s fair share of troubles running on RODCs and so do other apps that depend on SQL Server, like WSUS. So keep this in mind if you want to deploy branch offices – the single server approach worked with DCs, but it won’t with RODCs. Get two machines, one for the RODC, another one for the rest.

For branch office connectivity, we’ve always used DFS-N and DFS-R, which has continued to work flawlessly on WS08.

In our Edge environment, i’ve deployed an Exchange 2007 Edge Server, an OCS 2007 Edge Server and an ISA 2006 server. The latter two are still running on WS03, which i plan to upgrade as soon as it is possible.

I currently only have one unresolved issue, which is NTLM Authentication for Outlook Anywhere. UR4 should have resolved it, but i haven’t gotten around to test this.

As for the clients: We run three quarters Vista, one quarter XP. The XP machines only remain because i don’t have any jurisdiction over them, there are no technical reasons why they shouldn’t get upgraded.

So, after this you will probably assume that i got paid to write this. Well, i do work for a Microsoft Partner, so the Software cost associated with upgrading to WS08 was rather low, as we have Software Assurances for our Volume Licenses and we also get many internal use licenses through the MSPP.

The experience of deploying and running a production system has been a tremendous help for me to get acquainted with WS08 as a platform. I’m currently in the process of deploying my first SBS08 into production, about which i’ll write as soon as that project is done.

Still, i honestly believe that WS08 is ready to deployed. Not anywhere, mind you. Application Support is still an issue, and especially ERP vendors are slow to catch up (not us, though – we supported WS08 TS as platform from the start).

So, what do you think about WS08? Looked at it? Tried it? Running it?

While setting up a new SBS 2008, i ran into a problem with installing KB954960 – a WSUS Update. It immediately said “Abgebrochen” when trying to install it through Windows Update. Trying to install the update manually instantly showed the reason why it didn work, though.

The reason for this failure is that it expects the WSUS groups to have English names – until this is fixed, there is an easy workaround to install the update:

Rename the group “WSUS-Administratoren” to “WSUS Administrators”
Rename the group “WSUS-Berichterstatter” to “WSUS Reporters”

This will allow the update to install. It might make sense to rename the groups back to their original names, in case some other script depends on their names.

Another reason why running non-US versions of Windows is a bad idea in production environments – you just get additional trouble with zero advantages – but that’s for another post.

I’ve compared Hyper-V and ESXi in the past. Since then, the virtualization market has changed. I was also able to get more experience, and have two HP ML110 G5 that run ESXi (albeit unsupported) and Hyper-V Server.

I’ve ran a test environment, mostly to play with Citrix XenApp, SBS 2008 and EBS 2008. Especially the latter, as multi server solution, could be run with multiple roles distributed on Hyper-V and ESXi hosts, which made it interesting to deal with.

Hyper-V Server management

Well, i’ll be blunt. Hyper-V Server standalone management sucks. Big time. The problem here is that you need a Vista machine (which is a problem in the testlab which mostly consists of older PCs), and then you’ll need to create same-username same-password accounts to connect the two.

Also, if you just want to delegate specific VMs, you’ll need to dive into the depths of WMI.

Many of those problems vanish when you’re using a domain setup, with Hyper-V Server joined to the domain. But that’s usually not the case in a test lab.

Add to that that Server Core is still very young, and a lot of 3rd party hardware manufacturers do not have anything Server Core ready yet. Many don’t even announce whether certain tools are supported on Server Core or not.

This doesn’t mean Server Core is a bad idea – it just means that we will need to wait for hardware manufacturers to catch up – also management agents like Backup, Anti Virus, etc. will need to get up to speed for running on Server Core.

ESXi management

ESXi works very well with standalone management. You go to the website, download the VI Client, enter user and password, and you can manage the VM host – you can also delegate permissions easily.

From what i’ve read so far, using single signon and Active Directory with ESXi is more cumbersome than on Hyper-V server. Makes sense.

ESXi integrates a lot of hardware drivers and management. However, fewer servers are supported than with Hyper-V Server. On the other hand, you can use the native ESXi tools to create teams etc., something which you can’t on Server Core because the HW manufacturers do not ship the tools for Server Core.

My opinion

ESXi works very well in a standalone lab environment. Hyper-V is lackluster at best. What does that mean for you? If you want to build a test lab, go with hardware that supports ESXi, or if that is too expensive, go with hardware that at least works with ESXi.