Lukas Beeler's IT Blog

After this weekends cablecom hispeed business fiasco, i talked with cablecom about offering us a more reliable service.

Our current cablecom hispeed business line is ADSL2+ with 20/2 megabits. While the upstream is too low for my taste, i haven’t really seen better offers.

I talked with a sales on the phone – for about 200 CHF more, we could get 20/2 SDSL (which sounded strange) and a 20/2 DOCSIS backup line, together with a “Bronze” level SLA. This sounded very attractive to me and i told the sales to send me the offer.

In the written offer, the ominous 20/2 SDSL was downgraded to 4/4 SDSL (which made much more sense). Of course, downgrading our internet connection from 20/2 to 4/4 seemed like a rather bad idea. We have about 30 people working here everyday, and almost all of them really use the internet to do their job. We’ve upgraded from 6/.6 ADSL to the current cablecom connection, because 6 megabit downstream wasn’t fast enough.

So i asked what else they could offer us – for 500 CHF more than we pay today, we could get 8/8 SDSL with a 20/2 DOCSIS backup. That still didn’t sound interesting to me.

I, personally, think 1000 CHF per month would be okay for a redundant 20/20 connection or something in this direction. My current connection at home is 25/2.5 – for 75 CHF a month. It works well enough, and the last failure i had was fixed in three days. Just like the failure we had on our 500 CHF per month 20/2 connection. This should be a telltale sign that something is very wrong with either the pricing or the service level.

The next question i asked if they could do a 20/2 ADSL with a 20/2 DOCSIS backup. Apparently, that’s not technically possible right now, but they might introduce this later this year. That sounds attractive to me.

All in all, i still think that cablecom hispeed business sucks. They can’t be bothered to do a 5 minute fix in a 2 hour time window on Friday evening. Then, they make one ludicrous offer that noone can take serious after the other.

I’m pretty sure that cablecom doesn’t really understand what small businesses need.

As a side note, if you work for an ISP and think you can make us a better offer than cablecom, i’d be very much interested. Send your stuff to l dot beeler at acommit dot ch. We will be moving to Horgen/ZH at Seestrasse 202 in March 2010 and need 32 static IP addresses.

Since about one and a half year, we’ve been using cablecom hispeed business for internet access.

Shortly after installing the line back in 2008, we’ve ran into an issue where cablecom hispeed business blocks GRE packets. After almost three days and speaking with a variety of technicians, they were finally able to resolve the issue.

Now, we’ve run into another, much more grave problem. Since about 15:45, a variety of hosts on the Internet aren’t reachable and of course several other hosts can’t reach us.

Of course this isn’t a clear-cut “my DSL modem has no link” issue – so cablecom currently isn’t even trying to fix the problem. I’ve been on the phone twice, never get any callbacks and don’t get any updates on the state of the problem resolution.

Fact is, some hosts can reach our OWA 2010 and some can’t. Nasty thing is, Swisscom’s GPRS/UMTS IP addresses can’t – this means no push-email for all 35 of our employees. Since we’re working for a rather important project (ERP and POS implementation) this weekend, this is a big issue for us.

It also looks interesting in a tcpdump – some packets just get lost – and from other hosts it works without any issues.

The 77. addresses are cablecom hispeed business, the 217. are my cablecom residential connection. In the first part, we see a TCP connection to port 80. In the second part, we see a ping -t. As you can see, there are a lot of dropped packets.

23:12:12.629457 IP 217.162.252.98.18417 > 77.59.216.227.80: S 4006182815:4006182815(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
23:12:12.629479 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:15.826736 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:22.026734 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>
23:12:34.026733 IP 77.59.216.227.80 > 217.162.252.98.18417: S 1280362581:1280362581(0) ack 4006182816 win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 6>

08:51:49.642995 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 65
08:51:49.643024 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 65
08:52:00.641330 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 68
08:52:00.641345 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 68
08:53:16.641813 IP 217.162.252.98 > 77.59.216.227: icmp 40: echo request seq 84
08:53:16.641829 IP 77.59.216.227 > 217.162.252.98: icmp 40: echo reply seq 84

Cablecom gets 180 CHF per month for 24/7 support. The case has now been open for 7 hours, with no resolution in sight. There’s no escalation path and there are no workarounds – we don’t have redundant connections.

Interestingly, one of our customers who also uses cablecom hispeed business had a similar issue, that lasted for roughly three weeks – one of their IP addresses wasn’t reachable externally, from one minute to the other. Unfortunately for us, all of our public IP addresses are affected by this issue, so we don’t have an easy workaround.

Of course, for some part we’re also to blame. Luckily i’m not one of the higher ups who gambled with non-redundant internet connections and lost.

Have you made negative experiences with cablecom hispeed business? Positive ones? Was support able to fix your issues quickly?

Update: I’ve called cablecom again on Saturday at 09:00. Apparently, these sort of issues are supported on a best-effort base from 9 to 5, and not covered by our 24/7 support contract. We will have to wait until monday – they will not look at this issue further during the weekend.

Update: Monday morning, 11:00. Problem is still unsolved.

--- hor-fw-01.acommit.ch ping statistics ---
20 packets transmitted, 3 received, 85% packet loss, time 19012ms
rtt min/avg/max/mdev = 20.490/21.360/22.585/0.891 ms

Update: Monday morning, 11:36. Problem is now solved. According to the Tech i talked to, the he fixed the issue in 5 minutes. He could’ve done that on Friday, but apparently noone at cablecom felt like doing so. The issue was that cablecom configured our new line for the planned office move and configured load sharing between the new line for the new office and the old line. Since the new line didn’t physically exist yet, half of the packets were dropped.

Thanks to the Tech who fixed the issue, no thanks to cablecom in general for wasting an entire weekend on what could’ve been a five minute fix on Friday evening.

The company i’m working for, Acommit AG, is hiring again.

Currently, we’re looking for:

Project Manager (PDF)
Sales (PDF)

Exchange 2010 was released last Monday, the 9th. Today, we have Saturday the 14th – and i’m done with the Migration to Exchange 2010.

Sure, there are loads of MVPs and TAP-Members that have migrated to Exchange 2010 a long time ago, but i’m still proud of this.

At a starting point, i had a Exchange 2007 SP2 machine, with one Mailbox database, no public folders and 35 Mailboxes that used up 25GB of space. Moving this is simple enough, but the issue is that our Exchange isn’t virtualized, and i couldn’t get my hands on new hardware since the current box was only a year old.

Since in-place upgrades are not supported, i needed a temporary server for the migration. I used an HP ML110 from the Lab, which offered enough space to migrate.

Another issue was BackupExec 12.5, which did not support Exchange 2010 yet. Fortunately, Exchange 2010 (and 2007 SP2) can be backed up by using Windows Server Backup. So my goal was to just let WSB backup to a file server, and have BackupExec pickup the files from there. This way, i will get a reliable, clean and supported Exchange backup, and still have it on tape.

To Migration itself was straightforward and easy. There’s already _lots_ of content on the web about Exchange 2010, most of it from the RCs or Beta of course.

I followed the Migration Guide from TechNet, which worked out well enough. Unfortunately, the iPhone does not support Exchange 2010/2007 coexistence, which made it necessary for several people to manually reconfigure their phone.

Removing Exchange 2007 worked without issues, but after moving all the Exchange 2010 data back to the real hardware and removing the temporary server i ran into the issue of moving arbitration mailboxes, which fortunately was already documented widely on the web.

In the end, upgrading from Exchange 2007 to 2010 while keeping the same hardware is not difficult, it just needs a bit more time.

The iPhone does not properly support coexistence between Exchange 2010/Exchange 2007. See this TechNet Posting.

The error message in the IIS Log looks like this:

RdirTo:https%3a%2f%2flegacy.contoso.com%2fMicrosoft-Server-ActiveSync_LdapC2_LdapL15_Error:MisconfiguredDevice_Budget

A long time ago, i wrote a review of the HP ML110. In the comments, Paul indicated that the Performance of the E200 controllers was pretty bad, and i promised i would do benchmarks of that. Now we have a year later, and i indeed finally got the time and did those benchmarks.

For the benchmarks, i’ve used the free version of HDtune. I’ve benchmarked four systems, and five different disk configurations. Note that the free version only does benchmarks for disk reads, and it’s a not a very pervasive test. None of these benchmarks are scientific. They should serve as a general indicator of performance, not as a final world on this topic. I don’t have that much clue about benchmarking.

The first system is my computer at home: It has an i7-920 CPU at stock speed, with 3x2GB RAM at 1333 Mhz (which is a slight overclock, but within the spec of the memory i purchased). Attached to it’s ICH10R controller are an Intel X25-M G2 160GB (Firmware 02HA) and a WD1001FALS (1TB, 7×24), running Windows 7 x64.

The next system is my work laptop, which is a ThinkPad W500 with a 2.53 Ghz T9400 C2D CPU, with 4GB of RAM. Attached to it’s onboard controller is an OCZ Vertex 120GB (Firmware 1.40), running Windows 7 x64.

The third system is our Exchange Edge server, on which i dared to install a benchmark utility. It’s an IBM x3250 with two 70GB 15kRPM 2.5″ SAS drives installed, attached to an onboard LSI1064E SAS controller. The system has a Xeon 3040 2.4Ghz Dualcore CPU and 5 GB RAM. It is running Windows Server 2008 x64 SP2.

And the final system is a HP ML110 G5 with a 2.33 Ghz Xeon 3065 CPU, 8GB of RAM and a E200 with the latest firmware (1.78). Attached to that are 4 WD1001FALS drives in a RAID10 configuration. The E200 has a backup battery and 128MB of cache installed. The system is running Windows Server 2008 R2.

Please note that none of these benchmarks are scientific. They were done on real systems, with workload minimized as much as possible, but virus scanners and other mandatory background applications active. Both the laptop and the desktop have not been formatted since Windows 7 RC was installed (i migrated to Windows 7 RTM using Windows.old), but the ML110 was freshly setup and the only application that’s been installed so far is the HP ACU and Forefront Client Security. The Exchange Edge server has been in use since May 2008. As such, the ML110 is the “cleanest” machine out of these four.

Intel’s X25-M G2 160GB on an ICH10R (AHCI Mode)

This is how a graph should look. It’s nice, it’s clean, it’s fast. Intel’s X25-M G2 shows how a modern SSD and storage subsystem should behave. Clean, predictable performance.

OCZ’s Vertex 160GB on an ICH7 (AHCI Mode)

Here’s the OCZ Vertex. It’s running on a machine that’s a lot slower than the one the X25-M is attached to, and it’s storage controller is also quite a bit older. It still shows remarkably good performance. It should also be considered that this Vertex is quite a bit older – it was bought in May 09. It’s still very fast and responsive and a good SSD.

2x IBM’s 73GB 15kRPM 2.5″ SAS Disks on an LSI Logic 1064E SAS Controller

As you can see, this is the performance you get from the server hard disks on an entry-level controller in an entry-level system. It’s not astonishing, but the performance is very well acceptable.

Western Digital’s 1001FALS 1TB on an ICH10R (AHCI Mode)

Here’s how the Western Digital disk behaves on a proper controller. Please note that this is a single disk, not part of a RAID array. The performance is quite good.

4x WD’s 1001FALS 1TB on an HP E200 in RAID10

And here’s how it shouldn’t look. Compare this to the stand-alone disks above, which exhibits better performance. HP fucked up bad on this one, and there’s no fix in sight. Stay away from the E200.

And as a final word: I really don’t have much of a clue about benchmarking. If you see an obvious error here, please state what you think. If possible, i will try to correct it.

Update: As requested in the comments, i upgraded the E200 to Firmware 1.84 and redid the benchmark. It looks roughly the same.

Exchange 2010 will be out soon, and i’ve been preparing for the migration. One of the more important parts is that you will need to have both Exchange 2007 and Exchange 2010 client access servers accessible from the Internet.

If you’re following the recommended deployment method for Exchange 2007, you’ll already be using a SAN certificate in order to publish AutoDiscovery and OWA. For coexistence of Exchange 2007 and Exchange 2010, an additional name will need to be added to your SAN certificate.

With most CAs, this is a pretty straightforward process that can be done using their web interface, since the private key doesn’t need to be touched. After modifying this, you will get a new .crt file containing the certificate, but no private key (which is correct).

However, importing this into Exchange 2007 using Import-ExchangeCertificate doesn’t work – Windows won’t know which private key is associated with the newly imported certificate. When you try to use Enable-ExchangeCertificate, you will receive the following error message:

Enable-ExchangeCertificate : The certificate with thumbprint 1234 was found but is
not valid for use with Exchange Server (reason: PrivateKeyMissing).

I searched high and low on how to replace a certificate without touching the private key, but i didn’t find anything. So i turned to the community for support – MCSEBoard.de is an excellent Windows community for those who speak German.

Unfortunately, noone knew an easy way either – the suggestion was to use OpenSSL to create a new keystore.

This was rather easy, but i didn’t find any guides on the net on how to do this, so i’m publishing this here in the hope that it will help others with the same issue.

• First, you need to export the key including the private key using the Windows certificate manager. Open an elevated MMC, add the Certificate snap-in and focus on the Computer certificate. Click “Personal”, and then export the certificate with the private key.
• Download and Install OpenSSL for Windows
• Issue the following command: openssl pkcs12 -in mykey.pfx > out.txt
• Open out.txt using an LF-aware text editor, such as Notepad++. Save the PRIVATE KEY part to a textfile called key.pem.
• Save the certificate to a file called cert.crt
• Issue to the following command: openssl pkcs12 -export -in cert.crt -inkey key.pem -out newcert.p12
• Copy the newly created newcert.p12 to the Exchange server.
• Open PowerShell and run the following command: $secureString = ConvertTo-SecureString "blubb" -AsPlainText -Force – Replace “blubb” with the Passphrase you used in the step before
• Run Import-ExchangeCertificate -path newcert.p12 -pass $secureString to import the certificate back into Exchange
• The rest is as usual – use Enable-ExchangeCertificate to enable the certificate.

And that’s it. It might be a bit cumbersome – and i really hope that there is an easier way to to this. If you know, let me know so i can update this page.

Microsoft has finally offered a fix to the OCS issue described here

See here for the fix and it’s description KB974571

Click here to download the ocsasnfix.exe directly, which will fix the incorrect ASN License data – something which i already guessed about in my previous post about this issue.

SBS 2008 is out for roughly a year. In this time, i did four deployments of SBS 2008, each with 15-30 users.

During this time, i’ve gained valuable experience, which i’ll try to share here so that others can profit from it. Take all this with a grain of salt, as some observations may simply be my fault. Also, as times changes these things might change too.

Software

• Make sure to install Windows Server 2008 SP2 after installing SBS 2008. Some media may come with SP2 already preloaded. You can use the normal SP2 package that’s also used for Vista and the normal Server 2008
• Do not install SBS rollup updates before completing the configuration wizard. This is extremely counter-intuitive, but is described on the Official SBS blog
• Installing Exchange 2007 SP2 requires you to follow special considerations Here
• Installing WSUS 3.0 SP2, which is needed to support Windows 7, is currently not recommended. I was able to do this without issues on my lab machines, but others have reported issues doing this on machines that were in production. If you’re deploying a new SBS server, this should probably be safe to go. But make sure to test functionality afterward.
• Always use the answer file to deploy SBS 2008. This will make it possible to choose a custom domain name. Read my post about choosing your AD DNS namespace
• Do whatever tasks you can do using the SBS console. Resist of using the normal administration tools as much as possible, as you can break SBS with them easily.
• Ensure that the AV software you install is compatible with WS08 x64. Symantec Endpoint Protection Manager works well – Forefront Client Security on the other hand requires a seperate server running 32bit Windows for management. You may consider deploying FCS unmanaged in smaller environments, and configure FCS using the FCS ADM File

Hardware

• Use servers with the new Xeon 5500 CPUs. Read my x3650 M2 tips to find more about them. Consider using an E5530 or faster CPU. Using two CPUs (for a total of 16 virtual and 8 physical cores) makes little sense.
• Buy enough memory. Lots of it. Really. I mean it. You’ll need lots and lots of memory. I would consider 12GB to bare minimum. In a 3x4GB configuration which makes the most sense for the Xeon 5500 setups, this is quite cheap. Consider more memory if you intend to run SQL Server as, consider bumping the memory to 24GB. Remember that you can only use the first 8 slots in a single socket machine.
• Buy enough disks. A good starting layout is 8x147GB 2.5″ disks. Use a RAID 1 for the OS, another RAID1 for Exchange and Sharepoint, and a RAID10 for Data and WSUS. This is all up for debate of course, and it might make sense to consider other disk layouts.

If you have any additions, think i’m wrong somewhere just send in a comment.

Update: See here for Microsoft’s description of this issue KB974571

Security updates are important. And as we’re currently an evaluation setup for OCS 2007 R2, i’ve decided to install todays batch of security updates on these lesser important machines first. And after a reboot, OCS 2007 R2 was broken.

A quick view into the event log revealed that OCS 2007 R2′s evaluation license has expired. Now, this seemed very strange as i’ve installed from volume license media. I’ve the checked the media again, but they weren’t evaluation media.

Here’s the message in all it’s glory:

Event source: OCS Server
Event id: 12290
Event text: The evaluation period for Microsoft Office Communications Server 2007 R2 has expired. Please upgrade from the evaluation version to the full released version of the product.

Maybe i really did use other media to install it? I doubted myself, because that’s usually the most reasonable approach to take. The error is usually behind the keyboard.

Luckily, Microsoft has published documentation on how to upgrade an evaluation version to a full version. Unfortunately, this didn’t work, because as it appears i was running a Volume license version of OCS.

EVALTOFULL parameter cannot be used with currently installed license type Volume

At this point, i was pretty sure that this wasn’t my fault. There has been an issue with the OCS 2007 R2 Evaluation Media expiring at the wrong point in time, but apparently this has been sorted out and did never affect the full versions of OCS 2007 R2.

So i was bummed. A quick view using process monitor revealed that the licensing information was most likely to be stored here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RtcSrv\InstallInfo\ValidationData

I created a backup of that part of the registry, and then renamed the key. I got a file not found error, and created a new key of the same type and wrote binary data of the same length into it. This yielded the following error:

The service is shutting down due to an internal error.

Error Code: 80093102 (ASN1 unexpected end of data.)

At that point, i was pretty sure what might’ve caused this – the MS Crypto API security update KB974571.

I removed the update, rebooted the machine, and OCS 2007 R2 was up and running again, without any issues.

I’ve already opened a case with Microsoft to get this sorted out.

Update:
Appears that this is an official issue: See here