Lukas Beeler's IT Blog

Exchange 2010 was released last Monday, the 9th. Today, we have Saturday the 14th – and i’m done with the Migration to Exchange 2010.

Sure, there are loads of MVPs and TAP-Members that have migrated to Exchange 2010 a long time ago, but i’m still proud of this.

At a starting point, i had a Exchange 2007 SP2 machine, with one Mailbox database, no public folders and 35 Mailboxes that used up 25GB of space. Moving this is simple enough, but the issue is that our Exchange isn’t virtualized, and i couldn’t get my hands on new hardware since the current box was only a year old.

Since in-place upgrades are not supported, i needed a temporary server for the migration. I used an HP ML110 from the Lab, which offered enough space to migrate.

Another issue was BackupExec 12.5, which did not support Exchange 2010 yet. Fortunately, Exchange 2010 (and 2007 SP2) can be backed up by using Windows Server Backup. So my goal was to just let WSB backup to a file server, and have BackupExec pickup the files from there. This way, i will get a reliable, clean and supported Exchange backup, and still have it on tape.

To Migration itself was straightforward and easy. There’s already _lots_ of content on the web about Exchange 2010, most of it from the RCs or Beta of course.

I followed the Migration Guide from TechNet, which worked out well enough. Unfortunately, the iPhone does not support Exchange 2010/2007 coexistence, which made it necessary for several people to manually reconfigure their phone.

Removing Exchange 2007 worked without issues, but after moving all the Exchange 2010 data back to the real hardware and removing the temporary server i ran into the issue of moving arbitration mailboxes, which fortunately was already documented widely on the web.

In the end, upgrading from Exchange 2007 to 2010 while keeping the same hardware is not difficult, it just needs a bit more time.

The iPhone does not properly support coexistence between Exchange 2010/Exchange 2007. See this TechNet Posting.

The error message in the IIS Log looks like this:

RdirTo:https%3a%2f%2flegacy.contoso.com%2fMicrosoft-Server-ActiveSync_LdapC2_LdapL15_Error:MisconfiguredDevice_Budget

Finally, Office 2010 is now available officially.

I’m currently running Windows 7 Build 7100 x64 on my laptop, so i decided to uninstall Office 2007 and install Office 2010, the x64 version.

Unfortunately, the x64 version does not support running any x32 plugins. This is unfortunate, since one of the key features of our ERP package is a tight integration with Microsoft Office, which even i used from time to time. We call it the Office Solution Pack, which has been completely rewritten as a native .NET Office 2007 Plugin. The screenshot to the right is slightly outdated, but it clearly shows the integration we had.

I’m already pushing our developers for a x64 version of the plugin, but i suspect it will take a lot of time until all applications we are using (e.G. System i Access) are running on Office 2010 x64.

Also, its important to know that having 32bit plugins around after installing Office x64 will lead to several nasty error messages. You will need to uninstall the 32bit plugins before installing Office x64. Interestingly, the Primary Interop Assemblies come now shipped with Office by default. This alleviates installing them, streamlining the plugin installation process.

This also means that Office 2010 x64 will be a very niche product, and most people will opt to deploy the 32bit version. This isn’t bad – now that an x64 version is around, plugin developers can start adopting and we may see larger deployments of Office 2010.Next x64.

I’m now running my productive mailbox against Office 2010. Yep, might be risky, but it also gives me a very easy way to learn a new product before we’re actually pushing it to customers. I’ve found that i’ve never learned all that well in a separate environment.

So, whats new in Office 2010? Well, the only application i use daily is Outlook. And we will see how it plays out. Outlook 2010 now also comes with the new ribbon interface, and so far i like it. We will see how this plays out.

I’ve had my share of experiences with ZyXEL equipment, like the ZyWALL vs. Exchange post i did a few years ago.

But today i experienced the most grave issue with their equipment that critically impacted a customers business.

The customer has two sites – an HQ with an SBS 2008 and a branch office with two Lenovo SFF machines running Windows Vista Business. Both sites are using 20/2 VDSL lines from Swisscom, with ZyXEL P-2802HWL routers.

There is an IPsec VPN configured between these two sites. This has been working fine since January.

Now, about a month ago a telecom service company installed VoIP telephones in the branch office, and enabled QoS on both ZyXEL routers.

Since then, Outlook was unable to synchronize correctly with the SBS server. Unfortunately, the customers personnel isn’t that technically savy, so they weren’t able to tell that they had a problem – because smaller e-mails were able to successfully synchronize, but larger ones failed. This led to very inconsistent states of the OST files, with some mails there and some mails not there.

When i arrived at the branch office i didn’t have a single clue what the issue was or may be. At first i suspected an Outlook problem, so i deleted the OST file. But from there on, nothing happened – Outlook wasn’t able to download anything.

Next, i tried to copy a 50kbyte Excel file from a share to the local computer. This worked. So i tried a 2 megabyte Word file. This failed about halfway through, with Explorer just hanging there and doing nothing. From that point on, i suspected a network issue, but the fact that copying a 50kbyte file worked and a 2 megabyte file didn’t was very odd.

Using Outlook with Outlook Anywhere also worked (when the VPN tunnel was downed).

Whenever i’m confronted with strange network problems, i suspect MTU issues (which was my first “real” network problem i solved back on my first ADSL line – took me weeks for a simple fix). ping -l 5000 CUSTSBS01 worked. ping -l 15000 CUSTSBS01 worked, too. So thought it wasn’t an MTU issue.

Disabling QoS on the ZyXEL router fixed the issue, but made the phones unusuable while Outlook was filling it’s OST files.

So i ran through the usual check points – tcp checksum offloading, chimney, receive window autotuning, reboots, etc. Nothing helped. At the end i was just changing network settings at will. But nothing helped.

Out of any reasonable ideas, i changed the MTU to 1300. That fixed it – with QoS enabled and the NIC MTU of the two machines, everything was working as it should. File transfers worked, Outlook worked, Phones worked.

Don’t buy ZyXEL.

HP recently had a special offering for an ML110 G5 hardware bundle, that consisted of the following parts:

• Intel Xeon 3065 2.33Ghz 4MB L2
• 512MB ECC RAM
• E200 SAS Controller (8 Ports, 128MB BBWC)
• 2x 160GB 7.2kRPM SATA Disks

For less than 400 CHF. As i needed a machine to run SBS2008 at home, and my current one wasn’t 64bit capable, this seemed like a good buy, especially because the E200 with BBWC alone is worth around 300 CHF.

Of course, i needed more RAM and disk space. I also ordered 4x 2GB memory modules (with ECC) from a third party memory manufacturer (Transcend) – priced at around 80 CHF each. I also ordered 4x Western Digital 1TB disks that are optimized for 24 hour use, priced at around 180 CHF.

This brought me to a total price of around 1500 CHF. I had two 160GB disks that i didn’t have any use for (except throwing them at people i don’t like).

1500 CHF is a lot of money for me, but for a company it’s nothing – still, this is ideal for experimentation. The free ESXi supports the E200 SAS controller, making it easy to build a test lab based on VMware – also, Windows Server 2008 x64 and Hyper-V also run flawlessly on the machine.

The machine is also very quiet, making it possible to use it in a normal appartment or in your office.

You get what you pay for still applies – the machine has no remote management features, only a single network port, forcing you to use the same port for management and virtual machine traffic, which can be acceptable in a test environment. HP’s System Insight Manager is not supported on this machine, either.

The case is very small, resembling a normal HP client minitower. The mainboard supports ECC memory, which is becoming more and more important with todays memory sizes. Unfortunately, it only offers four memory slots with a maximum capacity of 2GB per stick, maxing the machine out at only 8GB of RAM.

The integrated E200 SAS RAID Controller has a 128MB BBWC card, that allows it to use it as a write cache, and enables licensing to use RAID5. In my case, i used RAID10. The disk performance is better than anticipated, even though i’m using slow consumer drives, the performance for running VMs is acceptable.

The machine has three x8 PCI-E slots and a single PCI slot. One of the x8 slots is used by the E200 controller.

This offer is still available under HP Part# 470064-639, and there are still some companies that are selling it for the lower promotion price.

I’m currently running SBS2008 directly on the hardware, with not virtualization in-between. The performance is good, but i’d still never use such a setup for a production deployment at a customer – the management options, hardware flexibility, redundancy etc. just aren’t fit for production.

Update: I was asked about Linux compatibility on this machine. See the official HP Linux compatibility list. The E200 SAS RAID Controller is supported by the cciss driver, which is in the vanilla linux kernel. So most distributions will be able to install on this box – support is another matter, though.

There is no easy way to get official support for non-corporate versions of Linux, like Ubuntu. My usual way in those scenarios is to run Linux as a VM under ESXi, but that doesn’t work with the ML110 as ESXi is not supported (but works).

Microsoft released the Windows Small Business Server 2008 RC0 today.

For those of you who do not know SBS: SBS has traditionally been a single server setup with Exchange, SQL Server and ISA Server. It consolidates all “big” Microsoft technologies on a single server. This contradicts most “Best Practices” published by Microsoft, and as such SBS has always been seen as the red-headed stepchild in the Windows Server Family. SBS 2008 aims to improve several of these points (especially with the Premium Edition shipping with TWO server licenses).

After a 6 hour downloaded that trickled in at a few meager 200kbyte/s, i was finally able to get started with it.

SBS 2008 now demands x64 hardware – so for testing i used an IBM x3650 running Windows Server 2008 Enterprise with the Hyper-V RC1. Hyper-V supports 64bit guests. Other hardware requirements have also gotten steeper – you’ll need 4GB RAM minimum (though i launched the VM with only 2GB). The Premium Edition now comes with licenses for two servers – finally making it possible to have redundant domain controllers even in a Small Business setup without paying for full server licenses.

The first half of the setup is similar to what you know from Windows Server 2008 and Windows Vista – you boot, select the disk, have the chance to enter a product key, and finally start the installation. After that, the WIM image is expanded to the harddrive. The machine reboots after installation, and this is where things get different.

After booting, you’ll land in “Install Windows Small Business Server 2008″ Wizard. This can be mostly automated using an Answer file, which is mandatory when migrating from earlier versions. I will check that out later and proceed with a simple installation without using an Answer file.

I get nagged by a “Insufficient Hardware Screen”, reminding me that my (virtual) machine only has 2GB RAM. After acknowledging the warning, i can setup my date and time. I choose the CEST timezone, and move onwards.

Next, a screen confronts me with the fact that i don’t have a NIC – which is true. The machine is running on Hyper-V RC1, and i wasn’t able to install the integration components yet. Luckily, there is a “Browse” Button, where i can launch the Integration Services setup. Installation of the Integration Components worked fine, the machine rebooted. I hope Microsoft packs the Hyper-V RTM bits into SBS RTM. This would make it easier to install it into a VM, but as you can see, it’s not much of a hassle.

I was back at the beginning, at the start of the SBS Wizard. Luckily, i was now able to use the mouse after installing the Hyper-V IC. Next, i get an Update Dialog, asking me if i want to update my server. I choose yes and have to wait.

Next, i was asked to enter my company information. Next, i was able to name my server and the NETBIOS name of the Domain. I was not able to choose a DNS Name for the Domain (This is only possible if using an Answer File). Interestingly, Dashes “-” were not accepted as part of the server name. I wonder why – our production setup uses dashes extensively in server names, and so does Microsoft (judging from their Mail headers).

Then i was asked to create an an administrative account – a good idea. The “Administrator” account shouldn’t be used in a production setup, instead each user with administrative rights should have their own account. SBS enforces this – a very good idea.

After confirming Server name, Domain name and Company name, the installation continued on it’s own. This took a good amount of time, during which the server restarted several times – of course completely unattended. No need to play disc jockey or logon – much better than SBS 2003.

After the installation, i was greeted with a screen that told me that it was unable to install some critical updates. Clicking on that bar revealed an IE7 404. I checked the IP configuration – the server was configured to use 192.168.0.2, and didn’t have a DHCP server installed. There was no default gateway set yet.

Next, i launched the “Connect to the Internet Wizard” which told me that i was already running a DHCP server – which makes sense. After choosing “Postpone”, the Wizard aborted. That wasn’t quite what i was hoping for.

I shut down the VM and reconfigured it to use a private LAN. That way, it wouldn’t have a connection to the internet, but it wouldn’t have to deal with a DHCP Server either. But SBS didn’t like that either – it wanted a router. So i setup a second VM running IPcop (which works flawlessly on Hyper-V using Legacy NICs and a small virtual hard drive).

It was interesting to see using “tcpdump” what SBS did under the covers to detect the router. ARP scanning, IPv6 Discovery, Everything. This seems rather well designed. It was sucessfully able to detect my IPcop VM which didn’t have a DHCP server.

Next, i started the wizard to enable my domain name. It seems that SBS will be able to do some of these things automatically if you live in the US. Here of course we have to do things manually.

So far i don’t like that SBS tells me very very few technical details. But this might be because Microsoft somehow thinks that a Small Business Owner will setup SBS on their own (which just seems a horribly stupid design decision).

Next, it told me that i couldn’t configure my Internet Router properly (my IPcop instance didn’t have UPNP support enabled). It’s interesting to see that it wants to forward port 25 to the server. It looks like the POP3 Connector was finally killed off for good. That’s very good to hear! Unfortunately, it’s still there. Just hidden.

I also had to configure outbound email properly, with the ability to configure a smarthost or use direct sending. There is also a wizard to easily create a properly signed official SSL certificate – nicely done and will surely improve the security of the many SBS setups that are out there.

SBS 2008 also ships with OneCare for Servers already preinstalled. You can just activate it with a few clicks. I don’t see this very positively – I’ve made a few bad experiences with ForeFront Client Security, which OneCare is based on. We’ve been using McAfee for the past. So in the future for SBS setups we will have to either remove OneCare from the SBS, or deal with having multiple virus scanners on the network (a management nightmare).

Another interesting tidbit is that UAC is enabled in approval mode, just like on standard Windows Server 2008 installations when not using the Administrator account. This is annoying, IMHO. I don’t have a problem with UAC on my desktop because i usually use my desktop to work and not change settings – but when i’m logged onto a server, i want to change settings all the time.

That’s it for the first impressions. I will have a closer look at SBS 2008 over the following days and will keep you all updated.

Pictures are here:

Outlook Anywhere / Outlook Autodiscovery on Windows 2008 still has some problems.
Read this most excellent post that has all the details.

Long story shorts: Modify the hosts file, remove the IPv6 localhost (::1) and then add hosts entries for your server. I would recommend against disabling IPv6 on the Exchange server, as this is probably not a recommended or supported configuration.

The root cause is that Outlook 2007 can’t contact a DC/Domain Controller using RPC over HTTP/Outlook Anywhere when used on Windows Server 2008.

Also note that NTLM Authentification for Outlook 2007/Outlook Anywhere is broken on Windows Server 2008.

So i’ve been playing with Office Communication Server 2007 to pass time. Thanks to the Microsoft Partner Licensing Program we can use this software internally, in production, without paying anything.

After playing with it in VMs for a few days, I decided to deploy it internally. Of course, the current deployment is not very integrated – our PBX is years old, and we have no chance to get any decent sort of integration, and we’re not yet on Exchange 2007 (though this is planned). As such, I didn’t expect to much usefulness out of. Boy, was I wrong.

OCS 2007 is several products in one, and it has a few drawbacks in a small business deployment (because it was designed for bigger environments). The price of the product isn’t prohibitive for a small business – 1500 CHF for the server, and 100 CHF per CAL (for Standard versions – the Enterprise versions are more expensive).

So, what features can one expect from OCS2007?

Services

Instant Messaging

One of the OCS2007 functions is an internal Instant Messaging server, with all the standard features you probably already know from ICQ, MSN et al. This part could easily provided by using e.G. an internal Jabber server and a Windows Jabber client like Pidgin. So why use OCS2007 for instant messaging? The reason is easy: Integration. The server software integrates into your Active Directory environment. You extend the AD schema, and all the user information is stored directly in Active Directory, with no need to maintain yet another user database. While that’s an advantage, it not much of a selling point (because the CEO usually doesn’t care if need 3 more minutes to add a user).

So let’s talk about integration on the client. After installing Office Communicator (the IM/VOIP client for OCS) on the client, you will notice full IM integration into Outlook, see the status of all the recipients and senders of the mail. This is a very nice feature, because it offers you information at a glance, without having to open the IM GUI to see whether someone is available for a quick follow up or not. But it gets better: this Integration also works in Sharepoint Services 3.0 and MOSS 2007. Also, the Unified Messaging part of Exchange Server 2007 integrates nicely into OCS 2007. You can check your voicemail using OCS 2007, with a fully graphical interface (similar to how the iPhone handles it’s voicemail)

Besides the ability for instant messaging, there is another very important feature – at least in our company: availability and presence. We have a HQ and a branch office, and our HQ is split over three floors. So usually it’s not easy to tell if someone is at his workplace or not. While Outlooks calender helps to establish the general whereabouts of a person, its not at-a-glance, and it doesn’t help if the person just isn’t at his desk (for whatever reason).

Office Communicator sets your presence to away at the instant you lock your machine, which people do when they walk away from their desk. As such, you can tell whether someone is currently working at his desk or not. This is very cool, and helps to save time on unnecessary phone calls to which no one answers.

There’s also a web client – Office Communicator Web Access. At the first glance, it is indistinguishable from the full desktop client, so the web interface is very nicely done.

Voice over IP (SIP)

OCS 2007 is also a fully blown VoIP solution. I can’t talk about this part too much – i haven’t worked with the mediation server or more enterprise VoIP integration (as said, our PBX doesn’t support that).

The Softphone client, integrated into the Office Communicator works nicely though, the voice quality is normal, and we didn’t have much problems using it over WAN lines.

You can also connect hardware IP phones to OCS2007, which should work with standard SIP phones – not having one, i didn’t test this. There are some very nice looking OCS specific IP phones out there.

Live Meeting

I’ve attended a few Webcasts done using Live Meeting 2005. With OCS 2007, you can now host Live Meetings (using the 2007 client) directly in your company, with no need for any hosted services. This feature might not be terribly useful if you’re working for a single-location Small Business, but it can be a timesaver when spread across the country (or world). Live Meeting also integrates into Outlook (see the above screenshot).

It works flawlessly, and i had few problems using Live Meeting. Didn’t really deploy this into production yet, though.

And more

OCS 2007 can also do a lot more stuff than i mentioned here. Most of this, like CDR and Archival is not necessary (or financially viable) in Small Businesses, so I didn’t invest too much time.

Drawbacks

So, what are the drawbacks of OCS 2007 in a Small Business? The main point i see here is that you need at least three servers – a Standard Server (hosting all the services), a mediation server for connecting to your PBX, and an Edge server offering internet connectivity. These are at least three OS instances that need to be maintained. Add to that the cost of either a proper virtualization server, or a few 1U boxes, and you’ll get into unviable price regions pretty soon.

For basic functionality, you can leave both the Mediation and the Edge server away. This means no integration with your PBX, and no external access to your server – at least in theory.

If you just need external access to IM, you can create appropriate SRV records in your public DNS, and forward port 5061. This will not result in a clean service, but it’s better than nothing. But without a proper edge server, you won’t be able to access other IM networks. Not cool.

Microsoft should really make single-server deployment possible, but probably we’re too small of a market to make this financially viable.

So what’s my conclusion? If you’re an SMB, give OCS2007 a try. It’s a very cool software, and the basic IM functionality isn’t that expensive.

Today i’ve encountered a very interesting problem that’s very hard to track down exactly.

A small business customer was running an Exchange 2003 server behind a ZyXEL ZyWALL 5 with AntiSpam installed and enabled. The ZyWALL forwarded port 25 to the Exchange server. This worked, for the most, flawlessly. But a few hosts (i’ve found no distinct differences between the source hosts – ADSL, Leased Lines, Colocated, Europe, USA) failed to get an SMTP greeting (220 customer.example.com Microsoft ESMTP MAIL Service, Version: 6.0.xx ready at Thu, xx Sep 2007 xx:xx:xx +0200).

When i disabled the Anti-Spam and pressed enter (in a telnet session to port 25), the SMTP greeting appeared. If anti-spam was enabled, it never appeared. But that didn’t help – Postfix still couldn’t send mails:

postfix/smtp[25010]: C65AA88075: conversation with customer.example.com[256.256.256.256] timed out while receiving the initial server greeting

I’ve looked at every setting on both the ZyWALL and the Exchange server, but didn’t find any unusual DNS etc. setting. I even disabled all the DNS lookups done on the Exchange server, but to no avail.

But after upgrading the ZyXEL ZyWALL 5′s firmware to the latest version (V4.02(XD.2)), the problem disappeared. While this wasn’t exactly what i was hoping for, at least the problem was now solved.

Source: MSExchangeSA
Event ID: 9188

Microsoft Exchange System Attendant failed to read the membership of group ‘cn=Exchange Domain Servers,cn=Users,dc=your domain’. Error code ’80072030′.

Please check whether the local computer is a member of the group. If it is not, stop all the Microsoft Exchange services, add the local computer into the group manually and restart all the services.

This error message sounds very serious, but it is entirely possible that the situation is not as grave as it sounds.

If you encountered this error message, you should first read all the hints over at EventID.Net. This shows the most common causes for this problem in bigger environments.

I’ve encountered this error message in smaller environments if specific factors were true: DC and Exchange were on seperate servers, and the Exchange wasn’t a DC (which is correct). However, there is only one DC on the network. In this case, the error message above might appear immediately after you restarted the first DC.

This seems to be a minor problem in Exchange which doesn’t fully recognize that the DC is back up. Restarting the Microsoft Exchange System Attendant service will solve the problem. This is not really a full solution to the problem, as you have to restart the System Attendant service everytime you restart the only domain controller.

The right way to fix this is to install a second domain controller. If you can’t do that, scripting the System Attendant restart makes sense – my approach would be srvany.exe and a plain cmd script. I didn’t write such a script yet, and i don’t really intend to do. Just buy a second DC.