Lukas Beeler's IT Blog

After migrating several users into our Exchange server, which were first defined as an external contacts (for proper forwarding), everything worked fine.

However, one user was unable to send mail to the new users. It generated a bounce/NDA, and failed to deliver. I first checked the Exchange Server, and everything seemed to be perfect. I then walked this user through the procedure of selecting the user from the address book, and then sending the mail. This worked.

At first, i was baffled, but then i remembered about the external contacts – the external contacts where defined as X.400 addresses, which coexisted with the new users for a short while. so the external contacts had different X.400 addresses than the new users. (I did handle the SMTP Addresses correctly, but not the X.400 ones).

But the external contacts where deleted, and no longer in the GAL or the OAB. But Outlooks address cache for type-ahead finding of contacts is independent of these two address sources. And that’s what the user used. You can delete entries in this cache by selecting them with the cursor keys, and then pressing the delete key. Seems logical, but i haven’t seen this mentioned anywhere explicitly.

Every major server manufacturer offers some sort of service processor for their server. I’ve got experience with IBM’s RSA II cards, and with HP’s iLO. Other server manufacturers like Dell have similar solutions, but i’m not familiar with them.

These things aren’t usually that costly (around 500CHF), and allow you to power on/off the server from remote – this comes quite handy if the server is at a remote site. You can also access the local keyboard/mouse from remote, this usually requires USB support in the OS (not a problem anymore, i think). Both iLO and RSA II support remote KVM, remote floppy (for upgrading the BIOS or similar tasks). They also both support secure remote access through SSH and HTTPS.

Another very important thing is access to the system error log, usually only visible through the BIOS or the monster management Applications (Insight and Director).

There are some key differences between iLO and RSA II:

IBM’s RSA II

IBM’s RSA II slimline is an add in card, usually located directly on the motherboard and connected through a proprietary connector. It allows remote alerts through SMTP, and comes with an outdated-looking web interface, but it offers all the usual features like remote KVM, system log display, remote storage, integration into LDAP directories, etc. The ethernet connector is usually integrated into the main system board, but only active with an appropriate RSA card installed. There’s also a non-slimline version of the RSA II card, which offers a bit more functionality, but i’ve never used one of them. RSA II slimline is a bit cheaper than the iLO advanced license.

HP’s iLO

HP’s iLO, integrated into all better HP servers doesn’t cost a penny in the standard version, and is sold with the server. No need to install hardware. You can activate features like remote KVM, and LDAP directory integration using license keys. This approach is in my opinion a lot better than IBMs, because you do not need to purchase and install additional hardware. What iLO can’t do is to make notifications by mail. Of course, SNMP is supported, but smaller businesses might lack the infrastructure for SNMP traps.

As you can see, both products have their own advantages and disadvantages. I think IBM should polish the look of it’s RSA Webinterface a bit, and HP should add alert support through E-Mails. Both products lack time sync through SNTP, for some reason. Maybe they use the systems internal clock, but i wasn’t able to find much about this topic (i didn’t look very far, either).

Both HP and IBM make excellent servers – IBM seems to be a bit slower in technology adaption than HP, though they are leading on other fields like Blade servers.

I recently ran into an interesting problem related to Active Sync and Cached Exchange Mode in Outlook 2003.

In this case, the user running Outlook 2003 was connected to the exchange server through a WAN link, and has just imported a rather large .pst file, which was being uploaded to the Exchange server.

At that point, i tried to configure Active Sync on the PDA (through a GPRS connection), but it just hung while retrieving the folder list. An Event from Source “Server ActiveSync” and ID 3005 was logged on the Exchange-Server. At first, i thought the device (running PocketPC 3.0) had issues with the Exchange server, but that wasn’t the case. When i quit Outlook (and it stopped uploading from the local store to the server), everything was fine.

I waited until Outlook finished uploading, and from that point on Active Sync didn’t have any issues.

It seems that Active Sync doesn’t work when the Exchange Server is receiving files from a client.

You can easily copy configurations from one IAS server to another, Microsoft documents this here.

Basically, you can use netsh aaaa dump > file.txt to dump the config to a file, and import it using netsh file < file.txt to import it.

But here comes the interesting part – after importing the configuration, my WLAN client failed to connect. But why? Simple, you will need to enter the PEAP-Dialog, and choose a certificate. After that, everything will work fine.

No, it didn’t.

But by default, DFS replication excludes certain file patterns. Some of these patterns (notably *.bak files) are used by the office 2003 installation source. You will need to edit the filter of the replication group, in order to enable replication of similar files. The replication group i’m using for my software packages doesn’t have any filters, and is only replicated at night.

Please remember that changing the filter, and the replication of the “missing” files may take up to an hour, due to lookup and replication delays by all replication group members.

I’ve run into a nasty little problem. The initial situation was as follows:

Windows Server 2003 R2 SP2, Single Domain Controller, which was also a CA. This worked fine, users could get their cert, the machine was able to register it’s own DC certs, and all machines were able to get their certs.

However, the second domain controller (also Windows Server 2003 R2 SP2) added to this domain was unable to register a certificate. I got the error message on the right, from source AutoEnrollment with Eventid 13, telling me “Permission denied”, with error code 0×80070005 – it was unable to register it’s own domain controller certificate.

This baffled me, but after searching around on the internet, i found out that there were two issues i was facing.

Members of the CERTSVC_DCOM_ACCESS group

The CERTSVC_DCOM_ACCESS group controls access to the certification service. In my case, the “Domain Controllers” group wasn’t a member of the CERTSVC_DCOM_ACCESS group. After adding it, the domain controller was able to talk to the CA service, but still wasn’t able to fetch a certificate.

According to some postings on the internet, this issue arises if you use a AD migrated over from a Windows 2000 functional level with a Windows 2000 CA, which was the case here, but i wasn’t able to find definitive answers.

Permissions on the MachineKeys directory

In %ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\RSA\MachineKeys, you will find a directory which has to have special permissions. There’s a Microsoft KB entry Default permissions for the MachineKeys Folder which describe how to fix this issue.

No idea why these permissions were wrong, this was a fresh install with SP2.

After fixing the permissions according to this documentation, everything was up and running again.

I’ve encountered an interesting problem, where i wasn’t really able to track down the source (but found a solution nonetheless.

It was a new Vista Business machine, running Office 2007, freshly set up by me about a week ago. The problem was that Adobe Flash didn’t work. Or, didn’t work completely. Sites like youtube.com didn’t work, but the flash install site showed the animated bar.

When looking at the add ons settings in IE7, you could see that it was successfully installed. I ran the Flash uninstall Tool, which didn’t work quite right – i had to launch it with administrative privileges manually in order for it to succeed. After that, i revisited the flash install site, the plugin was installed again, and everything worked fine.

And here comes the really interesting thing – the person that was using this machine swore that he never saw the ActiveX installation dialog. I looked at the installed software, and Adobe Reader 8 was part of it. Maybe Adobe Reader installed a slightly broken version of the flash plugin? I didn’t bother to investigate this more closely since the problem was already solved, but this might cause some more trouble down the road.

Electronic Service Agent supports VPN connections with NAT pass through since i5/OS V5R4 (there was a “beta”-version of this in later i5/OS V5R3 CUM-Packages, earlier versions, like OS/400 V5R2 didn’t support VPN connections for ESA). Ensuring that your System i (or iSeries, or AS/400) can communicate with IBM is necessary if you have a HW-Maintenance or SW-Maintenance contract.

Now, if you want to allow this traffic through a firewall, you will see that IBM doesn’t exactly document this too much.

Generally, you will need to allow these things from your System i:

• Outgoing ICMP echo request (PING)
• Outgoing UDP traffic to Ports 500 and 4500
• Outgoing ESP packets (Protocol 50)
• Accept incoming packets from established connections

If that’s to broad for you, you can restrict the above traffic to these two IP adresses:

• Boulder: 207.25.252.196
• Rochester: 129.42.160.16

Note that these IP Adresses may change in the future.

I’ve been asked many times why some applications are slow over a VPN connection.

In order to understand why this is the case, we must first understand what type of slow is normal, and what isn’t.

The most popular problem with VPN connections are MTU issues. Try a ping -l 1500 internal-ip. If that doesn’t work, you have an MTU issue.

The next step is to test the raw speed – copying a file with FTP or SCP to and from a server. Is the speed okay? If the speed is according to the lines used, your problem probably isn’t with the VPN connection.

The main problem with VPN connections is latency. The average latency for a roundtrip from ADSL to ADSL here in Switzerland is about 80ms. This is a lot of time, especially if you’re taking multiple round trips. If we look at other connectivity methods like UMTS (with about 500ms Latency), it gets a lot worse. 10 Roundtrips für ADSL are a second, but 10 Roundtrips for UMTS are 5 seconds.

You can use Wireshark to look at the traffic on the interface in order to see if the application that is slow uses multiple round trips to achieve it’s goal.

There are several well known applications that have problem with using multiple roundtrips:

The most popular one is SMB – it uses 5 – 10 Roundtrips for a directory list under certain circumstances. This means 0.5 – 1 second wasted on network latency (and this doesn’t even include the transmission time). The solution for this problem varies – if you need access from a site, use DFS-R and a server local to that site, if you want road warriors to have faster access, use Sharepoint.

Another thing causing many problems are ERP applications built onto the “Fat Client” design principle (Business Logic in Client), which queries the database directly – this ensures many, many roundtrips for every bit of information displayed. (Note that DIAS-iS doesn’t have this problem because it’s a Thin Client application, with the Business Logic in the server). Another remedy is an ERP application which can have servers in multiple sites, but this is probably not SMB software anymore.

There’s not really a solution for this, because it means switching ERP applications, or getting your software provider to start supporting high latency links.. A bit drastic for latency problems.

An interesting application regarding this is Outlook – Outlook is a hybrid. With Cached Exchange mode enabled, it behaves like a Thin Client, because it has a local replica of the database. So, the solution for Outlook is to always use Cached Exchange mode – a good idea anyway because it reduces load on the Exchange server.

When Windows Server 2003 R2 came out, many people thought there was nothing spectacularly new in it. But that’s not true.

While many of the new R2 features were meant for enterprise customers, there’s an important feature that can be used by anyone, as long as you’re not using the SBS version of Windows Server.

DFS consists of two seperate technologies, that can be used independently – the first one can be used no matter how small your business is:

DFS Namespaces

DFS Namespaces make it easy to provide a fault tolerant, single namespace for all your shares. With the time, clients probably have more and more shares mapped to network drives, which is both a hard to manage, and difficult to understand for the end users.

With DFS, you can show all your shares, no matter if they map to a NAS device, a Windows Box, an i5/OS instance running Netserver in a single, unified tree.

Note that this feature does not require anything except a Server running R2 and Active Directory. No additional licenses, no multiple domain controllers, etc. needed.

If you look at the picture to the right, you can see that that we have a unified structure, linking multiple shares into a single tree. You do not need to use replication or anything else.

Please note that you can still map the DFS root (here: \\int.dataline.ch\Public\) to a drive letter, in case you prefer that. The official stance is that you should use the UNC path everywhere, though.

Using a domain hosted DFS root has another big advantage – if you move servers, the links on your users won’t change. For example, if you offload all the Multimedia content from the server, and you move it to an Active Directory integrated NAS Box (like Snap Server), you just have to change the link in the DFS root – the users won’t even notice that it has moved to another server.

A domain hosted DFS root, hosted on multiple domain controllers gives you a fault tolerant redirection structure. Together with DFS-Replication, this gives you application redundancy (which is cool).

As you can see, there are several advantages with DFS Namespaces alone, that might make it worth the time it takes to implement DFS, no matter how small your Business is. Even if you have only one server, DFS can help you get a simpler directory structure for your users, and it can also allow you to make managment and moving of data easier. NAS boxes are becoming more and more popular in SMBs, because they offer very cheap storage.

DFS Replication

DFS Replication is a true multi master file replication implementation. DFS-R can be used over WAN and over LAN links – while the LAN variant is usually used to provide fault tolerance (which isn’t always implemented in a SMB), the WAN variant comes into play as soon as you have branch offices.

The DFS-R FAQ answers many questions related to DFS replication.

DFS over WAN links uses a technology called RDC – if you are familiar with Linux, you will soon notice that RDC is very similar to RSYNC, except that it’s a whole lot better. Why?

Cross-file RDC allows DFS replication to use RDC even when a file with the same name does not exist at the client end (provided either the client or server is running Windows Server 2003 R2, Enterprise Edition). Cross-file RDC uses a heuristic to determine files that are similar to the file that needs to be replicated, and uses blocks of the similar files that are identical to the replicating file to minimize the amount of data transferred over the WAN. Cross-file RDC can use blocks of up to five similar files in this process.

This is a very, very cool feature, that i’ve never seen anywhere else before.

You probably won’t need DFS-R if you only have a single office, and are not using application redundancy. But if you have branch offices, you can use DFS-R to have all the data in all your offices available. This makes it much more convenient to work, because the data will always be available from a local fileserver. Please note that you can’t really use direct SMB over high latency WAN links – SMB makes too many roundtrips, which will make everything feel very slow.

Conclusions

DFS Namespaces and DFS Replication can be used to provide HA fileshares – to provide a unified naming structure – to replicate content to and from branch offices while saving bandwidth. This is a whole lot of things that you get when you buy Windows Server 2003 R2.

Sadly, not many SMB admins know about DFS, or use DFS. But everyone should. There’s no reason not to use DFS.

Microsoft offers a nice Demonstration of the DFS technology – something you can even show your boss.