Lukas Beeler's IT Blog

As discussed in the previous post, you can gather temperature data from RSA II or iLO cards using SNMP quite easily.

While the data itself can be good enough to make a decision, executives in a company always like nice diagrams. So my first try was to load the CSV-like datafile generated using said script into Excel, and make a diagram out of it. But Excel is restricted to 255 parameters per axis, which was severely limiting.

I’ve been using Cacti for quite some time, but wasn’t willing to implement it because we’re mostly a Windows shop, and my plan was to integrate the linux boxes into Operations Manager 2007. Cacti uses Tobi Oetiker’s rrdtool to create the graphs.

Creating graphs using rrdtool is quite easy, actually. I wrote a simple script that handled this:

makerrd

Creates the appropriate rrd file. Replace the unix timestamp as appropriate. The last value on the RRA lines is the number of values saved into the data file.

#!/bin/sh
rrdtool create test.rrd           \\
           --start 1176465000     \\
           -s 300                 \\
           DS:temp:GAUGE:600:U:U  \\
           RRA:AVERAGE:0:1:5000

inputrrd

Loads the data from the simple CSV-like file into the RRD file. The more elegant approach would be to load the data directly from SNMP into the rrd database, but i’m no programmer.

#!/bin/zsh
while IFS=';' read timestamp temp ; do
        temp=`echo $temp | sed 's/\\..*//;'`
        rrdtool update test.rrd ${timestamp}:${temp}
        if [ $? != 0 ] ; then
                rrdtool failed
        fi
done < machine

makegraph

Creates a graph from the data in the rrd file. The HRULE lines create lines for error margins. In this case 35C and 30C.

#!/bin/sh
rrdtool graph temp.png                       \\
        --start 1176465385 --end `date +%s`  \\
        DEF:mytemp=test.rrd:temp:AVERAGE     \\
        LINE2:mytemp#0000FF                  \\
        HRULE:35#FF0000                      \\
        HRULE:30#FFA500

See the created graph to the right. Of course, rrdtool has much more options and can create much nicer graphs.

Another SMB topic, as most enterprises are obviously capable of doing this by the book.

Summer seems to be starting, with the days here getting warmer and warmer. A particular problem that seems to crop up every summer is servers shutting down or failing due to excessive temperatures. The tolerances of these machines to temperatures is actually quite low, even with redundant fans installed.

Most Small Businesses actually don’t follow any kind of strategy when choosing a place for servers, and usually try to ignore the AC problem – this works quite well when the new systems are installed during cold times.

While it might be possible to operate a server room without AC, this only works in rather rare circumstances:

• No windows, or very small windows
• Room is only during direct sunlight for a short time of the day
• A very small number of machines installed in the room (one or two)

So, in general you will need an AC. But what are acceptable temperatures in a server room? The ideal would be 22C during the entire year. But it’s possible to run a server in a bit hotter environment. These specs usually depend on the server itself. Consider that there is other temperature sensitive equipment in the room – tape drives, UPSs, etc.

Start reviewing the spec sheets of your server to see what is acceptable. Here is an example for an IBM System x3650:

• Air temperature:
  • Server on:
    10° to 35°C (50.0° to 95.0°F); altitude: 0 to 914.4 m (3000 ft). Decrease system temperature by 0.75°C for every 1000-foot increase in altitude.
  • Server off:
    10° to 43°C (50.0° to 109.4°F); maximum altitude: 2133 m (7000 ft)

As you can see, the maximum temperature during operation is 35C. With outside temperatures reaching this level during summer, an AC is almost always necessary. A UPS like the Powerware 9125 is specified to work from 0C to 40 C. This is a bit more generous than the x3650, but it’s still easy to get up to 40C with several servers in a room.

In order to figure out if you need an UPS, the best way to figure this out now is monitoring your server. If you are using IBM Director or HP Insight Manager, these tools already have this functionality integrated. I personally don’t like these two products (and they’re usually overdesigned for less than 10 servers). If you have an iLO or RSA II card in your server, you can use SNMP to get the temperature, write it to a file, and get a graph from this later.

I wrote a quick and dirty script to this. It runs on linux, but the same would be easily implementable in PowerShell or VB.

#!/bin/sh
while true ; do
        echo -n "`date +%s`;" >> ~/tempmon/machine
        snmpget -Onqv -c public -v 1 \
                machine.rsa.int.dataline.ch \
                SNMPv2-SMI::enterprises.2.3.51.1.2.1.5.1.0 |
                sed 's/"//g;s/Centigrade//;s/ //g' >> ~/tempmon/machine
        sleep 5m
done

Ugly? Yes. But it works fine. You can later load this “CSV” into Excel, and create appropriate graphs from the data. And get management to buy the AC before your servers die a fiery death. If you want to monitor this long term, you could integrate the appropriate values into cacti quite easily.

A sidenote about ACs from my personal experience: The same points as for servers apply – you get what you pay for. Buying self install ACs from Fust, MediaMarkt or some other chain in that direction won’t do you much good. Get a decent, two component AC, and let it get installed by a professional. This also avoids building damage. Also, let a professional size the system, provide him with the maximal output of your servers (measured in BTU), and then double that value just to be sure. Network managed ACs are usually not available for Small Business-acceptable pricing,

The second last day of the Microsoft TechDays 2007 in Switzerland.

I’ve attended 4 speaks:

• Remote Access & Security
• Longhorn Virtualization
• System Center Configuration Manager 2007
• System Center Operations Manager 2007

All three talks were very interesting.

Remote Access & Security

A lot about RPC over HTTPS, two factor authentication, and TS over HTTPS. It’s funny that everyone starts tunneling stuff over HTTPS now, just wondering when people start to block this, or start implementing an ALG for HTTPS.

Longhorn Virtualization

This was particularly interesting, because this guy obviously knew the underlying architecture, and explained it pretty well.

What i found rather interesting here is that Microsoft repeats the same mistakes IBM made when they started with virtualization (which IBM calls Logical Partitioning, or LPAR for short). Windows Server Virtualization uses a primary partition – even though Longhorn mitigates the risk by preferring “Longhorn Core” as the primary partition.

IBM only uses a complete Hypervisor, with no primary Partition in it’s current System i5 lineup. But the previous, Power4 generation used a primary partition, with a few obvious drawbacks.

But there are technical reasons why Microsoft can’t do this. And it’s simple: Drivers. If you’re using Longhorn Core as the primary partition, you essentially get a very, very big hypervisor ;)

System Center Configuration Manager 2007

It was pretty interesting to see how big businesses handle their desktops. SCCM is recommended for Business with more than 500 desktops, so it’s way out of league for me. There’s an alternative product for smaller businesses, called System Center Essentials, but these geared down versions usually aren’t that funny.

System Center Operations Manager 2007

I’ve had big expectations for this speak, because i’m planning to use SCOM internally. However, most of the talk covered things which are also published as Webcasts. So it was a bit of a letdown since i’ve already seen this stuff, but the talk itself wasn’t bad.

Personal Sidenote:

After leaving TechDays, and getting myself a very nice bag, i’ve lost my wallet in the parking garage. When i returned a few minutes later, it wasn’t there anymore. I’ve spent two hours cancelling all my cards, badges, etc. I even went to the police station to file a report (but that was declined, i had to wait three days). Just a few minutes ago i got a call that my wallet was found. I don’t know if i should be happy about that or not, because i’ve already wasted lots of bucks on new cards.

I’ve spent the entire day at the Kongresshaus Zürich, attending Microsoft TechDays 2007.

In general, i enjoyed this first part of the event. As always, the Kongresshaus has severe logistical problems, they don’t have enough parking space. Microsoft offered free parking at Albisgüetli, but that is about 15 minutes away.

So instead i choose to go early, and parked right next to the Kongresshaus – arrived at 08:15, and there was still plenty of parking space available. At 27.- CHF for the entire day, it didn’t come cheap, though.

Microsoft seems to have moved back from stickers to lanyards, and there were croissons available in the morning.

The Keynote was very good, held by a guy named Rafal Lukawiecki. A review on security in the past, and in the future. Very interesting, and very well done.

The next on the list was about Windows Sharepoint Services, but aimed at developers (which i didn’t really pay attention too, when choosing). Even though i’m not a developer, it was very interesting to see the architecture behind WSS.

I then attended a talk about Exchange HA & Disaster Recovery – i enjoyed this one too, but i think the title was a bit misleading, it concentrated on backups and reasons for disasters.

It was interesting to hear that he saw incorrectly configured anti virus software as the main reason for exchange full restore – i couldn’t really believe this, because it didn’t look like this guy worked with Small Businesses. He even wrote an Article about this topic on msexchange.org. While listening, i enjoyed that it seems i did everything right in the setups i did in the past, which was very reassuring.

Next was something about Windows Storage Server, or file serving from windows in general. It was very well, with a very competent speaker. It was also interesting to see that Microsoft is entering the Backup Market with DPM – at competitive prices, even though DPM requires a seperate server, which is a bit of a letdown. DFS/DFS-R was also covered very well, though there wasn’t anything there new to me.

What i wondered was that it seems that even though most people attending TechDays aren’t exacly working in SMBs didn’t know much about DFS. I assumed that most bigger companies were already using this.

At that point, i called it a day and went home. Stay tuned for the next part.

McAfee’s ProtectionPilot works well, at least most of the time.

During my regular checkup, i noticed that the virus definitions haven’t been updated for a few hours. When viewing the the scheduled tasks pane, all the scheduled times showed up as “Unknown”.

I tried a manual update, which worked fine.

I scanned the eventlog, and saw that the McAfee Framework service has crashed. After starting it, the updates worked automatically, as they should.

The Framework service didn’t have automatic restart & recovery enabled by default. I enabled it, but i don’t think that this will happen again – it has been working flawless for almost two years.

Sizing memory is easy – but i’ve seen many people run into trouble with it and buying more ram than they can run, and lot’s of other troubles.

There are several limits to the amount of memory you can use, because of several factors. These factors are:

• Hardware
• Operating System
• Application

When buying hardware from a distributor (and not preconfigured systems directly from the manufacturer like enterprises), you usually get a base memory of 1GB, in the form of 2x 512MB.

For smaller 1U machines with only 4 memory slots, the most economic configuration is thus 3GB (with an additional 2x1GB feature set) of memory, which as we later see is supported by all OS/Application combinations.

For bigger machines, with 8 or 12 memory slots, you can get a lot more RAM. 2x1GB is still somewhat cheaper than 2x2GB. At this point, OS and applications become a factor.

Microsoft has set the following limits for it’s Operating Systems:

• Windows Server 2003 for Small Business: 4 GB
• Windows Server 2003 Standard Edition: 4 GB
• Windows Server 2003 Enterprise Edition: 64 GB
• Windows Server 2003 Standard Edition x64: 32 GB
• Windows Server 2003 Enterprise Edition x64: 1 TB

Source: KB889654

These are the constraints by Microsoft. However, there are additional constraints on the maximum amount of memory, inflicted by the architecture itself.

Even for full 4 GB on a single server, you might need to enable PAE by setting the /PAE flag in the boot.ini file – this is necessary because there are virtual address spaces used by the PCI-Bus and similar equipment in your server. Note that /PAE is supported on SBS and Standard Edition, though you can’t use more than 4 GB of physical RAM.

Accessing more than 4GB of memory on 32bit Platforms requires certain tricks, especially if you want to access more than 4GB of RAM in a single process.

This leads us to the next set of problem – application support.

For example, a single exchange server running with 64GB of memory will not really make use of it – sure, the excess memory can be used as a disk cache, but store.exe won’t be able to use more than 4GB of memory.

There are certain application which supports AWE, most notable Microsoft SQL Server. AWE allows a single process to access more than 4GB of memory, using even more tricks. These usually slow down performance a lot.

So, having more than 4GB of RAM only makes sense if the application you are running consists of multiple, independent processes. There are other considerations such as kernel memory, which mostly come into play with a terminal server environment (which i have no experience with).

For Small Business Server, 4GB is the maximum, and in my opinion also the minimum. A SBS Server with only 1 GB of memory will be very, very slow and swapping constantly. With 2GB, it will probably work fine. With 4GB, you have the maximum amount of memory supported, and the server will probably need it, giving you an extra speed boost. The most economic way for this is usually 4x512MB, 2x 1GB.

If you think you need more than 4GB of memory, enterprise edition is very, very expensive and can lead to other performance problems when using AWE. So, if you think you need more than 4GB, go for 64bit. Exchange 2007 even requires 64bit, in contrast, Exchange 2003 doesn’t even support 64bit.

Remember that 32bit Standard Edition supports 4GB, but the 64bit Standard Edition supports 32GB.

In closing, it’s not that difficult if you can wrap your head around all these limitations. Here are my general sizing rules:

• Windows Server 2003 for Small Business – get 4GB, you will need them
• Windows Server 2003 Standard – get 3GB if that’s enough, or 4GB if you need the extra Gigabyte
• Windows Server 2003 Standard x64 – Decide how much you need according to application. 8GB is a good starting point
• Windows Server 2003 Enterprise – Don’t consider them for a Small Business, too expensive and too much hassles

Hope this braindump help someone. I won’t be writing on Friday and Monday, because i don’t have to work then.

A few days ago, i got hands on my first IBM System x3250. The x3250 isn’t a middle class server like the x3650, it’s IBMs low end rack server. You will see the difference on the pictures – there’s also a large pricing difference. This machine was to serve as a router/firewall/vpn concentrator, and thus doesn’t have any demands toward hardware. The OS installed was Debian GNU/Linux 3.1, which has it’s own set of problems.

The x3250 doesn’t have Light Path diagnostics, hot pluggable fans, or even hot pluggable hard disks. You can order them with 2.5″ HP SAS disks though, but that makes it a lot more expensive (to the point where an x3550 might be the better choice).

Here’s the configuration ordered:

• System x3250 Xeon 1.83Ghz DC, with 2×512MB Base Memory, 3.5″ Simple Swap SATA
• 2x 80GB SS-SATA

Unpacking and opening

This machine came packaged nicely into a big box, secured on a wooden pallet. It contained the usual low cost rack mount kit, without the facility the remove the machine halfway out of the rack, there was no cable tray, and no rails. It’s hard to see on the picture, because of the missing frame of reference, but the x3250 is very, very short. It would probably fit in a Telco Rack.

The disk blanks fit nicely (they have to – you don’t remove them, even if there are disks installed). Interestingly this machine still has PS/2 inputs, and the case is the same as the one of it’s predecessor, the xSeries 306m. Of course now with Intel Xeon DC. Even though a baseline model, you still have the ability to install an IBM RSA II card for remote maintenance. It also has a dedicated slot for installing a SAS/SATA raid controller, allowing you to do real hardware RAID without loosing a precious PCI-E slot.

Interiors

As you can see on the pictures, it’s clearly visible that this machine belongs to another price class than the x3650. While all cables are nicely tied together, and nothing is flying around, it’s still different from a middle class machine. The fans aren’t hot pluggable, neither are the disks. You can only install 4 DIMMs in total.

There’s an interesting heat pipe attached to the CPU, which i haven’t seen before – not even the x3650 has a heat pipe. Documentation however, is still top notch. The included documentation on the inside of the upper lid is is very detailed, and contains all the information you probably need.

Installing options

The cheapest x3250 has so called Simple Swap SATA disks. You can install and replace them while the server is mounted into the rack, but they aren’t hot pluggable. You don’t require any tools for this tasks, too. I think this was solved much better than HPs approach in their baseline machines – they use screws, and you will need to remove the machine from the rack.

Installing the SS-SATA disks is easy – just remove the filler pannel, and insert the disk till it clicks. Then place the filler panel pack into the server. Removing the disks is a breeze too, just pull on the blue latches attached to the disk.

Booting the server

The baseline x3250 doesn’t have a hardware RAID controller, just a standard Intel AHCI SATA controller, which is well supported on Linux. And by Linux, i mean “not Debian”. The current stable release of Debian doesn’t support AHCI SATA. This isn’t such a big problem, because you can install the OS using IDE emulation, build or install a newer kernel, and then switch the system to AHCI SATA mode.

However, this proved to be much more of a problem than i initially thought. Linux was able to recognize the disks, but after configuring the software RAID, the machine become really, really slow. Like 386 16Mhz slow. The RAID was rebuilding in the background, with about 2Mbytes per Minute. While this installing was very, very slowly skipping ahead, i built a proper kernel on another machine.

After the install finished, i quickly installed the new kernel, booting the machine in AHCI mode – thanks to Linux SW RAID autodection, there was no need to reconfigure anything. The RAID finished rebuilding with 50Mbyte/s, which i found much more acceptable – no slowdowns either.

Resumee

The x3250 is a cheap baseline model, and it’s visible. But i still think it trumps the alternative models from HP and Dell, while being similar in pricing.

Also, the obligatory plug to DATALINE AG which sells this server and other IBM System x or System i servers.

A few weeks ago, i deployed my first IBM x3650. In general, i think the x3650 is a very nicely made machine. I’m not going to talk about features which are pretty standard in all middle class IBM machines, like light path diagnostics.
Here’s the configuration i ordered:

• System x3650 Xeon 2Ghz DC, with 2x512MB Base Memory, 2.5″ SAS Open Bay
• 2GB additional Memory
• RSA II slimline
• 4x 73GB 2.5″ SAS Disks
• Redundant PSU Kit
• PCI-X Riser Card
• PCI-X U320 SCSI Adapter

Unpacking and opening

The server arrived on a standard wooden pallet, secured in nicely in the box, with styropor for dampening around it. The box also contained an UPS powercable, a detailed manual for installing options, and of course the rack mount kit with cable tray.

After removing the server from the packaging, which was rather easy to do, i inspected the machine for damage or faults. There were none to be found. One of the interesting think was that the 2.5″ SAS Disk blanks don’t hold very well in the system. It didn’t look good, but it’s not a real problem.

One of the first things you will notice is that this server no longer has PS/2 ports. This might be a problem if you have an older KVM switch in operation.

Installing options

After opening the server, everything had the usual IBM color codes for hot-plug components and touch points. There is a detailed diagram of the server on the inside of the upper lid. Near the CPU/RAM was a table that shows valid memory configurations, which is very nice when you have to upgrade memory.

It’s very nice to see that IBM keeps the in-server documentation on a very high level. For almost every part, there is a short but very concise description on how to maintain, configure or remove it. This is especially helpful if you experience a problem, and have to swap out a component.

Installing memory into the system was a breeze. Went without any problems. As you can see, the x3650 can take up to 12 sticks of memory, also allowing for highly redundant memory with features like Memory Mirroring and Memory Spares. I’m not going to waste my memory on such a feature, though.

I also had to replace the internal PCI-E riser card with a PCI-X riser card, because IBM doesn’t have any PCI-E SCSI adapters. This wasn’t a problem to do either.

Together with the redundant power supply, there also comes a supply of redundant fans. The system ships with 5 hot plug fans installed, and the redundant power supply kit comes with 5 fans extra, for a total of 10. They aren’t even that loud, since the usually idle at around 40%. The fans have a very slick spring mechanism, which makes them bump up in their casing, to allow for easy removal.

Installing the RSA card was a bit tricky. It didn’t click into the board as easily as i thought. I had to apply way more force than i had to on previous servers. It went in fine though, and is now working without a hitch.

Booting the server

The hassles begin when booting the server for the first time. Luckily IBM has redesigned it’s server support pages, allowing quick and easy access to all firmware updates at once. As usual, you need to flash a custom firmware to the RSA card, to allow proper support. This time around, i also had to update all the SAS Disks firmware, which was quite a hassle because this didn’t play well with the RSA II CD drive, i had to use the local one.

Installing Windows Server 2003 R2 was uninteresting, went fine without any troubles.

Resumee

Another good x86 from IBM server i can only recommend. If you’re interested in more server reviews like this one, please leave a comment. Currently, the only other machine i could write about is the IBM System x 3250.

Oh, and if you’re interested in buying an IBM System x 3650, give my current employer DATALINE AG a call – we’re an IBM Premier Business Partner.

Security has always been a major point in the past few years in IT. I’ve seen several SMBs without a DHCP-Server, because this supposedly helps security.

Of course, just because you don’t have a DHCP server doesn’t mean that nobody can access your network. There’s a much better technology available that helps to prevent unauthorized devices from accessing your network. It’s called 802.1x.

In case you’re wondering, 802.1x can also be used with Wireless LAN. I’ve written a little HOWTO about Wireless 802.1x with Windows Server 2003 and Cisco APs.

Most access points support 802.1x, but with switches the functionality is a bit more scarce, especially if you’re looking at SMB equipment. But there are several vendors that offer 802.1x functionality in their switches, for affordable prices.

But what is 802.1x? It’s a technology that allows authentication at the link level to take place – it requires a RADIUS server as a backend (such as Microsoft IAS, which allows you to authenticate against active directory). The switch or AP just works as an intermediary between the RADIUS server and the client – this makes the switch/AP agnostic to the authentication method used.

802.1x can be great to enhance your network security. It prevents people from installing unauthenticated devices into your network. Of course, you will need to either whitelist older devices without 802.1x support by MAC-Address, or upgrade to newer devices with 802.1x support. For example, all newer Lexmark/IBM printers fully support 802.1x out of the box.

Especially with the dawn of Windows Vista, which can configure 802.1x LAN authentication through GPO (XP could only configure Wireless LAN 802.1x through GPO), it makes sense to start implementing 802.1x.

Allied Telesis

Even new Allied Telesis (formerly Allied Telesyn) switches look like they’re at least 10 years old. But they offer all features usually needed at nice pricing points. Their software is not always consistent between different product lines, but this shouldn’t be a problem for SMBs. For example the AT 9000/24 offers 802.1x support and 24 Gigabit Ethernet ports. Retail price in switzerland is about 1000 CHF.

Linksys

Linksys, a brand of Cisco, also offers business switches with 802.1x support. The SRW2024 can be had for about 600CHF, featuring 24 10/100mbit Ports and 802.1x support.