Up until now, we’ve been using PPTP for our VPN needs. PPTP is easy and painless to setup, but can cause several problems on customers site because it needs GRE. Many overzealous firewalls block GRE.

In the future, we are intending to use SonicWALLs Global VPN Client, that uses IPsec with it’s NAT-Traversal over UDP. Also, the SonicWALL GVC solution is able to plug directly into Active Directory for central authentication.

I intended to keep PPTP running for some time after the migration, in order to ease the transition. But as it looks now, Cablecom blocks OUTBOUND GRE packets. Mighty strange, because inbound GRE-Packets work.

Here’s how this looks in tcpdump:

10:58:13.927888 IP 77.59.216.227 > 194.88.212.200: off 0×5858 [|gre]
10:58:13.947131 IP 77.59.216.225 > 77.59.216.227: icmp 52: host 194.88.212.200 unreachable

.225 is the Cablecom CPE, and .227 is the Linux machine running the PPTP server.

I’ve already opened a support case with Cablecom, in the hope of having this issue sorted out quickly. So far, i haven’t heard back from them, even though i reported the issue almost a day ago. It’s not like we pay 180 CHF a month for 24/7 support.

Update: Cablecom was able to resolve the issue today. Apparently, it was a config issue on the router.

I’ve seen the “big” ESX in a few places and worked a bit with it, but i decided to refresh my knowledge on VMware a bit. For this, i first had to scrounge up a machine that was able to pass the rigorous HCL from VMware.

Unfortunately i didn’t find something that was really a Small Business machine - i used a HS21 Blade from my BladeCenter S testing environment.

The HS21 blade has 4GB RAM, a 2.66 Ghz QuadCore CPU and two 500GB SATA Harddisks attached to an LSI1064 SAS Controller. Fortunately, this configuration is supported.

Installing ESXi

Similar to the installation of Windows Server 2008 or Windows Vista, the ESXi installation is extremely streamlined. All you have to do is pop the CD in, select the disk where you want to install ESXi and then let it continue. The whole setup took around 15 minutes, most of the delay owed to the extremely slow Laptop CD Drive installed in the BladeCenter S.

After installation, the Blade rebooted and you will be greeted by an extremely simplistic interface that allows you to change basics like the password of ESXi and reconfigure the management network interface and also display a few logfiles. On first startup, it also showed my a Web address where i can download the VI Client that is used to manage ESXi.

A very pleasant experience.

Installing the VI Client

After accessing the ESXi host through HTTP, i could then download the VI Client. Installation on another Blade running WS2008 was smooth. It also installed an Update Service that allows me to upgrade ESXi.

Configuring ESXi for the first time

After logging on using the VI Client to ESXi, i was greeted with a nicely detailed instructions that i would need to create a datastore. After few clicks i had a datastore created on the RAID1 that ESXi was installed.

The VI Client looks very impressive and neat. It looks like ESXi can read diagnostic information from the Blade, and can monitor RAID, Fan and other stati easily. One of the things i really like about this is that you get a standardized interface for monitoring your hardware - on Windows you usually have to use tools like IBM Director that are just one big mess to handle. Here, i didn’t have to configure anything - it just worked.

After entering licensing information, configurating a static IP Address, changing hostname and DNS information, i rebooted the blade.

Creating the first Virtual Machine

I decided to create a first virtual machine - the blade i killed for running ESXi was previously running Exchange 2007. As this is just a demonstration setup, i decided to recover the preexisting Exchange server into a VM, in order continue having a full featured demo setup.

So i created a new Virtual Machine, configured for running Windows Server 2008 x64. Now, i didn’t have WDS setup in the Demo Environment, so i had to find a way to boot the Blade from an ISO. Previously i used scp to copy the ISO to the ESX Management Partition, but that didn’t work on ESXi. Luckily, the VI Client has a “Datastore Browser” that allowed me to upload files to the vmfs3 filesystem.

After uploading the ISO, i booted from it. The installation was pretty slow, but comparisons to my Hyper-V hosts aren’t fair as those run 10kRPM 147GB SAS Disks in a RAID5 configuration instead of the slow-as-molasses 500GB 7.2kRPM SATA Disks.

After OS installation, i immediately installed the VMware tools. One reboot later, i had a working Windows Server 2008 machine.

One of the things i noticed: When running WS08 virtualized on Hyper-V with 4 virtual CPUs on a Quadcore machine, WS08 thinks i have on Quadcore. On VMware, WS08 thinks i have 4 real CPUs (Sockets). This can bite you if you want to give a WS08 Std Machine more than 4 Cores - as WS08 Std is only licensed to four sockets.

The next step obviously is restoring the Exchange server, but that doesn’t really have to do all that much with ESXi.

Conclusion

ESXi is great. One of the biggest advantages over Hyper-V is the VI Client that consolidates a lot of information that is all strewn about in Windows. For example, it has built-in performance metrics, raid status monitors, etc. You can get all the same information with a machine running Hyper-V, but you’ll have to use other tools for that (of course you can customize a MMC do include Perfmon, but it’s not exactly the same).

VMware shows that they have gained long term experience with Virtual Machines, and the VI Client clearly shows the maturity of their product.

Permission management seems much better than Hyper-V, but i didn’t find a way to use Active Directory integration. Maybe Virtual Center is required to this, or i just wasn’t able to find it in ESXi - it exists, because there are numerous references on the Web.

I’ll certainly consider using Hyper-V when i have to run non-Windows guests. For Windows guests, Hyper-V with it’s VMbus architecture seems better suited. For non-Windows guests, VMware can’t be beaten right now.

Virtualization has always been a topic with a lot of hype, but as of today we have a single customer that is using it (out of 150).

Why? Because virtualization is still expensive. For larger companies, it was possible to save money by using virtualization, for smaller companies that wasn’t really the case. You’ll still need to license the guest OS. You’ll still need to maintain it.

Most customers decided to just buy a Windows Small Business Server, and run all apps from that machine. Though that usually required a technician that knew what he was doing to get all the apps running together on a single machine, it saved money in licensing cost and hardware - and the most important application ran on a separate machine anyway (our ERP software on the IBM i).

With the release of Hyper-V and it’s inclusion in SBS 2008 Premium (on the second machine), Virtualization will probably get picked up even in small businesses. But is it the right way?

I’ve started gearing up my knowledge on virtualization as it will become a topic for our customers. For that, the most important other factor is VMware. VMware offers virtualization products for longer than Microsoft, and i’ve been using their Workstation product for a long time.

Microsofts Desktop product Virtual PC is lackluster at best. The performance is awful and it doesn’t offer many features. There was also Virtual Server 2005, which we’ve used internally since mid-2005 (when you still had to purchase GSX Server and we got VS2005 through the MSPP for free).

Now VMware has an offering that is free, Microsoft has an offering that is included into most Windows Server licenses and Citrix offers a very limited edition of their product for free (Max 4GB RAM, Max 4 VM).

And the big question would be - what product should a small business use today, and why?

I’ve found a few good blog posts on ESXi:

What’s the difference between free ESXi and licensed ESXi?

And on Xen:

Citrix XenServer for the ESX Engineer

And on Hyper-V:

Hyper-V for the ESX Engineer
More on Hyper-V for the ESX Engineer

On ESXi

ESXi Installable Edition Free (short: ESXi) only runs on certain certified systems. Of course you can still build a whitebox machine that runs ESXi, but that would be a rather stupid decision. Running supported hardware is important even in a small business.

ESXi doesn’t support many systems, especially our bestseller system, the IBM x3650 is not supported with ESXi installable edition. I expect the list of machines supported by ESXi to grow steadily, though.

On the other hand, ESXi supports a wide variety of guest operating systems that are supported by VMware. This is one of the main advantages VMware has over Hyper-V. However, most Small Businesses struggle with the complexity of using one operating system. They are unlikely to use multiple ones. On the other hand, VMware offers preconfigured appliances, which sounds like a good use. Important to know: Microsoft does not directly support running Windows on VMware unless you pay big for a Premium support contract.

ESXi can be managed by the VMware “VI Client”. This allows you to do all the everyday tasks of configuring and setting up virtual machines.

ESXi doesn’t have any restrictions that would prohibit production usage, but the management features are a bit limited - you can’t monitor it using SNMP, you can’t script it using the RCLI. If you want those features, you’ll have to pay.

VirtualCenter, which is VMware’s variant of System Center Virtual Machine Manager, is quite expensive. Of course, SCVMM is also quite expensive. So i doubt that either will be used in a Small Business. The disadvantage i see here over Hyper-V is the fact that it can’t be scripted or automated. While not a showstopper, it’s important to consider this.

On XEN

The free XEN version supports a maximum of 4 VMs and 4GB of RAM. With that, i think everything is said and done. These restrictions do not allow production usage. It’s more like a demo version for the full products.

On Hyper-V

Hyper-V only works on 64bit installations of Windows Server 2008 Standard, Enterprise or Datacenter. In SBS 2008 Premium, one license for Windows Server 2008 Standard is included. This allows small businesses to get started with Hyper-V. WS2008 Standard x64 supports up to 32GB RAM. If you use “just” Hyper-V on a WS2008 Standard installation, you can also install a single guest VM with WS2008 Standard without having to purchase an additional license. Be aware that it does not work this way if you run any other software like SQL Server on the Hyper-V host.

Hyper-V can run on a lot of hardware, as described in the Windows Server Catalog. It is also a lot more flexible when it comes to storage configurations, as Windows supports more disk controllers than ESXi.

Hyper-V can be automated using WMI, there is no direct PowerShell support (though you can use PowerShells WMI support).

You can deploy Hyper-V on Windows Server Core, as a dedicated VM host. Managing Hyper-V in this scenario requires a machine running Windows Server 2008 or Windows Vista with the Hyper-V management tools installed. This is the recommended deployment mode.

You can also install Hyper-V on a full Windows installation. Though not recommended, this allows you to logon to the machine using RDP and manage the VMs directly on the server using the same Hyper-V management tools.

Here is one of the biggest advantages Hyper-V has over ESXi. For example, if you setup the WS2008 Standard Server as a SQL Server, you can install Hyper-V after the fact with a simple reboot. Though this is not what Microsoft recommends, the reality is that most Small Businesses have to achieve a lot with less equipment. Running such a configuration can help fix business problems without having to reinstall a machine.

System Center Virtual Machine Manager allows you to manage Hyper-V centrally. It’s quite expensive, so i doubt many small businesses will start using it. Maybe the next version of System Center Essentials will include a subset of SCVMM functionality.

Conclusion

Hyper-V supports more hardware, and is more flexible when it comes to it’s deployment. For me, this makes Hyper-V the better choice for a Small Business than ESXi. XEN Express is absolutely unusuable in a production deployment.

Now, Enterprise admins will probably slap me for the “flexible” deployment of Hyper-V, and they are right. But for most small businesses, being able to cut corners in IT is more important than running “recommended” configurations.

I’m using Hyper-V standalone on a machine in a hosting center to run my private infrastructure (where i plan on moving this blog to), and it’s also a full Windows installation. Hyper-V runs flawlessly in such a scenario.

I also didn’t talk about Vmotion, HA, DRS and all the other fancy features that VMware has and Hyper-V doesn’t have yet - simply because they do not matter to a small business.

As almost all data from that RODC is replicated through DFS-R, backing it up wasn’t that important, we had a few more business needs that couldn’t be solved by using DFS-R to backup in our HQ in Horgen.

So we purchased a BackupExec Media Server license, and i tried installing BackupExec. It reminded me that installing on an RODC requires a seperate Windows installation that runs SQL Server. Well, we have Hyper-V and enough Windows licenses to do this, so i didn’t think of this as a big deal.

I’ve setup a VM with WS08, installed SQL Server Express with an Instance called “BKUPEXEC” and tried installing BackupExec, pointing it at the remote SQL Server Express (that was configured to allow remote connections).

The RODC is called LYS-RODC-01. The SQL Server Express VM is called LYS-SQLE-01, with a SQL Server Instance called BKUPEXEC.

It didn’t work:

08-08-2008,23:22:58 : There is no MSSQL$BKUPEXEC Service
08-08-2008,23:22:58 : V-225-212: Unable to connect to SQL Server. ***To search for information about this error, click here
08-08-2008,23:22:58 : Failed to configure SQL instance LYS-RODC-01\BKUPEXEC SQL instance to allow updates.
08-08-2008,23:22:58 : Action ended 23:22:58: InstallFinalize. Return value 2.
08-08-2008,23:22:59 : Action 23:22:59: Rollback. Rolling back action:

The error message seems strange. Why does it connect to the RODC - there is no SQL Server on the RODC, and i configured it correctly in the setup.

I read through the logfile multiple times. Didn’t find a mistake. Reinstalled the SQL Server VM a few times using a variety of SQL Server and OS combinations.

I contacted Symantec Support (which was a bit of a letdown, first i had to talk someone in one of the Eastern European countries who could barely speak German, and next i had to talk to someone from India who could barely speak English, much less German). After almost a month, i still wasn’t anywhere near a solution.

I’ve spent a few more days playing around until i finally tried something that worked.

I changed the name of the SQL Server instance from BKUPEXEC to SQLEXPRESS.

This fixed the problem.

I’m still baffled by this.

With just a bit of work, this has worked out (the number of sales we made also played into that).

If you’re wondering how we got 120 points with us being a rather small company, it’s quite easy:

• Get two competencies - for this you’ll need:
  • Two MCPs with relevant certs
  • The Information Worker & Network Infrastructure competencies are the easiest
  • Three customer references per competency minimum - 10 customer references total to get full points
• A total of 10 customer references - you’ll need only 6 if you have 18 points for Sales Performance
• Microsoft Small Business Specialist for 5 extra Points - for this you’ll need 70-282 and an online exam.
• A minimum of 7 points through MCPs - two MCITP/MCSE and one MCP will give you that
• A minimum of 10 points of Sales Performance

This should give you a total of 122 points - more than enough for a Gold Certified Partner!

As a side note, my current employer Acommit AG also has a job opening in a Systems Engineer position. If you have strong Windows, Exchange and IBM i skills and thinking of working for a company near Lake Zurich, apply now!

It’s the successor of several System p systems (which i know nothing about), and of the System i Model 515 (9407-515). As such, it targets small businesses.

Yesterday i received the first M15, to be installed in for our SaaS (Software as a Service, the IBM Slang for Application Service Provider) Project. This is the first standalone POWER System that i got my hands on, but i’ve already tested running IBM i on Blades.

The M15 is a standard 19″ 4U Server at half depth that can hold dual power supplies for redundant power, has a variety of expansions slots and supports up to 6 internal 3.5″ SAS Disks. The integrated SAS RAID Controller has support for a battery backed write cache. You can also install several PCI-X and PCI-E cards. The system has 2 8x PCI-E Slots, one 16x PCI-E Slot and two PCI-X slots.

In this case, the system came with a HMC - a normal System x3550 configured with a single 320GB SATA Disk drive. Interestingly, we had ordered the same HMC (7310-CR4) half a year ago. Back then, it shipped with a 80GB SATA Drive and an external modem. This unit shipped with increased capacity in the SATA Disk drive, and an internal Modem. Though i have no idea why anyone would still use the modem.

The HMC isn’t very interesting from a hardware perspective either, so the focus is purely on the M15.

The first view on the front shows a new green bar, that symbolizes POWER6. I think it could use a bit of improvement, doesn’t look that nice. Much more interesting is that the control panel has essentially vanished. Like the light path diagnostics model in the System x, it has to be pulled out from the System to be of any use. This was probably done to save space.

The new control panel isn’t an improvement, unfortunately. Of course on systems with the a HMC attached you don’t really need it anymore - but most of the Systems we are going to ship will not have a HMC. The buttons are hard to use and hard to reach - it’ll be interesting doing procedures like 65+21 on those. This isn’t a deal breaker - but from IBM i expect them to get even details as this right - all in all, this isn’t a 1k Dell Server - it’s a 20k High-End IBM Server.

Much better in my opinion is that with the POWER Systems, the IBM i has finally moved to the year 2008 in regard to IO technologies. PCI-E and SAS is finally here. What i do not understand is why the M15 uses 3.5″ SAS Drives, and not 2.5″ SAS Drives. This would allow to fit more arms into the same chassis. e.G. the 2U System x3650 ships with the possibility to install up to 8 2.5″ arms. A 4U machine could have up to 16 arms - without needing more space.

The power supplies have been moved to the front of the unit, similar to the PCI Expansion units. I like it - the new power supplies are bit smaller than the older ones, and replacement is easier, thanks to them sitting in the front.

>The machine internals still seem a bit sketchy to me. The M15 wastes a lot of space that is used for the second CPU in larger machines. But it also has a completely new fan design, with four centrifugal fans in the back of the machine.

The new fan hot plug mechanism is very sturdy, and is comparable to the high quality fan design used in the System x3650. This was one of the biggest downfalls of the POWER5/5+ 520/515/525 hardware platforms that has been fixed in the new hardware. RAM accessibility still isn’t optimal in my opinion - you’ll still need to remove the fans to access the memory. IBM has better solutions for this - just look at the x3550 and x3650.

The fans are very, very loud. The unit we have here is a rack unit and the fact that there is no conversion option like for the POWER5 models might mean that the tower and rack units have different acoustic configurations or different dampeners. While loudness is a complete non-issue in a server room, smaller customers sometimes have the machine in their office. As soon as i get the first tower model i’ll write about their loudness level.

One of the things that’s completely new to the POWER platform is the HEA - the host ethernet adapter. It allows to share a physical NIC with other partitions - that’s a very good feature, but i wasn’t able to play with it yet - this machine is not partitioned.

The BBWC in this machine is now hot pluggable. It’s great to see this, but in my opinion it wasn’t really necessary. There’s a reminder directly on top of it that you need to set the disk cache into an error state before replacing it - and that reminder is very important. If you don’t pay attention to it, you might have to make an unexpected test of your DR strategy.

The expansion and console capabilities of the M15 are artificially restricted - you can not have a PCI-Expansion Unit (no HSL/12X Ports) and you cannot have IOPs in the base unit. The conclusion: No Twinax, No U320 Tapedrives.

Especially the Twinax bit is, in my opinion, a good move. It will force customers stuck in the AS/400 days to get current with all their other hardware like printers. On the other hand, it might also cause those customers to stay stuck with their model 270 or model 800. As for me personally, i’ll have to deal less with Twinax - which has to be a good thing.

The Thin Console, a very good option for Small Businesses, is gone (because HP bought Neoware). With Twinax also gone you now have the choice to either get a HMC for 6k or get a Windows PC and use LAN Console. Both options aren’t really what a Small Business needs - a console that “just works”. The TC and Twinax console fit those criteria. The LAN console has issues on their own (The whole Systam i Access package is … aging) and the HMC requires a boatload of expertise that SMB operators just don’t have. We will go with the LAN console, mainly due to the HMC pricing, but i’m not really content with the console situation on the new systems.

This picture shows the rear of the unit, as you can see the cabling in my lab environment is always top notch. The HEA offers 4 ports by default, a bit much for a standalone system. I’ve implemented a Virtual IP Address setup to create redundant network connections. Not as cool as native Teaming/Bonding support, but it works well enough.

In general, the new model has improved several things on the old hardware, left one or two things in the same state, and has two new issues (the control panel, noise level). All in all, a good solid deal of hardware.

Questions? Comments?

Recently upgraded to Symantec Endpoint Protection 11, around 1 Month ago. A week ago, the customer complained that one of the Terminal Servers crashed constantly, requiring a reboot to recover.

Quick investigation showed that the machine was running out of paged pool.

Event ID 2020
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Description:
The server was unable to allocate from the system paged pool because the pool was empty.

I’m not proficient with Terminal Servers or Windows 2000, but debugging this issue was mostly similar to what you do when debugging pool issues on Windows Server 2003. First you need to enable Pool Tagging, which is enabled by default on Windows Server 2003 but not on Windows 2000. KB177415 explains how.

After that, install the Windows 2000 Support Tools, and run poolmon /p /p /b.

In my case, the output looked like this:

The limit for Windows 2000 Terminal Servers is 160 MB. As you can see, the machine here is idle and without any users on it. And we’re already at 132MB utilisation.

There are two culprits: “CM” and “SavE”. The Pooltag “SavE” is the Symantec Endpoint Protection Virus Scanner Driver. It clocks in at 50MB. The other Pooltag “CM” stands for “Configuration Manager”, and is the registry. It is 67MB big.

This is not normal - the other Terminal Server, the CM tag is a lot smaller, only 35MB. The “SavE” tag is still 50MB. This explains why the other TS does not have the same problems as this one. But we don’t know why one registry is so much bigger than the other.

This can be found out by using the dureg.exe tool, which can help us resolve the issue.

As you can see from the picture above, the enlarged registry is also caused by Symantec.

C:\Programme\Resource Kit>dureg /lm “SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Quarantine”
Size of HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Quarantine: 26111494

Clocks in at 26MB. The Quarantine key contained around 20′000 subkeys, each with a simple number below. Each was about the same .doc file.

After deleting the Quarantine key, the CM pooltag went down from 67MB to 35MB - just like the other TS.

The next step was obvious. Remove Symantec Endpoint Protection which something that doesn’t suck as bad: McAfee AntiVirus Enterprise. I downloaded an Evaluation Version, and installed it.

And the results were obvious:

Do you see the pooltag “NAI0″ somewhere in this list? I don’t. It’s there, but somewhere around Page 400, and surely not eating away 50MB of my paged pool.

So if you have problems with your machines running out of paged pool, frequently showing Event 2020 with Source Srv, check the registry size and replace Symantec Endpoint Protection with something that doesn’t suck that much.

Setting up the hardware

Finally, after almost two months i was able to get my hands on a SAS cable with the correct pinout to connect the TS3100 library to the SAS Connectivity Module. After plugging in the cable, both link lights went up. The TS3100 was connected - physically at least.

I logged into the Storage Configuration Manager, and dared to assign the HS21 Windows Blade and the JS12 Blade to the external SAS Port. When looking at the Windows Device Manager, it immediately recognized the Tape Drive, but didn’t recognize the media changer. No bummer, i haven’t installed BackupExec yet.

It’s important to notice that the IBM i OS will not see your tape drive. You cannot connect the tape drive to the IBM i OS, only to VIOS. This means that you always will do a disk to disk backup to the VIOS partition, and then use VIOS to save the D2D Image to tape. Restoring works the other way around - you restore from tape to VIOS disk, then load that disk into IBM i, and then run your restore or even IPL from the virtual optical medium.

This is not optimal, as this means that you cannot use BRMS to manage media (you can still use it for saving, though). It also adds another layer of indirection that makes automating backups more difficult. Another important point is that other i machines will not be able to read the VIOS created tapes, and that your i Blade will not be able to read tapes created by standalone POWER machines running IBM i.

Configuring VIOS

So i went to logon VIOS, running on the JS12 blade. I ran “lsdev | grep rmt0″, but apparently there was no tape drive to be seen. I ran “cfgdev”, to let the operating system configure devices, but that wasn’t met with success either.

This time, i chose the easy way out. I just rebooted the entire blade, and finally, the tapedrive showed up:

$ lsdev -dev rmt0 -attr
attribute      value            description                               user_settable

block_size     262144           BLOCK size (0=variable length)            True
delay          45               Set delay after a FAILED command          True
density_set_1  0                DENSITY setting #1                        True
density_set_2  0                DENSITY setting #2                        True
extfm          yes              Use EXTENDED file marks                   True
mode           yes              Use DEVICE BUFFERS during writes          True
res_support    no               RESERVE/RELEASE support                   True
ret_error      no               RETURN error on tape change or reset      True
rwtimeout      144              Set timeout for the READ or WRITE command True
var_block_size 0                BLOCK SIZE for variable length support    True
ww_id          5000e1111c878001 World Wide Identifier                     False

Okay. So as the TS3100 is a LTO4 Tape Library, the next step obviously was to load a tape into the tape drive. Now this can be done through the Web Interface or the Control Panel of the library, but that’s not the way you want to go during day to day backup - media moving should be handled by the backup software (called BRMS on IBM i).

But alas, this is not as easy on VIOS. Basically, you can get an AIX root shell on VIOS by typing “oem_setup_env” - and then there are several AIX commands to manage a tape library.

There was only one problem:

# mtlib
ksh: mtlib:  not found.
# tapeutil
ksh: tapeutil:  not found.

They’re not there. A quick google search revealed nothing. I didn’t know how IBM thought how we should use a tape library. In sequential mode? Or is there some way to manage a tape library in VIOS? If you know, tell me!

So, the next step was to create writable optical media, so i could start with creating a Save 21 of the system. This seemed as a sane first step.

Initializing the media on the i side

First, i needed to create a virtual optical volume in a volume library on VIOS - i already had the volume library from the IBM i installation, so all i needed to do was to add a writable optical volume. The system used about 40GB DASD, so i created a virtual optical device with a size of 80GB to leave room for future growth. This process took around 2 hours, probably because VIOS pre-blanked the file (it effectively used up 80GB on disk). A rate of 0.6GB/min. During this operation, the VIOS webinterface grinded to halt (didn’t respond), but SSH was still available and responded very slowly to each command. I suppose this is some issue with the disk controller, either the driver or firmware.

According to vmstat, most of the time is spent in system or IO wait context:

kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 1  1 203749  6107   0   0   0 2485 3279   0 150 29483 6542  6  9 77  7  0.18  17.5
 1  1 203732  6167   0   0   0 2464 2869   0 153 30479 6333  7  9 74 10  0.18  17.6

The next step was to load the newly created virtual media into the virtual optical drive that was attached to the IBM i partition. This was rather easy to do, just click through the web interface.

Now, we need to initialize the optical volume in IBM i OS:

INZOPT NEWVOL(’Save21′)
DEV(OPT01)
CHECK(*NO)
TYPE(*PRIMARY)

This finished rather quickly, and i then started the Save 21.

The Save 21

GO SAVE/21. The performance wasn’t much better, though. the vmstat looked the same as above, indicating the same problem.

Further debugging using iostat revealed that a volume group is not a RAID1 array - but nonetheless, the disk subsystem is behaving oddly:

hdisk0, hdisk1: VIOS installation - part of it is mirrored VIOS, other part is volume group that is spanned across these two disks
hdisk2-5: SCSI passthrough to the IBM i

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        614.0                0.0   7.7   72.2     20.1   0.1    8.3

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           1.0       8.0       2.0          0         8
hdisk1          85.0     20036.0     179.0      10080      9956
hdisk2           4.0     926.0      50.0        922         4
hdisk3          14.0     4481.0      98.0       4477         4
hdisk4          13.0     3730.0      96.0       3726         4
hdisk5           3.0     652.0      36.0        648         4

This is pure sequential IO - why is it reading from the disk? A similar picture was seen throughout the whole backup - even when backing up image catalogs from the IFS. hdisk1 consistently showed strong write activity. No idea why.

My AIX skills are weak, and i didn’t know a way to see on which files the write IO happened - however it’s important to know that read and write always showed the same numbers. To me, this looks like a problem - either in my config, firmware levels, or even a problem on IBMs side.

Either way, the Save 21 completed in 45 minutes. At around 40GB, this brings us to 0.9GB/min.

Backing up the virtual image to tape

The next step is to backup to our LTO4 tape.

Here’s how the backup itself looks:

# find /var/vio/VMLibrary/D2D_1 -print | backup -ivqf /dev/rmt0 -b 512
Backing up to /dev/rmt0.
Cluster 262144 bytes (512 blocks).
Volume 1 on /dev/rmt0
Backup finished on Mon Jul 21 20:45:18 CEST 2008; there are 167772672 blocks on 1 volumes.

And the sequential IO performance is much more reasonable:

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        615.0               11.2  29.0   58.2      1.6   0.4   40.9

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           0.0       0.0       0.0          0         0
hdisk1          99.0     81920.0     160.0      81920         0
hdisk2          17.0     2250.0      29.0          0      2250
hdisk3          12.0     2236.0      28.0          0      2236
hdisk4          12.0     2385.0      30.0          0      2385
hdisk5          16.0     2250.0      29.0          0      2250

That’s roughly 5GB per Minute. A very decent performance.

Interoperability

Now, this is where it gets interesting. Attached to the BladeCenter S, we have a TS3100 with a single drive, in the BladeCenter S we have three Intel Blades running Windows Server 2008 and one POWER Blade running IBM i.

We need to back up all this to the TS3100 - on the Windows Side, i’ll be using BackupExec 12, on the i Side VIOS. How do i make sure that the tape drive can be used from both sides, without to much interaction?

The SAS Connectivity Module can attach the same port to multiple Blades. So installed BackupExec on one of the Windows Blades, just to see how interoperability would work out.

I ran a backup & restore on another tape, from BackupExec. This worked fine. The next step was loading the tape from the i Save back, and then run a test restore from that. Unfortunately, i couldn’t use BackupExec to just move the tape in the drive, so i had to use the TS3100 Web Interface again.

I looked at the tape drive from VIOS, which also seemed okay. I also saw how much space was used on the VIOS tape after checking it up with BackupExec (this is stored on a small RFID Chip on the Tape itself). But the real test was yet to come:

Restoring from Tape

I started the restore from tape.

# restore -xvqf /dev/rmt0 -b 512 /var/vio/VMLibrary/D2D_1
New volume on /dev/rmt0:
Cluster size is 262144 bytes (512 blocks).
The volume number is 1.
The backup date is: Mon Jul 21 20:24:04 CEST 2008
Files are backed up by name.
The user is root.
x  85899345920 /var/vio/VMLibrary/D2D_1
The total size is 85899345920 bytes.
The number of restored files is 1.

iostat also told me that the performance on restore was bit worse than when backing up:

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        611.0               14.7  85.0    0.0      0.2   1.0  101.9

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           0.0       0.0       0.0          0         0
hdisk1         100.0     48216.0      68.0          0     48216
hdisk2           0.0       0.0       0.0          0         0
hdisk3           0.0       0.0       0.0          0         0
hdisk4           0.0       0.0       0.0          0         0
hdisk5           0.0       0.0       0.0          0         0

So after having restored the image file to VIOS, i could IPL from it and run a complete system restore. Nice. I didn’t want to scratch my whole setup and wait until a full restore, so i tried something simpler.

In the end, it turned out to be around 2GB/min. This was a lot faster than the creation of the file, which seems really odd to me.

Running a Test restore on the IBM i side

After restoring the file in the VMLibrary, it automatically appeared again in VIOS. I only had to mount it to my IBM i partition. This could easily be done through the VIOS web interface.

I ran a simple restore:

RSTLIB SAVLIB(AVNEDIAS) DEV(OPT01)

With simple results:

12 Objekt(e) von AVNEDIAS nach AVNEDIAS zurückgespeichert.

I also tried IPLing from the virtual optical media, which brought me into the limited paging DST. Nice!

Is this good?

AS you can see, this is not your fathers AS/400. This is a POWER Blade running IBM i. You’ll need to learn a bit about VIOS and AIX in order to make any sense on how this whole stuff works. But it’s not rocket science - i only know a bit about Linux, and was able to figure out the tasks i needed to do.

But now, how should one run this in production?

The current configuration i have seems unsuitable to production to me.

• VIOS/AIX can’t handle the tape library in random mode. This is a big letdown.
• Backing up to tape and restoring seems very, umm, basic to me
• The fact that you are using virtual optical devices, with no ability on the i side to change media, makes a usuable backup procedure hard to implement
• Automation on the VIOS side could be implemented by the i running the ssh client in command execution mode (similar to how this is used with the HMC
• Integration between the Windows and VIOS seems cumbersome

Solutions

Two Half-Height Tapedrives in a partitioned library

The TS3100 can be partitioned, and we could install two half-height LTO4 tapes. Reserving 12 Slots for Windows in Random Mode, and 12 Slots for VIOS in Sequential Mode. This would work, but there’s one huge downside: Price

Backing up on Windows only

Instead of using VIOS and clumsy hacks to get a halfway decent functionality, you can use Windows to backup everything. Save 21 and Systems Saves would run through VIOS in order to create bootable media, and then retrieved on the Windows side using SFTP.

For daily backups, we can use savefiles directly on the i, which is probably easier to deal with for most IBM i admins. These can be retrieved from the Windows side using FTP/TLS. The downside: If you have i and Windows people that work well together, i don’t see much of an issue. But if not, you got a big mess in your hands.

SAS Passthrough to IBM i

Unfortunately, this does option not exist yet. This is something that IBM should work on intensively. It will allow i admins to use well established backup processes with full library integration using BRMS.

Conclusions

Backing up the IBM i on Blade isn’t exactly easier than backing up a standalone POWER machine. In fact, it’s more difficult and requires additional skills.

Before buying a JS12 blade running IBM i, make sure that you think your disaster recovery strategy through completely. Your business partner should be able to help you with this.

Planning is crucial - Backup & Restore on the blade is different, and you’ll need to deal with VIOS when creating a procedure for fully automated Save 21 backups.

Any questions? What do you think about the situation? Want me to test something for you? Just leave a comment!

I also created a “i on Blade” category. Look at it if you want to see all my posts about this subject.

In the meantime i wasn’t just wasting my time on trivial things such as getting actual customer work done, but also playing a bit further with the JS12 Blade.

I’ve installed the software my company produces (DIAS-iS) on the JS12 blade, and ran a few very unscientific benchmarks. But first let’s talk about the disk situation in with IBM i on a JS12 blade in a BladeCenter S (i really like those convoluted product names).

Managing disks under IBM i on a JS12 Blade

As i found out the hard way during the initial bladecenter setup, the JS12 blade only supports SAS disks, and can cause issues if you have SATA disks zoned to it.

There are a few important considerations when thinking about the IBM i/JS12/BladeCenter S combination: First off, disks are directly attached to VIOS, and then virtualized by VIOS for the IBM i as SCSI disks. It’s important to note here that you do not have any (supported) options of RAIDing the disks before the IBM i sees them. So all disks are mapped through 1:1 to the IBM i OS, and then mirrored using IBM i mirrored protection. This is entirely different from the approach you would use in a BladeCenter H with a FC attached SAN.

Just to be clear: There is no cache on the JS12 and there is no way to use any disk protection except IBM i mirrored protection. You can’t use RAID5, RAID6 or hotspares. You can’t VIOS mirrored volume groups either, because it’s unsupported.

I’m thinking about removing one of the disks from the BladeCenter in order to test how recovery from a disk failure would look like, but i’m afraid of wasting a lot of work that i’ve already invested in this system - i’ll try this shortly before i have to give everything back.

I’m not sure what the virtualization by VIOS exactly entails, but i would assume it’s fairly similar to what Hyper-V/ESX do when you create “Passthrough disks”. This probably means that things like Predictive Failure Analysis (PFA) will probably not work.

Another, rather obvious, drawback is that you cannot install any expansion cards (well, there is the odd one you can install into the blade). But it also means there is no Twinax, no SNA directly over Ethernet, no Modems, etc. Not a big issue for us, as we’re urging our customers to stay current on technology, but not everyone is an IBM i shop - there are still lots of AS/400 shops out there.

If you access the System i Navigators SST/Disk management function, it will not be able to help you with disk locations. I haven’t found out on how to call disk locations in IVM/VIOS, but then again i don’t really know much about IVM/VIOS.

Installing DIAS-iS on the JS12 Blade

I’ve installed our software without a hitch, and loaded our 30GB Test/Benchmarking database on it. I ran several benchmarks, and the JS12 with it’s four SAS 15kRPM 147GB Arms in a mirrored configuration and one core and 13GB RAM in the IBM i LPAR was a bit slower (less than 5%) than our System i515 with four U320 15kRPM 70GB Arms in a RAID5 configuration and 3.5 GB in the IBM i LPAR.

Unfortunately, i do not have a M15 to pit against the JS12, as these two would be using comparable technology. It would also be interesting to see an M15 with four 147GB SAS disks in a mirrored configuration to compare the systems 1:1, especially regarding disk performance.

Next steps to go?

What’s next? Well, Backup obviously. If you’ve read the i on Blade manual you’ll see that saving to tape will be interesting to say the least. I already have a TS3100 ready to go, but i’m currently missing a SAS for attaching it. As soon as i have the cable, expect a big post about saving and restoring.

Questions? Suggestions? Any specific questions about i on Blade? Want me to test something for you?

Leave a comment or drop me a mail. I’ll be happy to help.

Luckily, this was easy to remedy. Just access IVM and remove one of the cores assigned to the LPAR running IBM i.

IBM licenses it Software based on numbers of Core, not number of Sockets like most other vendors do. This is important to note, because i though the message was wrong as the blade only has a single CPU.

Just two hours ago, i’ve received a shipment of four 3.5″ 147GB 15kRPM SAS Disks. I installed them into the BladeCenter S immediately.

I used SCM to assign the disks in the DSM to the JS12, and then booted the blade.

VIOS was already installed. So all i had to do was to create a new partition.

After creating the partition, i didn’t IPL it just yet. I needed System i Access in order to provide a console. The Operations Console part of System i Access is not supported on Server versions of Windows, so i couldn’t install it on one of the other blades running Windows Server 2008. At least not directly. So i installed the Hyper-V Role on one of the blades, and installed Windows XP and System i Access on it.

I then IPLed the partition and a minute later i was standing there with a lit attention light. I forgot about the CD drive. Bummer. I assigned the media tray to the JS12 blade, but it couldn’t see the CD drive. This must’ve worked before, because i installed VIOS using a CD. I restarted the JS12, but that wasn’t helping. Still no CD drive that i could assign to a partition. Didn’t find much on the web about this problem either, so i decided to use virtual media to install the operating system.

I logged into VIOS using SSH, downloaded the I_BASE_01 CD Image from our production system using FTP, and imported it into VIOS’s media library. I activated the I_BASE_01 CD Image, and booted. I also enabled the operations console connection (which is fairly straightforward, with just in this case the first IBM i instance having the partition ID 2).

After 10 minutes, the signon screen for the operations console finally appeared. That was kind of a Heureka! moment for me, altough i didn’t really do that much stuff yet.

I chose to install the LIC, and i was presented with a screen that i haven’t seen before - i was able to select the which disk i want to be the load source.

After that, the system started to initialize the hard drive. This was really slow on my system, taking around 5 hours for a single 147GB 15kRPM drive. I hope this isn’t indicative of the IO speed we will see when the IBM i OS is running.

While waiting for the formatting to complete, i tried to find a way on how to turn off the attention light that was lit because of my earlier mistake when trying to boot the partition. There is a detailed IBM document about turning off the attention light using IVM/VIOS. It’s a simple command: chled -r sa -t virtualsys -o off

As you can see from the screenshot (took during the middle of the run), it took quite long. In fact, it even exceeded the three hours it estimated and took 4.5 hours. I have an issue with that - the Intel blades do not need that much for initiating a RAID1 set, or NTFS formatting the disks. Even though they’re using slower 500GB SATA disks. It’s just leaving a bad impression for no reason. And it’s also an issue with disaster recovery.

Installing the LIC had a more reasonable speed, took 2 minutes. After the system IPLd again, i was able to add the three other disks. Adding the disks proceeded at a much more reasonable speed, but then it hung at 99%. After two hours, the system was still stuck at 99%. At that time, I went to bed, hoping the system would be finished in the morning.

And it really was finished in the morning. The next step was to start mirrored protection. It even complained that i was running virtual disks, and a failure of VIOS would lead to the system crashing anyway. I proceeded. As always. the first part was pretty quick, and i proceeded with the OS installation.

As the LIC started, initialization of mirroring began. The first time estimate was fifteen minutes. The next one was 6 hours, 10 minutes, then 8 hours. But then i jumped back to 5 hours. I left for a customer, and looked at the status occasionally. I took somewhere between 3 and 4 hours to complete.

Next, i had to change the virtual media in order to allow the OS installation to proceed further. It’s important to know that this has to be done in the partition configuration, not in the virtual media tab. And that you’ll need to acknowledge the partition change in order to make the media change active (the AJAXy web interface doesn’t make this entirely clear).

After that, the IBM i installation started and proceeded at an acceptable speed. In around an hour, the basic operating system was installed.

After that, installation of the licensed programs started. It halted after just half an hour and telling me i had a screen error (MSG CPF3D92). I suspected a problem with the operations console, restarted the XP machine running OpsCon, and retried the installation (with just the base system). The problem happened again. This seemed odd.

Having no idea on how to proceed further, i retried again. This time it worked(?). I figured it was an OpsConsole problem, probably related to the fact that the machine running OpsCon was virtualized. I quickly installed the TCP/IP utilities, IPLd the system and installed the remaining programs using a 5250/Telnet connection.

While the installation happened, i used an additional session to explore the system. The disks where shown to the system as virtual disks, similar to SAN attached disks. But one of the more interesting parts was looking at the Hardware Service Manager in SST/DST - it was completely empty, and didn’t contain any hardware. For me, this was a moment that was quite indicative of the whole experience - i on Blade is not “AS/400 in Blade Form”. It’s a completely new environment that you’ll need to learn to deal with. You got another layer of indirection (VIOS) with it’s own platform (AIX), plus you have the blade management in itself.

The whole setup took me roughly 24 hours (i started a day ago at 16:00). Of course, the system wasn’t always busy because i didn’t give him any work, but it’s worth to note that setting up a JS12 blade takes considerably longer than setting up a model 515 or M15.

I will now continue setting up our ERP application and make further tests with the hardware. If you have any requests for screenshots or want me to test something out, tell me!

We first started our internal virtualization stuff when both VMware GSX and Virtual Server 2005 still cost money. So we used VS2005 because we could get it for free since we were in the Microsoft Partner Program.

So, with the release of Hyper-V we finally had a chance to move to a more robust and faster virtualization solution - however, not everything has improved with Hyper-V, for example delegating permissions which was easy in VS2005 has now become much more complex. Probably because Microsoft wants to sell SCVMM 2008 that will automate a lot of this.

We have a few development VMs that are used for QA purposes by our development team - and we just have a single machine running Hyper-V. So i want to delegate a few of the VMs to the development team, without them being able to manage the Hyper-V server or virtual machines that do not belong to the development team.

I’ve found an excellent resource regarding setting up remote management for Hyper-V from John Howard. He has an excellent 5 Part series on how to enable remote management.

Part 1 Part 2 Part 3 Part 4 Part 5

What is not described in these links is how to delegate specific VMs. For doing this, you’ll need a script from Andrzej.

Hyper-V Azman Scope Scripts

Here’s a basic rundown of the general steps you’ll need to do:

• Create an appropriate Active Directory group for the users you want to give access to. If necessary, nest the groups according to your organizations group strategy
• The following two steps are detailed in Part IV from John Howard
  • Add to the group to the local “Distributed COM Users” group on the Hyper-V host
  • Grant the group permissions on the Root\CIMV2 and Root\Virtualization WMI Namespaces
• For detailed instructions for these three steps, see below.
  • Run azman.msc and create a new scope
  • Use the SetScope VBS script to assign the VM to scopes.
  • Run azman.msc and delegate appropriate permissions to Windows Groups using newly created scope

Creating scopes in AzMan and assigning VMs to scopes

First, you’ll need to start azman.msc and open to following Authorization Store: C:\ProgramData\Microsoft\Windows\Hyper-V\InitialStore.xml

Then, you’ll need to right click “IntialStore.xml” and choose “New Scope”. In my case, i named the new scope “Dev”.

Next, you will need to create a role in the top-level of the authorization store. This role is needed so that the Hyper-V Management tool can even connect. I called mine “View Only”, as it does not grant any specific permissions. It should look like this:

You’ll also need to add the Windows Group to this azman role in order for it to be of any use:

Next, we need to create a role that grants the necessary VM management skills to the Dev scope. It should look like this:

You’ll also need to add a Windows Group to this role.

After you’ve come so far, we will need to assign the VMs to the newly created scope. You can find the scripts here: Andrzej’s Hyper-V Scripts.

Assigning a VM to a scope is simple.

For example, if you want to assign the VM “dev-hdi-xp-01″ to the scope “Dev”, use this command.

setscope.vbs dev-hdi-xp-01 Dev

There will be three popup Windows - the first two don’t matter, and the latter will contain a single number. If the number is 4096 (or anything else), it failed. If the number is “0″, it succeeded.

You can verify scope membership using getscope.vbs

getscope.vbs dev-hdi-xp-01

The result should look like this:

If my posting is entirely correct, and you followed it correctly, the end result should look like this:

Here, we’re logged on as an admin. All VM’s are visible:

Here, we’re logged on as a normal user. It does not have any special privileges on the Hyper-V box, except the WMI / DCOM and AzMan changes. You’ll only see the two Development VMs.

So, this is quite a bit more complex than VS2005. But also a lot more cool.

I hope there are no mistakes in this post. If you find any, please tell me. If you found this post helpful, tell me too. Thanks for reading!

So far, we have mostly used ZyXEL’s ZyWALL products to serve our Small Business customers, however the ZyWALL Line wasn’t always very satisfying when moving to the upper end of the Small Business spectrum. Thus, we had a look at SonicWALL - i’ve been using them for quite some time.

There are a few things about SonicWALL that is different about people which are used to the low-end market (like the ZyXEL products).

• You’ll need to purchase Software Maintenance in order to be able to download newer Firmware versions
• The old SonicWall Hardware Generations (TZ / PRO) have “Standard” and “Enhanced” Firmware images - the Standard versions are stripped down and less flexible - the NSA Models just have “Enhanced”
• Registration on MySonicWall is mandatory

One of the things fixed with the release of SonicOS 5.0 was the graphical user interface - the new GUI is completely revamped, and looks like something that belongs to the Year 2008. Other improvements include a completely redesigned hardware, that uses multi-core CPUs to provide real-time traffic analysis.

The NSA Series ship with basic Firewall/VPN features that are licensed as part of the base hardware. Additional features like Anti-Virus Scanning, Content Filtering, Anti-Spam, Intrusion Detection and Prevention all require extra expenses. This model is similar to what other UTM appliances like the ZyWALL 5 UTM uses.

SonicWALL Global VPN Client is a IPsec compatible VPN client, that works pretty well. There is not 64bit Version yet, and it doesn’t work with other VPN Clients running on the same PC. If you do not want to use SonicWALLs GVC, the SonicWALL also offers the ability to use L2TP and your Operating Systems native VPN functionality. While L2TP connections are mostly unrestricted, the number of GVC Licenses can be pretty low (e.G. 10 for the NSA2400).

One of the main advantages over the ZyWALL Line of products is the object-based configuration, and the ability to have multiple, Gigabit interfaces on the hardware - the NSA 2400 offers 6 Gigabit interfaces with the ability to use 802.1q VLANs to create even more logical interfaces. Even the low-end NSA 2400 can offer quite a lot of throughput (I’ve measured up to 30 Megabyte / s), which is important if you have Servers deployed in your DMZ.

Other cool features include the “SonicPoint” Management, which is basically the same as Symbol’s or Cisco’s Lightweight Wireless Access Points. This is a very cool feature in Smaller Businesses that do not want to buy separate Hardware to maintain their Wireless Infrastructure.

You can even access Live Demo of the SonicWALL Web Interface to see for yourself.

Advantages

• Very flexible configuration
• Streamlined GUI with useful features like Packet Capturing and self updating Log views
• Lightweight VPN Client and the ability to use Standard L2TP
• Lightweight Access Point Deployment using the NSA as a base
• LDAP Integration, preconfigured for Active Directory
• 6 Gigabit Interfaces
• High Performance

Disadvantages

• High price of Hardware (List: 2700 CHF)
• High price of mandatory service contracts for Firmware updates (List: 1300 CHF for 3Y 7×24 and HW Advance Replacement)
• High price of UTM features licenses (List: Starting at 1700 CHF for 3Y AS/AV/IPS)
• Incomplete user authentication solution (based on an Agent using WMI to query logged on userinstead of using secure Kerberos authentication)
• No redundant PSU or Fans to compensate for high hardware price (the NSA 7500 has redundant Fan/PSU)

The BladeCenter S

The BladeCenter S i received came with 10 500GB SATA Disks and two DSMs, four power supplies, an Advanced Management Module, a Server Connectivity Module and a SAS Connectivity Module. The power supplies use standard 230V type 23 plugs, which do require a little special installation, but much less so than industrial plugs used with the bigger BladeCenters.

The big point about the BladeCenter S is that it does not require an external SAN to provide Storage to the Blade - an integrated SAS Switch that allows very flexible disk configurations is integrated. Configuration can be done using a Webbrowser against the SAS Connectivity Web Interface, using SSH/Telnet to access the SAS Connectivity Commandline, or using a fully graphical interface using IBM’s Storage Configuration Manager. There are some predefined configurations, but none of them suited my configuration - creating new configurations using SCM is easy enough though.

The disks in the BladeCenter’s DSMs (Disk Storage Module) are hot swappable - currently, only 3.5″ DSMs are available, with a 2.5″ DSM in the pipeline. Most of the blades support one or two internal disks, but the problem here is that these disks are not hot swappable. Depending on your Blade loadout, 12 disks might not be enough. For example, the HS21 XM Blades only fit one internal disk, and running without RAID on the System partition seems pointless, so you would be using at least 6 disks (without hotspares) for a basic Exchange deployment.

The Webinterface on the AMM is nicely done, although it lacks a bit of flashiness. That’s not a requirement though, it does a very solid job at what it needs to do.

After powering up the BladeCenter S for the first time, i connected to it using a web browser and upgraded all the firmwares. There are quite a lot of them (AMM, SAS, Server Connectivity), but it all worked out flawlessly. Time to move on to the real course: the Blades.

The HS21 and the HS21 XM

Starting with the familiar first, i started with the HS21 Intel Blades first. The two HS21 Blades both had a 2.66 Ghz Quadcore and 4GB RAM, the HS21 XM Blade had a 2.5 Ghz Quadcore and 9GB of RAM (more about that later).

When starting the first HS21 Blade, after configuring all the storage using SCM, it failed to POST it’s LSI Logic SAS/RAID Controller. I searched for the error message on the net, assuming that i screwed up the configuration. I didn’t find anything meaningful, so i tried to do what everyone else would do in this situation: Apply every Firmware update for the Blade i could find.

Of course it wasn’t as easy as i wanted it to be. The controller not POSTing was an endless loop, i couldn’t get the machine to start from the AMM virtual floppy drive. I used SCM to disconnect the storage (by disabling the Blade’s SAS port). Now, the blade booted flawlessly, indicating that i probably had a problem with my disks. When browsing the IBM website, it became obvious that only newer firmwares support SATA drives. After upgrading the SAS Firmware, i was able to boot the blade without disabling the Blade’s SAS port. Unfortunately, the onboard SAS controller only supports RAID level 1 and 10. Probably owed to the fact that most blades are using SAN storage - IBM promised that there would be SAS RAID adapter that supports other RAID levels - these are especially important for the cost-conscious SMB market.

I booted a Windows PE 2.0 using WDS, and was able to install Windows Server 2008 x64 without any issues.

The HS21 XM blade on the other hand complained when booted for the first time that it’s memory configuration was invalid - it only supports 2, 4 and 8 DIMM configurations - 6 DIMM configurations are not supported. I removed two 512MB modules and booted the Blade with 8GB - it worked flawlessly and without complaining.

The JS12

First, read this document about i on Blade. It explains everything better than i ever could.

The JS12 is a POWER6 based blade that is able to run IBM i. The first time i turned on the blade, all the HS21 blades (already running Windows Server 2008) crashed hard. When rebooting, they no longer found their drives. I turned off all the blades, disconnected the JS12’s SAS port and turned everything on again. The Intel blades booted, and after i was sure that they’re up and running again, i powered on the JS12 again. This time, no issue arised. I tried to reproduce the behaviour i’ve seen before, and the same thing happened again.

My current assumption is that the issues were caused by the SAS Controller which does not have a Firmware update yet, and can’t deal with the SATA drives located in the DSMs. Further investigation told me that there’s no firmware upgrade for the SAS Controller in the POWER6 blade, and that SATA drives are not supported when running IBM i on the blade anyway. I ordered 4 147GB SAS drives, disabled the SAS port on the blade, and tried booting the POWER6 blade again. It booted flawlessly again.

The next step was to install VIOS - this is a rather complicated multi-step process. First, you have to turn on “Serial over LAN” aka SOL, then logon to the AMM using SSH, connect to the POWER blade using serial passthrough and then boot the blade from the VIOS CD. The install is pretty self explanatory, but takes forever. Expect 3 to 4 hours.

Next is connecting to the Integrated Virtualization Manager (IVM) running on the VIOS partition. The IVM is basically a HMC light minus the console functionality. The only way to get a console on the JS12 blade is using a LAN console (which can only run on consumer versions of Windows, and is not supported on most of the Blades).

I installed the latest VIOS patches (around 4GB) and enabled mirroring on the two 147GB SAS disks in the blade itself. The next step will be installing IBM i, with which i have to wait until i receive the ordered SAS Disks. Installing the patches also takes quite some time, around 30 minutes.

Preliminary Summary

The BladeCenter S is great. Yep, not everything ran flawlessly from the start, but nobody’s perfect from the beginning. The BladeCenter brings an innovative new perspective to the SMB market. The problems that IBM needs to address are the addition of 2.5″ DSMs (already in the works) and more capable RAID controllers (also in the works). A BladeCenter S with the ability to use around 20-40 disks could prove interesting.

The POWER6 Blade is interesting, and while VIOS adds complexity, it is as streamlined as possible. I’m interested about seeing IBM i running on the machine.

If you have any other question about the BladeCenter S - or anything you would like to see in detail, post a comment. I’ll try to figure it out.

In General, my impression was that the exam was pretty solid but certainly “Enterprise Heavy” in focus. There were a lot of questions regarding appropriate configurations for failover clustering, and also several pieces of SCVMM 2008 (the latter though were never hard - anyone who has toyed with SCVMM and browsed through the main functionality should be able to answer them).

I’ve seen a few questions that weren’t worded 100% precisely, but that can always happen - the quality was generally high.

Other areas that were featured heavily:

• Clusters (as mentioned above)
• Snapshots - especially pay close attention on how Snapshots can be reverted, reused, etc. Snapshots can also be used in deployment scenarios
• Integration between SCOM and SCVMM
• Disk configuration - the available options for VHD files, their advantages and disadvantages, the usage of physical disks from the host and of course the use of iSCSI disks that are directly attached in the VM
• Hardware requirements and configuration requirements when setting up Hyper-V - pay close attention on how you configure the Windows Bootloader, and what necessary steps need to be taken when enabling hardware assisted virtualization in the BIOS
• Proper VM hardware configuration - remember which controllers in Hyper-V are bootable and which are not. Also, think about very old legacy applications that might have problems with newer CPU features available on modern CPUs and about the implications of running an OS that does not support synthetic hardware
• Network configuration - pay close attention to bigger scenarios involving the cluster heartbeat link, iSCSI connections from the host, iSCSI connections from the VMs themselves, Quorum disks in cluster scenarios. Also, remember the difference between internal and private network interfaces

Did i pass? I’m not sure. There were many cluster questions, and i never had much contact with those since i primarily work with Small Business customers.

So if you intend to go at this exam, make sure you’ve toyed around with SCVMM (SCOM knowledge not necessary, just look up on how these two can be integrated). Also, make sure you’ve setup a Hyper-V cluster at least once. You can emulate an iSCSI SAN by using an open source appliance like FreeNAS that can export disks using iSCSI. None of the questions i’ve seen seemed “hard” to me, but i was guessing at a few because i didn’t know about the topic.

Good luck!

I have to remedy that - when i checked my email this morning, i already got notice from Prometric asking for my MCP and Testing ID - i replied quickly, and got a an answer back in just a few minutes. This is good!

I’ll be going this Friday and see how it was.

You are invited to take beta exam 70-652: TS: Windows Server Virtualization, Configuring. You were specifically chosen to participate in this beta because of your current Microsoft Certification status or previous participation with Microsoft Learning. If you pass the beta exam, the exam credit will be added to your transcript and you will not need to take the exam in its released form. The 71-xxx identifier is used for registering for beta versions of MCP exams, when the exam is released in its final form the 70-xxx identifier is used for registration.

By participating in beta exams, you have the opportunity to provide the Microsoft Certification program with feedback about exam content, which is integral to development of exams in their released version. We depend on the contributions of experienced IT professionals and developers as we continually improve exam content and maintain the value of Microsoft certifications.

70-652: TS: Windows Server Virtualization, Configuring counts as credit towards the following certification(s).
• TS: Windows Server Virtualization, Configuration

So i tried to sign up for the exam. But i wasn’t even able to logon to my Prometric account.

Got the following error message:

Duplicate emails. Please call customer service.

So, i tried calling customer services. It’s a toll free Swiss number in a call center located at some other part of the earth. Unfortunately, i wasn’t even able to place a call

The number you’re calling is currently unavailable. Please check the number and dial again

So i mailed Prometric support and i’m hoping for an answer now.

If Prometric won’t fix it, at least i can ask Helmer what was in the exam. If you have a working Prometric account, you can get the invite code for the exam from Trika’s Blog

Update: Prometric fixed the problem quickly

For those of you who do not know SBS: SBS has traditionally been a single server setup with Exchange, SQL Server and ISA Server. It consolidates all “big” Microsoft technologies on a single server. This contradicts most “Best Practices” published by Microsoft, and as such SBS has always been seen as the red-headed stepchild in the Windows Server Family. SBS 2008 aims to improve several of these points (especially with the Premium Edition shipping with TWO server licenses).

After a 6 hour downloaded that trickled in at a few meager 200kbyte/s, i was finally able to get started with it.

SBS 2008 now demands x64 hardware - so for testing i used an IBM x3650 running Windows Server 2008 Enterprise with the Hyper-V RC1. Hyper-V supports 64bit guests. Other hardware requirements have also gotten steeper - you’ll need 4GB RAM minimum (though i launched the VM with only 2GB). The Premium Edition now comes with licenses for two servers - finally making it possible to have redundant domain controllers even in a Small Business setup without paying for full server licenses.

The first half of the setup is similar to what you know from Windows Server 2008 and Windows Vista - you boot, select the disk, have the chance to enter a product key, and finally start the installation. After that, the WIM image is expanded to the harddrive. The machine reboots after installation, and this is where things get different.

After booting, you’ll land in “Install Windows Small Business Server 2008″ Wizard. This can be mostly automated using an Answer file, which is mandatory when migrating from earlier versions. I will check that out later and proceed with a simple installation without using an Answer file.

I get nagged by a “Insufficient Hardware Screen”, reminding me that my (virtual) machine only has 2GB RAM. After acknowledging the warning, i can setup my date and time. I choose the CEST timezone, and move onwards.

Next, a screen confronts me with the fact that i don’t have a NIC - which is true. The machine is running on Hyper-V RC1, and i wasn’t able to install the integration components yet. Luckily, there is a “Browse” Button, where i can launch the Integration Services setup. Installation of the Integration Components worked fine, the machine rebooted. I hope Microsoft packs the Hyper-V RTM bits into SBS RTM. This would make it easier to install it into a VM, but as you can see, it’s not much of a hassle.

I was back at the beginning, at the start of the SBS Wizard. Luckily, i was now able to use the mouse after installing the Hyper-V IC. Next, i get an Update Dialog, asking me if i want to update my server. I choose yes and have to wait.

Next, i was asked to enter my company information. Next, i was able to name my server and the NETBIOS name of the Domain. I was not able to choose a DNS Name for the Domain (This is only possible if using an Answer File). Interestingly, Dashes “-” were not accepted as part of the server name. I wonder why - our production setup uses dashes extensively in server names, and so does Microsoft (judging from their Mail headers).

Then i was asked to create an an administrative account - a good idea. The “Administrator” account shouldn’t be used in a production setup, instead each user with administrative rights should have their own account. SBS enforces this - a very good idea.

After confirming Server name, Domain name and Company name, the installation continued on it’s own. This took a good amount of time, during which the server restarted several times - of course completely unattended. No need to play disc jockey or logon - much better than SBS 2003.

After the installation, i was greeted with a screen that told me that it was unable to install some critical updates. Clicking on that bar revealed an IE7 404. I checked the IP configuration - the server was configured to use 192.168.0.2, and didn’t have a DHCP server installed. There was no default gateway set yet.

Next, i launched the “Connect to the Internet Wizard” which told me that i was already running a DHCP server - which makes sense. After choosing “Postpone”, the Wizard aborted. That wasn’t quite what i was hoping for.

I shut down the VM and reconfigured it to use a private LAN. That way, it wouldn’t have a connection to the internet, but it wouldn’t have to deal with a DHCP Server either. But SBS didn’t like that either - it wanted a router. So i setup a second VM running IPcop (which works flawlessly on Hyper-V using Legacy NICs and a small virtual hard drive).

It was interesting to see using “tcpdump” what SBS did under the covers to detect the router. ARP scanning, IPv6 Discovery, Everything. This seems rather well designed. It was sucessfully able to detect my IPcop VM which didn’t have a DHCP server.

Next, i started the wizard to enable my domain name. It seems that SBS will be able to do some of these things automatically if you live in the US. Here of course we have to do things manually.

So far i don’t like that SBS tells me very very few technical details. But this might be because Microsoft somehow thinks that a Small Business Owner will setup SBS on their own (which just seems a horribly stupid design decision).

Next, it told me that i couldn’t configure my Internet Router properly (my IPcop instance didn’t have UPNP support enabled). It’s interesting to see that it wants to forward port 25 to the server. It looks like the POP3 Connector was finally killed off for good. That’s very good to hear!

I also had to configure outbound email properly, with the ability to configure a smarthost or use direct sending. There is also a wizard to easily create a properly signed official SSL certificate - nicely done and will surely improve the security of the many SBS setups that are out there.

SBS 2008 also ships with OneCare for Servers already preinstalled. You can just activate it with a few clicks. I don’t see this very positively - I’ve made a few bad experiences with ForeFront Client Security, which OneCare is based on. We’ve been using McAfee for the past. So in the future for SBS setups we will have to either remove OneCare from the SBS, or deal with having multiple virus scanners on the network (a management nightmare).

Another interesting tidbit is that UAC is enabled in approval mode, just like on standard Windows Server 2008 installations when not using the Administrator account. This is annoying, IMHO. I don’t have a problem with UAC on my desktop because i usually use my desktop to work and not change settings - but when i’m logged onto a server, i want to change settings all the time.

That’s it for the first impressions. I will have a closer look at SBS 2008 over the following days and will keep you all updated.

Pictures are here:

Mergers and acquisitions have always been a big part of what the important people do.

Of course, theoretically, that wouldn’t be my problem. However, one wouldn’t believe on how much work it is to get rid of an old name throughout a whole network.

First comes Active Directory - it uses DNS as a primary means of identification. Renaming an Active Directory domain is purely theoretical, e.G. it doesn’t work and it’s not supported if you’re running Exchange 2007.

Then there’s numerous other stuff that depends on DNS, names and everything. But in the end, i did my part of this deal. It has been an interested time and now i really wonder how big companies like Swissair handled their renames - or are they still running their infrastructure under the old name?

My employer is now called Acommit AG.

Of course, the whole rename proved to be a really good argument to buy new servers and upgrade straight to Windows Server 2008 - that means on my side the renaming thing has worked rather well. I still don’t know a lot about the company we bought (Futura Retail Solution GmbH), but their main market is selling POS related products.

If you need the full details, you can get them here:

Futura Retail Solution GmbH
Acommit AG (soon to be updated)

Long story shorts: Modify the hosts file, remove the IPv6 localhost (::1) and then add hosts entries for your server. I would recommend against disabling IPv6 on the Exchange server, as this is probably not a recommended or supported configuration.

The root cause is that Outlook 2007 can’t contact a DC/Domain Controller using RPC over HTTP/Outlook Anywhere when used on Windows Server 2008.

Also note that NTLM Authentification for Outlook 2007/Outlook Anywhere is broken on Windows Server 2008.