Lukas Beeler’s IT Blog

The IBM POWER Model 520 9407-M15 or in short the M15 is a one core, 4.2 Ghz POWER6 server.

It’s the successor of several System p systems (which i know nothing about), and of the System i Model 515 (9407-515). As such, it targets small businesses.

Yesterday i received the first M15, to be installed in for our SaaS (Software as a Service, the IBM Slang for Application Service Provider) Project. This is the first standalone POWER System that i got my hands on, but i’ve already tested running IBM i on Blades.

The M15 is a standard 19″ 4U Server at half depth that can hold dual power supplies for redundant power, has a variety of expansions slots and supports up to 6 internal 3.5″ SAS Disks. The integrated SAS RAID Controller has support for a battery backed write cache. You can also install several PCI-X and PCI-E cards. The system has 2 8x PCI-E Slots, one 16x PCI-E Slot and two PCI-X slots.

In this case, the system came with a HMC - a normal System x3550 configured with a single 320GB SATA Disk drive. Interestingly, we had ordered the same HMC (7310-CR4) half a year ago. Back then, it shipped with a 80GB SATA Drive and an external modem. This unit shipped with increased capacity in the SATA Disk drive, and an internal Modem. Though i have no idea why anyone would still use the modem.

The HMC isn’t very interesting from a hardware perspective either, so the focus is purely on the M15.

The first view on the front shows a new green bar, that symbolizes POWER6. I think it could use a bit of improvement, doesn’t look that nice. Much more interesting is that the control panel has essentially vanished. Like the light path diagnostics model in the System x, it has to be pulled out from the System to be of any use. This was probably done to save space.

The new control panel isn’t an improvement, unfortunately. Of course on systems with the a HMC attached you don’t really need it anymore - but most of the Systems we are going to ship will not have a HMC. The buttons are hard to use and hard to reach - it’ll be interesting doing procedures like 65+21 on those. This isn’t a deal breaker - but from IBM i expect them to get even details as this right - all in all, this isn’t a 1k Dell Server - it’s a 20k High-End IBM Server.

Much better in my opinion is that with the POWER Systems, the IBM i has finally moved to the year 2008 in regard to IO technologies. PCI-E and SAS is finally here. What i do not understand is why the M15 uses 3.5″ SAS Drives, and not 2.5″ SAS Drives. This would allow to fit more arms into the same chassis. e.G. the 2U System x3650 ships with the possibility to install up to 8 2.5″ arms. A 4U machine could have up to 16 arms - without needing more space.

The power supplies have been moved to the front of the unit, similar to the PCI Expansion units. I like it - the new power supplies are bit smaller than the older ones, and replacement is easier, thanks to them sitting in the front.

>The machine internals still seem a bit sketchy to me. The M15 wastes a lot of space that is used for the second CPU in larger machines. But it also has a completely new fan design, with four centrifugal fans in the back of the machine.

The new fan hot plug mechanism is very sturdy, and is comparable to the high quality fan design used in the System x3650. This was one of the biggest downfalls of the POWER5/5+ 520/515/525 hardware platforms that has been fixed in the new hardware. RAM accessibility still isn’t optimal in my opinion - you’ll still need to remove the fans to access the memory. IBM has better solutions for this - just look at the x3550 and x3650.

The fans are very, very loud. The unit we have here is a rack unit and the fact that there is no conversion option like for the POWER5 models might mean that the tower and rack units have different acoustic configurations or different dampeners. While loudness is a complete non-issue in a server room, smaller customers sometimes have the machine in their office. As soon as i get the first tower model i’ll write about their loudness level.

One of the things that’s completely new to the POWER platform is the HEA - the host ethernet adapter. It allows to share a physical NIC with other partitions - that’s a very good feature, but i wasn’t able to play with it yet - this machine is not partitioned.

The BBWC in this machine is now hot pluggable. It’s great to see this, but in my opinion it wasn’t really necessary. There’s a reminder directly on top of it that you need to set the disk cache into an error state before replacing it - and that reminder is very important. If you don’t pay attention to it, you might have to make an unexpected test of your DR strategy.

The expansion and console capabilities of the M15 are artificially restricted - you can not have a PCI-Expansion Unit (no HSL/12X Ports) and you cannot have IOPs in the base unit. The conclusion: No Twinax, No U320 Tapedrives.

Especially the Twinax bit is, in my opinion, a good move. It will force customers stuck in the AS/400 days to get current with all their other hardware like printers. On the other hand, it might also cause those customers to stay stuck with their model 270 or model 800. As for me personally, i’ll have to deal less with Twinax - which has to be a good thing.

The Thin Console, a very good option for Small Businesses, is gone (because HP bought Neoware). With Twinax also gone you now have the choice to either get a HMC for 6k or get a Windows PC and use LAN Console. Both options aren’t really what a Small Business needs - a console that “just works”. The TC and Twinax console fit those criteria. The LAN console has issues on their own (The whole Systam i Access package is … aging) and the HMC requires a boatload of expertise that SMB operators just don’t have. We will go with the LAN console, mainly due to the HMC pricing, but i’m not really content with the console situation on the new systems.

This picture shows the rear of the unit, as you can see the cabling in my lab environment is always top notch. The HEA offers 4 ports by default, a bit much for a standalone system. I’ve implemented a Virtual IP Address setup to create redundant network connections. Not as cool as native Teaming/Bonding support, but it works well enough.

In general, the new model has improved several things on the old hardware, left one or two things in the same state, and has two new issues (the control panel, noise level). All in all, a good solid deal of hardware.

Questions? Comments?

Customer is running two Terminal Servers on Windows 2000 Server. 32bit. 4GB of RAM.

Recently upgraded to Symantec Endpoint Protection 11, around 1 Month ago. A week ago, the customer complained that one of the Terminal Servers crashed constantly, requiring a reboot to recover.

Quick investigation showed that the machine was running out of paged pool.

Event ID 2020
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Description:
The server was unable to allocate from the system paged pool because the pool was empty.

I’m not proficient with Terminal Servers or Windows 2000, but debugging this issue was mostly similar to what you do when debugging pool issues on Windows Server 2003. First you need to enable Pool Tagging, which is enabled by default on Windows Server 2003 but not on Windows 2000. KB177415 explains how.

After that, install the Windows 2000 Support Tools, and run poolmon /p /p /b.

In my case, the output looked like this:

The limit for Windows 2000 Terminal Servers is 160 MB. As you can see, the machine here is idle and without any users on it. And we’re already at 132MB utilisation.

There are two culprits: “CM” and “SavE”. The Pooltag “SavE” is the Symantec Endpoint Protection Virus Scanner Driver. It clocks in at 50MB. The other Pooltag “CM” stands for “Configuration Manager”, and is the registry. It is 67MB big.

This is not normal - the other Terminal Server, the CM tag is a lot smaller, only 35MB. The “SavE” tag is still 50MB. This explains why the other TS does not have the same problems as this one. But we don’t know why one registry is so much bigger than the other.

This can be found out by using the dureg.exe tool, which can help us resolve the issue.

As you can see from the picture above, the enlarged registry is also caused by Symantec.

C:\Programme\Resource Kit>dureg /lm “SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Quarantine”
Size of HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\Quarantine: 26111494

Clocks in at 26MB. The Quarantine key contained around 20′000 subkeys, each with a simple number below. Each was about the same .doc file.

After deleting the Quarantine key, the CM pooltag went down from 67MB to 35MB - just like the other TS.

The next step was obvious. Remove Symantec Endpoint Protection which something that doesn’t suck as bad: McAfee AntiVirus Enterprise. I downloaded an Evaluation Version, and installed it.

And the results were obvious:

Do you see the pooltag “NAI0″ somewhere in this list? I don’t. It’s there, but somewhere around Page 400, and surely not eating away 50MB of my paged pool.

So if you have problems with your machines running out of paged pool, frequently showing Event 2020 with Source Srv, check the registry size and replace Symantec Endpoint Protection with something that doesn’t suck that much.

Disclaimer: These are my personal experiences, not a “How to i on Blade”. If you’re looking for decent i on Blade documentation, look at the i on Blade Readme.

Setting up the hardware

Finally, after almost two months i was able to get my hands on a SAS cable with the correct pinout to connect the TS3100 library to the SAS Connectivity Module. After plugging in the cable, both link lights went up. The TS3100 was connected - physically at least.

I logged into the Storage Configuration Manager, and dared to assign the HS21 Windows Blade and the JS12 Blade to the external SAS Port. When looking at the Windows Device Manager, it immediately recognized the Tape Drive, but didn’t recognize the media changer. No bummer, i haven’t installed BackupExec yet.

It’s important to notice that the IBM i OS will not see your tape drive. You cannot connect the tape drive to the IBM i OS, only to VIOS. This means that you always will do a disk to disk backup to the VIOS partition, and then use VIOS to save the D2D Image to tape. Restoring works the other way around - you restore from tape to VIOS disk, then load that disk into IBM i, and then run your restore or even IPL from the virtual optical medium.

This is not optimal, as this means that you cannot use BRMS to manage media (you can still use it for saving, though). It also adds another layer of indirection that makes automating backups more difficult. Another important point is that other i machines will not be able to read the VIOS created tapes, and that your i Blade will not be able to read tapes created by standalone POWER machines running IBM i.

Configuring VIOS

So i went to logon VIOS, running on the JS12 blade. I ran “lsdev | grep rmt0″, but apparently there was no tape drive to be seen. I ran “cfgdev”, to let the operating system configure devices, but that wasn’t met with success either.

This time, i chose the easy way out. I just rebooted the entire blade, and finally, the tapedrive showed up:

$ lsdev -dev rmt0 -attr
attribute      value            description                               user_settable

block_size     262144           BLOCK size (0=variable length)            True
delay          45               Set delay after a FAILED command          True
density_set_1  0                DENSITY setting #1                        True
density_set_2  0                DENSITY setting #2                        True
extfm          yes              Use EXTENDED file marks                   True
mode           yes              Use DEVICE BUFFERS during writes          True
res_support    no               RESERVE/RELEASE support                   True
ret_error      no               RETURN error on tape change or reset      True
rwtimeout      144              Set timeout for the READ or WRITE command True
var_block_size 0                BLOCK SIZE for variable length support    True
ww_id          5000e1111c878001 World Wide Identifier                     False

Okay. So as the TS3100 is a LTO4 Tape Library, the next step obviously was to load a tape into the tape drive. Now this can be done through the Web Interface or the Control Panel of the library, but that’s not the way you want to go during day to day backup - media moving should be handled by the backup software (called BRMS on IBM i).

But alas, this is not as easy on VIOS. Basically, you can get an AIX root shell on VIOS by typing “oem_setup_env” - and then there are several AIX commands to manage a tape library.

There was only one problem:

# mtlib
ksh: mtlib:  not found.
# tapeutil
ksh: tapeutil:  not found.

They’re not there. A quick google search revealed nothing. I didn’t know how IBM thought how we should use a tape library. In sequential mode? Or is there some way to manage a tape library in VIOS? If you know, tell me!

So, the next step was to create writable optical media, so i could start with creating a Save 21 of the system. This seemed as a sane first step.

Initializing the media on the i side

First, i needed to create a virtual optical volume in a volume library on VIOS - i already had the volume library from the IBM i installation, so all i needed to do was to add a writable optical volume. The system used about 40GB DASD, so i created a virtual optical device with a size of 80GB to leave room for future growth. This process took around 2 hours, probably because VIOS pre-blanked the file (it effectively used up 80GB on disk). A rate of 0.6GB/min. During this operation, the VIOS webinterface grinded to halt (didn’t respond), but SSH was still available and responded very slowly to each command. I suppose this is some issue with the disk controller, either the driver or firmware.

According to vmstat, most of the time is spent in system or IO wait context:

kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 1  1 203749  6107   0   0   0 2485 3279   0 150 29483 6542  6  9 77  7  0.18  17.5
 1  1 203732  6167   0   0   0 2464 2869   0 153 30479 6333  7  9 74 10  0.18  17.6

The next step was to load the newly created virtual media into the virtual optical drive that was attached to the IBM i partition. This was rather easy to do, just click through the web interface.

Now, we need to initialize the optical volume in IBM i OS:

INZOPT NEWVOL(’Save21′)
DEV(OPT01)
CHECK(*NO)
TYPE(*PRIMARY)

This finished rather quickly, and i then started the Save 21.

The Save 21

GO SAVE/21. The performance wasn’t much better, though. the vmstat looked the same as above, indicating the same problem.

Further debugging using iostat revealed that a volume group is not a RAID1 array - but nonetheless, the disk subsystem is behaving oddly:

hdisk0, hdisk1: VIOS installation - part of it is mirrored VIOS, other part is volume group that is spanned across these two disks
hdisk2-5: SCSI passthrough to the IBM i

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        614.0                0.0   7.7   72.2     20.1   0.1    8.3

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           1.0       8.0       2.0          0         8
hdisk1          85.0     20036.0     179.0      10080      9956
hdisk2           4.0     926.0      50.0        922         4
hdisk3          14.0     4481.0      98.0       4477         4
hdisk4          13.0     3730.0      96.0       3726         4
hdisk5           3.0     652.0      36.0        648         4

This is pure sequential IO - why is it reading from the disk? A similar picture was seen throughout the whole backup - even when backing up image catalogs from the IFS. hdisk1 consistently showed strong write activity. No idea why.

My AIX skills are weak, and i didn’t know a way to see on which files the write IO happened - however it’s important to know that read and write always showed the same numbers. To me, this looks like a problem - either in my config, firmware levels, or even a problem on IBMs side.

Either way, the Save 21 completed in 45 minutes. At around 40GB, this brings us to 0.9GB/min.

Backing up the virtual image to tape

The next step is to backup to our LTO4 tape.

Here’s how the backup itself looks:

# find /var/vio/VMLibrary/D2D_1 -print | backup -ivqf /dev/rmt0 -b 512
Backing up to /dev/rmt0.
Cluster 262144 bytes (512 blocks).
Volume 1 on /dev/rmt0
Backup finished on Mon Jul 21 20:45:18 CEST 2008; there are 167772672 blocks on 1 volumes.

And the sequential IO performance is much more reasonable:

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        615.0               11.2  29.0   58.2      1.6   0.4   40.9

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           0.0       0.0       0.0          0         0
hdisk1          99.0     81920.0     160.0      81920         0
hdisk2          17.0     2250.0      29.0          0      2250
hdisk3          12.0     2236.0      28.0          0      2236
hdisk4          12.0     2385.0      30.0          0      2385
hdisk5          16.0     2250.0      29.0          0      2250

That’s roughly 5GB per Minute. A very decent performance.

Interoperability

Now, this is where it gets interesting. Attached to the BladeCenter S, we have a TS3100 with a single drive, in the BladeCenter S we have three Intel Blades running Windows Server 2008 and one POWER Blade running IBM i.

We need to back up all this to the TS3100 - on the Windows Side, i’ll be using BackupExec 12, on the i Side VIOS. How do i make sure that the tape drive can be used from both sides, without to much interaction?

The SAS Connectivity Module can attach the same port to multiple Blades. So installed BackupExec on one of the Windows Blades, just to see how interoperability would work out.

I ran a backup & restore on another tape, from BackupExec. This worked fine. The next step was loading the tape from the i Save back, and then run a test restore from that. Unfortunately, i couldn’t use BackupExec to just move the tape in the drive, so i had to use the TS3100 Web Interface again.

I looked at the tape drive from VIOS, which also seemed okay. I also saw how much space was used on the VIOS tape after checking it up with BackupExec (this is stored on a small RFID Chip on the Tape itself). But the real test was yet to come:

Restoring from Tape

I started the restore from tape.

# restore -xvqf /dev/rmt0 -b 512 /var/vio/VMLibrary/D2D_1
New volume on /dev/rmt0:
Cluster size is 262144 bytes (512 blocks).
The volume number is 1.
The backup date is: Mon Jul 21 20:24:04 CEST 2008
Files are backed up by name.
The user is root.
x  85899345920 /var/vio/VMLibrary/D2D_1
The total size is 85899345920 bytes.
The number of restored files is 1.

iostat also told me that the performance on restore was bit worse than when backing up:

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        611.0               14.7  85.0    0.0      0.2   1.0  101.9

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           0.0       0.0       0.0          0         0
hdisk1         100.0     48216.0      68.0          0     48216
hdisk2           0.0       0.0       0.0          0         0
hdisk3           0.0       0.0       0.0          0         0
hdisk4           0.0       0.0       0.0          0         0
hdisk5           0.0       0.0       0.0          0         0

So after having restored the image file to VIOS, i could IPL from it and run a complete system restore. Nice. I didn’t want to scratch my whole setup and wait until a full restore, so i tried something simpler.

In the end, it turned out to be around 2GB/min. This was a lot faster than the creation of the file, which seems really odd to me.

Running a Test restore on the IBM i side

After restoring the file in the VMLibrary, it automatically appeared again in VIOS. I only had to mount it to my IBM i partition. This could easily be done through the VIOS web interface.

I ran a simple restore:

RSTLIB SAVLIB(AVNEDIAS) DEV(OPT01)

With simple results:

12 Objekt(e) von AVNEDIAS nach AVNEDIAS zurückgespeichert.

I also tried IPLing from the virtual optical media, which brought me into the limited paging DST. Nice!

Is this good?

AS you can see, this is not your fathers AS/400. This is a POWER Blade running IBM i. You’ll need to learn a bit about VIOS and AIX in order to make any sense on how this whole stuff works. But it’s not rocket science - i only know a bit about Linux, and was able to figure out the tasks i needed to do.

But now, how should one run this in production?

The current configuration i have seems unsuitable to production to me.

• VIOS/AIX can’t handle the tape library in random mode. This is a big letdown.
• Backing up to tape and restoring seems very, umm, basic to me
• The fact that you are using virtual optical devices, with no ability on the i side to change media, makes a usuable backup procedure hard to implement
• Automation on the VIOS side could be implemented by the i running the ssh client in command execution mode (similar to how this is used with the HMC
• Integration between the Windows and VIOS seems cumbersome

Solutions

Two Half-Height Tapedrives in a partitioned library

The TS3100 can be partitioned, and we could install two half-height LTO4 tapes. Reserving 12 Slots for Windows in Random Mode, and 12 Slots for VIOS in Sequential Mode. This would work, but there’s one huge downside: Price

Backing up on Windows only

Instead of using VIOS and clumsy hacks to get a halfway decent functionality, you can use Windows to backup everything. Save 21 and Systems Saves would run through VIOS in order to create bootable media, and then retrieved on the Windows side using SFTP.

For daily backups, we can use savefiles directly on the i, which is probably easier to deal with for most IBM i admins. These can be retrieved from the Windows side using FTP/TLS. The downside: If you have i and Windows people that work well together, i don’t see much of an issue. But if not, you got a big mess in your hands.

SAS Passthrough to IBM i

Unfortunately, this does option not exist yet. This is something that IBM should work on intensively. It will allow i admins to use well established backup processes with full library integration using BRMS.

Conclusions

Backing up the IBM i on Blade isn’t exactly easier than backing up a standalone POWER machine. In fact, it’s more difficult and requires additional skills.

Before buying a JS12 blade running IBM i, make sure that you think your disaster recovery strategy through completely. Your business partner should be able to help you with this.

Planning is crucial - Backup & Restore on the blade is different, and you’ll need to deal with VIOS when creating a procedure for fully automated Save 21 backups.

Any questions? What do you think about the situation? Want me to test something for you? Just leave a comment!

I also created a “i on Blade” category. Look at it if you want to see all my posts about this subject.

The i Blade is up and running, and i’ve received quite a bit of feedback on the Installing the JS12 Blade post.

In the meantime i wasn’t just wasting my time on trivial things such as getting actual customer work done, but also playing a bit further with the JS12 Blade.

I’ve installed the software my company produces (DIAS-iS) on the JS12 blade, and ran a few very unscientific benchmarks. But first let’s talk about the disk situation in with IBM i on a JS12 blade in a BladeCenter S (i really like those convoluted product names).

Managing disks under IBM i on a JS12 Blade

As i found out the hard way during the initial bladecenter setup, the JS12 blade only supports SAS disks, and can cause issues if you have SATA disks zoned to it.

There are a few important considerations when thinking about the IBM i/JS12/BladeCenter S combination: First off, disks are directly attached to VIOS, and then virtualized by VIOS for the IBM i as SCSI disks. It’s important to note here that you do not have any (supported) options of RAIDing the disks before the IBM i sees them. So all disks are mapped through 1:1 to the IBM i OS, and then mirrored using IBM i mirrored protection. This is entirely different from the approach you would use in a BladeCenter H with a FC attached SAN.

Just to be clear: There is no cache on the JS12 and there is no way to use any disk protection except IBM i mirrored protection. You can’t use RAID5, RAID6 or hotspares. You can’t VIOS mirrored volume groups either, because it’s unsupported.

I’m thinking about removing one of the disks from the BladeCenter in order to test how recovery from a disk failure would look like, but i’m afraid of wasting a lot of work that i’ve already invested in this system - i’ll try this shortly before i have to give everything back.

I’m not sure what the virtualization by VIOS exactly entails, but i would assume it’s fairly similar to what Hyper-V/ESX do when you create “Passthrough disks”. This probably means that things like Predictive Failure Analysis (PFA) will probably not work.

Another, rather obvious, drawback is that you cannot install any expansion cards (well, there is the odd one you can install into the blade). But it also means there is no Twinax, no SNA directly over Ethernet, no Modems, etc. Not a big issue for us, as we’re urging our customers to stay current on technology, but not everyone is an IBM i shop - there are still lots of AS/400 shops out there.

If you access the System i Navigators SST/Disk management function, it will not be able to help you with disk locations. I haven’t found out on how to call disk locations in IVM/VIOS, but then again i don’t really know much about IVM/VIOS.

Installing DIAS-iS on the JS12 Blade

I’ve installed our software without a hitch, and loaded our 30GB Test/Benchmarking database on it. I ran several benchmarks, and the JS12 with it’s four SAS 15kRPM 147GB Arms in a mirrored configuration and one core and 13GB RAM in the IBM i LPAR was a bit slower (less than 5%) than our System i515 with four U320 15kRPM 70GB Arms in a RAID5 configuration and 3.5 GB in the IBM i LPAR.

Unfortunately, i do not have a M15 to pit against the JS12, as these two would be using comparable technology. It would also be interesting to see an M15 with four 147GB SAS disks in a mirrored configuration to compare the systems 1:1, especially regarding disk performance.

Next steps to go?

What’s next? Well, Backup obviously. If you’ve read the i on Blade manual you’ll see that saving to tape will be interesting to say the least. I already have a TS3100 ready to go, but i’m currently missing a SAS for attaching it. As soon as i have the cable, expect a big post about saving and restoring.

Questions? Suggestions? Any specific questions about i on Blade? Want me to test something for you?

Leave a comment or drop me a mail. I’ll be happy to help.

I noticed something interesting when installing the IBM license key on the JS12 blade running IBM i V6R1. I received Message CPF9E2D when trying to install one of the 5761-SS1 license key. Turns out this one controls the number of cores licensed in the machine.

Luckily, this was easy to remedy. Just access IVM and remove one of the cores assigned to the LPAR running IBM i.

IBM licenses it Software based on numbers of Core, not number of Sockets like most other vendors do. This is important to note, because i though the message was wrong as the blade only has a single CPU.

If you came to this post for detailed instructions on how to setup IBM i on a blade, read this offical IBM i on Blade document. I’ve wrote about my personal experiences, not detailed instructions.

Just two hours ago, i’ve received a shipment of four 3.5″ 147GB 15kRPM SAS Disks. I installed them into the BladeCenter S immediately.

I used SCM to assign the disks in the DSM to the JS12, and then booted the blade.

VIOS was already installed. So all i had to do was to create a new partition.

After creating the partition, i didn’t IPL it just yet. I needed System i Access in order to provide a console. The Operations Console part of System i Access is not supported on Server versions of Windows, so i couldn’t install it on one of the other blades running Windows Server 2008. At least not directly. So i installed the Hyper-V Role on one of the blades, and installed Windows XP and System i Access on it.

I then IPLed the partition and a minute later i was standing there with a lit attention light. I forgot about the CD drive. Bummer. I assigned the media tray to the JS12 blade, but it couldn’t see the CD drive. This must’ve worked before, because i installed VIOS using a CD. I restarted the JS12, but that wasn’t helping. Still no CD drive that i could assign to a partition. Didn’t find much on the web about this problem either, so i decided to use virtual media to install the operating system.

I logged into VIOS using SSH, downloaded the I_BASE_01 CD Image from our production system using FTP, and imported it into VIOS’s media library. I activated the I_BASE_01 CD Image, and booted. I also enabled the operations console connection (which is fairly straightforward, with just in this case the first IBM i instance having the partition ID 2).

After 10 minutes, the signon screen for the operations console finally appeared. That was kind of a Heureka! moment for me, altough i didn’t really do that much stuff yet.

I chose to install the LIC, and i was presented with a screen that i haven’t seen before - i was able to select the which disk i want to be the load source.

After that, the system started to initialize the hard drive. This was really slow on my system, taking around 5 hours for a single 147GB 15kRPM drive. I hope this isn’t indicative of the IO speed we will see when the IBM i OS is running.

While waiting for the formatting to complete, i tried to find a way on how to turn off the attention light that was lit because of my earlier mistake when trying to boot the partition. There is a detailed IBM document about turning off the attention light using IVM/VIOS. It’s a simple command: chled -r sa -t virtualsys -o off

As you can see from the screenshot (took during the middle of the run), it took quite long. In fact, it even exceeded the three hours it estimated and took 4.5 hours. I have an issue with that - the Intel blades do not need that much for initiating a RAID1 set, or NTFS formatting the disks. Even though they’re using slower 500GB SATA disks. It’s just leaving a bad impression for no reason. And it’s also an issue with disaster recovery.

Installing the LIC had a more reasonable speed, took 2 minutes. After the system IPLd again, i was able to add the three other disks. Adding the disks proceeded at a much more reasonable speed, but then it hung at 99%. After two hours, the system was still stuck at 99%. At that time, I went to bed, hoping the system would be finished in the morning.

And it really was finished in the morning. The next step was to start mirrored protection. It even complained that i was running virtual disks, and a failure of VIOS would lead to the system crashing anyway. I proceeded. As always. the first part was pretty quick, and i proceeded with the OS installation.

As the LIC started, initialization of mirroring began. The first time estimate was fifteen minutes. The next one was 6 hours, 10 minutes, then 8 hours. But then i jumped back to 5 hours. I left for a customer, and looked at the status occasionally. I took somewhere between 3 and 4 hours to complete.

Next, i had to change the virtual media in order to allow the OS installation to proceed further. It’s important to know that this has to be done in the partition configuration, not in the virtual media tab. And that you’ll need to acknowledge the partition change in order to make the media change active (the AJAXy web interface doesn’t make this entirely clear).

After that, the IBM i installation started and proceeded at an acceptable speed. In around an hour, the basic operating system was installed.

After that, installation of the licensed programs started. It halted after just half an hour and telling me i had a screen error (MSG CPF3D92). I suspected a problem with the operations console, restarted the XP machine running OpsCon, and retried the installation (with just the base system). The problem happened again. This seemed odd.

Having no idea on how to proceed further, i retried again. This time it worked(?). I figured it was an OpsConsole problem, probably related to the fact that the machine running OpsCon was virtualized. I quickly installed the TCP/IP utilities, IPLd the system and installed the remaining programs using a 5250/Telnet connection.

While the installation happened, i used an additional session to explore the system. The disks where shown to the system as virtual disks, similar to SAN attached disks. But one of the more interesting parts was looking at the Hardware Service Manager in SST/DST - it was completely empty, and didn’t contain any hardware. For me, this was a moment that was quite indicative of the whole experience - i on Blade is not “AS/400 in Blade Form”. It’s a completely new environment that you’ll need to learn to deal with. You got another layer of indirection (VIOS) with it’s own platform (AIX), plus you have the blade management in itself.

The whole setup took me roughly 24 hours (i started a day ago at 16:00). Of course, the system wasn’t always busy because i didn’t give him any work, but it’s worth to note that setting up a JS12 blade takes considerably longer than setting up a model 515 or M15.

I will now continue setting up our ERP application and make further tests with the hardware. If you have any requests for screenshots or want me to test something out, tell me!

I’m not exactly what one would call an “Enterprise” Admin - so i don’t really know all that much about WMI.

We first started our internal virtualization stuff when both VMware GSX and Virtual Server 2005 still cost money. So we used VS2005 because we could get it for free since we were in the Microsoft Partner Program.

So, with the release of Hyper-V we finally had a chance to move to a more robust and faster virtualization solution - however, not everything has improved with Hyper-V, for example delegating permissions which was easy in VS2005 has now become much more complex. Probably because Microsoft wants to sell SCVMM 2008 that will automate a lot of this.

We have a few development VMs that are used for QA purposes by our development team - and we just have a single machine running Hyper-V. So i want to delegate a few of the VMs to the development team, without them being able to manage the Hyper-V server or virtual machines that do not belong to the development team.

I’ve found an excellent resource regarding setting up remote management for Hyper-V from John Howard. He has an excellent 5 Part series on how to enable remote management.

Part 1 Part 2 Part 3 Part 4 Part 5

What is not described in these links is how to delegate specific VMs. For doing this, you’ll need a script from Andrzej.

Hyper-V Azman Scope Scripts

Here’s a basic rundown of the general steps you’ll need to do:

• Create an appropriate Active Directory group for the users you want to give access to. If necessary, nest the groups according to your organizations group strategy
• The following two steps are detailed in Part IV from John Howard
  • Add to the group to the local “Distributed COM Users” group on the Hyper-V host
  • Grant the group permissions on the Root\CIMV2 and Root\Virtualization WMI Namespaces
• For detailed instructions for these three steps, see below.
  • Run azman.msc and create a new scope
  • Use the SetScope VBS script to assign the VM to scopes.
  • Run azman.msc and delegate appropriate permissions to Windows Groups using newly created scope

Creating scopes in AzMan and assigning VMs to scopes

First, you’ll need to start azman.msc and open to following Authorization Store: C:\ProgramData\Microsoft\Windows\Hyper-V\InitialStore.xml

Then, you’ll need to right click “IntialStore.xml” and choose “New Scope”. In my case, i named the new scope “Dev”.

Next, you will need to create a role in the top-level of the authorization store. This role is needed so that the Hyper-V Management tool can even connect. I called mine “View Only”, as it does not grant any specific permissions. It should look like this:

You’ll also need to add the Windows Group to this azman role in order for it to be of any use:

Next, we need to create a role that grants the necessary VM management skills to the Dev scope. It should look like this:

You’ll also need to add a Windows Group to this role.

After you’ve come so far, we will need to assign the VMs to the newly created scope. You can find the scripts here: Andrzej’s Hyper-V Scripts.

Assigning a VM to a scope is simple.

For example, if you want to assign the VM “dev-hdi-xp-01″ to the scope “Dev”, use this command.

setscope.vbs dev-hdi-xp-01 Dev

There will be three popup Windows - the first two don’t matter, and the latter will contain a single number. If the number is 4096 (or anything else), it failed. If the number is “0″, it succeeded.

You can verify scope membership using getscope.vbs

getscope.vbs dev-hdi-xp-01

The result should look like this:

If my posting is entirely correct, and you followed it correctly, the end result should look like this:

Here, we’re logged on as an admin. All VM’s are visible:

Here, we’re logged on as a normal user. It does not have any special privileges on the Hyper-V box, except the WMI / DCOM and AzMan changes. You’ll only see the two Development VMs.

So, this is quite a bit more complex than VS2005. But also a lot more cool.

I hope there are no mistakes in this post. If you find any, please tell me. If you found this post helpful, tell me too. Thanks for reading!