First real world experiences with IBM’s x3650 M2
So, IBM’s x3650 M2 have been out for some time and in the meantime i’ve deployed three of them – two with SBS 2008, and one with Windows Server 2008. No pictures, since the camera in the office is broken.
The x3650 M2 comes with a new IMM (Integrated Management Module) that replaces both the BMC and the RSA II Slimline. In order to get remote KVM capability, a physical licensing key must be installed into the server. Standard features like remote power on/off are available without the licensing key – which is more expensive than the physical RSA II slimline adapter. With the IMM also comes UEFI, as a replacement to the aging BIOS.
The x3650 M2 also gets rid of the Adaptec ServeRAID 8k controllers, and introduces us to the ServeRAID 10 series manufactured by LSI. The ServeRAID 8k series have been plagued by several extremely heavy bugs that never caused me loss of data (but several other people), but nevertheless cost me a lot of my nerves.
The two standard RAID controllers in the x3650 M2 are the ServeRAID BR10i, which is the baseline controller without BBWC and without support for RAID5. The ServeRAID MR10i is the better model, which comes with 256MB BBWC and support for RAID5/RAID6. Unlike the ServeRAID 8k/8k-l story, these are entirely standalone controllers, that are located in a special daughterboard position with a standard PCI-e x8 interface.
Both controllers support only 8 drives – in order to get 12 drives, you need a special enabler kit that comes with a SAS expander hard and several of other stuff that doesn’t look all that trivial. I’ve used such a kit, and so i can’t comment on how it works exactly.
The power supplies have gotten a lot smaller, the server seems to look a lot more organized, the 2.5″ SAS HDDs are no longer as finicky as they were in the x3650 and now fit very well into their slots, the Lightpath diagnostics panel now looks like it belongs to an expensive server and locks into place securely.
Of course, all the usual changes that come with the new Nehalem based Xeons, triple channel DDR-3 memory, both processors needed for using more than 8 slots, using to many memory modules will downgrade the speed, etc.
So much for the general rundown – now for my assorted observations:
- The ServeRAID BR10i seems to be a slightly newer variant than the SAS RAID Controllers found in the x3250 and x3250 M2. The configuration interface is simple, but it works reasonably well.
- The ServeRAID MR10i is a controller i haven’t dealt with before. It does not offer a standard character based interface for configuration, only a graphical interface called WebBIOS. It’s completely awful and half done – half of the buttons have no text on them, every button press requires several seconds until something happens. Configuring a controller with this interface requires you to guess actions based on the manual, since they aren’t labeled onscreen.
- As usual, both LSI based controllers use the MegaRAID management software. Compared to the old Adaptec software, it’s really awful. It runs extremely slow – even on these new servers, is much more complicated than the old ServeRAID software and offers less options in terms of notification.
- The IMM webinterface has gotten even slower than the already slow RSA II interface. Web2.0 style “loading” icons have been added, but viewing the status screen can take up to a minute now. This is retarded, and clearly a step back. At least IMM standard now comes with every server.
- The IMM’s KVM capability have gotten a lot better. Instead of a java applet running in the browser, a java application is launched using java webstart. While IMM itself is slow, the remote KVM capability is actually very fast, and even works with decent speed through a VPN connection.
- While the ServeRAID 8k with newer firmware usually spent 2-3 minutes looking for the drives, the new LSI based ServeRAID 10 series now only takes a few seconds. This is compensated by UEFI which now takes roughly 2 minutes instead of the 10 seconds the old BIOS took. With this, IBM is successfully keeping the server at roughly 4 minutes until OS boot
- The IMM connects to the OS using an USB LAN interface. This is a real problem on Windows, since it confuses the Windows Firewall (switching it into “Public network” mode) and the Windows DNS client. Install the driver and disable it in Windows. Ensure you never enable it on DCs! Run the IMM firmware updates from CD. Clearly a step back.
- The UEFI configuration screens act a lot slower than the old BIOSes. But the options available are decent.
- You can install Windows Server 2008 Standard in UEFI mode. I did that on our internal x3650 M2 which is going to run our WebFOCUS deployment
- You cannot install SBS 2008 in UEFI mode. It will work fine with legacy BIOS emulation, though.
- The IMM can’t send alert messages to email addresses with a – (Dash) in them. Retarded.
- The machine is extremely silent. Compared to the Power 520, which will kill your ears within minutes, they’re a blessing.
- Just like the RSA II interface, the IMM web interface has a tendency to lock up randomly and stop working. Requires a physical power cycle on the server to fix.
That’s it for now. Lot’s of negative stuff in here, but the machines are actually extremely good performers. I hope that IBM will fix those outstanding bugs soon.
Chris Z:
How was the documentation?
5. July, 2009, 03:53Lukas Beeler:
Chris,
As always from IBM – overly elaborate and missing the critical information you’d need the most ;)
5. July, 2009, 07:39Matt:
Hey there I’ve just deployed one of these X3650 M2 Servers and Server 2008 64bit
OS on 2 x 140GIG SAS HDD’s (Mirrored) / Data on 6 x 300GIG SAS HDD’s (RAID 10)
ServerRAID MR10i + Extra Hotswap Expansion
24GIG’s RAM currently only 1 x 2.2GIG CPU
and I’ve got a few problems maybe someone can shed some light.
1) For whatever reason the IBM ServerGuide install CD wouldn’t work for me (it didn’t pickup my drives?)
I had to install straight from my original Server 2008 disks
(so maybe some drivers aren’t loaded correctly, but I have used the ServerXpress updater anyway)
(it’s completely failed to load the VGA drivers which I had to add manually [Matrox something or other])
2) The Performance of the server seems low, I expect should be snappy, things seem to lag a bit.
(Can’t help but wonder if something is running in a kinda 32bit compatablity mode somewhere?)
The MEGARaid Storage Manager(MSM) is UNBELIEVABLY slow (makes me wonder if there is a controller fault)
3) After installing the option for me to enter the WEBbios raid config util (during post) has disappeared and I can’t back into it?
Any feedback would be GREATLY appreciated.
13. July, 2009, 21:54Cheers
-Matt
Lukas Beeler:
Heya matt,
1) Yep, will probably be fixed. Also noticed the fact that the Matrox driver didn’t get installed automatically, but i thought it was my mistake. Looks like it wasn’t.
2) WS08 using RDP has always been a bit laggy – WoW64 (the 32bit compatability mode) doesn’t cause such issues, though. The performance of WS08 itself is fine, though. MSM is just that slow, it’s completely horrible. I’ve had to deal with it on x3250 M2, and it’s a complete PITA.
3) Hmm, shouldn’t have happened. You can try entering the controller setup using UEFI and then selecting the controller configuration option in UEFI itself.
13. July, 2009, 21:58Matt:
Hi there,
Thanks for the awesomely quick response & feedback,
I’m a bit relieved that this hasn’t been something I’ve done horribly wrong I’ve been really stressed the last few days.
I’m setting up my server for SQL 2008 64Bit
My laggy response seems to happen even when using the machine locally not just thru an RDP session.
(I must check this again and I’ll try give proper benchmark example)
Should I be worried that there are several items in the taskmgr that have *32 next to them even thou I haven’t loaded any software (apart from ServerXpress & service packs)
Should I load the IBM director(I haven’t yet) will it help the server in anyway or will it just provide me with better monitoring?
and lastly pardon my ignorance but what is or where do I find “UEFI” ?
13. July, 2009, 22:42Cheers
-Matt
Lukas Beeler:
Yeah, don’t worry about the laggy GUI. It’s normal, and it doesn’t impact actual performance of the server.
Seeing x32 processes is completely normal – there is no performance decrease from processes using WoW64, and most applications still run partially in x32, even if they’re called “64 bit” versions by their vendors.
I don’t like IBM Director, it’s overkill. For basic monitoring, make sure to configure IMM and allow alerts to be delivered by mail. Also, configure the monitoring functionality in MSM. This is uses almost no resources and is rather hassle free, compared to IBM Director which requires a dedicated management server. Of course, IBM Director is the better choice in big deployments.
UEFI is the sucessor to the BIOS.
13. July, 2009, 23:01Nathan:
Good comments Lukas. FYI; I forwarded your comments to our RAID/Storage, IMM, and UEFI leads.
Matt – Like Lukas suggested, you can access the webbios configuration utility for the RAID controller from F1 Setup->System Settings->Adapters and UEFI Drivers (look for a LSI entry and press enter to launch its config util. One particularly nice thing about UEFI-based servers and UEFI-complaint adapters is you can manage those adapters within F1 Setup. Note: if you boot in pure UEFI-mode (WS2008,WIN7, etc.), you will not see the legacy OPROM banner (with ctrl-c option).
-Nathan
17. July, 2009, 04:06Lukas Beeler:
Nathan,
Thanks for the feedback. It’s nice to see that even a big company still listens to their customers!
17. July, 2009, 10:55Jeff:
Just got my first batch in today. Disappointing cable management. The tray is to small to handle anything other than your basic load out. Lets fill it up with 2 power, 4 Ethernet, 2 fiber and a KVM option. It also forces the extra slack in the lines into the rack which is not an ideal situation. The removable zip-ties are also a pain. Bring back the Velcro!
Just wondering on your thoughts to this.
Performance wise they are right up there as I would expect.
21. July, 2009, 00:22Christoph:
OMG! This LSI MegaRAID Storage Manager is SH!T!!! Takes 5 minutes to open the connect screen, its been 15 minutes “Scanning Adaptor”, what the hell! Surely this is not normal…
Also, i’m running Win 2008 x64, why does the server keep failing to boot from the onboard SAS controllers when my SAN is connected. I’ve had to add “Legacy Mode” to the boot order. Am I missing something here. I’ve been through every option on the QLogic FC adaptor settings in UEFI settings and can’t find a way to tell it not to boot from this device.
22. July, 2009, 10:02Martin:
Hi,
5. August, 2009, 09:00We just got two of these and are experiencing many network issues. When we try and get DPM communicating from these servers to virtual servers on other hosts we are getting massive delays which are causing installation etc. problems.
Has anyone heard/had any of these kind of issues?
Also have you noticed the delay in task manager when going from any tab to the networking tab and back? We think we have fixed this by modifying the settings of the management nic in network settings.
Lukas Beeler:
Hello Martin,
I’m quite sure that your problem is that you installed the Intel networking driver using UpdateXpress (which happens by default – even without any Intel NIC installed). This will enable TCP Offloading, which _will_ cause issues.
To verify:
Type netstat -t in the console. Verify that all connections show up as “InHost”. If they do not, use netsh to disable all the offloading stuff.
Here’s the IBM Support stuff about this topic:
http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5081260&brandind=5000008
5. August, 2009, 09:15Con:
Your MegaRaid Storage Manager problem will be fixed by updating your IMM package to the latest on the web. In particular the rndis inf file contained in the latest release fixes the issue.
11. August, 2009, 00:04http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5079600&brandind=5000008
Michael D'Angelo:
Had the same issue with a Win 2008 x64 on x3550M2. I disabled the “IBM USB Remote NDIS Network Device’ in the device manager to fix it.
14. August, 2009, 02:31Scott:
SYMPTOM
After installing drivers for the IBM RNDIS LAN-over-USB device, system performance may be severely degraded when the RNDIS device LAN interface is queried by network applications. This issue has been seen when running the MegaRAID Storage Manager utility and while running Windows Performance Monitor. This issue could affect any network application
FIX:
Fixed in Windows 2008 R2 by updated RNDIS drivers.
There is currently no fix for Windows 2008. A solution is under investigation with Microsoft.
WORKAROUND:
Unbind “QoS Packet Scheduler” from the RNDIS network interface.
To accomplish this, complete the following steps
1. From Network Properties, right-click on the RNDIS network interface
2. Clear the QoS Packet Scheduler check box
3. Click OK.
DETAILS:
The QoS filter driver in Win 2008 (pacer.sys) is an NDIS 6.0 driver. The RNDIS drivers supplied in Windows 2008 are NDIS 5.1 drivers. This difference is the root cause of the performance issue. The Windows 2008 R2 RNDIS driver is NDIS 6.0 compliant. The QoS Packet Scheduler is not necessary for the proper operation of the IBM LAN-over-USB device.
19. August, 2009, 19:26Investigation is ongoing with Microsoft to obtain a solution for Windows 2008.
Seth Conn:
The M2 is a nice system, but we nearly sent it back due to OS load issues. The latest ServerGuide disk would not boot, at all. Nor would our Windows 2008 disks. Both would attempt to boot into the Windows PE for setup, but would not be able to find “\windows\system32\boot\winload.exe” and would not boot into the install screens. Tech support gave us a 2 week long run around, telling us (initially) that the ServerGuide wasn’t supported on this system (not according to ServerGuide’s readme), the DVD drive was bad (same exact issue when using Virtual Media ISO mounts), that we needed a Windows 2008 retail disk, the disks/isos were bad (disks and ISOs work as expected on other machines), that I needed a software support contract (which we bought, pulled strings to get through the system, then was IMMEDIATELY told by software support that it wasn’t a software issue).
We read around and someone somewhere suggested that we try a newer version of the Windows 2008 disk. Downloaded SP2 disk from Microsoft’s volume licensing site, same issue. Finally I tried an Windows 2008 SP2 iso from the MSDN site, dated 5/11/2009, and the system installed just fine (filename en_windows_server_2008_datacenter_enterprise_standard_sp2_x86_dvd_342333.iso). I hope our pain helps someone else, because Support will not.
We’re loading up this machine right now and I’m really impressed the the Remote Control and the Virtual Media features. I have a few Avocent KVMs, which the interface is similar (do they license this from Avocent?), but it’s really nice to be able to power the machine from the KVM screen. The Virtual Media is incredible, especially if you’re mounting isos from the same LAN. You can do it across the WAN, but slower links = slower access (obviously, but it does work).
I agree with everyone else that the MegaRaid “WebBios” has a horrible interface, but I was able to do the configurations without too much pain. The MegaRaid Storage Manager, in Windows, is an exercise in patience. Even after upgrading the IMM firmware as suggested in these comments the GUI was still way to slow to be of any use at all. It’s a shame; IBM seems to make good hardware, but their software is some of the worse I’ve ever used (it’s always been this way). Hopefully they’ll change that someday.
20. August, 2009, 17:23Lukas Beeler:
Seth,
Thanks for your reply.
I’ve installed WS08 x64 and WS08R2 x64 without any issues. The x64 versions are EFI enabled, the x86 versions are not (but they should still run – SBS08 isn’t EFI enabled either). May i ask why you’re running x86?
KVM stuff is indeed from Avocent, it’s even in the “Info” menu of the applet.
The ServerGuide has always had issues with new machines. The Bootable Media Builder fails on x3400 M2. We’ve had baseboard failures on both x3550 M2 and x3400 M2.
Ah well, i wouldn’t have a job if there weren’t any issues :)
20. August, 2009, 17:28jivetolkein:
re: buggy RSA cards needing a server restart – I’ve found 99/100 times you can reset the card by telnetting the address, logging in as usual and issuing a resetsp command – careful, as reset DOES restart the server, resetsp does not.
Just got our first M2 in, looks good to start with but a few bits missing like the IMM module for KVM ^^
25. August, 2009, 18:16James Davis:
The megaraid software speeds up 100X after you disable the QOS
For some strange reason i have to disable the NDIS adaptor and restart services every time i want to connect to the MSM otherwise it trys to connect to some strange ip….
SBS 08..
28. August, 2009, 09:57JT:
Wow, this MegaRaid Storage Manager is killing me. Have updatet IMM FW to latest (yuoo32f) and disablet QOS as described above. Still, management is slow and I didnt manage to make a new array from MSM. Help was no good. Finally I managed to make a new array from adapter BIOS. I must be missing something here?
2. September, 2009, 12:18James Davis:
GLITCH!
SBS 2008 x64
x3550 M2
Had the Second Nic plugged in instead of the first (without thinking to much about it)
Megaraid storage manager would not work!!
Plug in first nic and BAM – MSM finds controller straight away.
Teamed and still working fine!
As said above Disable the QOS on the NDIS device and its a million times faster.
2. September, 2009, 23:16Busta Move:
Christoph – did you ever find a solution to this issue? – having same issue. Boots fine without san fibre attached, BIOS disabled in HBA, will not see local storage when san connected.
+++ Also, iââ¬â¢m running Win 2008 x64, why does the server keep failing to boot from the onboard SAS controllers when my SAN is connected. Iââ¬â¢ve had to add ââ¬ÅLegacy Modeââ¬Â to the boot order. Am I missing something here. Iââ¬â¢ve been through every option on the QLogic FC adaptor settings in UEFI settings and canââ¬â¢t find a way to tell it not to boot from this device.+++
17. September, 2009, 04:01gigi contra:
The new IMM is a nightmare for remote administration.
2. October, 2009, 10:02Slow login page load: 1-2 minutes.
Slow login: 15 minutes untill main page is loaded. And this works only if you have an open connection to the console of the IMM.
Latest firmware update makes it even slower
Working as designed …
band:
Hello,
I am having issues kickstarting 3650 M2. The PXE/DHCP doesn’t work. The installation crashes. Any idea?
Thanks,
2. October, 2009, 20:24band
Roei:
this server is excellent but :
* The basic controller that came with the server LSI br10i not supported in more than 2 raid array on x3650 M2 (maximum 4 disks 2 arrays).
* the raid-1 not really functional in the real time (2 disks only), even all of the array is bootable !!!
as well :
* the raid controller is very slow and support only raid 0 1 and 1E
if you buying this server you should upgrade it to better raid controller as LSI MR10i
Good Luck
6. October, 2009, 15:33russellb:
Can’t believe how slow the storage manager is. This is really bad, even for megaraid.
I updated IMM (and all updates) per tech support but it did not help.
The software is unusable. Any idea when this will be fixed?
8. October, 2009, 04:20jtc:
howdy, try this for issues with booting when SAN is attached, if your still having problems.
http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5081613&brandind=5000008
13. October, 2009, 17:28milo:
Also had trouble booting many x3650M2 servers after being attached to a San.
Used ‘Legacy’ method for booting from non-EFI OS’s eg Vmware ESX – http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5081613&brandind=5000008
For EFI (2008 Server x64) – http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5079636&brandind=5000008 – NOTE: This will only boot the machine. Need to use same process in Boot Manager to add a new boot entry then change order of entry
27. October, 2009, 02:18Tyler Rosolowski:
What IBM don’t mention is that the old 8k adapters which are 2/3 the price (in NZ at least) also come with battery backed cache as standard.
The new MR10i’s DON’T, which drags down RAID performance on small data patterns to less than 10% of a 7200rpm hard drive from our testing.
The MR10i’s battery is 1/2 the cost of the card again, which means that the RAID cards have in effect doubled in price.
God knows why if performance is so affected they don’t ship the cache batteries as standard as they do with the 8k controllers.
27. October, 2009, 03:38Tawney Follett:
MR10i FW & DD Update along with Installing JRE 6 Update 16 fixes the horrid MSM Slowness.
27. October, 2009, 04:41craig:
Hi,
31. October, 2009, 22:07The new IMM does not work for me.
Login Page Error: I am unable to login to the IMM with default login and password. After 20 mins i am getting an error saying “Error handling with http://www.ibm.com/BMC/sp/Monitors: Not proper XML: A string literal was not closed.”
tried to update the IMM FW from 1.04 to 1.05 with IBM bootable media, but it is taking 1.04 by default.
These are brand new 3650 M2′s in our organization, no OS on the systems.
Any help would be most appreciated.
Lukas Beeler:
Hi craig,
I’m not sure what you issue is. Which web browser are you using?
Can login using telnet or SSH?
31. October, 2009, 22:49craig:
I am using IE 6.0 and its through local intranet.
4. November, 2009, 05:14mark:
Anyone having problems with degradation of the virtual drivers on on drives 0 and 1 … We updated to latest firmware and drivers but everyday a raid error shows. Raid controller it teh MR10i
Thanks
5. November, 2009, 17:56Philibuster:
Hey, the docs say to press ctrl a at the prompt to access the server raid configuration bios app. But, I don’t see any such prompt. I would sure like to configure the RAID. This server is a box stock server no additional raid card. Supposedly, I should be able to setup RAID1.
I tried standing on my head, and waiting until midnight of the full moon, and still no prompt. They really made it a secret for some reason! lol
9. November, 2009, 17:27vincent:
Your review was VERY helpful in resolving our no OS when we attached to a SAN problems / panic.
However, I learned after hours of trying that just b/c you install Windows 2008, it may not be setup with the /EFI/.. directory items to be fully UEFI aware at that boot level. I ran into this problem which was likely caused b/c I used an “installer” wrapper process instead of the raw Windows 2008 DVD. Sometimes these do not allow the creation of the items as expected b/c they were not part of prior Windows 2003 etc.. install processes. An example of a wrapper would be “Server Guide”.
As for the IBM USB Remote NDIS Network – After reading these docs, I was under the impression this LAN/USB adapter is used to communicate with the IMM to the installed OS. In some of the documentation, I found that it suggests setting this as a fixed IP of 169.254.95.118 or 130 to prevent NDIS from grabbing the actual IP of the machine from the other adapters (which I had happen) or to prevent NDIS from using DHCP and obtaining some other IP address. This was all new to us but it is in these docs.
Here are the IBM docs on the IMM / setup and the transition to UEFI:
ftp://ftp.software.ibm.com/systems/support/system_x_pdf/imm_users_guide.pdf
http://www-07.ibm.com/systems/in/info/x/newgeneration/hs22/pdf/XSW03031USEN.PDF
Cheers.
12. November, 2009, 13:33Rudi:
W2008R2, installed in UEFI mode (GPT not MBR)
18. November, 2009, 23:34For anybody intends to add the SAN disks later, please add the first disk with W2008R2 up and running.
You even need to write a signature(initialize) before the next restart.
If you do so once, you will not run in trouble again, even removal of the disks and remap of new created volumes was possible without ever seen the boot hang again.
.
There is a problem in W2008R2 as published here: http://support.microsoft.com/kb/975535 but if you download and install the patches, you will not be able to install GPT again – only MBR possible.
.
If you are fine with legacy (MBR) Boot, you can use the patch but take care that “lagacy” is on top of the boot list in uefi.
jiten_tech:
Hi,
I have a X3400 M2 and after the BIOS upgrade, i am not able to boot either Win2k8 X64 versions, neither Windows 7 X64 versions. All other X86 versions are working and installing fine.
This is 1.01 version for BIOS and everything was working fine uptil i upgraded the BIOS. The system would not recognise X64 versions as bootable media, is someone else facing an issue. Most posts here are pointing towards using x86 versions but I cant run Hyper V profile in x86 mode.
Thanks
25. November, 2009, 17:45Jitender
raid_newbie:
Hi there,
We bought recently a x3650 M2 with a MR10i controller (PCI-e), we have 2 x 73 gb disks and 4 x 146 gb disks. We are using the first 6 hdd slots, and the problem is that i can manage only the first 4 disks with the BR10i controller (integrated) and the other 2 disks cannot be used for any kind of array. We want to use the 4 x 146 gb in a raid 5 array, or at least 2 arrays raid 1 (2 x 146 gb each).
25. November, 2009, 23:12The only way to use the RAID 5 capabilities of the MR10i is phisically connecting the disks to that controller? do we need any aditional wires?
please help asap,
i’m in a hurry
Lukas Beeler:
jiten_tech,
Have you tried booting from the CD using the “File” boot option? x64 versions of W7/WS08R2 are EFI-enabled.
raid_newbie,
You need to remove the BR10i controller and replace it with the MR10i. Then plug both SAS cables into the MR10i. The MR10i has the ports switched around compared to the BR10i, so if you want the correct slot numbers in the management tools you need to switch them around as well.
26. November, 2009, 09:13Dan:
Can someone explain to me what I neeed to do to configure the (IML)Management Port , without going through the Bios , No OS is installed on the servers , I was thinking telent in, set the IP address and go through the Web Console and mount a virtual disk to install the os, having a horrible time with the Management port geting Upgraded Bios to 1.05 , management key is installed, If I could only get in , Please help reply with the steps that would be fine
6. December, 2009, 02:43Naaman:
Hi Lukas,
8. December, 2009, 13:39We installed X3650 M2 with 6 SAS disks on RAID 10. The webbios only allow RAID 10 config in 3 groups of disk and in the end we ended up with 300GB x 2 = 600GB of usable disk space. Shouldnt we get 300GB x 3 = 900GB of usable disk space in RAID10. Thanks.
DAN:
If you are having issues getting your management modules to work, you can try changing the link speed to 100mb fullduplex and turn off auto negoitate if you are lucky to get in , if not this will have to be done on the switch , worked for me
17. December, 2009, 21:18Jim Bucks:
I agree, the LSI / MR10i controller is horrible! I’m trying to get a 12 disk 3650m2 set up with a 2 drive (raid1) and a 10 drive (raid6) configuration with a Linux operating system (as an internal samba server). The blasted MR10i will not let me set up a hot spare drive, will not automatically acknowledge and import a replaced drive, will not let me define an email address to send PFA’s to, will not let me define levels of alerts, and on and on….
The lashup IBM has us go though to put the MR10i in place to use all 12 drives is another “interesting” configuration. The “how to” doc is there, but is buried deep in the IBM web site. You have to really go hunting for it in order to find it. I’m still left with a couple of parts that are not even mentioned on the docs I found.
And don’t get me started on the HORRIBLE WebBIOS interface the LSI MR10i card has (as others have mentioned). I’ve seen better / more complete user interfaces on my old Commodore 64 computer back in the 80′s.
Also, IBM will not talk to me about these problems. I’ve been through several levels of management, since the support lines will no longer talk with me unless I pay another $500 – $700 per year to talk with the people in Atlanta (this is a new server, was bought about 2 months ago and has NEVER been placed into production because of these problems). I did get a real good reaction from one (supposed) VP when I sent him a picture phone photo of the missing text on the radio buttons of the WebBIOS interface.
As you can tell, I’m not a very happy camper with the M2 series of machines. I’ve got about 20 of the non M2 / older machines (which I have nothing but GOOD to say about). Unless this can get fixed / resolved by IBM shortly, I’ll be checking out other small / mid server machines for my next company standards.
Jim
22. December, 2009, 20:42Florian Stähr:
This is my second M2 Server setting UP. The first one was a 3250 M2, the second one is a 3650 M2, both of them got a crappy Matrox G200 8MB … the GUI is very slow and i cannot accept this. If there is no resolution for this problem, the next machines wont be IBM.
8. January, 2010, 16:22Lukas Beeler:
Florian,
If you spend more than 10 minutes in front of the server per year, you’re doing something wrong ;)
8. January, 2010, 16:55Florian Stähr:
Its a matter of principle, not the time someone spend in front of a server per year. The Matrox G200 chipset is from 1998, i dont know if the G200eV which is manufactured in the new IBM M2 series is a new one, but regarding to the performance on server 2008 R2 i dont think so. I like it perfect, and this is everything else, but not perfect ;)
9. January, 2010, 20:18Lukas Beeler:
Florian,
The G200eV is a embedded graphics chip with dedicated memory. IBM isn’t the only one using them, i’ve seen them in several HP servers as well. The x3650 used an ATI chip, which was just as slow. Be aware that those servers need a graphics chip which can interface with IMM/iLO and doesn’t use system memory (like Intel’s solutions). There aren’t many choices in this field.
High performance graphics card can even cause issues in servers, for example:
Hyper-V with high performance graphics cards
9. January, 2010, 20:22Florian Stähr:
I see, i will have to deal with it.
11. January, 2010, 08:24Kabeer Siddiqui:
Dears,
I have a 3650M2 server with LSI and SAS MR10i controller. when I click CTRL+H to go in WEBHOST, it doesn’t show me any disk and automatically configuration is not allowed.
All drives are shown @ startup but not in webhost console. Can anyone help me to configure RAID5 on it.
Regards,
11. January, 2010, 12:44Kabeer Siddiqui
Kym Charlesworth:
The comments about disabling the ââ¬ÅIBM USB Remote NDIS Network Deviceââ¬â¢ or giving it the IBM recommended static IP address (169.254.95.130/16) is all well and good unless you are deploying using Microsoft sysprep because that resets all the NIC’s. The %$#% thing grabs the DHCP address reserved for the Broadcom NIC from time to time which screws up a deployment good and proper. This is my first time on site with IBM kit and software and this is one of many negative experiences – would never recommend this vendor.
13. January, 2010, 21:58hphuhtin:
Another nice tidbit.. if your IMM loses network settings (static IP becomes 0.0.0.0) during firmware upgrade, and will not appear on network anymore, the fix is to set it on DHCP, login to that DHCP IP via web interface and set the static IP address via the web UI.
What a piece of junk, really. It seems to me IBM has taken to the web 2.0 “beta” concept and brought it over to enterprise server hardware.
14. January, 2010, 21:04SSD:
Recently bought this server with SATA disks and M5015 raid controller.
Updated the UEFI to 1.05, IMM to 1.07…still have problem making PXE legacy boot working with a RHEL 5 tftp server. From tftpd log, I can see it RRQ filename linux-install/pxelinux.cfg then nothing else..and server will reboot.
Any luck?
15. January, 2010, 22:21SSD:
After another day playing with the UEFI configuration..I found out the trick for PXE boot and hope to share it to others who may suffer the same problem.
To make the legacy PXE boot working, the trick is:
- make sure all onboard NIC port set up to enable Legacy-only in PXE
- Of course for Non-UEFI mode for each NIC, you select PXE instead of NONE
- then using the regular “PXE Network” boot via F12 or in your boot list will be able to boot to PXE legacy mode.
At the beginning, I only enable PXE (Legacy-only) in NIC port 1 and disable the rest…and it keeps to have boot error…where request the pxelinux.0 file from tftpd but not downloading it..so this is the main reason why it’s not working. (BTW, in my lab, dhcpd and tftpd are on the same server)
Hope this help
18. January, 2010, 02:16SSD:
Another thing to share:
For IMM, I’m using static IP only and disable dhcp. I found out that:
If I update the IMM firmware via SLES linux (xxxx.bin -s), it’ll semi-reset the IMM back to factory default. Firmware update will reboot IMM, then you CAN still go to web as your ‘first’ static IP..but after you login, when you browse the IMM, all settings are gone..so if you restart IMM from web now, you’ll revert back to dhcp or default 192.168 IP.
However, if I update the IMM, UEFI and DSA via IMM web interface, then IMM setting will remain.
18. January, 2010, 02:21Eddy:
I ah having some problem with my x3650M2 hope you guyz can lend me a hand. i have a x3650M2 with 10Gb of memory memory is 6 x 1gb and 2x 2gb, 2.4ghz single processor, 5 disk raid 5. The OS install was windows std 2008 R2 x 64 with SQL 2008 Dev. Edition. when i start up my server the memory using was around 4gb the mechine is stable, in less than 30 minute the memory goes up to 7gb in resource monitering tool on windows 2008 and the whole mechine started laging. i have around 20 user connection to this mechine for the SQL.
21. January, 2010, 01:09Local operating system boot fails when external storage is attached « Ilantz’s Weblog:
[...] First real world experiences with IBM’s x3650 M2 [...]
14. February, 2010, 21:55Nathan:
For those interested, there is a IBM System x UEFI whitepaper available @ http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5083207&brandind=5000008
-Nathan
16. February, 2010, 05:28Jim Bucks:
Has anyone tried using an Adaptec RAID 51245 – storage controller (RAID) – SATA-300 / SAS – PCI Express Mfg. Part: 2258400-R in a 3560 M2???
I’m to the point of trying this combination and throwing out the LSI (MR10i) card.
Thanks,
Jim
16. February, 2010, 22:04Eric Lo:
I am having the same issue with IMM login as Craig posted in October. “Error handling wsdl http://www.ibm.com/IBMC/sp/Monitors: Not proper XML: A string literal was not closed.” I am using web interface.
Any help will be appreciated.
22. February, 2010, 23:26thanks!
Jace Faulkner:
Hi,
1. March, 2010, 17:59We have just purchased a new X3650 M2 with a M5015 Raid controller. Do I remove the BR10i and replace it with the M5015 or use the other slot? I ask as I notice the two leads that are connected to the BR10i are fine but the M5015 does not have these ports so do you leave them unconnected?
Thanks.
Jace Faulkner:
The supplier shipped the wrong card. Ignore this.
2. March, 2010, 12:59Jim Bucks:
I’ve given up on the lsi cards. (MR10i).
I *was* able to get all the prompts to be visible by re-downloading and forcing a re-install of all the x3650M2 firmwares.
I *was* able to designate a global hot spare drive (slot #12).
I was *NOT* able to make the global hot spare drive “kick in” when I simulated a drive failure (pulled the drive in slot # 8).
I’ve ordered the Adaptec 51245 and will share the results after installation & test.
Jim
4. March, 2010, 17:59sans:
Hey Eric try to check the Baud rate,speed and check whether they are same as that on the network switch side.
4. March, 2010, 21:44Steffen:
Hey,
Ran into much of what people have already experienced, one thing I did find though, is that on an LCD monitor, if the contrast was 80 or above, I was unable to see many of the text labels in the webbios’, once I turned this down they displayed normally – might just be our crap monitor though.
Also the new “updateexpress” system is not really working that well, for me it worked okay once I had installed windows, but their boot cd creater tool went into an infinite loop with some ata1 device and never got as far as actually updating any firmware.
Very dissapointed with this model so far.
17. March, 2010, 18:54Daren:
After attending a US Govt event where IBM presented their “commitment to solving your support needs” and hearing how “hardware is a commodity – but IBM produces the best-integrated systems – and Support and Service is what IBM *is*” and how the wonders of ServerGuide, Systems Director and ServeRAID are the path to Server Nirvana…
Just setting up 4 brand new x3650 M2′s.
Worked with IBM’s pre-sales to spec servers per our operating and OS requirements.
Brand new out-of-the-box as-shipped directly from IBM (Feb 2010).
Well, I’m in OS install purgatory and IBM support’s inferno together with four new servers (boat anchors?) that won’t boot to anything but BIOS, RAID WebBIOS/CLI or ServerGuide. (With these boat-anchors-shaped-like-rack-mount-servers I can’t even get across Styx to spend some time in Hades!)
To begin: IBM shipped a version of ServerGuide that doesn’t even support the server hardware with which it shipped. !?!?
…Three versions of ServerGuide later…
Only positives so far: WebBIOS worked well to config RAID 5 across 4 300GB SAS drives. ServerGuide seemed straight-forward and worked well for RAID setup and until OS install step.
Now: Much trouble there is! Uncertain is the future. But, almost certain it is that IBM it does not contain.
ServerGuide (latest version(s) d/l’ed from IBM – @v.8.21 now) hangs at OS install (W2k8R2) with unspecified “Critical Error.” Only solution is to shut down machine.
Server won’t boot to any media but ServerGuide CD.
Unable to boot directly to Win install DVD, USB or any other media at all.
Received the exact same run-around from IBM support described by Seth Conn above: “Must be: faulty install media, faulty DVD/Optical drive, faulty power supply, ServerGuide not supported on this system (It is according to ServerGuide page and ReadMe.), faulty .iso’s, need updated BIOS/Firmware/UEFI on MB/Raid/IMM/OpticalDrive, need updated drivers… Paid for extra support. Hardware support insists it must be some software issue. Software team is convinced it’s hardware. Both say the other doesn’t know what they’re talking about. ?!?
What? Is this how IBM does integrated service and support these days?
I am seeing a uniform response now every time someone new from IBM has looked at our trouble: “Well, something is clearly wrong! Let’s see what we can do…” [after several iterations of trying what we’ve already tried several times] “Ah, well, since [none of what we just did worked], it must be a [software/hardware : failure/issue/glitch/bug (pick one that has nothing to do with this support person)] and you need to [install this patch/update -or- contact (insert IBM support organ that has nothing to do with this support person)]… It would be really trivial to emulate *all* the IBM support I’ve had so far (and had my fill, to be sure) with fairly simple software… Um, you don’t think???… No, that’s just not possible. They couldn’t really get away with…
I’ve up- and down-graded and flashed every firmware and software component IBM can specify; replaced: 6 power supplies, 4 optical drives, 2 RAID cards, 2 MB batteries and 8 SAS HDD’s; and tried every OS install from Red Hat to MSWinServer2000 to Debian to MSWin2003R2 to Ubuntu to Win2008R2 on every media from DVD/CD to USB HDD to USB Flash to SAN to network .iso, with 9 or more different versions of ServerGuide (which were all proposed as “the answer” “to get the system up and running, so we can see what’s really going on,”) all to no avail. No OS will boot, let alone install.
I’d been led to believe a lot of these problems had been solved / solutions improved by IBM. But, unless I find a solution that doesn’t involve a “consultant” or IBM’s pricy add-on “support,” I will move back to HP or Dell for our small / mid size servers.
Just too typical. Poorly documented overkill complexity that doesn’t perform in even the most basic configuration or set-up scenarios.
My new HP and Dell servers all boot the MS Win install media just fine. !?!
I really can’t believe these servers won’t install Server 2003 or R2 or 2008 or it’s R2.
How do IBM think these servers are going to be used?
I’ve asked IBM support (and even gone back to the pre-sales guys I worked with) to give me some – any! – supported base OS and configuration. Nothing. “We don’t really have reference configurations for other vendors’ software.” O.K., how about IBM software? Nothing. So, what am I supposed to do with these servers?
I’ve gone from really believing I’d have at least one of these servers in production this month and that IBM would really deliver, to being ready to RMA all four and move on to some hardware that works.
I’m done.
(Ironic biological double entendre ain’t my forte – but I got one out!)
23. March, 2010, 16:55Lukas Beeler:
Hi Daren,
I can clearly hear your frustration.
I’ve had my troubles with getting UEFI working right (as it sometimes refused to boot correctly, just as you’ve said), but all of these issues have been worked out mostly.
It sounds like you purchased your hardware directly from IBM? That’s not possible here in Switzerland – customers purchase the hardware from an IBM Business Partner (like us) which purchase the Hardware from an IBM authorized Distributor (Avnet), which purchases from IBM.
If one BP can’t help you, choose another one. That doesn’t look like advice that helps you, though.
I have no idea what’s wrong with your configuration. If you were in Switzerland, i’d come onsite and help you get your servers to boot as it should for free – just to make sure you can enjoy your IBM hardware.
As it is right now (according to my docs), i’ve deployed 12 x3650 M2 (2x SBS08, 10x WS08/R2), 2 x3550 M2 (WS08R2), 6 x3400 M2 (SBS 08).
All the SBS deployments, i did without the ServerGuide CD. Some of the other deployments, i did directly with install media, some with USB, some with ServerGuide.
Do you have an IMM KVM capability enabler installed in your server? If yes, i might be able to help your remotely. Contact me on skype – my username is “lukasbeeler”.
Regards,
Lukas
23. March, 2010, 17:17Luan:
We have here a brand-new 3650-M2 System and additionally we bought the M5015 Controller. Isn´t it possible to replace the integrated BR10i card with the M5015??? Asking because the slot fits nicely, but the blue rentention module, which formerly hold the BR10i don´t fit with the new 5015. Is the 5015 only to to be installed in the back of the server in one of the usual expansion slots? But, if so, how to connect the backplane cables (a wy to short to go to the back). Any help is welcome. Thanks.
30. March, 2010, 15:03zuhaz:
I have purchased 4 x3650 M2 for our client, I am having a very hard time as the raid controller which we purchased along with that is M5014(FRU: 46M0918), the configuration for RAID 5 is missing on it. But the hardware is compatible as per IBM serverproven system: http://www-03.ibm.com/systems/info/x86servers/serverproven/compat/us/xseries/controllers/matrix2.html .
31. March, 2010, 09:42Fabrizio:
Hello everyone,
I’m going slightly mad trying to install Windows 2008 Enterprise R2 into “my” new IBM x3650 M2 (7947-96G). The SAS controller is a M5015.
If I launch the IBM ServerGuide 8.21 CD, at the end of the process, when the system asks for the Windows DVD, after having put it into the DVD bay, it comes up the error “D:\Sources\Setup.exe is not a valid Win32 application” and I have to shut down the server.
If I try installing Windows without the ServerGuide, instead, the installation process starts fine, but then, when the system asks for the RAID controller drivers, I get stuck.
I had a chat with an IBM support member here in Italy, he drove me first to the IBM ToolsCenter Bootable Media Creator v2.0 for Microsoft Windows and then to the SAS Controller drivers’ page (http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5082161&brandind=5000008).
I extracted the drivers into a USB pendrive and attached to the server. But the system doesn’t like none of the 3 choices.
What shall I do next? What I’m doing wrong?
Thank you for your help.
1. April, 2010, 11:07Fabrizio.
Jerry:
To Luan.
11. April, 2010, 02:13It’s possible to change RAID controller to M5015. Only thing you have to do is change blue retention module with the one which is located over power supplies next to other black clips.
Steffen:
Just got confirmation from IBM in Denmark that if you have an additional external sas controller plugged into M2 models and have external storage connected, their update cd will not work until you unplug the external storage.
19. April, 2010, 11:57Toby:
I have just installed a 3650M2 and have run into problems with the SAS 4 pac HDD and I was very surprised that no one else on here has reported the same.
I followed the instructions to the letter but the drives connected in the expansion unit don’t work. I have swapped the drives over to rule out doa HDD’s, swapped the back planes and cables over to rule that out but I can’t get them to work.
18. May, 2010, 17:15Tyler Rosolowski:
I called up IBM to find out how to do the online RAID migration as even though the IBM Raid Card manual lists this a key feature, the only place it made reference to it was in the key features page. IBM also refer to this feature as logical drive expansion and something else.
IBM Phone Support refused to help me as this was a config issue, not a hardware issue. Sucky. HP’s kit makes this so much easier, and the much worse software within Windows to configure the RAID card has the feature listed, has no information in the help and grays out the option when you select the drive.
From the limited information IBM provide about this feature, is that “Online” means reboot, go into the BIOS and configure the online RAID expansion. I don’t know if you can then reboot and it will continue or if you have to wait until it’s done (I left it until it was done – 10 hours offline.)
My definition of ‘online’ doesn’t include ‘rebooting into the BIOS’.
IBM have quickly released x3650 M3 as I predicted as the M2 was such a rushed to market, v1 EFI BIOS bug filled release. I Imagine Warranty and Support were happy when the M3 came out.
This server is so horrific with it’s lack of cache battery and 5 minute EFI initialization times that I’m looking at HP Servers for the first time in 10 years.
13. June, 2010, 01:08PTCruiser:
I have had much the same experience as Tyler. “Sucky” sums it up perfectly. I don’t know what happened to IBM these last 4 years. They were on top of their game, especially with their blade servers.
My organization began switching back to HP servers. Interestingly, HP servers are totally different than 10 years ago. No more Netservers and TopTools management software. Now HP has Proliant servers and Insight Manager software, thanks to their merger with Compaq 9 years ago.
Sure, every vendor has their problems. Maybe IBM will make a comeback, but with HP growing every quarter, and having more than half the server market for blades, I just don’t see it happening anytime soon.
18. June, 2010, 21:13Salman:
I have two IBM x3550M2-7946 servers with 2 x 146GB SAS HDD and MR10i RAID Controller. Unable to access the WebBios of MR10i neither from UEFI nor I can see anything on the screen after POST. Any help out there?
1. July, 2010, 22:13Pat:
lukas beeler I have a question for you. How were you able to load the OS from a USB key? I’ve been trying to do this for two days now. My boss ordered two IBM x3650 M3 machines without cd/DVD drives by accident. I used IBM’s “Bootable Media Creator” to bring the servers up to the latest firmware updates using an external cd/dvd drive that was connected with USB. Everything installed great. When I try to install Server 2008 R2 I get an error when windows try’s to expand the files. I’ve read on forums that the files become missing/corrupt during the copy process. People have had success with installing the OS from a flash drive. I went into my systems BIOS and added “Legacy Only” in the top of the boot order followed by CD/DVD, Floppy and the Hard Drive. I have also added “USB Storage” to the top thinking it would force the system to boot from the USB device. I haven’t had any luck. Do you have a link or know of a video that I can follow to try and get my system to boot to my USB key so I can Install Server 2K8 R2? Thanks!
Pat
29. July, 2010, 14:56Ivan:
Hi. lukas,
11. August, 2010, 04:50I have faced a similiar case as Pat. I have an usb key to install RHEL5 and the usb key has proved to work in some other server systems like HP and DELL. However, the new IBM x3560 M3 system cannot boot from the device (or cannot detect the device) when we select ‘USB storage’ to boot from. Also, the system has waited too long to boot up …. pending for 5-10 mins everytime when the screen display ‘initializing USB legacy device’.
As we have brought over 40 servers recently, it is really time consuming to boot up the system. Is there any solutions about it ?
thanks!!!!
Ivan
Pat:
Ivan,
11. August, 2010, 13:25The way I usually set a server up is as follows: Firmware updates, Serverguide and then install the OS. I took a different approach with the x3650 M3′s. Since I was able to attach an external DVD drive by USB I was able to do the firmware updates. I restarted the server and installed the serverguide but when it came time to install the OS from the DVD thats when I was having errors. I wanted to learn/try to install everything from a USB key. I was unable to get the serverguide to boot off a USB key. What I did was setup the RAID configuration in WEBbios and then restart the server with the OS on a USB key using “WINTOFLASH”. The OS was able to install succesfully but I never had the opportunity to use serverguide to load whatever drivers it usually installs. I’m not sure if it makes a difference. Hope I could be of assistance.