But i don’t think that there is anything out there that is completely good and done, or perfected. As such, i always find things that need improvement or are implemented the wrong way for a particular use (or any use in some cases). And in other cases, there are also bugs that need to be resolved.

This is no problem in general, because many software manufacturers embrace (valid, constructive) criticism of their products, in order to provide better service and better products for their customers thus giving them more money and market share in the long term. Unfortunately, not all of them see it this way.

There are other problems in the IT industry (and some of these probably exist in other industries as well). One of them is unnecessary paperwork, or administrativa that prevent me from doing what i’m supposed to do (get something done). Of course, some of this is not avoidable and necessary – but in those cases it should be made in an easy, self explaining way with very fast turnaround times in case something goes wrong. Thus preventing the waste of time on administrativa and allowing to get the job done.

Those that know me personally know that I generally like to complain about stuff that doesn’t work right or better, doesn’t work the way i think it should, which is not always the same ;)

In many cases, this can help getting things done faster. Unfortunately, not everyone thinks that way. I don’t like making happy faces to a mess.

I’ve been in the receiving end of criticism more than i’d like to. No, i don’t like getting told that something i did was wrong. But shooting the messenger is exactly the wrong thing to do (even though i sometimes think that it would help my ego). In the end, criticism helps me to do my job better. But i still don’t like it.

You know who you are…

My focus here is clearly on Microsoft and IBM certifications, because those are the only ones i’ve got experience with.

I know many people that are afraid of certification exams – i was too, until i got some experience with them.

The first IT exam i took was the Microsoft exam 70-294. Two years ago, i’ve found my company lacking enough Microsoft certified personnel to make us stay in the Microsoft Partner Program. Something had to be done, and Microsoft even offered special Bootcamps for Partners. I booked the one with the nearest date and a topic that i was at least familiar with.

The bootcamp itself was very interesting. I, and some guy that was sent there by the RAV were the only attendees. The trainer however, know his topic very well, and i’ve learned quite a few things. A day later, i attended my exam and passed with just a few points over the required score. All was well.

Fast forward two years later, our CEO decided that we should earn the Microsoft Small Business Specialist. Again i’ve booked a one day bootcamp for exam 70-282. The attendance here was very interesting – and MCSE, an MCT and two sales guys. The course itself was rather worthless, i didn’t learn anything i didn’t know before. On the following monday, i passed the exam with a lot of room to spare. I didn’t learn for it, i didn’t do anything to prepare myself. And yet, i passed, with a good score and lots of time to spare.

I’ve got four years of experience dealing with Microsoft’s Small Business Server, and all of it’s related products. And that’s exactly what the exam tested. I knew all the stuff, because i’ve done it all before, for customers, in reality. Not in a virtual environment, not by reading a textbook, but at work.

At that point i decided to try something – why shouldn’t i just go and complete an MCSE certification? I registered for the next Exam online through Prometric. Exam 70-270 covered Windows XP. Again, it was all just trivia that i’ve done before on the job – of course i didn’t know all answers, but you can always take an educated guess. I passed.

After a few more exams, i had my MCSE. Without too much learning, and especially without cheating. One of the problems that IT certifications face are cheaters – people that use Braindumps to memorize the answers, and are then able to pass an exam.

I’ve seen people with enough technical knowledge to pass the exam still using braindumps. The problem here is two fold:

• People are afraid of exams, and want to make sure that they pass
• Exams sometimes have out-of-this-world questions

Not much can be done about the first point – just to repeat that failing an exam isn’t the same as having bad grades in school. You won’t get in any trouble, you will just have lost a bit of money.

The second point however is fixable. Microsoft did a lot in this respect, but there are still several questions which refer to ntbackup. Nobody uses that in a production environment, and even if they do, they probably don’t know what the difference between a “full backup” and a “daily backup” is (the latter is not incremental or differential, it uses the timestamp on the files). That’s why even people that are good at their jobs need to prepare for exams.

You get what you pay for – at least in the hardware world. If you do not have enough information, it is always difficult to say what you should buy, and how you should compare prices.

The problem is that it is sometimes hard for a customer to understand why there are so many ways to fulfill one single goal, and why the different solutions might have gigantic price differences. If you’re a sales type of guy, it’s important to make sure that you offer the right components, and make sure that the competition does the same.

Example

I need 1 TB of Storage.

Okay, this sounds simple enough. You can now make an offer for a simple 1TB Disk and an USB2 disk case. Totals at about 600 CHF including MWST.

Does this solve the customers problem? Does it help him? Is it what the customer wants?
Maybe. You don’t know.

You can generally leave a better impression if you ask the customer what he needs the storage for, and also think about secondary problems like backups.

I need 1TB of Storage for our ERP-Database running on one of our IBM x3650 servers. We have no idea on how we should back it up, though.

Now you already know a lot more, and know that the 600CHF solution with a single disk won’t even get close to the solution. You still don’t know enough to know everything (i.E. you could get 1TB with RAID10, but also with RAID5). You could also try to upsell to a SAN solution, though it wouldn’t be necessary. What you do have though is the possibility to upsell a backup solution – but there’s a catch here too.

ERP software is usually very important. Can the customer afford the degraded performance associated with an online backup? Or is there a need to use database mirroring and then backup the mirror? Problems like these are usually overlooked when shopping for hardware, and if you react right you have the chance to sell more hardware, sell services, and all this while helping the customer – you’re doing the right thing and making more money.

Real problems

I see this problem mostly with Windows Small Business Server 2003. Yes, you can get a fully working machine including licenses for 3000 CHF, but the question is if that’s what the customer really wants.

SBS is a product that makes many compromises in order to offer a highly competitive pricing, it even works against many Best Practices recommended by Microsoft. That doesn’t mean that SBS is a bad product, just that it’s a mixed bag. Selling two servers is usually impossible for Small Business customers, even though it would be best for Domain Controllers.

An SBS machine usually runs the whole company – it functions as a domain controller, file server, print server, Exchange server, and sometimes even ERP server or Internet gateway (i would recommend to seperate the latter two roles on seperate machines/appliances). If the SBS server is down, none of the information workers can do anything. That’s why you shouldn’t skimp on the SBS hardware. Hardware RAID, dual power supplies, brand components, updates, etc. are all important. Don’t buy whiteboxes or low end servers.

Why? You could buy an IBM System x3250, stick two 500GB disks and 4GB of RAM in it, and you have the same basic attributes – the problem is that such a machine is much slower IO-wise (you run Exchange and a Fileserver on this thing!), and has much worse reliability than better 2U servers like the IBM System x3650.

But if you’re running a bigger environment, go out and buy two x3250 and use them as Domain Controllers (and ONLY Domain Controllers).

Competition

There are many companies offering IT services to small businesses. While many of them have competent personnel that knows what they’re doing, sometimes they’re more sales than goal oriented. They will offer the wrong hardware for the job, and you’re the one who has to explain to the customer why your offer is more expensive than the one from your competition.

That’s why it’s important that even a sales understands what he’s selling.

As soon as a small business has more than one or two people, there comes a need for professional management of business resources. This includes bookkeeping, management of projects, management of orders, articles on stock, etc. pp. To do this, you will need an ERP software.

Let’s start with the cold reality – ERP software, if you want good, well integrated, well documented and well developed solution is expensive. More expensive than a good server. There’s also cheaper software available, but they’re not really meant for small business use, instead oriented towards clubs and one-man-shops.

The market for ERP software is very, very big. There are many software packages to choose from, and even multiple companies offering the same packages.

If you are doing IT in your small business, it is very important that you are not the one choosing the ERP software – while this may be the usual case for choosing infrastructure products like server OS, groupware, etc., this approach shouldn’t be used when choosing an ERP software. Instead, get the people involved which will be using the software, and get them to write appropriate functional specifications.

In general, these are the steps you’re going to take:

• Writing functional specifications for the software itself
• Writing technical specifications or requirements
• Evaluate products by list of features (on your own or with a consultant)
• Refining your specifications
• Evaluating products (together with their vendors)
• Refining your specifications
• Choosing a partner for implementing the software

Functional specifications for the software

It’s very important that you have very clearly laid out requests of functionality. In german, this is called a “Pflichtenheft”. Never buy ERP software without having written a “Pflichtenheft” or functional specification before buying software or talking to a vendor. This can be a preliminary version which you can refine later together with a vendor.

You need to know what you want, before you go shopping. If you’re a normal small business, with few established processes, most standard software available on the market should work fine for you. Maybe a little bit of customizing is needed in order to get everything fitted to your processes, but this is normal.

Technical specifications

The technical side with ERP software is different, and there are many ways leading to the target. Software can come web based, logic in client, logic in server, using Java, using VB6, using a combination of their own client server technology, etc. There’s also a big question on how to integrate the software into other technology your using – the mostly interesting candidate here is Microsoft Office.

Our Software DIAS-iS offers a very nice suite of macros to integrate Office into the software called OSP. It’s also based on a lightweight client, which allows to upgrade most of the functionality in one place, and having no business logic in the client itself – we call this technology the DIAS-iS Network Client. Due to the nature of this protocol, it’s very easy to use DIAS-iS over WAN connections, since one screen just needs one roundtrip.

If you have some Linux or Mac machines, it’s very important to have a multiplatform client available. For this, we offer Java Client. Please note that due to the implementation of DIAS-iS, with all the business logic on the server, the functionality of the DIAS-iS Java Client and the DIAS-iS Network is almost identical. This allows you to choose your client platform at will.

We also offer integration into a lot of other sectors, like Telephony, Point of Sale, Mobile data access and e-Commerce.

Our server platform is the IBM System i. If you don’t know the System i yet, i suggest you to read up on it. It’s a technically very nice platform, offering advanced features like a hardware hypervisor, well integrated systems management and other nice features. And if you can administrate a windows platform, you can also administrate a System i. (It’s always a good idea to know more than one platform anyway).

Evaluating products

At the first step, you can evaluate products you’ve found from a variety of sources (online databases, references, advertisements) based upon their list of features, requirements, etc. This allows you to cut a down a list of several products into a much smaller list of vendors you’re interested in dealing with. At this point, you should get in touch with these vendors, and get offers. Note that this will need quite some involvement from your side, in order to get a fair and complete offer.

This is as far as i know about this topic – here comes a lot of stuff that is usually done by executives – the most important thing for you is to get a solution which fits all the needs of your company, not just those of the IT people. This will help you to get a solution which is accepted in the whole company.

Power is down again for one row of racks, hosting the DATALINE machines – Sylon’s machines are, obviously, not affected. Outage started at 20:48, or so says Nagios. The Layer One people are already working on the issue, but i do wonder how do managed to get it this unreliable. Three consecutive power failures in one and a half day.

UPDATE:

Power is back at 22:10. Now back to repairing MyISAM tables.

UPDATE:

Power went down for all racks at about 23:00. There was a short time with power avaiable at approximately 23:02, some machines completed their power on cycle, but wasn’t long enough for Nagios to notice. It’s back up now at 00:45.

This makes it 5 consecutive power failures.

Nine Status Ticket offers some more insight into this problem. I’ve had Init7 Pikett on Phone two times, (at the first and the second major interruption), but they told me they didn’t know what happened.

Um 23:00 Uhr stellten wir einen generellen Unterbruch des Standorts Letzigraben fest. Techniker waren in weniger als 10 Minuten vor Ort und trafen einen vollständigen Stromausfall aller Räume – TIC und LayerOne – an.

Nach ersten Erkenntnissen hat eine Stromleiste in dem Raum der LayerOne einen Kurzschluss ausgelöst, welcher zu einer Störung der USV-Anlage führte. Dies wiederum führte zu einer Abschaltung der gesamten Stromversorgung.

Um 00:32 Uhr konnte die Stromversorgung wieder in Betrieb genommen werden. Sämtliche Systeme wurden erfolgreich überprüft. Sollte ein Dienst wider Erwarten nicht funktionieren, so nehmen Sie bitte mit uns Kontakt auf.

Wir möchten uns für die Unanehmlichkeiten entschuldigen. Der Rechenzentrums-Betreiber TIC hat auf Freitag, den 23.02.07 eine Stellungsnahme in Aussicht gestellt. Sobald wir diese erhalten, werden wir weitere Details über diese Plattform kommunizieren.

Update 23.02.2007 04:39: Im Raum der LayerOne fiel im Verlauf des 22.02.2007 eine 64A Sicherung einer ganzen Stromschiene aus. Ein Techniker hat dann ohne irgendwelche Massnahmen versucht, die Schiene wieder einzuschalten. Beim ersten mal war dies erfolgreich, nach dem zweiten Anspringen der Sicherung am Abend hat der Techniker erneut versucht, die Sicherung wieder einzuschalten. Leider hat der Techniker nicht gewusst, dass solche Schienen (alle Schienen dieses Typs wurden im TIC Raum nach Druck von Nine.ch ersetzt) durchschmoren koennen. Beim wiedereinschalten war die Schiene schon so stark geschmort, dass dies zu einem Kurzschluss gefuehrt hat, welcher durch die naechst groessere Sicherung vor der USV abgefangen wurde. Durch diesen Vorgang wurde die Verbindung zwischen USV und Dieselgenerator getrennt, weshalb um 23:00 der Strom fuer 2 Sekunden ausfiel und das Haus auf USV Betrieb umschaltete. Weshalb dieser 2 Sekunden lange Ausfall auftritt, ist noch nicht klar. Nach rund 6 Minuten USV-Laufzeit waren die Batterien leer und das ganze Haus offline. Nine.ch war sehr schnell klar, was das Problem war. Leider hat der Pikettdienst der Swisscom/Simag knappe 1h20 gebraucht um vor Ort zu kommen (war mit Taxi in angetrunkenem Zustand gekommen) damit er den Zugang zur Hauptverteilung gewaehren konnte um die Sicherung wieder einzuschalten!

Heute 23.02.2007 findet eine Sitzung der Mieterparteien im Haus statt um diese unbefriedigende Situation raschmoeglichst zu verbessern.

UPDATE:

Nine edited their ticket and removed the references to the drunk technician from Swisscom/Simag.

A quick call to our bandwidth provider Init7 confirmed that there was another power outage. If i counted right, this is the third power outage in the ~2 years that Sylon is located at Layer One. It happened because of power maintenance in another floor. Both power circuits went down, because they were plugged into the same UPS. That makes perfect sense.

Fortunately, there were few problems. Unfortunately, one of the problems we had required attention on site. One of the new DL360G5 decided to stay asleep. iLO wasn’t responding either. On site, i just power toggled the machine, and it worked fine. Strange.

There were a few (obvious) problems with customers still using MyISAM tables for data storage – and where myisamchk didn’t fix it, i’ve just restored the most recent backup made with bontmia.

UPDATE:
Power failed again last night for about 20 minutes. It didn’t effect all Racks (DATALINE’s machines went down, but Sylon’s didn’t). They didn’t know why this happened.

The DATALINE IT infrastructure was never designed with multiple sites in mind. However, it’s easy to migrate this, if you’re willing to dig through some docs, and had your GPOs sorted neatly from the beginning. You will need to assign some Site-based GPO’s, for WSUS Servers and similar local caches likes proxies, switch your file sharing infrastructure to DFS, make sure that replication works (still easy with just two sites), etc.

While it is easily possible for a remote branch to use our central Exchange server, thanks to Outlook’s Cached Exchange mode, this is not the case with file shares. Luckily, Microsoft has an excellent technology available to deal with this problem, called DFS. DFS can both provide replication for identical file shares, hosted in multiple sites, and also allow clients to automatically choose the closest server.

All this works using Active Directory sites. There’s also the need for some non-DFS replicated shares, containing for example heavy multimedia content.

For connection, a business ADSL connection, for example from Init7 is more than enough, because user’s don’t have to access file shares in the remote location. I’ve made several good experiences with OpenVPN for site-to-site VPN setups. A machine running Linux just offers so much more flexibility as a router for a small price – at least compared to gear from Cisco or Juniper. Using SOHO equipment like ZyXEL ZyWALLs will usually lead you to a wall, since these products often don’t have the flexibility that might be needed.

But what about infrastructure at the remote location? Yes, this is where we leave the purely technical side, and also get to deal with budget problems. Using IBM’s Standalone Solutions Configuration Tool, i’ve calculated what the minimum acceptable configuration that would be needed.

Yes, that’s about 40’000 CHF list price. I’ve tried to save as much as i can, but for a proper infrastructure, the prices can’t get any lower. I’ve tried to get comparable quotes using Dell’s Online Configurator, but didn’t come out much cheaper.

Getting money for this infrastructure is going to be a lot of work, but that’s also part of my job.

Without money, it is impossible to build a setup which is designed to last. And good equipment is expensive. You can save on a tape library, replacing it with a tape drive. But then you get the situation that backup reliability drops since administrative personnel never really cared about proper IT infrastructure. You can buy cheaper servers without remote console capability, risking long outages. You can save on redundancy, and remove the second PSU, further enhancing the risks for outages.