As you may have noted, configuration was standard through the department, but not through the company, as two different departments often had different policies and implementation. So the company went on and imposed "THE" standard, through an acronym-based-name project, that I will refer to here as Project SOB. At this time, the company moved all it's IT employees into one of it's subsidiary, and sold it to a big-time IT firm. Thus, maintaining our jobs was equating to the company becoming a customer of the IT firm. But as Project SOB was taking form, it was clear that all management and decision making involved in this project was made by the higher-up of the IT firm, without seeking knowledge from the people already on the field (they made us fill repetitive forms about what software was installed on each individual computer, all with version info, forms that we had to verify for mistakes after they type it in a database, but they failed to listen when we were trying to raise real issues).
If I stick to software deployment issues, here's what went bad. One part of Project SOB was to replace all desktop computers in the company to have a standardized computer base for all the company, hardware and software. Nothing wrong with that, of course. But the "standard image" of a SOB computer was far from being perfect. Remember that Clipart quirk I told you about earlier? They didn't do it. Antivirus software was configured with (poor) default configuration. And on and on. They haven't been tested at all (not by me anyway, even if I clearly and definitely required so) and were pushed on a rush by "higher-ups". I previously made arrangements with the deployment team to install 3 or 4 stations, so I could at least test them in a real environment before giving it my go. Then I wanted the deployment to be done during work hours, where it is easy to speak with the user and figure out that she has a folder on her C: drive where she keeps all her work (even if it is against policy), or to transfer existing e-mail (some people used more than one e-mail software, as the company standards moved from none to Lotus Notes to Netscape in less than 18 months). Instead, they pushed the installation of 88 desktop computers over the week-end, figuring they "should be all done by Monday, since they're pre-configured and all." As we were deploying machines, we were finding all the misconfigurations (like that SMS client they installed, because they planned to have SMS servers up in 6 months. Because no SMS server was available, this caused the machines to hang solid for about 30 seconds every time at start-up. A year later, there was still not a single SMS server installed). So new bugs found meant that we had to go over each previously installed machine to fix them too. That delayed the deployment a lot. And when this was done, we (my boss and I) had to go over each machine again to put the "local settings" in the machines (template folder on shared network drive for MS-Office, local network printers, etc.). That was Sunday night, 9PM. We knew we had a big night of work (after being there for the past three days), but we knew we could manage to get all 88 PCs to be fully functioning by Monday morning. As we were doing or third and fourth machine, Murphy's law proved true once again: the network went down. This is a bummer, because when you put network information on Windows, like a printer share name, it will always try to connect. And if it can't find it, it will refuse to install. We called network support (a voice mailbox), which meant that the problem could only be fixed in the early business hours of Monday morning. In a way, that pleased us, because we knew that we could go home and sleep. But on the other way, we knew the shit would hit the fan the morning after...
Monday morning, 84 people (remember, we could at least do 4) were not able to work for the whole day (we had to attend emergency meetings in order to explain this big mess instead of being on the floor to fix it), and the Chief of the Department wants blood. We calmly explain to him the fiasco that happened over the weekend, how we valiantly fought against it, how we lost (politics, my boss is bigger than yours), how we did everything we could to clean the mess. This story continues in the next chapter, "And the ugly", because it does get ugly. I'm sorry to cut this story clean, but I would like to focus this chapter on software installation and deal with the "brown stuff" later.
So that leads me a few months later, when my department computer base is once again under my control (after much proscribed-but-necessary-for-stability tweaking of the SOB standard), came the time of software upgrades. I have to admit we were due indeed. Office 97 had two service packs out by then, and it became accepted by the SOB standard to be the new office suite (we were on Office 95 until then). Antivirus went on a major upgrade release as well, and some utility software were available to all users in the company, thanks to company-wide site license or freeware (stuff like WinZip, Acrobat Reader, ftp tool, etc.). So they wanted to try to do these installations automatically, with as less human intervention as possible. A noble cause, I must agree. They approached the problem with an in-house utility interface on top of WinInstall (if I remember the name correctly). I wasn't involved in the creation of the software packages that were to be distributed with this tool (even if I requested it), so I don't how they are created. But for debugging them (because they were poorly tested again, some people never learn) and going up to identifying what causes the problems in his packages (because he had no clue why it was acting up), I could deduct that it was script based. It is probably a sequential list of all single items to be done in order to install a specific software. Each single file, directory or registry entry created or deleted (because you could use it also to de-install software, useful when performing automatic software upgrades). The list was then parsed and the tool performed the actions specified in the list. Since it was sequential, you can guess that some of the bugs were relating to some file trying to be copied in a folder that does not yet exists, which resulted in the file failing to be copied. The installation of a particular software, Windows Media player caused the computer to freeze sometimes (most of the times), but rebooting the computer and re-launching the install process fixed it (it always ran fine the second time). For some reason, the guy building the package could not make a working one for McAfee (I offered him some help, after all, I knew quite a bit about this product, as you can see in my paper "Virus protection in a Microsoft Windows network, or How to stand a chance"), so he used the tool to launch the original setup files (with default configuration, as this was the SOB standard). But for the WinInstall tool then proceeded immediately to the next item in the list, even if the McAfee install was not finished yet. So He had to include in the list a Notepad file that would pop-up, and that we had to close when the antivirus software was done installing. Closing notepad would let WinInstall to continue his tasks. As you can see, this was more of a semi-automatic system than automatic.
We upgraded a whole building this way (about 1000 PCs, ten people to do it). It was a pain. Other people in my position encountered bugs different than mine. All because the only testing done was done on a clean test machine, not an in-the-real-world-production PC. And because this tool wasn't efficient at all, even if I have to disagree with the project reports on that one. The worse is, in another part of the big monster that Project SOB had become, another team was working exactly on the same thing as us, with a different tool that worked pretty much in the same way as ours. They used the other solution to upgrade previously deployed PC (and encountered similar problems), while the newly deployed machines had the newer versions on them.
Another funny thing with Project SOB was with the new browser/e-mail standard. They chose Netscape to run internal and external e-mail, but they didn't want all the features available. They also did some customization so it would be pre-configured to the right mail server. What they did was one of the strangest things I've seen so far (and I've done quite a few myself): the installation program they crafted for this one was a self-extract .zip file, containing the original Netscape setup files along with a slew of batch files. The self-extract would launch one of these batch files, which took care of determining OS (95 or NT), then to create some folders where the user profiles would be kept, along with .bin file containing the server addresses. Then it would proceed to install Netscape like it normally would, then some other batch file finished the job by cleaning up the parts we were not to use. These batch files were very elaborate, and they took some things from granted in the batch files code. I found this out because this package wouldn't install on a particular machine. The batch file thingy would bust me out before the software would really install, without giving me any clue as to what was going on. So using WinZip, I opened the self-extract file and open the batch files one by one, trying to figure out what was happening. It turns out that somewhere they hardcoded C:\TEMP, and the machine that I had problems with, the temp folder was C:\TMP. Silly, isn't it? They should have used the environment variable instead.
2. The good...
4. And the ugly
Table of contents