This chapter may seem a bit off-topic, but I decided to include it here because it causes great interference on smooth and efficient software and hardware installation and configuration management: internal politics. And this topic is wide in meaning. It's the project managers who plan *our* deployment without consulting us to prevent and avoid potential problems, it's the push from higher up who wants the most possible machines deployed in the shortest amount of time, no matter what the level of quality of these installation is (because it leads to a bonus for the Project manager, but not a dime more for the guys who make them look good by cleaning up the mess), because some group is under union and the other is not (more on that below), the fact that managers are not from the IT background (more later on that one also), you name it, you got it. Here, I'll pass under silence some incidents with one sick-minded individual that also happened to be one of the managers involved in the SOB project. He caused a disruption in Internet service for over 30 of my users, all because of his poor planning. A new machine also meant a new network connection, switching from static IP that needs approbation for Internet access to a DHCP address that was allowed access no matter who. As they deployed 88 machines, the 30-or so laptops were not received yet, but the network guys planned them for deployment, so the Internet access for the static IP's was removed. So, on Monday morning, add to the 84 non-working PCs, 30 laptop users who are sure glad that they can still work on their machines, but they have lost Internet access in the process. Instead of fixing this relatively simple problem, he tried to get me fired with false accusations. These users got their Internet access 1 month later, after many complaints for the service to be restored, when they got their new laptop with a DHCP address. I had other problems with this individual, so I consider this to be a single event, while the rest of this chapter is the kind of brown stuff that happens all the time in big corporations. As for this sick-minded guy, he got promoted to the position that would have made him my boss, had I not resigned for another job a month before.
Back to our botched weekend deployment. The deployment team comes from another subsidiary from the same BIG company, and these guys have union. That means that their jobs is to get the machines out of the box, put them on the desk, connect and test the new network connection, and disconnect the old PC that will be picked up a few days later in order to have a chance to transfer over data we may have missed at deployment time. Pretty simple, repetitive. The networking stuff is their turf, I don't argue with that. But to go faster, we would have liked to help them put the machines on the desks, so we could start our tweaking earlier. But we couldn't, because that would "steal job" from these guys, although they were working overtime big time, and really wished they'd be home early (a comprehensive feeling). Unionizing is a good thing, but today's unions are only a tool in the hands of the bosses to get more squeeze from the employees, and this is one fine example. The union argument is that if there is simply too much work for them to do, then the employer simply has to hire more people. But in the meantime, it is the employees themselves who are putting nearly impossible time sheets for months, just because they're shortstaffed and nobody can give them a hand. And then they send us to "teamwork oriented" meetings to increase our productivity. Sheesh. At least, they agreed to help us to transfer the files from the old machine over to the new one, because we were only two guys doing it, and they had to wait after us to disconnect the old machines. Since they weren't feeling like spending the whole weekend there (which eventually happened anyway), they helped us do it. After all, these guys were victims just as my boss and I. All in all, we were about 15 people total (guys in the field and managers alike) failing miserably what 4 of us (my boss, 2 other guys and myself) had successfully achieved numerous times (at one time, the department was split up in 7 different sites).
On Monday morning, like I said, shit hit the fan big time. Head of the department wanted... a head. So him, VP Tech Support of Big IT firm, Boss of the subsidiary that was delivering the PCs, Project manager, network manager (the sick puppy), my boss and I are all gathered together for an emergency meeting in order to figure out what went wrong (as all Hell was breaking lose outside the doors, because over 100 people couldn't work properly, if at all). So we spent most of the day explaining everything, how the systems were poorly configured, lack of testing, and most of all our repetitive tries to prevent all this by raising potential issues. At the end of the day, that was interpreted as "lack of communication between all people involved", meaning we're as guilty as them because *they* didn't listen to us(!), and that another meeting would follow between people involved in the project in order to identify the root causes of all this, and how it can be avoided in the future. No one got reprimanded for all this crap. And they agreed that the fact that the network went down was not predictable, and they concluded that in the long run, this is what prevented us to get the machines in working condition in time (!!!).
The story behind this was that the Head of IT was getting a bonus if a ridiculous number of PCs would be delivered on a certain amount of time. How many? Well, the first phase (in which I took part) was to install 400 PCs in a matter of approximately 2 months. Our 88 were part of that 400. This is why it was rushed like that. Then Phase 2 was supposed to be able to deploy 1000 machine per week, on a 40 000 computer base. You read it right; it's not a typo. Do I need to precise that Head of IT "resigned" somewhere in between Phase 1 and Phase 2? The massive deployment progressed at a much slower rate (but I don't have numbers), and they had much bigger problems at other sites, where the IT personnel was less experienced.
Another problem with corporate politics is power. Not the power to do things, the power represented in numbers. Power in headcount (at this stage, they forgot we are human beings), power in budget (the bigger, the better, even if many of these costs could have been avoided and would actually saved money for the company). But not the power to do things. Example (BTW, all of the examples I brought in this document are any other are real and factual, even if some of them may appear quite incredible): neither the custom Netscape package or the base image for the new machines being deployed are configured to use the proxy server in order to access the Internet. This means that after the "automatic" installation, we still had to go and configure it properly in order to work. When I realize this, I call up the SOB e-mail team (they were taking care of the whole Netscape software):
ME: Hi there. I just noticed that although the standard configuration goes into great lengths to have the software self-configured in order to easily access e-mail profiles and all, but you guys forgot to put the proxy information.
SOB rep: I'm sorry, we don't deal with the proxy, it's not part of our mandate. For proxy problems, call the networking department.
ME: But it's a configuration that you have to put IN Netscape for it in order to work. The proxy itself doesn't have any problem, it's Netscape who's misconfigured.
SOB rep: I'm sorry, I can't help you. Please call Networking.
Well, how could I win with these kinds of arguments? No need to say I did not call networking. I would have sounded like an idiot dweeb if I'd done this, because they would have told me "Call the SOB team", of course. Again, the problem here is not technical, it's lack of leadership from the different groups to take their responsibilities. Then again, who said that power means leadership?
Another example of lack of leadership in a project: 6 months after they started deploying Windows 95 PCs, they changed the SOB OS standards to Windows NT (somehow, I managed to predict that 6 months earlier, what a psychic!). Now get this: the SOB standard specified no Local Administrator password on the machines, leaving it to the tech support and NT admins to set one of their choice, if they felt like putting one(!!!). I cried foul at this nonsense, but that didn't change a thing. You would say that this doesn't make these machines more vulnerable than a Windows 95 machine, and you would be right. But why switch to NT if you don't intend to use its capabilities? So of course, 6 months later, it is a mess, there are as many passwords as there are people in the tech workforce, each password being something hard to guess like agf$vm87 (of course, now that they are security conscious), and no one can practically login on a machine as Administrator on the first try (or even the first 5 tries, sometimes not at all). This is a bummer, because you often have to log in as Administrator to install software or to modify configuration settings as part of the troubleshooting process. It is then that they decided to put standardized local Admin passwords on building basis (each building have its own Admin password), but they implement it on the machines only as the techs are called to work on these machines as part of the support duties. That means that a machine whose user doesn't call tech support for problems won't be updated. So much for standardization. It looks nice on paper, but in the real world, nothing is changed. The problem here is that the Mother-company Security department should have taken ownership (pun intended) of the configuration policies of these machines. Computer security is the only thing they haven't outsourced, and what they had was a Security group not involved in company-wide projects, and an IT firm that administers servers and domains with no concern over security because it is not their mandate. Lack of leadership, seems like no one wanted that hot potato.
I also identified as part of the "politics" shenanigan the lack of IT knowledge on the part of project managers. And it takes more than getting a MCSE to claim to have knowledge. 6 months before they started Phase 1 of Project SOB, they presented a document that detailed every single part of configuration that will make the SOB Desktop Standard. One of these configuration items was quite ironic/moronic (you choose): it was specified that the PCs be delivered without any floppy disk drive, in an effort to protect the company's computers from computer viruses and preventing rogue employees to get confidential information out. Did I tell you that these machines are all connected to Internet, with full browser-mail-newsgroup-ftp capabilities (and more)? CQFD! Of course, they changed this one, as this would have been a ridiculous implementation. Let's just say that it was common practice for the employees of that big IT firm to forward virus hoaxes, even after numerous reply-back on my part mentioning that this was a false alert, all with links to the hoax pages of McAfee and Symantec. I guess this one explains the rest.
Examples like this are numerous, and you can see even more by checking the List Of The Day on Dilbert's website (www.dilbert.com).
3. The bad...
5. InstallRite
Table of contents