The Trouble with Canceling Flights…
At any given moment, there are only five people in the world who have the authority to cancel United Airlines’ flights – four controllers and their shift-manager. Out of nearly 100,000 employees, only five can cancel flights. So you see, canceling a flight is serious business.
More accurately, I should have said five people and one software application. That application is Artemis. As a result of the most recent corporate reorganization, I had been saddled with the title of project manager for Artemis – the computer program that decides which flights to cancel during “irregular operations.”
That day, back in May of 2000, the color of the sky could have been described as Mediterranean blue when I drove in to work. The weather forecast had warned of a developing thunderstorm, but I was too new to irregular operations to understand that bad weather meant I had to be worried.
Usually, my pager would go off whenever there was any disruption in running Artemis. This had already happened twice in just the last few days. But on that day, for reasons unknown to me, my pager had not buzzed. When I sauntered in to work (usually between 9.30 and 10.00) there were three frantic messages on my voice mail, including one from our Managing Director’s secretary. My cubicle neighbor said that my own department secretary had looked for me twice. I learned that a major run of the cancellation software Artemis was being planned, and my MD wanted me to personally make sure that things went smoothly this time. This time, he wanted me to make sure that he didn’t have to deal with any irate phone calls from our internal clients.
There is a lot of misconception around how software systems work. For any complex system to succeed, every part has to work in synchrony -- the mainframe computer in the back, the databases, the communication links, the network, the application itself, and the front-end user interface. During the previous two attempts to run Artemis, the communication piece between the software and United’s mainframe computer had broken down. But because I am in charge of the application and the front-end, I had to face the music no matter which component failed.
I skipped my first cup of office coffee (the prerequisite eye-opener for me to function at all) and ran up to the third floor of the other building which is where our Operations Control Center (the OCC) is located.
Artemis is run in the OCC. The operating head of the OCC floor is the shift manager. He gets to sit in a corner, overlooking the sea of desks of the people who literally run the airline. In the OCC there are tinted plastic sheets on all the glass windows, giving the place a darker feel. The low lighting, I was told, helped give a calmer feel to a place. Calmness is a prerequisite because the OCC is the focal point of the central nervous system of the entire airline. All operational decisions are taken there and implemented there, and then the rest of the world is informed.
The shift manager on the floor reports to the head honcho, the Director of the OCC. Dan Johnsdell was the director, the emperor of the OCC as it were. If you have seen the movie Apollo-13, then you remember the role that Ed Harris plays – that of the head of mission control. Dan Johnsdell was that same guy for our airline. He might have been described as hard-boiled in Western novels. Possibly due to the unrelenting pressure of his job he was just as likely to be found outside the building smoking as he was to be found inside the OCC.
Johnsdell had started working for the airline before I was born. When I was still in seventh grade, running around in shorts in Madras, he had become the director of the OCC. The very first time I had been introduced to him, he asked me for a clarification about something I had said regarding airplane taxi times. Later, I realized that this had been a sort of a quiz. Fortunately for me, I had blurted out the right response, and based on that sole data point Dan had concluded that I knew a lot about airline operations. He seemed to like me, in as much as Dan can be said to have liked anybody. But I had been around long enough to have learned that this kind of liking evaporates quickly when the software systems that my team supports so much as starts to hiccup.
When I entered the OCC, Dan didn’t acknowledge me. I stood a few feet away from Ed, the controller who would be running the program.
“Ram, it better work today,” one crew scheduler told me, leaving the consequence phrase of the ultimatum unspoken.
“Oh, you bet it will!” I assured him. I had been around long enough to learn that if I didn’t exude confidence, these guys would eat me for lunch and burp.
Flight cancellations result in a major disservice to the passengers, of course. One rule of thumb is that for each flight cancellation, 100 passengers are disrupted. The airline can curse the weather, but sometimes there are no alternatives to canceling flights. In the tightly-fought battle for statistics among airlines, a cancellation really hurts. A cancellation is like not even attempting to answer a multiple-choice question in an exam. You know you are going to score low, and you can only pray that your competitors are having the same problems you do.
Dan and the controllers went into a group huddle to have a discussion. I stood by the side and hoped that someone would update me on what was going on. Since I had inherited the program only recently, I didn’t know the first thing about fixing it when things went wrong. I shouldn’t have been there alone. I had assigned two support people specifically to Artemis. Amin Rahman was the team lead and Jinwen Lu was the programmer, but they hadn’t showed up at the OCC. I suspected that they had not come in to work yet. I didn’t have a cell phone, so I looked for an empty desk in the OCC to call them. The OCC phones are complicated, and the one I found had over 100 buttons. It looked more like a music mixer than a telephone and I was afraid of punching the wrong buttons. But the situation was getting desperate, so I lifted the receiver, managed to find the dial tone, and left messages for both of them. As an added precaution I called another person in my group and asked him to leave Post-It notes on both their computer screens, asking them to come to the OCC.
“We’ve decided to do the sag at 11.15, Ram,” one of the aircraft routers who was passing by told me. It was not yet 11.00. A few minutes later, I saw Amin walking into the OCC, all nervous. I know it makes me a lesser man, but I was glad to see him. If something broke, he would be the one sweating.
Johnsdell walked out of the OCC. I was sure that he was off for a smoke. Sensing a break, I updated Amin and rushed out to the nearest vending area for a cup of much-needed coffee, reminding myself that I had to at least attempt to overcome this addiction to coffee.
When I returned carrying my cup of coffee, the controllers had already started going through their pre-cancellation checks. Apparently, they were starting earlier than 11:15. I noticed one of the controllers, Ed, point me out to the shift manager. Ed would be the one running Artemis. I gave a weak smile, hoping that I looked confident, because my stomach was knotting itself inside.
“Three minutes. We are starting the sag in three minutes.” The shift-manger wasn’t shouting, but he was able to throw his voice to every corner of the room. “We are taking out 160. If you have any last minute requests, get them in right now!”
Translation: They would be running Artemis in 3 minutes, and they would be canceling 160 flights. Last minute requests are usually of two types: Please cancel this flight because I am not able to find an aircraft or a pilot for it; or please don’t cancel this flight because there is a VIP or a big group of school kids flying in that one.
I moved a little closer to Ed’s desk. I could overhear the Denver airport manager on the phone pleading with Ed. Ed played the hungry God, willing to be appeased so long as there was another sacrifice. “Gimme another one then,” he told Denver. They were sparing one flight and canceling another.
Sagging 160 flights was a huge reduction. It was almost 10% of our day’s operation. It meant over 16,000 irate passengers all over the country. If, for any reason, Artemis crapped out on us, all of the 160 flights would have to be cancelled individually. Manually. And I would have hell to pay. I looked at Amin and saw fear in his eyes. He knew.
“Alright everybody, stop working. We are canceling now!” The shift manager raised both hands above his head. His striped-shirt added to the baseball umpire image. He then brought his right hand down and pointed to Ed the controller. A little showmanship never hurts, if you can pull it off. Ed had already marked the 160 that he wanted cancelled. He clicked a red button on his screen, and he was done.
If things went well, in under twenty minutes, in airports from Anchorage to Zurich, from Omaha to Osaka passengers who were paying close attention to the arrival/departure monitors would see their flight’s status change. Many would groan when they noticed the letters CNCLD pop up next to the flights they were hoping to take.
While Artemis was busy chugging away, most of the OCC folks would not have anything to do for a little while, so they started browsing the Web. Amin logged into a free workstation and he opened a screen so that we could monitor the process. I calculated that it would take 15 minutes at the max for the 160 cancellations to percolate through the systems. There was nothing for us to do but watch the screen with a horrified fascination.
To make the wait bearable, Jinwen had added a status message after every 3 minutes – an indicator to say the cancellation process had not died on us. Two of those messages popped up, six minutes gone. Amin informed me that there was a 20-minute time out for this process. If the cancellations were taking more than 20 minutes, something inside had gone wrong and Artemis would be aborted. The third message showed up, 9 minutes up. We then noticed that Unimatic, the mainframe was painfully slow. The fourth message appeared, 12 minutes up. I was hoping it would to complete any minute now. I felt cold in my forehead and touched it. The air-conditioning was chilling the beaded-up sweat.
15 minutes up, nothing yet. Then the sixth status message showed up, still nothing. It had already been 18 minutes. There wouldn’t be a seventh message. The whole thing would time out, and the cursing and finger-pointing would start. These controllers would have to perform the cancellations manually, painstakingly, one at a time, which would take hours. I was mentally rehearsing a way to break the news to Johnsdell.
And then, miracle of miracles, calm as everything, the cancellations went through. Thank you, sweet Jesus. To me, it felt as if the OCC chamber was physically depressurizing. I walked over to Johnsdell, made my voice sound casual, and mentioned to him that the cancellations were done and that the recovery would be done in another five minutes. Johnsdell nodded and walked out. To smoke again, I was sure. I imagined that the relief I was feeling was exactly what smokers hungered for.
The room buzzed back to activity. Pilots had to paged or called and given their new assignments. Hundreds of flight attendants had to be tracked down. When things work, support people are completely ignored.
Amin and myself were not needed anymore. We walked out, not speaking. Both aware that we had been incredibly lucky, both dreading the next bad-weather day. From the glass-paneled spine connecting the OCC building to ours, I glanced out. One controller, referring to what was brewing had said, “It’ll come up to be a real thunderstorm.” I knew that soon the sky would start to turn gray.
But right then, it was still the same Mediterranean blue with lots of coiled cirrus clouds floating peacefully. It looked just like any other glorious mid-May day.
Ram Prasad
October 2002