I had just finished a routine shift, handing off the on-call baton to Ashley on the morning crew when the call came in. I literally had my jacket on and was walking out the door, but the first words out of Ashley’s mouth made me stop and take a seat to listen and maybe be helpful.
Ashley’s perfectly competent, but first shift is where new people go: absolutely nobody wants to be in first thing in the morning, so the new person gets the gig, and for most of their shift the senior staff is in to back them up.
For two hours, though, from six or so to eight, they’re mostly on their own; which is generally fine, it’s the lowest call volume of the day and it gives the newbies a taste of what it’s like to be on your own. But it means that first thing in the morning is the worst possible time for something to go wrong.
“NOC, this is Ashley,” said Ashley, and then, “Calm down, speak slowly, tell me what’s wrong.” She reached for a notebook and pen to jot notes with; I usually open a ticket at this point and type directly into the input field, but some people think better with a pen in their hand.
This, by the way, was the point in the call where I turned around, hand still on the doorknob, and stepped back into the room, quietly taking a seat in one of the guest chairs.
“Okay,” said Ashley, “I see that temperatures in the server room there are elevated, can you check the server room for me?”
‘Elevated’ was an understatement. The servers in that room were showing internal temperatures which would be well into the red, if the monitors were on, which they weren’t.
There was a hurricane in south Florida; this wasn’t the most unusual occurrence, but it meant that the site there was supposed to have been shut down; everybody was supposed to have gone home and the server room taken offline. I’d watched Henry from swing take the monitoring offline when I arrived last night, because everything was supposed to be down.
“What do you mean, wet?”
I reached over past Ashley and pressed the ‘speaker’ button on the phone.
“This is Ken, the graveyard tech,” I said, “I’m here with Ashley, I put you on speaker so we can both hear you. Can you repeat what you just said?”
Ashley looked up at me with this mix of annoyance and gratitude written across her face.
“There’s about an inch of water on the floor of the server room,” said a woman on the other end of the phone, “And all the servers are beeping.”
That would be the internal temp alarms. Ashley was bringing the alarming back online so we could see what was going on with these machines; I wouldn’t have done that -- heat will cause all sorts of problems, so it can look like you’ve got a lot more problems than just ‘the servers are overheated.’
“Does it seem particularly warm in the room?” Judging from the numbers, I’d have expected her to find a small fire going in there.
“No, not really, it’s just... wet.”
Ashley had called up a search of issues related to this particular site in the ticket database; sure enough, last night, one of the factors in deciding to shut down the site was that there was a leak in the server room, and water was running down one of the walls.
There was a followup note from a different tech noting that the site personnel had called back in and reported that it was fine, that they’d taken care of it.
Obviously, that report had been overoptimistic.
The sounds of servers beeping -- some of them madly beep-beep-beeping away, some of them producing a long steady tone, altogether a cacaphonic nightmare of badness -- came over the speakerphone in bursts, the sound cancelation making it even more chaotic.
“Shut them down,” I said quietly. “They’re supposed to be offline anyway, and they’re not doing anybody any good by overheating.”
Ashley nodded.
“Okay,” she said, “Can you do me a favor, I’m going to shut these servers down from this end, I need you to go around and make sure they’re all unplugged after I do that. I’ll read out the names of the machines as I see them finish their shutdown sequence, you should find the names on labels in the upper left hand corner of each machine’s front and rear; I just need you to remove the power cables from each machine, and then make sure the cables don’t end up dangling into the water, can you do that?”
“Uhhhh...” The hesitation was normal, on-site personnel are mostly really leery of screwing with the servers, which they regard as voodoo magic. What she said next was not normal.
“I can’t read the labels,” she said, “without unwrapping the plastic.”
“Plastic?” Ashley leaned over the speaker, as though she could make it make sense by looking through the phone.
“Yeah,” the woman at the other end of the phone said, “All the servers are wrapped in plastic sheeting. Frank did it last night, because of the leak.”
“You... Frank... wrapped them in...?” Ashley sounded like she was having trouble formulating a coherent question. I knew how she felt.
“Yeah, we figured... if they couldn’t get wet, we wouldn’t have to shut them down...”
“Okay,” said Ashley. She flexed her fingers, rolled her shoulders, took a deep breath. I could recognize a self-calming routine when I saw one and stayed quiet.
“Okay,” she said again, and her fingers started flying over the keyboard. “I’m just going to shut them all down, and when I see them all offline, I’ll have you pull power on all of them, okay?”
“Okay,” said the woman at the other end of the line.
“EPO,” I said, quietly. It’s pronounced “ee-poe.” Ashley looked up at me sharply, then her eyes widened a bit and she nodded.
“Actually,” she said. She looked down at the screen, where about half the servers were shut down. “Actually, I want you to look for a big red button on the wall, labeled ‘Emergency Power Off.’” Can you see it?
“Ahhh... yeah, I see it,” said the woman. “You want me to hit it?”
“Go stand by it... there’s a clear plastic cover over the button. Lift that cover, and when I say, go ahead and hit the big red button, Okay?”
“Okay...”
We watched as the last of the servers ground through its shutdown routine and hung halfway through. Ashley looked up at me, and I shrugged. “Heat,” I said. “Shut it down.”
“All right, Barbara, go ahead and hit the button.” Ashley and I were both watching the gauge that showed power consumption by the server room; as soon as Barbara hit the button, it should drop to zero.
Instead, it spiked, jumping way up into the red for a second, and then dropped to zero. At the same time, there was a sound like a strangled scream and a splash from the other end of the phone.
“Barbara?” Ashley leaned into the phone again, a sudden edge in her voice. “Barbara, can you hear me? What’s going on?”
Nothing from the other end of the phone; then the line went dead.
“Fuck,” I said, and leaned back in my chair.
Ashley’s fingers went to work on the keyboard, and seconds later she was dialing the number for Dade County emergency services. Not that they didn’t already have enough to do.
Ashley’s perfectly competent, but first shift is where new people go: absolutely nobody wants to be in first thing in the morning, so the new person gets the gig, and for most of their shift the senior staff is in to back them up.
For two hours, though, from six or so to eight, they’re mostly on their own; which is generally fine, it’s the lowest call volume of the day and it gives the newbies a taste of what it’s like to be on your own. But it means that first thing in the morning is the worst possible time for something to go wrong.
“NOC, this is Ashley,” said Ashley, and then, “Calm down, speak slowly, tell me what’s wrong.” She reached for a notebook and pen to jot notes with; I usually open a ticket at this point and type directly into the input field, but some people think better with a pen in their hand.
This, by the way, was the point in the call where I turned around, hand still on the doorknob, and stepped back into the room, quietly taking a seat in one of the guest chairs.
“Okay,” said Ashley, “I see that temperatures in the server room there are elevated, can you check the server room for me?”
‘Elevated’ was an understatement. The servers in that room were showing internal temperatures which would be well into the red, if the monitors were on, which they weren’t.
There was a hurricane in south Florida; this wasn’t the most unusual occurrence, but it meant that the site there was supposed to have been shut down; everybody was supposed to have gone home and the server room taken offline. I’d watched Henry from swing take the monitoring offline when I arrived last night, because everything was supposed to be down.
“What do you mean, wet?”
I reached over past Ashley and pressed the ‘speaker’ button on the phone.
“This is Ken, the graveyard tech,” I said, “I’m here with Ashley, I put you on speaker so we can both hear you. Can you repeat what you just said?”
Ashley looked up at me with this mix of annoyance and gratitude written across her face.
“There’s about an inch of water on the floor of the server room,” said a woman on the other end of the phone, “And all the servers are beeping.”
That would be the internal temp alarms. Ashley was bringing the alarming back online so we could see what was going on with these machines; I wouldn’t have done that -- heat will cause all sorts of problems, so it can look like you’ve got a lot more problems than just ‘the servers are overheated.’
“Does it seem particularly warm in the room?” Judging from the numbers, I’d have expected her to find a small fire going in there.
“No, not really, it’s just... wet.”
Ashley had called up a search of issues related to this particular site in the ticket database; sure enough, last night, one of the factors in deciding to shut down the site was that there was a leak in the server room, and water was running down one of the walls.
There was a followup note from a different tech noting that the site personnel had called back in and reported that it was fine, that they’d taken care of it.
Obviously, that report had been overoptimistic.
The sounds of servers beeping -- some of them madly beep-beep-beeping away, some of them producing a long steady tone, altogether a cacaphonic nightmare of badness -- came over the speakerphone in bursts, the sound cancelation making it even more chaotic.
“Shut them down,” I said quietly. “They’re supposed to be offline anyway, and they’re not doing anybody any good by overheating.”
Ashley nodded.
“Okay,” she said, “Can you do me a favor, I’m going to shut these servers down from this end, I need you to go around and make sure they’re all unplugged after I do that. I’ll read out the names of the machines as I see them finish their shutdown sequence, you should find the names on labels in the upper left hand corner of each machine’s front and rear; I just need you to remove the power cables from each machine, and then make sure the cables don’t end up dangling into the water, can you do that?”
“Uhhhh...” The hesitation was normal, on-site personnel are mostly really leery of screwing with the servers, which they regard as voodoo magic. What she said next was not normal.
“I can’t read the labels,” she said, “without unwrapping the plastic.”
“Plastic?” Ashley leaned over the speaker, as though she could make it make sense by looking through the phone.
“Yeah,” the woman at the other end of the phone said, “All the servers are wrapped in plastic sheeting. Frank did it last night, because of the leak.”
“You... Frank... wrapped them in...?” Ashley sounded like she was having trouble formulating a coherent question. I knew how she felt.
“Yeah, we figured... if they couldn’t get wet, we wouldn’t have to shut them down...”
“Okay,” said Ashley. She flexed her fingers, rolled her shoulders, took a deep breath. I could recognize a self-calming routine when I saw one and stayed quiet.
“Okay,” she said again, and her fingers started flying over the keyboard. “I’m just going to shut them all down, and when I see them all offline, I’ll have you pull power on all of them, okay?”
“Okay,” said the woman at the other end of the line.
“EPO,” I said, quietly. It’s pronounced “ee-poe.” Ashley looked up at me sharply, then her eyes widened a bit and she nodded.
“Actually,” she said. She looked down at the screen, where about half the servers were shut down. “Actually, I want you to look for a big red button on the wall, labeled ‘Emergency Power Off.’” Can you see it?
“Ahhh... yeah, I see it,” said the woman. “You want me to hit it?”
“Go stand by it... there’s a clear plastic cover over the button. Lift that cover, and when I say, go ahead and hit the big red button, Okay?”
“Okay...”
We watched as the last of the servers ground through its shutdown routine and hung halfway through. Ashley looked up at me, and I shrugged. “Heat,” I said. “Shut it down.”
“All right, Barbara, go ahead and hit the button.” Ashley and I were both watching the gauge that showed power consumption by the server room; as soon as Barbara hit the button, it should drop to zero.
Instead, it spiked, jumping way up into the red for a second, and then dropped to zero. At the same time, there was a sound like a strangled scream and a splash from the other end of the phone.
“Barbara?” Ashley leaned into the phone again, a sudden edge in her voice. “Barbara, can you hear me? What’s going on?”
Nothing from the other end of the phone; then the line went dead.
“Fuck,” I said, and leaned back in my chair.
Ashley’s fingers went to work on the keyboard, and seconds later she was dialing the number for Dade County emergency services. Not that they didn’t already have enough to do.
No comments:
Post a Comment