Government Data Center
Max stood up a the desk and paused his typing to rub his hands together as they felt sore from the cold air of the server room. He looked over at Omori, who had rolled up his sleeves and was typing commands into a terminal with a focused intensity.
“Do you have the latest telemetry?” he asked Omori, barely louder than the loud hum of the server fans.
“Yeah, I have it in one window,” Omori turned his monitor at a slight angle so Max could see. “It’s still stuck in a bootloop without any ability to recover. Each time the load balancers try to redirect traffic, it just crashes again.”
“Damn it, something like this has never happened before. It’s like a one-in-a-billion crash. There’s nothing in our playbook for this. We set up our systems to be resilient against hardware failures, DDoS attacks, even power outages. But this... this is something else entirely.”
Max rubbed his temples and walked over to the primary console, the one that controlled the entire operations of their Olympus data center. For such an important computer, its interface was surprisingly minimalistic; just a tiny CRT with a keyboard at least a decade old. Several of the the keys had been worn smooth from years of use.
“If we shut down the entire system, we can restart it piece-by-piece to avoid the bootloop, but that will take all day,” Omori suggested.
“The outage has already been going on for nearly 24 hours,” Max replied. “I agree we need to fix this faster. I think we need to try something riskier.”
“Like what?” Omori asked, looking up from his terminal.
Max began typing rapidly on the keyboard, trying to balance her urgency with the need to not make any mistakes.
“I can run an override here as superuser. I’ve got the system permissions to do it. Then I can directly connect to each of the load balancers and manually tell them to drop the current traffic. Then, we can restart each server one at a time so that traffic going forward will be able to run smoothly.”
“You’re just going to run a bash script? But if you make any mistakes, you’d brick every single server here!” Omori exclaimed.
“Everyone, all our customers, are depending on us. If we don’t get this fixed soon, all of our clients will leave anyway. Either way, the only option we have now is to act.”
He took a deep breath and hit enter on the script, executing it. There was a tense silence in the room as they saw the individual server LEDs turn off and begin blinking one at a time, indicating they were restarting. The red status lights were beginning to blink green, indicating that they were back online and working properly.
There was a sudden ping on his phone. Then a second. The notification traffic began flooding into his phone, confirming that the script had managed to restore service to all of their clients.
“We did it,” Max let out a sigh of relief. “Now we just need to monitor this for a while and work on that post-mortem report.”
He pulled out his phone and began to look at his notifications again. One of them was a breaking news alert involving their data center outage. He tapped on it to open up the livestream video.
“...will be seizing this moment of national crisis...”
President Thompson’s voice was grave and slow. He was sounding serious, more so than usual.
“Our great outage has shown us that private actors cannot be trusted to manage infrastructure that today is all too critical to our economy and national security. Their malfeasance and unaccountability is unacceptable.”
“What’s he talking about?” Omori asked, looking over Max’s shoulder.
“Neglect? This was a once-in-a-billion crash. We’ve have five-nines of uptime for the last decade.”
“Effective immediately, I am using the Digital Sovereignty Act to nationalize the Olympus Data Center in Nevada. All their operations will be placed under federal control to ensure future stability.”
The blood drained from Max’s face. He looked at Omori, who was equally stunned. The green lights of nominal operations now seemed to be mocking them.
“I guess we should go pack up our stuff,” Omori said quietly. “We’re not welcome here anymore.”
Harley stormed into the room with the confidence of a conquistador. His suit pressed against his sizable frame, not quite fitting anymore. His silk tie was wrapped around his neck like a noose, but he refused to loosen it.
Max and Omori stood by the entrance, unwilling to make eye contact or step back into the room they had just saved. They had been summoned here by the government to participate in the transition of power.
“Mr. Goodwin and Mr. Taki, I assume,” Harley extended his hand to them. “I’m Harley Dunn, the president’s new czar of data centers. I’m pleased to meet you. I know this might not be the best of circumstances, but I am impressed by the work you and your team have built here.”
“Thank you, sir. Welcome to Olympus,” Max returned his handshake and tried to keep himself calm. He and Omori had built it. It didn’t seem right that it was now being taken away from them.
“Yes, well the first act of my tenure is to give this a rebrand. It is now F-SICK-WEST. That is, F S I C. A Federal Secure Information Center,” Harley flashed a smile. “Isn’t that catchy?”
“Uh, sure,” Max replied hesitantly.
“Well, it’s a name by committee. I didn’t choose it. I’d much rather have something like ‘Olympus’. That’s a grand name. Impressive,” Harley peered into the primary server room. “All these blinking boxes are really something. I’ve never seen this many computers in one place before. Now I know why the president called this a national security issue.”
“Sir, this is far more than just a series of computers. This is a highly complex distributed system that requires a deep understanding of networking, hardware, and software to operate effectively,” Max explained. “You’ve never managed a data center before, have you?”
“Not something of this size, no,” Harley admitted. “But that is why I’ve called you here. I’ll need a bit of your expertise. I assume you have some sort of documentation on how all this works?”
Max and Omori exchanged a glance and led Harley to a nearby conference room. They sat down at a long table in a glass-walled room. They could see the server room through the other side of the glass, blinking and spinning as usual, without any idea of what was going to happen to it.
A large binder was placed on the table in front of Harley labeled “Olympus Data Center Operations Manual”. It contained hundreds of printed out pages of documentation that Max and Omori had created over years of real-world experience. They did a print run every quarter so it kept up to date in the event that even their LaTeX file was corrupted. Redundancies were important.
Max opened the binder and started flipping through pages. He talked about the architecture of the data center, the network topology, the server configurations, and the disaster recovery procedures. Every so often, Harley would interrupt with a question. It was clear from his questions that he was out of his depth.
The playbook documented all kinds of failure modes and how to handle them. Max and Omori had seen all kinds of outages before, and had written detailed procedures for every one of them. Except for one.
“This is fascinating stuff,” Harley adjusted his tie up and down idlily. “But I’m just hearing a lot of technical jargon. ‘Load balancing’? ‘Boot loops’? The president brought me in to make sure this system runs smoothly. The American people need to be able to trust their infrastructure runs well.”
“But we build the system,” Max protested, his voice rising more than he intended. “We already designed it for resilience. We’ve had five-nines uptime for the last decade. This was a fluke.”
“We need a new operational philosophy,” Harley said firmly. “We need to improve our processes and assign accountability. We will keep your historical contributions in mind, but the failure of your private system has shown that a new approach is warranted.”
“A new approach without us,” Omori said flatly, not quite a question as much as a statement of fact.
“I think that would be best,” Harley agreed. “The federal government thanks you for your service, but now it is time for us to bring in new management. Please clear our the remaining personal items you have here. We’ll be removing your access to the systems by the end of the day.”
Max didn’t reply. He could barely even give a curt nod before storming out of the room. Omori gave Harley a long look before following Max out the door.
Max and Omori were soon standing in the parking lot with boxes in their hands. They looked back at the large, windowless monolith that represented her life’s work. She could no longer see the rows of machines or their blinking lights. Now it was out of her hands, for someone else who had no idea what they were doing.
It was summer now. The days were longer and the air was warmer. Max lived in a small apartment that could see the skyline of the city in the distance when the blinds weren’t down. He didn’t have much furniture. He didn’t need that much. He preferred the quiet of the countryside to the noise of the city.
He had gotten a new job as a consultant for a small Swiss startup that was trying to build a new decentralized cloud platform. It was nothing like the scale of Olympus, but it was something. The work was remote, meaning he would take a one-minute commute from his bedroom to the three monitors hovering above his desk.
One monitor displayed a grid of faces on a video call. Max kept his camera off. Rather than paying attention to each team member giving a short report for the standup, he had a social media feed pulled up on another monitor filled with news from a Senate hearing about the data center nationalization.
Harley was sitting at the witness desk with a new suit that fit better. Although Max didn’t hear the words, he saw the transcriptions being posted by an automatic captioning service. The little nameplate at the desk read “Harley Dunn, Federal Czar of Data Centers”.
“We have achieved, since we began four months ago, a 30% gain in efficiency,” Harley said proudly.
Max let out a loud scoff. Still, with a bit of curiosity, he decided to mute the call and turn on the audio from the hearing.
“We have eliminated redundant systems, renegotiated carrier contracts with our national monopsony power, and our staffing have worked hard to make significant efficiency gains.”
Max shook his head, unable to believe that Harley was testifying to Congress while saying things that didn’t make any sense.
“Redundancies? Carrier contracts?” Max muttered angrily. “If you shut off the backup systems and cut off the fiber optic cables, you could save money but at the cost of creating a single point of failure. You idiot.”
Dunn flashed a professional smile at the camera and then said, “We have already proven that the government can be more agile and responsible than private actors.”
Max slammed his fist on the desk and immediately clicked out of the tab playing the video. That was a lie so audacious that his blood boiled. Harley was not some genius engineering manager. He was going to dismantle the entire system bit-by-bit until an even larger disaster happened. In the meantime, he would be seen as a hero for “saving” the data center.
On Max’s third monitor, a chat window popped up. It was from Omori.
“Did you see the hearing?” Omori’s message read. “This guy is a joke.”
Max was about to reply when another message popped up. It was an automated message from one of the monitoring systems Max had set up years before, which scraped websites that he thought were interesting and sent him alerts if anything on those interesting websites was especially interesting.
“Space Weather Alert. X-Class Solar Flare Detected.”
There was a web link underneath that he clicked so he could get a better look. The page loaded a live feed from a solar observatory in Chile. A coronal mass ejection meant millions of tons of hyper-magnetized plasma was going to be hurled out of the sun and slam towards the planet. It would induce catastrophic geomagnetic storms that could fry unprotected electrical systems.
When Max had been building out the Olympus project, he had kept this in mind. Protocol 7B had been written specifically to deal with solar storms. He opened up another tab and opened up the PDF version of the operations manual he had stashed away on his personal server, a file he legally probably should not have had.
“First, you would need to do a manual disconnect from the grid,” he recalled. “Physically isolate the data center before the storm hits, or else the induced currents could fry all the transformers and power supplies.
“Second, prepare the diesel generators and make sure they’re already running at a low idle so that you don’t run into any cold start problems.”
Max wondered if Harley would want to do either of these steps, if he even heard about the storm. These required manual effort and wasted fuel and money. Harley probably wouldn’t care about any of that.
“The third step would be to reroute server connections away from fiber optics and to a hardened, encrypted satellite uplink. Even if the storm supercharged the fiber lines, there would be an air gap to protect the servers.”
As Max read through his own words again, he started to understand that Harley would not follow any of these steps. He would be more likely to just rely on the existing infrastructure and hope for the best. It would be the best way to save money. He knew better. He had to do something before the lack of preparation fried everything in the center and destroyed the years of work he and Omori had put into it.
He opened a blank document and began to type out a message that laid out clearly and concisely the problem and how it needed to be addressed urgently. A few diagrams were included to make sure that even a person as layman as Harley could understand it. The words were so simple a five-year old could understand them.
He attached the document to an email and sent it to Harley Dunn with the subject line: “URGENT: Imminent Solar Storm - Immediate Action Required to Protect F-SICK-WEST Data Center”. He CC’d several other government officials whose emails were publicly available in the case that Harley decided to ignore it. He hoped that someone would listen.
Yet as several days passed by, there was no reply. There wasn’t even a read receipt or a formulaic thank you for your email. He continued to monitor the solar storm alerts and projections, which showed the storm’s arrival counting down rapidly.
Finally the day of storm, Max found an email from a junior staffer at the top of his inbox. He tapped it open with anticipation.
Thank you for your submission. The Committee has discussed your concerns with Director Dunn of FSIC-West. He has assured us that all necessary preparations are in place and that the facility’s automated systems are more than capable of handling any potential power fluctuations. Your input has been noted and placed on file, but the Committee is wary of giving credence to unsubstantiated warnings that appear to originate from disgruntled former employees of the facility’s previous private ownership.
Max read that single paragraph several times, hoping that he had simply misread it. But no, it was clear. Harley saw him as nothing more than a disgruntled ex-employee. A nuisance. He was just a Cassandra, screaming that something was wrong but being ignored. He felt a wave of despair as he realized that there was nothing he could do. All he could do was wait.
As Max poured the noodles out of the pot into a red ceramic bowl for supper, he looked out the window. He could see the streetlamps flickering in the distance against the darkening sky. He felt a chill run down his spine. While the streetlights would be fine, something that the average person wouldn’t even notice, he knew that the data center was in grave danger.
A series of warnings appeared on the primary monitor inside the newly created command hub at FSIC-West. Willis, a young technician who was the week’s on-call, looked at the screen in confusion.
“What’s happening?” he asked his supervisor nervously, looking at the warning text without a clue what it meant.
“It’s just a minor fluctuation from the grid,” his supervisor answered calmly as he peered at the monitor way too close to Willis’s neck. “We have automated systems to handle the change in current. Nothing to worry about.”
Harley then entered the room with a frantic energy that was uncharacteristic for him. His tie was hanging loosely around his neck and he anxiously struggled to straighten it.
“Give me a status report,” he barked, his voice louder than necessary.
“The solar event has been causing some current instability coming in from the external grid. Sir.,” Willis replied adamantly.
“That was to be expected,” Harley dismissed. “But we just follow the protocols. The facility has an automated uninterruptible power supply which will kick in if there’s any problems.”
With a flair of dramatic irony, the main lights in their command hub flickered again for a few seconds before going out completely. The room was plunged into a dim red glow from the emergency lighting.”
“Willis, what’s happening?” Harley demanded, his voice rising in panic.
“The external grid has gone offline, sir!” Willis shouted back, tapping on the keyboard to try to bring up more information. All he could see were a line log of error messages flashing across the screen in red text.
“Why didn’t we switch to generator power?” Harley asked, his composure finally cracking.
“The ATS isn’t responding,” Willis tapped urgently, even though nothing was happening. “The maintenance logs show that the quarterly diagnostic check had been deferred.”
“It was deferred until the next fiscal quarter,” Harley responded as if that excused the problem. “Go find the manual override. Now.”
“Sir, the manual transfer is complicated,” Willis tried to explain shakily. “There are multiple safety checks built into it. If I happen to bypass the wrong one...”
Harley snatched a thick binder from a nearby shelf with the words “EMERGENCY POWER SYSTEMS” stenciled on the spine. He slammed it on the desk next to Willis with a loud bang that startled him. He started flipping through pages and pointing a flailing finger at diagrams.
“Are you telling me you are illiterate? That you can’t follow a manual?” Harley spat. “This page is for the ‘Emergency Bus Transfer’. It is just a simple throw of the breaker. Even you couldn’t mess that up. Now get out of here and do it!”
“But the message underneath... it says this bypasses the synchronization check!” Willis protested, his shaky voice reflecting how conflicted he felt. “If we connect to the out-of-phase generators before we set them up properly, the power surge could...”
“This is a direct order, not an invitation to a debate on theoretical physics!” Harley shouted, his face as red as the emergency lights. “If you don’t do it, I will find someone who will!”
Willis half got out of his chair and then hesitated. He didn’t know which outcome was worse: disobeying a direct order and ruining his career or potentially destroying the entire data center and ruining his career. He looked up at Harley’s furious face and knew there was no going back.
With a deep breath, he stood up straight and slowly made his way to a heavy steel panel on the other side of the room. He peered through the large glass window at the glaring face of Harley now illuminated in red to give him an intimidating silhouette.
He reached for the large, insulated handle and pulled.
There was a long clang from the handle, followed by a deep rumble that seemed to come from somewhere deep within the building itself. He could hear the massive diesel generators in the sub-basement grind as they struggled to come to life and power the sudden demand for electricity from the data center.
The lights in the command hub turned back on, bathing Harley in bright white light. There was a glistening smile on his face as he looked at Willis closely.
“I guess that managed to work,” Willis admitted nervously.
Then he heard a bang as loud as a thunderclap that reverberated through the entire facility. The out-of-phase power from the generators conflicted with the uninterruptible power supply batteries. Every screen in the command hub suddenly shut off. The air was filling with the awful stench of burning electronics and ozone.
Then the lights went out again, without any emergency lights to fall back on. Willis couldn’t see anything. He fumbled for the phone in his pocket and then turned on its flashlight. He managed to put his hand on the wall and guide himself back to the command hub. He saw his phone battery was at 15%. He had meant to charge it, but kept putting it off. Now it seemed like he was on a short timer.
“What just happened?” Harley asked as the phone’s light flashed over his face.
Nobody answered. They were afraid to admit fault or acknowledge the whining from servers now gasping for cool air that they would no longer have.
The thermal sensors would’ve reported catastrophic temperatures in the server halls if they were still online. Capacitors began to explode, creating a sound like a rifle being shot in the dark. Several more began to pop, making the server room sound like a gangster fight happening within a speakeasy.
What they could see was a small flame licking up the side of a server rack. The fire suppression system should’ve kicked in, but with the emergency power now offline, the fire had nothing to trigger it.
When the fire reached a bundle of network cables, coated with polyethylene, it erupted into a thick black smoke that quickly filled the room. The blaze leaped from rack to rack, consuming irreplaceable data and equipment.
Willis and the others hurriedly evacuated the building as the facility’s main substation let out a final surge of power backwards towards the city. Several transformers along the way, struggling themselves to handle the solar storm, blew up with an overwhelming display of sparks.
Max watched from his apartment as the lights of Boulder went out one by one. Block by block, the entire city was plunged into darkness. The only light he could see was the small orange glow from the burning data center.
His legacy was now turning into ashes. He had studied engineering and worked for years in the field to build something reliable. He thought he had considered every potential scenario and calculated every variable. But there was one variable that he could never engineer away.