Inside a Hacking Competition to Take Down a Water-Treatment Plant

Just before 2:30 p.m. last Friday, the water-treatment plant lost power. For minutes, as technicians scrambled to bring it back to life, it was still, its pumps quiet, its purifying chemicals inactive, its ultraviolet lamps dark.

The plant had been attacked by hackers, who had bombarded its power-control system with an unrelenting stream of data until, overwhelmed, it shut itself off. This same technique had been trained on a vital piece of the internet’s infrastructure earlier that very day—albeit at a much bigger scale—preventing millions of Americans from accessing vast swaths of the internet.

For the next two and a half hours, the water-treatment plant remained under siege from several different groups of hackers, who were attacking each other even as they delved deeper and deeper into the plant’s controls, causing absolute mayhem. At 2:45, a pair of revolving sirens threw blue beams around the room: The system that maintained the plant’s water levels had been disabled, and one of its tanks began to fill at an alarming rate.

“The float’s been submerged!” a technician called out from near the tanks. The float was supposed to cut off water flow the moment it became immersed.

“Is it still filling?” asked another, hunched over a laptop perched on his knees.

“Yes!”

“That’s bad.”

The workers powered down the plant again in order to drain the tank to a safe level.

Later, in a moment of calm after the attacks had subsided, I caught up with Joe Needleman, the technician who’d been wrestling with the plant’s settings. “It’s been insane,” he told me, nodding his head. But he wasn’t too shaken—actually, he was pretty excited.

Needleman had intended the plant to be attacked. In fact, it was built specifically for that purpose: He and four other students at California State Polytechnic University, Pomona, had spent the last month assembling it and writing 5,000 lines of code to control it, hewing as close as was practical to reality.

The result was a desk-sized model made up of three large plastic tubs, several aquarium water pumps, and PVC pipe. It was arranged on a table in a large, airy room at a co-working space in Washington, D.C., that was, on this particular Friday, completely overrun with wires. The plant was just one in a buffet of targets that had been set out for participants in a capture-the-flag-style hacking competition.

All afternoon, 13 teams of three or four hackers—mostly made up of college students, with a few professional security researchers sprinkled in—raced each other to accrue the most points by solving trivia questions, reverse-engineering computer programs to un-encrypt files, finding hidden messages encoded in digital images and songs, and, of course, attacking the model water-treatment plant.

Wade left a trail of destruction as he worked. After accessing one system, he deleted every user account except his own, changing the administrator password to “zachpwn.” In another, he found the controls that limit the water temperature in one of the treatment tanks, and raised the maximum from 100 to 1,000 degrees Fahrenheit. Stressful dubstep pounded in the background.

“Zach just went in and started burning things,” laughed Matthew Savage, one of his teammates. As the final half hour of the competition ticked down, Savage and the team’s third member, Corwin de Boor, tried to complete as many of the smaller challenges as possible, while Wade continued trouncing through the water plant’s control systems.

The way the team from Carnegie Mellon found vulnerabilities in the control systems and exploited them mirrored real-world patterns. Especially when it comes to sensitive infrastructure, hackers who break in may not even be intentionally targeting them, Levinson said. Instead, they might just be scanning the internet for vulnerable devices and networks, and not realize until they dig a little further that they’ve stumbled upon something bigger than a computer or a server—say, for example, a dam near New York City.

Keeping hackers out of critical infrastructure has become a priority for government agencies worried about a catastrophic cyberattack on an energy grid or, well, a water-treatment plant. Many installations still run on outdated computer systems, Levinson said. He’s concerned both about their capacity for holding up to attack, given the systems’ age, but also about the prospect of a bumpy transition to modern technology. If newer control systems aren’t installed and secured correctly, they could be just as vulnerable to attack.

The hacking competition played out in a friendly atmosphere: To Levinson’s surprise, the teams largely heeded his warning not to hack one another. But when many teams had trouble connecting to the website where the challenges had been posted, it was an ominous reminder of the cyberattack that was still making it hard to access the internet up and down the East Coast of the United States.

With threats like Friday’s disruptive hack emerging more and more often, private companies and the government are scrambling to hire and train the brightest young hackers to defend against cyberattacks. To that end, recruiters from Uber, Tenable, and Northrop Grumman were roaming the room as the competitors hacked away, dispensing advice and collecting resumes.

Before the competition began, Phyllis Schneck, the Department of Homeland Security’s top cybersecurity official, spoke about the importance of bringing computer-security skills into the government. One of the main obstacles to luring hackers into the public sector was illuminated during the question-and-answer session that followed, however, when one particularly distraught participant stood to protest the ways hacking laws can put white-hat researchers in jeopardy.

As the final minutes of the competition ticked off, the Plaid Parliament of Pwning was the only team to have put any points on the board for completing various elements of the water-treatment plant challenge. But at the buzzer, the trio came in second, falling to a team from the University of Virginia, who had accrued more points on other challenges. “We lost to the guys that taught us,” Wade told me: The three friends had gone to the same high school as some of the members of the UVA team.

“We gotta get ’em next time, Zach!” de Boor said. The team tossed Snapple bottles and 7-Up cans into the recycling, packed up their laptops, and left, turning around only to snatch up a Northrop Grumman business card they’d forgotten on the table.