Breach Patrol:
The Surprisingly ‘Not Sexy’ Life
of 24/7 White Hat Hackers
of 24/7 White Hat Hackers
Breach Patrol:
Rob Bagnall, sporting a cowboy hat, leather jacket and black boots, tries to remember which secure federal facilities, lavish office buildings in and around Washington, D.C., and coffee shops he has visited over the past week for meetings. Pages in a lined notebook flip back and forth; an iPad blinks awake.
A week ago, I’d first met Bagnall, 46, face-to-face. We introduced ourselves Jan. 30 at a Dunkin' Donuts in Fairfax, Virginia, off Route 50 — the heart of the thriving contracting scene outside the nation’s capital.
I slurped an iced tea, trying to decide if he would be a credible source for a profile I had envisioned. Did he really run a firm that defends the intelligence community and wealthy VIPs from hackers, as his website attests?
Bagnall, who claimed to be the founder of 10-year-old small business Maverick LLC, was there to gauge his comfort level with my probing his life. He was also there for one of the many cups of joe required to man a 24/7 operation.
The idea I’d originally pitched to my editors was to embed with black hat hackers, the kind hired by despots and thieves to crack open secure networks. And also the kind increasingly glamorized in pop culture —"CSI: Cyber," "Scorpion" and other Hollywood productions. Such an arrangement would offer a nose-close view of the cyberspace threats confronting this country. But the legality and safety of such an expedition gave me pause.
Instead, I set out to find a squad of code breakers who use their skills for good.
A little networking led to this cybersecurity upstart that was willing to let me hover. I had blindly emailed Bagnall in early January a proposed plan to shadow him and his employees as they serve the federal government and their high-profile clients.
Lots of people tell you whatever you want to hear about cyber and how sexy it is. In the end it’s ensuring the mission. We are sanitation engineers more than anything else. We’re taking out the trash. When we’re doing it right, it’s mostly not sexy; it’s like being a PI.
The observations I documented would be published as a longform, online narrative, I told him at the Dunkin' Donuts. We set some ground rules: I would not enter any secure facilities and could not identify his private sector clients in the story. Certain family details and the nature of some government projects would not be published. For the next six months, I dropped off the grid about one day a week to follow Bagnall’s team without disruption.
My observations happened to coincide with a particularly busy time in federal cybersecurity. The White House issued sanctions against North Korea for allegedly hacking into Sony and the Obama administration kicked off a major push on cybersecurity, holding a West Coast summit in February.
And then, near the end of my time shadowing the company, the government disclosed a historic breach of federal personnel data: 21.5 million current, former and prospective federal employees and contractors, as well as family members, had personal information from background investigation files stolen by hackers. Government officials have called it one of the largest known cybercrimes ever perpetrated against the government.
What I saw during outings were not 3-D computer graphics projecting from wall-length display screens or cyber geniuses sprinting around with thumb drives in hand to neutralize infected airplanes.
"Lots of people tell you whatever you want to hear about cyber and how sexy it is,” Bagnall says. “In the end, it’s ensuring the mission. We are sanitation engineers more than anything else. We’re taking out the trash. When we’re doing it right, it’s mostly not sexy; it’s like being a PI."
Like cyberspace, Bagnall's office has no boundaries. Depending on the day or the assignment, it spans the back of a Range Rover truck, various Dunkin' Donuts, Regus rental offices, his home, the American Tap Room at Reston Town Center, client work sites and, of course, the Internet.
"There's really no point in us going brick and mortar yet," Bagnall says. "At some point, we might need to. But the way we operate – everything's digital." Again, there is no computer forensics lab or McLean office tower in this story.
When the conversation on a FaceTime video call turns classified or client-sensitive, he'll say to the employee, "Let's take this offline" or "This is not a safe medium."
Top Secret work is discussed in a room accredited by the intelligence community called a "SCIF," or Sensitive Compartmented Information Facility. Internet-enabled devices are forbidden there, which, as it turns out, isn’t a bad thing. Some of Bagnall's best breakthroughs are recorded in SCIFs, on a notepad, he says.
There's really no point in us going brick and mortar yet. At some point, we might need to. But the way we operate – everything's digital.
"Constantly in the back of my head no matter where I am, things are rolling around, I instantly understand there is a better way to do it,” he says. “I gotta write it down.”
When he is not penning proposals for upcoming "Ft. Meade projects" – contracts at the headquarters of U.S. Cyber Command and the National Security Agency – he says he brushes up on his creative writing. The one-time English major is working on a riff off a Clive Cussler thriller involving an intelligence analyst.
Bagnall’s Range Rover sports license plates with characters resembling the word "MAVERICK.” It racks up an average of 100 to 200 miles a day, he says. One week, when we don’t have a chance to meet, he emails me the exact itinerary of his day to provide a peek into what my recorder missed. On Feb. 4, just to pick a representative day, the office roves from Fairfax to Leesburg to Reston to Sterling to Centreville to downtown D.C. to Leesburg to Fairfax.
The federal procurement process is probably one of the most uncool aspects of Washington area cyber work – and one unlikely to ever be depicted on the screen.
The upside of government contracting is that once you've been cleared to access classified intelligence and have scored a spot on a list of approved vendors, it typically is a tenured job.
But the bureaucratic and administrative hoops companies – particularly smaller ones – must jump through to make their way onto a lucrative contract can be a struggle; cash flow can often slow to a drip. But it’s same story whether you’re selling pencils to the government or providing cutting-edge consulting services. Not exactly drama fit for Hollywood.
Task orders – jobs that companies must compete for even after winning a spot on a contract – have been held up by agencies, Bagnall claims.
"The problem within the last year is they aren't completing the process," he says in January.
One subcontract Maverick tried to bid on to help prime contractor Battelle and the government study the threat of car-hacking ultimately fell through.
According to the agency in charge, the deal awarded to Battelle in May 2014 was not tied to a specific paying job. "This type of contract does not, in itself, fund any projects,” a National Highway Traffic Safety Administration official told me in an email this summer. Rather, when a project is identified, an order would be placed under this contract which would then fund the project.” And so far, no orders have been placed, according to NHTSA.
Ironically, the possibility of remotely hacking vehicles hit the news around the same time in a big way. Private researchers remotely hacked a Jeep rolling 70 miles per hour on a St. Louis highway and killed the transmission, leading to a 1.4 million car recall and demands on Capitol Hill for stronger security standards for connected cars.
Separate from the NHTSA deal, a $6 billion Defense Intelligence Agency IT effort, for which proposals were due May 2014, was delayed until July of this year. Maverick won work as a subcontractor. DIA spokesman James M. Kudla noted the project had been divided among about 50 vendors and a year-long wait or more is not unusual for a multibillion, multiyear IT gig.
While waiting for federal project funding to come through, Bagnall says he inked a deal with a multinational high-end financial advising firm that month.
Earlier, in April, a $475 million solicitation for contractors to support Cyber Command was briefly floated and then canceled the next month. Bagnall says he spent more than $9,000 on personnel to do the proposal prepwork. A rewritten request for proposals is expected to be released this fall.
Why stay in this business at all?
"If I had my druthers," the composition of Maverick's contracts would be a ratio of 75 to 25, commercial to federal, Bagnall says. In June, he had 13 private sector customers and five federal contracts. "All but one of the federal ones now are holdovers that haven’t been funded yet," Bagnall says.
Usually, Maverick partners with a "prime" contractor to compete for work and then supports that company as subcontractor or, even more informally, as part of a teaming agreement.
That might be why a Cyber Command official in the press shop is not able to find any records of Maverick as a contractor, when I try to confirm his federal assignments. Typically, lead contractors do not file the names of all lower-level subcontractors or consultants with agency customers. According to a purchase order Bagnall provided me, Maverick was supporting CYBERCOM as a second-tier subcontractor from at least 2011 to 2012.
To become a prime contractor, and be eligible for bigger projects and more money, Bagnall says the company recently invested $19,000 in a business system compliant with the Federal Acquisition Regulation, the bible of government contracting. Reports generated by the system are regularly audited to deter waste, fraud and abuse. Maintenance costs about $5,000 a month.
A separate contractor computer system – maintained by the Defense Department – adds an additional tangle to the knot of red tape surrounding federal contracting. The Pentagon’s “Joint Personnel Adjudication System," or JPAS, is an online tool that tracks employee background investigations. Before a federal worker or contractor is cleared to see U.S. secrets, the employee, relatives and contacts are vetted to identify financial situations and other private matters that might expose the individual to blackmail.
The system can be clunky to use. Bagnall calls it “convoluted and painful,” when I meet with him in March, sipping scalding green tea inside a small Chinese restaurant in a Herndon strip mall. We’re across the parking lot from a Dunkin' Donuts. A few months later, the online background check system was yanked offline for several weeks as officials raced to plug a security flaw discovered in the wake of the OPM breach.
One of Bagnall’s toughest responsibilities as the boss is to “replicate” himself, he says. There is a fierce battle to recruit information security brains across both industry and government.
When I meet Bagnall one afternoon in early April in a Foggy Bottom office building – the site of a recent client visit – he’s preparing for one member of his small staff to leave and another to onboard.
What does he look for in prospective hires?
"Curiosity – that’s what everyone is missing now,” he says. “That’s the thing that makes you think about this stuff when you are not at work, when you are on vacation, when you are on the bus, when you are on the toilet."
Many hackers, the criminals and the good kind that warn people about security vulnerabilities, are driven by a simple desire to understand how things work. "They are not going to get stumped easily, and when they do get stumped, they are going to churn it over until they are unstumped,” Bagnall says. “And those are the kind of people I need.”
He says he has a rough time finding cybersecurity professionals under age 30 with that type of inquisitiveness, a problem many organizations in Virginia are facing. There were 20,507 cyber job postings in the commonwealth in 2013, reflecting a 53 percent jump in openings between 2007 and 2013, according to a March 2014 report from market research firm Burning Glass. Virginia, home to the CIA and other government security outfits, had the second highest need for cyber talent in the country, just behind California, with 27,084 open positions.
Skilled, experienced cyber whizzes are scarce in the United States. Since 2012, the Pentagon has been trying to fill the ranks of a 6,200 Cyber Command force with civilian, military and contractor personnel. As of July 23, about half of those slots remained empty.
While observing Bagnall from January through June, I became acquainted with most of Maverick's entire eight-person team. Surprisingly, for such a male-dominated field, there were as many women on staff as men.
And, in fact, Bagnall’s right-hand men are actually women, like Shannon Praylow, the company's cyber operations subject matter expert.
Praylow first worked with Bagnall about a decade ago at federal contractor SAIC, but only recently joined the company. The two met when the then-22-year-old, straight out of the Virginia Army National Guard, landed a security operations center analyst position at SAIC. Even back then, Bagnall, a deputy director at the center, was nudging people out of their boxes, Praylow says.
His directions were to write a white paper on cyber incident prevention, she says.
"I was a kid, and I’m like, what? I don’t know how to write a white paper," she recalls at a National Harbor coffee shop on her way to a federal 24-hour cybersecurity watch center in D.C.
"He pushes me outside of my limits, my comfort zone, and makes me do different types of work that he knows I can do,” Praylow says. “He has the ability to see what we’re capable of, even though we’re not able to see that, or we’re not comfortable," Praylow says.
Right now, I’m a team lead of 10 males, I’m still trying to find a happy medium of let’s not be a bitch but let’s not be so nice that they think I’m a pushover.
Bagnall values formal education, but not as much as much as natural qualities like ambition and tenacity, he says.
"You can have all those letters and certificates and commas after your name. That doesn't make you good," he says.
Praylow holds no college degree.
In between her time at SAIC and rejoining Bagnall in 2013, the 32-year-old mother of two daughters has risen in the ranks.
"Before I worked with Rob, I worked at – I can’t really say where I worked,” Praylow says. "I’m not supposed to.” The departments of Defense, Homeland Security, Justice and State are all on her resume, sometimes concurrently. Now, she helps Rob hire talent, maintain a company blog that dissects the latest software vulnerabilities and oversees a team of federal contractors.
In her day job at the D.C. watch center, "right now, I’m a team lead of 10 males," Praylow says. "I’m still trying to find a happy medium of, ‘Let’s not be a bitch,’ but let’s not be so nice that they think I’m a pushover."
Women make up a mere 14 percent of federal government cyber personnel, according to an (ISC)2 global information security workforce study released in May. This is Praylow's first management position.
On a Friday afternoon in mid-February, the crew is at the Reston Regus building scrambling to type out proposals for possible Ft. Meade projects. A staffer going over paperwork and paying company bills with a co-worker unthinkingly reads aloud Bagnall's Social Security number to the other staffer.
Under normal circumstances, speaking face-to-face, rather than emailing or instant messaging, would be the most secure medium for communicating this sensitive data. But with a reporter in the room and an audio recorder on the table behind her, she panics. This is an example of bad operations security, or opsec, a military term for the protection of information of value to an adversary.
A healthy dose of paranoia is a job qualification at Maverick. Every piece of mail with Bagnall's personal information on the envelope is shredded (anachronistically, most of the firms that Maverick does business with mail financial statements, even if they offer online payment methods).
"Are you really putting those things in the trash can?" Bagnall shouts, as whole pieces of junk mail land in a trash bin. "What's that address there on the envelope?"
He adds, "Shred, people, come on," sounding more like a cheerleading coach than a drill sergeant.
Compare this data-protection tactic to the raging bonfires and racing heros Hollywood employs to destroy sensitive information.
After being asked about realistic portrayals of cyber whizzes on film, Bagnall guffaws and raises his eyebrows as if I had asked the dumbest question ever. “I actually saw an "NCIS" episode where, the chick with a ponytail and the tattoos, she makes a Faraday cage to put a laptop in so that she can look out for malware and [the virus] hacked her through the power cord.”
Typically, a Maverick employee's social media profile will display little information about where in the government that individual has been posted. Hackers often try to trick contractors and federal personnel into disclosing sensitive data by posing as colleagues and "friending" them online.
What Bagnall does not want is a breach on par with a 2011 incident that ultimately cost a CEO at cyber firm HBGary Federal his job. Aaron Barr reportedly identified the top trouble-makers at hacker activist collective Anonymous and contacted the FBI to share his findings with authorities. The hacktivists retaliated by commandeering his Twitter account and posting online thousands of Barr's emails containing agency contacts. Barr resigned within a month.
My meetings with Bagnall come to an abrupt halt in June, with revelations about the espionage campaign targeting U.S. intelligence personnel. The volume of email Bagnall and his customers receive hawking bogus ID theft protections doubles, he claims. The amount of articles on potential employee ID theft my news organization publishes triples.
By early September, the furor over the big breach has waned and we sit down at the American Tap Room back in Reston. There has been a steady stream of hacks divulged since we last met – the OPM data heist, a leak of client names from adultery-arranging site Ashley Madison, the Jeep hijacking, and continuing concerns about plane hacking. But none of these episodes drastically alter Bagnall’s profit-making, he says.
“Car hacking didn’t change it. OPM didn’t change it,” he says. “It takes something catastrophic.”
Maverick has not witnessed a surge in customer traffic in any sector, he says. Private industry dismisses the intrusions as the result of sloth at OPM. Federal agencies have released more solicitations for information security work, but it is unclear if there is a cause-and-effect relationship between the personnel records hack and the need for more cyber expertise.
Victims of the background check breach have yet to be notified, but Bagnall says it’s likely his data is in the hands of hackers. Because he has undergone an OPM background investigation since 2000, intimate details, including medical information, about him, his wife and his young daughter have been exposed, he says.
Knowing that adversaries might try to use his secrets and psychological profile against him does not seem to faze Bagnall. "It doesn’t change my mindset or my operations,” he said. “I am just as paranoid as ever.”
Knowing that adversaries might try to use information about his 8th-grade daughter against her, however, does upset him as a father and an American.
The compromised background check forms list every child of every applicant and, as Bagnall says, many children of government employees follow in their parents footsteps.
I ask what he is doing to protect his daughter. "What can you do?" Bagnall says.
Aliya Sternstein reports on cybersecurity and homeland security systems. She’s covered technology for more than a decade at such publications as National Journal's Technology Daily, Federal Computer Week and Forbes. Before joining Government Executive, Sternstein covered agriculture and derivatives trading for Congressional Quarterly. She’s been a guest commentator on C-SPAN, MSNBC, WAMU and Federal News Radio. Sternstein is a graduate of the University of Pennsylvania.
Icon credits: To go cup by retinaicon from the Noun Project; Creativity by Creative Stall from the Noun Project; Woman by Andrew Searles from the Noun Project; Paper Shredder by Dan Hetteix from the Noun Project.