How technical debt makes government software crap, and what we can do about it

Technical debt is all the cruft, hacks, and shortcuts that get baked into a piece of software over time, making it insecure, unreliable and hard to maintain.

To give a non-technical analogy, imagine a concise, well written policy. Everyone understands it and agrees that it's a good thing. Then things change, and the team that wrote it must quickly add an update before a deadline. They don't have time to update the original document, so it gets added as an addendum. Then, as the policy catches on, other organizations add their own input and recommendations, some of which conflict. Finally, some brave souls try to simplify the whole thing, but they also have a deadline and end up writing yet another incomplete or conflicting recommendation.

The result is a convoluted policy no one understands. By this point, it takes a deliberate, concerted effort from experts to carefully untangle all the threads and return the effort to coherence.

Exactly the same thing happens in software. The first version of a program might look fine, but then, to add a new feature, someone skips documenting their code. The next person who reads it decides that since the code has no documentation, they'll write their own version instead of reusing the original code. The next member of the team realizes there are no tests, but decides because there are so many slightly different versions of the same code, it's not worth the effort to add tests to all of it. After all, these people have deadlines and no one will reward them for going out of their way to pay off this still-invisible debt.

This process is a pattern that has persisted for decades. It causes our government to accumulate a massive amount of technical debt. Millions of lines of completely undocumented, copy-pasted code, written in languages that no one still knows, are running on systems whose creators are now dead. Some of these systems provide life-or-death services for real people, so the instability that comes from this technical debt has real consequences.

The engineers building this software are set up to fail by contracts that reward checking boxes over delivering working and maintainable software, making any effort to pay off technical debt effectively charity. This is not a great situation for anyone.

Why USDS is effective

USDSers have built, debugged, read and rewritten software. Rather than asking teams for management-level reports, we sit side by side with the engineers to read and run the code ourselves. We can engage with the engineers on a technical level and hear their deepest concerns (after reassuring them we are there to fix the problem, not to find people to blame).

Often, when we ask for very specific technical things, like "show us your deployment process" or "how do we run your automated tests?", we get a mixture of surprise, fear or even excitement. Many people have never been asked those questions before. Some are afraid that the comfortable status quo is being disrupted, while others are excited that someone in government finally cares about the issues they've been quietly keeping at bay (and averting disaster) for many years.

Using USDS lessons learned to fix this

Ultimately, the problem has two parts: hiring qualified people, and incentivizing good infrastructure.

In part, USDS helps the government hire skilled software engineers by writing "coding challenges" in which applicants for a contract must write code as part of the application. We can evaluate this code based on real objective results instead of how good their proposal sounds. A developer can claim to use "best practices," but if their code doesn't include any tests (or it does but, they don't work), the truth becomes obvious.

Once skilled software engineers are hired, they can be set up to succeed by rewarding measurable, real-world infrastructure outcomes. This might include security (measured by penetration testing and bug bounties), robustness (measured by load tests and simulated failures) and maintainability (measured by how long it takes to onboard a new person, and how many bugs are found late in the development process).

Why this matters

Understanding technical debt and knowing some of the strategies to pay it down are key to building robust, secure and maintainable software. The federal government provides critical services that affect people's lives in dramatic ways, and every hack or shortcut that gets added puts those same people at risk, now and for years to come.

Hiring skilled teams and incentivizing them to fix technical debt would go a long way toward bringing software in the federal government to a better place.