Solving the 'technical debt' problem

Technical Debt plagues IT systems, both public and private, by severely limiting the speed at which new functionality can be delivered while driving up costs. Although its genesis is badly constructed software, Technical Debt quickly becomes a management issue that affects how well IT will serve the mission.

Computer scientist Ward Cunningham coined the term Technical Debt in 1992 to describe the deferred costs of making sub-optimal design decisions to get software delivered fast. He argued the future corrective maintenance required to fix these problems represented a debt. You were borrowing development speed in the current release that must be repaid through increased corrective maintenance effort in future releases.

The Technical Debt metaphor is popular because it allows IT to express the cost of application quality in financial terms the business side can understand. The metaphor includes several different costs created by poorly constructed software. First, 'interest' is paid on the debt when overly complex software requires excessive effort for repair or enhancement. If the application runs slower because of performance flaws, then interest involves excessive resource costs, especially in the cloud. Operational problems such as outages, security breaches, or corrupted data represent 'liabilities' if caused by the flaws underlying Technical Debt. Finally, having to fix problems rather than add new business functionality represents an ‘opportunity cost’.

How does Technical Debt hurt IT and the mission? Let us count the ways.

• Code becomes more complex requiring more developer time to understand and change.
• Unnecessarily complex software requires more testing to avoid unintended side-effects.
• Developers take longer to release new functionality into production.
• Applications may not be scalable to greater mission requirements.
• Developers are more likely to inject new defects when working with excessively complex code.
• Poor quality code increases operational risks related to reliability, security, and performance.

Academic research on Technical Debt has remained mostly focused on Cunningham’s original definition. However, IT uses this metaphor to describe the cost of fixing known problems in software regardless of how they were created. IT managers want to predict the cost of corrective maintenance and quantify their operational risk. Several questions that can be addressed by measuring Technical Debt include:

• What are the likely future corrective maintenance costs for this application?
• Which applications will need the most remediation resources?
• Are my contractors delivering good software or expensive legacy?
• What is the cost of reducing the operational risk of our mission-critical applications?
• What must be improved to speed the delivery of new mission functionality?
• How much effort is needed to improve the scalability of this application?
• Should we fix this application or replace it?

How can Technical Debt be measured?

One technique for measuring Technical Debt involves analyzing the structural quality of an application using static analysis technology. Since Technical Debt implies costing the effort to fix problems, a measure can be developed from estimating the effort to fix only the severe weaknesses that an application team knows must be fixed. Problems that can be endlessly deferred do not constitute Technical Debt since they may never incur a remediation cost.

The Consortium for IT Software Quality (CISQ) co-founded by the Software Engineering Institute and the Object Management Group, an IT standards organization, has developed standards for measuring software Reliability, Security, Performance Efficiency, and Maintainability by counting severe violations of good architectural and coding practice in the source code. Building on these, CISQ developed an approved OMG standard for measuring Technical Debt based on aggregating the effort to fix occurrences of the 86 weaknesses included in the four quality attribute measures. Effort estimates were made by professional developers and include adjustment factors based on the structural complexity surrounding each weakness.

While not perfect, a Technical Debt measure contributes to guiding decisions where there are few fact-based indicators. Although Technical Debt numbers often seem alarmingly high, the CISQ measure provides a correlate of costs that can be calibrated and customized to local conditions.

How can Technical Debt be removed?

Quantifying Technical Debt is the easy part. Removing it presents the greatest challenge, and this challenge is only partly technical. Managing Technical Debt requires defeined tasks that are integrated into software development methods and processes. At a minimum these tasks should include:

• Establish targets and plans—based on initial analysis of the codebase, acceptable levels of Technical Debt should be established for each application, and application teams should plan actions across releases to achieve these targets.
• Analysis and measurement—during each sprint the integrated codebase should be analyzed at least once to detect structural flaws that should be fixed. Multiple analyses allow the most severe weaknesses to be addressed within the sprint. Measures of Technical Debt and the health of the codebase should be tracked.
• Prioritization and scheduling—severe weaknesses emerging from analysis should be prioritized and scheduled for remediation. The prioritized list should be added to the backlog of stories for future sprints. Remediation schedules must be enforced by management to ensure they do not succumb to unending requests for new functionality.
• Removal and verification—The software flaws constituting Technical Debt are remediated and the absence of undesired side-effects is verified.
• Communication—Measurable improvements in an application’s health are tracked by IT and periodically communicated to the business as reductions in development costs and operational risks.

Removing Technical Debt must be enforced by executive policy because demands from the agency for new functionality will always overrule intentions to remediate the codebase. Removing Technical Debt should be enforced either as a portion of the stories in every sprint or release, or as regularly scheduled remediation releases. Otherwise, the application team will increase the Technical Debt at each release, speeding degeneration into legacy code.

Although born in rushed coding and unintended mistakes, Technical Debt ultimately affects how well IT serves the mission. It becomes an issue of management priorities, tradeoffs, cost avoidance, and risk tolerance. Managing Technical Debt is an act of IT governance.