The perils of petascale IT projects

Good news: We no longer have to talk about megascale IT projects. Large-scale ventures that typically cost $1 billion or more, megaprojects used to be all the rage, but they are quickly being superseded by petascale IT initiatives. Those projects can cost even more, involve complexity on a truly massive scale and require petaflops of computer processing.

Despite the horrendous track record of delivering on even moderately complex IT projects, public-sector CIOs continue to embrace the design, planning and execution of petascale IT projects. Consider the Department of Homeland Security's SBInet, a component of the Secure Border Initiative. The $8 billion project commenced in 2006 and aimed to place surveillance systems along the southwest border of the U.S. to identify and prevent illegal crossings.

In 2010, SBInet was halted and many of the project's funds were diverted to other border initiatives. The project was riddled with problems, including cost overruns, poorly defined schedules, missed deadlines, inadequate oversight, an unclear strategy on how to actually secure the border or manage threats, poor engagement with contractor Boeing, and poor performance on the part of the company.

Calling a petascale IT project complex would be a huge understatement. Such projects not only have numerous stakeholders, they also span multiple time horizons (e.g., election cycles, brief CIO tenures at agencies, etc.). They rely on multiple physical and logical infrastructures -- each of which might be owned, controlled and operated by multiple players -- and they have overly restrictive and detailed budget allocations that prevent quick changes in response to events.

Furthermore, they are technologically complex from both innovation and design perspectives, and there is limited ability for leaders to learn from previous efforts because all projects of this scale are unique.

And those are just some of the issues that occur in petascale projects.

The high cost of complexity

Those challenges manifest themselves in the time, effort and cost involved in the projects. Cost overruns in particular are a significant way in which many projects are determined to be a success or failure. One notable megascale IT project that experienced a 560 percent cost overrun was the Medicare Transaction System. In the 1990s, this system was set to modernize Medicare to better serve the 38 million Americans who are over the age of 65 or have a disability.

The project was plagued with unnecessary risks, delays, unmet deliverables, poorly detailed plans, an over-reliance on contractors, fragmentation, and a poor understanding of the existing system and contractor duties.

More recent projects, such as the October 2013 launch of HealthCare.gov, show that IT failures are not an issue of the past. Other failed projects include DHS' software upgrade to its terrorist-tracking system. The $500 million Railhead project had a team of 862 contractors and yet was riddled with architectural and quality flaws.

In addition, the Air Force spent seven years and more than $1 billion to merge 240 legacy systems into 12. Unfortunately, the Expeditionary Combat Support System (ECSS) was halted due to extreme technical glitches and delays. The FBI's Virtual Case File system lost $170 million before being canceled, U.S. Citizenship and Immigration Services went $190 million over budget on an automation effort, and the Internal Revenue Service's Business Systems Modernization took more than 10 years to launch and cost $8 billion.

When the Air Force took on ECSS, the project was expected to automate and streamline logistics operations. It began in 2005 with a deployment date of 2012, and Oracle was contracted to develop the system for $88.5 million. Due to delays, the schedule was pushed back by four years to 2016, and it went from a three-phase to a four-phase project. In 2012, the Defense Department officially ended ECSS, saying it would cost another $1 billion to get the project up to military standards.

In a 2010 study undertaken while ECSS was still being developed, the Government Accountability Office found instances of "unusual logic" that caused activities not to be finished on time and led to the tracking of high-level milestones without a clear program structure. By the time the project was canceled, Air Force officials noted that they had received usable hardware and software worth less than $150 million.

According to GAO, 27 agencies planned to spend about $76 billion on 8,142 IT investments in fiscal 2014 -- $17.2 billion on development and acquisition and $58.9 billion on operations and maintenance. However, that total does not include the IT investments made by 58 independent executive branch agencies (including the CIA) or by the legislative or judicial branches.

Despite the big investments, large-scale projects are failing at an alarming cost to the government and taxpayers. The Office of Management and Budget reported in 2010 that the government had spent more than $600 billion on IT since 2000. However, OMB also reported that the federal government had achieved little productivity from its investments. In fact, IT projects account for a considerably higher number of failures than other types of government megaprojects.

OMB requires federal agencies to regularly provide information on their IT investments, capital asset plans and business cases. Over the years, reports from various agencies have revealed several obstacles to IT success that have wasted and continue to waste taxpayer funds and public resources.

One thing is certain: 99.9999 percent of petascale IT projects will fail. Even if they do end up nominally delivering on their intended objectives, they will not meet user expectations or be completed within budget or time constraints.

Moreover, the fallout from the failures of these projects can be severe -- not only for the IT personnel involved in the effort but for IT projects in other parts of the public sector. IT professionals will be looked upon unfavorably and seen as a liability, consultants can and might lose their jobs, and the overall standing of the IT community in the public sector will be tarnished.

It is no secret that most stakeholders involved in failed projects take a hit, to varying degrees. With HealthCare.gov, government administrators were investigated by congressional oversight committees and some even resigned, most notably Department of Health and Human Services Secretary Kathleen Sebelius. The main contractor involved, CGI Federal, experienced media scrutiny regarding its role in the debacle, and its past public-sector project failures were magnified.

Given this reality, we must ask: Why are we so obsessed -- some might say to the point of being delusional -- in pursuing projects of this scale and scope and hoping they will succeed? The oft-misattributed quote about the definition of insanity being "doing the same thing over and over again and expecting different results" applies here.

Four steps to failure on a grand scale

Over the next few months, we will be studying petascale IT projects in the public sector with the goal of providing some insights into why these beasts are so difficult to tame and what can be done about it. In this initial piece, we explore how political shenanigans contribute to the failure of these projects.

Political activity plays a crucial role in a project's success. First, consider the way in which agencies acquire the money to initiate massive projects. Those funds come from the political bartering that happens among members of the executive and legislative branches. Budgets are allocated largely based on the political capital of agencies and their champions.

When cost overruns occur, Congress must agree to pay additional funds to the contractors to complete the work. When the FBI initiated the Virtual Case File phase of its Trilogy IT modernization project in 2000, it was scheduled to take three years and cost $380 million. At the end of 2002, the project was behind schedule and the FBI asked Congress to increase funding by $123 million. Congress obliged partly because of the 2001 terrorist attacks and the resulting political pressures to pay more attention to anti-terrorism efforts.

Second, politics is a heavy influencer of the way projects are carried out. During the development of the HealthCare.gov website and prior to the 2012 presidential election, the White House reportedly asked the agencies involved in the site's development to avoid committing any website or health insurance exchange ideas to paper so that those ideas could not be stolen and used against the Obama administration before the election. That informal directive would prove to be effective in preventing bad news from being leaked, but it also became detrimental to the development of HealthCare.gov because problems were not acknowledged early enough.

Third, political cycles create a certain atmosphere of change within the public sector. The speed with which political leaders and their focus change can be damaging to projects that last years longer than their champions' tenure at the agency. That makes it difficult to maintain governance of a project throughout its full life cycle. For instance, in 2013, California suspended work on its $200 million MyCalPays system, which was designed to modernize the state's payroll. The project's failure was largely attributed to chronic leadership turnover and lapses in due diligence.

Finally, public-sector managers are often forced to find ways to cope with political dysfunction. Constant partisan clashes and disruptive behavior by politicians force agencies into precarious positions because most of the time, it's more about politics than policy. Recent oversight hearings about HHS' role in the HealthCare.gov rollout and the IRS' review of tax-exempt groups are arguably designed to bring attention to problems for political gain rather than to resolve them. After all, the HealthCare.gov fumble was not the first of its kind, and the project was salvaged soon after its launch. Other projects in recent history have not been so lucky, yet they drew a fraction of the oversight.

Political factors also play a large role in the governance of petascale IT projects. Governance issues are so varied and wide-ranging that finding an appropriate mix is a significant challenge. With proper governance, staff and stakeholders can discharge their duties in a way that allows the organization to function efficiently and effectively. Without it, the organization and the project suffer, and the resulting loss of funds and technological failures are physical manifestations of the lack of governance.

What can be done?

Strong governance of complex, petascale projects is critical for the success of IT projects. We offer five recommendations for CIOs to consider:

1. Overestimate the cost and time required to complete the project by a factor of at least three.

2. Document, document and document the risks inherent in the project and ensure that they are communicated clearly and frequently. Also make sure that the press knows that you called attention to those risks even if the political apparatus chose to ignore them.

3. If at all possible, convince political players to scale down the effort and consider an option-based approach that involves funding each stage of the project with a clause to continue funding if results are achieved and the project is still valuable.

4. Limit the number of petascale IT projects that your agency is involved in or has to support at any given time.

5. Study the lessons learned and postmortem reports from numerous other petascale IT projects as part of your due diligence.

The truth about IT megaprojects is that although some are successful, many are failures. Those failures have been repeated over and over, which leads us to the rather obvious conclusion that a new approach must be taken. Although most public managers cannot do anything about the dysfunctional political climate, they can make efforts to not fall victim to the dysfunction. The rapidly changing technological and political landscape reminds us that there is constant pressure to stay alert and ready to conquer the IT challenges that lie ahead.

We would like to hear from you about governance challenges associated with petascale IT projects that you have encountered. What did they look like up close?

As we learn more through our research, we will share our findings here in FCW. If you are interested in participating in our research project, get in touch via Twitter (@KevDesouza) or KevinDesouza.net.

Note: This article was updated on Sept. 1 to remove reference to Centers for Medicare and Medicaid Services CIO Tony Trenkle resigning. Trenkle retired shortly after the launch of HealthCare.gov for a position in the private sector.