Kelman: Let the data talk

Those who want to go back to the procurement methods of the 1980s believe subsequent reforms have led to rapacious contractors getting huge government giveaways. Because of defense spending’s visibility and its ties to the war in Iraq, much of the reform debate is about runaway increases in weapons costs. I’ll explain, but first let me repeat an idea that is common wisdom among social scientists: The plural of anecdote is data.Two studies offer data pertinent to the debate. The studies focus on the effect of procurement changes of the mid-to-late 1980s and acquisition reform in the ’90s on the growing weapons costs. Both look at weapons contracts completed in periods before and after the changes of the ’80s and ’90s. The studies examine several hundred contracts.One study by Christensen, Searle and Vickery, published in 1999 in the Acquisition Review Quarterly, looks at the first period. That was the time of the Project on Government Oversight (POGO) holds up as the last era of true procurement reform. But actually it was a time when competition rules were bureaucratic, laws made it harder for government and industry to communicate, oversight was the watchword, and bid protests flourished. The second study, a 2002 master’s thesis on cost analysis by Andrew Mosier, looks at the effect of acquisition reforms from the ’90s to the end of 2001. Here’s how I would summarize the two studies’ findings: The so-called reforms of the mid-to-late ’80s produced an increase in the average cost growth across all weapons systems compared with the previous period. By contrast, the procurement reforms of the ’90s produced a decrease in average cost growth compared with the previous period.The first study compared the growth of contract costs completed between 1988 and 1991 with those completed between 1992 and 1995. Given the time it takes for a weapons contract to be completed, one can surmise that contracts completed between 1988 and 1991 were undertaken before the reforms of the ’80s were implemented. The study of the ’90s took a similar approach to contracts completed between 1993 and 1997, comparing those started before the ‘90s reforms had taken hold with those completed between 1998 and 2001. One interesting finding is that cost growth following procurement changes in the ’80s was greater for development than for production contracts. By comparison, the drop in cost growth following acquisition changes in the ’90s was greater for development contracts. Those results are especially noteworthy because the success of development contracts depends on trust between the government and the contractor, a trust POGO dislikes.The two papers don’t tell the complete story. A scholar would want to know whether the results are sensitive to small changes in the selected years. However, taking into account the studies’ limitations, I think they show that data is better than anecdotes. Instead, people have let anecdotes dominate the debate about procurement changes.