Inside the IBM/Amazon protest

The Central Intelligence Agency determined that IBM could have built and managed a massive cloud computing infrastructure for the entire intelligence community cheaper than Amazon Web Services, but selected AWS anyway for its "superior technical solution," according to the Government Accountability Office.

The assessment is in GAO's formal ruling on a bid protest filed by IBM in February over the CIA's deal with AWS worth $600 million over four years. While much of the language released June 14 was scrubbed by AWS and IBM attorneys – and by the CIA itself – it highlights the CIA's evaluated price over four year periods for AWS at $148 million and IBM at about $94 million, or an annual difference of about $13.5 million per year. (The evaluated price is not the entire value of the contract, but rather a bid assessment of the companies based on six scenarios developed by the CIA. The remaining base value of the contract, which could grow further if the three-year option and two-year option are exercised, is based on the intelligence community’s potential requirements for the cloud.)

Yet the CIA picked AWS in January to build out a cloud that will help the CIA, the National Security Agency and the rest of the intelligence community better analyze its growing piles of information – a decision protested by IBM and sustained in part on June 6 by the GAO.

"While IBM's proposal offered an evaluated price advantage over 5 years, the source selection authority (SSA) concluded that this advantage was offset by Amazon's superior technical solution," according to the GAO ruling.

In its final consensus evaluation, the CIA rated IBM's overall proposed risk as "high" and AWS as "low," and rated AWS higher in technical approaches and service level agreements. The agency was also skeptical of IBM's ability to auto-scale all applications, which it called a "grave" concern. IBM called that assessment "irrational" in its protest, saying it promised the auto-scaling capabilities in its IC cloud. But GAO denied that aspect of the protest, stating the CIA "reasonably determined that IBM failed to clearly establish the capability of its existing public cloud to auto-scale all applications."

The GAO did sustain IBM's protest on two grounds, however: That the CIA did not evaluate prices comparably, and materially relaxed a solicitation term for AWS during post-selection negotiations.

In technical language outlined in the GAO ruling, IBM took issue with how the CIA adjusted the company's pricing based on test scenarios of "providing a hosting environment for applications which process vast amounts of information in parallel on large clusters (thousands of nodes) of commodity hardware" using MapReduce. Test runs were to assume clusters large enough to process 100 terabytes of raw input data, and clusters were to be provisioned for computation and segmentation through MapReduce, with various other details specified.

GAO stated that both companies "adopted materially different interpretations" of the scenario requirements, and said the CIA's attempt to evaluate the performance "using the same or otherwise comparable level of performance" was unreasonable.

In addition, GAO sided with IBM over the company's exception to the CIA agreeing to an AWS' post-selection provision to alter a "material requirement" for the contract stating "only software developed and provided by Amazon" would be subject to certain requirements.

"In sum, we find that the agency, without issuing a written amendment, materially relaxed the solicitation's requirements for Amazon without affording the other offerors an opportunity to propose to the modified requirements. Accordingly, the protest is sustained on this basis as well," according to the statement.

"IBM believes this decision reaffirms our position that there were inaccuracies in the government's assessment of our proposal," said IBM spokesman Clint Roswell. "IBM remains committed to providing enterprise-level secure and robust cloud solutions and looks forward to a renewed opportunity to show our capabilities to fulfill the requirements of this important agency."

AWS did not respond to requests for comment from FCW. In a previous report, though, AWS took issue with GAO's finding.

"The CIA selected AWS based on its superior technological platform, which will allow the agency to rapidly innovate while delivering the confidence and security assurance needed for mission-critical systems," an AWS spokesperson said then. "The agency conducted a very detailed, thorough procurement that took many months to award. We look forward to a fast resolution of the two issues raised by the GAO so the agency can move forward with this important contract."

The CIA has 60 days to decide whether it will follow GAO's recommendations. The CIA is still reviewing details of the decision, according to a CIA spokesperson.

Even in Washington, $600 million is real money -- but the disputed contract is drawing attention for its larger implications. AWS is the largest public cloud infrastructure provider in world, but relatively new to federal contracting, and looking to take advantage of a large federal market in cloud computing that is beginning to open up. AWS recently achieved compliance with the Federal Risk Authorization Management Program (FedRAMP) and already counts more than 300 federal, state and local agencies as customers, but a deal of this magnitude with the CIA would instantly make it a major federal player.

IBM, meanwhile, has a storied history as a federal contractor to civilian, defense and intelligence agencies alike, and has made recent moves to strengthen its ability to compete in the growing infrastructure-as-a-service market. For example, the firm expanded its portfolio of private, public and hybrid cloud offerings through the $2 billion purchase of Dallas-based SoftLayer Technologies, Inc., the world's largest privately held cloud computing infrastructure provider.

The deal is also important in the context of the Intelligence Community Information Technology Enterprise (IC ITE) strategy, which tasks the CIA and the National Security Agency with providing cloud computing services to the entire intelligence community -- 17 individual agencies -- including data, utility and storage clouds. Intelligence agencies base IT decisions on the IC ITE strategy, which enhances the community's ability to discover, access and share information across agencies, and encourages integrated cloud computing services.

Given the shared-services aspect of the strategy, with its emphasis on consolidation and greater efficiencies, the entire IC could benefit from the private cloud in several ways, including big savings. It remains unclear how the CIA will proceed, but it is not uncommon for large corporations to end up working together on large projects following protests. And given the growing importance of the cloud to federal IT, a deal like this – whether both companies end up working together or not – is guaranteed to send ripples through the tech community.

Note: This article was updated on June 21 to clarify the differences between the evaluated prices and the contract's full base value.