Agencies are advancing AI pilots but face scalability challenges, report finds

Federal agencies are quickly moving to operationalize artificial intelligence pilot projects but often face challenges when it comes to making the emerging technologies work at scale, according to a report released Thursday by General Dynamics Information Technology.

The Falls Church, Va.-based IT contractor surveyed 325 officials in technology and decisionmaking positions at defense, civilian and intelligence agencies to understand how governmentwide efforts to adopt AI tools are progressing. 

President Joe Biden’s October 2023 executive order on federal uses of AI helped outline the government’s safe and secure approach to using these emerging capabilities. During a media roundtable on Wednesday ahead of the report’s release, Colleen Kummet — a program director and data science consultant at GDIT — said the order and ensuing federal guidance resulted in more “workforce changes in the government to support AI.”

Many agencies’ initiatives preceded Biden’s order, however, with the report noting that a September 2023 inventory of the government’s AI programs listed 710 projects “across 21 unique government departments.”

The report found that agencies, on average, were moving successful AI initiatives from the pilot stage to production within 14 months. GDIT wrote that part of the reason for the quick pace of development, by government standards, had to do with the narrow focus of most of the pilots.

Dave Vennergrund — vice president of AI and data insights at GDIT — also said during the media roundtable that this expedited process had to do with the way agencies were structuring their pilots.

“They're taking advantage of cloud data services, tooling and other capabilities that they can build on,” Vennergrund said. “Rather than having to build the plumbing, they're building on the plumbing.”

Despite the quick pace of many of these agencies’ AI-focused initiatives, the report found that 58% of respondents said their pilot projects failed to move forward because of scalability issues. Surveyed officials said these challenges were primarily due to a lack of end-user adoption of the pilots, insufficient funding for their projects and the inability to receive an authorization to operate as a result of security concerns.

“This difficulty suggests that agencies may need to focus on developing robust, scalable AI architectures and strategies from the outset,” the report said. “Considering scalability during the pilot phase itself will help ensure smooth transitions to larger-scale implementations. These considerations include the cost to train and run models, inference volume, users and data volumes.”

The success or failure of agencies’ pilot projects advancing to the production phase also had a strong correlation with the underlying capabilities being tested, with GDIT finding that the “most mature” models had the most success. 

The report found that “projects leveraging predictive analytics, at 71%, and machine learning, at 70%, were the most likely to succeed from pilot to production.” Conversely, pilot initiatives testing generative AI (65%), chatbots (57%) and image analytics (55%) had the lowest recorded success rates.