5 pitfalls that derail EVM success

No one understands better than the Federal Aviation Administration how hard it is to manage large-scale information technology projects. In a notable stretch of rough sailing, FAA officials saw their agency on the Government Accountability Office’s high-risk list for 14 years in a row.

“It’s not a reputation that anyone is proud of,” said James Washington, FAA’s chief acquisition officer.

Fortunately, that run came to an end this year, and FAA credits its adoption of earned value management and other best practices for its newfound success in managing projects. The agency now applies EVM to all large-scale projects that require an Office of Management and Budget Exhibit 300 business case form, or about one-third of its $3 billion in capital investments.

Since 2007, FAA’s EVM early warning system ended a couple of expensive IT initiatives that were beyond saving even if FAA had poured more money into them.

“We reached decisions fairly early on that we needed to cut our losses and channel our investments in other directions,” Washington said.

That success represents EVM at its best: A program management tool that sets baseline expectations for IT projects and then compares the value of work completed against expected progress and actual costs on an ongoing basis.

By using simple graphs to display data about schedules and spending, EVM systems show when projects risk becoming late or exceeding their budget. Most agencies base their approach on an industry-standard EVM model called ANSI/EIA 748. It specifies quarterly performance reviews by program managers, stakeholders and project implementers. Many government program managers go further and monitor performance continuously.

“EVM techniques are designed to make sure projects are performing the way they’re supposed to or identify early and frequently when something challenges their success,” said Kevin Plexico, senior vice president of research and analysis at government IT researcher Input.

With praise from GAO and other positive reviews, FAA is becoming a poster child for EVM success, but the agency also stands out because many of its peers still struggle to get EVM right. Common problems include lukewarm commitment by agency executives and contractors and problems with the accuracy of performance data.

Despite the inconsistencies in its application, EVM appears to be here to stay. The Defense Department has used EVM to some degree for decades, while Federal Acquisition Regulation rules and OMB mandates require that all federal agencies adopt the discipline for large-scale IT projects.

Last spring, some senators introduced legislation that would require OMB to create a Web site that uses EVM data to track the quarterly progress of major IT projects. Subsequently, OMB incorporated EVM elements into the IT Dashboard it launched last summer to summarize the performance of federal IT investments.

So with EVM appearing to gain steam, how can agencies circumvent its hazards and turn it into a valuable management tool? EVM veterans said organizations can start by steering clear of the following pitfalls.

Pitfall 1: Executive ambivalence

Compliance fatigue, personnel turnover, deadline pressures and other factors can keep senior executives and project managers from fully committing to EVM.

“Historically, earned value has had a bad knock as just a bunch of numbers that the finance geeks use,” said Daniel Butler, chairman of the National Defense Industry Association’s Program Management Systems Committee. The industry and government association promotes EVM and maintains the ANSI/EIA 748 standard.

Experts say executives who downplay the importance of EVM indirectly tell junior staff members and contractors to take a similarly casual approach, which is why EVM can be doomed unless it becomes ingrained in an agency’s or contractor’s operations.

“If the program manager is talking in the language of earned value, then obviously it’s part of the culture,” said David Kester, director of the Defense Contract Management Agency’s EVM Center, which certifies contractors for DOD projects worth $50 million or more according to their ability to meet ANSI/EIA 748 guidelines.

Solution: To achieve widespread buy-in, executives must first understand what EVM isn’t, said Jon Mihalic, a board member at the Project Management Institute, a professional association.

“It’s not an accounting tool. It’s not a financial management tool,” he said. “It’s part of your project management approach that can help drive your programs.”

PMI’s certification program and training manual, “A Guide to the Project Management Body of Knowledge,” include discussions of how earned value affects decisions about project and risk management.

Because EVM is more than just a regulatory check-off item, experts say senior managers must also understand the significant investments EVM requires.

“Most new folks coming [in] don’t understand the time and resources that it takes to stand something like this up,” Kester said. Many agencies devote 16 months or more to EVM training and system development, he added.

Pitfall 2: Poor data

EVM analyses are only as good as the numbers they crunch. Large IT projects produce volumes of rapidly changing data compiled by agency employees, contractors and subcontractors. As a result, there are plenty of opportunities for incorrect or outdated information to cloud performance reports.

Solution: After receiving sharp criticism for its handling of the problem-riddled FAA Telecommunications Infrastructure program, FAA re-examined its EVM data and found anomalies. With the help of an outside consultant, a 12-member team of FAA employees now reviews information submitted by contractors about planned work versus actual performance.

“Rather than taking the initial EVM data runs at face value, we validate the original set of data so we are certain what that data is telling us,” Washington said. “This gives us the assurance that we are looking for.”

Pitfall 3: Data overload

Accuracy isn’t the only problem when it comes to EVM data. Agencies also must avoid getting buried in data.

Too much detail can make it harder to spot the significant cost and schedule variances, said Fred Atwood, a certified Project Management Professional and program control manager at Vangent, an information management company that has used EVM for projects with the Labor, Commerce and Education departments and DOD’s Military Health System.

“Although you are getting more data, the additional work involved in setup, managing and reporting doesn’t always yield useful information,” Atwood said.

Solution: Focus on indicators that reveal trends for main project components. For example, project managers who oversee the development of a new software application might be expected to monitor progress on the user interface. But if they examine details at the level of individual menu items within that interface, they risk information overload.

“Rather than showing an agency 50 very small problems, [EVM analyses] should say, ‘Here are three major problems that are significant, and here’s why,’ to focus management attention on the things that are most critical,” Atwood said.

Pitfall 4: Poorly chosen software

A number of commercial software packages are available to automate data collection and analysis. Popular tools include Open Plan, Cobra and wInsight from Deltek; Oracle Primavera; and Caldera CostView.

However, misusing an application can undermine an organization’s management processes and skew EVM results.

Solution: Agencies should evaluate potential tools for their ability to integrate with the organization’s existing IT environment, Mihalic said. He added that even the best tools won’t guarantee success if they aren’t a proper fit.

“The biggest mistake I see is agencies buying a tool [and] building processes to fit the tool,” he said. “That’s when earned value becomes a problem.”

Instead, agencies should delay technology decisions until they’ve created well-defined processes that accommodate ANSI/EIA 748 guidelines and internal project management requirements. Then they should evaluate tools based on how easily they accommodate those processes.

Pitfall 5: Inexperienced contractors

Since OMB mandated the use of EVM, there has been no shortage of contractors that tout their EVM expertise.

“Everywhere you look, there are [EVM] consultants, but there is a huge disparity in their qualifications,” Butler said.

Solution: When drafting requests for proposals, agencies should include an excerpt of the EVM-related FAR clause in their solicitations. FAR requires contractors to show EVM certification, such as evaluations by DCMA, or document specifics about how they implement their EVM systems, Butler said.

The best candidates will have longtime EVM experience and perhaps DCMA certifications and projects in environments that adhere to recognized standards, Kester said.

“Find someone who is not just taking something off the Internet and walking through the basics of earned value,” he said.

As many agencies have learned, EVM is not a difficult concept to understand, but using it effectively requires more than a partial commitment.