Did USAJobs fix make matters worse?

In its efforts to fix significant operating and traffic problems at the USAJobs job search website, the Office of Personnel Management may have just made the problems worse with a misdirected fix, according to an industry critic.

On Oct. 18, OPM officials said they had installed a “governor” application on USAJobs to limit traffic, effectively blocking about 6 percent of the incoming visitors on their first try. The visitors were urged to hit the refresh button to keep trying. The purpose is to reduce the demand on the site's servers by forcing some users to wait for access.

Related stories:

OPM slows access to USAJobs to lighten load

OPM says USAJobs fixed, but complaints continue

The action was the latest effort to address technical problems in the eight days since OPM debuted the new job search site on Oct. 11. Thousands of users have reported searches returning incorrect results, timeouts and lost data. OPM has blamed the poor performance on extraordinary high traffic for the site.

However, OPM’s governor application may be shutting out more website visitors than anticipated, Linda Rix, co-chief executive of Avue Technologies Corp., which provides job application services to federal agencies, said on Oct. 19.

Instead of blocking 6 percent of incoming traffic—as OPM advertised—the governor application may be blocking a lot more, Rix said, based on her company’s authorized tests of the USAJobs system.

Avue’s tests conducted on Oct. 19 showed 86 failures to enter the USAJobs website in 100 tries, an 86 percent failure rate, Rix said.

“The governor may be hyperactive, and because of the way that OPM does caching, some people might be locked out for days,” Rix said. “That may be why some users are getting frustrated.”

Meanwhile, OPM officials said they were very focused on USAJobs concerns. The agency posted videos on its home page and the USAJobs Facebook page on Oct. 19 reassuring users that it is working around the clock addressing the system malfunctions.

OPM Director John Berry issued a statement emphasizing that the agency is making USAJobs operation a high priority. He also said many users have successfully submitted applications.

“During the last eight days, traffic has greatly exceeded previously reported volumes. To address this demand, we are adding additional server capacity to support this extremely high volume,” Berry said..

On Monday, OPM recorded a milestone: more than 100,000 applications submitted through the new system, Berry said.

"To be exact, as of 4 p.m. yesterday, Americans have submitted 141,289 applications for Federal jobs through the new USAJOBS. So we know the system is working for a lot of people,” he said.  “But I also know that’s cold comfort for those whose applications aren’t getting through. That's why we’re working with the Chief Human Capital Officers to extend non-emergency job openings for three weeks."

However, Rix suggested that the problems could go deeper and may not have been completely identified.

In an interview with Federal Computer Week, Rix outlined several other technical issues with USAJobs that her company has uncovered in testing and in conversation with industry experts and OPM staffers.

Avue manages job applications for several federal agencies and competes with OPM’s USAStaffing application, and Rix is considered a critic because of her writings on Huffington Post that preceded the USAJobs 3.0 problems. However, Rix also has substantial industry expertise in job search systems and Avue is authorized to perform certain operational tests on USAJobs.

Rix pointed to several other potential sources of bottlenecks in USAJobs. For one, OPM’s recent installation of additional capacity data transmission“pipes” for USAJobs may be inadequate, based on a comparison with similar tools used by Avue, Rix said.

While OPM, with the additional pipes, appears to have the equivalent of about 200 T1 lines to handle incoming traffic to the USAJobs website, Avue has the equivalent of about 2,700 T1 lines to handle similar or less traffic, she said.

“That suggests that OPM underestimated their capacity needs and essentially are bottlenecking the front end,” Rix said.

Rix also said she performed comparison tests using the Barracuda application that also suggest another capacity problem at USAJobs.

Her tests showed that for each visitor who visits the USAJobs home page, there are 16 hits on backend servers. By comparison, she tested the job search aggregator Indeed.com, which has a similar search configuration to USAJobs. Rix said the tests on Indeed.com returned a single hit per visitor.

“That suggests that their record high traffic on USAJobs really is not pent-up demand, it is an architecture issue,” Rix said. In essence, each visit results in 16 hits to the server, which she said strains the system and causes it to reach overcapacity on a frequent basis. It also could be overestimating traffic to the site by a factor of 16, she suggested.

Based on her testing, Rix said there are indications that OPM is not fully addressing all the capacity issues and operating issues yet. “They are not aware that they have more problems than what they are attempting to fix,” she said.