DARPA builds open-source rankings

Security for open-source software has always benefited from the fact that its code is public and anyone can attempt to fix a breach or, at the very least, make others aware of the problem.

The traditional reward for auditing open-source software and thus finding or fixing a security hole? Self-satisfaction and maybe an electronic pat on the back from a colleague — until now.

The Defense Advanced Research Projects Agency is funding a project, formally launched last month, to develop a Web-based rating system for volunteer auditors that will grade them based on the amount of open-source code they examine and the number of security holes they identify.

There's no gold medal for being the top security auditor, but points are awarded for solid code review and discovering bugs. Points will be deducted if a separate, future audit finds a missed glitch. Members of the open-source community hold good auditing skills in high esteem, and a good score will add to that reputation.

"It's an objective way for machines to rank who is more elite than whom. That will make people work harder," said Crispin Cowan, chief scientist at WireX Communications Inc., a Portland, Ore.-based provider of server software solutions.

Cowan created the project and named it Sardonix because "sardonic skepticism is a hallmark of the experienced system security practitioner."

DARPA is funding Sardonix with a two-year, $250,000 grant awarded last July, according to agency spokeswoman Jan Walker. "DARPA is interested in the open-source technology because we envision the proliferation of open-source systems and their use within the [Defense Department]," Walker said.

The foundation of the project is the Sardonix Security Portal, which went live Feb. 4 and will track which open-source code has been audited and by whom, Cowan said. The site (www. sardonix.org) received 6,600 unique visitors in the first three weeks and now has more than 225 people on its mailing list.

"The fundamental innovation is the scoring scheme," Cowan said. "That's what separates it from other open-source projects." Based on feedback he has received, Cowan believes the open-source community agrees in principle with the structure he proposed and said it's up to him to write the formulas, ask for feedback and then amend them.

DARPA would also like to see its investment succeed for DOD and commercial use, Walker said.

"By ensuring the use of sound development principles for effective security mechanisms and other secure system functions, DARPA will forge the way toward discovering how to build trustworthy systems and networks using open-source technol.ogy," she said.

Cowan would like to see Sardonix operational by the end of March, but that is not a final deadline. "The great thing about open source is that you ship it when it works, not on deadline, and that comes from [Linux creator] Linus [Torvalds]," he said.

By capturing the incentives that make code-sharing work and applying them to security audits, the Sardonix project is on the right track, said Terry Bollinger, principal information systems engineer for Mitre Corp., a think tank that conducts federally funded research.

"Instead of audits being incidental aspects of open source for which contributors receive little or no credit, the portal makes audits into enduring building blocks that can be shared by a community," Bollinger said.

It will be important for Sardonix to develop a style of auditing and contribution that lets people feel their work is making a difference, Bollinger added. "Without that, many of them will lose interest the first time they realize, 'Hey, didn't I audit that same piece of code last month?' " he said.