Searching the Web, NSA style

The National Security Agency just released "Untangling the Web," an unclassified how-to guide to Internet search. It's a sprawling document, clocking in at over 650 pages, and is the product of many years of research and updating by a NSA information specialist whose name is redacted on the official release, but who is identified as Robyn Winder of the Center for Digital Content on the Freedom of Information Act request that led to its release.

It' is a droll document on many levels. First and foremost, it's funny to think of officials who control some of the most sophisticated supercomputers and satellites ever invented turning to a .pdf file for tricks on how to track down domain name system information on an enemy website. But "Untangling the Web" isn't for code-breakers or wire-tappers. The target audience seems to be staffers looking for basic factual information, like the preferred spelling of Kazakhstan, or telephonic prefix information for East Timor.

Still, the author seems unable to resist frequent allusions to the cloak-and-dagger work of the NSA in the screen grabs she uses as illustrations. A demo of how to uses Windows Live Search features a request for information on Uranium-235. A how-to on the use of video aggregator Blinkx is keyed to a search for Iranian president Mahmoud Ahmadinejad. The domain-lookup function of the WHOIS database is explained with a look at the DNS information of security firm Kaspersky Labs. Instructions on a complicated Google feature that allows users to view pages returned in searches stripped of any images is based on a search for DPRK.

Perhaps by using this technique, an NSA searcher could get info from the North Korean government website without tell-tale pings to the host network. But it seems unlikely that the puzzle-masters and cryptographers at NSA would find themselves in that sort of predicament.

It was last updated in February 2007, and the six year lag gives "Untangling the Web" the feel of an unearthed time capsule. In 2007, before the financial crisis, there was a mini-boom in specialized search, and this is reflected in the many now-defunct search services that are listed in the guide. The author, clearly a seasoned web searcher, has strong opinions about the utility and futility of certain techniques, and goes out of her way to tsk-tsk the "mess" that was Yahoo's front page at the time.

The document contains some very useful information on searching the semantic or deep web – the disjointed mass of information and documents buried in databases that typically don't show up in Internet searches, and these are worth a read for people interested in snooping for spreadsheets, slide decks, and documents that businesses, governments, and large institutions may have made available online inadvertently.