You can now read 5 articles in a month for free on BostonGlobe.com. Read as much as you want anywhere and anytime for just 99¢.

Edward Snowden made use of simple software

Bested NSA with low-cost tools

WASHINGTON — Intelligence officials investigating how Edward J. Snowden gained access to roughly 1.7 million of the country’s most highly classified documents say they have determined that he used inexpensive and widely available software to “scrape” the National Security Agency’s networks, and he kept at it even after he was briefly challenged by agency officials.

Using “Web crawler” software designed to search, index, and back up a website, Snowden “scraped data out of our systems” while he went about his day job, according to a senior intelligence official.

Continue reading below

“We do not believe this was an individual sitting at a machine and downloading this much material in sequence,” the official said. The process, he added, was “quite automated.”

The findings are striking because the NSA’s mission includes protecting the nation’s most sensitive military and intelligence computer systems from cyberattacks, especially the sophisticated attacks that emanate from Russia and China. Snowden’s “insider attack,” by contrast, was hardly sophisticated and should have been easily detected, investigators found.

Moreover, Snowden succeeded nearly three years after the WikiLeaks disclosures, in which military and State Department files, of far less sensitivity, were taken using similar techniques.

Snowden had access to the NSA’s complete files because he was working as a technology contractor for the agency in Hawaii, helping to manage the agency’s computer systems in an outpost that focuses on China and North Korea.

A Web crawler, also called a spider, automatically moves from website to website, following links embedded in each document, and can be programmed to copy everything in its path.

Snowden appears to have set the parameters for the searches, including which subjects to look for and how deeply to follow links to documents and other data on the NSA’s internal networks.

Among the materials prominent in the Snowden files are the agency’s shared “wikis,” databases to which intelligence analysts, operatives, and others contributed their knowledge.

Some of that material indicates that Snowden “accessed” the documents. But experts say the documents may well have been downloaded not by him but by the program acting on his behalf.

Agency officials insist that if Snowden had been working from NSA headquarters at Fort Meade, Md., he almost certainly would have been caught.

But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of data raised few alarms.

In at least one instance when he was questioned, Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities: As a systems administrator he was responsible for conducting routine network maintenance. That could include backing up the computer systems and moving information to local servers, investigators were told.

Snowden learned something critical about the NSA’s culture: While the organization built enormously high electronic barriers to keep out foreign invaders, it had rudimentary protections against insiders.

Through his lawyer at the American Civil Liberties Union, Snowden did not specifically address the government’s theory of how he obtained the files.

Loading comments...
Subscriber Log In

We hope you've enjoyed your 5 free articles'

Stay informed with unlimited access to Boston’s trusted news source.

  • High-quality journalism from the region’s largest newsroom
  • Convenient access across all of your devices
  • Today’s Headlines daily newsletter
  • Subscriber-only access to exclusive offers, events, contests, eBooks, and more
  • Less than 25¢ a week
Marketing image of BostonGlobe.com
Marketing image of BostonGlobe.com
Already a subscriber?
Your city. Your stories. Your Globe.
Yours FREE for two weeks.
Enjoy free unlimited access to Globe.com for the next two weeks.
Limited time only - No credit card required!
BostonGlobe.com complimentary digital access has been provided to you, without a subscription, for free starting today and ending in 14 days. After the free trial period, your free BostonGlobe.com digital access will stop immediately unless you sign up for BostonGlobe.com digital subscription. Current print and digital subscribers are not eligible for the free trial.
Thanks & Welcome to Globe.com
You now have unlimited access for the next two weeks.
BostonGlobe.com complimentary digital access has been provided to you, without a subscription, for free starting today and ending in 14 days. After the free trial period, your free BostonGlobe.com digital access will stop immediately unless you sign up for BostonGlobe.com digital subscription. Current print and digital subscribers are not eligible for the free trial.