NASA is indexing the Deep Web

NASA is building a search engine

I would say “Wow” . Do you know: “There are hidden places on the internet even google can’t access“.It is private, or illicit, or simply unknown but NASA wants to help you reach it. The space agency announced on May-2015 along with the Defense Advanced Research Projects Agency (DARPA) it would help everyone discover the Internet commonly referred to as the Deep or Dark Web. Most Internet users first heard about it, if they’ve heard about it at all, in the context of Silk Road, which was one of the biggest online drug marketplace hosted in the deep web through hidden web services. Silk Road was only accessible using the anonymity-enhancing browser The Onion Router or TOR.

Now, NASA’s mission to explore the universe includes the furthest reaches of cyberspace. “It’s uncharted territory,” Chris Mattmann, NASA’s lead on the project. In a press release, NASA explained that it will help DARPA with its Memex program, which is working to “access and catalog this mysterious online world.”

Most of the Internet is hidden in the #DeepWeb. We’re making tools to search it http://t.co/dCPmMr9Wri @DARPA #MEMEX pic.twitter.com/CU5TSrwE2s

— NASA JPL (@NASAJPL) May 22, 2015

Government agencies are planning to clean the place up.If you go round the corner of the Deep Web for scoring illicit drugs, the Internet’s hidden channels have given birth to some pretty nasty illegal actors, including contract killers and pedophiles.

Related: Grams Search engine for Dark Web

Deep web contains the 96% of the whole internet and it has nothing to do with TOR and is inaccessible for more humdrum reasons. Some sites are not indexed by google because it thinks they are not worth its effort its time or behind paywalls generally scientific data. Why NASA’s Jet Propulsion Laboratory is interested in this information is all because of “Information sent by their spacecraft back to earth get dumped here“.

The idea is to organize access to the Deep Web’s content, and build a search engine alternative to Google, that will give NASA a better way to access data being uploaded by their machines.So the byproduct of this would be access to the hidden parts of the Internet to everyone in a smoother manner.

Related: TORCH Tor Search Engine

The goal is to not just build “The Best Search Engine for the Baddest things on Hidden Internet” JPL’s principal investigator on Memex, Chris Mattmann, told us. The NASA spacecraft sends a file format where it isn’t that much understandable by Google “[Those files] get in the second class of the web, that normally we call the Deep Dark Web…if you go to Google, you’re probably 10-30 clicks away from the science data—the actual information.”

NASA data is dumped in this gloomy, elusive (but not inaccessible) part of the web because the data make sense to humans but not to the crawlers. With Memex, Mattmann said, Web surfers will be “just one to four clicks away from the science data.”

This screen shot from the Image Space and ImageCat applications shows how the search tool can organize a large amount of information and look for relationships between data hidden in images. Credit: NASA/JPL-Caltech/Kitware/Continuum — This screenshot from the Image Space and ImageCat applications shows how the search tool can organize a large amount of information and look for relationships between data hidden in images. Credit: NASA/JPL-Caltech/Kitware/Continuum

Building a search engine for a particular domain is very much easy which are perfectly done by sites like Fandango and Yelp and other to come but a central search engine for searching the deep web across various different domains is where the real skill and complications come in.

And, Mattmann said, Memex is going through “the same kinds of search engine growing pains” as all search engines. “Being able to understand which sites are relevant, where to start crawling from. A lot of these crawl operations can take days or weeks… Even Google didn’t develop rank initially.”

There are other simpler versions of Memex already available. “If you’ve ever used the the Internet Archive’s Wayback Machine,” which gives you past versions of a website not accessible through Google, then you’ve technically searched the Deep Web, said Mattmann.

But once Memex, launched in September of last year, is fully realized, it could be a viable alternative to Google—maybe. “I don’t know if any government program could be a competitor to a commercial entity,” Mattman said, but Memex is doing in which Google has no interest “It’s not in their bottom line,” Mattmann explained, to do the type of web crawling DARPA and NASA are prepared to do.

For DARPA, this is coming full circle: Previously known as ARPA, the agency developed something called ARPANET in the late 1960s. ARPANET was an early version of the Internet as we know it. Now DARPA will help make it easier to mine.