Deep Web And The Dark Web: Venturing Into The Unknown

There are a lot of things the internet has to provide. Built by humans and populated by them, information is abundance. There are a lot to know and a whole more to learn, and those only scratch the surface of its knowledge.

The web we usually know, is clear and bright. Everything seems to be accessible with a few keywords to start with. Web pages are glamorous, and things get pretty easy here. This is what the Surface Web is, the part of the web where search engines and regular browsers can go.

If we go deeper than that, we venture to the more unknown place, called the Deep Web. This is everything online that is not indexed by search engines. Then there is the Dark Web that generally allows users to remain anonymous and require particular software to access.

So what's the differences between those two? "Deep Web" and "Dark Web" have been mentioned many times since Ross Ulbricht created the Silk Road, and many people consider them synonymous. But actually, the two aren't the same thing.

Surface Web - Deep Web - Dark Web

Deep Web

The Deep Web is also called the invisible web or the hidden web. It's parts of the World Wide Web where contents are not indexed by standard search engines for any reasons. Deep Web is the total opposite of the Surface Web, and it includes many common websites to web mail, online banking and also paid services with paywall, video-on-demand, and many more.

First coined by computer scientist Mike Bergman in 2000, he described the Deep Web as searching the internet by comparing to drag a net across the surface of the ocean. Many can get caught in the net, but the wealth of information and data that are deeper should be missed.

Most of the web's information is buried deep inside websites, and no search engines can find it. Traditional search engines only work on the Surface Web, and they cannot see or retrieve any information in the Deep Web.

In 2001, the Deep Web is said to have orders of magnitude larger than the Surface Web. This means is that no matter how much Google has in its indexed database, they only scratch the surface of the whole web.

The reason why search engines couldn't index the Deep Web is because:

  • Contextual web pages have different content varying for different access contents.
  • Dynamic web pages that respond differently when submitted a query. They can be only accessed through form or input elements.
  • Websites that limit access to their pages through Robots Exclusion Standard or CAPTCHAs, or other means to prevent search engines from indexing them.
  • Web pages that were created as non-HTML/text content, or using file formats not known by search engines.
  • Websites that require registration and login to access.
  • Web pages that have scripted contents. May use JavaScript, AJAX or Flash.
  • Websites that were created intentionally hidden from the regular internet. Accessing them require special software, such as Tor, I2P and others.
  • Web pages that have no linked content to them (no backlinks). This may prevent search engines to ever find them.
  • Web achieves that are stored inside database.

What search engines can't see, are considered parts of the Deep Web. It's huge, but accessing them require tools or manual efforts.

Other examples of Deep Web contents can also be found when navigating the web without using search engines. government database, libraries, medical records, user data and others are also considered contents that are parts of the Deep Web.

Dark Web

The Dark Web, likes its name, it's "dark". It generally consists of websites that wanted to remain anonymous and require special software to access. It's an overlay network that needs configurations or authorization.

Dark Web is a small part of the Deep Web, and just like the Deep Web, it's not a place for search engines to be. Anonymity is the unifying factor of the Dark Web, and it's used for the foundation of its whereabouts.

The most famous content that resides on the Dark Web is found in the TOR network. The TOR network is an anonymous network that can only be accessed with a web browser called the TOR browser. Because websites created intentionally hidden and have anonymity in mind associated withe the TOR network, Dark Web is most widely known for its illicit activities.

From whistleblowing, gambling, anonymous chat, abuse, books, directory, cryptocurrency, porn, puzzles, hosting, forums, wiki, mail, bitcoin and illegal and criminal activities such as blackmarkets for guns, drugs and narcotics, software exploits, fraud and counterfeiting services and even to the more nefarious human trafficking, hitmen for hire and child porn.

Others include small, friend-to-friend peer-to-peer networks, as well as large, popular networks like Freenet, I2P, and Tor, operated by organizations and individuals.

Tor Dark Web may be referred to as onionland, a reference to its top level domain suffix .onion and the traffic anonymization technique of onion routing.

Related: Seeing the Invisible Web