The Invisible Web
aka the Hidden Web
aka the Deep Web
by Jess Nevins
7 March 01
The "Invisible Internet" or "Invisible Web" are those pages which are not
covered and indexed by normal search engines.
Search engines only cover a limited amount of the Web and the Internet.
In 1999 one study showed that no single engine covered more than 19% of
the Internet, and cumulatively all of the search engines only covered 42%
of the Internet. In mid-2000 there were an estimated 1.5 billion pages
on the Internet, with the "distilled" or indexed pages numbering between
350 and 500 million. And a study early this year concluded that there are
around 2.5 billion pages on the visible Web and an astounding 1 trillion
pages uncatalogued, or invisible.
This obviously leaves a substantial amount of the Internet uncatalogued,
and although the situation is improving there is still much on the Internet
not covered by search engines.
There are a few basic types of invisible information on the 'Net:
pages that are not catalogued by a search engine when the engine crawls
pages whose access is restricted and require permission to view
pages that the search engines simply fail to find although they are linked
to and exist
pages which have been removed
pages which have data types, such as graphics, CGI scripts, Macromedia
flash, or PDF files, which most search engines do not index
Examples of "invisible web" information include much that is on various
Internet databases, information only found via ftp, and non-text information,
including photos, sound and image files.
The situation continues to improve, as more effort is put into making the
Invisible Web accessible, but as always users need to keep in mind that
the Internet should not be the beginning or the end of your search.
Good Resources for Finding Invisible Information
Engine Watch's Specialty Search Engines Page, found at
is a very good list of specialized search engines.
Search, found at http://gwis2.circ.gwu.edu/~gprice/direct.htm,
should be the first place you go to find Invisible Web information.
Search Engine Watch is an evaluative guide to various search engines and
provides comparisons and ratings of them.
The Specialty Search Engines Page is essentially an annotated directory
of specialty search engines--Yahoo! but for engines which search
the Invisible Web.
Does not have large numbers of specialty search engines listed, but those
included tend to be useful.
Web, found at http://www.invisibleweb.com/,
is another good resource for searching the Invisible Web.
direct search lists a very large number of specialized search engines and
databases, many of which cover the Invisible Web and are not otherwise
The engines included here are not annotated but are grouped by topics.
found at http://www.calvin.edu/library/searreso/internet/as/,
is another good resource.
It was originally part of Lycos' catalogue but was spun off in October
1999 on to its own site.
It is essentially an annotated, classified directory of Invisible Web sources
Alpha Search is a directory of gateway sites, those that collect and organize
subject-specific web sites. These, of course, include Invisible Web sites.