As of January, 2012, this site is no longer being updated, due to work and health issues Related Topics

Web-Wide Search Tools

Search tools covered on this site index a single site, an Intranet or a limited set of sites creating a portal. Web-wide search engines attempt to cover the entire Web, indexing billions of pages.

Web-Wide Search Engines

Google (link-analysis means that the most useful pages are often first)

HotBot (default "and" on multiple terms)

AltaVista (lots of options like link: and url:)

AllTheWeb (useful results using the FAST search engine)

Yahoo, because librarians do rule.

AskJeeves, because people are smarter than machines

Deep Web Searching

Many web pages with useful information are stored in databases and generated as HTML pages only when searched, so they are not indexed by the standard robot spiders. This is known as the Invisible Web or the Deep Web. Some companies are now attempting to provide search access to these databases without making searchers go to the sites themselves. For example, La Scala's travel schedule or the Where the Wild Things Are video. Auction metasearch engines are a form of deep web searching.

Many search tools will index all available URLs, even if they have a question mark in the path. This allows them to combine the deep and surface web and create a comprehensive index on a particular topic. All they need is a set of links to the contents of the database, or a direct connection to a database, like AltaVista and Excalibur RetrievalWare.

The Invisible Web for Educators

The Invisible Web, June 11, 1999 by Chris Sherman - Nice overview of the problems with accessing information in databases.

Direct search Gary Price - Links to search interfaces for databases and other special information resources.

MetaSearch Engines

Meta searching involves sending queries to multiple webwide search engines and grouping the results in a useful way. This function can be performed by an engine at a web site, by a local application, or by a site server application.

MetaSearch Sites

Dogpile - shows you the top results from various search engines

MetaCrawler - searches financial information and shareware as well as search engines

Web Sites That Help in Your MetaSearch San Francisco Chronicle, October 20, 1998 by Avi Rappoport


Copernic 2000 -Windows application that combines search results in a window.

Sherlock - part of Mac OS, many plug-ins for additional search sites available.

Searching Every Internet Cranny / New tools let you query multiple engines and combine results San Francisco Chronicle, October 20, 1998 by Henry Norr

MetaSearch Servers for Sites

SearchTools report on adding a meta search engine to a web site or Intranet.

Other Useful Sites for Web-Wide Search

Search Engine Watch
The ultimate analysis and news site for information on web-wide search engines, by journalist Danny Sullivan.

Search Engine Showdown
Technical analysis of webwide search engines from eminent web librarian Greg Notess.
Cybermetrics: Web Searching Analysis
Skeptical discussion of web wide searching, based on quantitative analysis of search engines

Page Updated 2002-03-29