Topic
Servers, Hosts and Domains
For best results, you should work with the servers and conform to their expectations, derived from the behavior of other search engines. But you'll also need to defend against some tricks that have been developed to improve rankings (search engine spam).
Virtual Hosts and Shadow Servers
Virtual hosts and virtual domains allow one server and IP address to act as though it is many servers. To access this, be sure to include the HTTP/1.1 "Host" field in all requests. Most web hosting services use these features to accommodate client hosts. In those cases, you should be sure to accept these URLs and index them using their own host name, although the IP address may be the same.
However, a few search engine spammers will create multiple hosts and even domains, and point them all at the same pages (occupying more of the desirable high rankings in the results). In addition, they may submit or create links to the IP address and even a hex or ten-digit version of the address. The alternate version is sometimes used to get around firewalls and proxies, but it's also used for search engine spam. You may want to do a random check on IP addresses and make sure that similar IP address pages are not just duplicating data, or at least design for future spam checks on this issue.
User-Agent
Referer Field
Another helpful way of working with webmasters is to include the referring page in the HTTP header "Referer" field (yes, it's misspelled). This is the page containing the link you are following. You may want to add this only when you are doing a first crawl, so you don't have to include it in the index database. In any case, it will help webmasters who read their logs to locate bad links and generally understand what you are doing and how you got where you are.
Accept
Robots.txt
First, read the contents and links from the Robots Exclusion Protocol and Web Robots FAQ and the Guidelines for Robot Writers. They're old but still definitive. There are are a couple of additional checks you may find useful:
- Make sure you can read the robots.txt file whether the line-break characters are LF, CR or CR/LF.
- Assume that the web server is not case-sensitive (better to be conservative)
- Some people forget to put a slash / to indicate the root directory: if you see a disallow without a slash, assume it starts at the root.
META Robots Tag
In addition to the robots.txt file, the Robots Exclusion Protocol allows page creators to set up robot controls within the header of each page using the Robots META tag. Be sure you recognize these options:
- <META NAME="robots" CONTENT="noindex"> do not index the contents of the page, but do follow the links.
- <META NAME="robots" CONTENT="nofollow">, you can index the page but do not follow the links
- <META NAME="robots" CONTENT="noindex,nofollow">, you should neither index the contents nor follow the links.
- <META NAME="robots" CONTENT="index,follow">, not required, default behavior.
Be sure that you can handle spaces between noindex and nofollow, capitalization variations, and even a different order (nofollow, noindex).
Do not mark these pages as off-limits forever: the settings may change, so you should check them again in future index updates.
Indexing Speed
Although many web servers are perfectly capable of responding to hundreds of requests a second, your indexing robot should be relatively conservative. I recommend that you do one page per ten seconds per site: it will be much faster than most of the other search engines (they tend to do very slow crawls these days). One per ten seconds will make sense to whoever is reviewing the web server log.
Avoid hammering servers! web hosting providers may have multiple virtual hosts per IP address, and multiple IP addresses per physical machine. They are likely to disallow your robot via robots.txt or IP access controls if you overload them.
Server Mirroring and Clustering
Following Links
Simple HTML links are straightforward, however there are many that are not simple at all.
HREF Links
Here's a checklist of things to watch out for in HREF links:
- Port numbers (A HREF="http://www.domain.com:8010"). This is entirely legitimate and the robot should follow this.
- Anchor tags (A HREF="page.html#section). In this case, the robot should strip the text after the # and simply go to the page.
- Extra attributes such as mouseovers, which you should simply ignore
<A HREF="folder/mainpage.ht ml"
onMouseOver="display(6);display(8);self.status ='status message'; return true " onMouseOut="display(5);display(7);"><IMG SRC="image.gif" ALIGN="MIDDLE" BORDER="0" WIDTH="24" HEIGHT="23" NATURALSIZEFLAG="3"></A> - Quotes in URLs - some content-management programs will insert single or double quotes (', ")in a URL string. In general, stripping these characters should give you a working URL.
Relative Links
Absolute links (those starting with http://) are easy. Relative links are trickier, because page creators are fairly bad at understanding them, content-management programs generate weird ones and browsers go to great lengths to decode them.
The rules are described in RFC 1808 and RFC 2396: basically it is like Unix addressing, where the location starts from the current directory and go down through child directories using slashes (/) and up through parent directories using two dots (..). A variation starts with a slash, meaning the root directory of the host. With these tools, any page can refer to any other page on the site, but it can get confusing:
- some content-management systems add a dot (.), meaning the current directory, to relative links, even though it's not necessary. Ignore this.
- confused page creators can add excessive parent directories, accidentally directing clients above the root level of the host. Browsers attempt to compensate for this by rewriting the URL to start at the host, so your robot should do the same.
- other bizarre combinations can occur, such as this one reported by a search engine administrator: <a href="foldername/http://mydomain">. Your robot should be robust enough to ignore these without problems.
Capitalization and Case Sensitivity 
BASE Tag
Stored in the <HEAD> section of an HTML document, the BASE tag defines an alternate default location for relative links. So if the page is www.example.com but it includes <BASE href="http://www.example.com/subdir/index.html">, all relative links in this document should start from the directory subdir, rather than the root.
Other Kinds of Links
Link Tag
Links to Frames
The links to pages in framesets are in src attributes in the tag, like this: <FRAME src="page.html">, within a larger <FRAMESET> tag. Just treat the SRC as HREF and you're fine. There may also be normal text, including tags, within the <NOFRAMES> tag.
Note that the linked framed page may be somewhat unintelligible by itself, as when the search engine returns a link in a results list, and the end-user clicks on that link. Smart page creators work around this with navigation and JavaScript, but there's not much a search engine can do about it without storing a lot of context information.
Image Links
In general, IMG SRC attributes do not contain links to indexable pages, so requesting them would waste processor power, bandwidth and time.
Client Side Image Maps are HTML coordinates within an <AREA> tag or a larger <MAP> tag using the familiar HREF format. All you have to do is ignore the shape and coords attributes.
Server Side Image Maps are special files only existing on the server. They also include a coordinate dispatch system, but it's harder to get at. You have three choices
- ignore these links
- try sending all or a random set of coordinates to the server and see what happens (for example "http://www.acme.com/cgi-bin/competition?10,27")
- ask the server to send you the image map file and try to decode it yourself. For an example, see the NCSA tutorial.
Object Links
JavaScript Links
JavaScript can generate text using the commands document.write and document.writeln. If you see those commands, I'd recommend parsing through to the end of the JavaScript and extracting any links you can locate, probably indicated by HREF, http://, .html or .htm.
JavaScript is also used for menus and scrolling navigation links. In this example, the JavaScript command is "onChange" and you can see the HREF is dynamically combined with the links.
<p>Choose a Page:</p>
<form name="jsMenu" >
<select name="select"
onChange= "if(options[selectedIndex].value)
window.location.href= (options[selectedIndex].value)" >
<option value="..index.html" selected>Home</option>
<option value="a.html">Page A</option>
<option value="b.html">Page B</option>
<option value="c.html">Page C</option>
</select>
</form>
Again, the best that you're likely to be able to do is parse through this text and locate the ".html" (and other known page extensions).
Page Redirection
There are two ways that webmasters can indicate to the clients that a page has moved or is not really available at the original URL.
Server Redirects
Meta Refresh
The META refresh tag is designed to update the contents of a page, perhaps after performing processing or as part of a graphic interaction. Some sites use the META refresh tag to redirect clients from one page to another, although the HTML 4.0.1 spec says not to do this. However, it may be the only option for page creators without access to server redirects. The syntax is:
<META http-equiv="Refresh" content="1; URL=page.html">
The slightly odd construction indicates how long before the refresh and the link to the target page: note that it does not have quotes after the URL= but does at the end of the string. Your robot should follow this link, but I'm not sure if it should index the contents of the referring page. Perhaps if it's over a certain length, or if the refresh interval is over a certain time. Some search engine optimization gateway pages use this technique to improve their rankings in the engine while still showing browser clients the full contents of their pages.
Directory Listings
File Name Extensions
The robot should accept pages which end in ".txt" with a MIME type of text/plain -- they are almost always useful and worth indexing. However, if the page ends in ".log", it's probably a web server log file, which should not be indexed.
Most HTML pages with a MIME type of text/html which do not end in ".htm" or ".html" are dynamically generated by a script or program on the server. But they are generally straightforward HTML and should be indexed normally.
Common file name extensions include:
- .ssi and .shtml - Server-Side Includes
- .pl - Perl
- .cfm - Cold Fusion
- .asp - Active Server Pages
- .lasso - Lasso Web Application Server
- .nclk - NetCloak
- .xml - XML text files (MIME type text/xml, becoming increasingly important!)
Dynamic Data Issues
Entry IDs
Session IDs
Cookies
Cookies are a much more sophisticated way to store state information about a client. For more information, see the Builder.com article. In general, if you include a way to store cookies from a site and send them back on request, interaction with sites that use them will be smoother and more straightforward. Excalibur stores cookies and sends them back correctly.
Domino URLs
Infinite Links
Infinite Loops
Default Pages
Sites will probably have links to a single page as both a directory URL (www.example.com/dir/) and a default name within the directory (www.example.com/dir/index.html). The server will automatically serve the default name when the directory is requested. The most common names are:
- index.html or index.htm
- default.html or default.htm or default.asp
- main.html or main.htm
- home.html or home.htm
You may want to index these separately or check for duplicates and delete one version or the other.
Problem Link Status Codes
2xx: Successful
Web servers will reply with a 200 when they serve a normal page, or when the client sends an "If-Modified-Since" request and the page has been changed since the indicated date. Other 2xx codes can be safely treated the same way.
404: Page Not Found and 410: Gone
Other 4xx Status Codes
5xx Status Codes
Updating the Index
To update the index, you should revisit the pages periodically to locate new and changed pages.
It's not clear how often is right. Many search engines track how often various sites and pages change, and revisit them according to their own internal schedules. Note that some servers pay transmission fees by the byte, especially those in low-technology regions, so constant revisiting can cost them significant amounts of money.
Expires
HTTP/1.1 includes an Expires field which tells you when the information on the page is no longer current. This is mainly used for caching but is also good for indexers, which can revisit the site on the expiration date.
If-Modified-Since
Modification Date Problems
Some servers, especially those sending dynamic data, always set the modification date to the current date and time. Some search engines check all pages against the contents of the index before adding them -- if the page is the same as one that is in the index, the new page is considered a duplicate and is not added. Other engines simply re-index the page with the new date.
Note: many pages change only in the content of the links to advertising banners. To avoid excess index updating load, the robot could ignore offsite link changes in IMG tags when computing a checksum.