As of January, 2012, this site is no longer being updated, due to work and health issues Related Topics

Distributed Indexing with SOIF and RDM

Harvest and WAIS pioneered distributed indexing technology, allowing local servers to gather and index data and then pass it on to search servers. This allows indexes to work together and update as needed, rather than forcing each search indexer to crawl each site separately. This can improve site searching of very large sites, as well as reducing server overhead.

Harvest's data format and syntax is called SOIF (Summary Object Interchange Format). Netscape spent some time creating the Resource Description Messaging (RDM) mechanism, which uses SOIF as it's syntax to define data and HTTP to transmit it. RDM also supports a schema to describe each specific SOIF, an RDM Server Description and a flexible mechanism to browser or search the data without query language dependence.

This was a hot topic in late 1996 and early 1997, but has a very low profile now. It's not clear to me how SOIF and RDM work with XML, RDF and other new standards.