The mySpider site specific search engine, which I created and programmed in Perl, offers many features to make your web site Administrator's job easier and more efficient.



Two text files allow the Administrator to easily configure the spider and search tools. Both files contain explanations and examples of the values for the configuration variables.

MySpider can spider and search multiple domain names and web servers.

Where necessary, custom coding of the spider engine ensures indexing of pages other search engines cannot "see".

As your site grows, it can become increasingly difficult to continuously redesign your navigation to keep it simple for the typical user to locate the wealth of information on your site.

With the mySpider search tool, your visitors will be able to quickly locate what they are looking for, at a fraction of the cost of the Google appliance.

The Administrator can configure the search tool's user interface to display categories for visitors to use in their searches.

As the spider engine creates or updates the search index, it reports:

• any errors it encounters during the spider process
• the total number and the URLs that contain redirects which were not added to the search index
• the total number and the URLs of stale web pages that are no longer linked from within the web site that have been removed from the search index
• the total number and the URLs of problem web pages that were spidered and should have been inserted into the search index but were not because of an error
• the total number and the URLs of any bad links, and the URLs of the pages they were found in
• the total number of URLs that were spidered for inserting into the search index
• the total number of URLs in the search index after the spider process was completed

Because stale pages that have been removed from the site get reported, the web site's Administrator will be able to quickly identify any anomalies that may have occurred on the web servers, such as sites being down or sections being unavailable.

With the Search Report tool you can also quickly monitor all of the search terms that have been used and compare similar search strings all in one simple console. Click here to view the Search Report Tool.

No matter how many different authors and editors there are adding content to the web site, these centralized administration tools can really help in the day to day administration of the site.

This web application/search tool offers many advantages to the web site's owner, visitors, and web master or web developer. For the complete details, visit the mySpider web site.

You can, of course, test the mySpider search function on this web site.

It has also been used on many other web sites. For example, the municipality of Chatham-Kent and the City of Pickering had for some time implemented it on their sites, but no longer do.

<< BACK    NEXT >>