Contents

Introduction

Outline of CyberSpyder Link Test`s Operation

Link Test Checks, Limitations and Future Directions

Starting CyberSpyder Link Test - Starting URL`s

Starting CyberSpyder Link Test - Set Options

Running CyberSpyder Link Test - The Main Screen

The Main Screen - Current URL Section

The Main Screen - Processing Status Section

The Main Screen - Controls Section

The Main Screen - Processing Section

The Reports - Summary Report

The Reports - Broken Link Report

The Reports - Can`t Check Report

The Reports - Warnings Report

Copyright and Trademark Notice

Program Registration


Introduction

One of the problems for anyone running a WWW site, is ensuring that all of the links on the site are operational.  In spite of being very careful in setting them up, almost every site has some `broken` ones.

Sometimes a link breaks because a page is removed or its location changed, and changes to one or more of the links to it are missed.  Other times links to other sites become broken because the destination document or image is no longer at the location expected.  Another common problem is missing quotation marks around a link reference - this has become more of a problem now that some browsers such as Netscape have become more particular about their parsing of HTML code.

CyberSpyder Link Test is a WebMaster`s friend as it will check all of the links on a site and report on any that no longer work!  When the program is started, one or more key URL`s are entered, CyberSpyder Link Test then works tirelessly until all the links have been tested and prepares a set of reports on any problems found.

The `WebMaster` may be responsible for all of the links in a server installation, or may be an individual or company with a large number of links under their `home page`. CyberSpyder Link Test may be used to handle either or both situations.


Outline of CyberSpyder Link Test`s Operation

When CyberSpyder Link Test is started, one or more initial URL`s are entered.  These URL`s should contain URL links which will result in CyberSpyder Link Test visiting every page within the site.  The starting pages do not need to contain links to all of these pages, but they should contain links which will point to other pages which will eventually lead to all pages on the site.

For example, although the `home` page at a site may not contain pointers to all other pages, it may contain a few links to some main pages for various areas of the site, which will contain pointers to the rest of the images and documents.  If a site is divided into two or more independent sections, then the appropriate starting URL`s would be to the `home` pages of these sections.  Similarly, if the server is `multi-homed`, then the appropriate starting URL`s would be the `home` pages for each of the virtual sites.

For private sites running on a service provider`s site, the starting URL`s should point to one or more pages within the private site area, but should not include any pages outside this area.

CyberSpyder Link Test examines these starting URL`s and extracts the host name (or IP address), the port number (if not included, the default port of 80 is assumed), and the path(s) to the starting page(s).  This information is used by CyberSpyder Link Test to decided whether a page referenced in a link should be parsed for additional links or if the link should just be tested to ensure that it works.  Pages referenced by links which include the same host, port and path as one of the starting URL`s are retrieved and tested for additional links, while those which do not include these are merely tested to ensure that they exist.

Some examples:

Starting URL - `http://my.host.com/index.html`

Referenced page -  `http://my.host.com/newstuff/new.html` - retrieved and parsed for additional links to be tested since it includes `http://my.host.com`

Referenced page - `http://not.my.host/special/great.html` - only tested to see that it exists (does not include `http://my.host.com)

Starting URL - `http://my.host.com/~personal/index.html`

Referenced page - `http://my.host.com/~personal/pics/special.html` - retrieved and parsed for additional links to be tested since it includes `http://my.host.com/~personal`

Referenced page - `http://my.host.com/~tim/index.html` - only tested to see that it exists (does not include `http://my.host.com/~personal)

The HTTP protocol supports various methods for retrieving linked items.  Two of these are used by CyberSpyder Link Test.  The `GET` method returns the file preceded with some header information added by the server.  Two of the items returned in this header are used by CyberSpyder Link Test when testing a link.  The first item is a response code - this will be 200 if the item is found or another code if it is not.  The second item identifies the document type - this is normally used by a browser to determine how to display the item.  The `HEAD` method returns the same header information as the `GET` method, but does not include the document.

For each link to be checked, CyberSpyder Link Test first requests the item from the server using the `HEAD` method.  If the item is found, and the document type is identified as an HTML document, and the URL is one to be tested for links, CyberSpyder Link Test uses the `GET` method to retrieve the actual document.  After retrieving a document, it is parsed and any links in it are extracted for testing.  The cycle repeats until there are no more links to test, at which time a set of reports are produced detailing the results of the tests.

The reports are produced as HTML documents and placed in a sub-directory `Reports` under the directory from which the program is run.  The summary document is called `index.htm`, and the other reports are linked from it.


Link Test Checks, Limitations and Future Direction

CyberSpyder Link Test tests only HTTP URL`s.  It will recognize other types of URL`s (mailto, gopher, ftp, etc.) and will optionally report on their existence.  Since the report is in HTML form, these may be checked manually from the report page, if desired.

CyberSpyder Link Test cannot check CGI URL`s (since it cannot know what data might have to be input by a user.  Any URL which contains `cgi` in it is assumed to be a `cgi script` URL and is not checked.  These will be included in the report along with the other types of URL`s which cannot be checked.

This version of CyberSpyder Link Test also cannot check URL`s with restricted access (those that require the entry of an ID and Password).

The program will also optionally warn of missing <html>, </html>, <head>, </head>, <body>, and </body> tags.  While most current browsers usually are not affected if these are missing, there is no guarantee that this will always remain the case, particularly as more and more features are added to the protocol and new browsers become available, so these missing tags should be added to your documents.

It is expected that future versions of CyberSpyder Link Test will include provision to permit entry of ID and Password for restricted URL`s so these can be tested.  Consideration is also being given to adding ftp URL`s to those which will can checked.

It should also be remembered that a full test of a set of pages can take considerable time since the program must check every link, and parse every page on the site being tested.  The actual time cannot be predicted as it depends mainly on the connection speed, the response speed of the site being tested, and the response speed of all of the remote sites referenced by the links.


Starting CyberSpyder Link Test - Starting URL`s

When CyberSpyder Link Test is first started, it will display a splash screen while it initializes, then the main operational screen appears, then the window for entering the starting URL`s:

The starting URL`s are entered in the way they would be entered for a browser (http://the.host.com/.......).  Pressing enter will add the URL to the list to be used to start the check.

The `Exit Program` button may be used to exit from the program, the `Set Options` button will cause a window to be displayed for setting user configurable options (See Starting CyberSpyder Link Test - Set Options).

Once all of the starting URL`s have been entered, click `Done` to exit this window and return to the main screen.


Starting CyberSpyder Link Test - Set Options

Selecting the `Set Options` button from the `Enter Starting URL`s` window or the `Options` button from the main screen will bring up the `Options` window:

The options are as follows:

Set Timeout:  This controls the length of time that CyberSpyder Link Test will wait for a connection to be established with a server, how long the program will wait for a response from the server and the delay allowed if the server apparently stops transmission part way through a document.  If the time limit is reached, the program will retry the operation three more times (for a total of four attempts) before reporting the link as broken.  The time is entered in seconds.  It is suggested that the default value of 60 seconds be used initially until some experience with the program has been gained.

Can`t Check Filter: CyberSpyder Link Test records a `can`t check` message for all none `http` URL`s.  This can result in a long report, particularly if every page on the site includes a `mailto` URL.  Any of these messages can be suppressed by selecting the check boxes for one or more of the URL types in this section.

Warning Filter: CyberSpyder Link Test produces warning messages if any of the tags shown here are missing in a document.  These warnings can be suppressed by checking the box in this section.

Any changes made in this window may be canceled by clicking the `Cancel` button, or the values may be reset to their defaults by clicking the `Defaults` button.


Running CyberSpyder Link Test - The Main Screen

Once all of the starting URL`s have been entered and the `Done` button has been selected, the main CyberSpyder Link Test screen will be displayed:

This screen allows the user to control the operation of the program and to monitor its progress. The various areas of this screen are each described in their own section.


The Main Screen - Current URL Section

This section shows the current URL being tested:

The items in this sections are:

Service:  The URL type.  In this version of CyberSpyder Link Test this will always be `http:`.

Host Name: The name of the host computer for this URL.  This may not be shown.  If a URL contains an IP address only, it is not always possible to get the host name.

Host IP: The IP address of this host.  Note that this will always be shown since it must be know before the host can be contacted.

Port: If the URL is of the form `http://the.host.name:xxxx`, the value following the semi-colon after the host name is the port number.  This value will be extracted and shown here.  If the URL does not contain the port number, the default http port value of 80 is used and will be shown here.

URL: This is the actual URL being processed.


The Main Screen - Processing Status Section

This section shows the current  processing status so the operation of the program can be monitored:

The items in this section are:

Current Operation: This shows the current operation being performed by CyberSpyder Link Test.

Response Timer: This is active during the `Contact Host`, `Wait Response` and `Receive` operations and shows the number of seconds remaining before the operation will be considered to have failed (resulting in a retry).

Retry Count: This shows the number of times the operation has been retried.  Failed operations will be retried 3 times (for a total of 4 attempts - the initial one plus the 3 retries).

In addition, when a document is retrieved and parsed, the blank area will display the number of bytes retrieved as the document is received, and the number of bytes remaining to be processed as the document is being parsed:


The Main Screen - Controls Section

This section is the `control panel` for the program.  After the starting URL`s have been entered, and before the program has been started it will appear as:

The items in this section are:

Elpased Time:  This show the elapsed time since the test was started as hh:mm.  The timer starts when the `Start` button is pressed and continues until the program completes.  The time will be stopped any time the program is paused.

Status:  This indicated the current status - Wait Start, Running, Waiting to Pause, Paused, etc.

Options button:  This will cause the Options window to be displayed.  This button is only available before a test has been started - once started this button is disabled.

Start button:  This starts the program.  The button is disabled once the program is started.

Pause button:  This button becomes active when the program is started.  Selecting this button while the program is running will cause the program to complete its current cycle, then processing will be paused and the caption on the button will change to `Go`.  Pressing it when the program is paused will cause processing to continue.

Exit button:  This will end the program.  This button is disabled when the program is started and remains disabled until the program is paused or completes the test.  If `Exit` is selected when the program is paused, the program will display a dialog to confirm that the exit was intended and will ask if the reports on the results to that point should be produced.

When the program is running, the Controls section will appear as shown below:


The Main Screen - Processing Section

This section is a counter display:

The counts displayed are:

Left:  The number of links left to check.  This will initially show the number of starting URL`s entered.  During processing, it is incremented each time a new link is found to be check, and decremented each time a link has been checked.

Checked:  The number of links that have been checked.

Good:	The number of checked links which are O.K.

Broken:  The number of links which do not work.

Can`t Check:  The number of links found which cannot be checked (mailto, ftp, etc.).  Not count is made for any items selected to be filtered out in the Options screen.

Warnings:  The number of warning messages recorded  (The warning count for missing tags will not be included if these were selected to be filtered out in the Options screen).


The Reports - Summary Report

CyberSpyder Link Test produces up to four reports.  The first report, a summary of the results, is always produced.  The other three, one for `Broken` links, one for `Can`t Check` links, and one for `Warnings` are produced only if there are any items to report.

The reports are produced as HTML documents and placed in a sub-directory `Reports` under the directory from which the program is run.  The summary document is called `index.htm`, the report for broken links is called `broke.htm`, the report for `can`t check` links is called `cant.htm` and the warnings report is called `warn.htm`.  These reports can be viewed using your favorite browser.  Load the `index.htm` file first.  It will contain links to the other reports.  The content of the summary report will appear as:


The Reports - Broken Link Report

If any broken links are found, a report containing entries, such as those shown here, will be produced by CyberSpyder Link Test.

The heading line will indicate the reason for the break.  If the error is internal to the document, or results from not being able to access the host computer, the heading will show three question marks followed by an explanation of the reason for the break.  If the break is indicated by the host response (Not Found, Moved, etc.), the heading will include the HTTP response code returned by the server along with the message returned.  (Different servers will have slightly different descriptions for the same response codes.)

The second line of the entry will identify the document containing the reference (as a clickable URL) with the title of the document in parentheses.

The third line will indicate the reference type (anchor, image source, etc.).

The fourth line will show the actual URL content as contained in the document.

The fifth line will show the URL as it would be translated by a browser (browsers turn partial URL`s into full URL`s for processing.


The Reports - Can`t Check Report

If any links are found which can`t be checked, a report containing entries, such as those shown here, will be produced CyberSpyder Link Test.

The heading line will indicate why the link cannot be checked (normally a service type not supported by the program)

The second line of the entry will identify the document containing the reference (as a clickable URL) with the title of the document in parentheses.

The remaining lines will identify the actual reference within the document.


The Reports - Warnings Report

If warning messages are produced, a report containing entries, such as that shown here, will be prepared by CyberSpyder Link Test.

The heading line will contain the warning message.

The second line of the entry will identify the document containing the reference (as a clickable URL) with the title of the document in parentheses.

Any remaining lines, if present, will contain more information about the warning.


Copyright and Trademark Notice

CyberSpyder and Aphrodite`s Software are trademarks of Thomas A. Aman.

IPPort (used as a component of this product) is a trademark of devSoft Inc.

CyberSpyder Link Test is copyright (1996) Thomas A. Aman, all rights reserved.

This software is protected by copyright law and international copyright treaty.

You may use the shareware version free for 60 days for evaluation purposes.  After that time, you must register it, or discontinue using it and remove it from your computer.  The shareware version may be freely distributed as long as all of the files are included and no other files are added (such distribution should be done using the original `.zip` file to ensure that this requirement is met).

The registered version must be treated like a book - it may be copied onto a computer and freely moved from one computer to another as long as their is no possibility of it being used by more than one person at a time.

You may transfer your rights to another person, provided you remove any copies you have and discontinue using it.

Limitation of Liability

This software and the accompanying files are sold as is and without warranties as to the performance or merchantability or any other warranties whether expressed or implied.  The extent of liability of the seller is hereby limited exclusively to product replacement or refund of purchase price.  In particular, in no event shall T. A. Aman be liable to you for any damages, including any loss of profits, loss of data, including but not limited to special, incidental, consequential, or indirect damages arising from the use of this software.

IPPort Custom Control

The IPPort Custom Control VBX included with this program is a licensed version and a license file (IPPORT.LIC)  is included as part of this program.  The license file may only be used with CyberSpyder Link Test.  You may, however, use this control for your own projects without the license file (the IPPort help file, which provides all of the necessary documentation is included with the package), but it is then subject to the terms and conditions as appear in the IPPort help file (30 days free use in your design environment and for evaluation purposes only).


Program Registration

CyberSpyder Link Test may be registered in two ways.

By Check or Money Order

Send a check or money order, payable to `T. Aman`, for $25.00US to

T.A. Aman

1600 Ainsley Dr.

Ottawa, Ontario

Canada

K2C 0S4

Note to Canadian residents:  The 25.00US price will be accepted as the full price including the GST and any applicable Provincial Taxes.

Through CompuServe Registration Service

You may register via CompuServe by going to the Shareware Registration Forum (GO SWREG) and following the forum instructions.  The Registration ID for CyberSpyder Link Test is 9702.

When registering, please ensure that you include both your postal address and your e-mail address, if you have one.

When your registration is received, you will be given a registration number to be entered in the startup screen which will remove the `nag` notice which displays at startup.  This will be sent to you via e-mail, if you have supplied an e-mail address, otherwise it will be sent by ordinary first class mail.

Registered users will be notified of any bug fixes which may be found necessary in this version and will receive free upgrades for a period of one year.  After that time they will be given special pricing on any future upgrades.  Registered users are entitled to support which will normally be provided via e-mail or Compuserve mail.

The author may be contacted at the above address, via Internet e-mail at thaman@magi.com or via Compuserve mail at 72667,3474.