Templeton's Known Limitations

Due to the vast options available on most web servers, Templeton is not capable of retrieving some types of documents. These known limitations effect virtually all types of web robots and are described below. Many of these limitations are expected to be corrected in later releases, although currently no dates have been announced.

Protocol Limitations

• Templeton attempts to identify file types using HEAD requests. Some servers (especially database systems) return errors for HEAD requests but complete documents for GET commands. Templeton may incorrectly determine a file as unretrievable if the HEAD request fails.
• Some servers do not return error 404 (or 4xx) for invalid documents. A few servers have been seen returning HTML documents containing a text error message and return code 200 (OK). Templeton will incorrectly process these documents; the text error message with return code 200 will be written to a file and any links within the document will be identified and processed. (In the case of imagemaps, these documents will not be processed.)
• Templeton does not support POST commands.
• Netscape "client pull" (aka refresh) tags are not supported. These tags are usually in the form: <META HTTP-EQUIV="Refresh" CONTENT="?;URL=URL">
• Templeton does not support text entry fields, selections, or pushbuttons.
• Image Maps are "clickable" pictures. By clicking on a portion of the picture, you activate a link to another HTML document. There are currently many ways to resolve imagemap URL addresses:
  1. Client-Side processing requires the URL to be identified by the client application. Templeton supports this method.
  2. CERN/NCSA server-side mappings use the URL path to describe the mapping program (generally imagemap) and the server's path to the map file. Templeton attempts to resolve the map file's path from the applications path, but may fail if the server does not return error 404 (not found) for invalid document paths.
  3. Imagemaps may refer to executables. Since Templeton cannot decompile executables, these imagemaps cannot be retrieved.
  4. Some servers support map directories where all files in the directory are processed as map resolution files. These servers do not permit the retrieval of the map document. Consequently, map directory references are not supported by Templeton.
  5. Some servers support mapping extensions. Usually files ending with ".map" are processed as image maps. Some of these maps are retrievable, while others are not. (It depends on how the server is configured.) Templeton can process some of these maps when they are retrievable.
• Templeton does not support cookies. A "cookie" is a string of data sent from the web server to the web client. The web client is expected to return the cookie information to the server. It has been rumored that some servers are denying access to documents if cookie information is not returned. To date, we have not found any such server, but if they exist, Templeton will not be able to retrieve these documents. (According to the HTTP 1.0 and 1.1 standards, cookies are optional information and cannot be "demanded".) Templeton may support cookies in a later release.

Remote Application Limitations

CGI (Common Gateway Interface)

• CGI applications which return a GIF or JPG file do not necessarily have the extension ".gif" or ".jpg". The file containing the retrieved data is the same as the CGI application's name+parameters. Filters such as "deny *.gif" will not work on them. Netscape, MS Internet Explorer, and WebExplorer properly view the file as a graphic.

  Prior to 1.970, probable CGI applications where placed in their own directories with the extension ".html" instead of ".gif" or ".jpg".

  The filename may be changed in a later release.

• CGI applications which take optional query arguments may be mistaken for directories or sent varied requests. CGI applications are specified in 3 ways:
  1. http://host/path/cgi
     In this case, Templeton will first test if the URL specifies a directory. This is done by appending a '/'. This may change the CGI application's interpretation of the request. If the request succeeds, then the application may be mistaken for a directory specification.
  2. http://host/path/cgi/arg1/...
     In this case, the directory structure is built and the final argument becomes the filename. This may cause problems when more optional arguments are identified. For example:
     http://host/path/cgi/arg1
     http://host/path/cgi/arg1/arg2
     There is no way for a web robot (or browser) to know the file specified is actually a CGI application. Templeton will attempt to first resolve "http://host/path/cgi/arg1/" (note the trailing '/'). If this succeeds, then the URL is assumed to specify a directory. This may also alter the CGI interpretation of the application.

     http://host/path/cgi?arg1...
     In this case, the application is correctly identified as a probable CGI application. Any "/" that exists within the query (after the "?") will be translated into a "_" so that directories will not be built. (If the "/" into "_" translation is not performed, then the same problem mentioned in #2 can occur.)
  The query may be modified when Templeton looks for a directory. It should be noted that most CGI applications tested so far return error code 200 (OK) when given invalid arguments. This leads to the protocol limitation (listed above) where a server returns 200 instead of 404 (or 4xx) for invalid documents.

  Scripts (Embedded programming code in the HTML document)

• Templeton does not support JavaScript or VBScript code. Scripts in HTML documents are not interpreted and may fail to work in retrieved files. All JavaScript entities found when processing a URL are noted in the log files. This feature may be available in later releases.
• Embedded <SCRIPT ...> ... </SCRIPT> code will not be interpreded by Templeton. This may break working embedded code when an explicit URL is used. The presence of scripts and embedded scripts are noted in the log files. This will be corrected in a later release.
• Embedded <% ... %> code (VBScript) will not be interpreted by Templeton. This may break working embedded code when an explicit URL is used. The presence of VBScript is not noted in the log files. This will be corrected in a later release.

  Object code (compiled code such as Java, located on a remote server)

• Object code which loads other objects from the server will not be fully retrieved. This is because Templeton does not decompile Java objects -- required for determining which other libraries or objects are needed by the applet.
• Object code which contains URLs are not modified (we retrieve the raw data but make no attempt to modify it). Java code which uses explicit URLs may fail to work.
• Object code which communicates with JavaScript may fail since we do not interpret JavaScript yet.

Network Limitations

• Templeton may seem to hang when the nameserver is unable to resolve a hostname. This "hanging" is the nameserver request timing out. This interval is system dependent, but is generally no longer than two minutes. Currently, Templeton is single-threaded and at the mercy of this limitation. This may be corrected in later releases.
• Templeton may run beyond the ending time specified by RestrictStopTime and RestrictDuration if the DNS query hangs. See previous network limitation.
• gethostname() under OS/2 does not resolve properly if the environment variable HOSTNAME is not set. This function also has problems when operated on a gateway using a private LAN and a PPP/SLIP interface, with hostnames that begin with numbers (such as those provided by Worldnet.att.net), and with machine names containing a '.' character. Currently, the user must set their username in the configuration file. This will be corrected in a future release.
• Templeton cannot determine if a mounted drive supports long filenames. It is assumed that the drive does support long filenames, but this may be a bad assumption. For example, a DOS machine can export a partition to a Linux system running Templeton. The user must set the FATFlag in the configuration file to overcome this problem.

Program Limitations

• Excluded URIs (path/files) that contain "*" as a text character will be incorrectly interpreted as a wild card "*". Thus, denying a file called "/*.gif" will be interpreted as excluding all ".gif" files. The exclusion can either occur in the Templeton configuration file, as a manual excluding during operation, or in a server's "robots.txt" file.
• Changed in version 2.200. RestrictHost can only be used to specify a single host. Multiple allowable hosts can be specified with the "Allow" configuration option or by entering "LA" (List restrictions and Add) while Templeton is running.
• Command_html, etc. specified in the configuration file cannot be longer than 511 characters. The expanded '%' commands cannot be longer that 1023 characters. Longer commands will be truncated.
  Note (1): scripts which require an ending character sequence will not receive the ending sequence if the command is truncated.
  Note (2): scripts and shells which cannot handle more than 255 characters in a command (e.g., DOS "command.com", OS/2 "cmd.exe", and Unix "sh") may produce errors. This will be changed in a future release.
• The URL used in Command_html, etc. (resolved from "%u", "%p", or "%r") may contain invalid command-line characters including "|", "&", and "?". The command is initiated in the default shell (csh, command.com, cmd.exe, etc.) and the shell characters may be interpreted. The user must ensure that the defined command string does not process these characters. This may be achieved by using double or single quotes, depending on the operating system.
• Changed in version 3.000 Templeton can handle "unlimited" length URLs. (The actual limit is only restricted by the amount of system memory.) But, URL's which map to filenames that are longer than names allowed by the operating system may not be saved. For example, OS/2 allows a filename to be no longer than 256 characters; URL's longer than 256 characters may not be saved properly. If the URL is longer than the maximum length filename for the operating system, then "FATFlag TRUE" should be used.
  In older versions of Templeton, the maximum URL length is 512 characters. Longer URLs are truncated.
  This may be changed in future releases.
• Templeton (and other web robots) assume that the previous password for a realm remains valid. This assumption is incorrect when different documents (usually on different servers) coincidently use the same realm string. Templeton only supports one (1) username:password per realm. This may be changed in later releases.
• Templeton performs single-pass processing on all documents. If a link is thought to exist (and placed in the processing queue) but later is unretrievable, all previously retrieved documents will contain links to a nonexisting file.
• Changed in version 1.960. Templeton only permits a 6K meta header from the server. Longer meta headers may be interpreted as part of the document. This is currently implemented so servers which do not return meta information will not have the entire (huge) file parsed. (The 6K assumes a maximum of 4K in cookie information and 2K is other meta information.) This will be modified in a future release.
• Changed in version 1.960. Because Templeton (and all other web robots) is unable to know the default file of a directory (it differs on each server), Templeton is unable to distinguish between '/' and '/default_index_name'. If the server's default index name matches Templeton's 'Index' name (defined in the configuration file) then the default directory file may not be retrieved. This may be modified in a future release.
  Rational:

  Suppose there are 2 files: Welcome.html and Index.html. Welcome.html is the server's default filename, while Index.html is configured as Templeton's default filename. A single HTML file may refer to "/", "/Welcome.html", and "/Index.html". Assuming that the default directory file is the same as Templeton's would incorrectly store a copy of Welcome.html in the local Index.html file. Retrieving the file twice would resolve this issue.

  Unfortunately, this also means a default server file called "Index.html" would be retrieved twice (once for "/" and once for "/Index.html"). Worse: if the default index is never specified by an HTML document, but a different remote file called "Index.html" is found, then the default directory data will be lost.

  Since this is a Lose-Lose situation, Templeton will retrieve the file "Index.html" only once. Note, prior to version 1.951, Templeton would retrieve the file twice. After much debate, this option was changed.

  How's that for a complicated situation?

• Regular HTML documents which contain a '?' may be incorrectly identified as a probable CGI application. (They shouldn't have '?' in the URI, but bad HTML code never ceases to amaze me.) A URI specified as "document.html" and "document.html?garbage" will generate two separate files containing the same data. (Most servers are smart enough to ignore the '?garbage' after a valid HTML document.)
• Version 1.990. When reusing Update-Files, the file must be listed on the command line before the "Update-File" command. The configuration option immediately rewrites the old file, destroying any previously existing data. The URLs to update may be overwritten before Templeton can load them. The current solution is to list the file containing the URLs on the command line before any other configuration files.
  Corrected in version 2.0. You can now reuse an update file without overwriting previous data.
• Version 1.990. When using an update-file, Templeton initially displays all the URLs that are being loaded from the update file. Pressing 'Q' or 'X' to leave the program will not necessarily exit.
• When running the unregistered Templeton with the setting "Interactive FALSE", the registration machine code is not displayed. To view the machine code, you must start Templeton with "Interactive TRUE" (the default setting), or use the "reginfo" application.
• URIs can specify a username:password. This is not used by HTTP, but can be used by other protocols (e.g. FTP). Templeton currently "throws-out" the username:password from HTTP URIs.
• HTML comments (<!-- ... -->) cannot be in files longer than LONG_MAX (a big number). This is system dependent, but for PC, SunOS, and HPUX operating systems, this is around 2 Gigabytes. Comments found beyond LONG_MAX bytes in a file may not be properly ignored.
• Documents that have been "moved" (HTTP return codes 301 and 302) are processed as normal URLs, and the contents of the moved document are saved to the retrieved file. This is different from how Netscape and Internet Explorer handle moved documents. Netscape and IE immediately follow the moved link, never displaying the contents of the moved page unless the destination link cannot be found. What this means:
  1. If the "moved" document contains links that would normally not be seen, then Templeton will still traverse these links.
  2. If the "moved" document does not contain a hyperlink to the new location (most do, but...) then there will be no link to the document's new location.
  3. If the "moved" document has no content, then Templeton will generate a zero-length retrieved file in place of the moved document.
  In all of these cases, Templeton will still process the "moved location", permitting it is not a restricted URL.