"tcpdl" history

Fixed a problem where the leading enclosing quotes around URLs in downloaded HTML files were omitted.

Added a NOWAIT option to allow tcpdl to exit without waiting for return to be pressed. This option can be used from the command line, or can be specified in the config file.

Added a size gadget to the status window which allows the width of the window to be adjusted (the height is fixed by the number of tasks specified).

Added a PUBSCREEN option to allow tcpdl to open its status window on a specified public screen. If no public screen is specified, then the Workbench screen will be used by default as in previous versions. This option can be used from the command line, or can be specified in the config file.

Added a FONT option to allow the font to be used by the tcpdl status window to be specified. By default the Xen font will be used, or the default font if Xen is not found. This option can be used from the command line, or can be specified in the config file.

Added a FONTSIZE option to allow the size of the font to be used by the tcpdl status window to be specified. By default a size of 9 is used. This option can be used from the command line, or can be specified in the config file.

Added a PRIORITY option to allow the maximum priority of the tcpdl tasks to be specified. By default a priority of 2 is used. This option can be used from the command line, or can be specified in the config file.

Fixed a bug that caused non-text files to be downloaded even when the previously downloaded version was still current. This wasted bandwidth unnecessarily. Thanks to Jon Wareham for reporting this problem.

Reworked the code that deals with the HTTP transfer. This should make the code more tolerant of strange replies from the server, and has paved the way for further improvements in the future.

Changed behaviour when URLs containing characters that are invalid in AmigaDOS file names are encountered. Any invalid characters are now encoded as %xx (where xx is the hex value of that character). This is transparent to Web browsers and servers, as this form of encoding is standard. The encoded filename will be used in HTML references, as well as in the file name itself. Previously, a CRC encoding was added to the file name, making it hard to determine what the original file name was. This ensured it was unique, but introduced its own problems.

Changed behaviour when filenames are longer than allowed by the filesystem being used. Filenames are now simply truncated at the maximum supported length (30 characters for FFS). The full URL without truncation will appear in HTML references. Previously, the filename was truncated and a CRC encoding was added. The new approach could lead to filename clashes (where more than one file in a directory has the same name) but this is expected to cause fewer problems than the CRC encoding. (Note: The encoding of characters (described above) that are invalid in filenames may cause filenames to exceed the length limit and be truncated.)

Added the NOSAVE command line option. When specified this prevents any downloaded files from being saved. This feature has been requested by users that use tcpdl to prime local proxy servers (e.g. "httpproxy") with files. If tcpdl also saved the files, there would be two copies stored locally.

Added the DEBUG command line option. When specified this stores a copy of the HTTP request in each "DATA" file, and leaves data files from failed transfers in the "TEMP" directory. This is only intended for use in investigating problems, and isn't intended for general use.

Added new status messages

• *BREAK* indicates that the task has noticed the ctrl-C
• Not Found indicates that the URL was not found on the server
• SRVR ERROR indicates that the server reported an error

Previously, some temporary file were written to "t:", which could cause problems in low memory situations. All files are now written to "tcpdldir:" so the user has control of the location of all files generated (even temporarily) by tcpdl.

Corrected display of current and total file sizes. Previously these were displayed only intermittently.

Corrected minor bug in the status display which caused columns not to be correctly aligned

Reduced memory usage slightly

Corrected a bug that could leave some files locked if an error occured during a download

Added URL command line option. This allows a URL to be downloaded to be specified on the command line rather than in the "urllist" file. See the Command Line Options section for details.

Added CONFIG command line option. This can be used to specify the configuration file to be used. If it is not specified, the default "tcpdldir:tcpdl.config" is used.

Added TASKS command line option to allow the number of tasks to be used in downloading URLs to be specified. Valid range is 1 to 15.

Added TASKS config option. If present, this value acts as the default number of tasks to be used in downloading URLs. Valid range is 1 to 15. This value can be overridden by use of the command line TASKS option.

Added USERAGENT config option. If present, this specifies the user agent name to be sent to the HTTP server. By default the user agent name used will be of the form "tcpdl/version", where version is the current version number. This can be used to spoof as some other HTTP client, for (broken) sites which only accept requests from certain browsers.

Reduced the default number of processes to 12, as 15 was too much for many machines.

Changed command line option handling to use AmigaDOS templates.

tcpdl sent HTTP commands that some hosts didn't like - it should now manage to talk to hosts using any version of HTTP.

The UPDATE flag wasn't correctly propogated to all the URLs referenced by the files marked with UPDATE in the urllist. This was a fundamental design flaw, and so the UPDATE flag has been removed and replaced by an UPDATE command line option which applies to all URLs in urllist.

The IMG flag didn't quite do what was advertised, so it has been amended slightly:

• "TEXT" means all URLs that are referenced by a "href" attribute
• "IMG" means all URLs that are referenced by an attribute other than "href"
• "ALL" means all URLs that are referenced

Added a configuration option USER, which allows the mail address to be used in the HTTP commands to be specified. By default tcpdl uses the username and host name used by the TCP/IP stack. If the USER option is specified with no mail address, then no "From:" header will be sent in HTTP commands.

tcpdl used to send the user's realname to the http host - there is no need to do this, so now it doesn't.

Fixed the DISK-ERR message, which looked untidy.

Made it possible to exit tcpdl when run from Workbench.

Fixed "Enforcer" hits when urllist contains a blank line.

Added support for proxy servers.

Added configuration options for connection, and HTTP transfer timeouts.

Added configuration option for the number of retries for failed transfers.

tcpdl now uses memory pools for its memory allocation. This improves the allocation times slightly, but has a dramatic effect on the deallocation time. It also reduces the risk of memory leakage when tcpdl is interrupted.

The "User Agent" name has been changed to conform to RFC2068.

The tcpdl.config file is now closed as soon as it has been read.

The environment variable USERNAME is now used if USER is not set.

Changed the ERROR status indicator to be more specific about the cause of the error.

Added tcpdl.config to allow the specification of file types that are not to be downloaded.

Fixed problem where URLs in the urllist file which were not followed by at least one space or tab character were ignored.

"tcpdlpp" history

Changed the handling of URLs with a leading "/". All URLs are now converted to relative form, so all local links should work when browsing off-line.

Minor optimizations.

Improved the handling of ".." in URLs.

Added optional translation of characters in URLs, by means of a new configuration file "tcpdldir:tcpdlpp.config".