Acorn User 1/AU_CD1.iso/internet/starterpak/!TCPIP/TCPIP_Doc/Notes/FTPClient

Text File | 1995-02-05 | 16KB | 475 lines

New facilities in FTP Client
============================

The FTP client in !TCPIP has undergone the first stage of improvements
aimed at sorted out UNIX -> RISC OS path convertion problems, and making
it a bit more automated.

These include:

  The addition of a highly configurable path name processor.
  The addition of a batch ftp facilty.
  The addition of an "mget" command.
  The addition of a "reget" command.
  The addition of a "quote" command.
  The addition of {ftp_data} and {ftp_list} variables.


Pathname Processor
------------------

The path handling is the main area of improvement, and now allows UNIX
style paths to be parsed for file type extraction from one or more file
name extensions and is able to work out appropriate processing based
upon the extensions present or elements in the basic file name.

On the basis of the file name and/or extensions, the ftp client can now
work out what file type to give the file, which directory to store it in
and whether to keep unconverted extensions, whether to keep directory
elements from the original remote path etc.

When there is more than one extension, it is able to sensible use one
extension to determine the file's type and another extension, or the
file's name, to determine in which directory the file should be placed.

These conversions are done according to a set of rules given in a new
!TCPIP resource file. (!TCPIPUser.PathEnt).

When downloading groups of files, for example, the entired contents of a
directory and subdirectories containing C source files, there are often
files which might end up being placed in a different directory to the
rest as a consequence of name/extension base placing. As this can be
undesirable, it is posible to switch off the fully automatic processing,
and force preset processing according to the type of download your are
doing. This is most useful when using the new batch facility.

Finally, just in case there are any problems that I have missed, the old
original path name handling is still available in a rather more useable
and improved form.

Whatever path processing options are in use, directories are
automatically created. The lack of this was one of the most annoying
problems with previous versions of the FTP client. Also filenames are no
longer actively truncated, allowing advantage to be taken of such
utilities as !LongFiles. With appropriate Risc OS configuration, overly
long filenames will still be truncated, but by Risc OS rather than the
FTP client.


PathEnt file
------------

The !TCPIPUser.PathEnt consist of a series of either options lines, or
name matching lines. When trying to match a name or extensions, three
types of matching are carried out. These are:

  1.  Match the file name to determine where to put the file.
  2.  Match an extension to determine where to put the file.
  3.  Match an extension to determine a file type.

The first match in each of the above categories is the one used. How
this effects the layout of the PathEnt file will become clear.

Each options line starts with a "$" character and is followed by
required options on the same line, each separated by a space.

There are three types of options line as follows:

$root <pathname>
"$root <FTP$Dir>"

This must be present, once only. The given pathname set the root
directory in which files are to be placed. Normally, this is set to
"<FTP$Dir>" which in turn is set to "!TCPIPUser.Downloads".

$default [<option>...]
Example: "$default nocase nfsext truncdir keepext in Misc type &fff"

This line must be present, once only. It sets the default path
processing options this are modified by options given later, or applied
directly to any paths that are not otherwise matched.

$ [<option>...]
Example: "$ rev1ext in Source"

This line contains options that are to be applied to subsequent
extension/filename decoding lines. Any options given are applied in
addition to those given in the default options line.

Lines starting with a "#" character, or blank lines are ignored.

Lines starting with a "." describe extensions matching. Other lines
describe filename matching.

Extension and file name matching lines are as follows:

<name>/.<ext>[|<name>/.<ext>...] <file type> [<options>]
Examples: ".c|.cpp|.c++|.h|.[1-8]|.s|.y|.l &fff"
          "rfc* &fff in RFCs"

"/" in the above is used to represent "or" in the syntax rather than the
usual "|" as "|" is literally used to separate extensions in a list.
There must be no spaces in the extensions part.

The use of wildcards is allowed in the name and extension as follows:

  "*"     matches any string of 0 or more characters.
  "?"     matches any single character.
  "[...]" matches a range of characters, for eg, [0-9] matches 0, 1, 2,
          3 ... 9.

The file type is a valid RISC OS file type, for example, "Text" "fff",
"&fff" etc.

Addition extension specific options may also be given by including the
option after the file type.


Options:

  nocase    Treat upper and lower case alike in matching

  keepext    Keep extensions as file/ext1/ext2 etc

  noext        Remove all extensions

  revext    Reverse extensions so that dir/file.a.b becomes
        dir.b.a.file

  rev1ext    Reverse 1st extension only, so that app/file.c.gz
        becomes app.c.file/gz

  nopath    Strip path prefix for remote name so.

  in <dir>      Places file in directory <dir>. Unless <dir> is a full
                path name, <dir> is assumed to be a sub-directory of the
                specified root directory.

  type <type>   Set the default type to be applied in the absence of any
                other information.

  nfsext        Enables the handling of Acorn NFS style ,xxx hex extensions.


Any option prefixed with "!" is switched off rather than on.

When a type is matched, the corresponding extension is allways removed.


An Example File


# Set the location of the downloads directory.

$root <FTP$Dir>


# Set default options as follows:
# case-insensitive matching, handling of NFS extensions, keep
# extensions as file/ext/ext unless told not to,
# place unmatched file in "<FTP$Dir>.Misc"
# set their type to Text (&fff)

$default nocase nfsext keepext in Misc type &fff


# These option are applied to the following type matches until
# another options lines is encountered. These options are applied in
# addition to those given on the defaults line above.
#
# Ensure directory elements in original remote path name are stripped.
# Ensure extensions are stripped, leaving just the filename.
# Place matched files in directory "<FTP$Dir>.Docs"
$ nopath noext in Docs
.txt         &fff
.doc         &fff
.asc         &fff
.ps          &ff5


# A new set of options to be applied. This completely replaces those
# given in the previous options line, and is applied to the following
# match lines.
#
# Basically as above, but put files in "<FTP$Dir>.Archives"

$ nopath noext in Archives
.arc         &ddc
.archive     &ddc
.spark       &ddc
.spk         &ddc
.tar.gz      &ddc
.tar.z       &ddc
.tar         &ddc
.gz          &ddc
.z           &ddc
.zip         &ddc


# A new set of options for Image files
# The files end of in "<FTP$Dir>.Archives"

$ nopath noext in Images
.jpeg|.jpg   &c85
.gif         &695
.tiff|.tif   &ff0


# A more complicate extensions set for dealing with source file downloads.
#
# this matches .c, .c++, .cpp, .h, .1, .2, .3, .4, .5, .6, .7, .8,
# .s, .y, and .l.
#
# The first extension and file name are swapped round so that you
# get c.*, h.* files.
#
# Also, as these options *replace* previous ones, directory elements in
# the original path name are retained. So applic/source.c becomes
# applic.c.source.

$ rev1ext in Source
.c|.cpp|.c++|.h|.[1-8]|.s|.y|.l &fff


# Any files with "rfc" at the start of the name are placed in
# "<FTP$Dir>.RFCs". Any files with "draft-" at the start of the name are
# placed in "<FTP$Dir>.IENs". Also path elements and extension are
# stripped.
#
# Note, that the following two match lines don't have a "." prefix, so
# they match the name rather than the extension. Also note the use of
# options in the match lines.

$ nopath noext
rfc*         &fff in RFCs
draft-*      &fff in IENs


Hopefully this commented example will now enable you to create you own
additional entries to suite your exact needs.


Path Command
------------

Syntax: path off | auto | type | <option> [<option>...]

"path" on its own tells you what state the path processor is in.
Separate state is maintained for each active FTP client session.

"path off"  switches the path processor off, result in the use of the old
path handling.

"path auto" switches the path processor on and causes it to use
information given in the "PathEnt" file as described above.

"path type" switches the type decoding only on and causes it to use
information given in the "PathEnt" file as described above.

"path ros"  causes the path processor to precisely mirror the name
conversions applied by the unix mode in !TCPIP's FTP server. This mode
is most useful when connected to a !TCPIP v2.02 FTP server and want to
use "mget" to transfer complete directory hierarchies intact.

"path <option> [<option>...]" presets a set of options independantly of
the PathEnt file to be applied to subsequence "get" operations.

The option given are exactly as per a PathEnt file options line, but you
dont need to prefix with the "$" and in fact you must not.

Example as used when fetching a the entire contents of a C sourrce
directory:

path nocase rev1ext in <FTP$Dir>.Source type Text

(See "batch" command for a real in-context example)

Notes:

The "in <dir>" option has a full name specified as the given directory
in this case is not treated as a subdirectory of a specified root
directory.

The path processor, when active, will allways be able to decode RISC OS
filetype name extensions as appended by !TCPIP's FTP server in unix
mode.

However, in order to ensure paths are kept exactly as on the server, use
the "path ros" command.



Batch FTP
---------

* Note to demon users: this is not to be confused with the Batch FTP
service that demon offer. This is something rather different, though the
batch ftp facility as provided by this client can be *very* usefully
used for fetching files from your private batch directory on demon.

Batch FTP is a very simple facility to allow an unlimited length list of
any FTP session commands (exept of course the batch command) to be place
in a file which is read with each command being executed upon completion
of the previous one.

In !TCPIP v2.0f onwards, dialogue scripts appended to an ftp or terminal
command could be used to partially achieve the effect of batch ftp, but
creating such scripts was rather fiddly for such a common operation.

Now that batch ftp is available properly, you can usefully use dialogues
appende to ftp commands to start off a batch operation. (See end for
details.)

Synyax: source <filename>

Either a full pathname may be given, (for eg, adfs::4.$...) or a
filename may be given. In the case of a filename, various places are
searched when looking for the specified file as follows:

!TCPIPUser.Scripts
!TCPIPUser
!TCPIPUser.^
$ on current drive
@ (the cirrect directory)

If it cant be found, then obviously an error is given.

A batch is simply a file containing a sequence of valid in-session FTP
commands, on per line, exept of course the "batch" command itself. Upon
completion of each command, the next command is read and executed until
the end of the file.

You will notice that the commands appear in the FTP session window in
the bold atrribute colour as they are being executed. Also the FTP
prompt changes to "batch> " while a batch file is being processed.

A batch file can be aborted in the same manner as a get command, by
entering "abort". This will terminate the tranfer in progress and close
the batch file.

An example:

# Set required path options
path nocase rev1ext in <FTP$Dir>.Source type Text

# Set required remote directory
cd /pub/NetBSD/NetBSD-current/src/gnu

# get each file, letting the path processor take care of the
# details.
get tar/Makefile
get tar/Makefile.gnu
get tar/README
get tar/buffer.c
get tar/create.c
get tar/diffarch.c
get tar/extract.c
get tar/fnmatch.c
get tar/fnmatch.h
get tar/getdate.y
get tar/getoldopt.c
get tar/getopt.c
get tar/getopt.h


To get the best out of batch ftp, it is advisable to get directory
listings of directories of interest from FTP sites of interest using the
"dir . <local file name>" command, and then to edit this listing using
the macro facilities in !Zap to create a list batch file containing the
required path, cd and get commands off-line. Next time you connect, FTP
to the site(s) and enter the batch command.

Also is possible to get the FTP session and batch request fully
automated by using a ftp connect dialogue similar to the following:

ftp ftp.{domain} \ @DemonFTP YE100 YV24 \
"name: " "anonymous\r\n" "assword: " "{ftpuser}@{host}\r\n"
"ftp> " "bin\r\n" "ftp> " "hash 1\r\n"
"ftp> " "source batchfile\r\n" "ftp> " "quit\r\n"

(Line is artificially broken for legibility)

and including this command at the end of the auto exec for a completly
un-attended ftp session.

Naturally, a number of similar commands can be included for multiple
similtaneous automatated ftp seesions.

Note that the "quit" comman is placed in the dialoque rather than the
batch file so that if the batch file fails for any reason, the quit
command will still get executed, allowing the tcp hangup timer to
continue to trigger a shutdown and exit.


MGET command
------------

This, like "get" is used for retreive files from an FTP server. Unlike
"get", "mget" can be given a short list of wildcarded names that are
each expanded into a file list, with each file in the resulting list
being fetched via "get".

Syxtax: mget <listspec> [<listspec>...]

Its operation is equivelent to entering ls <listspec> for each
<listspec>, and then doing an "get <file>" for each filename returned,
and in fact this is exactly what it does on your behalf, as you will see
from the information display during an mget operation.

For example, to fetch the entire contents of two directories on a server
"dira" and "dirb", you could use "mget dira dirb". To fetch the contents
of current directory, and immediate subdirectories, use "mget *".

Another way of using "mget" is to type ls at the "ftp> " prompt, and
then select some files from the resulting listing (upto around 50 or so)
and then apply the preset macro menu "Get file(s)".

Before using "mget", it is worth ensuring that suitable path processing
options are setup with the "path" command.


QUOTE command
-------------

This allows a command to be sent directly to an FTP server, bypassing
the normal client pre-processing of commands.

Syntax: quote <command> [<local file for listing>]

This is most useful for things like server secific variations of "ls"
that are handled like the unix "ls" command, for example:

quote "ls -C" listfile

sends "ls -C" to the server, and directs output to the local file
"listfile".

Note: as the command to be sent to the server will often need to contain
spaces, your will often have to enclose the command in quotes as above.

<listfile> should only be specified in situtations where output is
expected via a data connection.


REGET command
-------------

This command allows a get operation to be continued after a previous get
operation that was terminated early for some reason, for eg because your
internet connection was broken.

Syntax: reget <remote file> [<local file>]


WINOPEN and WINCLOSE
--------------------

These enable the FTP session windows to be opened and closed.

Syntax: winopen
        winclose

The main use of this is in FTP command files that may be triggered by an
automated FTP session initiated from another wimp application, in
particular, a FTP fetcher for WWW would use this.