World of A1200/World_Of_A1200.iso/programs/compress/misc/xfh/advanced_usage.doc

Text File | 1995-02-27 | 24KB | 494 lines

                  Xpk File System Handler v1.32

                      By Kristian Nielsen


This package may be distributed freely for non-commercial purposes along
with Xpk or on its own, provided that all files and subdirectories are
left intact. However, all rights remain with the author. And as usual,
the author will in no way be held responsible for any damage, directly,
implicit or otherwise, resulting from the use of this program. After
all, you get this program for free; use it or not as you like, and at
your own risk.

The file 'XFH.doc' (which should be distributed alone with this file)
gives information on how to get started quickly while this text gives
a more technical discussion of the XFH file system handler and
describe some of the more obscure options that are not accessible
through the gui. Some knowledge of the more technical aspects of the
Amiga is assumed. Also, this document is pretty unstructured, being
just a random collection of notes gathered together over the last year
of XFH development. Do read XFH.doc first to get an overview of what
XFH is. A lot of users probably don't wan't to read this file at all;
XFH.doc will tell them all they need to know.


                              --- O ---


What's possible.

   XFH, when used with Xpk, makes it possible to store data in compressed
format without this being visible to the user or to application programs -
XFH will make the compressed data appear like ordinary files. This is by
no means a new idea. In MS-DOS world, a (commercial) program called
'Stacker' has been available for some time, which makes it possible to
compress a whole partition on a harddisk. On the amiga, people have long
used powerpacker along with programs like PPMore to store data files in
packed formats (but relying on the application to recognise that the file
is in compressed format), and programs like PPPatch have been used to
change dos.library to recognise the powerpacker format automatically.
However, none of these approaches are perfect in all situations, and so
there is room for another alternative - the XFH.

   XFH works in conjuction with the Xpk approach to data compression. This
in itself gives a number of advantages - a flexible interface to the
compressing algorithms, lots of different packers available with the
possibility to add new ones etc. But compared to approaches like
'Stacker', there is an additional advantage: access to the compressed
data is not limited to XFH - most of the time, the XFH way will be the
most convenient way to access data, but if needed, the complete range of
Xpk applications is available to the user. The file orientated nature of
Xpk also means that XFH - like PPPatch etc., but unlike Stacker - will
co-exists nicely with any other Amiga filing system without the need to
set up any new partitions or prepare the disk with a special program. In
fact, if you happened to have a CD-Rom (read only) stuffed with
powerpacked files (or any other format that the current version of Xpk
supports), you could dump an XFH unit on top of it and largely forget
about the disk being compressed from then on.

   So, what is possible is to mount a XFH unit on top of each of one or
more standard AmigaDOS directories. It is now possible for programs to
access the directories as usual, data being decompressed and optionally
compressed as needed in a completely transparent way - programs will
never know the difference between compressed and uncompressed files -
while still having access to the compressed data using conventional
compress/uncompress programs. So the dream is that of doubling the
capacity of your hard- or floppydisks for zero cost.

   As an example of the possibilities of the XFH, I have been using it
for holding various doc files, metafont sources and little used emacs
scripts and shell commands on HD, saving in the order of perhaps 60%
file space with no noticeable degrading of overall system performance.


                              --- O ---


What is the catch?

   Of course, as everyone knows, nothing comes for free. Obviously,
there is a speed penalty to compressing or uncompressing a file.  This
speed penalty will be highly dependent on the actual compression
algorithm and data media used; for example an algebraic compression
scheme used with a superfast HD will probably be rather slow, while a
fast algorithm may actually result in a speed-up of floppy access since
the disk access time saved by the smaller file size more that accounts
for the time taken decompressing. And of course with the coming of ever
more powerful CPU's the speed will be less of a problem. Some things
will always be slow, though. Especially directory listning is a
problem, since every single file has to be opened in order to check
whether the file is compressed or not. In fact, I'll admit that running
XFH from floppy on an unaccelerated amiga will sometimes seem a bit
slow. However, XFH was not written to be as fast and small as possible,
but rather to be flexible and expandable. And I'm sure a lot of users
will find it very useful even on 'small' amigas. On an A3000, XFH runs
like a dream, of course. And there is still the possibility for speed
improvements in later versions.

   Aside from the problem of speed, V1.32 of the XFH comes with a few
other limitations that I'm hoping to remove in later versions. Most
important is the lack of most of the new 2.0 packets, a lack that will
become more severe as more 2.0-only programs start to depend on these
packets. Another problem is that XFH in the present version is somewhat
memory-hungry, in that it will keep any compressed file completely
unpacked in memory as long as that file is open. This is mostly due to
limitations in the current Xpk interface, and I'm hoping to remove this
in a later release. However, currently this means that it is impossible
to open a compressed file if it is larger than available memory.
[One way to solve this would be to have XFH use one of the 'virtual
memory' programs that are starting to appear. In fact, I have had one
or two reports about this actually working. If someone wants to
discuss this I'd very much wellcome it.]

   A number of other problems to look out for are detailed in a later
section in this doc file. Most of these are things that could be fixed in
later versions.



                              --- O ---


Instructions for use.

   XFH is implemented as an AmigaDOS device handler. The 'L:'
directory contains examples of other such device handlers, and this is
usually also the best place to put XFH (the file 'XFH-Handler', to be
precise). Like other handlers, XFH must be mounted before it can be
used. This can be done by creating an entry for it in the
'DEVS:Mountlist' file (just the mountlist for short) and issuing the
command 'Mount <device>', where <device> is the name given to the
device in the mountlist. The supplied file 'Devs/Mountlist.custom'
contains example entries for the XFH; it might be convenient to append
this file to the end of 'DEVS:Mountlist' (the entries assume an assign
'xfhdir:' to the XFH distribution directory). The most important part
of each mountlist entry is the Device name (XH1: etc) and the 'Startup'
entry, since these are used to configure the various options of XFH..
Further information on the mechanism of mountlists and handlers in
general can be found in various places; I will not attempt any lenghty
explanation here.

   Unlike most other 'normal' device handlers, XFH needs additional
information to function correctly. Most importantly it needs to be told
what directory it should use as its root directory (that is, where it
should look for compressed files). In the simplest case, this can be done
by having an assigning (or device volume name)  'nn:' to this
directory and mounting an XFH partition with the name 'Xnn:' (any
letter will do, not just 'X'). However, XFH also provides 'options',
some of which cannot be controlled by the gui but only through the
Mountlist or by using Arexx.

Options can be specified in the mountlist or in an option file, and
can be changed dynamically using AREXX. An option is specified as a
string following the same conventions as the TOOLTYPES of the
Workbench.

If the number of options that should be set are limited, it can be
convenient to put them directly in the mountlist entry. This is done
by using the 'Startup' keyword. Use a line of the form

    Startup = "!<opt1>=<value1>!<opt2>=<value2>...!<optN>=<valueN>"

(that is, a list of option assignments preceded by '!' (no spaces).
The '"'s are optional and the '='s can be replaced with '&'s (this is
nessesary to be compatible with some mount commands). See the entry
for 'XH0:' in 'Devs/Mountlist.custom' for an example.

For a larger number of options, option files should be used. XFH has
the concept of primary and secondary option files.  The primary option
file is used by putting a line of the form

    Startup = xfhdir:optionfiles/.xfhrc_1

into the mountlist entry (distinguised from the other use of this
keyword by the fact that no '!' appears). The filename can be anything
you like, of course. The secondary option file is named '.xfhrc' and
is placed in the root directory of the XFH unit (uncompressed!).  The
settings in the secondary option file overrides any settings in the
primary option file, but there are some restrictions on the options
that can be used in the secondary option file, see below.  Both files
are optional, the handler will pick default values for any options
that are not set by the user.

In the option files, options are given each on a line of its own. Some
examples are given in the directory 'Optionfiles'. Be careful when
using the secondary option file; this is also used by XFH itself to
store information that must be preserved across reboots (notably the
volume name for Relabel).

When the handler is mounted, most options can be changed by sending an
approproate AREXX command to the handler. The name of the port is the
same as the name of the device, though this can be changed (see
below). Note that XFH will refuce to mount if it cannot open the AREXX
port because of a naming conflict. To set an option, send a command

    SETOPTION <optstring>

to the port, where <optstring> should be in the same format as that
used in option files. The directory 'Arexx/' contains example AREXX
scripts that may be usefull. For example the command

    rx SetAutocompress XDH1 ON

will enable automatic compression on XDH1:.


   Boolean options can be specified with "NO" / "OFF" or. "YES" / "ON".
String options are specified by simply putting the string after the
'OPTION=' bit, no quotes are needed.

The available options are detailed below:

   ROOTDIR
   VOLUMENAME
   AUTOCOMPRESS
   PACKMODE
   STEPDOWN
   PASSWORD
   XPKPRIORITY
   TRUNCATEONPACK
   FAILONEXNEXT
   KILLSTARTUP
   COMPRESSREADWRITE
   ALLOWAPPEND
   PORTNAME

Option ROOTDIR:

   This option is used to set the name of the directory that XFH: is
to reside in. For example 'ROOTDIR=Work:xfh' would cause XFH: to use
that directory as root. Note that this option can only be used in the
primary option file or directly in the mountlist. When used elsewhere
it will simply be ignored. The default is to use the name of the XFH
device itself with the first character removed (so XDH1: becomes
DH1:). [Note that this is different from XFH v1.12 and earlier.]

Option VOLUMENAME:

   This option is used to set the name of the XFH: volume. This is the
name that will be used in absolute path specifications, as well as the
one returned by Info(). Ie. if you go 'VOLUMENAME=Manuals', access can
be by 'Manuals:' as well as by 'XHn:' (or whatever). If this option is
not specified, the default is to use the name of the directory in the
underlying file system, or (if this is itself the root of a volume) to
use the name with 'XFH_' prepended. Again, this is mainly for
compatibility with early versions of XFH:. An alternative to using
this option is to use the normal Relabel (from shell or WB); this will
automatically create an entry for this option in the secondary option
file (replacing any existing entry).

Option AUTOCOMPRESS:

   This options tells whether XFH: should attempt to compress the
files written through it. When this option is set, everytime a file
written to the XFH: is closed, an attempt will be made to compress the
file to a temporary file using Xpk. If this is succesfull, the
temporary file will be renamed to the original name and the
uncompressed file will be deleted. If the compression fails for any
reason, the uncompressed file will simply remain intact. This also
means that if anything should go wrong during compression (like a disk
full error), it is unlikely that any data will be lost since at least
one of the two files should be intact (though possibly with a wierd
name). This option is OFF by default.

Option PACKMODE:

   This option selects the mode that Xpk should use when compressing
files. It is specified in the usual way when using Xpk. For example,
to use 12-bit BLZW compression, 'PACKMODE=BLZW.12' would be used. The
default is to use the NUKE compression ('PACKMODE=NUKE'). Of course,
to use a specific compression method, the nessesary sublibrary must be
available in LIBS:.

Option STEPDOWN:

   This option controls the Xpk flag 'XPK_StepDown' during packing. If
set, it means that Xpk is allowed to reduce packing efficiency if
nessesary to save memory. Refer to the Xpk documentation for details.
Default is OFF.

Option PASSWORD:

   This option is used to set the password that XFH: should pass on to
Xpk when compressing or uncompressing. It should be noted that there
is no attempt to keep this password safe from 'memory peekers'. Of
course, storing the password in the option file isn't a good idea. A
better idea is to use AREXX; the AREXX script 'SetPassword.rexx' in
the directory 'Arexx/' may be helpful here. Note that if AUTOCOMPRESS
is requested, files may still be saved unencrypted in low-memory or
low-diskspace situations, and (depending on the underlying file system
/ disk device) part of the unencrypted data may still be physically
stored on the disk after the deletion of the unencrypted file.

   If an attempt to open a file is made when XFH has been given the
wrong password, the open will fail with error code 212 (Object wrong
type).

Option XPKPRIORITY:

   This option is used to set the task priority that should be used
when doing Xpk operations (compress/uncompress etc). Setting this to
zero or less will prevent XFH from stealing all CPU-time from tasks
running at a 'normal' priority. Note that it is possible to set the
priority of the handler itself in the mountlist. If this option is not
used, the handler will keep whatever priority it is running at when
calling the Xpk library.

Option TRUNCATEONPACK:

   This option is somewhat technical in nature and can be safely
ignored.  It is only used in case of an error occuring during the
compression of a file. In this case, the compressed file has to be
deleted, and if this option is set, the handler will try to call
SetFileSize() first to truncate the file to 0 bytes (perhaps saving
the flushing of a few buffers).  However, due to sparse documentation
I'm uncertain whether this feature is stable, and hence it is OFF by
default. Again, unless you are curious and don't mind risking
crashes/data losses, forget about this option.

Option FAILONEXNEXT:

This option was included to solve a problem with some (or more likely
most) programs that perform directory scanning.  The problem appears
when listing a directory containing a file that is opened with an
exclusive lock (for example, MODE_NEWFILE). In this case, XFH cannot
determine the correct size of the file and thus fails with an
appropriate error code. However, many programs just assume that the
end of the directory has been reached. Setting this option to OFF will
prevent XFH from reporting failure, returning a fileinfoblock with the
wrong file size instead. Since this could result in data loss because
programs will se the wrong file sizes, this option is ON by default.

Option KILLSTARTUP:

This option has been included to fix some problems with programs like
'Info' that examine data about file system handlers. It seems that
these programs assume that the 'Startup' entry in the device node of
any handler that supports volumes will be a FileSystemStartupMsg.
However, in the case of XFH, it is a string. This behaviour seems to
me to be completely unresonable, and thus a bug in these programs. The
only way I could find to fix this was to have XFH manually erase the
Startup field in the device node. This is not likely to be an
officially supported way of poking the device node. Nevertheless, I
had to agree that the problem is intolerable, and thus this option is
on by default. However, it can be turned off if you have a mount
command that does not like handlers that modify their device node.

Option COMPRESSREADWRITE:

This option controls whether new files opened with MODE_READWRITE will
be compressed upon Close(). It is ON by default.

Option ALLOWAPPEND:

This option must be set to allow the writing to existing files (using
MODE_READWRITE or MODE_OLDFILE). It is my personal opinion that
compressed files shouldn't really be updated in place, since it
requires first uncompressing the file on disk, then doing the Write(),
then compressing the file again. However, some programs need this
ability. This option is ON by default.

Option PORTNAME:

This option can be used to set the name of the AREXX port. This option
can only be set in the option file or directly in the mountlist (no,
you cannot set it using AREXX...). The default name is the same as the
name of the XFH device.


In case of an error during the scanning of the option files the handler
will fail its initialisation and hence refuse to load with error code 114
(Bad Template). Sorry, but there are currently no real error messages
implented (this will hopefully be fixed in a later version). If the handler
refuses to work for no apparent reason, be sure to tripple-check your
option files for errors.




                              --- O ---


Limitations and known bugs.

   The error detection code in the initialisation part of the handler is
somewhat flaky - I've tried to make it resonably safe, but documentation
on the right way to start a handler is hard to find. What it means is
that it is a good idea to make sure that the handler is placed in L: and
that the nessesary Xpk libraries are placed in libs: before starting the
handler (remember, XFH won't be able to tell you the reason if it was
unable to initialise for some reason). Another subtle problem is that due
to a quirk of the device list locking, it is vital (using XFH v1.32) that
the handler for the directory that XFH is to sit in is already loaded.
Usually this will not be a problem; however, if you are using 3rd party
filesystem handlers that are mounted after boot-up, you can avoid
problems by accessing them before mounting XFH (for example by creating
an assign (not late-binding) to them).

   It should be noted that the way the XFH makes the same files
available by two different routes is not without its problems. One
problem is connected to the volume name - XFH tries to be smart about
it, but it will sometimes create a duplicate volume name which is a
bad idea. To solve this problem the VOLUMENAME option should be used
in an option file, or the XFH volume should be changed with Relabel.
Another problem when mounting XFH in the root dir appears when using
the 'Leave Out' feature of the 2.0 Workbench. Here, the '.backdrop' is
duplicated in both volumes, making the left-out icons appear twice.
I'm working on a decent solution to this problem. Meanwhile, I would
recommend that XFH is mounted only on top of subdirectories.

To provide maximum transparency for application programs, both of the
options AUTOCOMPRESS and ALLOWAPPEND should be set. This will make XFH
compress files that are written to it 'on the fly', even for programs
that update existing files (ie MODE_READWRITE, 'append mode', shell
'>>' redirection etc). However, a bit of care is advised when using
the ALLOWAPPEND option. For example, it is obviously not a good idea
to have two programs writing to the same compressed file at the same
time. Even worse, if one program is accessing a file on the underlying
file system at the same time that another program is writing to the
same file using XFH, XFH is unable to guard completely against data
loss. Bevare of this situation. Another thing is that writing to a
compressed file opened with MODE_OLDFILE (like Lha does) relies on the
ACTION_CHANGE_MODE packet (to partially solve the problems just
mentioned), and thus won't work using KS1.3 file systems or some file
systems written before the apperance of KS2.0.

   Needless to say (but I'll say it anyway), you should not assign libs:
to a XFH unit unless you are absolutely sure of what you are doing. A
nice trap is to have the XFH call (and wait for) Xpk, which will then wait
for XFH to load a particular library for it.

   It should be noted that any given unit of the XFH binds to a directory,
not to a DOS device. This means that, currently, it is not possible to
have a XFH unit working like DFx: - any access will refer to the disk that
was in the drive when the handler was started, not to the disk currently
in the drive.

   The figures reported by the shell 'Info' command are somewhat strange.
The problem is that it isn't really possible to give sensible figures for
'NumBlocks' and 'NumBlocksUsed'. Currently, their values are the same as
those for the underlying file system.

   XFH has problems with exclusive locks (for example trying to obtain an
exclusive lock on "/" or "foo//" will always fail).

   Some people have experienced problems when using XFH with the
arp.library. This is because a bug/feature in the apr.library function
CompareLock() (it declares two locks equal only if the lock->fl_Key
fields are equal, which is illegal according to the 2.0 DOS manual).
Under 2.0, the program in the directory 'patcharp' can be used to patch
arp to use the correct 2.0 SameLock() call. Also, I know of a program
that remaps the arp calls to the corresponding 2.0 dos.library calls
(though I haven't tried it), it might help too.



                              --- O ---


Theory of operation.

   XFH works by installing itself in the system as a file system handler
like DF0: or RAM:. However, unlike most file system handlers, which sit on
top of a device (or rest in themselves like RAM:), XFH sits on top of an
underlying file system handler (abbrevated to UFS) containing a mixture
of normal and compressed files. After initialising itself, XFH sits in a
loop receiving packets from AmigaDOS and other applications. Each packet
is examined and the appropriate action taken. For example, an open request
will cause XFH to open the given file, check whether it is compressed, and
if so unpack it to memory for later read requests.

   XFH is currently single-threaded, unlike the Commodore file systems
(this means that it is not possible for the XFH to service other requests
while waiting for the UFS). I'm hoping to fix this in some later version.



                              --- O ---


Acknowledgments.

   The author wishes to thank all the people that have participated in
the development of Xpk without which the XFH would not have been the
same. I am especially grateful to Urban D. Müller for helping me start
the whole concept of the XFH back in the summer of 1991 - without his
help the XFH is not likely to have been realised. Thanks also to the many
beta testers who helped me iron out as many bugs as possible before
release; your help work has been very valuable to me. And thanks must
go, of course, to the guys at Commodore for bringing to us the wonderful
Amiga.