You can achieve quite respectable backup and restore speeds with Ftape: a Colorado DJ-20 and an Adaptec 1542CF controller, has been measured at 4.25Mbyte/min sustained data transfer rate (no compression) across a 70Mbyte tar archive, while comparing the archive on the tape with data on an IDE disk. The speed of Ftape is mostly dependent on the data transfer rate of your FDC: The AHA1542CF has a ``post-1991 82077'' FDC, and it will push 1Mbit/sec at the tape drive. If you have an FDC which can only deliver 500Kbit/sec data rates, you will see half the transfer rate (well, roughly).
There has been a few reports of "shoeshining". This is when the tape just seems to run back and forth endlessly. This has been seen on a Jumbo 250 (74407.3051@compuserve.com) and on an Iomega 250 Ditto Insider (tom@opus.cais.com). In the latter case it has been narrowed own to using an ELF Linux and running off a SCSI hard disk (connected to an Adaptec 1542cf). Please contact me if you have an update to this problem.
<from the Ftape-Howto>
Probably not. If you are backing up a large number of < 2K files, you're just going to have to live with it. In this event, the repositions are caused by file system access overhead. If you are backing up a normal system's files, this may be caused by slop or media stretching in the tape cartridge. By simply retensioning the tape, you should see this go away. Try
ftmt -f /dev/zqft0 reten
<answer from Tim Jones>
If you use afio as your backup tool you can set it to write a very large number of buffers in one hit by using the -c flag. Make it large enough so that you supply enough data for most of a single end-to-end pass over the tape. For my system, the following streams quite nicely - stopping relatively few times per tape pass on an unloaded system:
find /usr/local -xdev -print | afio -o -v -f -b 10240 -c 800 /dev/qft0
Presumably other backup utilities could be modified to use a similar technique.
<answer by Michael Hamilton>
GNU tar doesn't use buffering in this way. The commercial backup program "bru" is able to multi-buffer using shared memory; this works only when writing compressed archive with bru (regardless whether you use Ftape's builtin compression)
Another way to overcome the problem might be to use more dma buffers in the Ftape kernel driver like:
mt -f /dev/qft0 setdrvbuffer $((6*32786))
<answer by Claus Heine>
Not if you are using the latest version of the Ftape drivers from the Ftape Home Page.
To format a QIC-80, TR-1, TR-3, QICWide 3010 or 3020 tape, get the
latest version of ftape and the latest version of the
ftape-tools package (from the same location) and read the
documentation of the ftformat utility which is included in the
ftape-tools package.
<answers from Tim Jones and Claus Heine>
It isn't possible to format Ditto 2GB tapes with Ditto 2GB
tape drive, and it isn't possible at all to re-format Ditto 2GB
tapes in a way that they still can be used by a Ditto 2GB tape
drive.
This is a hardware limitation of the Ditto 2GB tape drive. It
can't be helped at the software level, i.e. it isn't ftape's
fault.
No, the Ditto Max can't format tapes.
This is a hardware limitation of the Ditto Max (Pro) tape drive. It
can't be helped at the software level, i.e. it isn't ftape's
fault.
If you look at the difference, you will notice that Ftape always detects 2784 sectors more than DOS.
The number that Ftape reports is correct (of course :-). Each
correctly formatted QIC-3020 tape has 2784 sectors at fixed positions
that are marked in the bad sector map. To quote from the specs:
Tracks 5,7,9,11,13,15,17,19,21,23,25 and 27 within 4 segments of either EOT or BOT are prone to increased error rates due to hole imprints. Therefore, these regions shall be mapped as bad at format time and entered in the bad sector map by indicating that all sectors within the identified segments are bad.
This gives 12 tracks * 2 * 4 segments * 29 sectors == 2784 sectors.
So Ftape choose to report the real number of sectors that cannot be used on the tape, while DOS gives a more optimistic number giving a better indication of tape quality. (Ftape's behavior might change in the future to detect correct formatting and display the separate numbers. It has rather low priority though).
QIC-3010 are alike QIC-3020 tapes regarding this.
<from the Ftape-Howto>
Yes. The driver merely updates an internal counter when those commands are issues. The tape should move to the proper location on the next read or write access to the tape drive.
<from the Ftape-Howto>
zftape requires the data to be written in multiples of a fixed minimal
block size. This is a very usual behavior for a tape device. There are three
ways to get rid of those errors:
mt -f /dev/qft0 setblk 5120
mt -f /dev/qft0 setblk 0
afio -b 10k ...
You may want to read the section "Tape blocks" of the manual (use its "Concept index" to directly jump to that section)
When using GNU tar's builtin compression with GNU tar versions prior to
tar-1.12 one needs to run tar with the --block-compress switch to
re-block the output to the tape. Otherwise tar will compress the data
it reads, and write it in arbitrary portions to the tape.
Example :
tar -czvf /dev/qft0 --block-compress /etc
WARNING: One shouldn't use tar's builtin compression with large backups as it makes the entire data stream one huge compressed block. If such archives are corrupted right at the beginning it will be very difficult to recover.
<answer by Claus Heine>
When you get next messages, this could be interesting for you !
The explanations:
"FDC" menas "Floppy Disk Controller". The problem is that your floppy disk controller must be able to support something that is called "perpendicular mode" to be able to read and write QIC-3020/QIC-3010 cartridges (i.e. TR-3 cartridges). To my knowledge all FDCs that are capable of at least 1Mbit/sec data transfer rate also support "perpendicular mode" ("perpendicular" refers to the direction of magnetization of the ferro-magnetic particles on the tape).
This means that you need to purchase another FDC. Either look around some computer stores and ask for an IO controller cards that is able to support 2.88 Mb floppies (which imlies 1Mbit data transfer rate and perpendicular mode).
Or get one of the so called "high speed" controllers that even support 2Mbit/sec data transfer rate. Those controllers are based on an Intel 82078 FDC. Iomega sells such a card under the name "Ditto Dash". I think Exabyte sells their 2Mbit controllers separately, too, whereas Seagate ships its TR-3 drives (i.e. the TST-3200) together with such a controller.
<answer from Claus Heine>
I assume that the following is the problem: The Ftape module is loaded OK into the kernel:
/usr/src/ftape-3.03b-970603# lsmod
Module Pages Used by
ftape 22 0
$ ftmt -f /dev/qft0 status
ftmt: /dev/qft0: No such device
Solution You need to load the zftape.o module as well. With Ftape-3.* the ftape.o module doesn't implement the VFS interface. This is done by zftape.o.
<answer from Claus Heine>
The "device busy" messages can only occur while the Ftape devices are still held open by some program. As soon as the close() system call has completed the busy flag is cleared. May be "bru" or some other program has still forked off a child that dies delayed?
Yes, this will reproduce the problem, it seems:
tar -cvvzf /dev/nqft0 --block-compress ; mt rewind
However, this is not a bug of Ftape. It seems that the parent tar process exits before its child has closed the tape device. I know, however, from hacking the tar code ages ago, that tar properly waits for its parent to die.
However, the busy message simply means that the "busy" variable is still held at 1 (zftape/zftape-init.c). And this simply means that there still is a process hanging around that holds the tape device open.
I think I have it (only for the case of tar 'cause I have the source code.
If on uses tar with compression, then it forks a child which will become the compressor bei execing "gzip" or whatever. Before the call to execlp() the child will fork off a grand child of its parent tar. That grandchild will do the actual tape I/O.
tar - fork() - write to child tar
|
child tar - fork() - gzip (will pipe to grand child tar)
|
grand child tar - open archive.
Now, parent tar only waits for its child to die. gzip surely doesn't wait for the grand child as the gzip is a result of an execlp().
What I don't know is whether the grand child should be implicitly waited for by the parent tar, or if the wait() function also waits for grand childs.
But this seems to be the problem: the parent tar already has exited while its grandchild still is busy closing the archive. One hardly will notice this problem if the close() happens fast (i.e. regular files, block devices, also other tape devices?), but it isn't a bug in Ftape, but either in the backup programs or in the kernel or maybe libc exit code.
Don't know if the considerations above also apply to bru. If there is no grandchild and the parent process properly waits for its childs then there shouldn't be a problem.
<answer from Claus Heine>
These are really tar questions: Please read the man page and
the info page. If you have not got it either, try
tar --help 2>&1 | less
If your version of tar is v1.11.1 or earlier, consider
upgrading to v1.11.8 - This version can call GNU zip directly
(i.e.: it supports the -z option) and has an elaborate help
included. Also, it compiles right out of the box on Linux.
<from the Ftape-Howto>
When using compression, and in all general, it can be a benefit to
specify to tar, that it should block the output into chunks.
Since Ftape cuts things into 29Kbyte blocks, saying `-b58'
should be optimum.
"Why 29Kbyte?", I hear you cry. Well, the QIC-80 standard specifies that all data should be protected by an Error Correcting Code (ECC) code. The code specified in the QIC-80 standard is known as a Reed-Solomon (R-S) code. The R-S code takes 29 data bytes and generates 3 parity bytes. To increase the performance of the ECC code, the parity bytes are generated across 29 1Kbyte sectors. Thus, Ftape takes 29Kbytes of data, adds 3Kbytes of ECC parity, and writes 32Kbytes to the tape at a time. For this reason, Ftape will always read and write 32K byte blocks to be able to detect (and correct) data errors.
If you are curious, and wish to know more, look in the ecc.c and
ecc.h files, for an explanation of the code and a reference to a
textbook on Reed-Solomon codes.
<from the Ftape-Howto>
All of these tools have been developed by the GNU project, and the
source (and man page) can be fetched from just-about any ftp site in
the world (including ftp.funet.fi, tsx-11.mit.edu, and
sunsite.unc.edu). In any case they can be fetched from the
official GNU home site: prep.ai.mit.edu [18.71.0.38]:/pub/gnu. The latest versions (as of September 12
1996) are:
cpio: 2.4.2 (cpio-2.4.2.tar.gz)
dd: 3.13 (fileutils-3.13.tar.gz)
mt: 2.4.2 (cpio-2.4.2.tar.gz)
tar: 1.11.8 (tar-1.11.8.tar.gz)
gzip: 1.2.4 (gzip-1.2.4.tar.gz)
They all compile out of the box on Linux v1.0.4 / libc
v4.5.19 / gcc v2.5.8.
<from the Ftape-Howto>
It is not bad as such to compress data twice (which would be the case when using tapers compression together with zftape's compression) but it doesn't make any sense. You won't gain much further compression, but only waste CPU cycles.
Tapers compression should be quite safe, as taper compresses single files; in contrast to tar -czf ... which makes the entire data stream a large compressed block of data, which is really a bad thing with serious backups as a single bad byte at the beginning of the archive can make the entire archive unusable, well, it will be at least quite difficult to recover.
<Answer from Claus Heine>
gzip -9 is better (i.e. one gains higher compression). zftape's compression is comparable with the Un*x compress program, but should be faster, and is faster than gzip.
<Answer from Claus Heine>
Use the zftape interface, but don't load the zft-compressor module.
The device then becomes /dev/qft0.
<answer from Tim Jones>
You get this complaint if you haven't erased your freshly formatted tape. This is because Ftape expect a "magic header" on the tape, to be able that it is allowed to interpret the header segment in its own way (eg: file marks). To remove the problem, say
mt -f /dev/nftape erase
<from the Ftape-Howto>
No. The DOS software conforms to the QIC-80 specs about the layout of the DOS filesystem, and it should(?) be a small problem to write a program that can read/write the DOS format. In fact, I'd bet that creating a nice user interface would be a bigger problem.
<From the Ftape-Howto>
(EOM is "End Of recorded Media", the position right after all data already recorded to the tape)
One cannot use tape "files" like files on an ordinary file system.
In principle, a tape doesn't allow anything but appending new data at EOM. However, if one positiones just in the middle of the already recorded data AND starts writing, then the driver first deletes all following files (thus moving the EOM to the actual position) and then starts writing.
Thus, the new EOM after finishing the write process, is then after the newly recorded data.
One of the consequences of the above is, of course, that writing to the tape in the middle of the already recorded area, is destructive in the sense, that it not only overwrites the "file" the tape is positioned at, but also deletes all following files.
<from the Ftape-Howto> <Answer from Claus Heine>
It probably didn't work before because you didn't use a
mt -f /dev/rft0 erase
But, hey, what does mt fsf? Tape drives don't store files in the sense that you can use
cp somefile /dev/my_what_ever_tapeor be able to mount the tape drive like you could mount a harddisk. You can't do nothing with a tape drive but write data to it in a sequential manner.
As this is quite inconvenient, somebody invented something which is known under the name file mark or eof mark (eof == End Of File). Those marks don't separate files that have been backed up to the tape device, but only separate blocks of data (whatever data that might be).
Normally, the kernel tape device drivers take care of writing file marks when the tape device is closed, i.e.
tar -cf /dev/nqft0 /bin
tar -cf /dev/nqft0 /etc
mt -f /dev/nqft0 rewind
Now, the sense of those file marks is, that it is possible to skip between different archives on the tape more quickly than would be possible with reading the data back.
The commands to do that are:
fast skip to the next file mark towards EOT (End Of Tape)
fast skip to the next file marks towards BOT (Begin Of Tape)
mt -f /dev/nqft0 rewind
mt -f /dev/nqft0 fsf
tar -xvf /dev/nqft0
<Answer from Claus Heine>
When Ftape was young there were two versions of the floppy tape driver, one of them was called zftape because of its built-in user-transparent on-the-fly compression. Whether such a thing is a feature or a bug ('cause this needn't be done in kernel space) is another question. However, the ioctl interface and file mark handling provided by zftape was much better and had less bugs. And zftape allows to use floppy tape cartridges with different OS. Well, you can't exchange data, but zftape won't overwrite volumes created by your Windoze program, and vice versa.
Nowadays, Ftape is name of the entire floppy tape driver package AND ftape.o is the file-name of the kernel module that implements the low-level hardware support. zftape has ceased to exist as a separate package, but the new Ftape versions (since ftape-3.00) contain a zftape.o module that needs to be loaded on top of ftape.o (i.e. you need to load BOTH modules to be able to access your floppy tape drive) and implements the file system interface and the advanced (?) features of the previous verions zftape.
<Answer from Claus Heine>
Well, the rewinding tape devices rewind the tape to BOT (Begin Of Tape) when the device is closed, i.e.
tar -cvf /dev/qft0 /bin
tar -cvf /dev/nqft0 /bin
Rewinding devices should be used when performing a single backup, non-rewinding devices can be useful when doing multiple backups as one doesn't need to space to EOM (End Of recorded Media) before appending another archive.
Non-rewinding devices MUST be used when sending any of the tape motion command to the tape drive, such as
mt -f /dev/nqft0 fsf
<Answer from Claus Heine>
Well, it depends. If the tape is still positioned inside the volume just written, "mt bsf 1" (or equivalently "mt bsf") will backspace just to the beginning of that volume (this is how "tar --verify" works). If the tape is already positioned AFTER the filemark that marks the end of the last written volume, then you need to issue "mt bsf 2"
The logic behind this is as follows:
"MTBSF count" backspaces over count file marks, stops, and then
positions on the EOT side of the last skipped file mark. This means,
an "mt bsf 2" will position right at the beginning of the previous
volume.
<answer form Claus Heine>
You are right: auto-rewind means, the tape is rewound when the tape device is closed, non-rewinding means, the tape isn't automatically rewound when the tape device is closed (but you can, of course, use the tape motion commands BSF/FSF etc. to position the tape head at every position you like).
<answer form Claus Heine>
A record is the minimal amount of bytes that will be accepted by the tape in one read/write operation (except in "variable block size mode" where it just should be the amount of data actually written in a single write operation??).
For zftape every read and write access has to be a multiple of a fixed
block size (fixed, but tunable with MTSETBLK). This block size is a
"tape record" (as mentioned in the GNU mt man page and defaults to
10kb for zftape.
A "file" (in the sense of the mt man page) is a, well, misleading terminus. What is meant is an area of the tape between two file marks. This is not a file like a file on the file system, in the sense that it could have a name, file access modes, could be moved or copied with cp, mv, rm etc.
Instead, It simply is the area of the tape that was recorded in one
backup session, its end is marked by a tape file mark, and its
beginning is delimited by either BOT or the file mark of the previous
tape "file". That tape "files" are the things that can be skipped with
the MTBSF/FSF commands.
<answer form Claus Heine>
We try to answer the followong questions :
If you want to "erase" an entire cartridge, then simply do:
mt -f /dev/qft0 erase
This will erase the volume table (i.e. the "file marks").
Pre-ftape-3.x releases of zftape and ftape used to allow overwriting of already existing volumes on a cartridge. I have removed this feature as it was reported that it already has caused data-loss with some backup programs.
If you indeed need to remove some volumes on the tape then you should use the
vtblc
program that comes with the ftape-tools package which can be
down-loaded from the same locations as the ftape kernel driver
package. Please refer to the documentation which is contained in the
ftape-tools package for more information.
If you simply want to reuse old tapes, then it suffices to do
mt rewind
If the tape is at BOT (Begin Of Tape) then every write access to the tape will silently erase all file marks and overwrite the data already existing on the tape.
<answer by Claus Heine>
Here is as little perl/bash script that lists the contents of a cartridge using the zftape specific "volinfo" ioctl. Hope this shows how to handle this kind of stuff.
What it basically does is the following:
claus@thales:~$ mt volinfo
file number = 1
block size = 10240
physical space used = 522.0 kilobytes
real size of volume = 520.0 kilobytes
The Perl Script
#!/usr/bin/perl
#
# Copyright (C) 1997 Claus-Justus Heine
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; see the file COPYING. If not, write to
# the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
#
# This script implements a simple contents listing for the zftape
# package using the MTIOCVOLINFO ioctl.
#
$version = <<EOT;
listtape-1.0 -- a perl script to list the contents of a floppy tape cartridge
under Linux using the zftape driver
RCS \$Revision: 1.2 $
RCS \$Date: 1998/08/30 13:44:03 $
EOT
$tapedev = "/dev/tape";
$usage = <<EOT;
Usage: listtape [options ...]
Mandatory or optional arguments to long options are mandatory or optional
for short options too.
-f, --file=FILE Tape device to use. Default is "/dev/tape".
-h, --help Print this help.
-? Same as '-h'.
--usage Same as '-h'.
-V, --version Print version information.
Author: Claus-Justus Heine <claus\@momo.math.rwth-aachen.de>
EOT
while ($ARGV[0] =~ /^-/) {
$_ = shift;
if (/--file/) {$_ = shift; $tapedev = $_; next;}
if (/-f/) {$_ = shift; $tapedev = $_; next;}
if (/--help/) { print $usage; exit 0; }
if (/-h/) { print $usage; exit 0; }
if (/--usage/) { print $usage; exit 0; }
if (/-\?/) { print $usage; exit 0; }
if (/--version/) { print $version; exit 0; }
if (/-V/) { print $version; exit 0; }
die $usage;
}
&open_tape($tapedev, "status");
while(<FTMT>)
{
$online = 1 if (/.*online.*/);
}
if (! $online) { die "No cartridge present.\n"; }
&mtop($tapedev, "rewind");
printf "%11s%12s%20s%20s\n",
"file number", "block size", "volume size", "tape space";
while (1)
{
&open_tape($tapedev, "volinfo");
while (<FTMT>) {
if (/^file number\s*=\s*([0-9]*)$/) { $filenumber = $1; }
if (/^block size\s*=\s*([0-9]*)$/) { $blocksize = $1; }
if (/^physical space used\s*=\s*([[0-9]*.*)/) { $rawsize = $1; }
if (/^real size of volume\s*=\s*([[0-9]*.*)/) { $size = $1; }
}
close(FTMT);
if (&mtop($tapedev, "fsf 1") != 0) {
&mtop($tapedev,"rewind");
print "\nRemaining space: $rawsize\n";
print "Tape block size: $blocksize\n";
exit 0;
}
printf "%6d %5d %20s%20s\n",
$filenumber, $blocksize, $size, $rawsize;
}
sub mtop
{
local ($tape, $operation) = @_;
local ($exitval);
system "ftmt -f $tape $operation > /dev/null 2>&1";
}
sub open_tape
{
local ($tape, $operation) = @_;
local ($command);
$command = "ftmt -f " . $tape . " " . $operation . " |";
open(FTMT, $command) || die "Couldn't open $command -- $!\n";
}
The Bash Script
#! /bin/bash
#
# Copyright (C) 1997 Claus-Justus Heine
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; see the file COPYING. If not, write to
# the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
#
# This script implements a simple contents listing for the zftape
# package using the MTIOCVOLINFO ioctl.
#
#
# insert better option parsing here
#
TAPEDEV=${1-/dev/tape}
if ! echo $TAPEDEV | grep "/dev/n"
then
TAPEDEV=/dev/n$(basename $TAPEDEV)
fi
if ! [ -c $TAPEDEV ]
then
echo $TAPEDEV is not a character device! 1>&2
exit 1
fi
if ! mt -f $TAPEDEV rewind
then
echo Could not rewind $TAPEDEV - no cartridge present? 1>&2
exit 1
fi
echo -e "\nContents of $TAPEDEV:\n"
printf "%11s%12s%20s%20s\n" "file number" "block size" "volume size" "tape space"
trap "rm -f /tmp/$0.$$" exit
while true
do
if ! foo=$(mt -f $TAPEDEV volinfo |cut -f 2 -d =)
then
echo $TAPEDEV doesn\'t seem to be a floppy tape device 1>&2
exit 1
fi
#
# "echo foo | read foo" will not work as the "read foo" is executed in
# another shell.
#
echo $foo > /tmp/$0.$$
read file blksz used usedunit size sizeunit < /tmp/$0.$$
if ! mt -f $TAPEDEV fsf 1 > /dev/null 2>&1
then
echo -e "\nRemaining space: $used $usedunit"
echo -e "Tape block size: $blksz"
if ! mt -f $TAPEDEV rewind
then
echo Rewind of $TAPEDEV failed 1>&2
exit 1
fi
exit 0
fi
printf "%6d %5d %20s%20s\n"\
$file $blksz "$size $sizeunit" "$used $usedunit"
done
<answer from Claus Heine>