ftp.wwiv.com/ftp.wwiv.com.zip/ftp.wwiv.com/pub/MISC/Y2KTOOL5.ZIP/pktdate.txt

Text File | 2000-01-09 | 24KB | 550 lines

PKTDATE Rev. 1.4 written by Tobias Ernst @ 2:2476/418
-----------------------------------------------------

Contents
1. About this Document
2. What you do need Pktdate for
3. Using Pktdate
4. Installation Hints
5. Fine-Tuning
6. Reason Codes
7. Caveats
8. Final Words
9. Credits

1. About this Document
----------------------

This document describes the advanced features of the Pktdate program.  Pktdate
is, to put it short, a tool to fix Fidonet PKT files that are broken due to
Y2K bugs in the creating software.  If you want to learn more about the year
2000 problem in Fidonet in general and how you can test your system, please
read year2000.txt first.


2. What you do need Pktdate for
-------------------------------

Pktdate is a program that can analyse PKT files for structural errors that
result from year 2000 related bugs in the creating software.  In standard mode
(like it is being used in year2000.txt), pktdate simply displays all date
fields in the pkt file and tells you what kinds of errors have been found.
However, pktdate is more powerful:  It can read pkt files even if they are
severely broken, correct the problems, and write a new pkt file that contains
correct date fields.

Pktdate currently applies two strategies in fixing pkt files.  First, it
checks the pkt file for structural errors that may have occurred, and tries to
resolve them.  A PKT file can contain structural errors that make it unreadable
for standard fidonet software (e.g. if the FTSC date field spills and the
subject has zero length, most software will not be able to interpret the
packet any more).  Pktdate will read those broken files, extract all messages,
and create a new, correct, pkt, that can be read by any Fidonet software.

Second, you can specify a certain tolerance level for the date fields.  If
a date seems to be too far in the future, or too far in the past, Pktdate
will assume that the date is incorrect due to a year 2000 related bug.  It
then will try to calculate the correct date (by undoing the bug that the
packet creator did), or if this is not possible, replace the incorrect date
field with the current date so that the message is ordered at least roughly
at the right position.  This will also prevent messages with incorrect date
fields from being purged too soon, or being bounced, etc. pp.

Pktdate is intended for use at hub level in Fidonet to assure smooth operation
of the net even if individual nodes or points do not have upgraded their
software to be Y2K capable.  I have written the tool in the belief that until
1/1/2000, still a major share of nodes and points will work with defective
software, partly because they haven't cared for the problem, and partly
because even if they performed Y2K test, they might not have realised all of
the tricky problems that can be buried in the binary PKT file format (see
year200.txt for more information).

If at least every Hub in Fidonet will install the Pktdate utility, there is a
chance that the net keeps on working even if a major share of the nodes do not
use fully compliant software.  As such, Pktdate will alleviate the effects of
the Y2K bug and probably help Fidonet to survive in the 21st century.

Of course, "normal" nodes are also encouraged to use this tool if they have
points that are using software that is not year 2000 ready.

As a last note, you see that pktdate is intended to fix broken packets that
*others* create.  It assumes that your *own* system is working correctly
(see year2000.txt on how to test this).  If you have problems with your
*own* system, you might want to take a look at other tools from other
authors:

- YEARCONV von Frederik Retsema.  It bascially allows you to set your
  system date to a year before 2000, and is then intended to run on every
  packet that you receive and most importantly, on every packet that you
  create, to fix the year field (either decrease it before import or
  increase it after export).

- Y2KDATE.ZIP by David Rance.  It does a job similar to that of Pktdate,
  but works on *.MSG files.  You could let this tool process your primary
  netmail folder, bevor packing the netmail into your mail queue or Binkley
  outbound, and then continue to use software with Y2K flaws on your own
  system.  However, please note that such software might have other
  relevant bugs than just a spilling date field in a *.MSG file.
  Y2KDATE.ZIP is available via fidonet f'req at 2:242/110, or by writing an
  e-mail to [e-mail].


3. Using Pktdate
----------------

If you simply invoke pktdate with a file name (note that wildcards are not
supported by pktdate, but you can specify multiple filenames as arguments),
like in

  pktdate 12345678.PKT

it will log all date fields found in the pkt file to standard output.  This is
good for interactive debugging.  If you want Pktdate to fix broken pkts,
you must instruct Pktdate to correct problems and re-write the fixed pkt file
to disk by issuing the -c option:

  pktdate -c 12345678.PKT

In an automated environment, e.g. if calling pktdate from your tosser batch
file, you will want to have a log file.  You can do this by specifying the
filename of the logfile as argument to the -L command line parameter.  You can
also change the log level with the -l parameter.  If you specify a log level
of 2, you will only see warnings and errors.  I.E. you will have a chance to
inform your downlinks about the problems, but you will not be bothered by
informational messages:

  pktdate -L e:\fido\log\y2k.log -l 2 -c 12345678.PKT

There is yet another problem:  In the logfile y2k.log, you will only see
that for example mail number 47 in the paket named 12345678.PKT had a flawy
date field (and what exactly in the date field was wrong).  You don't see
exactly who the originator of the mail was, nor any other information.
Therefore, starting with revision 1.3 of pkdate, there is a new command
line option "-k" ("keep").  If you specify it, pktdate will, (exactly) in
the case that it has to write corrections to a PKT file, make a backup copy
of the original (flawy) PKT file with file extension ".Y2K" (or .Y21, .Y22
and so on if .Y2K already exists), before writing anything.  So you can, at
a later time, when you see in the log file that a flawy PKT file passed
your system, use a PKT viwer to inspect the ".Y2K" backup file and see who
the originator of the particular mail in question was.  (Please note that
it was not necessarily the originator of the mail who cause the problem, it
could also have happened anywhere on the path from him to you):

  pktdate -L e:\fido\log\y2k.log -l 2 -c -k 12345678.PKT

Pkdate has yet some more command line options.  A complete list of
supported command line options, that also allows you to change tolerance
boundaries etc., can be obtained by invoking pktdate without arguments.

When calling pktdate from your tosser batch, you of course do not know the
name of the pkt file when writing the batch file, but you will want to have
pktdate process all pkt files in a certain directory.  The problem is that
Pktdate does not support wildcards, because supporting wild card expansion
would have required to use non ANSI-C features, so that pktdate would not be as
portable as it is now.  You can help yourself as follows:

- On OS/2, create the file FIXALL.CMD that looks like follows:

  FOR %%I IN (%1) DO PKTDATE -k -c -L y2k.log -l 2 %%I

  You can then say "CALL FIXALL.CMD E:\SOMEPATH\*.PKT" in your tosser batch
  or CMD file, and the command "pktdate -k -c -L y2k.log -l 2" will be issued
  on each filename found in E:\SOMEPATH.

- On DOS or Windows, create the file FIXALL.BAT with the following contents:

  FOR %%I IN (%1) DO PKTDATE -k -c -L y2k.log -l 2 %%I

  You can then say "CALL FIXALL.BAT E:\SOMEPATH\*.PKT" in your tosser batch
  file, and the command "pktdate -k -c -L y2k.log -l 2" will be issued on each
  file found in E:\SOMEPATH.

- On UNIX, you can use the wildcard expansion feature of the shell and call
  pktdate like

  pktdate -k -c -L /var/log/fidoy2k.log -l 2 /var/spool/inbound/*.pkt

- On Amiga, create the file Fixall that looks as follows:

.key PAT/A
.bra {
.ket }

; $VER: PktPat 1.0 (15.3.99) by MkM
; derived from SPat 40.1 (9.2.93)

FailAt 21

List >T:q{$$} {PAT} LFORMAT "pktdate -k -c -L y2k.log -l 2 *"%s%s*"" SORT NAME

IF NOT FAIL
  Execute T:q{$$}
Else
  Echo "{PAT} not found"
EndIF

Delete >NIL: T:q{$$} QUIET
Quit

  You can then say "(Execute) Fixall INBOUND:#?.pkt" in your tosser batch
  file, and the command "pktdate -k -c -L y2k.log -l 2" will be issued on each
  file matching INBOUND:#?.pkt.

  Please also have a look at the README.AMI file in the Amiga subdirectory,
  it contains some additional information by the person who compiled the
  Amiga binaries, and a somewhat different variant of a DOS script that
  calls pktdate.


4. Installation Hints
---------------------

If you install Pktdate and want it to automatically process and possibly fix
ALL pkt files that you receive from your links (which is highly recommended),
you must ensure that pktdate is called AFTER the tosser has run the
decompression programs for the arcmail bundles, but BEFORE the tosser starts
tossing.  Pktdate must check all *.PKT files in the protected, unprotected,
local and temporary inbound (the temporary inbound is where the decompressed
PKT files from arcmail bundles go) directories, Therefore, this chapter gives
installation instructions on how to best integrate Pktdate with various tosser
software.  The following tosser software has been tested by me:

- Fastecho
- Squish
- hpt

If you are using another tosser, you can still use Pktdate.  Just take the
information from chapter 3 and figure out how to best integrate Pktdate with
your system.  You might also take the methods describe in this chapter (4) as
an inspiration on how to integrate Pktdate with your tosser.

4.1 Fastecho
------------

- Copy PKTDATE.EXE to a convenient location, preferably in the search PATH.
- Create a file named PKTCHECK.CMD (OS/2) or PKTCHECK.BAT (DOS) using your
  favourite text editor that has the following contents:

@ECHO OFF
for %%i in (c:\inbound\*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i
for %%i in (c:\secure\*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i
for %%i in (c:\tempin\*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i

  Of course you have to adapt the paths to point to your temporary inbound
  (the one configured in FESETUP under System/Pathnames/Temp.  Inbound), your
  normal inbound and your secure inbound, and of course your log file
  directory.  You can also add lines to process local and insecure inbound
  directories, if you wish).

- Start FESETUP.EXE or FESETUP2.EXE.  In System/External Programs/After
  Unpack, enter the file name name and path of PKTCHECK.BAT or
  PKTCHECK.CMD.

- Poll your uplinks and watch your tosser task running.  You will see pktdate
  being executed multiple times, once for each PKT file.  If everything works,
  you might want to change the "-l3" parameter to "-l2" in order to only see
  log messages for pkt files that really contain problematic messages.

4.2 Squish
----------

- Step 1: Place pktdate.exe somewhere in your search PATH.

- Step 2: Processing uncompressed pkt files

  You probably use a batch file (.BAT or .CMD) to call Squish.  Edit this
  batch file and find the place where squish is invoked with the IN parameter.
  Right before this call, place the command line to invoke pktdate on your
  inbound, and possibly local and/or unprotected inbound directories:

  for %%i in (c:\inbound\*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i
  for %%i in (c:\secure\*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i

  Of course, you have to adapt the paths to unsecure and secure inbound, and
  to the log file to match your system configuration.

  If on OS/2, you use a .CMD REXX script instead of a plain CMD batch file,
  you have to enclose each line in quotation marks (").

- Step 3: Processing compressed pkt files

  Unlike Fastecho, Squish does not support calling of an external program
  after the decompression of arcmail bundles, but before the start of the
  tossing process.  Therefore, we have to do a trick:  We write a little
  script that first calls the unpacker and then calls pktdate, and make Squish
  believe that this script is the actual unpacker.  Start your favourite
  editor and create a file named PKTCHKSQ.CMD (OS/2) or PKTCHKSQ.BAT (DOS)
  which should be located in a directory which is in your search PATH.  Its
  contents should look like this:

@ECHO OFF

SET COMMANDLINE=
:LOOP
IF ~%1~==~~ GOTO ENDLOOP
SET COMMANDLINE=%COMMANDLINE% %1
SHIFT
GOTO LOOP
:ENDLOOP
%COMMANDLINE%
IF ERRORLEVEL 1 GOTO FEHLER
SET COMMANDLINE=

FOR %%i IN (*.PKT) DO PKTDATE -k -c -l3 -LC:\LOG\PKTDATE.LOG %%i
GOTO ENDE

:FEHLER
echo Unpack error!!!
exit 1

:ENDE

  Again, you have to adapt the path to the log file to suite your needs.

  The first part of this batch file parses the command line and executes
  everything that it has been passed as arguments verbatim.  After that, it
  calls pktdate to process all PKT files in the current directory.  (Yes, I
  know that in REXX the script would look much nicer, but not everybody
  has REXX installed, so I restricted myself to the plain batch language).

  [ For the REXX fanatics among us, I also present two REXX solutions for
    OS/2 that do exactly the same thing as the batch file shown above.
    My proposal:

    /**/
    parse arg commandline
    commandline
    "for %%i in (*.pkt) do pktdate -k -c -l3 -Lc:\log\pktdate.log %%i"

    From Oliver 'Attila' Grimm:  the pure REXX solutions, longer than my
    proposal, but according to him "to rexx or not to rexx" (very free
    translation, TE ...)

    /**/
    call RxFuncAdd 'SysLoadFuncs', 'RexxUtil', 'SysLoadFuncs'
    call SysLoadFuncs
    parse arg commandline
    ''commandline
    call SysFileTree '*.PKT', 'files', 'FO'
    do i = 1 to files.0
      'PKTDATE -k -c -l3 -LC:\LOG\PKTDATE.LOG 'files.i
    end
    return
  ]

  In order to activate the batch file, you must edit the COMPRESS.CFG file
  which Squish uses to learn which unpacker to use.  Locate all unpacker
  definitions and prefix all calls to the unpacker with PKTCHKSQ.  For
  example, if an entry in compress.cfg looks like this

DOS     Extract         pkunzip -n %a %f
OS2     Extract         unzip -C -s -j -o %a %f

  you should modify it to read

DOS     Extract         pktchksq pkunzip -n %a %f
OS2     Extract         pktchksq unzip -C -s -j -o %a %f

  Do so for every "Extract" statement found in compress.cfg

- Step 4: Testing

  For testing purposes, you should remove/uncomment the "QuietArc" keyword
  from your Squish configuration file in order to see what's going on during
  the decompression process.  Then poll your uplink and carefully watch the
  tosser task.  You should see that pktdate is being called before squish is
  invoked if any uncompressed PKT file have been received.  It should also be
  invoked multiple times when Squish is uncompressing arcmail bundles, once
  for each PKT file.  If everything works, you might want to change the "-l3"
  parameter to "-l2" in order to only see log messages for pkt files that
  really contain problematic messages.

4.3 hpt
-------

You will probably have a shell script that at some place calls "hpt toss".
Modify this shell script similar to the following example:

 for i in /home/fido/spool/inbound/*.pkt /home/fido/spool/secure/*.pkt \
          /home/fido/spool/inbound/*.PKT /home/fido/spool/secure/*.PKT; do
   if [ -f $i ]; then
     /home/fido/sbin/pktdate -c -k -l3 -L/home/fido/spool/log/pktdate.log $i;
   fi;
 done

 hpt toss

This will process all unpacked PKT files that you have received in the
normal or secure inbound directory.  (If you also have a local inbound, add
that one as well). Adapt the paths to match your configuration, of course.

In order to also have PKT files checked that are inside arcmail bundles,
use the "afterUnpack" command in fidoconfig to call pktdate on all files in
your temporary inbound directory:

afterunpack for i in /home/fido/spool/temp/inbound/*.pkt; do if [ -f $i ];
 then /home/fido/sbin/pktdate -c -k -l3 -L/home/fido/spool/log/pktdate.log $i;
 fi; done

Type this into a SINGLE line in the fidoconfig file!  This command assumes
that you run your unpacker with the "convert everything into lower case"
parameter.  If you don't, you also need to give the paths with "*.PKT"
and "*.Pkt" as arguments after the "in" in the for loop.

Once everything works, you will wnat to change the "-l3" parameter to
"-l2", in order to avoid disk full errors because of the pktdate log file
...


5. Fine-Tuning
--------------

Apart from obvious structural errors - when exactly will pktdate consider a
message date to be unplausible and try to correct it?  And how is this
correction done?

Pktdate has two tolerance windows.  A tolerance is specified in terms of a
maximum allowable message age, ("past tolerance"), as well as a maximum
allowable number of days into the future.

The first tolerance window is the "error detection tolerance window".  You
can set this tolerance with the "-f" and "-p" command line options.
Specify the number of days as argument, or the number of months with a
trailing "m", or the number of years with a trailing "y".  The default
setting is equivalent to "-p 364 -f 1m", i.E. all mails that are not older
than 364 days are OK, as well as all mails that come from not more than 1
month (31 days) out of the future (relative to the current system date).
This will capture most error situations, including sysops that just set
their clock one year into the past to "circumvent" their Y2K troubles.

However, if you intend to do an echomail rescan of mail that is older than
365 days, you might have to adjust the past tolerance, e.G. by using
"-p 10y", saying that anything not older than 10 years is OK.

The future tolerance can probably be set to 1 day ("-f 1", not zero, in
order to allow for time zone shifts, as Fidonet always uses local times).

Once pktdate has found that a message date is illegal, it tries to correct
it.  Pktdate knows some common errors that non-compliant tossers may make,
and tries to undo the effects of these errors.  In order to determine if
such a "undo-calculation" was successful, pktdate uses a second tolerance
time frame, called the "error correction tolerance".  It can be configured
with the "-F" and "-P" options (analoguous to "-f" and "-p" for the error
detection tolerance), and should be set to more strict values than the
error detection tolerance. The default is "-F 1 -P 31".


6. Reason Codes
---------------

When pktdate fixes a message in a pkt file, it will display a reason code
number that explains why the message has been fixed. The following reason
codes can occur:

  Code   Explanation
  ------------------
  1      The FTSC date field does not conform 100% to the specifications
     exactly, but pktdate could still figure out it's meaning. An
         example: Month name is written all in upper case.
  2      The FTSC date field could not be interpreted at all.
  4      The PKT file has been corrupted because the trailing zero of the
         FTSC field spilled by one. The corruption could be solved.
  8      The date is not within the given tolerance boundaries and therefore
         assumed to be invalid.
  16     The FTSC date field contains a string that seems to be a Seadog
         style date string (like "Thu  1 Apr 99 10:05") instead of a FTSC
         style date string (like "01 Apr 99  10:05:00"). Pktdate will only
         complain about this if you use it with the -S command line
         parameter.
  32     Additional explanation for code 8: The date was probably corrupted
         by a 7-bit overflow in a DOS timestamp field.
  64     Additional explanation for code 8:  The date looks right if just the
         year number is adjusted, but day and month remain.  Maybe the
         creator just set his clock some years into the past.
  128    The timestamp in the FTSC date field could not be interpreted, but
         the date part was correct. (Example: "01 Jan 99  12:75:61" will
         become "01 Jan 99  12:00:00" im you instruct pktdate to fix it)
  256    The date field was not initialised.

If two or more reasons both apply to a single message, the reason code values
will be added. For example, if there is both a flaw in the FTSC date field
structure (code 1) and the date is unplausible (code 8), you will get code 9
as a result.


7. Caveats
----------

By the end of 1999, there will be an increasing number of people that turn
their system to a date beyond 2000 and broadcast test messages throughout the
date asking "does my date show correctly"?  Doing so is absolute nonsense,
because if the date will not show correctly, nobody will know why exactly this
is so.  Those people should rather read year2000.txt from this package and
follow the testing instructions there.

However, it is an inevitable fact that those test messages will be
broadcasted.  You need to understand how your system reacts to such test
messages if you use pktdate.  If pktdate receives a messages that has a date
that is more than a few days in the future, pktdate will consider this
message's date field broken and therefore patch the date field with the
current date at your system.  The result is that your downlinks will see a
different date than the date that the creator of the message has used.

You have two options on what to do about this.  First, you can simply live
with the problem.  If anybody asks why your system changes date fields, tell
them about pktdate and assure them that in the year 2000, your system will no
do these modifications any more.

The second option is to increase the future date tolerance of pktdate during
the year 1999.  If you invoke pktdate with the option "-f 15y", pktdate will
have a future tolerance of 15 years and will not modify pkt files with date
stamps in this range.  This should be enough for every silly test message that
might be posted in the net.  However, if you do it this way, you should
remember to remove the "-f 15y" switch as soon as you enter the year 2000,
because then, your system will probably receive lots of messages that have
future dates not because the author intended to set them, but because he or
some system on the way is using buggy software.

Another thing that I would like to point out that Pktdate is only a workaround
written in the desperate attempt to keep the net going the the 21st century.
Using Pktdate is definitely a good idea, but on the other hand, the fact that
you or your up-/downlinks use it does not free you from the burden of doing
Y2K tests on your own system and replacing broken software.  Pktdate is only
an additional measure.  In particular, Pktdate only fixes bugs in the PKT
format, i.E. the transport format used for exchanging mail packets between
systems.  It can't do anything if any of your programs accesses your internal
message base in a way that screws it up, nor can it fix any other problems
that occur internally in your system.


8. Final Words
--------------

Pktdate can only be a success if it can discover and correct every or at
least nearly every problem that occurs in pkt files. Of course, I can not
know about every problem that every software might have. Therefore, I ask you
to send all sort of broken PKT files that you managed to create with non-Y2K
ready software to me, so that I can build "support" for these specific bugs
into Pktdate. The more details you provide on the problem you discovered, the
better.



9. Credits
----------

Special thanks to:

- Bernard Henter (2:233/405) for suggestions, and for the JAM docs.
- Sergey Ozerov (2:5020/348.2), Serguei Trouchelle (2:464/4077) and Vasily
  Zakharov (2:5020/156.35) for bug reports and suggestions.
- Craig Hutchison (3:634/30) and Andy McArdle (3:634/300)
  for providing the Amiga exectables of 1.1 and up.
- Klaus Kulbarsch (2:2426/1205) for writing the German version of this text.
- Frederik Retsema (2:280/905) for his comments.
- Robert Pearson (1:121/45), who had the idea for this program.
- Markus Maier ([e-mail]), for providing the Amiga executables of 1.0 and
  "debugging" the documentation.

[EOF]