ftp.wwiv.com

home *** CD-ROM | disk | FTP | other *** search

/ ftp.wwiv.com / ftp.wwiv.com.zip / ftp.wwiv.com / pub / MISC / Y2KTOOL5.ZIP / year2000.txt < prev

Wrap

Text File | 2000-01-09 | 31KB | 662 lines

Y2KTOOLS Release 5 compiled by Tobias Ernst @ 2:2476/418 -------------------------------------------------------- Contents 0. What's new in this package. 1. Overview. 2. Where date stamps are used in Fidonet. 3. Using the tools to examine your system for Y2K readiness. 4. Recompiling the tools for other platforms. 5. Programmer's Note: How to correctly use a 2 digit date stamp. 0. What's new in this package ---------------------------- See the file HISTORY for more information. If you are only interested in installing the pktdate fix utility, you must not necessarily continue to read year2000.txt, but can step over to pktdate.txt, although year2000.txt provides quite some useful background. 1. Overview ----------- This package contains information on Fidonet Year 2000 issues. It describes general problems that may occur from the year 2000 on in various fidonet file formats. Tools are provided that you can use to examine various fidonet files (a Squish base, a JAM base, a PKT file, a MSG file) to determine if the programs that you use will fill in the various date fields correctly after the year of 2000. It is necessary that you use the tools provided in this packages to examine that date stamps of your mail files, because mostly, a fidonet mail has more than one timestamp. Not all fidonet readers display all timestamps that are contained a fidonet mail. Most choose only one to display and ignore the others. So it can happen that some program of yours fills garbage into some date stamp field, but you won't notice it because your reader ignores it. (Only your down- or uplinks will notice it if they get a grunged packet from you ...). This package does NOT provide a list of programs that are proven to be Y2K compliant or not. Such a list will be provided separately at a later time. German users will find Daniel Friedmann's list a valuable ressource. It is posted periodically in FIDO2000.GER. You may freely redistribute the contents of this package, but if you do any modifications, or have any critics or suggestions, please submit them to me. 2. Where date stamps are used in Fidonet ---------------------------------------- 2.1 Date stamp formats ---------------------- You may skip to section 2.2 or chapter 3 if you are not interested in these details. 2.1.1 The FTSC time stamp ------------------------- The FTSC time stamp is a 20 bytes character string. It is supposed to be filled in as follows: "dd mmm yy hh:mm:ss" plus a trailing zero byte where dd is the day number mmm is a three character abbreviation of the month name yy is a two digit (!) year number hh:mm:ss is the time Example: 01 Jan 98 22:10:05 22 Jun 10 08:00:00 The FTSC time stamp occurs in various file formats, among them PKT, MSG and Squish SQD files. The most common problem with it is that programs erroneously write a three digit number into the year field in the year 2000 (i.E. they write 100 there instead of 00), which spills the 20 character field and can yield undefined behaviour of other programs reading this file. On the other hand, the FTSC time stamp is the only time stamp used in fidonet, that in theory can be used for eternity without a day problem. If you are a programmer that implements FTSC time stamps, please read chapter 5. A lots of programmers that nowadays are fixing Y2K problems are fixing them in a wrong way (i.E., they are only delaying the problem for 50 or 80 years). Chapter 5 describes how you can fix a Y2K problem with a 2 digit year number permanently. 2.1.2 The UNIX time_t --------------------- The UNIX time_t is a 32 bit integer value that counts the numbers of seconds that have passed since the beginning of the epoch, which is 01 Jan 1970. The UNIX time_t is uncritical during the millennium transition from 1999 to 2000, but it will have an overflow in 2038 (or so). In the UNIX world, this problem is expected to be solved by expanding the integer to 64 bit. UNIX time_t timestamps are used in the JAM message base, for example. 2.1.3 The struct _stamp ----------------------- The struct _stamp is a 32 bit integer in packed format that is defined as follows: Bits 0 .. 4: day number (1 .. 31) Bits 5 .. 8: month number (1 .. 12) Bits 9 .. 15: years passed since 1980 (0 .. 127) Bits 16 .. 20: second divided by 2 (0 .. 29) Bits 21 .. 26: minute (0 .. 59) Bits 27 .. 31: hour (0 .. 23) The struct _stamp is generally stored in little endian notation (least significant byte comes first), and "Bit 0" in the table above refers to the least significant bit. The struct _stamp is used in Squish SQD files, and as "Opus Date" in *.MSG files. It is safe until the year of 2107. Then it will have an overflow. 2.1.4 Packet header format -------------------------- This timestamp format is used in the PKT file header. It is simply a sequence of three 16 bit integers stored in little endian notation. The first describes the year (e.g. 1998 or 2001), the second describes the month (1 .. 12) and the third describes the day (1..31). 2.2 File formats ----------------- 2.2.1 PKT files --------------- A fidonet PKT file contains a packet header and one or more packed messages. It is the file format that is used for exchanging mail between two fidonet nodes. In order to maintain operability of fidonet, it is absolutely mandatory that you assure that your system will emit correct date stamps in outgoing PKT files!!! A PKT file contains a header time stamp in the Packet header timestamp format (see 2.1.4) at offset 4, which denotes the packet creation date. It is not of much significance, but should be correct anyway. Each message stored in a PKT file contains a packed message header. Inside this packed header, at offset 0Eh, a FTSC time stamp (see 2.1.1) is stored. An incorrect FTSC time stamp at this place is the most common source of problems. For a detailed description of the PKT file format, refer to FTSC-0001. To sum it up: A PKT file contains a header date field and a date field for each individual message stored inside the PKT file. 2.2.2 Fido *.MSG files ---------------------- The most simplest form of fidonet message base is a message base in Fido *.MSG format. Each folder (netmail area, echomail area) is represented by a separate subdirectory, and each of these subdirectories contains a separate file in *.MSG format for each message stored in the area. Today, the most common usage of *.MSG is as Fastecho's primary netmail folder. A *.MSG type message contains three timestamps. First, there is the 20 byte FTSC time stamp (see 2.1.1) of the mail at offset 090h. A lot of programs today interpret and/or write this time stamp incorrectly after Y2K, which is a serious problem, because this timestamp will usually be promoted into the outbound PKT file *unchanged* by your tosser, so that you will created PKT files with grunged date field. Then, a *.MSG type message can contain two more timestamps, the so called "Opus time stamps". These are extensions that were not originally suggested by FTS-0001, but are commonly used by today's software. Both opus time stamps are in the struct _stamp format (see 2.1.3). The first opus timestamp at offset 0B0h contains the date when the message was written. It should normally show the same date as the FTSC time stamp. The second opus timestamp at offset 0B4h contains the date when the message has arrived at *your* system. Thus, this field is always filled in by *your* tosser, even if the message has been created on a remote system. To sum it up, a *.MSG-format file contains three timestamps: The FTSC "written" timestamp, the Opus "written" timestamp, and the Opus "arrived" timestamp. Each available fidonet reader uses only one of these timestamps, so you should not trust your reader's display when you try to examine your system, but rather use the tools provided in this archive. 2.2.3 The Squish Message Base ----------------------------- I won't explain the details of the Squish Message Base structure, because they are very complicated. A Squish Message Base stores three files for each area (netmail or echomail). If you access a Squish Message Base via the SMAPI, the MSGAPI, or own-written code, you can obtain three timestamps for a message, much in analogy to a *.MSG message base. First, each message stored in a Squish SQD file contains a FTSC type timestamp (see 2.1.1), that normally should only be set by the reader when writing a message and be promoted as is by the tosser to a PKT file. Here again, the problem arises that some fidonet readers create incorrect FTSC type time stamps. Then, each message contains a binary "written" and a binary "arrived" date field. Both of these fields are stored in the struct _stamp format (see 2.1.3). To sum it up, each message inside a squish echo- oder netmail area has three timestamps: FTSC "written" date, "Squish written" date, and "Squish arrived" date. Each available fidonet reader uses only one of these timestamps, so you should not trust your reader's display when you try to examine your system, but rather use the tools provided in this archive. 2.2.4 The JAM Base ------------------ The JAM message base type itself uses UNIX timestamps. These do not have an overflow at 1.1.2000 (but they have one in January 2038). The JAM message base type as for itself will not trigger any particular problems, but of course, if any program gets wrong dates from another source (or because of an internal bug), it can still store wrong dates to the JAM message base. 2.2.5 The Hudson/QuickBBS Base ------------------------------ I don't have exact information on this message base type yet. However, if you use a hex editor, you will see that it stores a textual date similar to the FTSC date, but in a different format. Thus, it will have similar problems with editors writing grunged three-digit year numbers. 3. Using the tools to examine your system for Y2K readiness ----------------------------------------------------------- 3.1 What tools are provided? ---------------------------- This archive provides the following three tools: pktdate will show all date stamps occurring in a PKT file Invoke it with the filename of a PKT file as parameter. It can also be used as a PKT filter that automatically corrects broken PKT files (!). See pktdate.txt for more info. msgdate will show all date stamps occurring in a MSG file Invoke it with the filename of a .MSG file as parameter. sqdate will show all date stamps occurring in a squish base. Invoke it with the base filename of the squish area (as defined in squish.cfg) as parameter. jamdate will show all date stamps occuring in a JAM message base. Invoke it with the base filename of the JAM area (as defined in FESETUP, or wherever you defined it, i.E. without the .J?? filename extension) as parameter. You will find precompiled versions for the most popular operating systems in the corresponding subdirectories. 3.2 Testing the outbound path ----------------------------- At first, you should test if mails that you create after the year 2000 will have correct date stamps. To do so, proceed as follows: 1. Shut down your mailers. Shut down any schedulers/crons that are running. Shut down any other services that might be affected by sudden date switches. Basically, make sure that nothing is running on your system that does not have to be. 2. Let your tosser export all messages that are still unexported in your message base, and let it toss everything that still might be dangling around in the inbound. We don't want to get important mail with wrong timestamps :-). 3. If you use sophisticated log file analysis utilities, make a backup copy of the relevant log files. 4. Create a new empty area in the Fido *.MSG format. (You will probably have to create a subdirectory manually, and add an entry for this area into your tosser configuration file). For the following, I will assume that the path of the subdirectory is e:\msgbase\y2ktest\ . 5. Create a new empty area in the Squish message format. (You will probably only have to add an entry for this area to your tosser configuration file). For the following, I will asume that the files for the squish area have a base name of e:\msgbase\y2ksq, that is, the files will be named e:\msgbase\y2ksq.sqd and so on as soon as the area will be created. 6. Create a new empty area in the JAM message base format, if you use this. I will assume a base name of e:\msgbase\y2kjam in the follwing, i.E. the files will be named e:\msgbase\y2kjam.jhr, y2kjam.jdt and so forth. 7. If you are using other message base formats, create an empty folder for each of them as well. We have no tools yet to examine these message bases directly, but we will at least be able to examine the output PKT file. 8. Switch the system to a date in the new millennium. It is better not to use 1/1/2000, but something more creative like 1/4/2003 or anything. 9. Use your favourite mail editor, and any other mail editors that are used on your system (don't forget your BBS), to write a new message into each of the folders created during steps 2 and 3. Write one message for each editor that is used on your system. 10. In order to examine the dates in the *.MSG folder, change to the directory that you have created (e.g. e:\msgbase\y2ktest), and type "dir". You will probably see something like the following: 1.01.10 8.00 345 0 1.msg 1.01.10 1.25 2 0 lastread Then apply the "msgdate" program from this archive to each of the files ending with *.MSG in this directory. A correct output should look similar to this one: [E:\msgbase\y2ktest]e:\y2ktools\msgdate 1.msg File: 1.msg FTSC-Date Opus Written Date Opus Arrived Date 01 Jan 10 01:25:21 01.01.2010, 01:25:20 01.01.2010, 08:00:12 Check if the dates are correct. Especially, check if the FTSC-Date has a two digit year number (as shown above). Check if the Opus Written Date matches the FTSC date. The Opus Arrived Date is not so important, because the message has not yet arrived anywhere. Repeat this step for very .MSG file you find in this directory. 11. Apply the sqdate program to the squish area that you have created. If you used the path names I gave as an example above, you would type the following command: sqdate e:\msgbase\y2ksq A correct output should look similar to the following: Area: y2ksq Msg# FTSC-Date Squish Written Date Squish Arrived Date 1 01 Jan 10 17:32:04 01.01.2010, 17:32:04 01.01.2010, 17:32:04 3 01 Jan 10 17:40:12 01.01.2010, 17:40:12 01.01.2010, 17:40:12 Again, check all the dates. The FTSC date field should have *two* digit year numbers, the other dates should have *four* digit ones. 12. If applicable, apply the jamdate program to the jam area that you have created. If you used the path names I gave as an example above, you would type the following command: jamdate e:\msgbase\y2kjam A correct output should look similar to the following: File: e:\msgbase\y2kjam.jhr Msg# Date Written Date Received Date Processed 4 20 Nov 1999 23:46:56 Please not that for *JAM*, *ALL* date stamps *MUST* have a four digit year number. The "Received" and "Processed" time stamps are allowed to be empty, but if they aren't, they must be valid, i.E. have the same format as the "Written" time stamp. 13. Now, you have to export the message you have written. If the areas you have created were echomail areas, you have to define one downlink for this area. Use your sysop point, if you have one, or create a dummy point number and link the area to this dummy point. The system that you choose to connect to your test area should presently not have any pending outbound mail, in order to make it easier for you to find the PKT files that were newly created. Then instruct your tosser to export the newly written messages. 14. Now, you have to locate the PKT files that your tosser created. This largely depends on what mailer system you are using. If you are using arcmail attach (Frontdoor or similar), you will have a netmail message in your primary netmail folder that has an arcmail file attached. Locate this file and unzip it. If you have a binkley outbound, then you either know how to find files in it because you're an expert, or you can use an outbound manager to locate the arcmail file, unzip it, and copy the unzipped PKT files to a temporary directory. In order to verify that you have found the right PKT files, you should perhaps use a PKT viewer, a hexdump tool or in the worst case the "type" command to check if the message text inside the PKT file is the same text that you typed in your test messages. 15. Now, apply the pktdate command to the pkt file. A correct output should look similar to the following: [e:\temp] pktdate 12832abc.pkt Packet: 12832abc.pkt Packet header date (D/M/Y): 01.01.2010 Msg# FTSC-Date 1 01 Jan 10 17:32:04 2 01 Jan 10 17:40:12 Verify that the dates are correct (i.E. 2 digit year numbers in the FTSC date field, and the correct four digit year number in the packet header date. 16. Don't forget to reset your system clock to the current true date, and restart your mailers and everything you had shut down in step 1. If you are using sophisticated log file analysis tools, you should, before restarting the mailers, also restore backup copies of your log files now, or just delete the current log files, because some of the tools will have written dates in the future to the log files during the tests that you did, and this could be a problem for the log file analysis tools. If all tests described above succeeded, you can be relatively sure that your system is Y2K safe. BTW, in the text above, I suggested to use echomail areas for testing. If you use a netmail tracker, you should repeat the steps above with netmail areas in order to verify the tracker. I assume that if you use a netmail tracker, you are an expert and know which files in which format are created at which state of the exporting process, so you can decide to apply which of my programs at what time on your own. 3.3 Testing the inbound path ---------------------------- In order to check if your system is able to correctly process mail that is coming to your system and has correct date fields that indicate a year of 2000 or above, proceed as follows: 1. You need some PKT files that have been generated in a process similar to 3.2 and contain correct date fields. I.E., you need an arcmail packet that contains mails that have been created with a system date of 2000 or later, and that have been created with Y2k-ready software. Preferably, these mails should be echomails in a test-echo that does not exist in reality, like "Y2K". The best way to obtain these files is to setup a "sysop point" system that uses Y2K-ready software. In order to get the files, you would shut down your node system's scheduler or tosser task. If you spawn your tosser from your mailer, you should temporarily modify the mailer config in order NOT to spawn the tosser. Then send the arcmail file from the sysop point to the node. The mailer will store it in its inbound or protected inbound directory, from where you should move it to a temporary directory. 2. Once you have a file containing messages like that, no matter how you have obtained it, unzip it. You will get one or more PKT files. Apply the pktdate tool to each of these files in order to verify that all date stamps in these PKT files are still correct. 3. Set up the test echomail area in which these areas are written in your tosser. For the beginning, set it up to be Fido *.MSG. Besides an entry in the tosser configuration, this requires that you create the corresponding directory manually. Make sure that the area is empty (don't use an existing area for this. 4. Copy the PKT files to your protected inbound directory and have the tosser toss them, but retain a copy of them elsewhere. 5. Similar to step 10 of section 3.2, you now have to apply the msgdate tool to each *.MSG file that has been created in the subdirectory that represents the test echomail area. Refer to step 7 of section 3.2 for further information. Make sure that the FTSC and Opus Written timestamp are correct, representing the date after 1.1.2000 on which the mails have been written. Make sure that the Opus Arrived Timestamp represents the system time of your system when the mail arrived there (normally a value < 2000, because we have not set our node system's clock to a date after 2000 for THIS test. 6. After this test succeeded, erase the test folder and recreate it, but this time with the squish storage type. 7. Copy the PKT files to your protected inbound directory again, and have the tosser toss them. You might have to disable the dupe checking facility of your tosser for this. 8. Similar to steps 11 of section 3.2, apply the sqdate programs to the squish test area, and make sure that all dates are valid. 9. Repeat steps 6 to 8 with a JAM test area and jamdate, if necessary. 3.4 Estimating the results of the test -------------------------------------- The most crucial point about the tests we have just performed is that the FTSC time stamp in the packed message in an outbound PKT file is correct. If you produce PKT files where this is not the case, you could become subject to a formal complaint. On the other hand, it is also important that all other dates in your message base are correct. I have heard people saying that they don't care if a mail is displayed with a wrong date or not, but I don't agree. Consider, for example, what could happen if the date fields of the "Arrived" time stamp are wrong and you instruct your tosser to purge your message base based on arrival date. Or consider what happens if a downlink of yours rescans an area from you. He will then probably not get the messages that he wanted. As a consequence, you should under all circumstances make sure that in all steps from section 3.1 and 3.2, my tools always report the correct dates on all messages in all fields. 4. Recompiling the tools for other platforms -------------------------------------------- The msgdate, jamdate and pktdate tools are especially easy to recompile. They don't need to be linked to anything, they are pure ANSI C, and they don't rely on a compilers packing modus for structures. They will compile with any ANSI compliant compiler and will yield the same result no matter with which compiler they have been created or on which platform they run on. Simply type something like "cc -o msgdate msgdate.c" to compile them. The sqdate tool is not as easy. I have refrained from writing my own routines to access the squish message base, but have decided to use Paul Edwards SMAPI, which is also used in MsgEd, for this purpose. The SMAPI is available for file request as SMAPISS.ZIP at my system (2:2476/418), or as smapi_latest.tar.gz from ftp://ftp.sar-gmbh.com/pub/freeware/husky. You will need to compile the SMAPI, because you will need to link it with sqdate.c if you whish to compile your program. The SMAPI will compile on OS/2 DOS, Windows, FreeBSD, Linux, some other Unices and Amiga. Please take care that you do use SMAPI version 1.5.6 or newer, because only this version is fully Y2K compliant. 5. Programmer's Note: How to correctly use a 2 digit date stamp. ---------------------------------------------------------------- When you write Fidonet software, you will generally have to deal with the problem that you have to translate a 2 digit year number, that you have extracted from a FTSC time stamp in a packed message, into a four digit year number, that you can use to write struct _stamp structs, or that you can use to display the full four digit year to the user. 2 digit year numbers were invented by people that did not think that their software and designs would survive the year 2000 anyway. They just used to put a 19 in front of the two digit number. This is obviously wrong. Nowadays, some people try to fix problems with two digit year numbers like this: They set up rules that say "If the number is greater than or equal to 80, then assume it is in the 20th century. If it is less then 80, then assume it is in the 21st century.". In my opinion, this is not the correct way to fix the problem, because a program like this assumes that it will not survive the year of 2080, just like it previously assumed it would not survive the year of 2000. But you'll never know ... There is a way to fix the problem much more elegantly, while you can still continue to use two digit year numbers. Suppose you are a human living in the 25th century, say, in the year of 2460, and a friend of yours says that he has been born in the year 20. You will immediately know that he means the year of 2420. He can't mean 2320, because he does not look like a 140-year-old "grufti", and he can't mean 2520, because even in the 25th century, time-travel has not yet been invented. The same intelligence that you have when you interpret that your friend must mean 2420 when he says 20 (or 1970 if he says 70 and you are living in our times ...), can be easily applied to your software as well: A fido PKT file that you receive on your system will probably never be older than 10, or lets even say 50 years. It will also usually not come from too far in the future. (It might come from the future because someone is testing their system, though). Therefore, I suggest that a two digit year number is always interpreted as follows: Add a century number to it in that way, that the difference between the date of the message and the date of the system you are running on is never bigger than 50 years. Here is an example in C. Suppose the parameter "current_year" holds the current system date's year, the parameter "two_digit_year" holds a two digit year number, and the return value of the function below should be the four_digit_year number that is "meant" by the two-digit_year parameter. The code is not optimised, but is is written as to make the algorithm clear: int interpret_two_digit_year(int current_year, short two_digit_year) { int four_digit_year; /* our return value */ int current_century; /* current century, like 19, 20, 21, ... */ /* determine the current century */ current_century = current_year / 100; /* at first, assume the two digit number */ /* represents a year in the current century */ four_digit_year = two_digit_year + (current_century * 100); /* then make adjustments if the four digit year */ /* that you we have calculated seems to be too old */ /* or too new */ if (current_year - four_digit_year > 50) { four_digit_year = four_digit_year + 100; } if (four_digit_year - current_year >= 50) { four_digit_year = four_digit_year - 100; } return four_digit_year; } In the time until the year of 2030, your program will behave exactly like it did if you used the simpler fix method ( >=80: 1900, <80: 2000), so it won't make a different at first. However, in the time after the year of 2030, your program will slowly begin to adapt to the "newer times", and in the year of 2080, your program, or programs derived from it, will still be fully function, while the program that used the simpler fix method would not work any more. I know that none of us will probably live in the year of 2080. However, our sons and daughters, or grandsons and granddaughters, will, and I think we should not let them run into the same problem in 2080 that we are presently having with 2000. BTW, this does not only apply to fidonet. There are a huge number of computer applications where you can use a concept like that described above. Sometimes, you don't have a -50/+50 - range, but perhaps a -95/+5 - range, but it still is a better idea than a static 80/20 fix. Sometimes though, you can't use a dynamic method, e.g. if you have large databases that track historical events, or if you track people's birthdays and life spans (they might be longer than 100 years), and similar. In that cases, though, you have to use 4 digit fields, there is no other way to fix *that* at all. But a fidonet PKT surely is not an application where you need to exactly know the absolute date of a 200 years old packet, so in fidonet packets, there is no problem with dynamic century determination. 6. Finally ---------- If you have any questions, you can reach me at: Tobias Ernst @ 2:2476/418 tobi@bland.fido.de [EOF]