Programmer 7500

home *** CD-ROM | disk | FTP | other *** search

/ Programmer 7500 / MAX_PROGRAMMERS.iso / PROGRAMS / UTILS / COMPRESS / IDC15.ZIP / IDCSHELL.DOC < prev next >

Wrap

Text File | 1989-07-02 | 69.9 KB | 1,532 lines

IDCshell - A DOS/SHELL AND ARCHIVE UTILITY (for revision history, see IDCSHELL.REV) Documentation for IDCshell.EXE Written in assembler by Gary Conway Infinity Design Concepts, Inc. Louisville, Kentucky Copyright 1987-89 Infinity Design Concepts, Inc. all rights reserved Version 2.1 IDCshell.EXE is a user supported shareware product. As such, you have the opportunity to test drive the program before deciding whether or not to buy. One of the beauties of the shareware concept, is this "try before you buy" idea, but note that this does not relieve you of your obligation to buy if you do find the program to be of value to you and continue using it. You are also encouraged to give copies to your friends for their evaluation, as long as the documentation and program(s) remain intact as they were released by the author and as long as you DO NOT distribute copies of ARCCFG.EXE. IDCshell.EXE is and will remain the property of Infinity Design Concepts, Inc. This program may not be used in any connection with commercial ventures, nor as a sales aid, without the expressed written consent of the author. All rights are reserved. Infinity Design Concepts, Inc. 1052 Parkway Drive Louisville, Kentucky 40217 Voice: (502)-636-1234 Data: (502) 635-5471 Member IEEE PCCL KKUG NSPE All new shareware releases of IDCshell.EXE and all other IDC software can be located -FIRST- on ; The SoftStone FOG #24 (supporting CP/M and MSDOS) (502)241-4109 60 MEG 300/1200/2400 baud 24 hrs. Louisville, Kentucky Curt Edwards - SYSOP Sponsored by: Kentucky Kaypro Users Group Accounting Computer Systems First Osborne Group REGISTRATION If you find yourself using IDCshell, please take the time to do the right thing and that is register your copy. You have been provided the opportunity to freely test the program before even thinking about registering. This is only fair, so, in fairness, you should reciprocate and register your copy, if you continue using the program. Why register if you use the program ? 1) You get the ARCCFG.EXE program for customizing IDCshell. 2) You get notification of updates to IDC software. 3) You get FREE net-mail services for contacting IDC. 4) It is the right thing to do. You will find the registration form in the ARChive with this document under the name REGISTER.FRM. Please use this form for registration. MASTERCARD AND VISA are accepted, but you MUST follow the instructions in the REGISTER.FRM, otherwise, the order will not be processed. THIS IS NOT A FREE PROGRAM ........................ ............................ ........................ TABLE OF CONTENTS ............................ ............................................................................. Page FOREWORD .......................................... 1 Running IDCshell .................................. 2 Environment Variables.............................. 2 GENERAL ........................................... 4 HELP .............................................. 5 Copy File(s) ...................................... 6 Directory Tree .................................... 6 Quick Tree Traversal .............................. 6 Erase file(s) ..................................... 7 Move file(s) ..................................... 7 Rename file(s) .................................... 7 Print file(s) ..................................... 8 View .............................................. 9 ANSI view feature ................................. 9 Wildcard tag ...................................... 10 Mouse ............................................. 10 Editor ............................................ 10 Call NARC ......................................... 10 DOS Shell ......................................... 11 File Attributes ................................... 11 ERRORS ............................................ 12 Finding a File .................................... 12 What is an ARChive ................................ 13 Compression Techniques ............................ 14 Storing ................................. 14 Packing (run length encoding) ........... 14 Squeezing ............................... 14 Crunching (LZW-8) ....................... 14 Squashing (LZW-9) ....................... 14 Tagging files ..................................... 18 Why TAG files anyway ? .................. 18 Tagged files and archives ............... 19 ARChive preview function .......................... 19 Function Keys ..................................... 21 ........................................................................ IDCshell (c) 1987-89 Infinity Design Concepts, Inc. all rights reserved ........................................................................ FOREWORD IDCshell.EXE is the denouement of many hours of rewarding and painful labor. The long belated release of IDCshell was the consequence of many factors, not the least of which, was a dedication to writing a sleek, powerful and fast utility. Many of the routines have been completely rewritten several times in the ultimate quest for speed and compactness. These are the very reasons that ALL IDC software is written in assembler. There were bugs that took months to fix, once located, others that subtly emerged and still others that lay waiting for that right set of circumstances to pop up. We have made a more than substantial effort to squash bugs in IDCshell and hope that we have hit the mark, one hesitates to be too certain in the software industry. There were a number of people who proved vital in the debugging phases and have made monumental contributions to the project. I would like to thank Curt Edwards for being so damned picky in offering suggestions for features and how they should work. Curt was an endless source of ideas (and more work for me). I would also like to thank Jerry Taylor for his uncanny gift for finding bugs that the entire universe would overlook. I would like to thank my mom and dad for not killing me at an early age and my wife for not killing me at a later age, although, I may not be out of the woods yet. IDCshell is the grandchild of B29.COM (first released Sept. 1985), a CP/M utility that was very similar to IDCshell and handled LBR files. Many of the features of IDCshell were taken from the original DOS version of B29, which I never released. The directory tree routine is a complete rewrite of IDCTREE.COM, which was released in late '86. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 1 IDCshell is written entirely in assembler using the Microsoft MASM 5.1. The program was written on a Kaypro PC and an ALR 386/20. The editor of choice was the Norton Editor from Norton Computing. Approximately 30 support programs were developed in QuickBASIC, QuickC or assembler to aid in coding, debugging and testing algorithms and subroutines used in IDCshell. Including these files the source code is well over 900 K and some 30,000 lines of code. Acknowledgments: We at IDC would like to thank the following people for their kind direct or indirect assistance in the development of IDCshell. Curt Edwards Bob Freed Jerry Taylor (Creative Applications) Terry Welch Chuck Crumpton Carlisle Crutcher Murphy Frank Roemer Programmer's thought: You are your own worst enemy. Gary Conway, Pres. RUNNING IDCshell When IDCshell is first invoked, it reads the current directory, sorts and formats the entries and displays them on the screen. This is called the FILES SCREEN. The box on the right hand side of the screen shows IDCshell's commands. Note that the F1 key will bring up a full screen of help. Some of IDCshell's commands can be accessed through the command bar at the bottom of the screen. The cursor keys or the mouse may be used to move the highlight bar left and right. Once the highlight bar is over the desired selection, the ENTER key will execute that command. Alternatively, you can use the first letter of each of the menu bar commands with the same effect. Throughout the program, the left mouse button simulates the ENTER key and the right mouse button simulates the ESC key. ENVIRONMENT VARIABLES If you don't know what an environment variable is, then continue reading. If you DO know, then skip to "The Goods" section below. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 2 What are they ? An environment variable is a convenient way for programs to communicate with DOS or with each other through a commonly shared spot(s) in memory. If you type "SET" <ENTER> at the DOS prompt, DOS will respond by showing you the current spot of shared memory which we call the environment. These text strings can be created,read, modified and deleted at any time by any program. Each variable has a certain format that must be adhered to. The format is as follows, variable_name = string An example would be IDCED=C:\UTILITY\WS.EXE Spaces are allowed before and after the equal sign. This variable will mean absolutely nothing to DOS, but, to IDCshell, it will show the way to your editor. The variable name here is IDCED and the string is C:\UTILITY\WS which is the pathname for finding WordStar on the disk. You can change this string to anything you want by simply typing --- SET IDCED= (and type your pathname for your editor). IDCshell uses several of these environment variables to communicate with you and you should set them up in your AUTOEXEC.BAT file. What do I do if DOS says, "Out of Environment Space" ? DOS reserves only so much space for the environment strings, and if you have a long search path or a lot of strings, then you may run out of space. I can offer two suggestions, one for DOS 3.1 and one for DOS 3.2. There is a "switch" that you can add to your CONFIG.SYS file (If you don't have one, then you should !). One of the lines in your config.sys file should read Shell=C:\COMMAND.COM or something similar. You can add the following to the end of this line to increase your environment space. DOS 3.1 - /E:nn where nn = 10 up to 62. This number times 16 is the length of your environment. example: Shell=c:\command.com c:\ /P/E:10 This would reserve 160 bytes. DOS 3.2 - add /E:nnnn where nnnn is the actual number of bytes to reserve. example: Shell=c:\command.com c:\ /P/E:160 This would reserve 160 bytes. Note that the minimum environment size is 160 bytes. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 3 The Goods .. IDCED=c:\editors\ws.exe Use this variable to tell IDCshell where your editor is located, so that when the F9 key is pressed to invoke the editor, IDCshell will know where to locate it. The drive and path and editors name must be given (in upper or lower case). The file extension (.COM or .EXE) MUST be given. NARCPATH=c:\utility\narc.exe Use this variable to tell IDCshell where to find NARC when the "N" key is pressed. As with IDCED, the drive and path and filename and extension must be given. IDCHELP=c:\utility\idchelp.hlp Use this variable to tell IDCshell where to find the IDCshell help file when the F1 key is pressed. IDCTEMP=e:\ramdisk Use this variable to tell IDCshell where it can place temporary work files. If you have a ramdisk set up, then this would be a good use for it since all file operations will be MUCH faster from the ramdisk. All of the above variables can be set up in your Autoexec.bat file. The format is as follows, SET variablename = pathname GENERAL IDCshell is laid out with many menus, windows, dialog boxes and screens, all emanating from the MAIN SCREEN. This screen displays the program name and release number in the upper left hand corner, followed by the current drive/path, this is the status line. Just below the status line are headers for information describing each file. The fields are, filename, filesize, file date, file time and file attributes. The tag status indicators are located at the right end of this line and show how many files in the current directory are tagged and how many bytes are represented by the tagged files. At the bottom of the screen is the command line. Here you may use the cursor keys or the mouse to SELECT any of the commands and then use the ENTER key or the left mouse button to EXECUTE the highlighted command. You may also use the letter in each command that is capitalized. The last line of the screen displays the name of the current archive (if one Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 4 has been selected). The remainder of the screen is broken vertically into two halves. The left half displays the filenames and directory names that are located in the current directory. Directory names are denoted with <dir> in the file size field and volume labels are denoted similarly with <vol>. The cursor keys, Pg Up, Pg Dn, Home and End keys allow easy navigation through the list. The mouse may also be used to simulate the up and down arrows. Note that you may find any file easily with the ALT-letter combination (see finding files). The right side of the screen is the command menu window showing most of the commands that IDCshell provides and a very brief description of each. Note that much more detailed help is always on line with the F1 (HELP) key. At the bottom of the menu window, the total number of files in the current directory are displayed. Note that this count DOES NOT include directory names or volume labels, nor does it include the "." and ".." entries (although IDCshell will not display them). At the top of the command menu window, the volume label of the current disk is displayed. The command menu window is an interesting story. The first version of IDCshell (then B29 and later IDC-ARC), looked very much different than the version you have now. There was a window for each of the commands on the command line and as you moved the cursor bar right and left, the menu window would change giving much more detailed information than is currently displayed. As commands were added, this windowing system proved inadequate and the IDC menu compiler was written so that a complete on-line help facility was almost always available. This new system gave even more details about the commands. It was also during this time, that the help facility was incorporated into NARC. THE HELP SCREEN From the files screen, it is always possible to call the context sensitive help system. Hitting the F1 key will cause a full screen of help to pop up that has information specific to the command that the cursor bar was pointing to on the command line. Once the help window is opened up, it is then possible to get help on ANY of the commands. The Home key will take you to the help index (as will the "I" key). You can put the window away at any time with the ESC key. Note that the environment variable IDCHELP must be set for the help feature to work correctly. See "Environment variables". Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 5 COPY FILES The copy command will allow copying files from one directory or disk to another. When this command is invoked, IDCshell first checks to see if there are any tagged files. If there are none, then it is assumes that you wish to copy the highlighted file. If there are tagged files, then IDCshell makes no assumptions and asks you whether you wish to copy the highlighted file or the tagged files. After resolving the above, IDCshell will display a directory tree from which you can select the destination drive or directory. If you wish to copy to another drive, simply use the "L" command to log a new disk, select the destination directory there and the copy will proceed. DIRECTORY TREE The directory tree reads the FAT (file allocation table) from the current disk and locates all sub-directories from this information. The sub-dirs are then sorted and sub-sorted and formatted for a visual tree presentation. The commands available when the tree is displayed are the following; ALT-letter this combination of keys will position the highlight bar on the first subdirectory that begins with "letter" Pg UP - goes to the previous page of the tree Pg DOWN - goes to the next page of the tree Home - goes to the top of the tree (root) End - goes to the end of the tree ^Pg UP - goes to the top of the current page ^Pg DOWN - goes to the bottom of the current page "\" key - finds the root (same as Home key) R - remove the highlighted directory from disk (only if directory is empty) M - make new directory, you are prompted for a name L - log new disk drive ENTER - go to highlighted directory ESC - return to where you were before without changing directories Quick Tree Traversal. BIG NOTE: There are four other keys that may be used when moving around your disk tree. When you are NOT on the directory screen mentioned above, that is, when you are at the FILES SCREEN, you may use the "-" or the "G" keys to move one directory toward the root. Conversely, if you highlight a directory entry (denoted with <dir> in the size field), Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 6 you can use the "+" or the "H" keys to select or move down the tree away from the root. In either case, the current directory is changed to the newly selected one and the screen is updated with the new files. ERASE FILES The "E" command will first check to see if there are any tagged files, if there are none, then IDCshell assumes that you wish to copy the highlighted file, if there are tagged files, then IDCshell will ask whether you wish to erase the highlighted or tagged files. Files that are found to have the Read Only attribute set, cause IDCshell to ask before deleting them. MOVE FILES Files can be moved from one subdirectory to another ON THE SAME DISK. Moving a file is similar to copying a file, except that moving is faster and the file is actually moved from one subdirectory to another, not just duplicated as in copying. The file will also, no longer exist in the source directory after the move. The one limitation to moving files, is that you cannot move a file from one disk to another, here, you must use copy. If IDCshell finds no tagged files, then the currently highlighted file will be moved. If there are tagged files, then IDCshell will ask whether you wish to move the highlighted or tagged files. Next IDCshell will take you to the directory tree, where you can select the destination directory for the move. RENAME FILE(S) The RENAME "R" command allows you to rename any file in the current directory . To rename a single file highlight the chosen file and press "R" IDCshell will prompt you with - "ENTER NEW FILE NAME (OR "*" FOR WILDCARD RENAME)" At this time you can enter your new file name and press <ENTER>. However, if you wanted to change several files that match a wildcard mask (BIG???.*), at the above mentioned prompt you would enter "*" , IDCshell would then prompt you with - "ENTER OLD NAME (WILDCARD OK)" Now you can ENTER something like *.PRG and IDCshell would Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 7 then prompt you with - "ENTER NEW NAME (WILDCARD OK)" Now you would ENTER something like *.CMD this action would result in all .PRG files being renamed to .CMD files. Note that the rename function can be used to rename sub- directories as well as normal files. Note also that the volume label cannot be chaned in this way. PRINT FILE(S) The print routine will print the highlighted or tagged file(s). You have several options here. The first option that comes to the screen is the printer string option. This allows you to select a character font for printing the file. There are four optional fonts. they are , 0 Send no printer string (use printer default) 1 Set printer to 10 cpi (pica) 2 Set printer to 12 cpi (elite) 3 Set printer to 17 cpi (compressed) These printer strings come with IDCshell configured for an EPSON printer. They can be changed to anything you like with ARCCFG.EXE. Each string has a maximum length of 9 bytes. The ESC key aborts the print operation completely. The second option tells IDCshell how to format the document. There are three options. Each option is toggled between YES and NO with the SPACE bar. the ESC key aborts the print operation completely and the enter key accepts the currently displayed options. Format - YES - This option tells IDCshell that you want the document formatted during printing. The formatting includes page numbers,page breaks and expanded tabs. NO - This option tells IDCshell not to perform any formatting at all during print. This is for use on files that have already been formatted with formfeeds etc. Strip High bit - YES - This option tells IDCshell to strip the high bit off of each character that is printed. This is to allow WordStar (c) type documents to be printed correctly since WordStar and other word processors use the high bit of some characters to denote paragraph alignment etc. NO - Do not strip the high bit from each character. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 8 Strip Control Characters - YES - This option tells IDCshell to remove all control characters from the document before printing. The exceptions to this are carriage returns,linefeeds and tabs. NO - Do not remove control characters. VIEW The VIEW "V" command allows you to view any file in the current directory by simply moving the highlight bar to the chosen file and pressing "V". This command is very useful for viewing ANY type of file, including binary. By using the MODE "M" function you may toggle between ASCII or HEX views. The FILTER "F" function toggles the filter so that the out put of some word processors can be displayed correctly. Last but not least is the SEARCH "S" function. This function can be used to find a key word or heading within a file. When the SEARCH function is chosen IDCshell will prompt you with- " ENTER TEXT TO FIND " Simply ENTER the text to be found IDCshell will locate the first listing of this text and highlight it onscreen. The search next function will locate the next occurrence of a matching string. If no matches are located, a message will appear. Note that CONTROL-R will repeat the last text when entering the string to find. ANSI VIEW FEATURE This feature is invoked by the F3 key and is UNIQUE to IDCshell. The highlighted file will be displayed on the screen as though it were a file containing ANSI escape sequences. You will note that there are no page up/down commands in this mode. The reason for this is simple, MOST ansi files are single page files and most of the editors that create ansi files, create single page files. You may PAUSE the screen output with the SPACE bar or abort with ESC or ENTER. This feature is provided primarily for SYSOPS of bulletin boards since most BBS software these days supports ANSI files when transceiving in color. This is an easy method of looking at your compliment of ANSI files. NOTE: You MUST have ANSI.SYS loaded in your CONFIG.SYS file for this feature to function correctly. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 9 WILDCARD TAG If you wish to tag several files that match a wildcard filemask (BIGTOOL?.*, for example), the current directory use the WILDCARD TAG "W" function. To use this function press "W" and IDCshell will prompt you with- ENTER WILDCARD MASK TO TAG : Entering something like *.PRG, would result in all files with the extension of .PRG being tagged. Note that tagged files are used when arcing, copying, deleting, renaming or printing files. MOUSE FUNCTION The mouse function is always enabled (providing of course that you have a mouse) and controls the highlight bar and command line. The left mouse button simulates the ENTER key and the right mouse button simulates the ESC key. Mouse sensitivity may be changed in ARCCFG by decreasing or increasing the values for vertical or horizontal movement using the appropriate function key. You may also disable the mouse function in ARCCFG if you so desire. INVOKING YOUR EDITOR IDCshell can call your editor with the F9 key from the main menu. IDCshell must first be told what your editors' name is and where to locate it. This is done via the environment variable IDCED. For complete information on how to set this environment variable, see the ENVIRONMENT VARIABLE section. Once the environment variable is set up, the F9 key will open the currently highlighted file with your editor. Note that the ALT-F9 key combination will call the editor and will not pass a filename to edit. CALL NARC To execute NARC you must make a change in your AUTOEXEC.BAT file so that IDCshell will be able to find NARC. The following line (with your drive and path filled in) should be put into your AUTOEXEC.BAT file. SET NARCPATH = DRIVE:\PATH This is called an environment variable if you don't fully understand this it is suggested that you read the section ; Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 10 ENVIRONMENT VARIABLES Once you have set up your autoexec.bat file it is very simple to call NARC by simply press "N" and NARC will be executed. If the highlighted file is an archive, then this filename will be passed to NARC and the archive opened. If the filename is NOT an archive, then no filename is passed, NARC comes up in its default condition. DOS SHELL If for some reason you wish to exit IDCshell temporarily this is easily accomplished by using the F5 (dos shell) command. THIS will allow you to run DOS commands outside of IDCshell and return by typing EXIT when you are finished. It is also possible to invoke a second copy of the DOS command processor by typing "COMMAND" at the first DOS prompt (if you have enough RAM left). FILE ATTRIBUTES Each file on your disk has one or more file attributes. Here is a brief description of these attributes. READ ONLY (R) The file cannot be deleted NOR can it be modified under normal circumstances. SYSTEM (S) Your MSDOS files are system files HIDDEN (H) These files can be executed but do not show up when you request DOS to show the directory via the DOS DIR command. ARCHIVE (A) Not to be confused with .ARC files. This attribute is used primarily for backing up harddisks. IDCshell shows these attributes in the "Attr" column next to each file in the directory using the abbreviations, R H S or A. If you have not tagged any files IDCshell will assume that you wish to change the highlighted file. In order to change a file's attributes press the "Y" key and IDCshell will display a dialog box showing each of the possible file attributes. Beneath each of the attributes is displayed either "YES" or "NO",indicating that the attribute will be either turned on "YES" or turned off "NO". The left and right arrow keys will move the highlight bar to the selections in the box. The SPACE bar toggles the attributes between the "YES" and "NO" states. The ENTER key will cause the file(s) attribute(s) to be set. The ESC key will abort the operation. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 11 Note that the attributes of the subdirectories may also be changed with this command, but only one directory at at time because you cannot tag subdirectories. ERRORS Requires DOS version 2.0 or above. When IDCshell is first invoked, it checks the DOS version, if it is found that the system is operating on a DOS version lower than 2.0, IDCshell exits with the above message. Insufficient Free Memory to Run IDCshell. When IDCshell is first invoked, it checks to see if there is enough RAM available to run the program, if not, IDCshell exits to DOS with the above message. Error: Can't Move a file to ITSELF, Hit ENTER. When trying to move files, if you do not select a destination directory different from the source, then the move is aborted with the above message. Error: Unable to Move File, Hit ENTER. DOS has reported an error in trying to move the file, disk removal etc. Error: Writing Destination File, Hit ENTER. Error: Reading Source File, Hit ENTER. Either of the above messages can be caused when trying to copy file(s) if the disk is removed. Unable to Open Source File, Hit ENTER. Destination Directory is Full, Hit ENTER. FINDING FILES The "F" command will allow you to quickly find any file in any subdirectory of a disk. This is the global find feature. Invoked by the"F" key, this function will ask for a wildcard filename and will proceed to locate a matching file on the disk. As always, ESCape or ENTER will abort the operation while entering the filename. Once the filename is given, a Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 12 directory tree appears, along with the search mask. The cursor bar moves down the tree as it hunts for your file. When one is found, a menu bar appears at the bottom of the screen asking for further instructions. At this point you may do one of three things; (J)ump to the directory where your file was found (C)ontinue the search, looking for the next match (Q)uit the search operation and return to the files screen Note that if the directory tree has not been previously scanned (and remembered) with the "D" command from the files screen, then it is scanned for the find operation. If the tree has already been read, then it is not read again, speeding up the find operation considerably. You may also find files in the current directory with the ALT-letter key combination. Here, to find a file that started with the letter "Q", for example, you would enter ALT-Q and the first file starting with the letter "Q" would be highlighted. Note also, that hitting ALT-Q again, will locate the next filename that starts with the letter "Q". WHAT IS AN ARCHIVE ? An archive file is a collection of files all rolled into one file. You could, for example, store all of last month's invoices in one ARC file for organizational purposes. Suppose that all of these invoices totaled 400K when adding their file sizes. If you only had 360K floppy drives, you could not put all of this information on a single floppy. With an archive, it is very possible that you can do just that. The arcing program will compress the files by as much as 95% and store them into one file, an ARC file, that will very likely be much smaller than 360K and thus fit on one floppy. Files are added to an archive sequentially in alphabetical order. Each file in the archive has a header that is 29 bytes long (25 for very old archives). This header gives statistical information about the file such as the filename, actual size and compressed size, what method of compression was used (if any), a CRC of the file, and the date and time of the file. History: In the CP/M world, we had what were called library files. These files were collections of ones rolled into one. There was no compression offered, unless you compressed the files BEFORE inserting them into the library (.LBR). Also, the number of entries in the library had to be specified at the Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 13 time the library was built. We had several programs to deal with libraries, LU, NULU and IDC's B29. Library files were soon ported to DOS. This was great until Thom Henderson of System Enhancement Associates (SEA) brought forward his version of library files, called archives (.ARC). What ARC.EXE offered was a new file format and compression of the files as they were inserted. Truly the library file had come of age, except we now called them .ARC files. STORING There is really no story to tell here, the input file is merely copied into the archive. This method is really a last resort. When none of the other compression methods will yield a smaller output file, then storing is used. You will notice that not many files in archives are simply stored and when they are they are usually VERY SMALL files. PACKING Packing or RUN LENGTH ENCODING as it is more formally called is a relatively old, time honored and proven method of compression. It is often mistakenly called NON-REPEAT- PACKING, this term is entirely erroneous, because it is actually REPEAT PACKING as it packs out repeated strings of characters. Suppose that we had a file that had 100 spaces in it and nothing else. Packing would compress that file down into just 3 characters. That's 97% compression, not bad ! The 3 characters would have the following format, SPACE,SPECIAL REPEAT FLAG CHAR,100 in HEX this would be 20,90,64. The only special case here, is when we encounter the SPECIAL REPEAT FLAG CHAR (90h) as data from th input file. This is handled by outputting 90h followed by a zero byte. SQUEEZING For a complete discussion of squeezing (Huffman coding, after David Huffman), see NARC.DOC. CRUNCHING AND SQUASHING The current crunching and squashing algorithms take their roots from an article by Terry Welch in IEEE COMPUTER, June 1984, entitled "A Technique for High Performance Data Compression". This article built on another article by J. Ziv and A. Lempel, entitled, "A universal algorithm for sequential data compression. The term "LZW" was coined in the article by Welch from the three authors' initials. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 14 Crunching and squashing are both variations of the LZW algorithm, since the original made no provision for variable length output codes, nor did it call for an adaptive reset. The LZW algorithm begins by constructing a table in memory, where each entry in the table contains one of the 256 "atomic" characters along with a pointer, thus the format for each entry is, <POINTER> <CHAR> Where POINTER is just that, a number that "points" to one of the other entries in the table. The reason for the pointer should become apparent as we proceed. The following IS the complete LZW algorithm. It appears, at first glance, to be very simple, but the ramifications get downright nasty very quickly. The LZW algorithm, Initialize the string table as described above. Get the first character from the input file (omega). Rept: Get the next character from input (K). IF no more characters then output code(omega): DONE IF omega-K exists in table then omega = omega-K: GO REPT ELSE omega-K not in table, output code(omega) insert omega-K into table omega = K GOTO REPT ENDIF This is the algorithm as originally described by Welch. Note that it does not make provision for the string table filling. It should be apparent that when the string table fills, no more strings (omega-K) could be added to the table. More on this later. What the algorithm does, in effect, is to look at the input file, byte by byte, looking for the longest sequence of characters that it "recognizes". A sequence of characters (a string) is "recognized" if it is located in the string table. The omega in the algorithm above, represents this string of characters and the K represents the last character read from the input file and appended to this string. As an example, take a look at the start of this paragraph and imagine that you are looking at the input file. The algorithm would get the first character "W" and call it omega. It would then get the second character "h" and call it K. We would then search the string table for an entry where <POINTER> points to the location in the table where the "W" entry was made at initialization AND <CHAR> = K. If we find this, then we are ready to get another character and add it to the omega Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 15 string. If we don't find a match in the table, then a new entry is made in the table and omega is sent to the output file as a "CODE". This sequence continues until the input file is exhausted. It is in examining the output "CODES" mentioned above, that we find one of the major differences between SQUASHING and CRUNCHING. Both of the algorithms vary the output code length, they both begin with 9 bits, but crunching ends with 12 bits and squashing ends with 13 bits. When we described the string table at the start of this discussion, we said that the table was initialized with 256 entries, one for each of the 256 atomic characters. Well, the algorithm has shown us that we are going to be adding strings to the table, so it seems that we will have more than just the 256 atomic entries. But how many entries ? In the case of crunching, remember, we said that our maximum code length was 12 bits, in binary, 12 bits can represent numbers up to 4096, thus we have our maximum number of entries in the table. Squashing has a maximum code length of 13 bits, which allows for 8192 entries. From this, is should be somewhat intuitive, that squashing allows learning more strings that does crunching, and indeed it does. But what happens when the string table fills ? The first LZW implementation as described in the algorithm, did not provide for this case and simply stopped "learning" strings, because there was no more room to add them to the string table. Later versions (and squashing) perform what is called an "adaptive reset" of the string table when it fills. This is done by sending a special code to the output file that will tell the uncompressor to reset its table and continue. The variable length output codes are explained by looking at the codes that are output. The first "learned" code is output as the number 257 and the second as 258 and so on up to the table size, 4096 (crunch) or 8192 (squash). As the numbers get larger, it takes more bits to represent them, obviously, you cannot represent 4096 with 9 bits. In actuality, one never lets the table fill at all, in fact, the best compression speeds are achieved by not letting the table get over 75-80% full. This is precipitated by the method for looking entries up in the table. Remember, that the algorithm says that we must see if omega-K exists in the table. This implies that we must search the table once for each byte of the input file. The EASIEST way to search the table in by looking at the first entry for a match, and then the next and so on, until we have examined the whole table. This is called a linear probe. To see the disastrous ramifications of a linear probe for LZW, we must turn to a little mathematics. MATHEMATICAL FACT 1. The smallest codeable loop (in assembler) for the linear probe takes 6.3 micro-seconds to execute on a 20mhz 386 Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 16 computer. MATHEMATICAL FACT 2. Our sample file is 15000 bytes long. MATHEMATICAL FACT 3. The table size is 4096 entries, so we CHOOSE an average figure of 2000 locations searched for each input byte. Multiplying, loop iteration time X avg. iterations/search X bytes in file we end up with 189 seconds or just over 3 minutes to compress our 15K file. Here we have definitely delved into the unlivable. Note also, that we have not taken into account the rest of the compression code overhead, we have only looked at how much time the compressor would spend/waste searching the table. It seems there must be a better way, and indeed there is, the hash function. Our table has a finite number of entries and all we wish to do is search the table for a match, so if we had a mathematical black box where we could input a number and have it output a unique number, our searches would be made a lot faster. We input a number (omega-K) and the black box outputs another number, which is an entry number in the table. Well, we run-amuck here again. <POINTER> is a word (2 bytes) and <CHAR> is 1 byte. The number of possible combinations of the two is 65536 times 256, which is a BIG number and left as an exercise for the reader. If our black box is to spit out UNIQUE numbers, then there will be as many output numbers as there are input numbers and our string table would have to be HUGE, in fact, it couldn't be run on any computer that exists in the world today! Then what we need is a black box (hash function) that will take a huge number of input combinations and output numbers that are within our table. The implication here is that we will have several input numbers that result in the same output number. This is better known as a "hash collision". We can help minimize the number of collisions by designing our hash as a double hash, where the second hash function is a function of <CHAR> and table size. Technically, what we are doing here, is minimizing what is called "clustering", that is, the tendency of certain sets of input numbers to "cluster" around a given output number. Collisions from the first hash are called "primary clustering" and those from the second hash are called "secondary clustering". Our hash function must be designed very carefully so that it will also "probe" every location in the table, if need be. Clustering is also avoided by making the table size a prime number instead of the 4096 or 8192 called for by the bit length. The "fill limit", as discussed above, is also selected as a prime number. Prime numbers are divisible by only two numbers, 1 and the number itself. It may be intuitively observable, that this should reduce clustering. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 17 The real beauty of hashing, is that when the string table is 25% full, the hash function will have to search an average of 1.15 to 1.33 table locations to know whether or not the omega-K exists in the table. With the linear probe that we spoke of before, would be searching an average of 500 locations for each input byte, quite a difference. As the table fills, however, the performance of the hash function decreases logarithmically, thus it is favorable, from the standpoint of speed, that we limit the load factor of the string table, that is we limit the number of entries that we will actually put into the table. When the table hits this limit, we arbitrarily call the table full and issue an adaptive reset code to the output file. The hash function in IDCshell was derived from a discussion in "Data Structures in Pascal", by Reingold/Hansen, pp. 397. Aside from the maximum bit length, the other major difference between crunching and squashing, is that crunching prepacks the file BEFORE crunching it. This prepacking or repeat packing or run length encoding, is actually storage method 2 in the archives. See "packing". One final note on the codes that are developed by the compressor and sent to the output file. These codes, you remember, vary in length from 9 to 12 or 13 bits and are strung together at bit level in the output file. The most significant bits from one code are shifted left into the next code. It does get a bit messy playing with the codes this way, but real compression has as its primary cost, coding headaches. The algorithms are actually very short when fully coded, a little over 1K each. And there you have it. It seems like a bit much and is not casually understood, but if you really want to know how they work, you should get the original article and study it along with the above and hopefully, you will gain more insight into the replete world of data compression. There are many other forms of data compression that have been in use for years, bit mapping, null suppression, half-byte packing, diatomic encoding, relative encoding, statistical encoding and pattern substitution to name a few and that is not to mention all the forms of redundancy and apriori structure compression. All things considered, LZW is definitely one of the best we have. TAGGING FILES Why tag files anyway ? Tagging files started back in the CP/M world with the sweep programs. The idea is to allow performing the same operation to more than one file at a time. You would tag the files that you wanted to copy, for example, and then invoke the copy command. This is a lot faster than copying each of the files Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 18 individually. This same reasoning applies to deleting, moving and printing. There are several ways to tag files in IDCshell. SPACE bar or "T" key - These two keys perform the same operation, they tag the currently highlighted file. Actually, they toggle the tag status of the currently highlighted file, that is if it was tagged, after the toggle, it will not and vice-versa. A diamond will appear next to the file on the screen denoting it's tagged status. The number of tagged files and tagged bytes will also be displayed on the screen, just above the command window. F6 - key This key will tag all files in the current directory. F7 - key This key will invert the tag status of all files. NOTE: Also see the WILDCARD TAG section. Tagged files and archives Tagged files are one method of adding/updating files in an archive. When you change directories or drives, IDCshell will remember the files that you had tagged and where they were so that when the "A" (archive) command is invoked, you will be shown a list of these tagged files to preview/edit before performing the archive function. Also, if you tag files and then proceed directly to the preview screen via the "A" command, the tagged files will be displayed for review. See the ARC preview section. ARCHIVE PREVIEW FUNCTION The preview screen will display each path where files were tagged and the files that were tagged in that directory. The arrow, home, end, page up and page down keys will move you through the list. The "D" key will delete the currently highlighted filename from the list -OR- if the highlight bar is pointing to a path and the "D" key is pressed, all files in that path will be deleted from the list. IDCshell allows moving into and out of the preview screen at any time, without having to choose the ARC file first. This allows for a certain degree of playing around before you get Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 19 your thoughts together. You can edit the list, return to select more files for inclusion, or proceed with the arc function. Each time that you leave the files screen, all of the tagged files are saved in memory for inclusion into an ARC. This method would normally display several copies of the tagged files in a directory if you had been into and out of that same directory more than once. IDCshell will remove any duplicated directories from the saved list and keep ONLY the last one. Once you have tweaked the preview list into just what you want, you can then hit ENTER and proceed with the archive operation. The next screen asks how you want to perform the arc function. This screen is vastly different from the usual -f, -u, -a etc. commands from the other arcers. The reasoning for this is simple, How many times have you used the -u command when you really meant to -f ? IDCshell uses a much more intuitive approach, I believe. The screen is as follows. ┌─────────────────────────────────────────────┐ │ Select Archive Options │ ├───────────────────────────────────────┬─────┤ │ Add new or update existing files │ YES │ │ Update existing files only │ NO │ ├───────────────────────────────────────┼─────┤ │ Replace with newer files only │ YES │ │ Replace files regardless of date │ NO │ ├───────────────────────────────────────┼─────┤ │ Overwrite existing files │ YES │ │ Do not overwrite existing files │ NO │ └───────────────────────────────────────┴─────┘ The arrow keys are used to move from field to field up and down. The SPACE bar is used to toggle between YES and NO. Notice that if you select NO for "add new or update existing files", that the last field disappears. This is because you are telling IDCshell to update existing files only, therefore you MUST overwrite them, but you still have the option of selecting date checking with the second field. Add new or update existing files - This selection is used if you are adding new files to an existing archive and/or updating files that already exist in the archive. Existing files will be replaced only if you have chosen to do so with the second selection in the box. Update existing files only - This selection allows you to update an archive without adding any new files. Suppose that you wanted to update an archive Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 20 and you did not know all the filenames that were in the archive, but you also knew that whatever their names were, they were in a certain directory. You could tag ALL files in that directory and then choose "update existing files only" and only the files that previously existed in the archive would be updated, no new files would be added. The "extra" tagged filenames would be ignored. This selection is also subject to the setting of the "replace" option. Replace with newer files only - This option, when YES, will cause any filename in the archive to be replaced with a matching tagged filename, ONLY if the tagged filename has a more current date/time stamp. Replace files regardless of date - This option, when YES, will cause filenames in the archive to be replaced with matching tagged filenames REGARDLESS of the date/time stamp. The net result here is that NEWER files may be replaced with OLDER files. Overwrite existing files - NOTE: This option only applies if the "add new or update existing files" option is set to YES. If this option is set to YES, then new files will be added to the archive and existing files will be updated. Note also, that this option is subject to the setting of the "replace with newer files only" option. Do not overwrite existing files - If the option is set to YES, then IDCshell will ONLY ADD NEW FILES to the archive, existing files will not be disturbed. Once the options are chosen, you may hit the ENTER key and the archiving process will begin. Each file is displayed at the top of the list as it is analyzed and compressed. The ESC key can be used to abort the operation. Note that the abort does not take place until the current file is finished. FUNCTION KEYS F1 - invokes the IDC on-line help system F2 - selects the currently highlighted archive for archive operations. F3 - ANSI view function Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 21 F4 - send a copy of the file list in the main screen to the printer. F5 - shell to DOS F6 - tag all files in the current directory F7 - invert tagged status of all files F8 - specify directory sort order F9 - edit the highlighted file with your editor F10- clear all files tagged for archiving ALT-F9 - call your editor ALT-F1 - change file display mask ALT-F2 - run highlighted program ALT-F3 - display machine status screen End of file. Copr. 1987-89 Infinity Design Concepts, Inc. All rights reserved Page 22