Black Box 4

home *** CD-ROM | disk | FTP | other *** search

/ Black Box 4 / BlackBox.cdr / archiver / cpk44.arj / MANUAL.DOC < prev next >

Wrap

Text File | 1991-11-01 | 41KB | 1,073 lines

COMPACK 4.4 DOD Program Compaction Software USER MANUAL CONTENTS ________ Page Installation.........................................1 Distribution.........................................3 Getting Started......................................3 Rationale............................................3 How COMPACK works....................................4 Real Time Envronment....................................5 Some typical compression statistics..................5 COMPACK Syntax.......................................6 Command Line Options.................................6 Examples.............................................6 Hardware and Software Requirements...................7 COMPACK Error Messages...............................7 COMPACK Warning Messages.............................7 Optimizing COMPACK...................................8 Compacting BIOS Files................................9 Problem Cases........................................9 Known Anomalies......................................9 Answers to some common questions about COMPACK.......11 BuildSFX - Self Expanding Program Creator............13 BUILDSFX Syntax...................................13 BUILDSFX Hardware and Software Requirements.......13 Example Use.......................................13 Execution of the SFX..............................13 Minimizing SFX Disk Size..........................13 Sample Commercial Use Licence........................14 INSTALLATION ____________ COMPACK is supplied as a self expanding file (created by our utility program BUILDSFX). To install the software select a suitable directory as the default and execute the program CPK43.EXE. This will automatically create all the files in the current directory (overwriting any files with the same name). Be sure to read the file README. 2 August 1991 Prominence Computer Services Ltd. Strada Sottopiazzo 18, 14056 Boglietto di Costigliole (AT), Italy. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 2 Fax. +39 (11) 9574117 Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 3 DISTRIBUTION ____________ COMPACK and BUILDSFX are the copyrighted property of Prominence Computer Services Ltd., and are distributed as Shareware. As such, you are free to use and copy the software for non commercial purposes. The software may not be modified. If you find the programs generated are small and fast, then a contribution of US $25 would be appreciated. If you send $40 or more you become a registered user and will be sent the latest version of COMPACK with printed manual. If you use COMPACK or COMPACKted programs in a commercial environment or government institution registration is obligatory. Commercial distribution of COMPACKted programs requires a written agreement (not required for compaction of public domain software or shareware so long as credits are given in the documentation or program). Commercial users wishing to distribute programs compressed by COMPACK or BUILDSFX should contact Prominence Computer Services Ltd. for further details. A sample agreement is appended to this manual. In Italy COMPACK is available from Polimatica SRL, Corso Francia 233 A, 10098 Cascine Vica, Rivoli (TO). Tel 011-9597222 (3 lines). FAX +39 (11) 9574117. GETTING STARTED _______________ COMPACK transforms your DOS programs, both EXE and COM files, to load faster and take up less space on disk! You don't need to read this manual to start using COMPACK. Like all good software, COMPACK comes with help already built in. Just enter "COMPACK" at the DOS prompt for a summary of the syntax and available options. The rest of this manual is really only for reference. RATIONALE _________ Hard disk are always too small, modem bills too high, and network response time too slow. The classic solution to these problems has been to compress the data prior to transmission or backup. There are a number of archive utilities to compress files into "LBR, "ZOO", "ZIP", "ARC", "PAK", "LZH" and similar incompatible formats. These utilities work fine when the objective is archiving. But access to data, (ie running a program) requires an additional decompression utility and this adds delay and complexity. In order to hide this complexity many software distributors are obliged to create bulky installation procedures, somewhat counter-productive when the aim is to reduce media costs. In fact most archive expanders are not suitable for software distribution. Firstly the expansion program itself takes up tens of thousands of bytes of overhead. Secondly file compression is compromised by the need to maintain the archive updatable. Neither of these "features" is of any use to the poor computer user who simply wants to use his software with the minimum of expense and delay. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 4 COMPACK therefore is not an archiving program. It is a utility which simply compresses DOS programs. The compressed programs will run the same as before without the use of any extra software or hardware. This is an important advantage because it means that compressed software is portable. There is no risk that a program will not run because an essential utility is not available. Unlike other software, the expansion code does not disable interrupts for extensive periods (less than 10uS) and consequently compacted programs work correctly even in a real time environment (such as a network). Furthermore no messages are embedded in the code so foreign language programs can be compressed without problems. With few exceptions COMPACK will compress programs smaller than the very best archive utilities. As for speed, COMPACKted programs actually load faster! COMPACKted programs running on floppy disk, CD-ROM, or over networks will typically respond in half the time of their bulky predecessors. At the other extreme, on a fast RAM disk, there may be a small additional delay (perhaps up to a few tenths of a second). Of course such delay is barely perceptible and is compensated by the ability to fit even more data in the RAM disk. HOW COMPACK WORKS _________________ In simple terms, COMPACK compresses the input program and sandwiches the result between header and trailer code. Under DOS 3.X and later the input file is opened DENY_WRITE preventing any other process (including COMPACK itself or other network users) from overwriting. Note that your operating system may not offer this protection unless SHARE is loaded. The extra header and trailer code amounts to just a few hundred bytes and is responsible for recreating the exact image of the uncompressed program and passing control to it. The code to expand COM files is smaller and faster than the code to expand EXE files. The compression algorithm is based on Lempel Ziv's first proposal, a sliding window algorithm. (This algorithm is quite distinct from their second proposal, also known as the Lempel Ziv Welch algorithm.) COMPACK replaces repeated byte strings by shorter references to previous identical strings. This is called substitutional compression (as opposed to statistical compression). There is nothing very original in this, almost all modern software or hardware compressors in the past few years have used a variation on this basic method. Common variations include a second phase of statistical compression whereby frequent references are encoded in fewer bits than less frequent ones. Statistical compressors have been applied successfully, for example, in the Huffman encoding of LZH archives (hence the name) and Shannon Fano encoding of ZIP archives. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 5 Combined with the prior phase of Lempel Ziv substitutional compression, these techniques achieve excellent compression but are not suitable for compressing programs for several reasons. Firstly the expansion algorithms are relatively complex, adding a significant amount of overhead. Secondly the algorithms tend to be slow. Thirdly memory requirements tend to be high, and there simply may not be sufficient room in memory for both tables and a large program simultaneously. It would certainly be a pity if large programs, which have the most to gain from compression, could not be expanded again! COMPACK manages to avoid these disadvantages whilst actually achieving even greater compression. There is no second phase of statistical compression. Memory requirements are minimised because there are no tables. The simplified expansion code takes up just a few hundred bytes adding very little overhead to the COMPACKted program. Simplicity implies speed: the expansion code executes between 3 and 20 times faster than similar archive utilities. Disadvantages? Well, to achieve the superior compression COMPACK analyses the program to be compressed in very considerable detail, and this takes time. We feel it is worth investing this time once and recouping the speed and space savings whenever the program is run. Real Time Environment Many PC applications are finding their way into networks, multi- tasking platforms and real time applications. In these environments a program which disables interrupts for a significant amount of time is considered ill behaved. For example, disabling interrupts could cause a resident communication program to lose a character, (and probably the entire packet). If you are already having problems of this kind with programs compressed by other utilities, COMPACK may be the answer. COMPACKted programs have no additional interrupt latency. Some typical compression statistics using COMPACK 4.4 _____________________________________________________ Original Compacted Reduction Size Size % CYCLONE.COM 5632 3650 35 COPYWRIT.COM 34784 17548 50 CED.COM 59206 34524 42 TCC.EXE 179917 85388 52 TC.EXE 290249 156972 46 Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 6 COMPACK - COM and EXE File Compacter program ____________________________________________ SYNTAX ______ COMPACK \path\program.in \path\program.out [msg] [-options] Note that the filename must be unambiguous (no '*' nor '?'). The third parameter is an optional message which is appended, uncompressed, to the output program. This might be used to include a serial number and or copyright notice. Copyright notices embedded in the program code will of course be compressed, and therefore may not be visible unless displayed by the program. Command Line Options ____________________ Options always begin with the "-" character and may occur in any order, even mixed up with the other arguments. The following letter may be in either upper or lower case. -r Allows program compaction to continue even if the space saving is not significant. Normally COMPACK aborts compaction if it cannot save at least 1024 bytes. This option can be useful where real objective is to make unauthorised disassembly or modification of the program difficult rather than compression. In addition this option retains the output program even if it contains excess debug information. Use this option with care. It is possible that the COMPACKted program will not work. -o Displays the order form on the screen. (Print with PrintScrn key). -i Causes COMPACK to abort compaction if it can identify a possible optimisation of the input program. See section on Optimising COMPACK. -g Speeds up compression (by about 50%) at the expense of program size (about 1% larger). -h Most linkers produce EXE headers which are a multiple of 512 bytes so there may be considerable unused space. Using the "-h" option normal EXE program compression is bypassed and COMPACK attempts to reduce the header to a smaller multiple of 16 bytes. This is a useful technique for EXE style device drivers or for very small EXE files. Examples ________ COMPACK bigprog.exe -g fastprog.exe for %q in (*.com) do COMPACK %q d:%q -r for %q in (*.exe) do COMPACK %q d:%q "Copyright (c) Fox 1991" Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 7 Hardware and Software Requirements (Version 4.4) ________________________________________________ COMPACK.EXE program.com program.exe Free Memory 256 k Bytes 65k Bytes Program + 16k DOS 2.0 or later Any Any CPU 8088 8088 8088 COMPACK Error Messages ______________________ Exit Code Insufficient memory 3 Error writing output program 3 EXE header too large at XXXX bytes 3 Unsuitable Overlay file 3 Unsuitable HIGH loading EXE file 3 Cannot open FILENAME 3 Insufficient Saving. Program not compacted. 3 COM file > 64k Bytes 3 EXE program contains excess debug info or overlays. 3 Invalid EXE header. Relocation overlap. 3 Input and output filenames must be different 3 COMMAND.COM should not be COMPACKted 3 Invalid EXE header. Actual length less than Reported 3 All the above error messages are fatal, causing the immediate termination of the program. Note the error message regarding excess debug information. This condition is noted when the DOS file size is significantly longer than the file size registered in the EXE file header. It is possible that the excess information is an overlay or other essential information which is read from disk at execution time. This condition is considered a warning if the "r" (retain) option is specified. Naturally such programs may not function correctly when compacted so testing is essential. COMPACK Warning Messages ________________________ Warning: Multi-segment COM file. Warning: File may have been already compressed with EXEPACK. Warning: It may be possible to convert this file to COM format. Warning: No significant saving - new program created anyway. Warning: Already COMPACKted. Warning: Probable Device Driver. Warning messages are purely informatory and point out some format of the source program file which deserves attention. For example, using our COM2EXE utility to convert multi-segment COM files back into the original EXE format. Similarly other utilities may strip off debug information. These are usually included with the compiler or operating system. In all cases even greater execution speed and smaller programs should be possible. In the next section we will see how to transform these programs into a more simple and suitable format for compression. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 8 Optimising COMPACK ___________________ COMPACK obtains best results when it compresses a program in its most simple format. By using the -i command line option COMPACK attempts identify what kind of program is being compressed and returns with a DOS error code. This can be used by a batch program to simplify the program (possibly expanding it) prior to re-compacting it again with COMPACK. Exit Code Program Type 16 Already compressed by PKLITE 15 Already compressed by LZEXE 14 Probable Device Driver 13 Normal EXE 12 Convertible to COM format with EXE2BIN 11 Packed by EXEPACK 10 Multi-segment COM File 9 COMPACKted File 0 Normal COM File 3 Error In order to automate the process of converting programs into a more efficient format prior to using COMPACK you can use the batch command file REPACK.BAT. REPACK uses utility programs supplied by third parties to perform the conversion. It is invoked in the same way as COMPACK :- REPACK bigprog.exe smallprog.exe EXE programs are convertible to COM format if there are no relocatable items, if there is no stack segment, if the starting execution address is 100H and the code fits into one 64K segment. The appropriate conversion program, EXE2BIN is supplied with many DOS operating systems. EXEPACK is a utility supplied by Microsoft which attempts simple compression of an EXE file. For example it performs run length encoding, compressing repeated bytes into a more efficient format. This compression is sometimes quite effective because many programs have data areas initialised to some constant value. Of course run length encoding is only special case of COMPACK's sliding window algorithm. In short anything EXEPACK can do COMPACK should be able to do rather better! Multi-segment COM files are small EXE files which have been converted to COM format by a proprietary utility. The format is used in various system programs supplied with PC-DOS and MS-DOS (for example CHKDSK.COM). The reason for using this format is not clear because it results in larger COM files which load slower than their EXE counterparts. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 9 Compacting BIOS Files _____________________ IO.SYS and IBMBIO.COM may be sucessfully compressed by COMPACK even though strictly speaking they are neither COM nor EXE style files. COMPACK takes care of all the details without any special options. Neither MSDOS.SYS nor IBMBIO.COM may be compacted. PROBLEM CASES _____________ Not all executable files may be compressed successfully: COMPACK may (see messages) abort compression if:- - the .EXE file is an overlay. - the .EXE file is a high memory loading program. - the .COM file is larger than 65000 bytes. - the .EXE file contains overlays or debug information - the program is a device driver (eg CACHE.EXE). Executable files should be stripped of any debugging information prior to compression as this is not automatically removed, and cannot be used by a debugger in compressed form. Naturally inferior compression results by including inaccessible debugging information in a compacted file. COMPACK attempts to detect this situation and aborts compaction if the program size is longer than specified in the EXE header. However compaction continues if the -r switch is used. Programs which rely on re-reading themselves from disk or checking their own size such as COMMAND.COM and certain protected software programs may fail to execute correctly when compressed. COMPACK cannot always detect such programs automatically, so thorough testing is essential. Always maintain a backup copy of the original program. COMPACK aborts compression if COMMAND.COM is specified as the input program unless the -r switch is used. Using the /P parameter the SHELL command in CONFIG.SYS can load COMMAND.COM permanently in memory so it should never be re-read and therefore works in compacted form. Unfortunately this does not always work with some operating system and application program combinations so again, careful testing is essential. Example:- SHELL=C:\COMMAND.COM /P KNOWN ANOMALIES _______________ 1) COMPACK aborts compaction of an EXE file with more than about 8500 relocatable segment references. Fortunately very few EXE files are this large. 2) COMPACK does not update the checksum field in the .EXE header. This field is not used by DOS, and consequently there should be no ill effects under compatible operating systems. 3) OS/2 and Microsoft (tm) Windows format files are not supported. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 10 Disclaimer __________ Prominence Computer Services Ltd. hereby disclaims all warranties relating to this software, whether express or implied, including without limitation any implied warranties of merchantability or fitness for a particular purpose. Prominence Computer Services Ltd. will not be liable for any special, incidental, consequential, indirect or similar damages due to program malfunction or any other reason. In no event shall the liability of Prominence Computer Services Ltd. for any damages exceed the price paid for the licence to use the software, regardless of the form of the claim. The person using the software bears all risks as to the quality and performance of the software. COMPACK has been tested on many thousands of files with successful results. Nevertheless the possibility of program errors can never be completely eliminated. Testing is the resonsability of the user. But if you have any comments, suggestions or complaints, please contact us. We will endeavour to assist all registered users. Naturally we intend to continue development of COMPACK and feedback from users is an essential part of this process. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 11 Answers to some common questions about COMPACK ______________________________________________ Q: How is it possible that COMPACK can compress programs better than the best archiving programs? A: COMPACK understands the contents of program files and is optimised to compress programs efficiently. For example COMPACK can realise that much of the space dedicated to an EXE file header is redundant. General purpose archivers apply general purpose algorithms, paying more attention to compression time and archive updating at the expense of compressed archive size and expansion time. Q: Will a compressed program always behave like the original? A: Apart from programs which attempt to read themselves, (such as COMMAND.COM, overlay programs, programs with debug information etc.) all programs will run similarly (probably faster) because the expansion code constructs an identical program image in memory just as if the program had been loaded in memory by DOS. Naturally there has to be a little extra memory for the expansion code to run but this is only a few kbytes. Unlike some other compressed programs, COMPACKted programs may be freely renamed because they do not attempt to read themselves. Q: However fast the expansion algorithm may be, surely it takes a finite amount of time and therefore compacted programs should take longer to execute? A: Almost all compacted programs execute faster because smaller programs load faster. The code expansion rate can typically exceed 10 Mbits per second on an 80386 and this is far faster than the overall throughput of a network or mechanical disk. Q: How can I be sure that COMPACK will work with my application program? A: If your application works with any other compacter such as Microsoft's EXEPACK, it will almost certainly work with COMPACK but with much better performance. Q: What programs are the best candidates for compression by COMPACK? A: The bigger the better! Small programs are compressed successfully but there will always be fixed overhead (about 300 bytes) of the expansion code to consider. This code becomes a smaller proportion of the whole for large programs. In addition the compression itself becomes more efficient for larger programs. Q. Much of our distribution software includes data files as well as programs. How can COMPACK be used to minimise distribution media costs? Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 12 A. Use COMPACK and our BUILDSFX utility to create self expanding files. The programs created by this combination are the smallest and fastest available, way ahead of any competitive solution. Q. Will my compacted application work on non-standard platforms? A. Almost certainly. COMPACTed programs make no extra DOS calls and access no DOS nor BIOS memory structures when expanding and so should be compatible even with non DOS operating systems which recognise standard EXE and COM programs. But note that COMPACK does not support Microsoft's Windows format nor other formats which will not run under DOS. Q. Why does my compacted application not work? A. It is possible your application uses non-initialised variables. Check its execution when running in different memory configurations, when running under a debugger etc. Also check there is sufficient RAM available. As a final resort contact Prominence with sample program. Q. How much extra memory does a compacted program use? A. A COM style program uses the full 64K byte segment regardless of how big the original file was. This is a very modest requirement as today most PCs come with at least 640 kBytes of RAM. An EXE style program needs an extra 0 to 16 kBytes extra (3k bytes is typical). You can find out the exact memory requirements of EXE programs using Microsoft's EXEMOD utility. Q. Why is it better to prefer COM files over EXE files? A. COM files contain no relocation tables and consequently relocation is not included with the expansion code, so compacted COM files are slightly smaller. Because all COM files must be smaller than 64 kBytes, the expansion code is faster as it is not necessary to cope with multiple segments. Q. Why does COMPACK think that my COM file is an EXE program (and vice versa)? A. Good question. An EXE file always starts with the signature bytes "M" "Z" and a COM file does not. The filename extension is ignored by both DOS and COMPACK in determining whether or not the program is EXE style. Q. How much does a licence cost to use COMPACK on software for corporate distribution? A. Just $125 one time fee permits unrestricted distribution of programs compacted by COMPACK. No licence requirements are necessary for user groups clubs etc who distribute software on a non profit making basis. Licencing of BUILDSFX is negotiable. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK 4.4 User Manual Page 13 BUILDSFX - Self Expanding Program Creator _________________________________________ Purpose _______ BUILDSFX is a utility program to concatenate multiple data files into a single EXE style program which re-creates the data files in the current directory when executed. COMPACK is automatically invoked to compress the program providing a very efficient means of distributing all kinds of data and program files. Benchmarks show that creating files in this way is far faster than conventional self expanding files (up to 20 times faster) but file size is smaller than the best archiving programs. In fact it is usually faster to execute a BUILDSFX file than use DOS COPY command! BUILDSFX Syntax _______________ BUILDSFX input-file output-program The first parameter is an ambiguous file specification, (ie it may optionally contain '*', or '?') defining the input files to be included in the program. The second parameter is the name of the output program which is created in COMPACKted EXE format. Both may contain pathnames. In order to work correctly it is essential that COMPACK.EXE is in the current path. BUILDSFX requires DOS 2.0 or later and 256k bytes of free memory. Example Use ___________ BUILDSFX D:\TXT\*.?OC E:MAKEDOC.EXE Execution of the SFX ____________________ When executed the SFX.EXE program will expand itself in memory and proceed to create its component data files (overwriting any files with the same name) in the current directory. Obviously there must be sufficient free memory available for SFX to run so care should be taken not to create excessively large SFX programs. If necessary multiple SFX programs. The SFX returns an error code of 0 if file creation is successful, otherwise a simple message is displayed and error code > 0 is returned. These error codes can be detected by a batch file or install program. Minimising SFX disk size ________________________ COMPACK works by compressing repeated byte strings so files which are likely to contain similar strings should be grouped together in the SFX. This can be done by reordering the files in the input directory. For example, it is often appropriate to sort the input directory first by filename extension, and then by increasing file size. Copyright (C) Prominence Computer Services Ltd. 1990,1991 COMPACK COMMERCIAL USE LICENCE ______________________________ MEMORANDUM OF AGREEMENT _______________________ between COMPANY. (hereinafter called the Licensee) of COMPANY ADDRESS, of the one part and Prominence Computer Services Ltd. (hereinafter called the Licenser) whose registered office is at 7 Old King Street, Bath BA1 2JW, England, of the other part. Definitions ___________ a) 'The Product', means the Licensee's computer software, known as APPLICATION. b) 'The Utilities', means the Licenser's computer software package known as COMPACK used for file compaction, and described in the documentation whether or not incorporated in whole or in part into other software. Where appropriate in context, the phrase includes all or any modifications substantially based on or derived from the software package whether entitled COMPACK or not. NOW IT IS HEREBY AGREED BETWEEN THE PARTIES HERETO AS FOLLOWS:- _______________________________________________________________ 1) The Licenser hereby grants to the Licensee the non- transferable and non-exclusive rights use the Utilities on Product for distribution and the Licensee agrees to pay the Licenser a one time fee of US $125. This agreement does not authorise the distribution of the Utilities for profit, nor are any rights are granted until the fee has been paid. The Licensee may terminate this agreement in writing within 30 days of its execution for a full refund of any fee already paid. 2) The Licenser hereby warrants that the Utilities are original and will in no way whatever infringe upon any existing copyright. The Licensee agrees that all components of the Product modified by the Utility shall be protected by copyright, and shall exercise at least the same care and diligence to protect the Licenser's intellectual property as it does to protect its own. 3) The Licensee agrees to acknowledge in the Utilities or in their documentation that the use of COMPACK has been licensed by the Licenser. 4) The parties to this Agreement undertake to maintain the confidentiality of any communication which is specified at the time as being confidential, and undertake not to disclose such information as sales figures, billing methods, pricing, algorithms, technical specifications, program code unless such information is already made public (other than by breach of this Agreement). Copyright (C) Prominence Computer Services Ltd. 1990,1991 5) Prominence Computer Services Ltd. hereby disclaims all warranties relating to the Utilities, whether express or implied, including without limitation any implied warranties of merchantability or fitness for a particular purpose. Prominence Computer Services Ltd. will not be liable for any special, incidental, consequential, indirect or similar damages due to program malfunction or any other reason. In no event shall the liability of Prominence Computer Services Ltd. for any damages exceed the price paid for the licence to use the software, regardless of the form of the claim. The person using the software bears all risks as to the quality and performance of the software. 6) This Agreement shall be subject to and construed according to the laws of England. __________________________________________ __________________ Authorised Signature (Title and Date) COMPANY. __________________________________________ Director Authorised Signature 1 January 1991 Prominence Computer Services Ltd. Copyright (C) Prominence Computer Services Ltd. 1990,1991 Copyright (C) Prominence Computer Services Ltd. 1990,1991