Simtel MSDOS 1992 September

home *** CD-ROM | disk | FTP | other *** search

/ Simtel MSDOS 1992 September / Simtel20_Sept92.cdr / bbs / zdcs / zdcswalk.txt < prev

Wrap

Text File | 1991-12-31 | 29.5 KB | 673 lines

Zipfile Duplicate Checking System (ZDCS) Ver. 1.65 Copyright (C) 1991, Michael W. Cocke ---------------------------------------------------------------------- A Walk-Through Guide for Installing and Using ZDCS on a PCBoard BBS TABLE OF CONTENTS ----------------- HELLO!..........................................................1 OVERVIEW OF ZDCS................................................2 ZDCS OPTIONS....................................................3 Deleting Duplicate Files...................................3 Allowed Dupicates..........................................3 BBS Ads....................................................4 Pre-Testing................................................4 INSTALLATION....................................................4 STEP 1 - Set up the config file.................................5 Line 1................................................5 Line 2................................................5 Line 3................................................5 Line 4................................................5 Line 5................................................6 Line 6................................................6 Line 7................................................6 Line 8................................................6 STEP 2 - Create the initial database............................7 STEP 3 - Create the bbs ads database............................8 STEP 4 - Create the list of allowed duplicates..................9 STEP 5 - Set up the upload file check...........................9 PRE-TESTING....................................................11 THAT'S ALL, FOLKS!.............................................12 HELLO! ------ ZDCS is a shareware set of utilities intended to help a PCBoard sysop deal with the problem of duplicate files, whether those files are already on the bbs or are being uploaded by a caller. It provides specific support for looking inside zipfiles (including PKZIP version 1.93) and self-extracting files made with PKZIP (*.SFX). Since ZIP files are by far more common that SFX files, and since they are both treated the same way by ZDCS, we'll use the term "zipfile" to refer to both of them. ZDCS also provides support for accepting unzipped GIFs. ZDCS thinks of such a GIF file as a poor zipfile with only one file in it. You'll find more detailed information on GIF files and others in the reference manual, ZDCS-REF.TXT. This walk-through will guide you in getting ZDCS up and running on your bbs. The reference manual, ZDCS-REF.TXT, has more detail on all sorts of things (probably more than you will want or need), but it's arranged to be the sort of file where you look things up when you want more information, not something you read all the way through. On the other hand, this walk-through is a friendly guide that will take you by the hand and show you how easy it is to install ZDCS. Don't be put off by the length of this file. ZDCS is quite simple to install. We've added lots of extra words to explain the process and your choices for how you want to run ZDCS. OVERVIEW OF ZDCS ---------------- There are two basic functional parts to ZDCS: checking an existing collection of files for duplicates, and checking newcomer files against the existing collection. It's set up this way to permit use on a dynamic collection like a bbs file system, but it can also be used to look for duplicates on other systems as well, such as a shareware CD-ROM or even multiple directories on your hard drive system. However, the focus is on bbs systems and that's how the documentation is written. To check an existing collection of files, you have to create a ZDCS database for that collection. Then you can use the duplicate report generator to tell you which files are duplicates and where they are found. A little housecleaning based on this report can weed out duplicates and recover hard drive storage space, something no one ever has enough of. When new files are added to the collection, you need to be able to do two things with them: add the files to the database, and find out if there are any duplicates among them. The most common addition of new files is via uploads to the bbs. ZDCS will enable you to process an upload as soon as it is received, and tells both you and your caller about duplicate files in that upload. It can be set to accept or decline an upload based on the percentage of dupes it finds. It also adds any new files to the ZDCS database to keep that up to date. There's a bit more, of course. ZDCS can be told to decline an upload, to automatically remove duplicate files, to delete those pesky little bbs ads from uploads, and to recognize "allowed duplicates" - or any combination of the above. There's even a pre-test capability that lets callers find out ahead of time whether or not their intended upload duplicates files already on your board. Installing ZDCS to work on your bbs is relatively simple. The hardest part may be deciding which options you want to enable. Let's look at those first. ZDCS OPTIONS ------------ Running ZDCS without any of the extra options enabled gives you a basic duplicate checking system that will not delete any files automatically. It will still tell you about all duplicate files, but it won't be able to distinguish between files you always want to delete (like bbs ads), files that duplicate exactly ones already on your board, and files that are duplicates but still important to keep (like VALIDATE.COM from Macafee's SCAN program). You can try out the different options and change your mind about which ones you want to use without re-installing ZDCS. Deleting Duplicate Files ------------------------ When a new zipfile is uploaded to the board, ZDCS checks it against the existing database to see if there is any duplication of files. You can permit ZDCS to not only flag the duplicates but also to delete them from inside the zipfile, leaving the rest of the upload intact. This deletion feature does not operate on a GIF file. This is a good time to point out one consequence of deleting all dupes. Some shareware authors issue updates that consist of both new files (executables, perhaps) and unchanged files (registration information, maybe). If you enable the deletion of all duplicate files in an uploaded zipfile, you can lose some of the files that belong in the author's package. This brings us to the concept of allowed duplicates. Allowed Duplicates ------------------ You can choose to tell ZDCS that certain files are allowed duplicates. When you create a list of allowed duplicates, ZDCS knows not to treat these files as dupes no matter how often they appear. You can add new files to the list or delete old ones with any text editor. Why would you want to have allowed duplicates? There are some files that reappear frequently as part of shareware or freeware packages, such as OMBUDSMN.ASP (found in ASP-ware), or VALIDATE.DOC and VALIDATE.COM (from Macafee's SCAN program). Especially in a case like Macafee's where new versions of the program come out frequently, each with certain standard files included, it would be useful to "recognize" these duplicate files as being acceptable. There are two places where enabling the allowed duplicates option makes a difference. It prevents these files from being deleted if you have selected the option to delete duplicate files. It also does not figure these files into the calculation for what percentage of an uploaded zipfile is duplicates. This feature does work for GIF files. If a GIF is included among the allowed duplicates, then a repeat upload of the same GIF will not be flagged as a duplicate, and the upload will be accepted. BBS Ads ------- There are some specific files that you may want to delete every time you run across them. The prime example of this would be those rude and annoying bbs ads from you-know-who that a few boards feel it necessary to add to every zipfile they carry. You might also have other problem files that you want to keep off your board, like some of the pyramid scheme get-rich-quick files that circulate from time to time. You can tell ZDCS to recognize these nuisances by creating a bbs ads database. (This is always referred to as the "bbs ads database" to distinguish it from the main ZDCS database, which is bigger and not the least bit optional.) When you run the upload file checker, ZDCS will flag any files it finds that match the ones listed in the bbs ads database. If you want to do more that just flag these pests, you can also tell ZDCS to delete all bbs ads automatically. Of course, you can include new bbs ads as they are perpetrated. This option is completely independent of the option to delete duplicate files, so you don't take a chance on removing authors' unchanged files from newer shareware versions. The bbs ads option does not operate on a GIF file. Pre-Testing ----------- You can decide whenever you like to allow or disallow pre-testing. Since it's very easy to enable and won't affect the installation of ZDCS at all, we've covered it in a separate section at the end of this walk-through. After you've gotten ZDCS installed on your bbs, you can take your time about making the decision to pre-test or not to pre- test. INSTALLATION ------------ There are five basic steps to installing ZDCS to work with the bbs: 1. Setting up the configuration file. 2. Creating the initial database. 3. Creating the bbs ads database (optional). 4. Creating the list of allowed duplicates (optional). 5. Setting up the check for uploaded duplicates. None of the steps is at all difficult. We'll cover them one by one. STEP 1 - Set up the config file ------------------------------- The first step in the installation is to create the ZDCS.CFG file for your configuration. This is a flat ASCII text file containing eight short lines. A sample configuration file is included in this package, so you might want to look at that. It takes more time to describe the file than it does to write one from scratch. Put the finished configuration file into the same directory that contains the executable ZDCS files, unless you're running DOS 2.x. If that's the case, either upgrade to at least DOS 3.x or go to the ZDCS reference manual ZDCS-REF.TXT and read the answer there. Line 1 ------ This line is the complete drive, path and filename of an ASCII text file. This is a file that you create listing all the pathnames, one on each line, that contain the zipfiles / GIFs to be included in the database. To process a new collection of files into the ZDCS database, like those on a CD-ROM, just change either this line or the contents of the file it points to. There is no upper limit on the number of pathnames that can be processed. But make sure that you've included the trailing backslash for each pathname. Line 2 ------ This line is the drive and pathname where you want the finished ZDCS database to be located. It makes no difference if you include the trailing backslash here or not. Line 3 ------ This line is either the letter "Y" or the letter "N". It controls whether you want ZDCS to delete all duplicate files from an upload (Y) or just flag them and leave them intact (N). Line 4 ------ This line is an integer - that's a whole number, no decimals - between 0 and 100. It sets the maximum percentage of dupes that your bbs will accept in an upload. ZDCS will calculate the actual percentage of duplicates in the upload and compare it to your maximum percentage. If the actual percentage is lower, the upload is accepted. If the actual percentage is equal to or higher than the maximum you specified, the upload is declined. (The PCBoard code takes care of moving these declined files into your private upload directory, where you can review them.) Setting the percentage to 100 effectively bypasses this filter, since it permits a duplicated GIF or a zipfile with nothing but duplicates to pass. At the other extreme, setting the percentage to 0 effectively requires that the uploaded GIFs and zipfiles have no duplicates at all. Line 5 ------ This line is the complete drive, path and filename you want ZDCS to use for the log file created by the upload file checker ZDCSFC. Here is where you'll find the messages telling you which files are duplicates, bbs ads, allowed duplicates, or absolutely new and fresh. Line 6 ------ This line is either the letter "Y" or the letter "N". It controls the switch to tell ZDCS whether to delete bbs ads (Y) in an uploaded zipfile or to just flag them (N). If you've decided not to enable any checking for bbs ads at all, just set this to N. Line 7 ------ This line is where you can express from 1 to 72 characters' worth of creativity. Whatever text you write here will be displayed to the caller who has just uploaded a file that ZDCS has declined. A boring but informative possibility is (without the quotes) "Too many duplicate files - upload must be reviewed by sysop." If you don't want to display any message to the caller, you can place one or more blank spaces on this line. Just don't leave the line completely blank! The PCBoard @codes are supported here. Line 8 ------ This line consists of the single letter "Y" or "N". It controls whether ZDCS displays the one line "registered to" message after the board receives an upload (Y) or turns off the display of this message (N). Either way, the caller still sees the file by file breakdown of the upload and the status (duplicate, bbs ads, etc.) of each file. This line is only recognized by the registered version of ZDCS. It has no effect on the three line message displayed by the unregistered version. It is also the only line of the configuration file that you can forget to include without causing major problems. If the line is missing, ZDCS defaults to (Y) and displays the message. STEP 2 - Create the initial database ------------------------------------ The next step after creating the configuration file is to create the initial database of CRC32 values. To do this, you simply run the database build program ZDCSDB.EXE. Assuming that you have created the configuration file properly, there is nothing more to be done until this program finishes processing the files. While ZDCSDB is running, the display points out that you may press the F10 key at any time to abort. This is the only safe way to abort the database build! Ignore this warning and the chances are good that you will be rewarded with lost and/or cross-linked clusters on your hard disk. (This has been an official warning.) There will also be other useful and esoteric bits of information on the screen, but you can read more about those in the reference manual ZDCS-REF.TXT. The one that will probably interest you the most is on the very last line. The end time appears here after the database build is complete. Usually you will be using ZDCSDB on a collection of files that have already passed a file integrity checker. If for some reason this is not the case, you can still use ZDCSDB on those files by making use of the T (for Test) switch. This slows the processing down tremendously. It's really a better solution to use a file integrity checker to handle this duty, but ZDCSDB T is a workable alternative if you need it. What happens when you use the "T"? ZDCS sends out to PKZIP to come in and test each zipfile. Any one that is damaged is marked and not processed by the database builder. GIF files are passed over by PKZIP, so they receive no file integrity checking at all. The GIFs are still processed by ZDCS to add them to the database. There is a log file called ZDCS-DBB.LOG created by the database build operation. This is where you would find messages about corrupt zipfiles ZDCS encounters while trying to build the database, for example. If you have any problems while running ZDCSDB, look in this log file for help in understanding what happened. It is entirely likely that when you first create the initial database you will already have some duplicate files in your collection of zipfiles and GIFs. To find out about them, use the database report generator ZDCSDR to generate a flat ASCII text file called ZDCS- DUP.LST. This will create the list of all duplicate files in the database, including the name and CRC32 of the duplicated file and the identity with full drive and pathname of the zipfile or GIF containing the dupe. The format of the ZDCS-DUP.LST file is the standard comma- separated variable to make it easy to import this file into a database or parse it into a .BAT file. When you run ZDCSDR, it asks you whether you want the results sorted by the CRC32, the individual file name, or the name of the zipfile or GIF containing the dupe. This sort is called the Wichita sort for very obscure historical reasons. (The culprits know who they are.) Note that no duplicate files are deleted by ZDCSDR when you create the initial database. The list of duplicates, ZDCS-DUP.LST, can be used by the sysop to remove any duplicate files in the system. After the sysop has cleaned up the file system and removed any duplicate files, it is possible to purge duplicate entries from the CRC32 database in order to reduce its size. This is covered in the reference manual ZDCS-REF.TXT. STEP 3 - Create the bbs ads database ------------------------------------ This step is optional. You need it only if you want to flag and/or delete bbs ads from uploaded zipfiles. If you want to complete the basic installation and do this step at a later date, that's fine too. ZDCS doesn't quite have the intelligence to recognize a bbs ad unless you tell it which files to look for. To do this, you have to create the bbs ads database. This consists of a single file, ZDCS-BBA,NDX, which will be located in the same directory as the rest of the ZDCS files. The easiest way to do this is to first collect all those nasty ads together and zip them up into one zipfile. Use whatever name you like for the ZIP; this example is going to call it BBS-ADS.ZIP. Then run the program ZDCSBA.EXE from the directory containing all the ZDCS files. The syntax (7% in NJ) is: ZDCSBA BBS-ADS.ZIP (If you have used a different name for your ZIP collection of bbs ads, just use that name in place of BBS-ADS.ZIP.) The program will create the database files and you will be ready to flag or delete all that free advertising. Whether or not you delete the bbs ads is controlled by line 6 of the configuration file. You did read the section on setting up the configuration file, right? If you want to create a new bbs ads database in the future, just delete the old database file (ZDCS-BBA.NDX) and follow these same steps to create the new one. If you don't delete the old database, then the new ads will be added to the old ones in the database, which is an easy way to add new bbs ads. To make it even easier to include single ads as they come on the market, you can also run this program on an individual bbs ad, whether it is zipped or unzipped. So, if a new bbs ad named HAPPY.BBS shows up, you can issue the command: ZDCSBA HAPPY.BBS This will add HAPPY.BBS directly to the bbs ads database. You can even include a bbs ad in this database without having the original file on hand. All you need is the CRC32 value for the bbs ad. In the command above, you would replace HAPPY.BBS with a dollar sign $ followed immediately by the eight character CRC32 value. STEP 4 - Create the list of allowed duplicates ---------------------------------------------- This step is also optional. You need to do this only if you want ZDCS to recognize certain files as allowed duplicates. Whether or not this option is enabled is controlled by the presence or absence of an ASCII text file called ZDCS.ADN in the ZDCS directory on your system. This file is one that you create with any text editor to list all the files, one per line, that are allowed duplicates. You can designate an allowed duplicate by either its file name or its CRC32. To specify an allowed duplicate by name, just type the dollar sign $ followed immediately by the name of the file with its extension. This preserves a file with a distinctive name (like OMBUDSMN.ASP) even if it undergoes some revisions. To specify an allowed file by its CRC32 value, type the pound sign # followed by the CRC32 for the file. If you don't know the CRC32 offhand, you can get this information from PKZIP. Issuing the PKZIP - V command on any zipfile gives you the CRC32 values for each individual file inside the zipfile. The ZDCS.ADN file may be up to 256 lines long, but must contain no blank lines and no blank spaces. STEP 5 - Set up the upload file check ------------------------------------- Almost done! Now that you have created the ZDCS duplicate file database, all you need to do is get the bbs to check all uploaded zipfiles and GIFs from now on. This will be done by processing the uploaded zipfiles and GIFs with the upload file checker ZDCSFC as the files are received. ZDCSFC.EXE is the real-time upload file checker. If the uploaded file was a GIF, it first calculates the CRC32 for it. Then ZDCSFC compares the CRC32 of the uploaded zipfile or GIF against the ZDCS database and the bbs ads database (if you've created the bbs ads database, that is). The results of this comparison are displayed file by file for the caller and logged into the ZDCSFC log file. (Remember back on line 5 of the configuration file, when you specified the pathname and filename for this log?) ZDCSFC calculates the actual percentage of duplicate files in the upload. Since a GIF is a single file, it will either be 0% (not a dupe) or 100% (a dupe). For zipfiles, this actual percentage can vary anywhere between 0 and 100. ZDCSFC compares this actual percentage against the maximum percentage you set in line 4 of the configuration file. (Boy, that configuration file is a handly little thing.) If the actual percentage is lower than the maximum, the upload is accepted. If the actual percentage is equal to or higher than the maximum you specified, the upload is declined. The file is never lost or dumped! PCBoard moves these declined files into your private upload directory, where you can review them. If you want to bypass this filter, set the percentage to 100. This permits a duplicated GIF or a zipfile with nothing but duplicates to pass the filter and never be declined. At the other extreme, you can set the percentage to 0. This effectively requires that the uploaded GIFs and zipfiles have no duplicates at all, or they will be declined. ZDCSFC also updates the duplicate files database with the CRC32s of all uploads. You do not have to do anything special to include this information in the database for comparisons with future uploads. ZDCSFC does not modify the bbs ads database at all. It's still not smart enough to recognize a bbs ad until you've pointed it out first - but it does remember them next time it sees them. To process the new uploads with ZDCSFC, it must be called by the PCBTEST.BAT file of PCBoard or by your file integrity checker. We are actively pursuing discussions with makers of file integrity checkers to provide a seamless integration with ZDCS. EXZTEST already has such a feature in version 2.x, which calls ZDCSFC directly and feeds it the necessary information to process uploads. To understand how to call ZDCSFC with the older (1.x) and newer (2.x) versions of EXZTEST, take a look at the sample files PCBTEST.BAT and PCBTEST.ALT, respectively. They are included in this package. For more information on the integration available in EXZTEST 2.x, please see the documentation for EXZTEST. To use ZDCS with the file integrity checker of your choice (even if it's just PKZIP -T), there are a couple of things you need to include in the PCBTEST.BAT file. First, it's highly recommended that you include the following three lines to clean out old copies of these files left over from processing other uploads: @IF EXIST PCBFAIL.TXT DEL PCBFAIL.TXT @IF EXIST PCBPASS.TXT DEL PCBPASS.TXT @IF EXIST ZDCS-DEL.LST DEL ZDCS-DEL.LST Second, call your integrity file checker to process the upload. (Your syntax may vary.) Third, call ZDCSFC to check the upload. The appropriate command is: ZDCSFC %1 %2 %3 If you are running a version of PCBoard older than the May 22, 1991 version 14.5A, PCBoard will provide only two parameters. Since the third one is needed for pre-testing, that means you must disable the pre-testing feature. Even if you don't want to use the pre-testing right now, it is still strongly recommended that you leave the third parameter in place. It does no harm and could save you some grief if you change your mind in the future. Other than that, ZDCS should be OK on those older versions. At the end of processing by ZDCSFC, you either have a new file called ZDCS-DEL.LST or you don't. This file contains the list of individual files within an uploaded zipfile that are targeted for deletion. If you haven't enabled any deletion of either duplicate files or bbs ads, this file won't be created. Even if you have enabled one or both of these deletions, the file still won't be created unless the upload that was just processed had something to be deleted. Fourth, skip to the end if there are no files to be deleted: IF NOT EXIST ZDCS-DEL.LST GOTO END Fifth, perform the deletion of files specified by ZDCSFC: PKZIP -D %1 @ZDCS-DEL.LST Note that the actual deletion is done by PKZIP. This is also the only time an existing AV stamp on an upload is affected by ZDCS. Sixth, don't forget to include the end: :END This concludes our tour of PCBTEST.BAT. In addition to creating the required PCBPASS.TXT and PCBFAIL.TXT files, ZDCSFC also sets the DOS error level when it exits. These levels are explained in the reference manual ZDCS-REF.TXT. PRE-TESTING ----------- Callers have been enthusiastic about duplicate checking on a bbs file system - until they discover that the huge file they've just finished sending your bbs at 2400 baud is a set of complete duplicates. Many of them asked for a way to pre-test an upload before actually transmitting the entire file. ZDCS gives them this capability without requiring any ongoing handholding, explanations or help from the sysop. Pre-testing is most valuable when it's simple enough that callers actually use it. ZDCS pre-testing requires nothing that might tax a relatively novice uploader's skills. There are no special files to download, no complicated operations to get right, no arcane rituals to perform. All that your callers need is PKZIP and enough skill to upload a file. There are four simple things for you to do so that ZDCS pre-testing runs smoothly on your bbs. 1. Make sure that your board code is no earlier than PCBoard version 14.5A from May 22, 1991. This and all subsequent versions of PCBoard support ZDCS pre-testing. 2. Confirm that the PCBTEST.BAT file contains the following line: ZDCSFC %1 %2 %3 You already set up the PCBTEST.BAT file as part of the ZDCS installation. Just check it one more time, OK? If you've inadvertently left out the third parameter, the pre-test will work just fine for the first caller and will barf all over subsequent callers, who will come complaining to you that their ZDCSTEST.CHK is being called a duplicate! 3. Make sure that *.CHK files may be uploaded to your bbs. This is done via the UPSEC file, part of PCBoard. If you don't permit callers to upload files with the .CHK extension, you've effectively prevented them from using the pre-test feature. 4. When you're ready, post the sample bulletin included in this release of ZDCS to explain to callers what the pre-test feature is all about. The bulletin holds your callers' hands through the whole process, which should take some of the load off you. If your bbs permits the uploading of SFX or GIF files, you might want to add those initials where you see ZIP in the bulletin. A ZIP- only bbs can use the canned bulletin right from the package. If you track your callers' uploads and downloads, you might like to know that the ZDCSTEST.CHK will not count as an upload. The bulletin mentions this little fact just in case some of your callers are a bit *too* enthusiastic about collecting upload credits. THAT'S ALL, FOLKS! ------------------ That's all there is to it, and there ain't no more. ZDCS is fully installed on your system. If you want to fiddle with the options, go right ahead. Most of them can be turned on or off with no more than a line change in the configuration file. If you have more questions or are just curious about parts of ZDCS, take a look at the reference manual ZDCS-REF.TXT. It has more excruciating and technical details than this walk-through does. You will also find information about registration (ZDCS is reasonably priced shareware - $25.00), support, access to free utilities and accessories and a host of other goodies in the reference manual - even how to reach us with comments and suggestions.