The SRE-Filter For GoServe Web Server
	Users guide

SRE-Filter is a HTTP (Web) Server for OS/2. Written in REXX, SRE-FILTER is designed to run under the IBM EWS GoServe program.

SRE-Filter is designed to be a relatively full-featured package for non-cutting edge, small-to-medium load sites being maintained by non-professionals

• Support for multiple hosts
• Remotely configurable
• Common-log auditing
• Caching support for NCSA HTTPD-style server side includes
• Support for server-side-processing using SRE-Filter add-ons and CGI-BIN scripts
• Bundled accessories include a text-file search engine, and a caching directory display processor.

Bored already? You may find the FAQ more informative, the manual more immediately useful, and the list of documentation files a useful index.
Not bored? You can proceed linearly through this documentation, or you can use the table of contents to jump to the desired section.

Please be aware that this guide strikes a balance between brevity and thoroughness. There will be features not mentioned. So repeat after me: the manual, the discussion of initialization paramters, and the list of documentation files are the places to go for more detailed discussion.

Are you looking for a quick start?

Let me add one proviso: if you want to use server side includes, you should carefully read the relevant section of the SRE-Filter manual. They can do a lot for you...

First, have you installed GoServe? I'll assume that you have.
Second do you have the most recent copy of SRE-Filter (if you have a slow connection, you might need to download SRE-Filter in pieces)? Assuming that you have, you should create (or clear out) a temporary directory, and UNZIP the (possibly several) SRE-Filter distribution files.

1. Look at the READ.ME file that you just unzipped -- it outlines the installation steps (more impatient souls can skip this file with little ill effect).
2. Ready to go? Run the INSTALL program from an OS/2 prompt (be sure you are in the temporary directory, or you might end up running somebody else's installation program!).
3. INSTALL is self explanatory. It does require that you enter the GoServe working directory and the GoServe data directory (see the GoServe documentation, or the SRE-Filter manual, for details).
4. INSTALL also let's you set a few parameters, and will ask if you want to add a SUPERUSER entry to SRE-Filter's username database. We especially recommend adding the SUPERUSER entry, without it you may have a hard time accessing SRE-Filter's configurator!

Some Basic Terminology

URL

Universal Resource Locator (URL) is a scheme for specifying Internet resources using a single line of printable ASCII characters. A URL should contain a protocol, domain name, port number (optional), the location of the resource, and an (optional) option list (following a ?).
Examples:

http://your.server.net/dir1/sample.htm

http://pet.store.com/prices?type=mammals&class=retail

Selector (also referred to as the request selector )

The selector is the location of the resource on the server -- it's the part of the URL after the domain name. Examples (assuming the above URLs):

dir1/sample.htm

prices?type=mammal&class=retail

In many cases, the connection between the selector and a server resource it refers to is simple (such as when the selector is a filename relative to the GoServe data directory). In other cases, it's completely virtual (as when the resource consists of the output of a program that's run using information provided by the client).

Selector-specific (usually written as SEL-specific )

SEL-specific refers to SRE-Filter configuration information that is specific to a selector -- such as resource privileges, or a virtual directory.

In much the same way that OS/2 extended attributes contain additional information about a file, SRE-Filter's SEL-specific information contain additional information about a selector ; or, to be bit more precise, about the server resource identified by the selector (needless to say, this information is only used by SRE-Filter).

The easiest way to add (and remove) hosts is through the Multiple Hosts section of the simple-mode configurator. You may also want to examine the description of the HOSTS parameter in INITFILT.DOC, and the discussion of hosts in the SRE-Filter manual.

Return to table of contents.

Since many sites will contain a continuum of private and public resources, this coarse level of control will often be inappropriate. SRE-Filter offers two finer methods of access controls: SEL-specific access controls, and the HTACCESS method.

SEL-specific access controls	Selector-specific access (aka SEL-specific access controls ) are based on a comparison between the request selector (the request selector is part of the http request string that the client sends to the server... ... it's the URL minus http://x.y.z/) and a a list of SEL-specific access controls. This list (which may contain wildcarded entries) dictates who can access the various resources (files, etc.) on your site. Given a match, SRE-Filter will then compare the list of resource privileges assigned to the selector to the client privileges granted to the client. In general, the client must have at least one client privilege that's also in the resource privilege list (though you can specify more complicated conditions). If no such matching privilege exists, SRE-Filter returns an unauthorized access response. This response can be a generic response, a customized-for-your-site response, or a customized-for-this- selector response. For a more detailed description of SRE-Filter's various access control mechanisms, see the the SRE-Filter FAQ	The Access Controls section of the simple-mode configurator lets you enable access controls, as well as choose between the generic and customized unauthorized access response. You can also use the simple-mode configurator to add and remove entries in the list of SEL-specific access controls. Note that when you use the simple-mode configurator to create a selector specific access control , you will be asked to provide the selector (with optional wildcards) a list of resource privileges, and an optional permission list. You should use the Modify SEL-specific Access Controls section of the intermediate mode configurator to: specify a selector-specific access-failure response (for details, see the description of ACCESS_FAIL_FILE in INITFILT.DOC), specify more complicated access-control conditions, grant SEL-specific permissions. specify an advanced options file Client privileges are granted when you set up user entries. You may also want to use the intermediate mode configurator to modify the INHOUSEIPS_PRIVS and PUBLIC_PRIVS parameters. If you want to get fancy, see ADDPRIVS.DOC for details on how you can specify dynamic client privileges.
The HTACCESS method	The HTACCESS method (which is something of a standard) uses special HTACCESS files located in the directory (and in the parent directories) of the requested file. This (or these) HTACCESS files contain information on who has access rights to files in the directory (and in child directories).	HTACCESS files can be created using your favorite text-editor. You might want to read some documentation on the HTACCESS method of access control. Alternatively, you can use the intermediate configurator's Modify HTACCESS Contols options to change HTACCESS files.

The choice between the SEL-specific and HTACCESS methods is a matter of taste and convenience -- controlling directories is a sure way to prevent access to files, but controlling selectors is more flexible (if your physical directory structure changes more frequently then your pages). In fact, these two methods (SEL-specific and directory-specific) can be used jointly, which implies multiple checks. But be careful when using both methods, it is possible to get stuck in authorization loops!

Since SEL-specific controls are native to SRE-Filter, they will tend to be faster (and better tested) then the HTACCESS method. Therefore, our recommendation is to use the SEL-specific method.

Return to table of contents.

Return to table of contents.

Return to table of contents.

Return to table of contents.

Return to table of contents.

Optimization Hints

That said, a lot has been done to improve throughput. In particular; the extensive use of macrospace, the use of independent threads to handle much of the mundane processing, and the incorporation of an SSI-cache help keep SRE-Filter's performance respectable. Still, it would be nice if it were faster.

Well, if you are willing to work on it, that's actually quite a bit you can do to improve it's performance. The following outlines some of the things you can do. If you want more specific information, the best place to go is the first section of INITFILT.DOC -- it contains a number of optimzation hints.

Action	Description
The GoServe Cache	By far the best optimization strategy is to take advantage of GoServe's cache (look at the Options-General tab under GoServe). Unfortunately, there is a potential drawback -- anything that's cached will be free of access controls (since GoServe sends back cached items without fussin' around). Therefore, if you have any security on your site, SRE-Filter will disable the cache. To work around this problem, you can use the CACHE access permission, or the PUBLIC_URLS public area identifiers.
The SREFQUIK variant of SRE-Filter	For many requests you won't need all the bells and whistles built into SRE-Filter. Recognizing this, SRE-Filter allows you to call a small/simple filter. This filter (SREFQUIK) will process a select set of requests, and pass the one's it can't handle to the standard version of SRE-Filter. In particular, SREFQUIK can handle "cached" requests, and it can handle literal PUBLIC_URLS. The SREFQUIK documentation thoroughly describes the use SREFQUIK. You should also read the description of PUBLIC_URLS in INITFILT.DOC.
Suppressing unneeded features	If you know you aren't going to use one of SRE-Filter's features; you can and should instruct SRE-Filter not to use it. In general, there are two ways of doing this: across the board, or on a selector specific basis. For example, you can turn off Virtual Directory lookup for everyone (by setting VIRTUAL_FILE=' '), you can use the NO_VIRTUAL permission to suppress virtual directory lookup on a case by case ( selector by selector ) basis, or you can set up a literal PUBLIC_URL. Another example is the use of SHTML files to indicate "SSI-capable" documents; normal HTML files will not be subject to SSI compilation (a time consuming action).

Return to table of contents.

• Multiple hosts (either unique IP addresses, or multiple aliases per IP address) are supported.

  Seperate data directories, aliases, etc. can be specified for each supported host

• Logon requirements may be required of no one, everyone, or the general public.
  • A set of public resources can be specifed, which will be available to all clients.
  • OWNERS (with SUPERUSER privileges) and IN-HOUSE users can be automatically detected (using their numeric IP address), with logon requirements waived.
  • A set of "unallowed" numeric IP addresses can be specified.
  • Wildcards can be used in the list of IN-HOUSE and "unallowed" numeric IP addresses
• Several tools for resolving and redirecting requests are provided
  • A "directory specific" document can be sent when a request selector specifying a directory, but no file is name, is recieved
  • Instead of a default document; a "directory listing" can be produced, with file descriptions generated for all plain, html, and .ZIP files. These directory listings can be cached, and used for subsequent requests.
  • ALIASes can be used to redirect requests for moved documents, to resolve commonly occuring "misspelled requests", and to transfer specific files from anywhere on your server.
  • A customized "not found" document (complete with suggested link) can be sent when the requested document is missing.
  • The ~ shorthand for the "home directory" is supported. In addition, ~ can point to www-specific subdirectories of the "home directory"
  • Transfer of files from outside the "default data directory" is easily supported through the use of "virtual directories".
  • Transfer, with optional processing, of files from remote servers (without using redirection) is supported through the use of "remote" virtual directories.
• A variety of server-side includes are easily achieved
  • A special caching algorithim selectively caches documents that contain server side includes
  • Headers and Footers can be included in all HTML documents
  • Recursively processed keyphrases, that dictate server side includes, can be placed in your HTML documents
    • Dynamic and Static String REPLACEments
    • Inclusion of files (from anywhere on your system)
    • Inclusion of OPTIONs that appear in the request selector (information after a ?)
    • Execution of REXX-code blocks, and insertion of the results
    • SELECTive excludes of HTML code blocks
  • The NSCA HTTPD server side include syntax is fully implemented
• Standard server features include:
  NCSA style IMAGEMAPS are supported, and easily requested.
  Searchable indices on text files are easy to setup
  A flexible mechanism for controlling access to explicit selectors (and the files they correspond to); including assignation of realms on a SEL-specific basis.
  Support for the HTACCESS method of access control
  Several mechanisms for recording hits
  Several forms of redirection can be specified
  A simple form of load balancing is supported
  Support for "common-log" format audit files (as well as browser and referer log files)
• Special SRE-Filter built-in features include:
  A file display and transfer facility with easy directory traversal
  A simple but powerful "text file" search facility
  Dynamic assignation of "short-term" privileges to clients
  Tracking of recent hits to prevent recording of repetitive requests
  Optional limits on frequency of unsuccesful "logon" attempts, with options for custom response files in the event of "logon" or "access" denial.
  Implementation of "byte range retrieval" (which is used by Adobe Acrobat to access subsections of documents)
  Reponse headers can be automatially generated, using LINK and META HTTP-EQUIV elements in the <HEAD> portion of the document.
• Several file upload methods are supported
  Client directed retrieval of files from other HTTP servers
  Browsers that support the type=file FORM attribute (such as NetScape 2.01), can upload files from their own machines.
  The PUT verb, with byte ranges, can be permitted on a per selector (or per "set of selectors") basis
  The DELETE method can be permitted on a per selector (or per "set of selectors") basis
• User written server side processing (gateway programs) options include:
  Support for SRE-Filter add-ons (REXX routines that return results directly to the client).
  CGI-BIN scripts (both REXX and binary executables, and PERL scripts if you have a PERL interpreter installed) are supported (using procedures adapted from the GoHTTP package).
• A "pre-filter" can be called prior to invoking SRE-Filter
  SRE-Filter comes with a special pre-filter that implements the the GoRemote package.
• A "post-filter" can be called after SRE-Filter has responded to the request.
  SRE-Filter comes with a special post-filter that uses socket calls, to an SMTP server, to e-mail "event specific" alerts.

SRE-Filter, and related products, are to be used at your own risk; the authors assume no liability for any damage that may be caused by the use or misuse of these software products.

SRE-Filter is copyright by Daniel Hellerstein, and hereby placed in the the public domain. You may use is it any manner you deem fit, up to and including use of the code in your own software.

  Copyright 1996,1997 by Daniel Hellerstein.
  Permission to use this program  for any purpose is hereby granted without fee,
  provided that the author's name not be used in advertising or publicity
  pertaining to distribution of the software without specific written
  prior permision.
  With some exception, this includes the right to subset and reuse the code,
  with proper attribution. The exceptions are two fold:
     1)  Portions of the code adapted from other author's work
        (these are noted where appropriate); you'll need to contact these other
        authors for appropriate permissions.
    2)  SRE-Filter uses Quercus System's REXXLIB procedure library.  The
        license for REXXLIB gives the author the right to distribute REXXLIB
        without charge.  This right may NOT extend to redistributors.
        Please contact Quercus Systems for details.
        SRE-Filter also uses Info-Zip's UNZIPAPI.  Although this is freely
        available software, you may wish to contact Info-Zip for details.
  Furthermore you may also charge a reasonable re-distribution fee for
  SRE-Filter; with the understanding that this does not remove the
  work from the public domain.

    THIS SOFTWARE PACKAGE IS PROVIDED "AS IS" WITHOUT EXPRESS
    OR IMPLIED WARRANTY.
    THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE PACKAGE,
    INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
    IN NO  EVENT SHALL THE AUTHOR (Daniel Hellerstein) OR ANY PERSON OR
    INSTITUTION ASSOCIATED WITH THIS PRODUCT BE LIABLE FOR ANY
    SPECIAL,INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
    RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    OF CONTRACT,NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
    IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE PACKAGE.


   SRE-FILTER was developed on the personal time of Daniel Hellerstein,
   and is not supported, approved, or in any way an official product
   of my employer (USDA/ERS).

Acknowlegments

• Don Meyer, author of the GoHTTP filter, was very gracious in allowing me to "build on his work". In particular, much of the source code for HTACCESS file access-controls, CGI-BIN script processing, and image map processing was adapted from his code.
• Derek Sims (author of the IntrFilt filter) for helpful advice regarding multi-host servers.
• Charles Daney for providing the REXXLIB library at a very reasonable price.
• Dave Briccetti for providing the "SMTP using socket calls" code.
• Ariel L. Szczupak for advice on working with queues and semaphores.
• Richard Hughes for useful thoughts on discussion group software
• The people at the INFO-ZIP project, for providing the UNZIPAPI dynamic link library.
• the several early adopters (may they rest in peace) for their patience in dealing with stupid bugs
• several individuals (mm, dr, sr, et al) for good suggestions that are now realities
• .... and (of course) Mike Cowlishaw for creating GoServe, and always finding time to try and answer.

Return to table of contents.