Apache API notes

Robert S. Thau

A few notes on general pedagogical style here. In the interest of conciseness, all structure declarations here are incomplete --- the real ones have more slots, that I'm not telling you about. For the most part, these are reserved to one component of the server core or another, and should be altered by modules with caution. However, in some cases, they really are things I just haven't gotten around to yet. Welcome to the bleeding edge.

Finally, here's an outline, to give you some bare idea of what's coming up, and in what order:

• Basic concepts.
  • Handlers, Modules, and Requests
  • A brief tour of a module
• How handlers work
  • A brief tour of the request_rec
  • Where request_rec structures come from
  • Handling requests, declining, and returning error codes
  • Special considerations for response handlers
  • Special considerations for authentication handlers
  • Special considerations for logging handlers
• Resource allocation and resource pools
• Configuration, commands and the like
  • Per-directory configuration structures
  • Command handling
  • Side notes --- per-server configuration, virtual servers, etc.

Basic concepts.

Handlers, Modules, and Requests

• URI -> Filename translation
• Auth ID checking [is the user who they say they are?]
• Auth access checking [is the user authorized here?]
• Access checking other than auth
• Determining MIME type of the object requested
• "Fixups" --- there aren't any of these yet, but the phase is intended as a hook for possible extensions like SetEnv, which don't really fit well elsewhere.
• Actually sending a response back to the client.
• Logging the request

• Handle the request, and indicate that it has done so by returning the magic constant OK.
• Decline to handle the request, by returning the magic integer constant DECLINED. In this case, the server behaves in all respects as if the handler simply hadn't been there.
• Signal an error, by returning one of the HTTP error codes. This terminates normal handling of the request, although an ErrorDocument may be invoked to try to mop up, and it will be logged in any case.

The handlers themselves are functions of one argument (a request_rec structure. vide infra), which returns an integer, as above.

A brief tour of a module

Let's begin with handlers. In order to handle the CGI scripts, the module declares a response handler for them. Because of ScriptAlias, it also has handlers for the name translation phase (to recognise ScriptAliased URI's), the type-checking phase (any ScriptAliased request is typed as a CGI script).

The module needs to maintain some per (virtual) server information, namely, the ScriptAliases in effect; the module structure therefore contains pointers to a functions which builds these structures, and to another which combines two of them (in case the main server and a virtual server both have ScriptAliases declared).

Finally, this module contains code to handle the ScriptAlias command itself. This particular module only declares one command, but there could be more, so modules have command tables which declare their commands, and describe where they are permitted, and how they are to be invoked.

A final note on the declared types of the arguments of some of these commands: a pool is a pointer to a resource pool structure; these are used by the server to keep track of the memory which has been allocated, files opened, etc., either to service a particular request, or to handle the process of configuring itself. That way, when the request is over (or, for the configuration pool, when the server is restarting), the memory can be freed, and the files closed, en masse, without anyone having to write explicit code to track them all down and dispose of them. Also, a cmd_parms structure contains various information about the config file being read, and other status information, which is sometimes of use to the function which processes a config-file command (such as ScriptAlias). With no further ado, the module itself:

/* Declarations of handlers. */

int translate_scriptalias (request_rec *);
int type_scriptalias (request_rec *);
int cgi_handler (request_rec *);

/* Subsdiary dispatch table for response-phase handlers, by MIME type */

handler_rec cgi_handlers[] = {
{ "application/x-httpd-cgi", cgi_handler },
{ NULL }
};

/* Declarations of routines to manipulate the module's configuration
 * info.  Note that these are returned, and passed in, as void *'s;
 * the server core keeps track of them, but it doesn't, and can't,
 * know their internal structure.
 */

void *make_cgi_server_config (pool *);
void *merge_cgi_server_config (pool *, void *, void *);

/* Declarations of routines to handle config-file commands */

char *script_alias (cmd_parms *, void *per_dir_config, char *fake, char *real);

command_rec cgi_cmds[] = {
{ "ScriptAlias", script_alias, NULL, RSRC_CONF, TAKE2,
    "a fakename and a realname"},
{ NULL }
};

module cgi_module = {
   STANDARD_MODULE_STUFF,
   NULL,			/* initializer */
   NULL,			/* dir config creater */
   NULL,			/* dir merger --- default is to override */
   make_cgi_server_config,      /* server config */
   merge_cgi_server_config,	/* merge server config */
   cgi_cmds,			/* command table */
   cgi_handlers,		/* handlers */
   translate_scriptalias,	/* filename translation */
   NULL,			/* check_user_id */
   NULL,			/* check auth */
   NULL,			/* check access */
   type_scriptalias,		/* type_checker */
   NULL,			/* fixups */
   NULL				/* logger */
};

How handlers work

A brief tour of the request_rec

The most important such information is a small set of character strings describing attributes of the object being requested, including its URI, filename, content-type and content-encoding (these being filled in by the translation and type-check handlers which handle the request, respectively).

Other commonly used data items are tables giving the MIME headers on the client's original request, MIME headers to be sent back with the ppppresponse (which modules can add to at will), and environment variables for any subprocesses which are spawned off in the course of servicing the request. These tables are manipulated using the table_get and table_set routines.

Finally, there are pointers to two data structures which, in turn, point to per-module configuration structures. Specifically, these hold pointers to the data structures which the module has built to describe the way it has been configured to operate in a given directory (via .htaccess files or <Directory> sections), for private data it has built in the course of servicing the request (so modules' handlers for one phase can pass "notes" to their handlers for other phases). There is another such configuration vector in the server_rec data structure pointed to by the request_rec, which contains per (virtual) server configuration data.

Here is an abridged declaration, giving the fields most commonly used:

struct request_rec {

  pool *pool;
  conn_rec *connection;
  server_rec *server;

  /* What object is being requested */
  
  char *uri;
  char *filename;
  char *path_info;
  char *args;			/* QUERY_ARGS, if any */
  struct stat finfo;		/* Set by server core;
                                 * st_mode set to zero if no such file */
  
  char *content_type;
  char *content_encoding;
  
  /* MIME header environments, in and out.  Also, an array containing
   * environment variables to be passed to subprocesses, so people can
   * write modules to add to that environment.
   *
   * The difference between headers_out and err_headers_out is that the
   * latter are printed even on error, and persist across internal redirects
   * (so the headers printed for ErrorDocument handlers will have them).
   */
  
  table *headers_in;
  table *headers_out;
  table *err_headers_out;
  table *subprocess_env;

  /* Info about the request itself... */
  
  int header_only;		/* HEAD request, as opposed to GET */
  char *protocol;		/* Protocol, as given to us, or HTTP/0.9 */
  char *method;			/* GET, HEAD, POST, etc. */
  int method_number;		/* M_GET, M_POST, etc. */

  /* Info for logging */

  char *the_request;
  int bytes_sent;

  /* A flag which modules can set, to indicate that the data being
   * returned is volatile, and clients should be told not to cache it.
   */

  int no_cache;

  /* Various other config info which may change with .htaccess files
   * These are config vectors, with one void* pointer for each module
   * (the thing pointed to being the module's business).
   */
  
  void *per_dir_config;		/* Options set in config files, etc. */
  void *request_config;		/* Notes on *this* request */
  
};

Where request_rec structures come from

• If the request is to an imagemap, a type map (i.e., a *.var file), or a CGI script which returned a local "Location:", then the resource which the user requested is going to be ultimately located by some URI other than what the client originally supplied. In this case, the server does an internal redirect, constructing a new request_rec for the new URI, and processing it almost exactly as if the client had requested the new URI directly.

• If some handler signaled an error, and an ErrorDocument is in scope, the same internal redirect machinery comes into play.

• Finally, a handler occasionally needs to investigate "what would happen if" some other request were run. For instance, the directory indexing module needs to know what MIME type would be assigned to a request for each directory entry, in order to figure out what icon to use.

  Such handlers can construct a sub-request, using the functions sub_req_lookup_file and sub_req_lookup_uri; this constructs a new request_rec structure and processes it as you would expect, up to but not including the point of actually sending a response. (These functions skip over the access checks if the sub-request is for a file in the same directory as the original request).

  (Server-side includes work by building sub-requests and then actually invoking the response handler for them, via the function run_sub_request).

Handling requests, declining, and returning error codes

• OK --- the request was handled successfully. This may or may not terminate the phase.
• DECLINED --- no erroneous condition exists, but the module declines to handle the phase; the server tries to find another.
• an HTTP error code, which aborts handling of the request.

Special considerations for response handlers

They should begin by sending an HTTP response header, using the function send_http_header. (You don't have to do anything special to skip sending the header for HTTP/0.9 requests; the function figures out on its own that it shouldn't do anything). If the request is marked header_only, that's all they should do; they should return after that, without attempting any further output.

Otherwise, they should produce a request body which responds to the client as appropriate. The primitives for this are rputc and rprintf, for internally generated output, and send_fd, to copy the contents of some FILE * straight to the client.

At this point, you should more or less understand the following piece of code, which is the handler which handles GET requests which have no more specific handler; it also shows how conditional GETs can be handled, if it's desirable to do so in a particular response handler --- set_last_modified checks agaisnt the If-modified-since value supplied by the client, if annny, and returns an appropriate code (which will, if nonzero, be USE_LOCAL_COPY). No similar considerations apply for set_content_length, but it returns an error code for symmetry.

int default_handler (request_rec *r)
{
    int errstatus;
    FILE *f;
    
    if (r->method_number != M_GET) return DECLINED;
    if (r->finfo.st_mode == 0) return NOT_FOUND;
	
    if ((errstatus = set_content_length (r, r->finfo.st_size))
	|| (errstatus = set_last_modified (r, r->finfo.st_mtime)))
        return errstatus;
    
    f = fopen (r->filename, "r");

    if (f == NULL) {
        log_reason("file permissions deny server access", r->filename, r);
        return FORBIDDEN;
    }
      
    register_timeout ("send", r);
    send_http_header (r);

    if (!r->header_only) send_fd (f, r);
    pfclose (r->pool, f);
    return OK;
}

(Invoking internal_redirect from handlers which are not response handlers will lead to serious confusion).

Special considerations for authentication handlers

• Authentication-phase handlers not invoked unless auth is configured for the directory.
• Common auth configuration stored in the core per-dir configuration; it has accessors auth_type, auth_name, and requires.
• Common routines, to handle the protocol end of things, at least for HTTP basic authentication (get_basic_auth_pw, which sets the connection->user structure field automatically, and note_basic_auth_failure, which arranges for the proper WWW-Authenticate: header to be sent back).

Special considerations for logging handlers

Resource allocation and resource pools

The way this works is as follows: the memory which is allocated, file opened, etc., to deal with a particular request are tied to a resource pool which is allocated for the request. The pool is a data structure which itself tracks the resources in question.

When the request has been processed, the pool is cleared. At that point, all the memory associated with it is released for reuse, all files associated with it are closed, and any other clean-up functions which are associated with the pool are run. When this is over, we can be confident that all the resource tied to the pool have been released, and that none of them have leaked.

Server restarts, and allocation of memory and resources for per-server configuration, are handled in a similar way. There is a configuration pool, which keeps track of resources which were allocated while reading the server configuration files, and handling the commands therein (for instance, the memory that was allocated for per-server module configuration, log files and other files that were opened, and so forth). When the server restarts, and has to reread the configuration files, the configuration pool is cleared, and so the memory and file descriptors which were taken up by reading them the last time are made available for reuse.

It should be noted that use of the pool machinery isn't generally obligatory, except for situations like logging handlers, where you really need to register cleanups to make sure that the log file gets closed when the server restarts (this is most easily done by using the function pfopen, which also arranges for the underlying file descriptor to be closed before any child processes, such as for CGI scripts, are execed), or in case you are using the timeout machinery (which isn't yet even documented here). However, there are two benefits to using it: resources allocated to a pool never leak (even if you allocate a scratch string, and just forget about it); also, for memory allocation, palloc is generally faster than malloc.

We begin here by describing how memory is allocated to pools, and then discuss how other resources are tracked by the resource pool machinery.

Allocation of memory in pools

int my_handler(request_rec *r)
{
    struct my_structure *foo;
    ...

    foo = (foo *)palloc (r->pool, sizeof(my_structure));
}

(It also raises the possibility that heavy use of palloc could cause a server process to grow excessively large. There are two ways to deal with this, which are dealt with below; briefly, you can use malloc, and try to be sure that all of the memory gets explicitly freed, or you can allocate a sub-pool of the main pool, allocate your memory in the sub-pool, and clear it out periodically. The latter technique is discussed in the section on sub-pools below, and is used in the directory-indexing code, in order to avoid excessive storage allocation when listing directories with thousands of files).

Allocating initialized memory

     pstrcat (r->pool, "foo", "/", "bar", NULL);

Tracking open files, etc.

     ...
     FILE *f = pfopen (r->pool, r->filename, "r");

     if (f == NULL) { ... } else { ... }

Unlike the case for memory, there are functions to close files allocated with pfopen, and popenf, namely pfclose and pclosef. (This is because, on many systems, the number of files which a single process can have open is quite limited). It is important to use these functions to close files allocated with pfopen and popenf, since to do otherwise could cause fatal errors on systems such as Linux, which react badly if the same FILE* is closed more than once.

(Using the close functions is not mandatory, since the file will eventually be closed regardless, but you should consider it in cases where your module is opening, or could open, a lot of files).

Other sorts of resources --- cleanup functions

Fine control --- creating and dealing with sub-pools, with a note on sub-requests

The primitive for creating a sub-pool is make_sub_pool, which takes another pool (the parent pool) as an argument. When the main pool is cleared, the sub-pool will be destroyed. The sub-pool may also be cleared or destroyed at any time, by calling the functions clear_pool and destroy_pool, respectively. (The difference is that clear_pool frees resources associated with the pool, while destroy_pool also deallocates the pool itself. In the former case, you can allocate new resources within the pool, and clear it again, and so forth; in the latter case, it is simply gone).

One final note --- sub-requests have their own resource pools, which are sub-pools of the resource pool for the main request. The polite way to reclaim the resources associated with a sub request which you have allocated (using the sub_req_lookup_... functions) is destroy_sub_request, which frees the resource pool. Before calling this function, be sure to copy anything that you care about which might be allocated in the sub-request's resource pool into someplace a little less volatile (for instance, the filename in its request_rec structure).

(Again, under most circumstances, you shouldn't feel obliged to call this function; only 2K of memory or so are allocated for a typical sub request, and it will be freed anyway when the main request pool is cleared. It is only when you are allocating many, many sub-requests for a single main request that you should seriously consider the destroy... functions).

Configuration, commands and the like

However, just giving the modules command tables is not enough to divorce them completely from the server core. The server has to remember the commands in order to act on them later. That involves maintaining data which is private to the modules, and which can be either per-server, or per-directory. Most things are per-directory, including in particular access control and authorization information, but also information on how to determine file types from suffixes, which can be modified by AddType and DefaultType directives, and so forth. In general, the governing philosophy is that anything which can be made configurable by directory should be; per-server information is generally used in the standard set of modules for information like Aliases and Redirects which come into play before the request is tied to a particular place in the underlying file system.

Another requirement for emulating the NCSA server is being able to handle the per-directory configuration files, generally called .htaccess files, though even in the NCSA server they can contain directives which have nothing at all to do with access control. Accordingly, after URI -> filename translation, but before performing any other phase, the server walks down the directory hierarchy of the underlying filesystem, following the translated pathname, to read any .htaccess files which might be present. The information which is read in then has to be merged with the applicable information from the server's own config files (either from the <Directory> sections in access.conf, or from defaults in srm.conf, which actually behaves for most purposes almost exactly like <Directory />).

Finally, after having served a request which involved reading .htaccess files, we need to discard the storage allocated for handling them. That is solved the same way it is solved wherever else similar problems come up, by tying those structures to the per-transaction resource pool.

Per-directory configuration structures

typedef struct {
    table *forced_types;	/* Additional AddTyped stuff */
    table *encoding_types;	/* Added with AddEncoding... */
} mime_dir_config;

(If we are reading a .htaccess file, that resource pool is the per-request resource pool for the request; otherwise it is a resource pool which is used for configuration data, and cleared on restarts. Either way, it is important for the structure being created to vanish when the pool is cleared, by registering a cleanup on the pool if necessary).

For the MIME module, the per-dir config creation function just pallocs the structure above, and a creates a couple of tables to fill it. That looks like this:

void *create_mime_dir_config (pool *p, char *dummy)
{
    mime_dir_config *new =
      (mime_dir_config *) palloc (p, sizeof(mime_dir_config));

    new->forced_types = make_table (p, 4);
    new->encoding_types = make_table (p, 4);
    
    return new;
}

To do that, the server invokes the module's per-directory config merge function, if one is present. That function takes three arguments: the two structures being merged, and a resource pool in which to allocate the result. For the MIME module, all that needs to be done is overlay the tables from the new per-directory config structure with those from the parent:

void *merge_mime_dir_configs (pool *p, void *parent_dirv, void *subdirv)
{
    mime_dir_config *parent_dir = (mime_dir_config *)parent_dirv;
    mime_dir_config *subdir = (mime_dir_config *)subdirv;
    mime_dir_config *new =
      (mime_dir_config *)palloc (p, sizeof(mime_dir_config));

    new->forced_types = overlay_tables (p, subdir->forced_types,
					parent_dir->forced_types);
    new->encoding_types = overlay_tables (p, subdir->encoding_types,
					  parent_dir->encoding_types);

    return new;
}

Command handling

char *add_type(cmd_parms *cmd, mime_dir_config *m, char *ct, char *ext)
{
    if (*ext == '.') ++ext;
    table_set (m->forced_types, ext, ct);
    return NULL;
}

Another way in which this particular command handler is unusually simple is that there are no error conditions which it can encounter. If there were, it could return an error message instead of NULL; this causes an error to be printed out on the server's stderr, followed by a quick exit, if it is in the main config files; for a .htaccess file, the syntax error is logged in the server error log (along with an indication of where it came from), and the request is bounced with a server error response (HTTP error status, code 500).

The MIME module's command table has entries for these commands, which look like this:

command_rec mime_cmds[] = {
{ "AddType", add_type, NULL, OR_FILEINFO, TAKE2, 
    "a mime type followed by a file extension" },
{ "AddEncoding", add_encoding, NULL, OR_FILEINFO, TAKE2, 
    "an encoding (e.g., gzip), followed by a file extension" },
{ NULL }
};

• The name of the command
• The function which handles it
• a (void *) pointer, which is passed in the cmd_parms structure to the command handler --- this is useful in case many similar commands are handled by the same function.
• A bit mask indicating where the command may appear. There are mask bits corresponding to each AllowOverride option, and an additional mask bit, RSRC_CONF, indicating that the command may appear in the server's own config files, but not in any .htaccess file.
• A flag indicating how many arguments the command handler wants preparsed, and how they should be passed in. TAKE2 indicates two preparsed arguments. Other options are TAKE1, which indicates one preparsed argument, FLAG, which indicates that the argument should be On or Off, and is passed in as a boolean flag, RAW_ARGS, which causes the server to give the command the raw, unparsed arguments (everything but the command name itself). There is also ITERATE, which means that the handler looks the same as TAKE1, but that if multiple arguments are present, it should be called multiple times, and finally ITERATE2, which indicates that the command handler looks like a TAKE2, but if more arguments are present, then it should be called multiple times, holding the first argument constant.
• Finally, we have a string which describes the arguments that should be present. If the arguments in the actual config file are not as required, this string will be used to help give a more specific error message. (You can safely leave this NULL).

int find_ct(request_rec *r)
{
    int i;
    char *fn = pstrdup (r->pool, r->filename);
    mime_dir_config *conf =
      (mime_dir_config *)get_module_config(r->per_dir_config, &mime_module);
    char *type;

    if (S_ISDIR(r->finfo.st_mode)) {
        r->content_type = DIR_MAGIC_TYPE;
	return OK;
    }
    
    if((i=rind(fn,'.')) < 0) return DECLINED;
    ++i;

    if ((type = table_get (conf->encoding_types, &fn[i])))
    {
        r->content_encoding = type;

	/* go back to previous extension to try to use it as a type */
	
        fn[i-1] = '\0';
	if((i=rind(fn,'.')) < 0) return OK;
	++i;
    }

    if ((type = table_get (conf->forced_types, &fn[i])))
    {
        r->content_type = type;
    }
    
    return OK;
}

Side notes --- per-server configuration, virtual servers, etc.

The only substantial difference is that when a command needs to configure the per-server private module data, it needs to go to the cmd_parms data to get at it. Here's an example, from the alias module, which also indicates how a syntax error can be returned (note that the per-directory configuration argument to the command handler is declared as a dummy, since the module doesn't actually have per-directory config data):

char *add_redirect(cmd_parms *cmd, void *dummy, char *f, char *url)
{
    server_rec *s = cmd->server;
    alias_server_conf *conf =
        (alias_server_conf *)get_module_config(s->module_config,&alias_module);
    alias_entry *new = push_array (conf->redirects);

    if (!is_url (url)) return "Redirect to non-URL";
    
    new->fake = f; new->real = url;
    return NULL;
}