ftp.cse.unsw.edu.au/2014.06.ftp.cse.unsw.edu.au.tar/ftp.cse.unsw.edu.au/pub/doc/papers/misc/config-WRPRC-X11R4/config-WRPRC.ms

Text File | 1992-10-18 | 20.9 KB | 692 lines

.\" format with
.\" xroff -ms % | lpr
.\"
.\" revision date - change whenever this file is edited
.ds RD 24 July 1990
.nr PO 1.2i    \" page offset 1.2 inches
.nr PD .7v    \" inter-paragraph distance
.\"
.EH 'Multiple-project Imake'- % -''
.OH ''- % -'Multiple-project Imake'
.OF 'Revision date:\0\0\*(RD'\s+2\*-DRAFT\*-\s-2'Printed:\0\0\n(dy \*(MO 19\n(yr'
.EF 'Revision date:\0\0\*(RD'\s+2\*-DRAFT\*-\s-2'Printed:\0\0\n(dy \*(MO 19\n(yr'
.\"
.\" I - italic font (taken from -ms and changed)
.de I
.nr PQ \\n(.f
.if t \&\\$3\\f2\\$1\\fP\&\\$2
.if n .if \\n(.$=1 \&\\$1
.if n .if \\n(.$>1 \&\\$1\c
.if n .if \\n(.$>1 \&\\$2
..
.TL
Using Imake for Multiple Projects
.sp .5v
(document version 1.02)
.AU
Paul DuBois
[e-mail]
.AI
Wisconsin Regional Primate Research Center
Revision date:\0\0\*(RD
.NH
Introduction
.LP
.I imake
is a tool for generating makefiles and is used to allow software projects
to be configured easily on a wide variety of target machines.
It is distributed with projects such as the X Window System and the Kerberos
authentication system.
The tool
.I makedepend
is typically used in conjunction with
.I imake
to generate dependencies in makefiles.
.I xmkmf
is also used, for bootstrapping makefiles from imakefiles.
.LP
This document describes the particular way
.I imake
is used to configure projects at WRPRC.
The method is patterned after the way
.I imake
is employed in the X Window System (version 11, release 4).
The X11R4 method is significantly more developed than in X11R3.
Presumably,
.I imake
as used in Kerberos (currently at protocol revision 4)
might have been used as a base, but I judged it
unsuitable for two reasons: (i) I learned
.I imake
from studying X, so I'm probably imprinted to it;
(ii) development of the configuration files in Kerberos appears to be
at a more primitive level even than in X11R3.
.LP
The X11R4 approach allows configuration files to be stored either
in a well-known system-wide location (e.g.,
.I /usr/lib/config ),
or in a
.I config
directory under the project top-level directory.
.LP
A more-than-passing acquaintance with
.I imake
is assumed here.
Helpful companion documentation includes ``Configuration Management in
the X Window System,'' the configuration and
.I README
files in the X11R4 distribution (in the
.I mit/config
directory), and the WRPRC configuration files (see below for
information on distribution and availability).
.NH
Statement of Problem
.LP
Because of the flexibility
.I imake
provides as a
configuration tool (once you figure out how to use it!),
I had begun to use it on all my projects.
Since these were all developed at the same site, the projects naturally
had quite a bit of configuration information in common.
On the other hand, different projects just as naturally had their own
individual requirements.
.LP
What I had been doing was creating a
.I config
directory in each project directory, with a copy of the configuration files
in each, modified as necessary for the project (otherwise known as ``using
brute force'').
I did not use configuration files in a system-wide location
because of differing project requirements.
At the same time, there was a great deal of redundancy between sets of
configuration files since projects were built and installed
in similar ways.
It seemed rather wasteful and silly not to take advantage of this overlap,
besides the fact that the multiple copies of files contributed to a
maintenance inefficiency when changes were made that affected all projects.
.LP
This (unsatisfactory) situation prompts the question:
how can the number of files to be maintained be minimized and how can
configuration files be shared among projects without sacrificing the ability
to specify project-specific information as necessary?
.NH
Solution to Problem
.LP
Since
.I imake
possessed the ability to look through multiple directories when looking
for configuration files, the simplest solution to the problem seemed to be:
.IP (i)
Create a set of standard configuration files and store them in a well-known
location (referred to here as StdConfigDir).
.IP (ii)
Allow projects to define an alternate configuration directory (referred to
here as ProjConfigDir, typically the
.I config
directory under the project directory).
.IP (iii)
Tell
.I imake
to look through ProjConfigDir, then StdConfigDir for configuration files.
This means changing
.I xmkmf ,
which executes
.I imake ,
and the configuration file definitions that construct
.I imake
commands.
.LP
In other words,
a standard set of configuration files is created and stored in a well-known
location for use by multiple projects.
However, should it be required,
individual projects place configuration files in their own
.I config
directory, which takes precedence over StdConfigDir when
searching for files.
In the degenerate case, projects need supply nothing if
the standard stuff is sufficient; only files in StdConfigDir will be
used.
.NH
Implementation of Solution
.NH 2
Imake.tmpl Architecture
.LP
The architecture of the
.I imake
template,
.I Imake.tmpl ,
as used in X11R4 is, roughly:
.LP
.DS
.ta 3i
header block section
#include <\fIplatform\fR.cf>    (platform-specific definitions)
#include <site.def>        (site-specific definitions)
system description, build and installation definitions
#include <Project.tmpl>    (project-specific definitions)
#include <Imake.rules>    (\fIimake\fR rules)
#include "./Imakefile"    (\fIImakefile\fR for current directory)
.DE
.LP
The header block section determines the machine type and the
platform-specific file
.I platform .cf
to be included.
.LP
To accommodate other project-specific files, extra
#include's are introduced into
.I Imake.tmpl .
The resulting modified architecture becomes:
.LP
.DS
header block section
.sp .2v
#include <\fIplatform\fR.pcf>
#include <\fIplatform\fR.cf>
.sp .2v
#include <site.pdef>
#include <site.def>
.sp .2v
system description, build and installation definitions
.sp .2v
#include <Project.ptmpl>
#include <Project.tmpl>
.sp .2v
#include <Imake.prules>
#include <Imake.rules>
.sp .2v
#include "./Imakefile"
.DE
.LP
The header block section now additionally determines the name of the
project-specific platform file
.I platform .pcf.
.LP
The modification is based on the notion of override files.
Before any of the platform, site, project or rules
files are included, another related file is
included which may contain project-specific information.
Standard versions of these project-specific files
.I *.pcf , (
.I site.pdef ,
.I Project.ptmpl
and
.I Imake.prules )
are present in
StdConfigDir (to prevent file-inclusion errors) but they are all empty.
If a given project has special configuration requirements, it may
supply its own private versions of any of these files in its
.I config
directory.
Since
.I imake
will search that directory before StdConfigDir, the standard (empty)
versions will be overridden.
The project does not have to actually replace them.
.LP
.B "Example 1:"
At the one extreme, when a project has
no special project-specific configuration requirements (no
.I config
directory, or an empty one),
only the files in StdConfigDir are used to create makefiles.
Since the default project-specific files are empty, the new
architecture of
.I Imake.tmpl
is functionally equivalent to the old.
At the other extreme, a project can replace
.I all
the configuration files; nothing from StdConfigDir is used then.
.LP
.B "Example 2:"
The standard
.I site.def
file might be used by the system administrator to set the value of
BinDir to the directory in which programs are installed.
A project might want them installed somewhere else; this could be done
by defining BinDir in the project-specific site file
.I site.pdef
to override the value in
.I site.def .
.LP
.B "Example 3:"
Override files may also be used to extend the standard configuration
by defining symbols not appearing at all in the standard files.
For instance, if some programs were to be installed in the standard
BinDir, but others were to be installed in a private directory, a
symbol such as ProjBinDir could be defined for the latter purpose in
.I Project.ptmpl .
.LP
One bothersome thing about this architecture is the naming of
.I Project.tmpl
and
.I Project.ptmpl .
If
.I Project.tmpl
contains ``project-specific'' information, then what does
.I Project.ptmpl
contain?
``Project-project-specific'' information?
Ugh.
I prefer to consider
.I Project.tmpl
to contain ``standard'' (i.e., best-guess) project configuration
parameters, and
.I Project.ptmpl
to contain the parameters which actually are specific to a given project.
.LP
The disadvantages of this architecture are obvious:
.IP \(bu
.I Imake.tmpl
is somewhat more complex.
.IP \(bu
StdConfigDir must be populated with the additional
.I *.pcf ,
.I site.pdef ,
.I Project.ptmpl
and
.I Imake.prules
files (although this is easy since they are all empty).
.LP
The advantages are:
.IP \(bu
Configuration information is centralized, but flexibility is maintained.
Projects may
override definitions in the the standard configuration files
without replacing those files directly, and multiple projects may use
files in StdConfigDir cooperatively without interfering with each other.
.IP \(bu
Override files minimize the need for individual
projects to modify the standard non-project specific files
.I Imake.tmpl , (
.I *.cf ,
.I site.def ,
.I Project.tmpl
and
.I Imake.rules )
directly.
The philosophy underlying the architecture is that the standard files
describe a common ``baseline'' configuration, and
individual projects need specify configuration information (using
private
.I *.pcf ,
.I site.pdef ,
.I Project.ptmpl
and
.I Imake.prules
files) only to the extent that they
.I differ
from the standard.
.IP \(bu
When improvements are made to the standard files, the changes are
immediately available to any project using them via a normal
configuration step, i.e., ``make Makefile;make Makefiles''.
(Of course, it also becomes possible to break a lot of projects if an
erroneous change is made.)
.IP \(bu
Overall, fewer files need to be maintained, and less space is used.
.NH 2
Other Changes
.LP
.I xmkmf
is used to bootstrap makefiles from imakefiles.
To accommodate the above scheme, it was
modified from the version distributed with X11R4 to search the project's
.I config
directory, then the standard configuration directory
(i.e., ProjConfigDir then StdConfigDir, not just StdConfigDir).
.LP
The definition of the makefile variable IMAKE_CMD in the configuration
files was also changed:
.LP
.DS
Old (from X11R4):
#ifdef UseInstalled
    IRULESRC = $(CONFIGDIR)
    IMAKE_CMD = $(IMAKE) -DUseInstalled \e
        -I$(IRULESRC) \e
        $(IMAKE_DEFINES)
#else
    IRULESRC = $(CONFIGSRC)
    IMAKE_CMD = $(NEWTOP)$(IMAKE) \e
        -I$(NEWTOP)$(IRULESRC) \e
        $(IMAKE_DEFINES)
#endif
.DE
.\" allow page break here if necessary...
.DS
New:
#ifdef UseInstalled
    IMAKE_CMD = $(IMAKE) -DUseInstalled \e
        -I$(NEWTOP)$(PROJCONFIGDIR) \e
        -I$(STDCONFIGDIR) \e
        $(IMAKE_DEFINES)
#else
    IMAKE_CMD = $(NEWTOP)$(IMAKE) \e
        -I$(NEWTOP)$(PROJCONFIGDIR) \e
        -I$(STDCONFIGDIR) \e
        $(IMAKE_DEFINES)
#endif
.DE
.LP
The old rule determines whether to use versions of
.I imake
(and
.I makedepend )
already installed in public directories or
build them in the source tree based on whether UseInstalled is defined or not.
That symbol also determines where to look for configuration files and
defines the value of IRULESRC accordingly.
If UseInstalled is undefined, searches look in CONFIGSRC, the project's
.I config
directory, otherwise in CONFIGDIR, the standard configuration directory.
.LP
This behavior is unsuitable for the purposes described here, since both of
those directories should be searched, if they exist.
The new rule makes UseInstalled relevant only for determining where
to find (and possibly build)
.I imake
and
.I makedepend ,
not where to look for configuration files.
.NH
Test of Implementation
.LP
After making the changes described above, a flight test was arranged.
Nine projects developed under six different logins were converted to use
the standard files.
Projects supplied override files where necessary.
These projects were all initially developed using configuration files
that bore a
fairly strong relationship to the standard set, so it was expected
that conversion would not be unbearably odious.
The degree of relationship broke down as follows:
.IP \(bu
Two projects used configuration files that were identical (or nearly
so) to the standards.
Each of these projects, unlike the other seven, comprised only a
single source directory.
As expected, conversion was completely trivial, once the top-level
.I Makefile
was rebuilt with
.I xmkmf .
.IP \(bu
Four projects required additional project-specific specification
beyond the standard configuration
(special include file directories, link libraries,
installation directories, etc.).
These usually required only
.I site.pdef
and
.I Project.tmpl
files in their local
.I config
directories.
One project required a
.I platform.pcf
file.
Project owner and group names (used for install operations) were moved
out of
.I site.def
files into
.I site.pdef .
.IP \(bu
Three projects diverged from the standard files fairly significantly,
because although their configuration files were initially set up from
files similar to the standards, less effort had been exerted to keep
them in sync as project development progressed.
Besides the changes made to the projects described in the previous
paragraph, there were additional special commands for compilation, special
.I Imakefile
rules, etc.
Nevertheless, conversion proceeded smoothly and without incident.
.LP
One thing I had done in the original
.I Project.tmpl
files was to define certain
makefile variables directly (e.g., ``INCLUDESRC=$(TOP)/h''), rather
than by using a previously #define'd
.I cpp
symbol (e.g., ``INCLUDESRC=IncludeSrc'', where IncludeSrc is #define'd
within a #ifndef/#endif block).
This was a short-cut, that works in a non-shared environment but
needed to be fixed to work in a shared environment.
A default value of IncludeSrc needs to be placed in
.I Project.tmpl ,
and individual projects redefine that symbol as necessary in
.I Project.ptmpl .
.LP
It was necessary to place a number of
.I cpp
symbol definitions in
.I Project.tmpl
within #ifndef/#endif blocks, since they became liable to being
overridden in
.I Project.ptmpl .
.LP
In the top-level imakefiles for the multiple source directory
projects, the ``make World'' rules had to be adjusted to account for
whether UseInstalled is defined or not.
If so, the build process does not need to try to build
.I imake
and
.I makedepend
in the project source tree.
(That would be a mistake since the source for those programs would not
necessarily be present.)
.LP
Overall, conversion was simple and straightforward.
I judged the conversion odiousness coefficient acceptably low.
.NH
Another Problem\*-Project Groups
.LP
The mechanism described above allows a group of similarly-configured
projects to share configuration files and selectively extend or
override them as necessary.
This provides a solution to the original problem (how to avoid
maintaining multiple full sets of files while maintaining flexibility),
but leads to another.
Suppose that instead of a group of projects, there exist several
groups of projects, such that projects within a group are similar, but
differences between groups are quite large.
This can happen, for instance, if projects designed and developed at one
site under certain conventions are installed at another
site where different conventions are used.
.LP
The difficulty here is that there are two sets of ``standard'' configuration
files, and they cannot both be installed in the same well-known
location.
This problem can be overcome quite simply.
Instead of using StdConfigDir as the single well-known location in which
the standards are installed, define StdConfigPath to be that
directory, and use it as the root of a set of
subdirectories, each of which corresponds to a project group.
Each group's standard configuration files are installed in the appropriate
directory, and StdConfigDir for a given project group
will be defined as that directory (e.g., StdConfigPath/\fIabc\fR for
project \fIabc\fR).
.LP
To allow
.I xmkmf
to be used for bootstrapping,
it was modified slightly again to select
one of the subdirectories as the hardwired-in default, but allow
another to be selected through a
.B \-C
option.
For convenience, the argument following
.B \-C
may be absolute or relative.
If absolute, it is used unchanged.
If relative, StdConfigDir is prepended.
.LP
This modification makes
.I xmkmf
incompatible with the version distributed with X11R4, which is a
problem for sites using that version.
Assuming X configuration files are installed in
.I /usr/lib/X11/config
and the value of StdConfigPath is
.I /usr/lib/config ,
this incompatibility can be obviated as follows:
.LP
.DS
mkdir /usr/lib/config
cd /usr/lib/config
ln -s /usr/lib/X11/config X11
ln -s X11 default
.DE
.LP
For sites without symbolic links, the files in
.I /usr/lib/X11/config
can be copied into
.I /usr/lib/config/X11 ,
and
.I xmkmf
can be hacked to use
.I /usr/lib/config/X11
instead of
.I /usr/lib/config/default
as the default.
Ugly, but workable.
(Also known as ``crude, but effective''.)
.LP
With these conventions,
projects within groups can share redundant information, as before, and
project groups can coexist peacefully.
.LP
The nine projects previously converted were tested against these
additional modifications.
They required only ``xmkmf;make Makefile;make Makefiles'' to properly
reset the configuration information in the makefiles.
.NH
Distribution of Projects
.LP
The mechanism described in this document appears to solve
the problem of reusing
configuration files while allowing for extension and override within
individual projects.
However, when a project is packaged for distribution, it's very little
use to include only those configuration files which are unique to the
project, if the receiving site doesn't have the standard files.
Several alternatives come to mind.
.IP \(bu
Provide the standards as a separate distribution, along with the
sources to
.I imake
and
.I makedepend .
This is perhaps inconvenient for the receiver, as it is another
distribution that needs to be retrieved.
.IP \(bu
Copy all the standard files into the project's
.I config
directory and distribute along with the project.
One has to be careful when doing the copying not to destroy the
project's override files,
the resulting distribution size is larger, and
distributing the standards is unnecessary if receiving site
has them already.
Also, it is not clear which files are actually unique to the project
when this is done.
.IP \(bu
Keep the project's unique configuration files in
.I config
(where they are normally anyway), and copies of the standard files in
.I config-std .
If the receiver has the latter already, they can be jettisoned easily,
otherwise installed in the project's StdConfigDir.
.LP
It is unclear whether any of these has any compelling advantage over
the others.
.NH
Beginning New Projects
.LP
If a new project has special configuration requirements, a file
.I README.{project-name}
should be created in
.I config
explaining what they are.
.NH 2
Define UseInstalled, or Not?
.LP
If a project wants to use the installed versions of
.I imake
and
.I makedepend ,
the initial makefile should be created from the the imakefile by executing
.I xmkmf
with no arguments (or just
.B \-C
.I configdir )
in the project top-level directory.
This will cause -DUseInstalled to be passed to
.I imake
which will in turn cause the version of IMAKE_CMD that
includes -DUseInstalled to be selected from the configuration files.
Thus -DUseInstalled will propagate nicely through all future ``make Makefile''
operations.
.LP
If
.I imake
and
.I makedepend
are to be built in the project source tree, UseInstalled should not be defined,
and you must have the source under the project.
.I imake
source is in
.I config
and
.I makedepend
source is in
.I util/makedepend
or
.I util/scripts ,
depending on the definition of UseCCMakeDepend.
Execute ``xmkmf \fB.\fR'' (note the \fB.\fR!) in the project top-level
directory.
This will run
.I imake
without -DUseInstalled, and the proper IMAKE_CMD in the config
files will be selected.
.LP
One problematic aspect of
.I xmkmf
is that it really should always pass -DUseInstalled, since it assumes there
is an installed version of
.I imake
anyway (it doesn't specify a path to it).
But then the problem becomes how to turn UseInstalled off later?
You can't run ``make Makefile'' because the version of IMAKE_CMD that
uses -DUseInstalled has already been selected.
.NH
Distribution and Availability
.LP
The standard WRPRC configuration files are (or will be soon)
available for anonymous
.I ftp
access in the
.I ~ftp/pub/imake-stuff
directory, in file
.I config-WRPRC.shar.Z
on host
.I indri.primate.wisc.edu
(Internet address 128.104.230.11).
These files illustrate the concepts discussed in this document.
.LP
This distribution also includes parts of the X11R4
.I mit/util
source tree (where
.I makedepend
and
.I xmkmf
are found) as well as
.I msub ,
a locally-developed utility for yanking the values of
variables directly from makefiles to produce target files from templates.
.LP
The X11R4 configuration files are in the same directory, in
.I config-X11R4.shar.Z .