Usenet 1994 January/usenetsourcesnewsgroupsinfomagicjanuary1994.iso/sources/std_unix/volume.30/text0088.txt

Text File | 1993-03-11 | 3.2 KB | 64 lines

Submitted-by: [e-mail] (Jeffrey Kegler)

Steve Walli and others have mentioned the difficulties of producing
good test method standards.  I'd like to take another approach to this
issue.  I suspect that test method standards always undermine their
base standards, and the better constructed the test method standard
is, in its own terms, the more it undermines its base standard and the
POSIX effort as a whole.

Here's the problem.  In the absence of a test method standard, the
implementor of the base standard must come as close as possible to the
original standard.  If a test method standard exists, however, he need
only implement to pass the tests in that standard.  In effect the test
method standard implies a second, and considerably weaker base
standard.  As I will show below, this is true regardless of how
excellent a job was done in drafting the test method standard.

The second base standard, implied by the test method standard, is
likely to overshadow the original base standard, becoming the "real"
standard as far as the marketplace is concerned.  This is for three
reasons.  First, as I shall show below, it is weaker and easier to
meet.  Second, because meeting it results in certification, while the
rewards for meeting the original base standard are more concentrated
in the afterlife.  Much as the students in a course might regard its
real subject matter as those topics covered in the final exam, rather
than the lofty generalities of the syllabus outline, so implementors
will regard the real requirement as that for which they will be
tested.  Third, because passing an officially specified test will
strike most people as superior proof of quality to claims of adherence
to a standard.  Meeting the base standard implied in the test methods
is a "hard fact", while meeting the original base standard seems a
mere claim, a "soft", unproven assertion.  One reason for the
popularity of benchmarks, even outdated and irrelevant ones, is that
"hard fact" benchmark results usually carry the day over vaguer
analyses, even when the latter are far more relevant.

Standardized test methods imply, perhaps paradoxically, a considerably
less rigorous standard than their original base standard.  Almost all
standards of POSIX complexity will contain requirements which are not
practically testable.  The standardized test method, in effect,
repeals these.

The mathematical argument that the base standard implied by the test
method standard must be less rigorous than the original is
straightforward.  Most POSIX standards will specify a sufficiently
complex system to implement a Turing machine.  Testing that all
programs for a Turing machine correctly execute on a given black box
is impossible, even in theory -- it implies the solution of whole sets
of undecidable problems.

A POSIX standard need not be complex enough to implement a Turing
machine to be theoretically untestable.  And untestability in
practical terms is even more quickly arrived at.

It would seem to me to be prudent to produce test method standards on
a limited, trial basis, if at all.
--
Jeffrey Kegler, Independent UNIX Consultant, Algorists, Inc.
[e-mail] or uunet!algor2!jeffrey
137 E Fremont AVE #122, Sunnyvale CA 94087

Volume-Number: Volume 30, Number 90