Usenet 1994 October/usenetsourcesnewsgroupsinfomagicoctober1994disk2.iso/std_unix/archive/text0006.txt

Text File | 1994-03-07 | 7.8 KB | 267 lines

Submitted-by: [e-mail] (Nicholas M. Stoughton)

               USENIX Standards Report Editor

   Nicholas M. Stoughton <[e-mail]>, Report Editor


POSIX Test Methods


Fred Zlotnick <[e-mail]> reports on the January
10-14, 1994 meeting in Irvine, Ca.:

The requirement that POSIX working groups develop test
methods in parallel with their standards was suspended at
the April 1993 meeting, and then finally withdrawn at the
following July meeting.  Nevertheless there are two active
test methods activities and more in the works.  The active
groups, which met at the October POSIX meeting in Bethesda
and at the January meeting in Irvine, are 23 (which is
revising POSIX.3, the standard that describes how to write
test methods) and 23.2 (which is developing test methods
for POSIX.2, Shell and Utilities). Well, technically it
wasn't the 23.2 working group that met, but it's the same
group of people.  More about that later.  Both of these
groups are chaired by Lowell Johnson of Unisys.

Revision to POSIX.3

The 23 working group has been working on a revision to
POSIX.3 for about a year and a half.  Although POSIX.3 has
been used successfully to write test methods for POSIX.1,
and its methodology has formed the basis for quite a few
commercial test suites, the use of this methodology has
exposed a number of problems.  The purpose of this revision
is to deal with these problems:

   o+ It is difficult to use POSIX.3 to write test methods
     for standards that modify other standards.  Real-time
     (POSIX.1b, which used to be POSIX.4) is a good example.
     Because the real-time standard consists of a collection
     of optional extensions to POSIX.1, every assertion for
     real-time must be conditional (type C or type D).  But
     there are other conditions within many real-time
     assertions, and this makes the statement of each
     assertion in POSIX.3 format rather cumbersome.
     Moreover, some of the options of POSIX.1b place
     additional semantic requirements on POSIX.1 interfaces
     such as _o_p_e_n().  Writing the assertions to test these
     requirements raises questions not adequately addressed
     in POSIX.3-1991:  How should they be numbered? How
     should they be conditioned?  How should they be
     classified (assertion-typed)?

   o+ A number of the users of POSIX.3 found the standard to
     be quite terse, and consequently difficult to











- 2 -



     understand.  A number of related but distinct concepts
     in POSIX.3 were often confused by users of the
     standard.

   o+ ISO had difficulty with the terminology of POSIX.3,
     which is not always consistent with that of some other
     test methods standards at the international level.

   o+ POSIX.3 was originally designed, and is specified, as
     only applying to POSIX standards.  The IEEE's Portable
     Applications Standards Committee (PASC) currently
     manages a number of projects, only some of which fall
     under the POSIX umbrella.  yet the test methods
     methodology of POSIX.3 applies to, and should be
     specified for, other PASC working groups such as P1327
     and P1328.  In general, the scope of POSIX.3 should be
     broadened.

The 23 working group began the week in Irvine with Draft
2.0 of the revised standard.  This draft had been completed
by the group's technical editor, Anthony Cincotta of NIST,
just prior to the meeting.  By the end of the week the group
had agreed on a set of changes that, when completed, will
produce Draft 3.  This draft should be suitable for mock
ballot.

The basic methodology of assertion based testing has not
changed in this revision, but the form in which assertions
are written has changed drastically.  The familiar,
frequently misunderstood and often vilified 2x2 matrix of
assertion types is gone.  The syntax of an assertion now
closely resembles a conditional sentence, with possibly many
nested conditions.  If an assertion applies only when
conformance to a particular version of a standard (e.g.
POSIX.1- 1993) is being tested, a condition indicates this
fact.  If an assertion depends on support of an option (e.g.
job control) a condition indicates this fact.  Sometimes an
assertion may specify required behavior but may only be
testable if the implementation supports optional features
(such as certain appropriate privileges). If so, a condition
indicates this fact.  Assertions are now labeled by
assertion IDs rather than assertion numbers; an assertion ID
is a string.

The new assertion structure promises to make assertion
writing easier and to allow the structure of test methods
standards to more closely parallel the base standards
against which they are written.

POSIX.2 Test Methods












- 3 -



The 23.2 working group has been working on ballot
resolution for the ballot of Draft 8 for the last four
meetings.  This means that, according to IEEE rules, it's
not really the 23.2 committee that's meeting.  Ballot
resolution is done not by the committee that wrote the
draft, but by a group of technical reviewers.  They just
happen (in this case, as in most) to be many of the same
people.

Ballot resolution for Draft 8 has been a slow job for a
number of reasons.  One is the size of the task: there are
almost 9 assertions in Draft 8.  Another is that despite
its size, Draft 8 was incomplete, and a number of ballot
objections made note of this fact.  Some of the missing
pieces (like assertions for _y_a_c_c) have not been easy to
supply.  Nevertheless, part of the task of resolving these
objections is to fill in those missing pieces.  Yet another
problem is that participation in this working group has not
been as regular as one might like, although the October and
January meetings were quite well attended.

Besides the incompleteness of Draft 8, major issues in
ballot included the fact that in many places the test
methods must be locale dependent but the draft frequently
addressed testing only in the POSIX locale.  Moreover, it
was often not explicit about this fact.  Other problems
included the omission of reasons for classifying assertions
as extended, and the omission of clear references for
reference assertions.

By the end of the October meeting the reviewers had made
enough progress to enable the technical editor, Andrew
Twigger of UniSoft, to produce interim Draft 8.5.  This will
not be balloted, but it was useful as a working document at
the January meeting.  At that meeting the technical
reviewers completed ballot resolution.  The technical editor
now has to integrate the resulting changes to produce Draft
9, which will go out to ballot.  The one thing of which we
can be certain is that Draft 9 will be much more complete,
and much larger, than Draft 8.

Test Methods for POSIX.1b

At the Irvine meeting the PASC Sponsor Executive Committee
(SEC) approved a new Project Authorization Request (PAR) for
test methods. The PAR creates a POSIX 23.1b project under
the direction of the 23 working group.  Its goal is to
write test methods for POSIX.1b.

You may recall that during the ``test methods wars'' the
POSIX.4 working group was grandfathered out of the











- 4 -



requirement (since lifted) to develop test methods along
with base standards.  Thus there are no test methods, even
in draft form, for POSIX.1b.  Yet there is substantial
interest in the development of conformance tests for POSIX
Real Time, and such tests need a specification.  In Irvine a
number of organizations, including representatives of
several DoD agencies (DISA, JITC), committed to provide
reop these test methods.  Ken Thomas of JITC
has offered to serve as vice-chair of 23 for the direction
of the 23.1b effort.  Bruce Weiner of Mindcraft has
offered to act as technical editor for the test methods
document.

















































Volume-Number: Volume 34, Number 7