home *** CD-ROM | disk | FTP | other *** search
Text File | 1991-12-22 | 149.0 KB | 5,589 lines |
- .rs
- .\" Troff code generated by TPS Convert from ITU Original Files
- .\" Not Copyright (~c) 1991
- .\"
- .\" Assumes tbl, eqn, MS macros, and lots of luck.
- .TA 1c 2c 3c 4c 5c 6c 7c 8c
- .ds CH
- .ds CF
- .EQ
- delim @@
- .EN
- .nr LL 40.5P
- .nr ll 40.5P
- .nr HM 3P
- .nr FM 6P
- .nr PO 4P
- .nr PD 9p
- .po 4P
-
- .rs
- \v'|.5i'
- .sp 1P
- .ce 1000
- \v'3P'
- SECTION\ 4
- .ce 0
- .sp 1P
- .ce 1000
- \fBCONFORMANCE\ TESTING\fR \v'1P'
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fBRecommendation X.290\fR
- .RT
- .sp 2P
- .ce 1000
- \fBOSI\ CONFORMANCE\ TESTING\ METHODOLOGY\ AND\ FRAMEWORK\ FOR\fR
- .EF '% Fascicle\ VIII.5\ \(em\ Rec.\ X.290''
- .OF '''Fascicle\ VIII.5\ \(em\ Rec.\ X.290 %'
- .ce 0
- .sp 1P
- .ce 1000
- \fBPROTOCOL\ RECOMMENDATIONS\ FOR\ CCITT\ APPLICATIONS\fR
- .FS
- Recommendation\ X.290 was developed in close collaboration with the ISO/IEC
- effort on OSI Conformance Testing Methodology and Framework. At the time
- of publication,
- Recommendation\ X.290 was aligned with the texts of DP\ 9646/1 and DP\ 9646/2.
- Since this work was at an early stage of development, changes should be
- expected. Consequently, users should exercise caution in applying this
- Recommendation.
- .FE
- .ce 0
- .sp 1P
- .ce 1000
- \fI(Melbourne, 1988)\fR
- .sp 9p
- .RT
- .ce 0
- .sp 1P
- .LP
- The\ CCITT,
- .sp 1P
- .RT
- .sp 2P
- .LP
- \fIconsidering\fR
- .sp 1P
- .RT
- .PP
- (a)
- that Recommendation X.200 defines the Reference Model of Open Systems for
- CCITT Applications;
- .PP
- (b)
- that the objective of OSI will not be completely
- achieved until systems dedicated to CCITT applications can be tested to
- determine whether they conform to the relevant OSI protocol Recommendations;
- .PP
- (c)
- that standardized test suites should be developed for
- each OSI protocol Recommendation as a means to:
- .LP
- \(em
- obtain wide acceptance and confidence in conformance test
- results produced by different testers,
- .LP
- \(em
- provide confidence in the interoperability of equipments
- which passed the standardized conformance
- tests;
- .PP
- (d)
- the need for defining an international Recommendation to specify the framework
- and general principles for the specification of
- conformance test suites and the testing of protocol implementations,
- .sp 2P
- .LP
- \fIunanimously recommends\fR
- .sp 1P
- .RT
- .PP
- (1)
- that the general principles, definition of terms and
- concepts of OSI protocol conformance testing shall be in accordance with
- Part\ 1 of this Recommendation;
- .PP
- (2)
- that the test methods, test suites and test notation
- shall be in accordance with Part\ 2 of this Recommendation.
- .LP
- .sp 3
- .bp
- .sp 1P
- .ce 1000
- CONTENTS
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fBPart\ 1\fR \ \(em\ \fBGeneral concepts\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 0
- Introduction
- .sp 9p
- .RT
- .LP
- 1
- Scope and field of application
- .LP
- 2
- References
- .sp 2P
- .LP
- SECTION\ 1\ \(em
- \fITerminology\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3
- Definitions
- .sp 9p
- .RT
- .LP
- 4
- Abbreviations
- .sp 2P
- .LP
- SECTION\ 2\ \(em
- \fIOverview\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 5
- The meaning of conformance in OSI*
- .sp 9p
- .RT
- .LP
- 6
- Conformance and testing
- .LP
- 7
- Test methods
- .LP
- 8
- Test suites
- .LP
- 9
- Relationships between, concepts and roles
- .LP
- 10
- Compliance
- .LP
- \fBPart\ 2\fR \ \(em\ \fBAbstract test suite specification\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 0
- Introduction
- .sp 9p
- .RT
- .LP
- 1
- Scope and field of application
- .LP
- 2
- References
- .LP
- 3
- Definitions
- .LP
- 4
- Abbreviations
- .LP
- 5
- Compliance
- .sp 2P
- .LP
- SECTION\ 1\ \(em
- \fIRequirements on protocol specifiers\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 6
- Conformance requirements in OSI* Recommendations*
- .sp 9p
- .RT
- .LP
- 7
- PICS proformas
- .sp 2P
- .LP
- SECTION\ 2\ \(em
- \fIRequirements on abstract test suite specifiers\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 8
- Test suite production process
- .sp 9p
- .RT
- .LP
- 9
- Determining the conformance requirements and PICS
- .LP
- 10
- Test suite structure
- .LP
- 11
- Generic test case specification
- .LP
- 12
- Abstract test methods
- .LP
- 13
- Specification of abstract test suites
- .LP
- 14
- Use of an abstract test suite specification
- .LP
- 15
- Test suite maintenance
- .sp 1P
- .LP
- \fIAnnex\ A\fR \(em
- Options
- .sp 9p
- .RT
- .LP
- \fIAnnex\ B\fR \(em
- Guidance for protocol Recommendations* writers
- .LP
- \fIAnnex\ C\fR \(em
- Incomplete static conformance requirements
- .LP
- \fIAnnex\ D\fR \(em
- Tree and tabular combined notation
- .sp 1P
- .LP
- \fIAppendix\ I\fR \(em
- Applicability of the test methods to
- OSI* protocols
- .sp 9p
- .RT
- .LP
- \fIAppendix\ II\fR
- \(em
- Index to definitions of terms
- .LP
- \fIAppendix\ III\fR \(em
- Examples for guidance for PICS proforma
- .LP
- \fIAppendix\ IV\fR
- \(em
- Example of choice of abstract
- test methods
- .bp
- .sp 1P
- .ce 1000
- \fBPart\ 1\ \(em\ General concepts\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB0\fR \fBIntroduction\fR
- .sp 1P
- .RT
- .PP
- The objective of OSI will not be completely achieved until systems can
- be tested to determine whether they conform to the relevant \*QOSI or related
- CCITT X\(hyseries or T\(hyseries\*U (hereafter abbreviated to \*QOSI*\*U)
- protocol
- \*Qstandard(s) or Recommendation(s)\*U (hereafter abbreviated to
- \*QRecommendation(s)*\*U).
- .PP
- Standardized test suites should be developed for each OSI*
- protocol Recommendation, for use by suppliers or implementors in self\(hytesting,
- by users of OSI products, by the Administrations* or other third party
- testers. This should lead to comparability and wide acceptance of test
- results produced by different testers, and thereby minimize the need for
- repeated
- conformance testing of the same system.
- .PP
- The standardization of test suites requires international
- definition and acceptance of a common testing methodology and appropriate
- testing methods and procedures. It is the purpose of this Recommendation to
- define the methodology, to provide a framework for specifying conformance
- test suites and define the procedures to be followed during testing.
- .PP
- Conformance testing involves testing both the capabilities and
- behaviour of an implementation and checking what is observed against the
- conformance requirements in the relevant Recommendation(s)* and against
- what the implementor states the implementation's capabilities are.
- .PP
- Conformance testing does not include assessment of the performance
- nor the robustness or reliability of an implementation. It cannot give
- judgements on the physical realization of the abstract service primitives,
- how a system is implemented, how it provides any requested service, nor
- the
- environment of the protocol implementation. It cannot, except in an indirect
- way, prove anything about the logical design of the protocol itself.
- .PP
- The purpose of conformance testing is to increase the probability that
- different implementations are able to interwork. This is achieved by verifying
- them by means of a protocol test suite, thereby increasing the confidence
- that each implementation conforms to the protocol specification. Confidence
- in
- conformance to a protocol specification is particularly important when
- equipment supplied by different vendors is required to interwork.
- .PP
- However, it should be borne in mind that the complexity of most
- protocols makes exhaustive testing impractical on both technical and economic
- grounds. Also, testing cannot guarantee conformance to a specification
- since it detects errors rather than their absence. Thus conformance to
- a test suite
- alone cannot guarantee interworking. What it does do is give confidence
- that an implementation has the required capabilities and that its behaviour
- conforms
- consistently in representative instances of communication.
- .PP
- It should be noted that the OSI reference model for CCITT
- applications (Recommendation\ X.200) states (in \(sc\ 4.3):
- .RT
- .LP
- \*QOnly the external behaviour of Open Systems is
- retained as the standard of behaviour of real Open
- Systems\*U.
- .PP
- This means that although aspects of both internal and external
- behaviour are described in OSI* Recommendations*, only the requirements on
- external behaviour have to be met by real open systems. Although some of the
- methods defined in this Recommendation do impose certain constraints on the
- implementor, for example that there be some means of realizing control and
- observation at one or more service access points, it should be noted that
- other methods defined herein do not impose such constraints.
- .PP
- However, in the case of partial OSI* end\(hysystems which
- provide OSI* protocols up to a specific layer boundary, it is desirable
- to test both the external behaviour of the implemented protocol entities and
- the potential of those entities for supporting correct external behaviour in
- higher layers.
- .PP
- Detailed investigation of relative benefits, efficiency and
- constraints of all methods is addressed in various parts of this
- Recommendation. However, any organization contemplating the use of test
- methods defined in this Recommendation in a context such as certification
- should
- carefully consider the constraints on applicability and the benefits of the
- different possible test methods.
- .PP
- Testing is voluntary as far as ISO/CCITT is concerned. Requirements
- for testing in procurement and other external contracts are not a matter for
- standardization.
- .bp
- .RT
- .sp 2P
- .LP
- \fB1\fR \fBScope and field of application\fR
- .sp 1P
- .RT
- .PP
- 1.1
- This Recommendation specifies a general methodology for testing the conformance
- to OSI* protocol Recommendation(s)* of products
- in which the Recommendation(s)* are claimed to be implemented. The
- methodology also applies to testing conformance to transfer syntax
- Recommendation(s)* to the extent that can be determined by testing each in
- combination with a specific OSI* protocol.
- .sp 9p
- .RT
- .PP
- 1.2
- This Recommendation is structured into two separate
- parts:
- .sp 9p
- .RT
- .PP
- \fIPart\ 1\fR \|identifies the different phases of conformance testing
- process, these phases being characterized by four major roles. These roles
- are:
- .LP
- a)
- the specification of abstract test suites for particular
- OSI* protocols;
- .LP
- b)
- the derivation of executable test suites and associated
- testing tools;
- .LP
- c)
- the role of a client of a test laboratory, having an
- implementation of OSI* protocols to be tested;
- .LP
- d)
- the operation of conformance testing, culminating in the
- production of a conformance test report which gives the results
- in terms of the Recommendation(s)* and the test suite(s)
- used.
- .PP
- Additionally, this Part provides tutorial material, together with definition
- of concepts and terms.
- .PP
- \fIPart\ 2\fR \| defines the requirements and guidance for the
- specification of abstract test suites for OSI* protocols.
- .RT
- .PP
- 1.3
- In both Parts of this Recommendation, the scope is limited to include only
- such information as is necessary to meet the following
- objectives:
- .sp 9p
- .RT
- .LP
- a)
- to achieve an adequate level of confidence in the tests as a guide to
- conformance;
- .LP
- b)
- to achieve comparability between the results of the
- corresponding tests applied in different places at different
- times;
- .LP
- c)
- to facilitate communication between the parties responsible for the roles
- described above.
- .PP
- 1.4
- One such aspect of this scope involves the framework for
- development of OSI* test suites. For example:
- .sp 9p
- .RT
- .LP
- a)
- how they should relate to the various types of conformance
- requirement;
- .LP
- b)
- the types of test to be standardized and the types not
- needing standardization;
- .LP
- c)
- the criteria for selecting tests for inclusion in a
- conformance test suite;
- .LP
- d)
- the notation to be used for defining tests;
- .LP
- e)
- the structure of a test suite.
- .PP
- 1.5
- Certification, an administrative procedure which may follow
- conformance testing, is outside the scope of this Recommendation. Requirements
- for procurement and contracts are also outside the scope of this
- Recommendation.
- .sp 9p
- .RT
- .PP
- 1.6
- The Physical layer and Media Access Control protocols are
- outside the field of application of this Recommendation.
- .sp 9p
- .RT
- .sp 2P
- .LP
- \fB2\fR \fBReferences\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- Recommendation\ X.200\ \(em
- \fIReference model of open systems\fR
- \fIinterconnection for CCITT applications\fR \|
- (see also ISO\ 7498).
- .sp 9p
- .RT
- .LP
- Recommendation\ X.210\ \(em
- \fIOpen systems interconnection layer\fR
- \fIservice definition conventions\fR \|
- (see also ISO TR\ 8509).
- .LP
- Recommendation\ X.209\ \(em
- \fISpecification of basic encoding rules for\fR
- \fIabstract syntax notation one (ASN.1)\fR \|
- (see also ISO\ 8825).
- .LP
- .rs
- .sp 2P
- .ad r
- BLANC
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .ce 1000
- SECTION\ 1\ \(em\ \fITerminology\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB3\fR \fBDefinitions\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.1
- \fIReference model definitions\fR
- .sp 9p
- .RT
- .PP
- This Recommendation is based upon the concepts developed in
- reference model of open systems interconnection for CCITT applications
- (CCITT X.200), and makes use of the following terms defined in that
- Recommendation:
- .RT
- .LP
- a)
- (N)\(hyentity
- .LP
- b)
- (N)\(hyservice
- .LP
- c)
- (N)\(hylayer
- .LP
- d)
- (N)\(hyprotocol
- .LP
- e)
- (N)\(hyservice\(hyaccess\(hypoint
- .LP
- f
- )
- (N)\(hyrelay
- .LP
- g)
- (N)\(hyprotocol\(hydata\(hyunit
- .LP
- h)
- (N)\(hyprotocol\(hycontrol\(hyinformation
- .LP
- i)
- (N)\(hyuser\(hydata
- .LP
- j)
- real open system
- .LP
- k)
- subnetwork
- .LP
- l)
- application\(hyentity
- .LP
- m)
- application\(hyservice\(hyelement
- .LP
- n)
- transfer syntax
- .LP
- o)
- physical layer
- .LP
- p)
- data link layer
- .LP
- q)
- network layer
- .LP
- r)
- transport layer
- .LP
- s)
- session layer
- .LP
- t)
- presentation layer
- .LP
- u)
- application layer
- .LP
- v)
- systems\(hymanagement
- .LP
- w)
- application\(hymanagement
- .LP
- x)
- layer\(hymanagement
- .sp 1P
- .LP
- 3.2
- \fITerms defined in other Recommendations\fR
- .sp 9p
- .RT
- .PP
- This Recommendation uses the following terms defined in the OSI
- Service Conventions (Recommendation\ X.210):
- .RT
- .LP
- a)
- service\(hyuser
- .LP
- b)
- service\(hyprovider
- .PP
- This Recommendation uses the following term defined in the ASN.1\ \(em
- Basic Encoding Rules Recommendation (Recommendation\ X.209):
- .LP
- c)
- encoding
- .sp 1P
- .LP
- 3.3
- \fIConformance testing definitions\fR
- .sp 9p
- .RT
- .PP
- For the purposes of this Recommendation the definitions in \(sc\(sc 3.4
- to 3.8 apply.
- .RT
- .sp 2P
- .LP
- 3.4
- \fIBasic terms\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.4.1
- \fBimplementation under test (IUT)\fR
- .sp 9p
- .RT
- .PP
- That part of a real open system which is to be studied by testing, which
- should be an implementation of one or more OSI* protocols in an adjacent
- user/provider relationship.
- .bp
- .RT
- .sp 1P
- .LP
- 3.4.2
- \fBsystem under test (SUT)\fR
- .sp 9p
- .RT
- .PP
- The real open system in which the IUT resides.
- .RT
- .sp 1P
- .LP
- 3.4.3
- \fBdynamic conformance requirements\fR
- .sp 9p
- .RT
- .PP
- All those requirements (and options) which determine what
- observable behaviour is permitted by the relevant OSI* Recommendation(s)* in
- instances of communication.
- .RT
- .sp 1P
- .LP
- 3.4.4
- \fBstatic conformance requirements\fR
- .sp 9p
- .RT
- .PP
- Constraints which are placed in OSI* Recommendations* to
- facilitate interworking by defining the requirements for the capabilities
- of an implementation.
- .PP
- \fINote\fR \ \(em\ Static conformance requirements may be at a broad level,
- such as the grouping of functional units and options into protocol classes,
- or at a detailed level, such as the ranges of values that are to be supported
- for
- specific parameters or timers.
- .RT
- .sp 1P
- .LP
- 3.4.5
- \fBcapabilities of an IUT\fR
- .sp 9p
- .RT
- .PP
- That set of functions and options in the relevant protocol(s) and, if appropriate,
- that set of facilities and options of the relevant service
- definition which are supported by the IUT.
- .RT
- .sp 1P
- .LP
- 3.4.6
- \fBprotocol implementation conformance statement (PICS)\fR
- .sp 9p
- .RT
- .PP
- A statement made by the supplier of an OSI* implementation or
- system, stating the capabilities and options which have been implemented,
- and any features which have been omitted.
- .RT
- .sp 1P
- .LP
- 3.4.7
- \fBPICS proforma\fR
- .sp 9p
- .RT
- .PP
- A document, in the form of a questionnaire, designed by the
- protocol specifier or conformance test suite specifier, which when completed
- for an OSI* implementation or system becomes the PICS.
- .RT
- .sp 1P
- .LP
- 3.4.8
- \fBprotocol implementation extra information for testing
- (PIXIT)\fR
- .sp 9p
- .RT
- .PP
- A statement made by a supplier or implementor of an IUT which
- contains or references all of the information (in addition to that given
- in the PICS) related to the IUT and its testing environment, which will
- enable the
- test laboratory to run the appropriate test suite against the IUT.
- .RT
- .sp 1P
- .LP
- 3.4.9
- \fBPIXIT proforma\fR
- .sp 9p
- .RT
- .PP
- A document, in the form of a questionnaire, provided by the test laboratory,
- which when completed during the preparation for testing becomes a PIXIT.
- .RT
- .sp 1P
- .LP
- 3.4.10
- \fBconforming implementation\fR
- .sp 9p
- .RT
- .PP
- An IUT which is shown to satisfy both static and dynamic
- conformance requirements, consistent with the capabilities stated in the
- PICS.
- .RT
- .sp 1P
- .LP
- 3.4.11
- \fBsystem conformance statement\fR
- .sp 9p
- .RT
- .PP
- A document summarizing which OSI* Recommendations* are implemented and
- to which conformance is claimed.
- .RT
- .sp 1P
- .LP
- 3.4.12
- \fBclient\fR
- .sp 9p
- .RT
- .PP
- The organization that submits a system or implementation for
- conformance testing.
- .RT
- .sp 1P
- .LP
- 3.4.13
- \fBtest laboratory\fR
- .sp 9p
- .RT
- .PP
- An organization that carries out conformance testing. This can be a third
- party, a user organization, an Administration*, or an identifiable part
- of the supplier organization.
- .bp
- .RT
- .sp 2P
- .LP
- 3.5
- \fITypes of testing\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.5.1
- \fBactive testing\fR
- .sp 9p
- .RT
- .PP
- The application of a test suite to an SUT, under controlled
- conditions, with the intention of observing the consequent actions of the
- IUT.
- .RT
- .sp 1P
- .LP
- 3.5.2
- \fBpassive testing\fR
- .sp 9p
- .RT
- .PP
- The observation of PDU activity on a link, and checking
- whether or not the observed behaviour is allowed by the relevant
- Recommendation(s)*.
- .RT
- .sp 1P
- .LP
- 3.5.3
- \fBmulti\(hylayer testing\fR
- .sp 9p
- .RT
- .PP
- Testing the behaviour of a multi\(hylayer IUT as a whole, rather than testing
- it layer by layer.
- .RT
- .sp 1P
- .LP
- 3.5.4
- \fBembedded testing\fR
- .sp 9p
- .RT
- .PP
- Testing the behaviour of a single layer within a multi\(hylayer IUT without
- accessing the layer boundaries for that layer within the IUT.
- .RT
- .sp 1P
- .LP
- 3.5.5
- \fBbasic interconnection testing\fR
- .sp 9p
- .RT
- .PP
- Limited testing of an IUT to determine whether or not there is
- sufficient conformance to the main features of the relevant protocol(s) for
- interconnection to be possible, without trying to perform thorough
- testing.
- .RT
- .sp 1P
- .LP
- 3.5.6
- \fBcapability testing\fR
- .sp 9p
- .RT
- .PP
- Testing to determine the capabilities of an IUT.
- .PP
- \fINote\fR \ \(em\ This involves checking all mandatory capabilities and
- those optional ones that are stated in the PICS as being supported, but
- not checking those optional ones which are stated in the PICS as not supported
- by the
- IUT.
- .RT
- .sp 1P
- .LP
- 3.5.7
- \fBstatic conformance review\fR
- .sp 9p
- .RT
- .PP
- A review of the extent to which the static conformance
- requirements are met by the IUT, by comparing the static conformance
- requirements expressed in the relevant Recommendation(s)* with the PICS
- and the results of any associated capability testing.
- .RT
- .sp 1P
- .LP
- 3.5.8
- \fBbehaviour testing\fR
- .sp 9p
- .RT
- .PP
- Testing the extent to which the dynamic conformance requirements are met
- by the IUT.
- .RT
- .sp 1P
- .LP
- 3.5.9
- \fBconformance testing\fR
- .sp 9p
- .RT
- .PP
- Testing the extent to which an IUT is a conforming
- implementation.
- .RT
- .sp 1P
- .LP
- 3.5.10
- \fBconformance assessment process\fR
- .sp 9p
- .RT
- .PP
- The complete process of accomplishing all conformance testing
- activities necessary to enable the conformance of an implementation or
- a system to one or more OSI* Recommendations* to be assessed. It includes
- the
- production of the PICS and PIXIT documents, preparation of the real tester
- and the SUT, the execution of one or more test suites, the analysis of
- the results and the production of the appropriate system and protocol conformance
- test
- reports.
- .RT
- .sp 2P
- .LP
- 3.6
- \fITerminology of test suites\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.6.1
- \fBabstract test method\fR
- .sp 9p
- .RT
- .PP
- The description of how an IUT is to be tested, given at an
- appropriate level of abstraction to make the description independent of any
- particular implementation of testing tools, but with enough detail to enable
- tests to be specified for this method.
- .bp
- .RT
- .sp 1P
- .LP
- 3.6.2
- \fBabstract testing methodology\fR
- .sp 9p
- .RT
- .PP
- An approach to describing and categorizing abstract test
- methods.
- .RT
- .sp 1P
- .LP
- 3.6.3
- \fBabstract test case\fR
- .sp 9p
- .RT
- .PP
- A complete and independent specification of the actions required to achieve
- a specific test purpose, defined at the level of abstraction of a
- particular abstract test method. It includes a preamble and a postamble to
- ensure starting and ending in a stable state (i.e.,\ a state which can be
- maintained almost indefinitely, such as the \*Qidle\*U state or \*Qdata
- transfer\*U
- state) and involves one or more consecutive or concurrent connections.
- .PP
- \fINote\ 1\fR \ \(em\ The specification should be complete in the sense
- that it is sufficient to enable a verdict to be assigned unambiguously
- to each
- potentially observable outcome (i.e.,\ sequence of test events).
- .PP
- \fINote\ 2\fR \ \(em\ The specification should be independent in the sense
- that it should be possible to execute the derived executable test case
- in isolation from other such test cases (i.e.,\ the specification should
- always include the possibility of starting and finishing in the \*Qidle\*U
- state \(em\ that is without any existing connections except permanent ones).
- For some test cases, there may be pre\(hyrequisites in the sense that execution
- might require some specific
- capabilities of the IUT, which should have been confirmed by results of the
- test cases executed earlier.
- .RT
- .sp 1P
- .LP
- 3.6.4
- \fBexecutable test case\fR
- .sp 9p
- .RT
- .PP
- A realization of an abstract test case.
- .PP
- \fINote\fR \ \(em\ In general the use of the word \*Qtest\*U will imply
- its normal English meaning. Sometimes it may be used as an abbreviation
- for abstract test case or executable test case. The context should make
- the meaning
- clear.
- .RT
- .sp 1P
- .LP
- 3.6.5
- \fBtest purpose\fR
- .sp 9p
- .RT
- .PP
- A description of the objective which an abstract test case is
- designed to achieve.
- .RT
- .sp 1P
- .LP
- 3.6.6
- \fBgeneric test case\fR
- .sp 9p
- .RT
- .PP
- A specification of the actions required to achieve a specific test purpose,
- defined by a test body together with a description of the initial
- state in which the test body is to start.
- .RT
- .sp 1P
- .LP
- 3.6.7
- \fBpreamble\fR
- .sp 9p
- .RT
- .PP
- The test steps needed to define the path from the starting stable state
- of the test case up to the initial state from which the test body will
- start.
- .RT
- .sp 1P
- .LP
- 3.6.8
- \fBtest body\fR
- .sp 9p
- .RT
- .PP
- The set of test steps that are essential in order to achieve the test purpose
- and assign verdicts to the possible outcomes.
- .RT
- .sp 1P
- .LP
- 3.6.9
- \fBpostamble\fR
- .sp 9p
- .RT
- .PP
- The test steps needed to define the paths from the end of the test body
- up to the finishing stable state for the test case.
- .RT
- .sp 1P
- .LP
- 3.6.10
- \fBtest step\fR
- .sp 9p
- .RT
- .PP
- A named subdivision of a test case, constructed from test events and/or
- other test steps, and used to modularize abstract test cases.
- .RT
- .sp 1P
- .LP
- 3.6.11
- \fBtest event\fR
- .sp 9p
- .RT
- .PP
- An indivisible unit of test specification at the level of
- abstraction of the specification (e.g.,\ sending or receiving a single
- PDU).
- .bp
- .RT
- .sp 1P
- .LP
- 3.6.12
- \fBtest suite\fR
- .sp 9p
- .RT
- .PP
- A complete set of test cases, possibly combined into nested test groups,
- that is necessary to perform conformance testing or basic
- interconnection testing for an IUT or protocol within an IUT.
- .RT
- .sp 1P
- .LP
- 3.6.13
- \fBtest case\fR
- .sp 9p
- .RT
- .PP
- A generic, abstract or executable test case.
- .RT
- .sp 1P
- .LP
- 3.6.14
- \fBtest group\fR
- .sp 9p
- .RT
- .PP
- A named set of related test cases.
- .RT
- .sp 1P
- .LP
- 3.6.15
- \fBgeneric test suite\fR
- .sp 9p
- .RT
- .PP
- A test suite composed of generic test cases, with the same
- coverage as the complete set of test purposes for the particular protocol,
- this being the set or a superset of the test purposes of any particular
- abstract test suite for the same protocol.
- .RT
- .sp 1P
- .LP
- 3.6.16
- \fBabstract test suite\fR
- .sp 9p
- .RT
- .PP
- A test suite composed of abstract test cases.
- .RT
- .sp 1P
- .LP
- 3.6.17
- \fBexecutable test suite\fR
- .sp 9p
- .RT
- .PP
- A test suite composed of executable test cases.
- .RT
- .sp 1P
- .LP
- 3.6.18
- \fBconformance test suite\fR
- .sp 9p
- .RT
- .PP
- A test suite for conformance testing of one or more
- OSI* protocols.
- .PP
- \fINote\fR \ \(em\ It should cover both capability testing and behaviour
- testing. It may be qualified by the adjectives: abstract, generic or
- executable, as appropriate. Unless stated otherwise, an \*Qabstract test
- suite\*U is meant.
- .RT
- .sp 1P
- .LP
- 3.6.19
- \fBbasic interconnection test suite\fR
- .sp 9p
- .RT
- .PP
- A test suite for basic interconnection testing of one or more
- OSI* protocols.
- .RT
- .sp 1P
- .LP
- 3.6.20
- \fBselected abstract test suite\fR
- .sp 9p
- .RT
- .PP
- The subset of an abstract test suite selected using a specific
- PICS.
- .RT
- .sp 1P
- .LP
- 3.6.21
- \fBselected executable test suite\fR
- .sp 9p
- .RT
- .PP
- The subset of an executable test suite selected using a specific PICS and
- corresponding to a selected abstract test suite.
- .RT
- .sp 1P
- .LP
- 3.6.22
- \fBparameterized abstract test case\fR
- .sp 9p
- .RT
- .PP
- An abstract test case in which all appropriate parameters have
- been supplied with values in accordance with a specific PICS and
- PIXIT.
- .RT
- .sp 1P
- .LP
- 3.6.23
- \fBparameterized executable test case\fR
- .sp 9p
- .RT
- .PP
- An executable test case in which all appropriate parameters have been supplied
- with values in accordance with a specific PICS and
- PIXIT.
- .RT
- .sp 1P
- .LP
- 3.6.24
- \fBparameterized abstract test suite\fR
- .sp 9p
- .RT
- .PP
- A selected abstract test suite in which all test cases have been made parameterized
- abstract test cases for the appropriate PICS and
- PIXIT.
- .bp
- .RT
- .sp 1P
- .LP
- 3.6.25
- \fBparameterized executable test suite\fR
- .sp 9p
- .RT
- .PP
- A selected executable test suite in which all test cases have been made
- parameterized executable test cases for the appropriate PICS and PIXIT,
- and corresponding to a parameterized abstract test suite.
- .RT
- .sp 2P
- .LP
- 3.7
- \fITerminology of results\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.7.1
- \fBrepeatability (of results)\fR
- .sp 9p
- .RT
- .PP
- Characteristic of a test case, such that repeated executions on
- the same IUT lead to the same verdict, and by extension a characteristic
- of a test suite.
- .RT
- .sp 1P
- .LP
- 3.7.2
- \fBcomparability (of results)\fR
- .sp 9p
- .RT
- .PP
- Characteristic of conformance assessment processes, such that
- their execution on the same IUT, in different test environments, leads
- to the same overall summary.
- .RT
- .sp 1P
- .LP
- 3.7.3
- \fBoutcome\fR
- .sp 9p
- .RT
- .PP
- A sequence of test events together with the associated
- input/output, either identified by an abstract test case specifier, or
- observed during test execution.
- .RT
- .sp 1P
- .LP
- 3.7.4
- \fBforeseen outcome\fR
- .sp 9p
- .RT
- .PP
- An outcome identified or categorized in the abstract test case
- specification.
- .RT
- .sp 1P
- .LP
- 3.7.5
- \fBunforeseen outcome\fR
- .sp 9p
- .RT
- .PP
- An outcome not identified or categorized in the abstract test case specification.
- .RT
- .sp 1P
- .LP
- 3.7.6
- \fBverdict\fR
- .sp 9p
- .RT
- .PP
- Statement of \*Qpass\*U, \*Qfail\*U or \*Qinconclusive\*U concerning
- conformance of an IUT with respect to a test case that has been executed and
- which is specified in the abstract test suite.
- .RT
- .sp 1P
- .LP
- 3.7.7
- \fBsystem conformance test report (SCTR)\fR
- .sp 9p
- .RT
- .PP
- A document written at the end of the conformance assessment
- process, giving the overall summary of the conformance of the system to
- the set of protocols for which conformance testing was carried out.
- .RT
- .sp 1P
- .LP
- 3.7.8
- \fBprotocol conformance test report (PCTR)\fR
- .sp 9p
- .RT
- .PP
- A document written at the end of the conformance assessment
- process, giving the details of the testing carried out for a particular
- protocol, including the identification of the abstract test cases for which
- corresponding executable test cases were run and for each test case the test
- purpose and verdict.
- .RT
- .sp 1P
- .LP
- 3.7.9
- \fBvalid test event\fR
- .sp 9p
- .RT
- .PP
- A test event which is allowed by the protocol Recommendation*,
- being both syntactically correct and occurring or arriving in an allowed
- context in an observed outcome.
- .RT
- .sp 1P
- .LP
- 3.7.10
- \fBsyntactically invalid test event\fR
- .sp 9p
- .RT
- .PP
- A test event which syntactically is not allowed by the protocol
- Recommendation*.
- .PP
- \fINote\fR \ \(em\ The use of \*Qinvalid test event\*U is deprecated.
- .RT
- .sp 1P
- .LP
- 3.7.11
- \fBinopportune test event\fR
- .sp 9p
- .RT
- .PP
- A test event which, although syntactically correct, occurs or
- arrives at a point in an observed outcome when not allowed to do so by the
- protocol Recommendation*.
- .bp
- .RT
- .sp 1P
- .LP
- 3.7.12
- \fB\*Qpass\*U verdict\fR
- .sp 9p
- .RT
- .PP
- A verdict given when the observed outcome satisfies the test
- purpose and is valid with respect to the relevant Recommendation(s)*
- and with respect to the PICS.
- .RT
- .sp 1P
- .LP
- 3.7.13
- \fB\*Qfail\*U verdict\fR
- .sp 9p
- .RT
- .PP
- A verdict given when the observed outcome is syntactically invalid or inopportune
- with respect to the relevant Recommendation(s)* or the
- PICS.
- .RT
- .sp 1P
- .LP
- 3.7.14
- \fB\*Qinconclusive\*U verdict\fR
- .sp 9p
- .RT
- .PP
- A verdict given when the observed outcome is valid with respect to the
- relevant Recommendation(s)* but prevents the test purpose from being
- accomplished.
- .RT
- .sp 1P
- .LP
- 3.7.15
- \fBconformance log\fR
- .sp 9p
- .RT
- .PP
- A record of sufficient information necessary to verify verdict
- assignments as a result of conformance testing.
- .RT
- .sp 2P
- .LP
- 3.8
- \fITerminology of test methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 3.8.1
- \fBpoint of control and observation (PCO)\fR
- .sp 9p
- .RT
- .PP
- A point at which control and observation is specified in a test
- case.
- .RT
- .sp 1P
- .LP
- 3.8.2
- \fBlower tester\fR
- .sp 9p
- .RT
- .PP
- The abstraction of the means of providing, during test execution, control
- and observation at the appropriate PCO either below the IUT or remote from
- the IUT, as defined by the chosen abstract test method.
- .RT
- .sp 1P
- .LP
- 3.8.3
- \fBupper tester\fR
- .sp 9p
- .RT
- .PP
- The abstraction of the means of providing, during test execution, control
- and observation of the upper service boundary of the IUT, plus the
- control and observation of any relevant abstract local primitive.
- .RT
- .sp 1P
- .LP
- 3.8.4
- \fBabstract (N)\(hyservice\(hyprimitive ((N)\(hyASP)\fR
- .sp 9p
- .RT
- .PP
- An implementation independent description of an interaction
- between a service\(hyuser and a service\(hyprovider at an (N)\(hyservice
- boundary, as
- defined in an OSI* service definition Recommendation*.
- .RT
- .sp 1P
- .LP
- 3.8.5
- \fBabstract local primitive (ALP)\fR
- .sp 9p
- .RT
- .PP
- An abbreviation for a description of control and/or observation to be performed
- by the upper tester, which cannot be described in terms of ASPs
- but which relates to events or states defined within the protocol
- Recommendation(s)* relevant to the IUT.
- .PP
- \fINote\fR \ \(em\ The PIXIT will indicate whether or not a particular
- ALP can be realized within the SUT. The ability of the SUT to support particular
- ALPs as specified in the PIXIT will be used as a criterion in the test
- selection
- process.
- .RT
- .sp 1P
- .LP
- 3.8.6
- \fBtest coordination procedures\fR
- .sp 9p
- .RT
- .PP
- The rules for cooperation between the lower and upper testers
- during testing.
- .RT
- .sp 1P
- .LP
- 3.8.7
- \fBtest management protocol\fR
- .sp 9p
- .RT
- .PP
- A protocol which is used as a realization of the test coordination procedures
- for a particular test suite.
- .RT
- .sp 1P
- .LP
- 3.8.8
- \fBlocal test methods\fR
- .sp 9p
- .RT
- .PP
- Abstract test methods in which the PCOs are directly at the layer boundaries
- of the IUT.
- .bp
- .RT
- .sp 1P
- .LP
- 3.8.9
- \fBexternal test methods\fR
- .sp 9p
- .RT
- .PP
- Abstract test methods in which the lower tester is separate from the SUT
- and communicates with it via an appropriate lower layer
- service\(hyprovider.
- .PP
- \fINote\fR \ \(em\ The service\(hyprovider is immediately beneath the (lowest
- layer) protocol which is the focus of the testing, and may involve multiple
- OSI layers.
- .RT
- .sp 1P
- .LP
- 3.8.10
- \fBdistributed test method\fR
- .sp 9p
- .RT
- .PP
- An external test method in which there is a PCO at the layer
- boundary at the top of the IUT.
- .RT
- .sp 1P
- .LP
- 3.8.11
- \fBcoordinated test method\fR
- .sp 9p
- .RT
- .PP
- An external test method for which a standardized test management protocol
- is defined as the realization of the test coordination procedures,
- enabling the control and observation to be specified solely in terms of the
- lower tester activity, including the control and observation of test management
- PDUs.
- .RT
- .sp 1P
- .LP
- 3.8.12
- \fBremote test method\fR
- .sp 9p
- .RT
- .PP
- An external method in which there is neither a PCO above the IUT nor a
- standardized test management protocol; some requirements for test
- coordination procedures may be implied or informally expressed in the abstract
- test suite but no assumption is made regarding their feasibility or
- realization.
- .RT
- .sp 1P
- .LP
- 3.8.13
- \fBreal tester\fR
- .sp 9p
- .RT
- .PP
- The realization of the lower tester, plus either the definition
- or the realization of the upper tester, plus the definition of the test
- coordination procedures, as appropriate to a particular test
- method.
- .RT
- .sp 1P
- .LP
- 3.8.14
- \fBtest realizer\fR
- .sp 9p
- .RT
- .PP
- An organization which takes responsibility for providing, in a
- form independent of client and IUT, the means of testing IUTs in conformance
- with the abstract test suite.
- .RT
- .sp 2P
- .LP
- \fB4\fR \fBAbbreviations\fR
- .sp 1P
- .RT
- .PP
- For the purposes of this Recommendation the following
- abbreviations apply:
- .RT
- .LP
- Administration*
- Administration or recognized private
- operating agency
- .LP
- ALP
- abstract local primitive
- .LP
- ASP
- abstract service primitive
- .LP
- DTE
- data terminal equipment
- .LP
- IUT
- implementation under test
- .LP
- OSI
- open systems interconnection
- .LP
- OSI*
- OSI or related CCITT X\(hyseries or T\(hyseries Recommendations
- .LP
- PCO
- point of control and observation
- .LP
- PCTR
- protocol conformance test report
- .LP
- PDU
- protocol data unit
- .LP
- PICS
- protocol implementation conformance statement
- .LP
- PIXIT
- protocol implementation extra information for testing
- .LP
- BBSAP
- service access point
- .LP
- SCTR
- system conformance test report
- .LP
- Recommendation*
- Standard or Recommendation
- .LP
- SUT
- system under test
- .LP
- TM\(hyPDU
- test management PDU
- .LP
- .rs
- .sp 02P
- .ad r
- BLANC
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .ce 1000
- SECTION\ 2\ \(em\ \fIOverview\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB5\fR \fBThe meaning of conformance in OSI\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 5.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- In the context of OSI*, a real system is said to exhibit
- conformance if it complies with the requirements of applicable OSI*
- Recommendations* in its communication with other real systems.
- .PP
- Applicable OSI* Recommendations* include protocol Recommendations*,
- and transfer syntax Recommendations* inasmuch as they are implemented
- in conjunction with protocols.
- .PP
- OSI* Recommendations* form a set of interrelated
- Recommendations* which together define behaviour of open systems in
- their communication. Conformance of a real system will, therefore, be expressed
- at two levels, conformance to each individual Recommendation*, and
- conformance to the set.
- .PP
- \fINote\fR \ \(em\ If the implementation is based on a predefined set of
- Recommendations*, often referred to as a functional standard or profile, the
- concept of conformance can be extended to specific requirements expressed in
- the functional standard or profile, as long as they do not conflict with the
- requirements of the base Recommendations*.
- .RT
- .sp 2P
- .LP
- 5.2
- \fIConformance requirements\fR
- .sp 1P
- .RT
- .PP
- 5.2.1
- The conformance requirements in a Recommendation* can be:
- .sp 9p
- .RT
- .LP
- a)
- mandatory requirements: these are to be observed in all
- cases;
- .LP
- b)
- conditional requirements: these are to be observed if the
- conditions set out in the Recommendation* apply;
- .LP
- c)
- options: these can be selected to suit the implementation, provided that
- any requirements applicable to the option are observed. More
- information on options is provided in Annex\ A.
- .PP
- For example, CCITT essential facilities are mandatory
- requirements; additional facilities can be either conditional or optional
- requirements.
- .PP
- \fINote\fR \ \(em\ The CCITT terms \*Qessential facilities\*U and \*Qadditional
- facilities\*U need to be considered in the context of the scope of the CCITT
- Recommendation concerned; in many cases, essential facilities are mandatory
- for networks but not for DTEs.
- .RT
- .PP
- 5.2.2
- Furthermore, conformance requirements in a Recommendation* can be stated
- .sp 9p
- .RT
- .LP
- a)
- positively: they state what shall be done;
- .LP
- b)
- negatively (prohibitions): they state what shall not be
- done.
- .PP
- 5.2.3
- Finally, conformance requirements fall into two
- groups:
- .sp 9p
- .RT
- .LP
- a)
- static conformance requirements;
- .LP
- b)
- dynamic conformance requirements;
- .LP
- these are discussed in \(sc\(sc 5.3. and 5.5, respectively.
- .sp 1P
- .LP
- 5.3
- \fIStatic conformance requirements\fR
- .sp 9p
- .RT
- .PP
- Static conformance requirements are those that define the allowed minimum
- capabilities of an implementation, in order to facilitate interworking.
- These requirements may be at a broad level, such as the grouping of functional
- units and options into protocol classes, or at a detailed level, such as
- a
- range of values that have to be supported for specific parameters of
- timers.
- .bp
- .PP
- Static conformance requirements and options in OSI* Recommendations* can
- be of two varieties:
- .RT
- .LP
- a)
- those which determine the capabilities to be included in
- the implementation of the particular protocol;
- .LP
- b)
- those which determine multi\(hylayer dependencies, e.g., those which
- place constraints on the capabilities of the underlying layers of the
- system in which the protocol implementation resides. These are likely to be
- found in upper layer Recommendations*.
- .PP
- All capabilities not explicitly stated as static conformance
- requirements are to be regarded as optional.
- .sp 1P
- .LP
- 5.4
- \fIProtocol implementation conformance statement (PICS)\fR
- .sp 9p
- .RT
- .PP
- To evaluate the conformance of a particular implementation, it is necessary
- to have a statement of the capabilities and options which have been implemented,
- and any features which have been omitted, so that the
- implementation can be tested for conformance against relevant requirements,
- and against those requirements only. Such a statement is called a Protocol
- Implementation Conformance Statement (PICS).
- .PP
- In a PICS there should be a distinction between the following
- categories of information which it may contain:
- .RT
- .LP
- a)
- information related to the mandatory, optional and
- conditional static conformance requirements of the protocol
- itself;
- .LP
- b)
- information related to the mandatory, optional and
- conditional static conformance requirements for multi\(hylayer
- dependencies.
- .PP
- If a set of interrelated OSI* protocol Recommendations* has been implemented
- in a system, a PICS is needed for each protocol. A System
- Conformance Statement will also be necessary, summarizing all protocols
- in the system for each of which a distinct PICS is provided.
- .sp 1P
- .LP
- 5.5
- \fIDynamic conformance requirements\fR
- .sp 9p
- .RT
- .PP
- Dynamic conformance requirements are all those requirements (and
- options) which determine what observable behaviour is permitted by the
- relevant OSI* Recommendation(s)* in instances of communication. They form
- the bulk
- of each OSI* protocol Recommendation*. They define the set of
- allowable behaviours of an implementation or real system. This set defines
- the maximum capability that a conforming implementation or real system
- can have
- within the terms of the OSI* protocol Recommendation*.
- .PP
- A system exhibits dynamic conformance in an instance of communication if
- its behaviour is a member of the set of all behaviours permitted by the
- relevant OSI* protocol Recommendation(s)* in a way which is consistent
- with the PICS.
- .RT
- .sp 1P
- .LP
- 5.6
- \fIA conforming system\fR
- .sp 9p
- .RT
- .PP
- A conforming system or implementation is one which is shown to
- satisfy both static and dynamic conformance requirements, consistent with
- the capabilities stated in the PICS, for each protocol declared in the
- System
- Conformance Statement.
- .RT
- .sp 2P
- .LP
- 5.7
- \fIInterworking and conformance\fR
- .sp 1P
- .RT
- .PP
- 5.7.1
- The primary purpose of conformance testing is to increase the
- probability that different implementations are able to interwork.
- .sp 9p
- .RT
- .PP
- Successful interworking of two or more real open systems is more likely
- to be achieved if they all conform to the same subset of an OSI*
- Recommendation*, or to the same selection of OSI* Recommendations*, than if
- they do not.
- .PP
- In order to prepare two or more systems to interwork successfully, it is
- recommended that a comparison be made of the System Conformance Statements
- and PICSs of these systems.
- .PP
- If there is more than one version of a relevant OSI* Recommendation* indicated
- in the PICSs, the differences between the versions need to be
- identified and their implications for consideration, including their use in
- combination with other Recommendations*.
- .RT
- .PP
- 5.7.2
- While conformance is a necessary condition, it is not on its
- own a sufficient condition to guarantee interworking capability. Even if two
- implementations conform to the same OSI* protocol Recommendation*, they may
- fail to interwork because of factors outside the scope of that
- Recommendation.
- .bp
- .sp 9p
- .RT
- .PP
- Trial interworking is recommended in order to detect these
- factors. Further information to assist interworking between two systems
- can be obtained by extending the PICS comparison to other relevant information,
- including test reports and PIXIT (see \(sc\ 6.2). The comparison can focus
- on:
- .LP
- a)
- additional mechanisms claimed to work around known
- ambiguities or deficiencies not yet corrected in the
- Recommendations* or in peer real systems, e.g.\ solution
- of multi\(hylayer problems;
- .LP
- b)
- selection of free options which are not taken into account in the static
- conformance requirements of the
- Recommendations*;
- .LP
- c)
- the existence of timers not specified in the
- Recommendation* and their associated values.
- .PP
- \fINote\fR \ \(em\ The comparison can be made between two individual
- systems, between two or more types of product, or, for the PICS comparison
- only, between two or more specifications for procurement, permissions to
- connect,\ etc.
- .LP
- \fB6\fR \fBConformance and testing\fR
- .sp 1P
- .RT
- .sp 2P
- .LP
- 6.1
- \fIObjectives of conformance testing\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 6.1.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- Conformance testing as discussed in this Recommendation is focused on testing
- for conformance to OSI* protocol Recommendations*. However, it also applies
- to testing for conformance to OSI* transfer syntax Recommendations*, to
- the extent that this can be carried out by testing the transfer syntax
- in
- combination with an OSI* protocol.
- .PP
- In principle, the objective of conformance testing is to establish
- whether the implementation being tested conforms to the specification in the
- relevant Recommendation*. Practical limitations make it impossible to
- be exhaustive, and economic considerations may restrict testing still
- further.
- .PP
- Therefore, this Recommendation distinguishes four types of testing,
- according to the extent to which they provide an indication of
- conformance:
- .RT
- .LP
- a)
- basic interconnection tests, which provide \fIprima facie\fR \|
- evidence that an IUT conforms;
- .LP
- b)
- capability tests, which check that the observable
- capabilities of the IUT are in accordance with the static conformance
- requirements and the capabilities claimed in the PICS;
- .LP
- c)
- behaviour tests, which endeavour to provide testing which
- is as comprehensive as possible over the full range of dynamic conformance
- requirements specified by the Recommendation*, within the capabilities
- of the IUT;
- .LP
- d)
- conformance resolution tests, which probe in depth the
- conformance of an IUT to particular requirements, to provide a definite
- yesB/Fno answer and diagnostic information in relation to specific conformance
- issues; such tests are not standardized.
- .sp 2P
- .LP
- 6.1.2
- \fIBasic interconnection tests\fR
- .sp 1P
- .RT
- .PP
- 6.1.2.1
- Basic interconnection tests provide limited testing of an IUT in relation
- to the main features in a Recommendation*, to establish that there is sufficient
- conformance for interconnection to be possible, without trying to
- perform thorough testing.
- .sp 9p
- .RT
- .PP
- 6.1.2.2
- Basic interconnection tests are appropriate:
- .sp 9p
- .RT
- .LP
- a)
- for detecting severe cases of non\(hyconformance;
- .LP
- b)
- as a preliminary filter before undertaking more costly
- tests;
- .LP
- c)
- to give a \fIprima facie\fR \| indication that an implementation which
- has passed full conformance tests in one environment still conforms in
- a new environment (e.g.\ before testing an (N)\(hyimplementation, to check
- that a
- tested (N\ \(em\ 1)\(hyimplementation has not undergone any severe change
- due to being linked to the (N)\(hyimplementation);
- .LP
- d)
- for use by users of implementations, to determine whether
- the implementations appear to be usable for communication with other conforming
- implementations, e.g.\ as a preliminary to data interchange.
- .bp
- .PP
- 6.1.2.3
- Basic interconnection tests are inappropriate:
- .sp 9p
- .RT
- .LP
- a)
- as a basis for claims of conformance by the supplier of an implementation;
- .LP
- b)
- as a means of arbitration to determine causes for
- communications failure.
- .PP
- 6.1.2.4
- Basic interconnection tests should be standardized as either a very small
- test suite or a subset of a conformance test suite (including
- capability and behaviour tests). They can be used on their own or together
- with a conformance test suite. The existence and execution of basic interconnection
- tests are optional.
- .sp 9p
- .RT
- .sp 2P
- .LP
- 6.1.3
- \fICapability tests\fR
- .sp 1P
- .RT
- .PP
- 6.1.3.1
- Capability tests provide limited testing of each of the static
- conformance requirements in a Recommendation*, to ascertain what capabilities
- of the IUT can be observed and to check that those observable capabilities
- are valid with respect to the static conformance requirements and the PICS.
- .sp 9p
- .RT
- .PP
- 6.1.3.2
- Capability tests are appropriate:
- .sp 9p
- .RT
- .LP
- a)
- to check as far as possible the consistency of the PICS
- with the IUT;
- .LP
- b)
- as a preliminary filter before undertaking more in\(hydepth
- and costly testing;
- .LP
- c)
- to check that the capabilities of the IUT are consistent
- with the static conformance requirements;
- .LP
- d)
- to enable efficient selection of behaviour tests to be made for a particular
- IUT;
- .LP
- e)
- when taken together with behaviour tests, as a basis for
- claims of conformance.
- .PP
- 6.1.3.3
- Capability tests are inappropriate:
- .sp 9p
- .RT
- .LP
- a)
- on their own, as a basis for claims of conformance by the
- supplier of an implementation;
- .LP
- b)
- for testing in detail the behaviour associated with each
- capability which has been implemented or not implemented;
- .LP
- c)
- for resolution of problems experienced during live usage or where other
- tests indicate possible non\(hyconformance even though the capability tests
- have been satisfied.
- .PP
- 6.1.3.4
- Capability tests are standardized within a conformance test
- suite. They can either be separated into their own test group(s) or merged
- with the behaviour tests.
- .sp 9p
- .RT
- .sp 2P
- .LP
- 6.1.4
- \fIBehaviour tests\fR
- .sp 1P
- .RT
- .PP
- 6.1.4.1
- Behaviour tests test an implementation as thoroughly as is
- practical, over the full range of dynamic conformance requirements
- specified in a Recommendation*. Since the number of possible combinations of
- events and timing of events is infinite, such testing cannot be exhaustive.
- There is a further limitation, namely that these tests are designed
- to be run collectively in a single test environment, so that any faults
- which are difficult or impossible to detect in that environment are likely
- to be
- missed. Therefore, it is possible that a non\(hyconforming implementation
- passes the conformance test suite; one aim of the test suite design is
- to minimize the number of times that this occurs.
- .sp 9p
- .RT
- .PP
- 6.1.4.2
- Behaviour tests are appropriate, when taken together with
- capability tests, as a basis for the conformance assessment process.
- .sp 9p
- .RT
- .PP
- 6.1.4.3
- Behaviour tests are inappropriate for resolution of problems
- experienced during live usage or where other tests indicate possible
- non\(hyconformance even though the behaviour tests have been satisfied.
- .sp 9p
- .RT
- .PP
- 6.1.4.4
- Behaviour tests are standardized as the bulk of a conformance
- test suite.
- .sp 9p
- .RT
- .PP
- \fINote\fR \ \(em\ Behaviour tests include tests for valid behaviour by
- the IUT in response to valid, inopportune and syntactically invalid protocol
- behaviour by the real tester. This includes testing the rejection by the
- IUT of attempts to use features (capabilities) which are stated in the
- PICS as being not implemented. Thus, capability tests do not need to include
- tests for
- capabilities omitted from the PICS.
- .bp
- .sp 2P
- .LP
- 6.1.5
- \fIConformance resolution tests\fR
- .sp 1P
- .RT
- .PP
- 6.1.5.1
- Conformance resolution tests provide diagnostic answers, as near to definitive
- as possible, to the resolution of whether an implementation
- satisfies particular requirements. Because of the problems of exhaustiveness
- noted in \(sc\ 6.1.4.1, the definite answers are gained at the expense
- of confining tests to a narrow field.
- .sp 9p
- .RT
- .PP
- 6.1.5.2
- The test architecture and test method will normally be
- chosen specifically for the requirements to be tested, and need not be ones
- that are generally useful for other requirements. They may even be ones that
- are regarded as being unacceptable for (standardized) abstract conformance
- test suites, e.g.\ involving implementation\(hyspecific methods using,
- say, the
- diagnostic and debugging facilities of the specific operating system.
- .sp 9p
- .RT
- .PP
- 6.1.5.3
- The distinction between behaviour tests and conformance
- resolution tests may be illustrated by the case of an event such as a Reset.
- The behaviour tests may include only a representative selection of conditions
- under which a Reset might occur, and may fail to detect incorrect behaviour
- in other circumstances. The conformance resolution tests would be confined
- to
- conditions under which incorrect behaviour was already suspected to occur,
- and would confirm whether or not the suspicions were correct.
- .sp 9p
- .RT
- .PP
- 6.1.5.4
- Conformance resolution tests are appropriate:
- .sp 9p
- .RT
- .LP
- a)
- for providing a yesB/Fno answer in a strictly confined and
- previously identified situation (e.g.\ during implementation
- development, to check whether a particular feature has been
- correctly implemented, or during operational use, to investigate the cause
- of problems);
- .LP
- b)
- as a means for identifying and offering resolutions for
- deficiencies in a current conformance test suite.
- .PP
- 6.1.5.5
- Conformance resolution tests are inappropriate as a
- basis for judging whether or not an implementation conforms
- overall.
- .sp 9p
- .RT
- .PP
- 6.1.5.6
- Conformance resolution tests are not standardized.
- .sp 9p
- .RT
- .PP
- \fINote\fR on \(sc 6.1\ \(em\ As a by\(hyproduct of conformance testing,
- errors and deficiencies in protocol Recommendations* may be identified.
- .sp 1P
- .LP
- 6.2
- \fIProtocol implementation extra information for testing\fR
- \fI(PIXIT)\fR
- .sp 9p
- .RT
- .PP
- In order to test a protocol implementation, the test laboratory
- will require information relating to the IUT and its testing environment in
- addition to that provided by the PICS. This \*QProtocol Implementation eXtra
- Information for Testing\*U (PIXIT) will be provided by the client submitting
- the implementation for testing, as a result of consultation with the test
- laboratory.
- .PP
- The PIXIT may contain the following information:
- .RT
- .LP
- a)
- information needed by the test laboratory in order to be
- able to run the appropriate test suite on the specific system
- (e.g.,\ information related to the test method to be used to run the test
- cases, addressing information);
- .LP
- b)
- information already mentioned in the PICS and which needs
- to be made precise (e.g.\ a timer value range which is declared as a parameter
- in the PICS should be specified in the PIXIT);
- .LP
- c)
- information to help determine which capabilities stated in the PICS as
- being supported are testable and which are untestable;
- .LP
- d)
- other administrative matters (e.g. the IUT identifier,
- reference to the related PICS).
- .PP
- The PIXIT should not conflict with the appropriate PICS.
- .PP
- The abstract test suite specifier, test realizer and test
- laboratory will all contribute to the development of the PIXIT
- proforma.
- .bp
- .RT
- .sp 2P
- .LP
- 6.3
- \fIConformance assessment process outline\fR
- .sp 1P
- .RT
- .PP
- 6.3.1
- The main feature of the conformance assessment process is a
- configuration of equipment allowing exchanges of information between
- the IUT and a real tester. These are controlled and observed by the real
- tester.
- .sp 9p
- .RT
- .PP
- 6.3.2
- In conceptual outline, conformance testing should include several steps,
- involving both static conformance reviews and live testing phases,
- culminating in the production of a test report which is as thorough as is
- practical.
- .sp 9p
- .RT
- .PP
- 6.3.3
- These steps are:
- .sp 9p
- .RT
- .LP
- a)
- analysis of the PICS;
- .LP
- b)
- test selection and parameterization;
- .LP
- c)
- basic interconnection testing (optional);
- .LP
- d)
- capability testing;
- .LP
- e)
- behaviour testing;
- .LP
- f
- )
- review and analysis of test results;
- .LP
- g)
- synthesis, conclusions and conformance test report
- production.
- .PP
- These are illustrated in Figure 1/X.290 Part\ 1.
- .PP
- Prior to the execution of any of the tests, the IUT's PICS and PIXIT are
- input to the test case selection and parameterization process.
- .RT
- .LP
- .rs
- .sp 33P
- .ad r
- \fBFigure 1/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .LP
- 6.4
- \fIAnalysis of results\fR
- .sp 1P
- .RT
- .sp 2P
- .LP
- 6.4.1
- \fIGeneral\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 6.4.1.1
- \fIOutcomes and verdicts\fR
- .sp 9p
- .RT
- .PP
- The observed outcome (of the test execution) is the series of
- events which occurred during execution of a test case; it includes all
- input to and output from the IUT at the points of control and observation.
- .PP
- The foreseen outcomes are identified and defined by the abstract
- test case specification taken in conjunction with the protocol
- Recommendation*. For each test case, there may be one or more foreseen
- outcomes. Foreseen outcomes are defined primarily in abstract terms.
- .PP
- A verdict is a statement of pass, fail or inconclusive to be
- associated with every foreseen outcome in the abstract test suite
- specification.
- .PP
- The analysis of results is performed by comparing the observed
- outcomes with foreseen outcomes.
- .PP
- The verdict assigned to an observed outcome is that associated with
- the matching foreseen outcome. If the observed outcome is unforeseen then
- the abstract test suite specification will state what default verdict shall
- be
- assigned.
- .PP
- The means by which the comparison of the observed outcomes with
- the foreseen outcomes is made is outside the scope of this Recommendation.
- .PP
- \fINote\fR \ \(em\ Amongst the possibilities are:
- .RT
- .LP
- a)
- manual or automated comparison (or a mixture);
- .LP
- b)
- comparison at or after execution time;
- .LP
- c)
- translating the observed outcomes into abstract terms for
- comparison with the foreseen outcomes or translating the
- foreseen outcomes into the terms used to record the observed
- outcomes.
- .PP
- The verdict will be pass, fail or inconclusive:
- .LP
- a)
- pass means that the observed outcome satisfies the test
- purpose and is valid with respect to the relevant Recommendation(s)* and
- with respect to the PICS;
- .LP
- b)
- fail means that the observed outcome is syntactically
- invalid or inopportune with respect to the relevant Recommendation(s)*
- or the PICS;
- .LP
- c)
- inconclusive means that the observed outcome is valid with respect to
- the relevant Recommendation(s)* but prevents the test purpose from being
- accomplished.
- .PP
- The verdict assigned to a particular outcome will depend on the
- test purpose and the validity of the observed protocol behaviour.
- .PP
- The verdicts made in respect of individual test cases will be
- synthesized into an overall summary for the IUT based on the test cases
- executed.
- .RT
- .sp 1P
- .LP
- 6.4.1.2
- \fIConformance test reports\fR
- .sp 9p
- .RT
- .PP
- The results of conformance testing will be documented in a set of conformance
- test reports. These reports will be of two types: a System
- Conformance Test Report (SCTR), and a Protocol Conformance Test Report
- (PCTR).
- .PP
- The SCTR, which will always be provided, gives an overall summary
- of the conformance status of the SUT, with respect to its single or multi\(hylayer
- IUT. A standard proforma for the SCTR is for further study.
- .PP
- The PCTR, one of which will be issued for each protocol tested in the SUT,
- documents all of the results of the test cases giving references to the
- conformance logs which contain the observed outcomes. The PCTR also gives
- reference to all necessary documents relating to the conduct of the conformance
- assessment process for that protocol.
- .PP
- A standard proforma for the PCTR is for further study. The ordered
- list of test cases to be used in the PCTR will be specified in the conformance
- test suite Recommendation*.
- .RT
- .sp 1P
- .LP
- 6.4.2
- \fIRepeatability of results\fR
- .sp 9p
- .RT
- .PP
- In order to achieve the objective of credible conformance testing, it is
- clear that the result of executing a test case on an IUT should be the
- same whenever it is performed. Statistically, it may not be possible to
- perform a complete conformance test suite and observe outcomes which are
- completely
- identical to those obtained on another occasion: unforeseen events do occur,
- and this is a feature of the environments involved. Nevertheless, at the
- test case level, it is very important that every effort is made by the
- test
- specifiers and test laboratories to minimize the possibility that a test
- case produces different outcomes on different occasions.
- .bp
- .RT
- .sp 1P
- .LP
- 6.4.3
- \fIComparability of results\fR
- .sp 9p
- .RT
- .PP
- In order to achieve the ultimate objectives of conformance testing, the
- overall summary concerning conformance of an IUT has to be independent
- of the test environment in which the testing takes place. That is to say,
- the
- standardization of all of the procedures concerned with conformance testing
- should result in a comparable overall summary being accorded to the IUT,
- whether the testing is done by the supplier, a user, or by any third party
- test house. There are a large number of factors to be studied to achieve
- this, of
- which some of the more important are:
- .RT
- .LP
- a)
- careful design of the abstract test case specification to
- give flexibility where appropriate, but show which
- requirements have to be met (which is the subject of this
- Recommendation);
- .LP
- b)
- careful specification of the real tester which should be
- used to run the test suite; again this specification should give flexibility
- where appropriate, but show which requirements have to be met, including all
- test coordination procedures (if any);
- .LP
- c)
- careful specification of the procedure to be followed in
- determining how the contents of the PICS are to be used in the analysis of
- outcomes of test cases; there should be no room for \*Qoptimistic\*U
- interpretation;
- .LP
- d)
- careful specification of the procedures to be followed by
- test laboratories as regards the repetition of a test case before making a
- final verdict for that test purpose;
- .LP
- e)
- a proforma for a conformance test report;
- .LP
- f
- )
- careful specification of the procedures necessary
- when synthesizing an overall summary.
- .sp 1P
- .LP
- 6.4.4
- \fIAuditability of results\fR
- .sp 9p
- .RT
- .PP
- For legal reasons, as well as others, it may be necessary to
- review the observed outcomes from the execution of a conformance test suite
- in order to make sure that all procedures have been correctly followed.
- Whether or not analysis has been carried out in a manual or automatic mode,
- it is
- essential that all inputs, outputs, and other test events are careful logged,
- and the analysis of the results recorded. In some cases this may be the
- responsibility of the test realizer, who may elect to include the test
- criteria in the conformance log, as well as all outcomes. In others, it
- may be the
- responsibility of the test laboratory, which might be required to follow all
- standard procedures concerning the recording of results.
- .PP
- \fINote\fR \ \(em\ As far as auditability is concerned, some automatic
- procedures would be preferred, but in the event it should be appreciated
- that from a legal standpoint such automatic procedures would have to be
- accredited themselves, if they are to be credible.
- .RT
- .sp 2P
- .LP
- \fB7\fR \fBTest methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 7.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- The testing of a given OSI* protocol can require the use of
- several test methods, as systems under test can come in several configurations,
- and vary in terms of their ability to allow ways of producing effects
- applicable to a layer boundary.
- .PP
- This section first characterizes the features of the system under
- test which are to be taken into consideration, next defines the possible
- test methods in abstract terms, and finally provides guidance on their
- applicability to real systems.
- .RT
- .LP
- 7.2
- \fIClassification of real open systems and IUTs for conformance\fR
- \fItesting\fR
- .sp 1P
- .RT
- .sp 2P
- .LP
- 7.2.1
- \fIClassification of systems under test\fR
- .sp 1P
- .RT
- .PP
- 7.2.1.1
- There is a relation between the test methods and the
- configurations of the real open systems to be tested. The appropriate test
- methods vary according to:
- .sp 9p
- .RT
- .LP
- a)
- the main function of the system (end\(hysystem or
- relay\(hysystem);
- .LP
- b)
- which layers use OSI* protocols;
- .LP
- c)
- whether the alternative of non\(hyOSI* protocols is
- also available.
- .bp
- .PP
- 7.2.1.2
- The following configurations of systems have been
- identified for the purposes of conformance testing, as illustrated in
- Figures\ 2/X.290 to\ 4/X.290, Part\ 1. Configurations 1 to 3 are the basic
- configurations of systems under test (SUTs):
- .sp 9p
- .RT
- .LP
- a)
- Configuration\ 1: 7\(hylayer open system (end\(hysystem)
- .LP
- These systems use OSI* Recommendation* protocols in
- all 7 layers.
- .LP
- b)
- Configuration\ 2: Partial (N)\(hyopen system (end\(hysystem)
- .LP
- These systems use OSI* Recommendation* protocols in
- layers 1 to N.
- .LP
- c)
- Configuration\ 3: Open relay\(hysystems
- .LP
- These use OSI* protocols in layers 1 to 3 (Network
- relay\(hysystems) or 1 to 7 (Application
- relay\(hysystems).
- .LP
- .rs
- .sp 15P
- .ad r
- \fBFigures 2/X.290, partie 1 \*`a 4/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .PP
- 7.2.1.3
- Other configurations can be derived from the basic
- configurations.
- .sp 9p
- .RT
- .PP
- A SUT can be a combination of basic configurations\ 1 and\ 2,
- allowing the alternative of using OSI* and non\(hyOSI* protocols above layer\ N
- (see Figure\ 5/X.290, Part\ 1).
- .LP
- .rs
- .sp 15P
- .ad r
- \fBFigure 5/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .LP
- 7.2.2
- \fIIdentification of the implementation under test (IUT)\fR
- .sp 9p
- .RT
- .PP
- An implementation under test (IUT) is that part of a real open
- system which is to be studied by conformance testing. It should be an
- implementation of one or more adjacent OSI* protocols.
- .PP
- IUTs can be defined for configurations 1 and 2 of SUTs as single\(hylayer
- IUTs (one single layer of the SUT is to be tested), or as multi\(hylayer
- IUTs (a set of any number of adjacent layers of the SUT to be tested in
- combination).
- .PP
- An IUT defined in an open relay\(hysystem will include at least the
- layer which provides the relay function.
- .PP
- When OSI* and non\(hyOSI* protocols exist in a system, the IUT(s) will
- be defined for the OSI* mode(s) of operation. Testing non\(hyOSI* protocols
- is
- outside the scope of this Recommendation.
- .PP
- Clients and test laboratories will agree on what part of the SUT
- will be considered to be the IUT.
- .RT
- .sp 1P
- .LP
- 7.3
- \fIAbstract testing methodology\fR \ \(em\ \fIGeneral\fR
- .sp 9p
- .RT
- .PP
- \fR Test methods need to refer to an abstract testing methodology,
- based upon the OSI reference model. Considering first end\(hysystems (7\(hylayer
- or partial (N)\(hyopen systems) and single layer IUTs within these systems,
- abstract test methods are described in terms of what outputs from the IUT
- are observed and what inputs to it can be controlled. More specifically,
- an abstract test
- method is described by identifying the points closest to the IUT at which
- control and observation are to be exercized.
- .PP
- The OSI* protocol Recommendations* define allowed behaviour of a
- protocol entity (i.e.\ the dynamic conformance requirements) in terms of the
- protocol data units (PDUs) and the abstract service primitives (ASPs) both
- above and below that entity. Thus the behaviour of an (N)\(hyASPs and (N\
- \(em\ 1)\(hyASPs (the latter including the (N)\(hyPDUs).
- .PP
- If an IUT comprises more than one protocol entity, the required
- behaviour can be defined in terms of the ASPs above and below the IUT,
- including the PDUs of the protocols in the IUT.
- .PP
- The starting point for developing test methods is the conceptual
- testing architecture, illustrated in Figure\ 6/X.290, Part\ 1. It is a
- \*Qblack\(hybox\*U active testing architecture, based on the definition
- of behaviour required of the IUT.
- .PP
- The action of the tester shown in Figure 6/X.290, Part\ 1, can either be
- applied
- locally, in which case there is a direct coupling within the system under
- test, or externally via a link or network. The two sets of interactions,
- above and
- below the IUT, can, in practice, be observed and controlled from several
- different points, locally or externally.
- .PP
- The possible points of control and observation (PCOs) are
- identified by three factors:
- .RT
- .LP
- a)
- whether it is the ASPs or PDUs which are observed and
- controlled;
- .LP
- b)
- the layer identity of the ASPs or PDUs concerned;
- .LP
- c)
- whether they are controlled and observed within the system under test
- or in a system remote from the system under test \(em if the latter
- then the ASPs are distinguished by the addition of a double\(hyquote
- character\ (\|\*U\|).
- .LP
- .rs
- .sp 14P
- .ad r
- \fBFigure 6/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .PP
- Possible PCOs within the SUT are illustrated in
- Figure\ 7a)/X.290, Part\ 1. Possible PCOs, when the activity below the IUT is
- controlled and observed externally are illustrated in Figure\ 7b)/X.290,
- Part\ 1. It can be seen from these figures that there is a multiplicity of
- possible PCOs in different layers, which offer different degrees of control
- and observation of IUT behaviour. This Recommendation makes a selection
- from this set of possible PCOs, defining a limited number of abstract test
- methods.
- .LP
- .rs
- .sp 14P
- .ad r
- \fBFigure 7a/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .rs
- .sp 14P
- .ad r
- \fBFigure 7b/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .PP
- If control and observation below, or external from the IUT is
- specified in terms of ASPs, it will include control and observation of
- the PDUs carried by those ASPs; but if it is specified in terms of PDUs
- (at layer\ N)
- then the ASPs (at layer\ N\ \(em\ 1) are not considered to be controlled or
- observed.
- .PP
- It is assumed that the ASP activity below the IUT can at least be
- observable and controllable via the peer activity in a remote testing system
- \(em i.e.\ the corresponding ASP\*Us. Thus when the ASPs below the IUT
- are not
- controllable nor observable locally, conformance testing can be carried out
- externally, provided that the underlying service offered is sufficiently
- reliable for control and observation to take place remotely.
- .PP
- It is possible that the ASP activity above the IUT might not be
- controllable nor observable, in which case this activity is said to be
- hidden.
- .PP
- SUTs are not required to provide access to layer boundaries. However, the
- possible provision of such access and the possible positions of such
- boundaries with respect to the layers of the IUT are factors to be taken
- into consideration in the definition of the test methods, which may take
- advantage of this access to define the test suites in terms of the corresponding
- ASPs. It does not matter whether the accessible boundaries are accessed
- via service
- access points (SAPs) or via some other PCOs.
- .bp
- .PP
- Figure 8/X.290, Part\ 1 shows examples of IUTs, with respect to layer boundary
- accessibility.
- .PP
- \fINote\fR \ \(em\ In addition, a conformance test suite Recommendation*\fR
- may define \*Uabstract local primitives\*U. These are used to specify control
- and observation of events or states which are referred to in the protocol
- Recommendation* but which are internal to the IUT and which cannot be
- expressed in terms of ASPs. They are abbreviations for text descriptions of
- control and observation to be performed by the upper tester.
- .PP
- Similar consideration apply to relay systems (see Part 2 of the
- Recommendation for details).
- .RT
- .LP
- .rs
- .sp 15P
- .ad r
- \fBFigure 8/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .sp 1P
- .LP
- 7.4
- \fIAbstract testing functions\fR
- .sp 9p
- .RT
- .PP
- The definition of abstract test methods requires that the PCOs be distributed
- over two abstract testing functions, the lower and upper testers.
- .PP
- The lower tester is the abstraction of the means of providing,
- during test execution, control and observation at the appropriate PCO either
- below the IUT or remote from the IUT, as defined by the chosen abstract test
- method. Thus, it is the testing function related to the control
- and observation of the lower boundary of the IUT. If the action of the
- tester is local to the SUT, the lower tester will take the place of the
- lower part of the SUT. If the action of the tester is external to the SUT,
- the lower tester will rely on the (N\ \(em\ 1)\(hyService, provided jointly
- by the lower tester itself, a link and the SUT.
- .PP
- The upper tester is the abstraction of the means of providing,
- during test execution, control and observation of the upper service boundary
- of the IUT and of any relevant abstract local primitive.
- .PP
- There is a need for cooperation between the upper tester and the
- lower tester; the rules for such cooperation are called the test coordination
- procedures.
- .PP
- The test methods will vary in the way that they specify the test
- coordination procedures. In some cases it is possible to define a test
- management protocol to provide the coordination between the upper and lower
- testers. In other cases it is only possible to describe the requirements
- to be met by the test coordination procedures without specifying what mechanisms
- might be used to realize them.
- .RT
- .sp 2P
- .LP
- 7.5
- \fIOverview of abstract test methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 7.5.1
- \fIEnd\(hysystem IUTs\fR
- .sp 9p
- .RT
- .PP
- For the IUTs defined within end\(hysystem SUTs (configurations\ 1 and 2
- in Figures\ 2/X.290, Part\ 1 and\ 3/X.290, Part\ 1) four categories of
- abstract
- test methods are defined, one local, and three external ones which assume
- the lower tester is located remotely from the SUT and connected to it by
- a link or network.
- .bp
- .RT
- .sp 1P
- .LP
- 7.5.2
- \fIThe local test methods\fR
- .sp 9p
- .RT
- .PP
- The local abstract test methods define the PCOs as being at the
- service boundaries above and below the IUT. The test events are specified in
- terms of the ASPs above the IUT and the ASPs and PDUs below the IUT, as
- illustrated in Figure\ 9a)/X.290, Part\ 1. Abstractly, a lower tester is
- considered to observe and control the ASPs and PDUs below the IUT, while an
- upper tester observes and controls the ASPs above the IUT. Requirements
- to be met by test coordination procedures used to coordinate the realizations
- of the upper and lower testers are defined in the abstract conformance
- test suites,
- although the test coordination procedures themselves are not.
- .RT
- .sp 1P
- .LP
- 7.5.3
- \fIExternal test methods\fR
- .sp 9p
- .RT
- .PP
- The external test methods use control and observation of the ASPs below
- the IUT by means of a lower tester separated from the SUT, together with
- control and observation of the ASPs above the IUT. Three different categories
- of external abstract test methods are defined, which are referred to as
- distributed, coordinated, and remote. They vary according to the level of
- requirement or standardization put on the test coordination procedures, on
- the access to the layer boundary above the IUT, and on the requirements on
- an upper tester. They are illustrated in Figures\ 9b), c) and\ d)/X.290,
- Part\ 1.
- .PP
- The coordinated test method requires that the test coordination
- procedures used to coordinate the realization of the upper and lower testers
- be achieved by means of test management protocols. The other two methods
- do not make any assumptions about the realization of the test coordination
- procedures.
- .PP
- The distributed and coordinated test methods require specific
- functions from the upper tester above the IUT. The remote method
- does not.
- .PP
- The distributed method requires access to the upper boundary of
- the IUT. The other two methods do not.
- .RT
- .LP
- .rs
- .sp 29P
- .ad r
- \fBFigure 9/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .LP
- 7.5.4
- \fIVariants of end\(hysystem test methods\fR
- .sp 9p
- .RT
- .PP
- Each category of test methods has variants which can be applied
- to single\(hylayer IUTs or to multi\(hylayer IUTs. For multi\(hylayer IUTs
- in which
- the protocols are to be tested layer by layer, embedded variants can be
- used.
- .PP
- All abstract test methods for end\(hysystems are fully specified in
- section\ 8 of Part\ 2 of this Recommendation, including their single\(hylayer,
- multi\(hylayer and embedded variants where applicable.
- .RT
- .sp 1P
- .LP
- 7.5.5
- \fIRelay\(hysystem IUTs\fR
- .sp 9p
- .RT
- .PP
- For open relay\(hysystems, two test methods are defined, loop\(hyback
- and transverse. These are fully specified in section\ 8 of Part\ 2 of this
- Recommendation.
- .RT
- .sp 1P
- .LP
- 7.6
- \fIApplicability of test methods to real open systems\fR
- .sp 9p
- .RT
- .PP
- The architecture and stage of development of a real open system
- determines the applicability of test methods to it.
- .PP
- Local test methods are useful for systems under development, when
- their architecture permits the isolation of an IUT, be it single\(hylayer or
- multi\(hylayer.
- .PP
- External test methods are useful for testing complete or partial
- end\(hysystems which can attach to telecommunications networks.
- .PP
- Coordinated test methods apply where it is possible to implement a
- standardized test management protocol in an upper tester in the SUT, above
- the IUT.
- .PP
- Remote test methods apply when it is possible to make use of some
- functions of the SUT to control the IUT during testing, instead of using a
- specific upper tester.
- .PP
- Distributed test methods apply when it is necessary to allow
- complete freedom for the implementation of the test coordination procedures
- between the SUT and the lower tester, but to specify in detail the control
- and observation requirements at both ends.
- .PP
- Single\(hylayer test methods are the most appropriate methods for testing
- the majority of the protocol conformance requirements.
- .PP
- Multi\(hylayer test methods will be used when conformance to true
- multi\(hylayer dynamic conformance requirements is to be tested.
- .PP
- Embedded test methods permit the application of single\(hylayer
- testing to all layers of a multi\(hylayer IUT.
- .PP
- For 7\(hylayer open systems, the preferred methods are the
- incremental use of appropriate external single\(hylayer embedded methods
- with the following PCOs:
- .RT
- .LP
- a)
- the upper interface of the application layer as provided
- by the 7\(hylayer open system, when applicable;
- .LP
- b)
- successively, each SAP (or corresponding PCO if there is no SAP as such)
- below the protocol which is the focus of the
- testing, as controlled and observed in the external
- lower tester, starting from the lowest protocol of the IUT
- and working upwards.
- .sp 1P
- .LP
- 7.7
- \fIApplicability of the test methods to OSI* protocols and layers\fR
- .sp 9p
- .RT
- .PP
- The test methods defined in this Recommendation apply to all
- layers, with the exception of the Physical layer and the Media Access Control
- protocols which are outside the scope of this Recommendation. Appendix\
- I of
- this Recommendation provides guidance on the applicability of test methods
- to all other layers.
- .RT
- .sp 2P
- .LP
- \fB8\fR \fBTest suites\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 8.1
- \fIStructure\fR
- .sp 9p
- .RT
- .PP
- Test suites have a hierarchical structure (see Figure\ 10/X.290,
- Part\ 1) in which an important level is the test case. Each test case has a
- narrowly defined purpose, such as that of verifying that the IUT has a
- certain required capability (e.g.\ the ability to support certain packet
- sizes) or
- exhibit a certain required behaviour (e.g.\ behave as required when a particular
- event occurs in a particular state).
- .bp
- .PP
- Within a test suite, nested test groups are used to provide a logical ordering
- of the test cases. Test groups may be nested to an arbitrary depth.
- They may be used to aid planning, development, understanding or execution of
- the test suite.
- .PP
- Test cases are modularized by using named subdivisions called test
- steps. Each test case comprises at least one test step: the ordering of
- events covered by the test purpose (\*Utest body\*U). It may include further
- test steps to put the IUT into the state required for the test body to
- start from (the
- \*Upreamble\*U) or return to a quiescent state after the test body has finished
- (the \*Upostamble\*U).
- .PP
- For practical reasons, common test steps may be grouped together
- into test step libraries. Test step libraries may be structured into nested
- sets of test steps, to any depth of nesting. Test step libraries may be
- associated with the whole test suite or with a particular test group or test
- case.
- .PP
- Furthermore, all test steps consist of an ordering of other test
- steps andB/For test events (e.g.\ the transfer of a single PDU or ASP to
- or from the IUT). All test steps are, therefore, equivalent to an ordering
- of test
- events (after expansion of the inner test steps).
- .RT
- .LP
- .rs
- .sp 38P
- .ad r
- \fBFigure 10/X.290, partie 1, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 2P
- .LP
- 8.2
- \fIGeneric, abstract and executable test cases\fR
- .sp 1P
- .RT
- .PP
- 8.2.1
- A generic test case is one which:
- .sp 9p
- .RT
- .LP
- a)
- provides a refinement of the test purpose;
- .LP
- b)
- specifies that all sequences of test events (paths)
- in the test body which correspond to verdicts of \*Qpass\*U, \*Qfail\*U and
- \*Qinconclusive\*U, using a specialized notation;
- .LP
- c)
- is used as the common root of corresponding abstract test
- cases for different abstract test suites for the
- same protocol;
- .LP
- d)
- includes a description of the initial state in which the
- test body should start, in lieu of a preamble;
- .LP
- e)
- need not describe the postamble;
- .LP
- f
- )
- is specified using the style of either the Remote or Distributed Single\(hylayer
- test methods.
- .PP
- 8.2.2
- An abstract test case is derived from a generic case and the
- relevant protocol specification; it:
- .sp 9p
- .RT
- .LP
- a)
- specifies the test case in terms of a particular test
- method;
- .LP
- b)
- adds a more precise specification for sequences of events
- which are only described informally in the generic
- test case;
- .LP
- c)
- adds the sequences of test events required to achieve the
- objectives of the preamble and postamble of the generic test case using a
- specialized notation.
- .PP
- 8.2.3
- An executable test case is derived from an abstract test case,
- and is in a form which allows it to be run on a real tester for testing
- a real implementation.
- .sp 9p
- .RT
- .PP
- 8.2.4
- The terms generic, abstract and executable are used to
- describe test suites, which comprise generic, abstract and executable test
- cases, respectively.
- .sp 9p
- .RT
- .PP
- 8.2.5
- A generic test suite has the coverage of the set or a
- superset of all possible abstract test suites for a particular
- protocol.
- .sp 9p
- .RT
- .sp 2P
- .LP
- \fB9\fR \fBRelationships between concepts and roles\fR
- .sp 1P
- .RT
- .PP
- Figure\ 11/X.290, Part\ 1 is a pictorial representation of the
- relation between the various Recommendations and the processes of producing
- generic, abstract and executable test suites and test reports.
- .PP
- Part\ 2 concerns the production of testable protocol
- Recommendations and abstract test suite Recommendations. Part\ 1 provides
- general concepts and definitions.
- .PP
- \fINote\fR \ \(em\ Other aspects of the conformance assessment process,
- such as executable test derivation, preparation of the IUT, PICS and PIXIT
- by the
- client and test laboratory role are for further study.
- .RT
- .sp 2P
- .LP
- \fB10\fR \fBCompliance\fR
- .sp 1P
- .RT
- .PP
- In this Recommendation, \*Qcompliance\*U refers to meeting the
- requirements specified by the Recommendation. This word is used as an
- attempt to eliminate confusion between \fIcompliance\fR to this Recommendation
- and \fIconformance\fR of a protocol implementation to protocol
- Recommendations.
- .PP
- This part of the Recommendation contains no compliance
- requirements.
- .RT
- .LP
- .rs
- .sp 06P
- .ad r
- BLANC
- .ad b
- .RT
- .LP
- .bp
- .LP
- .rs
- .sp 47P
- .ad r
- \fBFigure 11/X.290, partie 1, (N), p. 10\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .ce 1000
- \fBPart\ 2\ \(em\ \fR \fBAbstract test suite specification\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB0\fR \fBIntroduction\fR
- .sp 1P
- .RT
- .PP
- This part of the Recommendation provides a common approach to the specification
- of \*QOSI or related CCITT X\(hyseries or T\(hyseries\*U (hereafter
- abbreviated to \*QOSI*\*U) conformance test suites at a level which is
- independent of the means of executing those test suites (hereafter called
- \*Qabstract test suites\*U). This level of abstraction is suitable for
- standardization and facilitates the comparison of results produced by different
- organizations which run the corresponding executable test suites.
- .PP
- Section One recalls that there are requirements put on the OSI*
- protocol specifiers which have to be fulfilled before there can be an objective
- basis for the process of developing an abstract test suite. The need for
- consistent conformance sections and for PICS and PIXIT proformas in OSI*
- protocol Recommendations* is expressed.
- .PP
- Section Two describes the process of developing an abstract test
- suite, including the design criteria to be used and guidance on its structure
- and coverage. The possible abstract test methods are defined and guidance
- is
- given to help the test suite specifier to decide which method(s) to use
- in the production of a particular test suite. The relationship between
- abstract test suites for different methods is provided by a generic test
- suite which is
- independent of the test method. A test notation is defined and requirements
- and guidance are given on its use for specifying both generic and abstract
- test
- cases. These include the subdivision of test cases into test steps and the
- assignment of verdicts to outcomes.
- .PP
- The test suite specifier is also required to provide information
- to the test realizers, particularly on the constraints governing test case
- selection and ordering.
- .PP
- Finally, guidance and requirements are given on test suite
- maintenance.
- .RT
- .sp 2P
- .LP
- \fB1\fR \fBScope and field of application\fR
- .sp 1P
- .RT
- .PP
- This part of the Recommendation specifies the requirements and
- provides guidance for the production of system\(hyindependent conformance test
- suites for one or more OSI* Recommendations*. In particular, it applies
- to the production of all conformance test suite Recommendations* for OSI*
- protocols.
- .PP
- It applies to the production of conformance test cases which check
- the adherence of an implementation to the relevant static and/or dynamic
- conformance requirements by controlling and observing protocol behaviour.
- The abstract test methods included in this Recommendation are in fact capable
- of
- being used to specify any test case which can be expressed abstractly in
- terms of control and observation of protocol data units, abstract service
- primitives and abstract local primitives. Nevertheless, for some protocols,
- test cases may be needed which cannot be expressed in these terms. The
- specification of such test cases is outside the scope of this Recommendation,
- although those test
- cases may need to be included in a Recommendation* for a conformance test
- suite.
- .PP
- \fINote\fR \ \(em\ For example, some static conformance requirements related
- to an application service may require testing techniques which are specific
- to
- that particular application.
- .PP
- The production of conformance test suites for multi\(hypeer or
- physical layer protocols is outside the scope of this
- Recommendation.
- .PP
- The relation between abstract test specification and formal
- description techniques is outside the scope of this
- Recommendation.
- .RT
- .sp 2P
- .LP
- \fB2\fR \fBReferences\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- Recommendation\ X.200
- \fIReference model of open systems interconnection\fR \fIfor CCITT applications\fR
- \|
- (see also ISO\ 7498).
- .sp 9p
- .RT
- .LP
- Recommendation\ X.214
- \fITransport service definition for open systems\fR
- \fIinterconnection for CCITT applications\fR \|
- (see also ISO\ 8072).
- .LP
- Recommendation\ X.224
- \fITransport protocol specification for open\fR
- \fIsystems interconnection for CCITT applications\fR \|
- (see also ISO\ 8073).
- .bp
- .LP
- Recommendation\ X.210
- \fIOpen systems interconnection layer service\fR
- \fIdefinition conventions\fR \|
- (see also ISO\ TR\ 8509).
- .LP
- Recommendation\ X.208
- \fISpecification of abstract syntax notation one\fR
- \fI(ASN.1)\fR \|
- (see also ISO\ 8824).
- .LP
- Recommendation\ X.209
- \fISpecification of basic encoding rules for abstract\fR \fIsyntax notation
- one (ASN.1)\fR \|
- (see also ISO\ 8825).
- .LP
- Recommendation\ X.290/1
- \fIOSI conformance testing methodology and\fR
- \fIframework for protocol Recommendations for CCITT applications\ \(em\
- Part\ 1:\fR
- \fIGeneral concepts.\fR
- .LP
- ISO\ 8571\(hy4
- \fIInformation processing systems\ \(em\ open systems\fR
- \fIinterconnection\ \(em\ File protocol specification.\fR
- .sp 2P
- .LP
- \fB3\fR \fBDefinitions\fR
- .sp 1P
- .RT
- .PP
- For the purposes of this Part of the Recommendation, all the
- definitions in Part\ 1 of the Recommendation apply.
- .RT
- .sp 2P
- .LP
- \fB4\fR \fBAbbreviations\fR
- .sp 1P
- .RT
- .PP
- For the purposes of this Recommendation the abbreviations given in section\
- 4 of Part\ 1 of this Recommendation apply. The following abbreviations
- also apply to this Recommendation:
- .RT
- .LP
- ASP\*U
- the abstract service primitive on the side of the service
- provider remote from the IUT
- .LP
- CM
- coordinated multi\(hylayer (test method)
- .LP
- CS
- coordinated single\(hylayer (test method)
- .LP
- CSE
- coordinated single\(hylayer embedded (test method)
- .LP
- DM
- distributed multi\(hylayer (test method)
- .LP
- DS
- distributed single\(hylayer (test method)
- .LP
- DSE
- distributed single\(hylayer embedded (test method)
- .LP
- FDT
- formal description technique
- .LP
- LM
- local multi\(hylayer (test method)
- .LP
- LS
- local single\(hylayer (test method)
- .LP
- LSE
- local single\(hylayer embedded (test method)
- .LP
- RM
- remote multi\(hylayer (test method)
- .LP
- RS
- remote single\(hylayer (test method)
- .LP
- RSE
- remote single\(hylayer embedded (test method)
- .LP
- TTCN
- tree and tabular combined notation
- .LP
- YL
- loop\(hyback (test method)
- .LP
- YT
- transverse (test method)
- .sp 2P
- .LP
- \fB5\fR \fBCompliance\fR
- .sp 1P
- .RT
- .PP
- 5.1
- A protocol Recommendation* which complies with this Part of
- this Recommendation shall satisfy all the requirements stated in
- Section\ 1.
- .sp 9p
- .RT
- .PP
- \fINote\fR \ \(em\ Such compliance is a precondition for the
- protocol Recommendation* to be an effective basis for conformance testing of
- implementations.
- .PP
- 5.2
- An abstract test suite specification which complies with this part of this
- Recommendation:
- .sp 9p
- .RT
- .LP
- a)
- shall be a conformance test suite;
- .LP
- b)
- shall be specified in a test notation standardized by
- ISO or CCITT;
- .LP
- c)
- shall satisfy all the requirements stated in
- Section\ 2.
- .PP
- 5.3
- It is recommended that the test notation used be the Tree and Tabular Combined
- Notation (TTCN). If TTCN is used, the abstract test suite
- shall comply with all the requirements in Annex\ D.
- .bp
- .sp 9p
- .RT
- .sp 1P
- .ce 1000
- SECTION\ 1\ \(em\ \fIRequirements on protocol specifiers\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB6\fR \fBConformance requirements in OSI* Recommendations*\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 6.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- The meaning of conformance in OSI* is discussed in Part\ 1 of this Recommendation.
- It is necessary that there be an unambiguous and objective
- understanding of the conformance requirements of an OSI* protocol or transfer
- syntax Recommendation*, as a prerequisite to the production of an abstract
- test suite for that Recommendation*. This section states the requirements
- on
- protocol specifiers to ensure that there is such an understanding
- of the conformance requirements.
- .RT
- .sp 2P
- .LP
- 6.2
- \fIGeneral requirements\fR
- .sp 1P
- .RT
- .PP
- 6.2.1
- A clear distinction shall be made between static and dynamic
- conformance requirements. To avoid ambiguity, they should be stated separately
- from one another.
- .sp 9p
- .RT
- .PP
- 6.2.2
- It shall be clear what conformance to the Recommendation*
- means, in the sense of what shall be done, what is permitted but not mandatory,
- and what shall not be implemented, to conform to the Recommendation*.
- .sp 9p
- .RT
- .PP
- 6.2.3
- It shall always be decidable whether an instance of
- communication conforms dynamically or not.
- .sp 9p
- .RT
- .PP
- For example, one should be able to look at a record of PDU
- activity and decide whether it is valid.
- .PP
- 6.2.4
- Requirements on the need to produce and content of a PICS
- shall be stated separately from the requirements on the protocol implementation
- itself.
- .sp 9p
- .RT
- .sp 2P
- .LP
- 6.3
- \fIConformance sections\fR
- .sp 1P
- .RT
- .PP
- 6.3.1
- Each OSI* protocol and transfer syntax Recommendation* shall
- include a conformance section, which shall be expressed clearly and
- unambiguously.
- .sp 9p
- .RT
- .PP
- 6.3.2
- Conformance sections shall distinguish between the following
- categories of information that they may contain:
- .sp 9p
- .RT
- .LP
- a)
- references to sections which state dynamic conformance
- requirements;
- .LP
- b)
- static conformance requirements concerning the protocol
- implementation;
- .LP
- c)
- static conformance requirements concerning multi\(hylayer
- dependencies;
- .LP
- d)
- what has to be stated in the PICS concerning (b) above;
- .LP
- e)
- what has to be stated in the PICS concerning (c) above;
- .LP
- f
- )
- what other information shall be provided
- (e.g. to assist testing) and whether this should be in the PICS or
- elsewhere.
- .PP
- 6.3.3
- In connection\(hyoriented protocol Recommendations*, the
- conformance section should also include:
- .sp 9p
- .RT
- .LP
- a)
- the option to support either the initiation of a
- connection or the acceptance of a connection, or both;
- .LP
- b)
- the requirement to be able to accept all correct sequences
- of PDUs received from peers, and respond with correct PDU
- sequences appropriate to the defined state of the
- connection;
- .LP
- c)
- the requirement to be able to respond correctly to all
- incorrect sequences of PDUs received, the response being
- appropriate to the defined state of the
- connection.
- .bp
- .sp 1P
- .LP
- 6.4
- \fIAdditional guidance for new protocol Recommendations*\fR
- .sp 9p
- .RT
- .PP
- It is recognized that although existing protocol Recommendations* can be
- improved by the progression of an addendum to add a PICS proforma and
- make the conformance section align with the requirements stated above, it is
- unrealistic to expect any greater improvement. However, new protocol
- Recommendations* should be developed using the additional guidance given in
- Annex\ B to make the task of understanding the conformance requirements
- easier and less prone to ambiguity.
- .RT
- .LP
- \fB7\fR \fBPICS proformas\fR
- .sp 1P
- .RT
- .sp 2P
- .LP
- 7.1
- \fIRequirements on PICS proformas\fR
- .sp 1P
- .RT
- .PP
- 7.1.1
- The specific requirements to be met by suppliers in respect
- of each PICS they provide shall normally be stated in the relevant protocol
- Recommendation*. The specification of these requirements shall include
- a PICS proforma. In exceptional circumstances, the PICS proforma may be
- found in the abstract test suite Recommendation* rather than in protocol
- Recommendation*; in particular, this applies when the PICS proforma has
- to cover different versions of the same protocol coming from both ISO and
- CCITT. In normal circumstances, the PICS proforma should be found in an
- annex to the protocol Recommendation* and referenced in the conformance
- section, and, if necessary, progressed as an addendum rather than as part of
- the original Recommendation*.
- .sp 9p
- .RT
- .PP
- \fINote\fR \ \(em\ A PICS for a specific protocol implementation will need
- to be accompanied by administrative and documentary information relating
- to the supplier, the system and its intended environment.
- .PP
- 7.1.2
- The PICS proforma shall be in the form of a questionnaire or
- checklist to be completed by the supplier or implementor of an implementation
- of the relevant OSI* protocol.
- .sp 9p
- .RT
- .PP
- 7.1.3
- The PICS proforma shall cover all functions, elements of
- procedure, parameters, options, PDUs, timers, multi\(hylayer dependencies and
- other capabilities identified in the protocol Recommendation*, including
- optional and conditional capabilities. It is desirable, where practicable,
- to include all mandatory features. There shall be a well\(hydefined mapping
- between static conformance requirements and the PICS proforma and there
- shall be
- consistency between them.
- .sp 9p
- .RT
- .PP
- 7.1.4
- The PICS proforma shall be preceded by a section that
- states:
- .sp 9p
- .RT
- .LP
- \*QThe supplier of a protocol implementation which is claimed to
- conform to this Recommendation shall complete the following Protocol
- Implementation Conformance Statements (PICS) proforma and shall
- provide the information necessary to identify uniquely both the
- supplier and the implementation.\*U
- .PP
- 7.1.5
- The name, version and date of the relevant protocol
- Recommendation* shall be stated on each page of the PICS proforma.
- .sp 9p
- .RT
- .PP
- 7.1.6
- The PICS proforma for a specific protocol shall
- contain:
- .sp 9p
- .RT
- .LP
- a)
- explanations of special symbols, abbreviations, special
- terms, together with appropriate references;
- .LP
- b)
- explicit instructions for completing and interpreting the
- PICS;
- .LP
- c)
- provision for mentioning any mandatory feature that has
- not been implemented, with the appropriate rationale;
- .LP
- d)
- one or more tables (or other kinds of form as necessary)
- to be completed to state the capabilities of the implementation
- in detail, including:
- .LP
- 1)
- name of the feature, PDU type, timer, parameter, and
- other capabilities;
- .LP
- 2)
- a column stating whether each is mandatory, optional,
- negotiable, or conditional;
- .LP
- 3)
- wherever feasible, a column giving references to the
- relevant sections in the Recommendation;
- .LP
- 4)
- a column giving the permitted range or values, if
- appropriate;
- .LP
- 5)
- a column to be filled in to give the supported
- values or range of values, if appropriate;
- .LP
- 6)
- a column to be filled in to state if each capability
- has been implemented;
- .LP
- e)
- the proforma shall give a clear indication of the
- preferred data types (e.g.\ number bases, string types, octets,
- bits, seconds, minutes,\ etc.) for responses.
- .bp
- .PP
- 7.1.7
- The PICS proforma shall use the following abbreviations as
- appropriate, unless they conflict with the abbreviations used in the specific
- protocol Recommendation*:
- .sp 9p
- .RT
- .LP
- m
- mandatory
- .LP
- o
- optional
- .LP
- c
- conditional
- .LP
- n
- negotiable (using the protocol)
- .LP
- x
- exclusion of capability
- .LP
- \(em
- not applicable
- .LP
- s
- ability to send
- .LP
- r
- ability to receive
- .sp 1P
- .LP
- 7.2
- \fIGuidance on PICS proformas\fR
- .sp 9p
- .RT
- .PP
- Appendix\ III provides some general purpose examples to give
- guidance on the construction of PICS proformas.
- .RT
- .LP
- .rs
- .sp 38P
- .ad r
- BLANC
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .ce 1000
- SECTION\ 2\ \(em\ \fIRequirements on abstract test suite specifiers\fR
- .sp 1P
- .RT
- .ce 0
- .sp 1P
- .sp 2P
- .LP
- \fB8\fR \fBTest suite production process\fR
- .sp 1P
- .RT
- .PP
- In order to present the requirements and general guidance for
- abstract test suite specification, it is useful to assume a normal form
- of the process of test suite production. This section describes the process
- in just
- such a normal form. Abstract test suite specifiers are not required to
- follow this normal form exactly, however, they are recommended to use a
- similar
- process involving the same steps, possibly in a different order.
- .PP
- For the purposes of this Recommendation, the abstract test suite
- production process is assumed to be as follows:
- .RT
- .LP
- a)
- study the relevant Recommendation(s)* to determine
- what the conformance requirements (including options) are which
- need to be tested, and what needs to be stated in the PICS
- (see\ \(sc\ 9);
- .LP
- b)
- decide on the test groups which will be needed to achieve
- the appropriate coverage of the conformance requirements and
- refine the test groups into sets of test purposes
- (see\ \(sc\ 10);
- .LP
- c)
- specify generic test cases for each test purpose, using
- some appropriate test notation (see\ \(sc\ 11);
- .LP
- d)
- choose the test method(s) for which the complete abstract
- test cases need to be specified, and decide what restrictions
- need to be placed on the capabilities of the lower tester and
- [if appropriate to the chosen test method(s)] the
- upper tester and test coordination procedures
- (see\ \(sc\ 12);
- .LP
- e)
- choose the test notation for specifying the complete
- abstract test cases, and specify the complete abstract test
- cases, including the test step structure to be used
- (see\ \(sc\ 13);
- .LP
- f
- )
- specify the inter\(hyrelationships among the test cases
- and those between the test cases and the PICS and, as far as
- possible, the PIXIT, in order to determine the restrictions
- on the selection and parameterization of test cases for
- execution, and on the order in which they can be executed
- (see\ \(sc\ 14);
- .LP
- g)
- consider the procedures for maintaining the abstract test
- suite (see\ \(sc\ 15).
- .PP
- The remainder of this section provides requirements and guidance which
- relate to each step in the above process.
- .sp 2P
- .LP
- \fB9\fR \fBDetermining the conformance requirements and PICS\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 9.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- Before an abstract test suite can be specified, the test suite
- specifier shall first determine what the conformance requirements are for
- the relevant Recommendation(s)* and what is stated in the PICS proforma
- concerning the implementation of those Recommendation(s)*.
- .RT
- .sp 1P
- .LP
- 9.2
- \fIConformance requirements\fR
- .sp 9p
- .RT
- .PP
- Section\ 1 of this part of the Recommendation specifies the
- requirements to be met by protocol specifiers as a prerequisite to the
- production of an abstract test suite for a particular protocol.
- .PP
- In practice, early OSI* Recommendations* might not contain a clear
- specification of all the relevant conformance requirements. In particular,
- the static conformance requirements might be badly specified or even omitted.
- In
- such cases, the test suite specifier shall contribute to the production
- of an amendment or addendum to the relevant Recommendation(s)* to clarify
- the
- conformance requirements. If, however, an abstract test suite has to be
- produced in advance of the conformance requirements being clarified in the
- relevant Recommendation(s)*, then the test suite specifier shall adopt the
- short\(hyterm solution given in Annex\ C and state clearly in the test suite
- Recommendation* what the implications of this are (i.e.\ what is assumed
- to be mandatory, what conditional and what the conditions are, and what
- is assumed to be optional).
- .bp
- .RT
- .sp 1P
- .LP
- 9.3
- \fIPICS proforma\fR
- .sp 9p
- .RT
- .PP
- If no PICS proforma is included in a relevant protocol
- Recommendation*, the test suite specifier shall provide a PICS proforma
- to be processed as an addendum to that Recommendation*, or in exceptional
- circumstances (as discussed in \(sc\ 7.1.1) as an annex to the abstract
- test suite Recommendation*.
- .PP
- \fINote\fR \ \(em\ Progressing an addendum to an existing protocol
- Recommendation* may well be faster than progressing the abstract test
- suite Recommendation* because the PICS proforma is likely to be less
- controversial than the test suite and therefore is likely to require fewer
- revisions before a final text can be agreed.
- .RT
- .sp 2P
- .LP
- \fB10\fR \fBTest suite structure\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 10.1
- \fIBasic requirements\fR
- .sp 9p
- .RT
- .PP
- An abstract test suite shall comprise a number of test cases. The test
- cases shall be grouped into test groups, if necessary nested. The
- structure shall be hierarchical in the sense that an item at a lower level
- shall be completely contained within a higher level item. The structure need
- not, however, be strictly hierarchical in the sense that any one test case
- may occur in more than one test suite or test group. Similar test groups
- may occur in more than one higher level test group or test suite.
- .PP
- The abstract test suite specifier shall ensure that a subset of the
- test purposes of each abstract test suite is concerned with capability
- testing, and another subset is concerned with behaviour testing. This need
- not lead to distinct test cases for behaviour and capability testing because
- it may be
- possible to use the same test steps for both a behaviour test purpose and
- for a capability test purpose. The test suite specifier shall provide an
- explanation of how the test purposes are derived from or relate to the
- protocol
- Recommendation*. The test suite specifier shall also provide a summary
- of the coverage achieved by the test suite.
- .RT
- .sp 1P
- .LP
- 10.2
- \fIThe test group structure\fR
- .sp 9p
- .RT
- .PP
- In order to ensure that the resulting abstract test suite provides adequate
- coverage of the relevant conformance requirements, the test suite
- specifier is advised to design the test suite structure in terms of nested
- test groups in a top down manner (see Figure\ 1/X.290, Part\ 2).
- .PP
- There are many ways of structuring the same test suite into test
- groups; no one way is necessarily right and the best approach for one test
- suite may not be appropriate for another test suite. Nevertheless, the test
- suite specifier shall ensure that the test suite includes test cases for
- whichever of the following categories is relevant:
- .RT
- .LP
- a)
- capability tests (for static conformance requirements);
- .LP
- b)
- behaviour tests of valid behaviour;
- .LP
- c)
- behaviour tests of syntactically invalid behaviour;
- .LP
- d)
- behaviour tests of inopportune behaviour;
- .LP
- e)
- tests focusing on PDUs sent to the IUT;
- .LP
- f
- )
- tests focusing on PDUs received from the IUT;
- .LP
- g)
- tests focusing on interactions between what is sent and
- received;
- .LP
- h)
- tests related to each protocol mandatory feature;
- .LP
- i)
- tests related to each optional feature which is
- implemented;
- .LP
- j)
- tests related to each protocol phase;
- .LP
- k)
- variations in the test event occurring in a particular
- state;
- .LP
- l)
- timing and timer variations;
- .LP
- m)
- PDU encoding variations;
- .LP
- n)
- variations in values of individual parameters;
- .LP
- o)
- variations in combinations of parameter
- values.
- .bp
- .PP
- This list is not exhaustive; additional categories might be needed to ensure
- adequate coverage of the relevant conformance requirements for a
- specific test suite. Furthermore, these categories overlap one another
- and it is the task of the test suite specifier to put them into an appropriate
- hierarchical structure. The following structure is an example of a
- single\(hylayer test suite, provided for guidance:
- .LP
- A
- Capability tests
- .LP
- A.1
- Mandatory features
- .LP
- A.2
- Optional features said by the PICS to be
- supported
- .LP
- B
- Behaviour tests: response to valid behaviour by peer
- implementation
- .LP
- B.1
- Connection establishment phase (if relevant)
- .LP
- B.1.1
- Focus on what is sent to the IUT
- .LP
- B.1.1.1
- Test event variation in each
- state
- .LP
- B.1.1.2
- Timing/timer variation
- .LP
- B.1.1.3
- Encoding variation
- .LP
- B.1.1.4
- Individual parameter value
- variation
- .LP
- B.1.1.5
- Combination of parameter
- values
- .LP
- B.1.2
- Focus on what is received from the IUT
- .LP
- \(em\ substructured as B.1.1
- .LP
- B.1.3
- Focus on interactions
- .LP
- \(em\ substructured as B.1.1
- .LP
- B.2
- Data transfer phase
- .LP
- \(em\ substructured as B.1
- .LP
- B.3
- Connection release phase (if relevant)
- .LP
- \(em\ substructured as B.1
- .LP
- C
- Behaviour tests: response to syntactically invalid behaviour
- by peer implementation
- .LP
- C.1
- Connection establishment phase (if relevant)
- .LP
- C.1.1
- Focus on what is sent to the IUT
- .LP
- C.1.1.1
- Test event variation in each
- state
- .LP
- C.1.1.2
- Encoding variation of the
- invalid event
- .LP
- C.1.1.3
- Individual invalid parameter
- value variation
- .LP
- C.1.1.4
- Invalid parameter value
- combination variation
- .LP
- C.1.2
- Focus on what the IUT is requested to send
- .LP
- C.1.2.1
- Individual invalid parameter
- values
- .LP
- C.1.2.2
- Invalid combinations of
- parameter values
- .LP
- C.2
- Data transfer phase
- .LP
- \(em\ substructured as C.1
- .LP
- C.3
- Connection release phase (if relevant)
- .LP
- \(em\ substructured as C.1
- .LP
- D
- Behaviour tests: response to inopportune events by peer
- implementation
- .LP
- D.1
- Connection establishment phase (if relevant)
- .LP
- D.1.1
- Focus on what is sent to the IUT
- .LP
- D.1.1.1
- Test event variation in each
- state
- .LP
- D.1.1.2
- Timing/timer variation
- .LP
- D.1.1.3
- Special encoding variations
- .LP
- D.1.1.4
- Major individual parameter
- value variations
- .LP
- D.1.1.5
- Variation in major combination
- of parameter values
- .LP
- D.1.2
- Focus on what is requested to be sent by
- the IUT
- .LP
- \(em\ substructured as D.1.1
- .LP
- D.2
- Data transfer phase
- .LP
- \(em\ substructured as D.1
- .LP
- D.3
- Connection release phase (if relevant)
- .LP
- \(em\ substructured as D.1
- .bp
- .PP
- If the test suite is to cover more than one layer, then a
- single\(hylayer test suite structure such as this could be replicated for each
- layer concerned. In addition, a correspondingly detailed structure could be
- produced for testing the capabilities and behaviour of multiple layers taken
- as a whole, including the interaction between the activities in adjacent
- layers.
- .sp 1P
- .LP
- 10.3
- \fITest purposes\fR
- .sp 9p
- .RT
- .PP
- The test suite specifier shall ensure that, for each test case in an abstract
- test suite, there is a clear statement of the test purpose. It is suggested
- that these test purposes should be produced as the next refinement of the
- test suite, after its structure in terms of test groups has been defined.
- The test purposes could be produced directly from studying those sections
- in
- the relevant Recommendation(s)* which are appropriate to the test group
- concerned. For some test groups, the test purposes might be derivable directly
- from the protocol state table; for others, they might be derivable from
- the PDU encoding definitions or the descriptions of particular parameters,
- or from text which specifies the relevant conformance requirements. Alternatively,
- the test suite specifier could employ a formal description of the protocol(s)
- concerned and derive test purposes from that by means of some automated
- method.
- .PP
- Whatever method is used to derive the test purposes, the test
- suite specifier shall ensure that they provide an adequate coverage of the
- conformance requirements of the Recommendation(s)* concerned. There
- shall be at least one test purpose related to each distinct conformance
- requirement.
- .PP
- In addition, it is possible to give guidance on the meaning of
- \*Qadequate coverage\*U with reference to the above example. In order to
- express
- this, a shorthand notation will be used: the letter \*Qx\*U will represent all
- appropriate values for the first digit in the test group identifier, and
- similarly \*Qy\*U for the second digit, so that B.x.y.1 stands for B.1.1.1,
- B.1.2.1, B.1.3.1, B.2.1.1, B.2.2.1, B.2.3.1, B.3.1.1, B.3.2.1 and\ B.3.3.1.
- With this notation, a minimum \*Qadequate coverage\*U for the above example
- is
- considered to be as follows:
- .RT
- .LP
- a)
- for capability test groups (A.1, A.2):
- .LP
- 1)
- at least one test purpose per relevant feature, class
- or subset;
- .LP
- 2)
- at least one test purpose per relevant PDU type and
- each major variation of each such type, using \*Qnormal\*U or
- default values for each parameter;
- .LP
- b)
- for test groups concerned with test event variation in
- each state (B.x.y.1, C.x.1.1, D.x.y.1):
- .LP
- \(em
- at least one test purpose per relevant state/event
- combination;
- .LP
- c)
- for test groups concerned with timers and timing (B.x.y.2,
- D.x.y.2):
- .LP
- 1)
- at least one test purpose concerned with the expiry
- of each defined timer;
- .LP
- 2)
- at least one test purpose concerned with very rapid
- response for each relevant type of PDU;
- .LP
- 3)
- at least one test purpose concerned with very slow
- response for each relevant type of PDU;
- .LP
- d)
- for test groups concerned with encoding variations
- (B.x.y.3, C.x.1.2, D.x.y.3):
- .LP
- \(em
- at least one test purpose for each relevant kind of
- encoding variation per relevant PDU type;
- .LP
- e)
- for test groups concerned with valid individual parameter
- values (B.x.y.4, D.x.y.4):
- .LP
- 1)
- for each relevant integer parameter, test purposes
- concerned with the boundary values and one randomly
- selected mid\(hyrange value;
- .LP
- 2)
- for each relevant bitwise parameter, test purposes
- for as many values as practical, but not less than all the
- \*Qnormal\*U or common values;
- .LP
- 3)
- for other relevant parameters, at least one test
- purpose concerned with a value different from what is
- considered \*Qnormal\*U or default in other test
- groups;
- .LP
- f
- )
- for test groups concerned with invalid individual
- parameter values (C.x.1.3, C.x.2.1):
- .LP
- 1)
- for each relevant integer parameter, test purposes
- concerned with invalid values adjacent to the allowed
- boundary values plus one other randomly selected invalid
- value;
- .LP
- 2)
- for each relevant bitwise parameter, test purposes
- for as many invalid values as practical;
- .LP
- 3)
- for all other relevant types of parameter, at least
- one test purpose per parameter;
- .bp
- .LP
- g)
- for test groups concerned with combinations of parameter
- values (B.x.y.5, C.x.1.4, C.x.2.2, D.x.y.5):
- .LP
- 1)
- at least one test purpose per \*Qcritical\*U value
- pair;
- .LP
- 2)
- at least one test purpose per pair of interrelated
- parameters to test a random combination of relevant
- values.
- .sp 2P
- .LP
- \fB11\fR \fBGeneric test case specification\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 11.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- The test suite specifier is recommended to specify a generic test suite,
- particularly if there is an intention to produce more than one abstract
- test suite.
- .PP
- A generic test suite shall consist of one generic test case for
- each test purpose. Each generic test case will be a refinement of its test
- purpose, which can be used as a common root of corresponding abstract test
- cases for different test methods.
- .PP
- If a generic test suite is produced in advance of abstract test
- suites, then it will be a useful step in the design process. If the generic
- test suite is produced after the production of at least one abstract test
- suite, then it will provide a means of relating different test suites
- to one another and analyzing where there may be gaps in their
- coverage.
- .RT
- .sp 1P
- .LP
- 11.2
- \fIDescription of generic test cases\fR
- .sp 9p
- .RT
- .PP
- A generic test case consists of a test description of an \*Qinitial state\*U
- [of the test body] and a specification of the test body in a
- standardized test notation. The \*Qinitial state\*U includes not only the
- protocol state, but also any necessary information concerning the state
- of the SUT and the testing environment.
- .PP
- The test body is that part of a test case in which verdicts
- related to the test purpose are assigned to foreseen outcomes. The test
- body:
- .RT
- .LP
- a)
- shall be defined in the style of either the Remote or
- Distributed test method;
- .LP
- \fINote\fR \ \(em\ Generic test cases may use the full expressive
- power of these methods (see \(sc\ 12.3.3 for details), including the
- use of abstract local primitives.
- .LP
- b)
- shall assign verdicts to all possible outcomes; all
- outcomes yielding \*Qpass\*U verdicts shall be explicitly
- identified, whereas all outcomes yielding \*Qfail\*U or
- \*Qinconclusive\*U verdicts shall be either identified or
- categorized (which may include categorization of default
- verdicts);
- .LP
- c)
- shall be described using a standardized test notation; the
- Tree and Tabular Combined Notation (TTCN) (defined in Annex\ D)
- is recommended.
- .sp 1P
- .LP
- 11.3
- \fIRelation of generic to abstract test cases\fR
- .sp 9p
- .RT
- .PP
- For a particular abstract test method there may be many abstract
- test cases that can be derived from a single generic test case. A major
- difference between an abstract test case and a generic test case is that the
- abstract test case includes specifications of a preamble and a postamble.
- The preamble starts from a chosen stable state and leads to the required
- initial
- state of the test body. The postamble starts from the end of the test body
- and returns to a chosen stable state.
- .PP
- The preamble and postamble may be realized in different ways
- depending on the degree of control and observation provided by the test
- method used, or on the variety of different possible stable states which
- the derived abstract test case can start from and end in. These abstract
- test cases are
- simply different ways of achieving the same test purpose.
- .PP
- In addition, the test body of an abstract test case may be different from
- the corresponding generic test case if the test method used for the
- abstract test case is different from that used for the generic
- test case.
- .PP
- If a generic test suite is produced, it shall be used as the means
- of relating corresponding abstract test suites for different abstract test
- methods.
- .bp
- .RT
- .sp 2P
- .LP
- \fB12\fR \fBAbstract test methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 12.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- Each abstract test suite shall be specified in terms of the control and
- observation of events in accordance with one of the abstract test methods
- defined in this section. The chosen test method determines the points at
- which control and observation shall be specified and the categories of
- events which shall be used [e.g.\ (N\ \(em\ 1)\(hyASPs, (N)\(hyPDUs].
- .RT
- .sp 2P
- .LP
- 12.2
- \fIGeneral specification of the abstract test methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 12.2.1
- \fIEnd\(hysystem IUTs\fR
- .sp 9p
- .RT
- .PP
- For IUTs within end\(hysystem SUTs there are four categories of
- abstract test methods, one local, and three external ones which assume the
- lower tester is located remotely from the SUT and connected to it by a
- link or network.
- .PP
- For generality, the test methods are described with reference to
- an IUT in which the highest layer is numbered \*QN\dt\u\*U (for \*Qtop\*U)
- and in
- which the lowest layer protocol to be tested is numbered \*QN\db\u\*U (for
- \*Qbottom\*U). The IUT may implement protocols in layers lower than \*QN\db\u\*U,
- but these are not of interest in the test method descriptions. The description
- applies to single\(hylayer IUTs by taking N\dt\uto be equal to
- N\db\u.
- .RT
- .sp 1P
- .LP
- 12.2.2
- \fIAbstract local primitives\fR
- .sp 9p
- .RT
- .PP
- Abstract test suite specifiers may, if necessary, define a set of abstract
- local primitives (ALPs) to be used in specifying the test suite. An
- ALP is an abbreviation for a description of control and/or observation to be
- performed by the upper tester, which cannot be described in terms of ASPs
- but which initiate events or state changes defined within the relevant
- protocol
- Recommendation(s)*. ALPs may be used with any abstract test method for
- end\(hysystem IUTs, but, for simplicity, will not be shown in any of the
- figures which illustrate those methods.
- .PP
- Any test case which uses an ALP shall be optional in the abstract
- test suite.
- .RT
- .sp 1P
- .LP
- 12.2.3
- \fIThe local test methods\fR
- .sp 9p
- .RT
- .PP
- \fIAbbreviation: L\fR
- .PP
- In this method:
- .RT
- .LP
- a)
- the abstract test suite is specified in terms of control
- and observation of (N\db\u\ \(em\ 1)\(hyASPs and (N\db\u) to
- (N\dt\u)\(hyPDUs;
- .LP
- b)
- the abstract test suite is also specified in terms of
- control and observation of (N\dt\u)\(hyASPs;
- .LP
- c)
- the abstract test suite may also be specified in terms of
- control and observation by the upper tester of abstract
- local primitives (ALPs);
- .LP
- d)
- the method requires access to the lower and upper
- boundaries of the IUT and a mapping between the specified ASPs
- and their realization within the SUT;
- .LP
- e)
- the requirements for the test coordination procedures are
- specified in the abstract test suite but no assumption is made
- regarding their realization;
- .LP
- f
- )
- the upper and lower testers are required to achieve
- control and observation of the specified ASPs and the required
- test coordination procedures. They are assumed to be integrated
- into the SUT.
- .PP
- This method is illustrated in Figure\ 1/X.290, Part\ 2.
- .sp 2P
- .LP
- 12.2.4
- \fIExternal test methods\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 12.2.4.1
- \fIThe distributed test method\fR
- .sp 9p
- .RT
- .PP
- \fIAbbreviation: D\fR
- .PP
- In this method:
- .RT
- .LP
- a)
- the abstract test suite is specified in terms of control
- and observation of (N\db\u\ \(em\ 1)\(hyASP\*Us and (N\db\u) to
- (N\dt\u)\(hyPDUs;
- .bp
- .LP
- b)
- the abstract test suite is also specified in terms of
- control and observation of (N\dt\u)\(hyASPs; the method
- requires access to the upper boundary of the IUT and a mapping
- between the (N\dt\u)\(hyASPs and their realization within
- the SUT;
- .LP
- c)
- the abstract test suite may also be specified in terms of
- control and observation by the upper tester of ALPs;
- .LP
- d)
- the requirements for the test coordination procedures are
- specified in the abstract test suite but no assumption is made
- regarding their realization;
- .LP
- e)
- the upper tester is required to achieve control and
- observation of the specified (N\dt\u)\(hyASPs and to achieve
- the effects of the required test coordination procedures; no
- other assumptions are made;
- .LP
- f
- )
- the lower tester is required to achieve control and
- observation of specified (N\db\u)\(hyASP\*Us and specified
- PDUs and to achieve the required test coordination
- procedures.
- .PP
- This method is illustrated in Figure\ 2/X.290, Part\ 2.
- .sp 1P
- .LP
- 12.2.4.2
- \fIThe\fR
- \fIcoordinated test method\fR
- .sp 9p
- .RT
- .PP
- \fIAbbreviation: C\fR
- .PP
- In this method:
- .RT
- .LP
- a)
- the abstract test suite is specified in terms of control and
- observation of (N\db\u\ \(em\ 1)\(hyASP\*Us, (N\db\u) to
- (N\dt\u)\(hyPDUs and test management PDUs (TM\(hyPDUs);
- .LP
- b)
- (N\dt\u)\(hyASPs need not be used in the specification
- of the abstract test suite; no assumption is made about the
- existence of an upper boundary of the IUT;
- .LP
- c)
- the requirements for the test coordination procedures are
- specified in the abstract test suite by means of a standardized
- test management protocol;
- .LP
- d)
- TM\(hyPDUs may be defined which correspond to ALPs;
- .LP
- e)
- the upper tester is required to implement the test
- management protocol and achieve the appropriate effects on the
- IUT;
- .LP
- f
- )
- the lower tester is required to achieve control and
- observation of specified (N\db\u)\(hyASP\*Us and specified
- PDUs (including TM\(hyPDUs).
- .PP
- This method is illustrated in Figure\ 3/X.290, Part\ 2.
- .sp 1P
- .LP
- 12.2.4.3
- \fIThe\fR
- \fIremote test method\fR
- .sp 9p
- .RT
- .PP
- \fIAbbreviation: R\fR
- .PP
- In this method, provision is made for the case where it is not
- possible to observe and control the upper boundary of the IUT. Also in this
- method:
- .RT
- .LP
- a)
- the abstract test suite is specified in terms of control
- and observation of (N\db\u\ \(em\ 1)\(hyASP\*Us and (N\db\u) to
- (N\dt\u)\(hyPDUs ;
- .LP
- b)
- (N\dt\u)\(hyASPs are not used in the specification of
- the abstract test suite; no assumption is made about the
- existence of an upper boundary of the IUT;
- .LP
- c)
- the abstract test suite may also be described in terms of
- control and observation of ALPs within the SUT;
- .LP
- d)
- some requirements for test coordination procedures may be
- implied or informally expressed in the abstract test suite but
- no assumption is made regarding their feasibility or
- realization;
- .LP
- e)
- abstractly the SUT needs to carry out some upper tester
- functions to achieve whatever effects of test coordination
- procedures and whatever control andB/For observation of the IUT
- are implied, informally expressed or described by ALPs in the
- abstract test suite for a given protocol; these functions are
- not specified nor are any assumptions made regarding their
- feasibility or realization;
- .LP
- f
- )
- the lower tester is required to achieve control and
- observation of specified (N\db\u)\(hyASP\*Us and specified
- PDUs and should attempt to achieve the implied or informally
- expressed test coordination procedures in accordance with the
- relevant information in the PIXIT.
- .PP
- This method is illustrated in Figure\ 4/X.290, Part\ 2.
- .bp
- .LP
- .rs
- .sp 29P
- .ad r
- \fBFigures 1, 2, 3 et 4/X.290, partie\ 2, (N), p.\fB
- .sp 1P
- .RT
- .ad b
- .RT
- .sp 1P
- .LP
- 12.2.5
- \fISingle\(hylayer, multi\(hylayer and embedded variants\fR
- .sp 9p
- .RT
- .PP
- Each category of test methods has a variant which can be applied
- to single\(hylayer IUTs (abbreviation:\ S), and another which can be applied to
- multi\(hylayer IUTs (abbreviation:\ M), when the set of adjacent layers
- is to be
- tested in combination (as a whole).
- .PP
- For a multi\(hylayer IUT in which the protocols are to be tested
- layer by layer, an embedded variant of the test methods has been defined
- (abbreviation:\ E).
- .PP
- If control and observation are applied to a means of access to the
- upper boundary of the entities under test within SUT, then the test methods
- are normal (and no E is added to the abbreviated name). If, however, control
- and
- observation are applied through one or more OSI* layer entities above the
- entities under test, the test methods are called embedded (and an\ E is
- appended to the abbreviated name).
- .PP
- Names of particular variants of the test methods shall be formed
- as follows:
- .RT
- .ce 1000
- .sp 1
- |
- \ L\
- |
- .ce 0
- .ce 1000
- \ R\
- .ce 0
- .ce 1000
- |
- \ C\
- |
- .ce 0
- .ce 1000
- \ D\
- |
- \ S\
- |
- .ce 0
- .PP
- \ M\
- [\ E\ ]
- For example, DSE is the abbreviation for the \*Qdistributed
- single embedded\*U test method.
- .bp
- .sp 1P
- .LP
- 12.2.6
- \fIOpen relay\(hysystems\fR
- .sp 9p
- .RT
- .PP
- For open relay\(hysystems, loop\(hyback and transverse abstract test
- methods are defined. They are given the abbreviated names: YL and YT,
- respectively.
- .RT
- .sp 1P
- .LP
- 12.3
- \fISingle\(hylayer test methods for single\(hylayer IUTs in\fR
- \fIend\(hysystems\fR
- .sp 9p
- .RT
- .PP
- For single\(hylayer test methods, the abstract model of the IUT is
- called the (N)\(hyentity under test.
- .RT
- .sp 1P
- .LP
- 12.3.1
- \fIThe local single\(hylayer test method\fR
- .sp 9p
- .RT
- .PP
- The Local Single\(hylayer (LS) abstract test method defines the points
- of control and observation as being at the service boundaries above and
- below the (N)\(hyentity under test. The test events are specified in terms
- of
- (N)\(hyASPs above the IUT, and (N\ \(em\ 1)\(hyASPs and (N)\(hyPDUs below
- the IUT, as shown in Figure\ 5/X.290, Part\ 2. In addition, ALPs may be
- used as test events. Abstractly, a lower tester is considered to observe
- and control
- the (N\ \(em\ 1)\(hyASPs and (N)\(hyPDUs while an upper tester observes and
- controls the (N)\(hyASPs and ALPs. The requirements to be met by test
- coordination procedures used to coordinate the realizations of the upper and
- lower testers are defined in the abstract test suites, although the test
- coordination procedures themselves are not.
- .RT
- .sp 1P
- .LP
- 12.3.2
- \fIThe distributed single\(hylayer test method\fR
- .sp 9p
- .RT
- .PP
- The Distributed Single\(hylayer (DS) abstract test method defines the points
- of control and observation as being at the service boundaries above the
- (N)\(hyentity under test and above the (N\ \(em\ 1)\(hyService Provider
- at the SAP remote from the (N)\(hyentity under test. The test events are
- specified in terms of
- (N)\(hyASPs above the IUT and (N\ \(em\ 1)\(hyASP\*Us and (N)\(hyPDUs remotely,
- as shown in
- Figure\ 6/X.290, Part\ 2. In addition, ALPs may be used as test events.
- Abstractly, lower and upper testers are again considered to observe and
- control the behaviour at the respective points. The requirements to be
- met by the test coordination procedures are again defined in the abstract
- test suites, although the procedures themselves are not.
- .PP
- For lower layers (1\(hy3) where it may be unrealistic to specify
- observation and control of (N\ \(em\ 1)\(hyASP\*Us, the lower tester observation
- and
- control shall be specified in terms of (N)\(hyPDUs and, when necessary,
- changes in the state of the underlying connection.
- .PP
- \fINote\fR \ \(em\ For example, the state of the underlying connection
- could be changed by setting up a new connection, or resetting or closing
- an existing
- connection.
- .PP
- The observation and control to be performed by the lower tester can
- optionally be specified in terms of (N)\(hyASP\*Us where this will reduce
- the size of the test case specification without loss of required
- precision.
- .RT
- .LP
- .rs
- .sp 15P
- .ad r
- \fBFigures 5 et 6/X.290, partie\ 2, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .LP
- 12.3.3
- \fIThe coordinated single\(hylayer test method\fR
- .sp 9p
- .RT
- .PP
- The Coordinated Single\(hylayer (CS) abstract test method is an
- enhanced version of the DS method, using a standardized upper tester and the
- definition of a test management protocol to realize the test coordination
- procedures between the upper and lower testers. The same standardized upper
- tester and test management protocol are not necessarily applicable to all
- test suites which use the coordinated method.
- .PP
- Standardized upper testers and test management protocols are
- applicable to a particular standardized abstract test suite for the coordinated
- test method and may not be applicable to other abstract test suites for
- the
- coordinated method.
- .PP
- There is only a single point of control and observation, above the
- (N\ \(em\ 1)\(hyService Provider at the SAP remote from the (N)\(hyentity
- under test. Test events are specified in terms of (N\ \(em\ 1)\(hyASP\*Us,
- (N)\(hyPDUs and TM\(hyPDUs, as shown in Figure\ 7/X.290, Part\ 2.
- .PP
- For lower layers (1\(hy3) where it may be unrealistic to specify
- observation and control of (N\ \(em\ 1)\(hyASP\*Us, the observation and
- control shall
- be specified in terms of TM\(hyPDUs, (N)\(hyPDUs and, when necessary, changes
- in the state of the underlying connection.
- .PP
- Concerning the test management protocol:
- .RT
- .LP
- a)
- the test management protocol shall be implemented within
- the SUT directly above the abstract service boundary at the top
- of the IUT;
- .LP
- b)
- the IUT shall not be required to interpret TM\(hyPDUs, only
- pass them to and from the upper tester;
- .LP
- c)
- a test management protocol is only defined for testing a
- particular protocol and so does not need to be independent of
- the underlying protocol;
- .LP
- d)
- verdicts on test cases shall not be based on the ability
- of the SUT to exhibit any ASP or parameter of an ASP at the
- upper service boundary of the IUT, since this would contradict
- the definition of the coordinated method in that the upper
- service boundary of the IUT is not a point of control and
- observation in this method. However, it is recommended that
- the test management protocol be defined separately from the
- abstract test suite(s) in order to facilitate the task of the
- implementor of an upper tester. This definition (as with the
- definition of any OSI* protocol defined by ISO or CCITT) can
- refer to the ASPs of its underlying service (i.e.\ the ASPs at
- the upper service boundary of the IUT);
- .LP
- e)
- TM\(hyPDUs which correspond to ALPs are optional and there is
- no requirement for the upper tester to support
- them.
- .sp 1P
- .LP
- 12.3.4
- \fIThe remote single\(hylayer test method\fR
- .sp 9p
- .RT
- .PP
- The Remote Single\(hylayer (RS) abstract test method defines the point
- of control and observation as being above the (N\ \(em\ 1)\(hyService Provider
- at the SAP remote from the (N)\(hyentity under test. The test events are
- specified in
- terms of the (N\ \(em\ 1)\(hyASP\*Us and (N)\(hyPDUs remotely, as shown
- in Figure\ 8/X.290, Part\ 2. In addition, ALPs may be used as test events.
- Some requirements for
- test coordination procedures may be implied or informally expressed in the
- abstract test suites, but no assumptions shall be made regarding their
- feasibility or realization.
- .PP
- For the lower layers (1\(hy3) where it is unrealistic to specify
- observation and control of (N\ \(em\ 1)\(hyASP\*Us, the observation and
- control shall
- be specified in terms of (N)\(hyPDUs and when necessary the state of the
- underlying connection.
- .PP
- In addition, in order to overcome the lack of specification of
- behaviour above the (N)\(hyentity under test, where necessary the required
- behaviour of the system under test shall be specified in terms of the
- (N\ \(em\ 1)\(hyASP\*Us or (N)\(hyPDUs which need to be observed by the
- lower tester.
- This form of implicit specification shall be taken to mean \*Qdo whatever
- is necessary within the system under test in order to provoke this required
- behaviour\*U.
- .PP
- \fINote\fR \ \(em\ Such implicit specification is necessary with this test
- method for any test case which requires an IUT initiated event which cannot
- be initiated by means of an ALP. Since ALPs may only be defined if the
- same effect cannot be achieved by ASPs, then any PDU which can be initiated
- by an ASP needs implicit specification to allow it to be initiated in this
- test
- method.
- .PP
- However, it is possible that some of the test cases in the
- abstract test suite cannot be executed (e.g.\ transmission and maintenance of
- busy conditions, transmission of consecutive unacknowledged Data PDUs,\
- etc.). In such instances, it is left to the test laboratory and client
- to negotiate
- the method by which these tests can be accomplished.
- .bp
- .PP
- Even with such implicit specification of control of the IUT, in
- this method it is possible to specify control but not observation above the
- IUT, except through the use of ALPs. This is a major difference between this
- and the DS and CS test methods.
- .RT
- .LP
- .rs
- .sp 18P
- .ad r
- \fBFigures 7 et 8/X.290, partie\ 2, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .sp 1P
- .LP
- 12.4
- \fIMulti\(hylayer test methods for multi\(hylayer IUTs (LM, DM, CM, RM)\fR
- .sp 9p
- .RT
- .PP
- Multi\(hylayer testing, when the combined allowed behaviour of the
- multi\(hylayer implementation is known, involves testing all the layers of a
- multi\(hylayer IUT as a whole, without controlling or observing any of the
- inter\(hylayer boundaries within the IUT.
- .PP
- In the Local Multi\(hylayer (LM) test method the points of observation
- and control are the service boundaries directly above and below the IUT. The
- test events shall be specified in terms of the (N\dt\u)\(hyASPs and ALPs
- above the IUT and the (N\ \(em\ 1)\(hyASPs and (N) to (N\dt\u)\(hyPDUs below
- the IUT.
- .PP
- In the Distributed Multi\(hylayer (DM) test method the points of
- observation and control are at the service boundary above the IUT and above
- the (N\ \(em\ 1)\(hyService Provider at the SAP remote from the IUT. The
- test events
- shall be specified in terms of the (N\dt\u)\(hyASPs and ALPs above the
- IUT and the (N\ \(em\ 1)\(hyASP\*Us and (N) to (N\dt\u)\(hyPDUs remotely.
- .PP
- In the Coordinated Multi\(hylayer (CM) test method the point of
- observation and control is above the (N\ \(em\ 1)\(hyService Provider at
- the SAP
- remote from the IUT. The test events shall be specified in terms of the
- (N\ \(em\ 1)\(hyASP\*Us, the (N) to (N\dt\u)\(hyPDUs and the TM\(hyPDUs. The
- test management protocol shall be designed to operate over the
- (N\dt\u)\(hyService, where (N\dt\u) is the highest layer in the
- IUT.
- .PP
- In the Remote Multi\(hylayer (RM) test method the point of observation
- and control is above the (N\ \(em\ 1)\(hyService Provider at the SAP remote
- from the
- IUT. The test events shall be specified in terms of the (N\ \(em\ 1)\(hyASP\*Us
- and the (N) to (N\dt\u)\(hyPDUs, ALPs and the implicit specification of
- the
- control of the SUT behaviour when necessary. Some requirements for test
- coordination procedures may be implied or informally expressed, but no
- assumptions shall be made regarding their feasibility or
- realization.
- .RT
- .sp 1P
- .LP
- 12.5
- \fISingle\(hylayer testing of multi\(hylayer IUTs or SUTs (Embedded\fR
- \fImethods)\fR
- .sp 9p
- .RT
- .PP
- In embedded single\(hylayer test methods testing is specified for a
- single\(hylayer within a multi\(hylayer IUT, including the specification of the
- protocol activity in the layers above the one being tested but without
- specifying control or observation at service boundaries within the multi\(hylayer
- IUT. Thus in a multi\(hylayer IUT from layer (N) to (N\dt\u),
- abstract test cases for testing layer\ (N\di\u) shall include the
- specification of the PDUs in layers\ (N\di\u+1) to (N\dt\u) as well as
- those in layer\ (N\di\u).
- .bp
- .PP
- The Local Single\(hylayer Embedded (LSE) test method uses the same points
- of control and observation as the LM test method for the same set of layers.
- The test events shall also be specified in the same terms as for the LM test
- method. The difference is that the LSE test method concentrates on a
- single\(hylayer at a time, whereas the LM test method tests the multi\(hylayer
- IUT as a whole.
- .PP
- In the Distributed Single\(hylayer Embedded (DSE) test method for layer
- (N\di\u) within a multi\(hylayer IUT from layer (N) to (N\dt\u) the
- points of observation and control are at the service boundary above the
- IUT and above the (N\di\u\ \(em\ 1)\(hyService Provider at the SAP remote
- from the IUT, as
- illustrated in Figure\ 9a)/X.290, Part\ 2. The test events shall be specified
- in terms of (N\dt\u)\(hyASPs and ALPs above the IUT and (N\di\u\ \(em\
- 1)\(hyASP\*Us
- and (N\di\u) to (N\dt\u)\(hyPDUs remotely.
- .PP
- \fINote\fR \ \(em\ For the top layer in the multi\(hylayer IUT, (N\dt\u),
- this method is the same as the DS test method.
- .PP
- The Coordinated Single\(hylayer Embedded (CSE) test method uses features
- of both the CM and DSE test methods. The test events shall be specified
- in
- terms of (N\di\u\ \(em\ 1)\(hyASP\*Us, (N\di\u) to (N\dt\u)\(hyPDUs and
- TM\(hyPDUs and the test management protocol shall be designed to operate over
- the (N\dt\u)\(hyService. This is illustrated in Figure\ 9b)/X.290,
- Part\ 2.
- .PP
- The Remote Single\(hylayer Embedded (RSE) test method uses the same
- point of control and observation as the RS test method for the same layer,
- but differs from the RS test method in that (N\di\u+1) to (N\dt\u)\(hyPDUs
- shall be specified in test cases for layer (N\di\u).
- .PP
- Successive use of a single\(hylayer embedded test method (from layer
- (N) to (N\dt\u)) is called incremental testing of a
- multi\(hylayer IUT.
- .PP
- The DSE/CSE/RSE methods are defined for a single layer under test
- in a multi\(hylayer IUT. This does not mean that there cannot be accessible
- service boundaries within the multi\(hylayer IUT, just that no such boundaries
- are used in the test methods. Thus, all layers between the layer under
- test and the highest layer for which PDUs are expressed as test events
- in the abstract test suite shall be regarded as being part of the multi\(hylayer
- IUT.
- .PP
- \fINote\fR \ \(em\ DME/CME/RME test methods could theoretically be defined to
- test multiple layers as a whole within a larger multi\(hylayer IUT, but
- in order to avoid excess complexity, they are not detailed in this
- Recommendation.
- .RT
- .LP
- .rs
- .sp 25P
- .ad r
- \fBFigure 9/X.290, partie\ 2, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 1P
- .LP
- 12.6
- \fIRelay test methods\fR
- .sp 9p
- .RT
- .PP
- There are two abstract test methods for relay system
- testing:
- .RT
- .LP
- a)
- the loop\(hyback test method (YL): used for testing a relay
- system from one subnetwork.
- .LP
- This test method is illustrated in Figure\ 10/X.290,
- Part\ 2.
- .LP
- For this test method there are two points of observation and
- control on one subnetwork at SAPs remote from the (N)\(hyRelay.
- For connection\(hyoriented protocols, it requires that the two test
- connections are looped together on the far side of the
- (N)\(hyRelay, but it is not specified whether this looping is
- performed within the (N)\(hyRelay or in the second subnetwork. For
- connectionless protocols, it requires that the PDUs are looped
- back within the second subnetwork and addressed to return to the
- second point of observation and control.
- .LP
- b)
- the transverse test method (YT): used for testing a relay
- system from two subnetworks.
- .LP
- This test method is illustrated in Figure\ 11/X.290,
- Part\ 2.
- .LP
- In this test method there are two points of observation and
- control, one on each subnetwork, at SAPs remote from the
- (N)\(hyRelay.
- .LP
- .rs
- .sp 16P
- .ad r
- \fBFigure 10/X.290, partie\ 2, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .rs
- .sp 16P
- .ad r
- \fBFigure 11/X.290, partie\ 2, (N), p.\fR
- .sp 1P
- .RT
- .ad b
- .RT
- .LP
- .bp
- .sp 2P
- .LP
- 12.7
- \fIChoice of test method\fR
- .sp 1P
- .RT
- .sp 1P
- .LP
- 12.7.1
- \fIIntroduction\fR
- .sp 9p
- .RT
- .PP
- Before an abstract test suite can be defined, it is necessary to
- study all the environments in which the protocol is likely to be tested and
- establish accordingly the abstract test method(s) to be used for the production
- of one or more abstract test suites.
- .PP
- Abstract test suite specifiers shall place a requirement in the
- abstract test suite Recommendation* defining which abstract test method(s)
- shall be supported as a minimum by any organization claiming to provide a
- comprehensive testing service for the protocol(s) in question.
- .RT
- .sp 1P
- .LP
- 12.7.2
- \fIIUT environments\fR
- .sp 9p
- .RT
- .PP
- There is a relation between the test methods and the
- configurations of the real open systems to be tested.
- .PP
- Section 7.2 Part 1 of this Recommendation gives a full account of
- the classification of systems and IUTs.
- .PP
- When choosing a test method, the test suite specifiers should
- identify, if they have not already done so, whether the test suites they
- plan to produce are for IUTs which
- .RT
- .LP
- a)
- are single or multi\(hylayer;
- .LP
- b)
- belong to end or relay systems;
- .LP
- c)
- belong to complete or partial systems;
- .LP
- d)
- belong to fully open or mixed systems;
- .LP
- e)
- have service boundaries accessible or not;
- .LP
- f
- )
- are special purpose (i.e. to be used by a single
- application) or general purpose (i.e.\ to be used by several
- applications).
- .sp 1P
- .LP
- 12.7.3
- \fIApplicability of the abstract test methods\fR
- .sp 9p
- .RT
- .PP
- The possibility of developing an abstract test suite for a given
- abstract test method will depend on the protocol(s) being considered, together
- with the results of the study described in \(sc\ 12.7.2. This applies to
- the possibility of developing test suites for a given combination of methods
- (e.g.\ incremental) or a given variant of a method (e.g.\ embedded).
- .PP
- Some considerations concerning the applicability of the methods to
- different layers are discussed in Appendix\ I of Part\ 1 of this
- Recommendation.
- .PP
- One or more appropriate abstract test methods shall be selected
- for the protocol being considered.
- .PP
- Priorities should be assigned to the various test methods which
- have been identified, according to the configurations which are most likely
- to be found in real systems.
- .RT
- .sp 1P
- .LP
- 12.7.4
- \fIIllustrative examples\fR
- .sp 9p
- .RT
- .PP
- Appendix\ III provides an illustrative example of the choice of
- abstract test methods for given protocols.
- .RT
- .sp 1P
- .LP
- 12.8
- \fITest coordination procedures\fR
- .sp 9p
- .RT
- .PP
- For effective and reliable execution of conformance tests, some set of
- rules is required for the coordination of the test process between the
- lower tester and the upper tester. The general objective of these rules
- is to enable the lower tester to control remotely the operation of the
- upper tester or vice versa, in ways necessary to run the test suite selected
- for the IUT.
- .PP
- These rules lead to the development of test coordination procedures to
- achieve the synchronization between the lower tester and the upper tester
- and the management of information exchanged during the testing process.
- The details and how these effects are achieved are closely related to the
- characteristics of the SUT, as well as the external test methods.
- .PP
- For each abstract test suite using the coordinated, distributed or
- local test methods, a standard set of test coordination procedures should be
- developed. This is because those procedures depend on the functionality
- of the upper tester and definitions of test cases.
- .bp
- .PP
- In the case of the coordinated test methods (CS, CSE, CM) the test
- coordination procedures are realized by the standardization of a test
- management protocol. The test management protocol needs to be able to convey
- requests to the IUT to achieve the effect of a service primitive or ALP
- and to convey back to the lower tester the record of observations of effects
- equivalent to the occurrence of service primitives or ALPs. The upper tester
- should be an implementation of the test management protocol. It will be
- necessary to add test cases to the abstract test suite for the purpose of
- testing that the upper tester conforms to the requirements of the test
- management protocol specification. Such test cases do not contribute to the
- conformance assessment of the IUT.
- .PP
- When defining test cases for the local and distributed test methods, the
- test suite specifier shall record any constraints on the upper tester
- and/or test coordination procedures which may be necessary.
- .RT
- .LP
- \fB13\fR \fBSpecification of abstract test suites\fR
- .sp 1P
- .RT
- .sp 2P
- .LP
- 13.1
- \fITest cases\fR
- .sp 1P
- .RT
- .PP
- 13.1.1
- The abstract test suite specifier shall select an appropriate standardized
- notation in which to define the abstract test cases. The Tree and Tabular
- Combined Notation (TTCN), defined in Annex\ D, is defined for this
- purpose.
- .sp 9p
- .RT
- .PP
- 13.1.2
- Once the test notation and test method have been chosen,
- then the generic test cases can be expanded into abstract test cases. There
- are two main kinds of change required to convert a generic test case into
- an
- abstract test case. The first is to express the test body in terms of control
- and observation required by the test method, and, if relevant, include
- a
- description of the synchronization needed between upper and lower
- testers. The second kind of change is to specify the preamble and
- postamble.
- .sp 9p
- .RT
- .PP
- 13.1.3
- Specification of preambles and postambles may result in more
- than one test step for each. The preamble shall start in a stable state and
- progress to the desired state. Conversely, the postamble shall progress from
- the final state of the test body to a stable state. A small number of stable
- states shall be defined for the protocol concerned, always containing as a
- minimum the \*Qidle\*U state. Each abstract test case shall be capable
- of being run on its own and shall therefore include test steps to start
- the preamble from
- the \*Qidle\*U state and to end the postamble in the \*Qidle\*U state.
- .sp 9p
- .RT
- .PP
- However, other stable starting and final states for an abstract
- test case can be useful when efficient concatenation of abstract test cases
- is required.
- .PP
- Furthermore, in designing the test step structure within the abstract test
- cases, the test suite specifier can benefit from using the same test steps
- in several abstract test cases.
- .RT
- .PP
- 13.1.4
- In converting from generic test cases to abstract test
- cases, the test suite specifier shall ensure that the initial state for the
- test body is preserved, the pass paths through the test body are preserved
- and the assignment of verdicts to outcomes remains consistent.
- .sp 9p
- .RT
- .PP
- In order to maintain consistency, the following conditional
- requirements apply when assigning verdicts to outcomes:
- .LP
- a)
- if the behaviour of the preamble and the postamble are
- valid then the verdict assigned to a particular outcome shall be
- the same as that assigned to the corresponding outcome in the
- generic test case;
- .LP
- b)
- if the preamble results in the initial state of the test
- body not being reached, although there is no invalid behaviour,
- then the verdict shall be inconclusive;
- .LP
- c)
- if the preamble results in the initial state of the test
- body not being reached, because of invalid behaviour, then the
- verdict shall be \*Qfail but test purpose inconclusive\*U (\*Qfail
- type\ 3\*U);
- .LP
- d)
- if the postamble exhibits invalid behaviour and the
- generic test case (or test body) verdict is \*Qpass\*U or
- \*Qinconclusive\*U, then the verdict shall be \*Qfail but test purpose
- passed\*U (\*Qfail type\ 2\*U) or \*Qfail but test purpose inconclusive\*U
- (\*Qfail type\ 3\*U), respectively;
- .LP
- e)
- if the generic test case (or test body) verdict is \*Qfail\*U
- then invalid behaviour in the postamble shall not
- change the verdict (\*Qfail type\ 1\*U).
- .bp
- .PP
- 13.1.5
- The test suite specifier shall also ensure that each
- abstract test case explicitly identifies all the outcomes assigned the
- verdict \*Qpass\*U and identifies or categorizes all the remaining foreseen
- outcomes,
- assigning to each individual outcome or category a verdict \*Qfail\*U or
- \*Qinconclusive\*U. All unforeseen outcomes in the test body shall be assigned
- either:
- .sp 9p
- .RT
- .LP
- a)
- the verdict \*Qfail\*U; or
- .LP
- b)
- the verdict \*Qinconclusive\*U.
- .PP
- The test suite specifier shall ensure that the application of a) or b)
- is consistent throughout the abstract test suite. If\ a) is chosen, then
- any unforeseen outcome in the preamble shall be assigned the verdict \*Qfail
- but test purpose inconclusive\*U (\*Qfail type\ 3\*U), and any unforeseen
- outcome in the postamble shall be treated as a protocol violation, leading
- to the appropriate type of fail verdict.
- .PP
- If b) is chosen, then any unforeseen outcome in the preamble shall
- be assigned the verdict \*Qfail but test purpose inconclusive\*U (\*Qfail
- type\ 3\*U), and any unforeseen outcome in the postamble shall be assigned
- the appropriate type of fail verdict.
- .RT
- .sp 1P
- .LP
- 13.2
- \fITest suites\fR
- .sp 9p
- .RT
- .PP
- An abstract test suite specification comprises a set of test cases and
- test steps. Preceding the test cases themselves shall be the following
- information:
- .RT
- .LP
- a)
- abstract test suite name, date of origin and version
- number;
- .LP
- b)
- names (and version numbers) of the protocol
- Recommendation(s)* for which test cases are provided;
- .LP
- c)
- names (and version numbers) of the service
- Recommendation(s)* whose primitives are specified as
- being observed;
- .LP
- d)
- name (and version number) of the Recommendation*
- defining the test notation, or a reference to an annex for the
- same if it is not standardized elsewhere;
- .LP
- e)
- name of target test method;
- .LP
- f
- )
- description of the coverage of the test suite; for
- example, the functional subsets of the protocol
- Recommendation(s)* that are tested;
- .LP
- g)
- description of the structure of the test suite, in terms
- of test groups and their relationship to the protocol
- Recommendation(s)*;
- .LP
- h)
- description of the test coordination procedures (if
- applicable in the test method);
- .LP
- i)
- statement of which test cases are optional and which
- mandatory for conformance to the abstract test suite
- Recommendation*;
- .LP
- j)
- information to assist the test realizer and test laboratory
- in their use of the abstract test suite Recommendation* (see
- \(sc\ 14).
- .sp 2P
- .LP
- \fB14\fR \fBUse of an abstract test suite specification\fR
- .sp 1P
- .RT
- .PP
- The abstract test suite specifier shall provide information in the abstract
- test suite Recommendation* to assist the test realizer and
- test laboratory in their uses of the test suite. This information shall
- include, but is not limited to, the following:
- .RT
- .LP
- a)
- a mapping of the abstract test cases to the PICS proforma
- entries to determine whether or not each abstract test case is
- to be selected for the particular IUT; the mapping should be
- specified in a suitable notation for Boolean
- expressions;
- .LP
- b)
- a mapping of the abstract test cases to the PIXIT proforma
- entries, to the extent that they are known by the abstract test
- suite specifier, to parameterize the test suite for the
- particular IUT and to determine which selected test cases cannot
- be run with the particular IUT.
- .LP
- The test suite specifier shall define a partial PIXIT proforma.
- This shall contain a list of all the ALPs used in the test suite (or test
- management protocol) and a list of all parameters for which the test suite
- requires values. If any of the required parameter values will be found
- in the PICS, the PIXIT proforma entry for each such parameter shall reference
- the
- corresponding entry in the PICS proforma;
- .LP
- \fINote\fR \ \(em\ Other aspects of the PIXIT proforma are for further
- study;
- .bp
- .LP
- c)
- a list of the abstract test cases in the order that shall
- be used in the Protocol Conformance Test Report (PCTR), together
- with any information which shall be preserved in the PCTR on the
- status of each test case; this is a contribution to the
- development of a PCTR proforma;
- .LP
- d)
- any restrictions that there may be on the order in which
- the test cases may be executed;
- .LP
- e)
- reference to the description of the test coordination
- procedures (if applicable in the chosen test method);
- .LP
- f
- )
- any necessary timing information.
- .sp 2P
- .LP
- \fB15\fR \fBTest suite maintenance\fR
- .sp 1P
- .RT
- .PP
- Once an abstract test suite has been specified and is in use, it
- can be expected that errors and omissions in it will be detected by those
- who are using the test suite. The abstract test suite specifier shall in
- such
- circumstances progress the updating of the test suite via the relevant rapid
- amendment procedures.
- .PP
- In addition, from time\(hyto\(hytime, changes will be made to the protocol
- Recommendation(s)* to which the test suite relates. The abstract test suite
- specifier shall ensure that the test suite is updated as soon as possible
- after changes to the relevant protocol Recommendation* have been ratified.
- \v'6p'
- .RT
- .ce 1000
- ANNEX\ A
- .ce 0
- .ce 1000
- (to Recommendation X.290, Part 1)
- .sp 9p
- .RT
- .ce 0
- .ce 1000
- \fR \fBOptions\fR
- .sp 1P
- .RT
- .ce 0
- .PP
- A.1
- Options are those items in a Recommendation* for which the
- implementor may make a choice regarding the item to suit the implementation.
- .sp 1P
- .RT
- .PP
- A.2
- Such a choice is not truly free. There are requirements which
- specify the conditions under which the option applies and the limitations of
- the choice.
- .sp 9p
- .RT
- .PP
- Conversely, there may be mandatory or conditional requirements, or prohibitions,
- in a Recommendation* which are dependent on the choice made or on a combination
- of the choices already made.
- .PP
- A.3
- The following are examples of options and associated
- requirements; the list is not exhaustive:
- .sp 9p
- .RT
- .LP
- a)
- \*QBoolean\*U options: the option is \*Qdo or do not do\*U; the
- requirement is \*Qif do, then do as specified.\*U
- .LP
- b)
- Mutually exclusive options: the requirement is to do just
- one of \fIn\fR \ actions, the option is which of them to do.
- .LP
- c)
- Selectable options: the option is to do any \fIm\fR \| out of \fIn\fR \|
- actions, with a requirement to do at least one
- action (1\ \(=\ \fIm\fR \ \(=\ \fIn\fR and \fIn\fR \ \(>="\ 2).
- .PP
- A.4
- Options may apply to anything within the scope of a
- Recommendation* (e.g.\ static or dynamic aspects, use or provision of a
- service, actions to be taken, presence/absence or form of parameters,\
- etc.).
- .sp 9p
- .RT
- .PP
- A.5
- Another type of distinction is between service\(hyuser options and service\(hyprovider
- options.
- .sp 9p
- .RT
- .PP
- A.6
- In a wider context, the choice will be determined by conditions which lie
- outside the scope of the Recommendation* (e.g.\ other
- Recommendations* which apply to the implementation, the protocols used
- in the (N+1) and (N\ \(em\ 1) layers, the intended application, conditions of
- procurement, target price for the implementation,\ etc.). However, these have
- no bearing on conformance to the Recommendation* in which the option
- appears.
- .sp 9p
- .RT
- .LP
- .bp
-