NetNews Usenet Archive 1992 #30/NN_1992_30.iso/spool/comp/software/4998

Text File | 1992-12-16 | 4.8 KB | 109 lines

Newsgroups: comp.software-eng
Path: sparky!uunet!munnari.oz.au!ariel.ucs.unimelb.EDU.AU!kowari.cpsg.com.au!johnr
From: [e-mail] (John Reid)
Subject: Re: Value of High Code Coverage Metrics in Testing - Request for Opinion
Message-ID: <[e-mail]>
Sender: [e-mail] (USENET News System)
Nntp-Posting-Host: pongo.cpg.oz.au
Organization: Computer Power Software Group Pty Ltd
References: <[e-mail]> <[e-mail]>
Date: Wed, 16 Dec 1992 06:18:34 GMT
Lines: 96

[e-mail] (UK702 DavidAlexander) writes:

>In article <[e-mail]>, [e-mail] (Paul Bandler) writes:

><preamble deleted>
>
>|> I believe we have a tool to measure the %BFA 'Branch Flow Analysis' but
>|> of course the engineers are responsible for producing the test cases to
>|> exercise the code.
>|>
>|> I have 3 questions:-
>|>
>|> 1)  Do people think that this is a valuable metric?
>|> 2)  Is it a cost effective excersize to get engineers to achieve a particular
>|>     %BFA as a completion criteria?
>|> 3)  What is a realistic %BFA to aim for?
>|> Paul Bandler

>BFA is a valuable tool for testing. %BFA, as with all unitless numbers, is
>of a more dubious nature.

>%BFA does not tell the person looking at it which paths were not tested
>and why. This is important because engineers could, subconsciously or
>otherwise, use their quota of untested branches to avoid testing the more
>complex/niggly areas of code.

>The nature of the testing is also important - only if you are programming
>in an interpreted language where syntax must be checked by execution,
>is executing every line of code useful by itself. (You can
>run through every branch of a square root function but it won't
>test for what happens with a negative parameter. There is always a
>temptation to say that a line i++ 'passes because it increments i' without
>checking when it shouldn't.)

>As to cost effectiveness this is so dependant upon the available tools,
>maintainance costs, whether or not testability was built into the code etc.
>that any global statement would be rash.

>A realistic %BFA target is again dependant upon specific circumstances.
>I don't know if anyone has done any work to find out the %age of
>defects found, against testing coverage and time taken but I would
>be interested to find out. The benefits of higher percentages are not
>linear and certainly peaks.

>Setting %BFA as a part of completion criteria is probably worthwhile but
>getting dogmatic about it is probably not. BFA is very useful for helping
>engineers see where more testing is required it is not that useful as an
>exam mark for the testing.

>-- David

I agree very much with the above. All coverage measures are useful, but
they are only useful if the tests you are performing are independently
verified as "working correctly". In other words, executing the code is all
very well, but you need to know that the results you obtained were the
expected results. Adding 2 and 3 to get 7 may well take you down a path,
but if you leave it at that your program is not working so well...

Here we also spend time drawing up plans for testing, and try to design
tests to be easy to verify. I try to persuade those wwriting the code to
always do the following:

A.    Identify data values that should be accptable, and those that should
    not be accepted.

B.    Identify for selected "good" values what the expected outcome should
    be, and verify that that is what happens.

C.    Identify for selected "Bad" values what the expected outcome should
    be, and verify that that is what happens.

D.    As a guide values at the boundaries of acceptable ranges should always
    be tested (BOTH SIDES OF THE BOUNDARY).

    A value well away from any set boundary should also be tested.

I often have to persuade programmers to try data values that should never
be acceptable, but for commercial applications, your code should be robust
enough to deal with any values. If your input program, say, is accepting
data, and only numeric data can be used by the appliication, then you need
to know that if you are silly enough to enter "FRED" the program will
identify your deliberate mistake and tell you about it, rather than
blithely attempting to calculate the size of a concrete beam with a safe
working load of FRED tonnes (How heavy is Fred, anyway ??:-), assuming that
that was what your program was to do.

In summary, then coverage measures are useful (we use them a lot here,
mostly the C1 coverage metric), but they are not sufficient, at least I
don't believe they are.


--
John Reid                                   ACSnet : [e-mail]
CP Software Export Pty Ltd,                 Phone  : [phone]
19 Cato Street, Hawthorn 3123 Australia     Fax    : [phone]
ACN 006 640 133