The working group focused its discussion on the role of formal methods and certification as applied to reuse. The definition of formal methods included all tools and techniques that were mathematically based. The group used a broad definition of certification which included the use of documentation, testing methodology and tools, metrics, structured walkthrough, etc. In short, any method or tool that could be used to aid in increasing the quality or reliability of the reusable system or component was included in the scope of the discussions.
Goals
The goal of the group was to explore current accomplishments and research issues in applying formal methods and certification techniques and tools to reuse.
Formal Methods -- State of the Art and Practice
There are an increasing number of formal methods tools and techniques that are available for use by the reuse community. There have been a number of successful applications of formal methods to the software engineering of government and commercial systems documented in the literature.
The application of formal methods to reuse repositions or to internal reuse projects has been a rare occurrence, but there are a few existing efforts. We believe that increased efforts in the area of technology transfer of formal methods is necessary. The creation of high assurance reusable components may also be driven by consumer demands for high quality.
This is an especially important issue for those in the areas of building trusted software, safety critical systems and high assurance systems.
Certification -- State of the Art and Practice
As collections of reusable software assets have become increasingly widely available in recent years, concerns have grown about how to ensure that such assets conform to the quality requirements of the systems that will be using them. Asset certification has been proposed as a means of providing the software engineer with a level of confidence that an asset will conform to the quality requirements of the domain in which systems are being built and of the organization creating these systems. While a number of certification schemes have been proposed, there is still disagreement as to what it means for an asset to be ``certified'' and how such a scheme is best implemented.
The certification schemes that have been proposed to date fall into three categories - levelled, ranked, and enumerative. In a levelled scheme, assets are assigned quality ``levels'' according to the amount or type of quality assurance processing that has been applied to them. For example, a certification process might term a component that has had no quality assurance performed on it a ``level 0 component,'' one that compiles without error might be a ``level 1 component,'' one that has been unit tested might be a ``level 2 component,'' one that has been deployed and displays a mean time to failure within a certain bound might be a ``level 3 component,'' and one whose correctness has been formally verified might be a ``level 4 component''.
Levelled schemes suffer from two main drawbacks. First, it is not clear how an engineer can draw quality inferences about a system based simply on knowing that a given component has been certified according to a specific level. Second, if a component is modified, it is not clear how much effort should put into re-certifying the component, because there is no way to trace the specific properties that might have been altered in the course of the adaptation.
In a ranked scheme, scores are assigned to a component based on how well it adheres to certain quality criteria. Different criteria are assigned different weights, according to each criterion's perceived importance. A component's rank is a composite of its scores in each quality area. The goal is to enable the user, if he is presented with a set of similar assets, to choose the asset with the highest overall rank.
Ranked schemes suffer from the same problems as levelled schemes, along with an additional complication:- inconsistency of scoring and weighting. For example, a component might be assessed according to such aspects as memory usage, performance, and mean time to failure, each aspect of which would be scored separately, weighted, and then composed into an overall score. Another component might be assessed according to different attributes with different scores, or to the same attributes with different weights. Unless there is consistency in scoring, there is no way for a library user to know, based on looking at a component rank, whether one component is ``better'' than another.
Enumerative schemes are intended to address the short-comings of levelled and ranked methods. In an enumerative scheme, specific quality properties are verified by either formal or informal methods, and explicitly stated to the engineer in the form of a list. Quality properties might be trivial, such as ``Conforms to organizational formatting standards,'' or complex such as ``Free from memory leakage''. Enumerative schemes allow the programmer to understand the specific quality characteristic possessed by a component and to draw inferences about the systems that will use these components.
The main drawback of enumerative schemes is the logistics of organizing and presenting properties to the engineer. A component can possess an arbitrarily large number of properties and it is difficult for the engineer to discern whether all of the properties critical to his domain have been taken into consideration.