Metrics
The term ``metrics,'' in the context of certification, refers to the means by which certification criteria are established and assessed. The main research topics in this area are three-fold. First, there is no agreed-upon set of standards for measuring a component's quality. Second, there is little experience in applying certification standards to existing libraries. Third, it is not clear what set of mechanisms can be used to derive these metrics.
With respect to lack of agreed upon quality measurements, an example is the situation described earlier with ranked certification schemes. If the providers of a set of components do not agree on which quality criteria should be certified, or on the importance of a given certification property, the result is confusion to the component's user. Thus, emphasis is needed on determining which quality attributes of a component are important for specific domains, and determining appropriate measurements for these attributes.
With respect to lack of experience, more work is needed in documenting which schemes are most effective in a given situation. Specifically, work is needed to determine which schemes best enable engineers to make decisions about the components to use and how the component will impact the overall quality of systems built using the components. A framework for assessing the different certification schemes would prove quite useful at this stage of the technology.
With respect to mechanism, there are three different ways of determining metrics: formal, informal, and empirical. Formal methods involve using program proof techniques to assure that specific properties hold. Informal methods include inspection. Empirical methods include testing and statistical modeling. Guidelines are needed on which methods are best used to establish specific classes of properties. Furthermore, practitioners would welcome assessments of available tools to help establish certification properties.
Local Certifiability, Modularity, and Scalability
An important issue in reuse technology is the benefit that accrues when a component's certification is local. The working group concentrated on certification of correctness, but the idea applies to any property. Local certification of a component property means establishing that the property holds out of the context of any particular client of the component. There may be constraints on the client contexts in which a certification applies, but if these are stated explicitly as part of the component specification then a certification can (in principle) still be local. Unfortunately, just stating these constraints is not sufficient for local certifiability. Some of the position papers from WISR '93 and WISR '92 illustrate the technical problems with poorly-designed components and programming languages.
There are two major reasons for wanting to certify component properties locally. Both are directly related to reuse. The first is that local certifiability is tantamount to the ability to reason modularly, on a component-wise basis, about software systems. The intrinsic importance of local certifiability is based on the intractability of reasoning about the monolithic programs that would result from recursive source-code expansion in large software systems. It is simply a practical reality that if one cannot reason modularly about a large system, one cannot really hope to reason about it at all. Any scalable reuse technology, then, must be based on foundations that support local certifiability of component properties.
The second reason has to do with cost. Although there is evidence that formal methods can reduce life-cycle costs, there is still a potentially significant short-term expense that tends to limit their use in practical certification efforts. If a certification can be done locally, then even if the process is costly---as might be the case for, say, a formal proof of correctness---this cost can be amortized over the many future uses of the component. In this case the marginal cost of even an expensive certification becomes negligible for a heavily used component. But if a certification has to be redone for every use of a component, then incremental cost becomes a significant issue and one of the main potential benefits of reuse (i.e., amortized costs) is reduced or lost.
Reuse after modification vs. black box reuse
The group was divided on the issue of whether or not a reusable component or system could be modified once it certification was complete. The two positions were discussed as follows.
Reuse after modification
Research on local certifiability and modularity is an ongoing topic to enable such modifications without requiring the reuser to recertify or reverify the whole system or component. This is obviously an important issue in terms of cost. If the recertification could be done locally, then user modifications would not destroy the integrity of the system or component.
Black Box Reuse
Software components that are stored in a repository for software reuse must be used exactly as is, and not modified before they are integrated into their host system. There are two arguments for this stance, a conceptual argument and a pragmatic argument.
More work needs to be done on connections between certification, verification, reuse, programming languages
We view the techniques from testing, reviews and inspections, software metrics, certification, and formal verification as contributing to verifying that a given software artifact has desirable properties such as correctness, functionality, performance, structure, and conformance to standards (of documentation, of layout, of use of language, etc). It is not clear which properties correlate best with reusability of the artifacts; more experimental evidence is required. Furthermore, it is not clear that formal methods are the most appropriate way to verify these properties, despite an intuition that ``correct'' software is more reusable than ``incorrect'' software.
The application of some techniques leads to undecidable, intractable, or impractical problems for automatic tools. Aliasing and pointers are common examples. These problems are sometimes inherent in the logic and programming language constructs used, rather than reflections of our ignorance of better techniques. Can, and should, we remove these difficulties by designing languages and methodologies which avoid the use of problematic constructs and properties?
Our discussion needs to go beyond code reuse and address the connections between formal methods and reusable design, and reusable specification.