Black Box 4/BlackBox.cdr/progc/adacplus.arj/ADACPLUS.HLP

Text File | 1991-12-02 | 24KB | 429 lines

                       OVERVIEW OF U.S. AIR FORCE REPORT
                     Ada AND C++: A BUSINESS CASE ANALYSIS

Form G75-1191
AdaCPLUS.HLP

Ada Information Clearinghouse, 1-800-AdaIC-11, 1-703-685-1477

On July 9, 1991, the Air Force released to the public a report of a business
case they conducted to determine under what circumstances a waiver to the DoD
Ada requirement might be warranted for use of C++, particularly in DoD's
Corporate Information Management (CIM) program.  The report is titled, "Ada
and C++: A Business Case Analysis."

Since its release the report has received a great deal of publicity in various
newspapers and journals.  The information that follows was excerpted from
remarks made by Mr. Lloyd K. Mosemann, II Deputy Assistant Secretary of the
Air Force (Communications, Computers, and Logistics), at a press conference
held July 9, 1991.

I.  Introduction

There has never been any intention to question DoD's commitment to Ada, but
only to identify when waivers for C++ might be warranted.  This business case
will support the development of DoD programming language policy for
information systems and C3 systems.

I might say at the outset that language comparison is not merely a scientific
issue: it evokes strong emotions as well, in that to a certain extent people
adopt "favorite" languages for reasons other than purely dispassionate
analysis, much as one might not be able to explain why he/she roots for the
Chicago Cubs or drinks Coke rather than Pepsi.  The task is also rendered
difficult because there are yet no well-established and standard methods for
conducting such comparisons.  For these reasons, we endeavored to make our
study as quantitative as possible, asking the best experts we could find to
use a variety of methods that have historically been used for business
analysis in such tasks.  We felt that by using a variety of methods and
comparing their results, we would avoid the skewing that might result from the
sole use of a single method.

In our business case, therefore, several different approaches were undertaken
to identify, from a business perspective, when the life cycle cost
effectiveness of C++ might be greater than that of Ada.

--    The first, conducted by the Institute for Defense Analyses (IDA),
      examined quantitatively the availability of tools and training for the
      two languages.

--    The second, conducted by the Software Engineering Institute, applied to
      this problem a quantitative language selection methodology developed by
      IBM for the Federal Aviation Administration (FAA).

--    The third, conducted by CTA, Inc., analyzed cost and cost analysis).

--    And the fourth, conducted by the TRW Corporation, applied a standard
      cost model in depth to both languages for a typical information
      systems/C3 project (micro analysis).

--    In addition, the Naval Postgraduate School (NPS) was asked to address
      the overall policy issue of Ada, particularly in the context of emerging
      fourth-generation language (4GL) software technology.

Each of the substudies reached the same conclusion: there are no compelling
reasons to waive the Ada requirement to use C++.

The business case analysis was directed at information systems and C3 systems.
However, there is no reason to believe the results would differ for computer
programs embedded in weapons systems.

Let me now summarize for you the salient quantitative results of each study,
and I think you will understand more fully how we arrived at our conclusion.

II.  Substudy Results.

A.   Tools, Environments, and Training: IDA Substudy.

The Institute for Defense Analyses (IDA) collected and analyzed information on
the market availability of commercial- off-the-shelf products available from
U.S. sources for Ada and C++ compilers, tools, education, and training.

The study provided a large quantity of demographic data. For example, there
are 28 companies located in the U.S. that have Ada compilers with currently
validated status; 18 vendors offer C++ compilers.  The Ada compiler vendors
are more likely to have been in business five years or more.  Ada "validation"
is more rigorous than that of other high order languages: only Ada is
monitored and approved for conformity to a standard, without supersets or
subsets, by a government-controlled process.  By contrast, no validation or
even a standard of any kind exists for C++, although a standard by 1994 is
expected.

Both languages are supported on PCs and workstations.  Ada is also supported
on mainframes.  Ada, but not C++, has cross compilation systems.

Ada is supported with program engineering tools.  Compiler vendors provide a
rich set.  Code generators exist for Ada but none so far for C++.  There is
considerable variability among C++ products in language features supported and
libraries provided.

Ada is taught in 43 states at 223 universities and 13 DoD installations.  C++
is taught in four states at four universities and no DoD installations.  There
are more Ada than C++ courses available.  The cost of training is about equal,
but Ada course variety is wider.

B.   Faceted IBM Language Selection Methodology: SEI Substudy.

The Federal Aviation Administration (FAA) contracted with IBM in the mid-1980s
to evaluate high order languages for use on its Advanced Automation System
(AAS) Program.  In response, IBM developed a formal, quantitative faceted
methodology comparing 48 language features (criteria) in six categories.  This
IBM study concluded that use of Ada was "in the ultimate best interest of the
AAS program and its goals, and that warrants coping with the temporary
risks/problems that loom large in the near term in order to reap the
significant benefits/payoffs over the long term."


Using this same methodology for each of the 48 criteria, the Software
Engineering Institute (SEI) evaluated Ada and C++ for application in
information systems/C3 systems.  The original FAA weighted scores for the six
criteria categories were as shown in this matrix:

      Category       Max.  Ada    C   Pascal  JOVIAL  FORTRAN
                     Score

      Capability      16.7  6.1   9.6   10.4     7.6     3.9
      Efficiency      16.4  8.0  11.8   10.8    11.0    11.1
      Availability/   22.6 21.5  11.6   14.5    15.6    10.3
        Reliability
      Maintainability/17.4 14.0  10.2   12.2     6.8     8.3
        Extensibility
      Lifecycle cost  11.3  8.2   7.4    7.8     4.9     5.2
      Risk            15.6  8.8   8.9    7.6     9.6     8.2

      Total          100.0 76.6  59.5   63.3    55.5    47.0

The 1991 weighted scores for the six criteria categories were:

      Category             Max. Score  Ada   C++

      Capability               16.7   15.3   11.3
      Efficiency               16.4   10.7   10.9
      Availability/Reliability 22.6   19.1   12.6
      Maintainability/Extensibility17.413.6  11.4
      Lifecycle cost           11.3    8.4    8.0
      Risk                     15.6   11.7    9.8

      Total                   100.0   78.8   63.9

In 1985 Ada was considered considerably more capable than C. Today, the SEI
study found there is still a significant difference between Ada and C++, C's
successor.  The relative efficiency of Ada has improved markedly; Ada still
scores significantly higher in availability/reliability; the Ada advantage in
maintainability/extensibility persists; and from a position of parity in 1985,
Ada has attained in 1991 a significant advantage over C++ in lowered risk.

An attachment lists numerous major Ada information systems/C3 systems.  It is
not widely appreciated that such extensive use is now being made of Ada: in
fact, the Ada 9X Project Office reports that the U.S. Ada market, excluding
training, services, and government research/development, now exceeds $1
billion.

C.   Macro Cost Analysis: CTA Substudy.

CTA compiled and compared available productivity and cost data of Ada and C++.
Much of their data comes from Reifer Consultants' extensive database, one of
the best, largest, and most current programming language cost databases now
available.


Average productivity across the four domains for which data exists
(environment/tools, telecommunications, test (with simulators) and other) for
both Ada and C++ projects is shown in this matrix.  Note the productivity
advantage for Ada:

                        Productivity   Number of
                          (SLOC/MM)   Data Points
      *  Norm (all languages) 183         543
      *  Average (Ada)        210         153
      *  Average (C++)        187          23
      *  First project (Ada)  152          38
      *  First project (C++)  161           7

The C++ project data reflected information on 23 projects taken from seven
firms who had been using C++, Unix, and object-oriented techniques for over 2
years.  All projects were new developments.  Application size ranged from 25
to 500 KSLOCs (thousand source lines of code).  Average size was about 100
KSLOC.

The average costs across the four domains for both Ada and C++ projects are
shown in this matrix.

                            Cost       Number of
                          ($/SLOC)    Data Points
      *  Cost (all languages)$ 70         543
      *  Average (Ada)         65         153
      *  Average (C++)         55          23

Typically, the Ada developments were performed in accordance with military
standards and incorporated formal reviews, additional documentation, and
additional engineering support activities such as formal quality assurance
(QA) and configuration management (CM).  Most C++ projects are commercial and
do not extensively incorporate such activities.  Additionally, on such
projects developers are typically intimately involved with users, resulting in
considerably less requirements engineering effort.  Consequently, applications
on which C++ is used are inherently less costly, so that the reported
productivity rates are favorably skewed toward C++.

The average error rates across the four domains for both Ada  and C++ projects
were:

                       Integration       FQT        Number of
                       Error Rates   Error Rates      Data
                     (Errors/KSLOC)(Errors/KSLOC)    Points
      *  Norm (all languages)33           3            543
      *  Average (Ada)     24             1            153
      *  Average (C++)     31             3            23

The integration error rates include all errors caught in test from start of
integration testing until completion of software Formal Qualification Test
(FQT).  The FQT error rate includes only those errors found during the FQT
process.

A so-called "transition state analysis" performed by Reifer's group indicates
that 26 of the 38 firms within the Ada database had successfully made the
changeover to effective use of Ada, while none of the 7 firms in the C++
database had made the transition.  Also, none of the 7 firms were fully using
C++'s inheritance and other advanced features.

The standardization maturity of Ada was found by the CTA to be particularly
important.  While Ada has a firm and well policed standard, allowing neither
supersets nor subsets, it will be years before a stable C++ language
specification is established.  New features are being considered for the
latest standard C++ release.  Vendors are likely to offer their own enhanced
versions of C++ compilers and CASE tools, complicating portability and reuse.

Finally, the original arguments for establishing a single programming language
for military applications were found to remain.  Common training, tools,
understanding, and standards simplify acquisition, support, and maintenance.
The study concluded that after maturing for a decade, Ada's benefits have been
proven for all application classes.  Ada projects have reported 15% higher
productivity with increased quality and double the average size.  Normalizing
these data to comparable size projects would result in an expected Ada
productivity advantage of about 35%.  Ada should be the near term language of
choice.  C++, the study felt, still needs significant maturing before it is a
low risk solution for a large DoD application.

D.   Micro Cost Analysis: TRW Substudy.

TRW performed a tradeoff analysis that generalized recent corporate cost
analyses on a typical real-world information systems/C3 systems project.
Their study defined a set of maximally independent criteria, judged each
language with respect to those criteria, and then translated those judgments
into cost impacts to emphasize the importance of each criterion from a
lifecycle cost perspective.  Results were translated into perturbations of
Boehm's Ada COCOMO cost model.


Rankings of the two languages based on this analysis are shown in this matrix
(0 = no support; 5 = excellent support), followed by a total score, a weighted
sum of the rankings based on weights determined by an expert panel:

                                               Ada     C++
Reliable S/W Engineering                       4.5     3.2
Maintainable S/W Engineering                   4.4     3.2
Reusable S/W Engineering                       4.1     3.8
Realtime S/W Engineering                       4.1     2.8
Portable S/W Engineering                       3.6     2.9
Runtime Performance                            3.0     3.6
Compile-Time Performance                       2.3     3.1
Multilingual Support                           3.1     2.4
OOD/Abstraction Support                        3.9     4.6
Program Support Environment                    4.1     2.1
Readability                                    4.4     2.9
Writeability                                   3.4     3.5
Large Scale S/W Engineering                    4.9     3.3
COTS S/W Integration                           2.8     3.6
Precedent Experience                           3.6     1.5
Popularity                                     2.8     4.0
Existing Skill Base                            3.0     1.8
Acceptance                                     2.5     3.3

TOTAL SCORE FOR MGT INFO SYSTEMS              1631    1324
    (Ada score is 23% higher)

TOTAL SCORE FOR C3 SYSTEMS                    1738    1401
    (Ada score is 24% higher)

The study concluded that both Ada and C++ represent improved vehicles for
software engineering of higher quality products.  Currently, C++ was estimated
to be approximately 3 years behind Ada in its maturity and tool support.  The
case study used in this report (the Command Center Processing and Display
System-- Replacement) demonstrated development cost advantages for Ada on the
order of 35% and maintenance cost advantages for Ada on the order of 70% under
today's technologies.  In the far term (1994+), the study felt, this Ada
advantage might erode to approximately a 10% advantage in development costs
and 30% in maintenance costs for a typical development intensive system.

The study listed the primary strengths of Ada as its support for realtime
domains and large scale program development.  Its primary weaknesses are its
compile-time and runtime efficiency.  The primary strengths of C++ listed were
its support for better object oriented design, support for COTS integration,
and its compile-time and runtime efficiency.  Its main weaknesses were
identified as its support for reliability and large scale program development.
In general, the study felt Ada's weaknesses to be solved by ever-increasing
hardware performance and compiler technology advancement.  C++ weaknesses, on
the other hand, remain to be solved by advances in its support environment.


E.   Ada Policy Issues: NPS Study.

Concurrently with the preparation of this Ada and C++ Business Case Analysis,
the Naval Postgraduate School (NPS) reported on policy issues on the use of
Ada for Management Information Systems.  Their report, an analysis of the need
to see Ada in a total and evolving context, is an important vision statement
leading from Ada as the primary third-generation language (3GL) to its
conception as the basis for evolving to higher levels of productivity in
so-called 3 1/2 GL and 4GL environments.

Rather than concentrating on programming language selection, the NPS report
focuses on and argues for needed advances in software development technology.
In particular, the Report contends, while traditional factors such as
programming language selection, better training, and computer-assisted
software engineering (CASE) tools can enhance productivity modestly, a
fundamental change in the software development paradigm will be necessary to
achieve an order of magnitude gain.  Such a gain is possible through use of
4GLs, languages that will ultimately enable the developer to define the
complete design of an application entirely in the 4GL's own high-level
specification language.  The specification is then translated automatically by
the 4GL into an executable program.  When accompanied by a productive
development environment, an evolutionary implementation methodology, and well
trained development teams, the report asserts, 4GLs can provide a tenfold gain
in productivity.

An intermediate step cited by the report in the movement to 4GLs is 3 1/2 GL
programming, a term referring to the extensive use of CASE tools coupled with
a high level of code reuse.  The 3 1/2 GL approach requires a strong
commitment to codifying and accrediting code modules, to the point where it
becomes easier and more desirable to reuse code than to rewrite it.

Although experience with 4GLs has not yet been extensive (with existing
experience limited largely to specific functional domains such as financial
management and transaction processing), 4GLs are attractive for several
reasons.  One is their robustness under change: changes made to the
application, for whatever reason, are made at the specification level and then
re- translated automatically into executable code.  Another is the facility
with which they can be integrated into tightly knit and full-featured
development environments.  For these reasons, the report strongly recommends
that the DoD discourage use of traditional 3GL programming and take bold steps
to incorporate the 4GL paradigm.

Finally, the report recommends that, given the importance of Ada to DoD
software, greater effort and funding should be provided for the key Ada
initiatives: the Ada Technology Improvement Program, Ada 9X, and Ada education
initiatives.

Two issues on 3 1/2 GLs and 4GLs related to this business case were outside
the scope of the NPS report.  The first of these is that, for the foreseeable
future, state-of-the-art limitations will probably keep 4GLs from generating
more than half the total code required by most applications.  In such cases,
where a substantial amount of 3GL programming will be required to complete
application development, use of a 3 1/2 GL approach, rather than a 4GL
approach, is preferable.

Another issue outside the scope of the NPS Report was the evaluation of the
relative merits of Ada and C++ as target (output) languages for 4GL
application generators.  However, as section V.C of the NPS report points out,
a "standard, stable target language portable to a variety of hardware
platforms" with good software reuse and interface definition capabilities is
appealing.  Although more study of the characteristics desired in 4GL target
languages is warranted, the SEI and TRW substudies suggest no particular
advantage of C++ over Ada in software reuse and interface definition, so there
appears no reason to waive DoD's Ada requirement in favor of C++ as a target
language for 4GLs.

3.  Conclusions.

All four substudies which specifically compared Ada and C++ (IDA, SEI, CTA,
TRW) report a significant near term Ada advantage over C++ for all categories
of systems.  This advantage could be eroded as C++ and its supporting
environments mature over the next few years.  On the other hand, as aggressive
overseas Ada initiatives stimulate even wider domestic Ada interest, as Ada
tools/environments further mature, and when the Ada update (Ada 9X) is
complete, the balance could tip further in Ada's favor.

Adding to the case for Ada is the fact that the Ada scoring so well in the
business case was Ada's 1983 version, MIL-STD-1815A.  Just as C++ incorporates
into C certain software engineering concepts already in Ada (e.g., modularity,
strong typing, specification of interfaces), so an Ada update now underway
will bring into Ada selected features now included in C++.  This update, known
as the Ada 9X Project, is targeted for completion in 1993.  The product of
extensive community involvement (including the C3 and MIS communities), Ada 9X
will bring to Ada such improvements as decimal arithmetic, international
character sets, improved input/output, support for calls between Ada and other
languages, further representation specifications, and inheritance/polymorphism
(popular features of C++).  The Ada 9X Project Office lists one of the goals
of Ada 9X as "to provide all the flexibility of C++ with the safety,
reliability, and understandability of Ada 83."

At the same time, Ada 9X has been designed so that neither existing Ada
benefits nor performance will be lost.  For example, Ada 9X inheritance will
be controlled so as not to reduce lifecycle supportability.  Some have
criticized OOP features such as inheritance as potentially dangerous to DOD
software mission goals (such as safety, reliability, and dependability).
Bjarne Stroustrup himself, the originator of C++, has been quoted as follows:
"C makes it easy for you to shoot yourself in the foot.  C++ makes that
harder, but when you do, it blows away your whole leg."

In summary, it is not possible to make a credible case for the existence of
classes of "more cost effective" C++ systems compared to Ada.  Business cost
effectiveness data collected for this study are typified by the TRW study's
conclusion that Ada provides development cost advantages on the order of 35%
and maintenance cost advantages on the order of 70%.  In terms of full
lifecycle costs, it will be some time before data exists which could justify a
cost savings for C++.  Today, there is limited lifecycle data available for
Ada and almost none for C++.

For the foreseeable future, then, this business case shows that there are more
than enough reasons for the DoD to stick firmly with Ada, both for all high
order language (3GL and 3 1/2 GL) development and for exclusive use as a
target language of 4GL application generators in the large class of
applications for which 3GL code must supplement generated code.

"Ada and C++: A Business Case Analysis" is available in hard copy form from
Software Technology Support Center, OOALC/TISAC, Attn: Gary Peterson, Hill
AFB, Utah 84056, DSN 458-7703, 801/777-7703.  It is available in electronic
form in the ada-info directory on the AJPO machine on the Defense Data Network
and on the AdaIC electronic bulletin board [phone] or [phone].