Black Box 4/BlackBox.cdr/textinfo/risks8d.arj/RISKS868.TXT

Text File | 1989-05-09 | 23.3 KB | 481 lines

RISKS-LIST: RISKS-FORUM Digest  Monday 8 May 1989   Volume 8 : Issue 68

        FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS
   ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Contents:
  Low-Probability / High-Consequence Accidents -- and the Midland 737? (PGN)
  "Probing Boeing's crossed Connections" (Werner Uhrig)
  An Atlantis spacecraft computer problem resolved nicely (PGN)
  "Life's Risks: Balancing Fear Against Reality of Statistics"
    (Marc Rotenberg, Jerry Leichter)
  Hear No Evil (Kevin Driscoll)
  Computer Ethics Course/Resource Volunteers Wanted (long)  (Bob Barger)

----------------------------------------------------------------------

Date: Mon, 8 May 1989 8:34:12 PDT
From: Peter Neumann <[e-mail]>
Subject: Low-Probability / High-Consequence Accidents -- and the Midland 737?

I would like to consider here a class of problems that has not been addressed
specifically in RISKS, although its components are familiar.  The RISKS Forum
has addressed alarm systems that could not adequately be debugged under truly
real circumstances.  There was also the example of the earliest Antarctic ozone
depletion data, which was systematically rejected by the analysis program for
being *too* anomalous.  The potential for a combination of these two types of
problems might occur in aircraft monitoring during flight, as follows.

   Sensitive sensors in hostile environments (such as engines) sometimes
   report unrealistic or off-scale readings due to noise or interference.
   Consequently software monitoring the sensor may be programmed to ignore
   values beyond a certain threshold, on the grounds that such extreme
   readings must be the results of extraneous events.  If the ignored sensor
   reading was "real", however, other more remote sensors might pick up -- and
   accept -- less extreme readings.  This appears to be a potential problem in
   a variety of control systems.

In the absence of any definitive information about the British Midland 737
crash, such a hypothesis seems just as plausible as any other.  The *left*
engine was reportedly vibrating wildly (possibly due to a broken fan blade),
but the pilots for some reason(s) shut off the (good) right engine.  The
extreme vibration in the left engine might indeed have produced hitherto
unexperienced sensor readings that designers -- or the software folks -- felt
would have to be impossible.  The vibration from the left engine would have
been transmitted -- much attenuated -- through the entire airframe, and might
have been reported at a much more "reasonable" intensity by the vibration
sensors of the right-hand engine.  It does not take much of a leap in
imagination for the computer program to conclude that it was the *right* engine
that was malfunctioning.

In any event, this possible fault mode represents another case of
LOW-PROBABILITY / HIGH-CONSEQUENCE ACCIDENTS [1], and thus deserves explicit
attention.  Unfortunately it is just one more such combinatory fault mode.


  [1] See Koshland's editorial (title above, in CAPS) in Science, vol 244 no
4903, 28 April 1989, p. 405, discussing the Exxon Valdez spill and conclusions
that should be drawn from it.

------------------------------

Date: Mon, 8 May 1989 4:53:45 CDT
From: Werner Uhrig <[e-mail]>
Subject: ``Probing Boeing's crossed connections''

[The title is that of an article in IEEE Spectrum, May 1989, pp. 30-35,
subtitled ``Misconnected circuits and hoses found on 94 in-service Boeing
aircraft raise concern about design, test, and maintenance of aircraft
safety systems''.  Author is Karen Fitzgerald.]

At the very end of the article is a further reference of interest to this
group:

  For a minute-by-minute account of the British Midland crash from knowledge
  gathered to date, see Special Bulletin S2/89 of the Air Accidents
  Investigation Branch of the Department of Transport in Farnborough, England,
  March 20, 1989.
                                         [I recommend the Spectrum article, and
                                         would like to see the Bulletin.  PGN]

------------------------------

Date: Mon, 8 May 1989 12:13:25 PDT
From: Peter Neumann <[e-mail]>
Subject: An Atlantis spacecraft computer problem resolved nicely

One of Atlantis' main computers (one of the processors in the two pairs of the
2x2 + 1 backup architecture) failed on 7 May.  For the first time ever the
astronauts made repairs -- in this case by substituting a spare processor.  It
took them about 3.5 hours to gain access to the computer systems by removing a
row of lockers on the shuttle mid-deck, and another 1.5 hours to check out the
replacement computer.

It is ironic that such a replacement was so difficult, but not surprising.  My
old friend Al Hopkins, who at MIT Instrumentation Lab (now Draper Lab) designed
the Apollo on-board guidance computer, told me years ago how carefully they had
planned the packaging so that the astronauts would be able to make repairs on
the fly (as it were).  NASA officials would have none of it, and buried the
computer several layers underneath other equipment.  Apparently that tradition
has continued.  Perhaps the success of the Atlantis crew will change things.

During STS-9, Nov-Dec 83, multiple primary computers on the Columbia failed at
the same time, and delayed the return to earth.  On one hand, the calculations
say that losing three processors would be a rare event.  However, here we have
another example of a low-probability / high-consequence accident -- especially
if it involved the backup and one of each of the pairs.  Furthermore, since the
software is the same in all four of 2x2 the main processors, they would all
have failed consistently, and been deemed correct.  (And we just reported the
serious problem in the Magellan software caught before Atlantis' launch, noted
in RISKS-8.67!) In the case of pairwise disagreement among both pairs, there is
always the fifth, backup, computer, separately programmed.  As far as I know,
the shuttles have never had to rely on the backup computer software, so it
might be preferable to make processor replacements among the main four rather
than resort to the backup...

------------------------------

Date: Mon, 8 May 89 12:14:37 -0700
From: [e-mail] [Marc Rotenberg]
Subject: "Life's Risks: Balancing Fear Against Reality of Statistics"

Excerpted from today's New York Times:

  Is the slight risk of contracting cancer from Alar too high a price to pay
for crisper apples?  Is the dramatic increase in milk production available
through genetically engineered growth hormones worth the unknown risk to
children's health?  If a few aging aircraft suffer explosive decompressions,
should all old airlines be grounded?

  Risks to health and safety and the complex questions of public policy they
create are seemingly everywhere these days.  And while there is little
statistical evidence that the hazards of daily life are on the rise, a wide
range of academic and business experts believe that American's perception of
increased peril is stifling technology, wasting billions of dollars, and,
ironically, making it more difficult to contain the most serious risks.

  ... by broad statistical measures, Americans have never been safer ...

  Even the high-profile threats have not changed the risks of untimely death or
injury.  The skies may be crowded, the planes aging and the pilots
inexperienced, but the trend in aircraft fatalities is downward. ...

  Life-saving medicines have been less dramatically affected, but even here,
the measures to compensate for risk can radically change the economic of
distribution ...

  The Environmental Protection Agency also regards itself as handicapped by
Congressional and public misperception of relative risk. ...

  What explains the public's decreasing tolerance of some risks and apparent
indifference to others? ... perceived risk is not always related to the
probability of injury.

  Easily tolerated risks include ones that people can choose to avoid (chain
saws, skiing), that are familiar to those exposed (smoking), or that have been
around for a long term (fireworks).  Poorly tolerated risks are involuntary
(exposure to nuclear waste), have long delayed effects (pesticides), or unknown
effects (genetic engineering).

  ... nuclear and chemical technologies fare especially badly in such
subjective rankings.  Indeed the general acceleration of technical change and
integration of new technology in products helps to explain the increase in
public anxiety about risk.  ...

------------------------------

Date: Mon, 8 May 89 17:17 EDT
From: "Jerry Leichter ([e-mail])"
Subject: Life's Risks ...

Today's New York Times (Monday 8 May) has a front-page article title "Life's
Risks:  Balancing Fear Against Reality of Statistics".  It's the first of two
articles on "risk and public policy".

The article is ... well worth reading.  Here's an interesting quotation:

  Peter W. Huber, and engineer, lawyer and author of "The Legal Revolution and
  its Consequences" notes that ... "safety taxes" [extra costs charged by
  suppliers to pay for potential lawsuits] are added to the price of thousands
  of ... goods and services, distorting production and reducing living
  standards.  By Mr. Huber's reckoning, the safety tax represents 30 percent of
  the cost of a step ladder, one-third the cost of a ride on a Long Island tour
  bus and $300 of the cost of giving birth in New York City.
                                         -- Jerry

------------------------------

Date: 7 May 89 22:44:01 GMT
From: [e-mail] (Kevin Driscoll)
Subject: Hear No Evil

On a recent flight, the cabin crew was a bit late in starting the in-flight
movie.  The flight took less time than expected, so the movie's climactic
showdown scene began just after plane touched down.  Many of the passengers
became noticeably irritated at the flight attendants pre- and post-landing
announcements which interrupted the movie's audio.  This was a tow-in gate
so the engines were shut down well before arriving at the gate.  Without
engine power, an APU supplies electrical power.  On the switch-over,
however, the power glitch reset the audio channel controllers to the default
channel (8) which is silent.  It is common on commercial aircraft to have
"unimportant" control systems (such as the individual seat lighting and
audio) reset on power glitches.  This is not a safety problem.  Is it?

When the audio went dead on this flight, most of the passengers didn't know
what happened and pushed their flight attendant call buttons.  Same of the
more irate passengers repeatedly pushed it, causing the alert tone to sound
almost continuously.  (This was what I could see in first class. I can only
imagine what was happening in the coach cabin where passengers had to
explicitly pay extra for headsets and where there were more passengers.)

I would suspect that the official justification for the flight attendant
call button system is to alert the crew to emergencies.  During this
incident, any signaling of an emergency would not have been noticed.  I also
suspect that a failure analysis of the audio system did not foresee the
implications of a power glitch reseting the channel.  This is an example of
the most common reason for safety problems; the designers don't see all the
possible circumstances that the design will face, particularly where people
are involved.

The fix to this problem is trivial; make the default channel one with some
material on it, preferably one of the movie channels (1 through 4).  I
wonder if the current design was to save some small amount of power.

Another disconcerting observation was that the cabin crew did not seem to
understand what had happened either.  They seemed unable to help the
passengers.  They made repeated visits to the passengers who contined to
re-press their call buttons.  All that had to be done was to switch the channel
back to where it had been.

Disclaimer:  I don't represent Honeywell, neither should Don Dodgen.

Kevin R. Driscoll, Principal Research Scientist  [phone]  FAX: -7438
POST:  Honeywell M/S MN65-2500; 3660 Technology Drive; Mpls, MN 55418-1006

------------------------------

Date:     Wed 03 May 1989 13:51 CDT
From: Bob Barger <[e-mail]>
Subject:  Computer Ethics Course/Resource Volunteers Wanted

Two drafts of the following course were previously printed in RISKS digests.
These brought a host of suggestions from readers. Almost all these suggestions
were incorporated into the final version below. Volunteers are now being sought
to participate in the course this Fall (see Section 3. b. 2. below). These
volunteers could contribute items relating to computer ethics for posting on the
class bulletin board, correspond by e-mail with individual students on course
topics, and/or comment on students' postings on the class bulletin board.
The course will run from late August to early December. No money is presently
available as compensation for this service, but I will gladly contribute
letters of appropriate recognition for those who participate as resource persons
in all or part of the course. If interested, send a brief "vita" to Bob Barger
at [e-mail].


                          SENIOR SEMINAR
                   EASTERN ILLINOIS UNIVERSITY

1.  Catalog Description

   a. Course Number: EIU 4050

   b. Title: Computer Ethics

   c. Credit: 2-0-2    [2 hrs per week/one semester]

   d. Term to be offered: On Demand

   e. Short title: Computer Ethics

   f. Course Description: The course will investigate current
ethical issues involving computers.  While it is not a "computer
course," students will make frequent use of postings on the
electronic bulletin board of the ECN mainframe computer to
research and discuss ethical issues.

   g. Prerequisites: 75 Semester Hours and previous experience
with computers. [Class size limit = 15 students for Fall, 1989,
semester].

   h. Exclusions: None.

2.  Outline of topics :

    Week        Topic

     1         Orientation to the course (introduction,
               explanation of course content, class procedures,
               and evaluation methodology). Consideration of
               ethical theory: examination of the metaphysical
               bases and resultant ethical norms of the idealist
               and naturalist theories.

     2         Consideration of ethical theory (continued):
               examination of the metaphysical bases and
               resultant ethical norms of the consequentialist
               and existentialist theories.

     3         On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     4         Consideration of professional ethics:
               responsibilities between employer/employee,
               client/professional, professional/peer, and
               professional/society.

     5         On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     6         Consideration of liability for software design,
               manufacture, and use: legal liability; truth-in-
               advertising; contracts; warranties; software as
               product or service?

     7         On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     8         Consideration of privacy issues: individual
               privacy rights; institutional "right-to-know"
               concerns; system security concerns; data-banking
               concerns.

     9         On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     10        Consideration of power/control issues: the
               computer as agent of centralization or
               decentralization? the computer as agent of
               conservation or change? the computer as agent of
               alienation?

     11        On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     12        Consideration of ownership and theft issues:
               copyrights; fair usage; patents; trade secrecy and
               competition; considerations unique to the computer
               market.

     13        On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     14        On-line reading of the "Discussion of Ethics in
               Computing" list, the "Forum on Risks to the Public
               in Computers and Related Systems" digest, and the
               "Computers and Society" list (all are available on
               the ECN bulletin board); written reactions to
               these readings, and written commentary on other
               students' reactions. [The instructor will insure
               that these activities equate to the activities of
               a traditional two hour class meeting].

     15        Seminar members will reconvene as a group for the
               last meeting to allow for group reflection on the
               seminar experience and course evaluation.

   Exam week   Final examination

    Writing component

   Students will type thirteen 30-to-50 line (i.e., one-to-two
   page) reactions to the on-line electronic bulletin board
   readings. Students will "post" these reactions (i.e.,
   electronically send them to the mainframe computer bulletin
   board set aside for members of this seminar). In their
   reactions, students will: 1) identify the particular
   publication or publications to which they are reacting, 2)
   identify the particular issue or issues raised in the
   publication(s), 3) identify the ethical implications of the
   issue or issues, 4) identify the ethical paradigm used by the
   author, 5) add their own reasons for agreement or disagreement
   with the viewpoint of the publication's author, 6) and,
   finally, offer an alternative solution or viewpoint to that
   presented by the author, or present other appropriate
   considerations not raised by the author or covered in their
   own (i.e., the student's own) previous comments. The
   instructor will send weekly, by confidential electronic mail,
   a grade on the student's posted reaction, together with
   whatever comments the instructor thinks helpful. The student's
   original posted reaction will also be open to public comment
   by the other students in the seminar [this is accomplished by
   posting notes to the bulletin board, referencing the original
   posted reaction]. These latter comments by the other students
   in the seminar will be considered along with classroom
   discussion in computing the "participation" factor of the
   student's semester grade.

    Evaluation

   Each student's semester grade for the seminar will be
   calculated according to the following weighted formula:

      - 13 posted reactions (at 5% each)    = 65%

      - Participation (based on class
        discussion and posted comments
        on other students' reactions)       = 20%

      - Final Exam                          = 15%

3.  Implementation :

   a. This course will be taught by: Robert N. Barger, Ph.D.

   b. Materials in the course will include:

      1) Texts:
         a) Deborah Johnson,  Computer Ethics  (Englewood
            Cliffs, NJ: Prentice Hall, 1985)
         b) Notes on Systematic Philosophies from Dr. Barger's
            Philosophy 1800 class (furnished without charge to
            seminar members)
         c) Postings on the above-mentioned ECN electronic
            bulletin board lists.

      2) Resource people: Computer professionals (e.g.,
      administrators, systems analysts, programmers, etc.) will
      be utilized as guest contributors to the class. This will
      be accomplished by personal appearances, as well as by
      electronically mediated conferencing (e.g., postings, e-
      mail, relay round-tables, etc.).

   c. Exceptional costs: None, unless the student wishes to use a
      modem to access the computer. In this case the student will
      be responsible for any personal equipment costs and/or long
      distance phone charges.

   d. Effective date: Fall, 1989.


 Date approved by Senior Seminar Committee:  February 24, 1989.

 Date approved by Council on Academic Affairs:  April 20, 1989.

------------------------------

End of RISKS-FORUM Digest 8.68
************************
-------