NetNews Usenet Archive 1993 #1/NN_1993_1.iso/spool/comp/risks/145

Internet Message Format | 1993-01-11 | 30.6 KB

Path: sparky!uunet!zaphod.mps.ohio-state.edu!cis.ohio-state.edu!ucbvax!CSL.SRI.COM!risks
From: [e-mail] (RISKS Forum)
Newsgroups: comp.risks
Subject: RISKS DIGEST 14.25
Message-ID: <[e-mail]>
Date: 11 Jan 93 23:55:20 GMT
Sender: [e-mail]
Reply-To: [e-mail]
Distribution: world
Organization: The Internet
Lines: 525
Approved: [e-mail]

RISKS-LIST: RISKS-FORUM Digest  Monday 11 January 1993  Volume 14 : Issue 25

        FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS
   ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

  Contents:
Organizational Analysis in Computer Science -- PART TWO (Rob Kling)
  [PART ONE is in RISKS-14.24.]

 The RISKS Forum is moderated.  Contributions should be relevant, sound, in
 good taste, objective, coherent, concise, and nonrepetitious.  Diversity is
 welcome.  CONTRIBUTIONS to [e-mail], with relevant, substantive
 "Subject:" line.  Others may be ignored!  Contributions will not be ACKed.
 The load is too great.  **PLEASE** INCLUDE YOUR NAME & INTERNET FROM: ADDRESS,
 especially .UUCP folks.  REQUESTS please to [e-mail].

 ALL CONTRIBUTIONS CONSIDERED AS PERSONAL COMMENTS; USUAL DISCLAIMERS APPLY.
 Relevant contributions may appear in the RISKS section of regular issues
 of ACM SIGSOFT's SOFTWARE ENGINEERING NOTES, unless you state otherwise.

----------------------------------------------------------------------

   [CONTINUED FROM RISKS-14.24]

System Security and Reliability

In a simplified engineering model of computing, the reliability of products is
assured through extensive testing in a development lab. The social world of
technology use is not perceived as shaping the reliability of systems, except
through irascible human factors, such as "operator errors." An interesting and
tragic illustration of the limitations of this view can be found in some
recent studies of the causes of death and maiming by an electron accelerator
which was designed to help cure cancer, the Therac-25 (Jacky, 1991, Leveson
and Turner, 1992).

The Therac-25 was designed and marketed in the mid 1980s by a Canadian firm
AECL as an advanced medical technology. It featured complete software control
over all major functions (supported by a DEC PDP-11), among other innovations.
Previous machines included electro-mechanical interlocks to raise and lower
radiation shields. Several thousand people were effectively treated with the
Therac-25 each year. However, between 1985 and 1987 there were six known
accidents in which several people died in the US. Other were seriously maimed
or injured.

Both studies concur that there were subtle but important flaws in the design
of the Therac-25's software and hardware. AECL's engineers tried to patch the
existing hardware and (finally) software when they learned of some of the
mishaps. But they treated each fix as the final repair.

Both studies show how the continuing series of mishaps was exacerbated by
diverse organizational arrangements. Jacky claims that pressures for speedy
work by low-skilled machine operators coupled with an interface design that
did not enhance important error messages was one of many causes of the
accidents. Leveson and Turner differ in downplaying the working conditions of
the Therac-25's operators and emphasize the flawed social system for
communicating the seriousness of problems to Federal regulators and other
hospitals. Both studies observe that it is unlikely for the best of companies
to develop perfect error-free systems without high quality feedback from
users. Their recommendations differ: Jacky emphasizes the licensing of system
developers to improve minimal standards of competence. Leveson and Turner
propose extensive education and training of software engineers and more
effective communication between manufacturers and their customers.

However, both studies indicate that an understanding of the safety of computer
systems must go beyond the laboratory and extend into the organizational
settings where it is used. In the case of the Therac-25, it required
understanding a complex web of interorganizational relationships, as well as
the technical design and operation of the equipment. The need for this kind of
organizational understanding is unfortunately slighted in the CS academic
world today. CTF discusses only those aspects of computer system reliability
which are amenable to understanding through laboratory-like studies (Hartmanis
and Lin, 1992:110-111). But cases of safety critical systems, like the
Therac-25, indicate why some Computer Scientists must be willing to undertake
(and teach) organizational analysis.

   [From the title of the above section, I presume Rob had not yet gotten
   around to commenting on system security in complementary context to the
   discussion on human safety.  But it seems that similar conclusions can
   be drawn.  Apologies for the interjection.  PGN]

Worldviews and Surprises about Computerization

These few paragraphs barely sketch the highlights of a fertile and significant
body of research about computer systems in use.  Perhaps the most important
simplification for traditional computer scientists is to appreciate how people
and their organizations are situated in a social world and consequently
compute within a social world. People act in relationship to others in various
ways and concerns of belonging, status, resources, and power are often
central. The web of people's relationships extend beyond various formally
defined group and organizational boundaries (Kling and Scacchi, 1982; Kling,
1992).  People construct their worlds, including the meanings and uses of
information technologies, through their social interactions.

This view is, of course, not new to social scientists. On the other hand,
there is no specific body of social theory which can easily be specialized for
"the case of computing," and swiftly produce good theories for Organizational
Informatics as trivial deductions. The best research in Organizational
Informatics draws upon diverse theoretical and methodological approaches
within the social sciences with a strong effort to select those which best
explain diverse aspects of computerization.

       ORGANIZATIONAL INFORMATICS WITHIN COMPUTER SCIENCE

CTF places dual responsibilities on Computer Scientists. One responsibility is
to produce a significant body of applicable research. The other responsibility
is to educate a significant fraction of CS students to be more effective in
conceiving and implementing systems that will enhance organizational
performance. It may be possible to organize research and instruction so as to
decouple these responsibilities. For example, molecular biologists play only a
small role in training doctors. However, CS departments act like an integrated
Medical school and Biology department. They are the primary academic locations
for training degreed computing specialists, and they conduct a diverse array
of less applicable and more applicable research. In practice, the research
interests of CS faculty shape the range of topics taught in CS departments,
especially the 150 PhD granting departments. CS curricula mirror major areas
of CS research and the topics which CS faculty understand through their own
educations and subsequent research. As a consequence, CS courses are likely to
avoid important CS topics which appear a bit foreign to the instructor.

An interesting example of this coupling can be illustrated by CTF, in a brief
description of public-key encryption systems and digital signatures (Hartmanis
and Lin, 1992:27). In the simple example, Bob and Alice can send messages
reliably if each maintains a secret key. Nothing is said about the social
complications of actually keeping keys secret. The practical problems are
similar to those of managing passwords. In real organizations, people lose or
forget their passwords. Also, some passwords can be shared by a group of with
shifting membership, and the "secret key" can readily become semi-public. In
practice, the management of keys is a critical element of system security.
But Computer Scientists are prone to teach courses on cryptography as
exercises in applied mathematics, such as number theory and Galois theory, and
to skirt the vexing practical problems of making encryption a practical
organizational activity.

Today, most of the 40,000 people who obtain BS and MS degrees in CS each year
in the U.S. have no opportunities for systematic exposure to reliable
knowledge about the best design strategies, common uses, effective
implementation, and assessments of value of computing in a social world
(Lewis, 1989). Yet a substantial fraction of these students go on to work for
organizations attempting to produce or maintain systems that improve
organizational performance without a good conceptual basis for their work.
Consequently, many of them develop systems that underperform in organizational
terms even when they are technically refined. They also recommend ineffective
implementation procedures and are sometimes even counterproductive.

One defensible alternative to my position is that CS departments should not
take on any form of organizational analysis. They should aggressively take a
role akin to Biology departments rather than taking on any instructional or
research roles like Medical schools. To be sincere, this position requires a
high level of restraint by academic Computer Scientists. First and foremost,
they should cease from talking about the uses, value or even problems of
computerized systems that would be used in any organizational setting.
Research proposals would be mute about any conceivable application of research
results. Further, they should make effective efforts to insure that anyone who
employs their graduates should be aware that they may have no special skills
in understanding organizational computing. It would take an aggressive "truth
in advertising" campaign to help make it clear that Computer Scientists have
no effective methods for understanding computerization in the social world.
Further, Computer Scientists would forsake their commitments to subfields like
software engineering which tacitly deals with ways to support teams of systems
developers to work effectively (Curtis, et. al. 1988). Computer Scientists, in
this view, would remove themselves from addressing organizational and human
behavior, in the same way that molecular biologists are removed from
professionally commenting on the practices of cardiologists and obstetricians.
CTF argues that this view would be self-defeating.  But it would be internally
consistent and have a distinctive integrity.

In contrast, CS faculty are often reluctant to wholly embrace Organizational
Informatics. But some CS subfields, such as software engineering, depend upon
organizational analysis (Curtis, et. al., 1988). Further, CS faculty do little
to advertise the distinctive limitations in the analytical skills of our
programs' graduates. Part of the dilemma develops because many CS faculty are
ambivalent about systematic studies of human behavior. Applied mathematics and
other modes of inquiry which seem to yield concise, crisp and concrete results
are often the most cherished. As a consequence, those who conduct behaviorally
oriented research in CS departments are often inappropriately marginalized.
Their students and the discipline suffers as a result.

Between 1986 and 1989, the total number of BS and MS CS degrees awarded
annually in the US declined from about 50,000 to approximately 40,000. The
number of students majoring in CS rapidly declined at a time when
computerization was becoming widespread in many fields. A significant fraction
of the decline can be attributed to many students finding CS programs insular
and indifferent to many exciting forms of computerization. The decline of
military R&D in the U.S. can amplify these trends or stimulate a more
cosmopolitan view in CS departments. The decline in military R&D is shifting
the job market for new CS graduates towards a markedly more civilian
orientation. This shift, along with the trend towards computing distributed
into diverse work groups, is leading to more job opportunities for people with
CS education who know Organizational Informatics.

The situation of CS departments has some parallels with Statistics
departments. Statistics are widely used and taught in many academic
disciplines. But Statistics departments have often maintained a monkish
isolation from "applications." Consequently, the application of statistics
thrives while Statistics departments have few students and modest resources.
Might the status of Statistics indicate a future possibility for an insular
approach to CS?

The best Organizational Informatics research in North America is conducted by
faculty in the Information Systems departments in business schools and by
scattered social scientists (cf. Boland and Hirschheim, 1987; Galegher, Kraut
and Egido, 1990; Cotterman and Senn, 1992; Sproull and Kiesler, 1991). But
Computer Scientists cannot effectively delegate the research and teaching of
Organizational Informatics to business Schools or social science departments.

Like Computer Scientists, faculty in these other disciplines prefer to focus
on their own self-defined issues.  Computer Scientists are much more likely to
ask questions with attention to fine grained technological nuances that
influence designs. For example, the professional discussions of computer risks
have been best developed through activities sponsored by the ACM's Special
Interest Group on Software (SIGSOFT). They are outside the purview of business
school faculty and, at best, only a few social scientists are interested in
them. Generally, technology plays a minor role in social science theorizing.
And when social scientists study technologies, they see a world of
possibilities: energy technologies, transportation technologies, communication
technologies (including television), medicinal drugs and devices, and so on.
They see little reason to give computer-related information technologies a
privileged role within this cornucopia. As a consequence, the few social
scientists who take a keen interest in studying computerization are
unfortunately placed in marginal positions within their own disciplines. Often
they must link their studies to mainstream concerns as defined by the
tastemakers of their own fields, and the resulting publications appear
irrelevant to Computer Scientists.

Further, faculty in these other disciplines are not organized to effectively
teach tens of thousands of CS students, students who are steeped in technology
and usually very naive about organizations, about systems development and use
in organizations. In North America there is no well developed institutional
arrangement for educating students who can effectively take leadership roles
in conceptualizing and developing complex organizational computing projects
(Lewis, 1989).

CTF is permeated with interesting claims about the social value of recent and
emerging computer-based technologies. While many of these observations should
rest on an empirically grounded scientific footing, Computer Scientists have
deprived themselves of access to such research. For example, the discussion of
systems risks in the ACM rests on a large and varied collection of examples
and anecdotes. But there is no significant research program to help better
understand the conditions under which organizations are more likely to develop
systems using the best risk-reducing practices. There is an interesting body
of professional lore, but little scholarship to ground it (See Appendix).

Computer Scientists have virtually no scholarship to utilize in understanding
when high performance networks, like the National Research and Education
Network, will catalyze social value proportional to their costs. Consequently,
many of the "obvious" claims about the value of various computing technologies
that we Computer Scientists make are more akin to the lore of home remedies
for curing illness. Some are valid, others are unfounded speculation. More
seriously, the theoretical bases for recommending home medical remedies and
new computer technologies can not advance without having sound research
programs.

                         WHAT IS NEEDED

CTF sets the stage for developing Organizational Informatics as a
strong subfield within Computer Science. CTF bases the expansion
of the discipline on a rich array of applications in which many
of the effective technologies must be conceived in relationship
to plausible uses in order provide attractive social value for
multi-billion dollar public investments.

The CS community needs an institutionalized research capability to produce a
reliable body of knowledge about the usability and value of computerized
systems and the conditions under which computer systems improve organizational
performance. In Western Europe there are research projects about
Organizational Informatics in a few Computer Science departments and research
funding through the EEC's Espirit program (Bubenko, 1992; Iivari, 1991; Kyng
and Greenbaum, 1991). These new research and instructional programs in Western
Europe give Organizational Informatics a significantly more effective place in
CS education and research than it now has in North America.

The CS community in the U.S. has 30 years of experience in institutionalizing
research programs, especially through the Defense Advanced Research Projects
Agency and the National Science Foundation (NSF). There are many approaches,
including national centers and individual investigator research grants. All
such programs aim to develop and sustain research fields with a combination of
direct research funds, the education of future researchers, and the
development of research infrastructure. They are all multimillion dollar
efforts. Today, NSF devotes about $125K annually to Organizational Informatics
as part of the Information Technology in Organizations program. This start is
far short of the level of funding required to develop this field within CS.

The North American CS curricula must also include opportunities for students
to learn the most reliable knowledge about the social dimensions of systems
development and use (Denning, 1992).  These opportunities, formed as courses,
can provide varied levels of sophistication. The most elementary courses
introduce students to some of the key topics in Organizational Informatics and
the limitations of Systems Rationalism as an organizing frame (for example,
Dunlop and Kling, 1991a). More advanced courses focus on specific topics, such
as those I have listed above. They teach about substantive problems and
theoretical approaches for analyzing them. While many of these approaches are
anchored in the sociological theory of organizations, CS students usually
won't grasp the importance of the theories without numerous computing examples
to work with. They also have trouble grasping the character of computing in
organizations without guided opportunities for observing and analyzing
computerization in practice. Consequently, some courses should offer
opportunities for studying issues of computerization in actual organizations.

Fortunately, a few CS departments offer some courses in Organizational
Informatics. In addition, some CS faculty who research and teach about human
behavior in areas like Human-Computer Interaction and Software Engineering
can help expand the range of research an instruction. Unfortunately, only a
fraction of the CS departments in the US. have faculty who study and teach
about computing and human behavior.

While the study of Organizational Informatics builds upon both the traditional
technological foundations of CS and the social sciences, the social sciences
at most universities will not develop it as an effective foundational topic
for CS. On specific campuses, CS faculty may be able to develop good
instructional programs along with colleagues in social sciences or Schools of
Management.

But delegating this inquiry to some other discipline does not provide a
national scale solution for CS. Other disciplines will not do our important
work for us. Mathematics departments may be willing to teach graph theory for
CS students, but the analysis of algorithms would be a much weaker field if it
could only be carried out within Mathematics Departments. For similar reasons,
it is time for academic Computer Science to embrace Organizational Informatics
as a key area of research and instruction.

                           REFERENCES

Bentley, Richard, Tom Rodden, Peter Sawyer, Ian Sommerville, John
     Hughes, David Randall and Dan Shapiro.  1992.
     "Ethnographically Informed Systems Design for Air Traffic
     Control." Proc. Conference on Computer-Supported Cooperative
     Work, Jon Turner and Robert Kraut (ed.) New York, ACM Press.
Boland, Richard and Rudy Hirschhiem (Ed). 1987.  Critical Issues
     in Information Systems, New York: John-Wiley.
Bullen, Christine and John Bennett. 1991.  Groupware in Practice:
     An Interpretation of Work Experience" in Dunlop and Kling 1991b.
Bubenko, Janis. 1992. "On the Evolution of Information Systems
     Modeling: A Scandinavian Perspective." in Lyytinen and
     Puuronen, 1992.
Cotterman, William and James Senn (Eds). 1992. Challenges and
     Strategies for Research in Systems Development. New York:
     John Wiley.
Curtis, Bill, Herb Krasner and Niel Iscoe.  1988. "A Field Study
     of the Software Design Process for Large Systems,"
     Communications. of the ACM. 31(11):1268-1287.
Denning, Peter. 1991. "Computing, Applications, and Computational
     Science." Communications of the ACM. (October)
     34(10):129-131.
Denning, Peter. 1992. "Educating a New Engineer" Communications
     of the ACM. (December) 35(12):83-97
Dunlop, Charles  and Rob Kling, 1991a. "Introduction to the
     Economic and Organizational Dimensions of Computerization."
     in Dunlop and Kling, 1991b.
Dunlop, Charles and Rob Kling (Ed). 1991b. Computerization and
     Controversy: Value Conflicts and Social Choices. Boston:
     Academic Press.
Ehn, Pelle. 1991. "The Art and Science of Designing Computer
     Artifacts." in Dunlop and Kling, 1991.
Fish, Robert S., Robert E. Kraut, Robert W. Root, and Ronald E.
     Rice. "Video as a Technology for Informed Communication."
     Communications of the ACM,36(1)(January 1993):48-61.
Galegher, Jolene, Robert Kraut, and Carmen Egido (Ed.) 1990.
     Intellectual Teamwork: Social and Intellectual Foundations
     of Cooperative Work.  Hillsdale, NJ: Lawrence Erlbaum.
Greif, Irene. ed. 1988. Computer Supported Cooperative Work: A
     Book of Readings. San Mateo, Ca: Morgan Kaufman.
Grudin, Jonathan. 1989. "Why Groupware Applications Fail:
     Problems in Design and Evaluation." Office: Technology and
     People. 4(3):245-264.
Hartmanis, Juris and Herbert Lin (Eds). 1992. Computing the
     Future: A Broader Agenda for Computer Science and
     Engineering.  Washington, DC. National Academy Press.
     [Briefly summarized in Communications of the ACM,35(11)
     November 1992]
       [[TO BE DISCUSSED AT ACM CSC in Indianapolis, 16-18 Feb 1992... PGN]]
Hewitt, Carl. 1986. "Offices are Open Systems" ACM Transactions
     on Office Information Systems. 4(3)(July):271-287.
Iivari, J. 1991."A Paradigmatic Analysis of Contemporary Schools
     of IS Development." European J. Information Systems
     1(4)(Dec): 249-272.
Jacky, Jonathan. 1991. "Safety-Critical Computing: Hazards,
     Practices, Standards, and Regulation" in Dunlop and Kling
     1991b.
Jarvinen, Pertti. 1992. "On Research into the Individual and
     Computing Systems," in Lyytinen and Puuronen, 1992.
King, John L. and Kenneth L. Kraemer. 1981. "Cost as a Social
     Impact of Telecommunications and Other Information
     Technologies." In Mitchell Moss (Ed.) Telecommunications and
     Productivity, New York: Addison-Wesley.
Kling, Rob. 1992. "Behind the Terminal: The Critical Role of
     Computing Infrastructure In Effective Information Systems'
     Development and Use." Chapter 10 in Challenges and
     Strategies for Research in Systems Development. edited by
     William Cotterman and James Senn. Pp. 153-201. New York:
     John Wiley.
Kling, Rob  and Charles Dunlop. 1993. "Controversies About
     Computerization and the Character of White Collar Worklife."
     The Information Society. 9(1) (Jan-Feb)
Kling, Rob and Lisa Covi. 1993. Review of Connections by Lee
     Sproull and Sara Kiesler. The Information Society, 9(1)
     (Jan-Feb, 1993).
Kling, Rob, Isaac Scherson, and Jonathan Allen. 1992. "Massively
     Parallel Computing and Information Capitalism" in A New Era
     of Computing. W. Daniel Hillis and James Bailey  (Ed.)
     Cambridge, Ma: The MIT Press.
Kling, Rob and Walt Scacchi. 1982. "The Web of Computing: Com-
     puting Technology as Social Organization", Advances in
     Computers. Vol. 21, Academic Press: New York.
Kraemer, Kenneth .L., Dickhoven, Siegfried, Fallows-Tierney,
     Susan, and King, John L. 1985.  Datawars: The Politics of
     Modeling in Federal Policymaking.  New York:  Columbia
     University Press.
Kyng, Morton and Joan Greenbaum. 1991. Design at Work:
     Cooperative Work of Computer Systems. Hillsdale, NJ.:
     Lawrence Erlbaum.
Ladner, Sharyn and Hope Tillman. 1992. "How Special Librarians
     Really Use the Internet: Summary of Findings and
     Implications for  the Library of the Future" Canadian
     Library Journal, 49(3), 211-216.
Leveson, Nancy G. and Clark S. Turner. 1992. "An Investigation of
     the Therac-25 Accidents".  Technical Report #92-108.
     Department of Information and Computer Science, University
     of California, Irvine.
Lewis, Philip M. 1989. "Information Systems as an Engineering
     Discipline."  Communications of the ACM
     32(9)(Sept):1045-1047.
Lucas, Henry C. 1981. Implementation : the Key to Successful
     Information Systems. New York: Columbia University Press.
Lyytinen, Kalle and Seppo Puuronen (Ed.) 1992. Computing in the
     Past, Present and Future: Issues and approaches in honor of
     the 25th anniversary of the Department of Computer Science
     and Information Systems. Jyvaskyla Finland, Dept. of CS and
     IS, University of Jyvaskyla.
Orlikowski, Wanda. 1992. "Learning from Notes: Organizational
     Issues in Groupware Implementation." Proc. Conference on
     Computer-Supported Cooperative Work, Jon Turner and Robert
     Kraut (Ed.) New York, ACM Press.
Sarmanto, Auvo.  1992. "Can Research and Education in the Field
     of Information Sciences Foresee the Future of Development?"
     in Lyytinen and Puuronen, 1992.
Sproull, Lee and Sara Kiesler. 1991. Connections: New Ways of
     Working in the Networked Organization. Cambridge, Mass.: MIT Press.
Suchman, Lucy. 1983. "Office Procedures as Practical Action: Models
     of Work and System Design."  ACM Transactions on Office
     Information Systems. 1(4)(October):320-328.
Winograd, Terry and Fernando Flores. 1986. Understanding
     Computers and Cognition. Norwood, NJ: Ablex Publishing.



                        ACKNOWLEDGEMENTS

This paper builds on ideas which I've developed over the last decade. But they
have been deepened by some recent events, such as the CTF report. They were
also sharpened through a lecture and followon discussion with colleagues at
the University of Toronto, including Ron Baeker, Andy Clement, Kelley
Gottlieb, and Marilyn Mantei.  Rick Weingarten suggested that I write a brief
position paper reflecting those ideas. At key points, Peter Denning and Peter
Neumann provided helpful encouragement and sage advice. I also appreciate the
efforts of numerous other friends and colleagues to help strengthen this paper
through their comments and critical assistance. The paper is immeasurably
stronger because of the prompt questions and suggestions that I received in
response to an evolving manuscript from the following people: Mark Ackerman,
Jonathan P. Allen, Bob Anderson, Lisa Covi, Brad Cox, Gordon Davis, Phillip
Fites, Simson Garfinkel, Les Gasser, Sy Goodman, Beki Grinter, Jonathan
Grudin, Pertti Jarvinen, John King, Heinz Klein, Trond Knudsen, Kenneth
Kraemer, Sharyn Ladner, Nancy Leveson, Lars Matthiesen, Colin Potts, Paul
Resnick, Larry Rosenberg, Tim Standish, John Tillquist, Carson Woo and Bill
Wulf.

                       APPENDIX

            Published Materials about Computer Risks

Unfortunately, there is no single good book or comprehensive review article
about the diverse risks of computerized systems to people and organizations,
and ways to mitigate them. The Internet board, comp.risks, is the richest
archive of diverse episodes and diverse discussions of their causes and cures.
While its moderator, Peter Neumann does a superb job of organizing discussions
of specific topics each year and also creates periodic indices, there is no
simple way to sift through the megabytes of accumulated comp.risks files.

Computerization and Controversy edited by Charles Dunlop and Rob Kling (1991)
includes two major sections on "security and reliability" and "privacy and
social control" which identify many key debates and reprint some key articles
and book excerpts which reflect different positions.  Another major source is
a series of articles, "Inside Risks, which Peter Neumann edits for
Communications of the ACM.

This is a list of this series of articles, to date:
(All articles are by Peter Neumann unless otherwise indicated.)

Jul 90.  1. Some Reflections on a Telephone Switching Problem
Aug 90.  2. Insecurity About Security?
Sep 90.  3. A Few Old Coincidences
Oct 90.  4. Ghosts, Mysteries, and Risks of Uncertainty
Nov 90.  5. Risks in computerized elections
Dec 90.  6. Computerized medical devices, Jon Jacky
Jan 91.  7. The Clock Grows at Midnight
Feb 91.  8. Certifying Programmers and Programs
Mar 91.  9. Putting on Your Best Interface
Apr 91. 10. Interpreting (Mis)information
May 91. 11. Expecting the Unexpected Mayday!
Jun 91. 12. The Risks With Risk Analysis, Robert N. Charette
Jul 91. 13. Computers, Ethics, and Values
Aug 91. 14. Mixed Signals About Social Responsibility, Ronni Rosenberg
Sep 91. 15. The Not-So-Accidental Holist
Oct 91. 16. A National Debate on Encryption Exportability, Clark Weissman
Nov 91. 17. The Human Element
Dec 91. 18. Collaborative Efforts
Jan 92. 19. What's in a Name?
Feb 92. 20. Political Activity and International Computer Networks, Sy Goodman
Mar 92. 21. Inside ``Risks of `Risks' ''
Apr 92. 22. Privacy Protection, Marc Rotenberg
May 92. 23. System Survivability
Jun 92. 24. Leaps and Bounds (Leap-year and distributed system problems)
Jul 92. 25. Aggravation by Computer: Life, Death, and Taxes,
Aug 92. 26. Fraud by Computer
Sep 92. 27. Accidental Financial Losses
Oct 92. 28. Where to Place Trust
Nov 92. 29. Voting-Machine Risks, Rebecca Mercuri
Dec 92. 30. Avoiding Weak Links
Jan 93. 31. Risks Considered Global(ly)
Feb 93. 32. Is Dependability Attainable?
Mar 93. 33. Risks of Technology

------------------------------

End of RISKS-FORUM Digest 14.25
************************