The California Collection/TheCaliforniaCollection.cdr/his094/isue11e.arj/ISUE11E.TXT

Text File | 1988-05-10 | 19.7 KB | 323 lines

                             NOT WITHOUT US

A Challenge to Computer Professionals to Help Bring the Present
Insanity to a Halt

By Joseph Weizenbaum, professor of computer science at MIT

(This is an English translation of a talk given in German to the
Association of Computer Professionals in West Germany in July 1986.
You are welcome to reproduce and distribute it.)


.    Whenever I come to Europe, especially to West Germany, I am
amazed by the normality of everyday life: superhighways, "music" that
assaults one in restaurants, the many parks, the forests of television
antennas on the roofs of houses and so on.  I am amazed because of
Europe's geographic position and all that follows from it.  In West
Germany, for example, there is the border with the other Germany,
dense with military installations of all sorts.  There are holes in
the street that are intended to be filled with nuclear land mines if
Russian tanks should come.  These are signs of Europe's physical and
psychological proximity to the final catastrophe.
.    We in America are, in a certain sense, no more distant from the
catastrophe than the Europeans are.  Not only Chernobyl, but also the
threat of war is everywhere.  And war is everyone's enemy.  In case of
war, regardless of whether unintentionally initiated by technology
allegedly designed to avert war, or by so-called statesmen or women
who thought it their duty to push the button, you may die ten minutes
earlier than we in fortress America, but we shall all die.
.    But we have no holes in our streets for atomic land mines that
are intended to delay Soviet tank regiments.  We see our missile silos
only now and then -- that is, only whenever it pleases someone to show
them to us on television.  No matter how passionately our government
tries to convince us that the nasty Soviets are effectively as near to
us as to Europeans, that they threaten us from Cuba and Nicaragua,
Americans are, on the whole, quite unconvinced and untroubled by such
efforts.  The American experience of war has allowed us to develop an
"it can't happen here" attitude, rather than a concrete fear of what
appears to be far removed from the immediate concerns of daily life.
.    We know that it is emotionally impossible for anyone to live for
very long in the face of immediate threats to existence without
bringing to bear psychological mechanisms that will exclude these
dangers from consciousness, permitting them to surface only rarely.
But when repression necessitates systematically misdirected efforts,
or excludes potentially life-saving behavior, then it is time to
replace it with a conscious effort to find the prod to correct action.
.    That time has come for computer professionals.  We now have the
power radically to turn the state of the world in directions conducive
to life.
.    In order to gain the necessary courage (not all of us are saints
or heroes) we have to understand that for us as individuals, as well
as for those we love, our present behavior is far more dangerous, even
life threatening, than what healthy common sense now demands of us.
None of the weapons that today threaten every human being with murder,
and whose design, manufacture and sale condemns countless people to
starvation, could be developed without the earnest cooperation of
computer professionals.  Without us, the arms race, especially the
qualitative arms race, cannot march another step.
.    What does this say to us?
.    First, that we computer experts -- as well as specialists in many
other technical domains -- share in the guilt of having brought about
the present dangerous state of the world.  Those among us who, perhaps
without being aware of it, devote our talents and strengths to death
rather than to life have little right to curse politicians, statesmen
and women for not bringing us peace.  It isn't enough to make pretty
posters that can be carried in demonstrations.  Those who carry them
must care whether their daily work helps to make possible the very
devices the use of which they are protesting.
.     At this point, the domain called Artificial Intelligence (AI)
comes especially to mind.  Many of the technical tasks and problems in
this subdiscipline of computer science stimulate the imagination and
creativity of technically oriented workers particularly strongly.
Goals like making a thinking being out of the computer, giving the
computer the ability to understand spoken language, making it possible
for the computer to see, offer nearly irresistible temptations to
those among us who have not fully sublimated our playful sandbox
fantasies, or who mean to satisfy our delusions of omnipotence on the
computer stage.  Such tasks are extraordinarily demanding and
interesting.  Robert Oppenheimer called them sweet.  Besides, research
projects in these areas are generously funded.  The required moneys
usually come out of the coffers of the military, at least in America.
.    It is enormously tempting and, in Artificial Intelligence work,
seductively simple to lose or hide oneself in details, in subproblems
and their subproblems and so on.  The actual problems on which one
works -- and which are so generously supported -- are disguised and
transformed until their representations are mere fables: harmless,
innocent, lovely fairy tales.
.    Here is an example.  A doctoral student characterized his
projected dissertation task as follows.  A child, six or seven years
old, sits in front of a computer display that shows a kitten and a
bear, in full color.  The kitten is playing with a ball.  The child
speaks to the computer system: "The bear should say 'thank you' when
someone gives him something."  The system responds in a synthetic, but
nevertheless pleasing voice: "Thank you, I understand."  Then the
child again: "Kitty, give your ball to your friend."  Immediately we
see the kitten on the computer display throw the ball to the bear.
Then we hear the bear say: "Thank you, my dear kitten."
.    This is the kernel of what the system, development of which is to
constitute the student's doctoral work, is to accomplish.  Seen from a
technical point of view, the system is to understand spoken
instructions -- that alone is not simple -- and translate them into a
computer program which it is then to integrate seamlessly into its own
computational structure.  Not at all trivial, and beyond that, quite
touching.
.    Now a translation to reality.  A fighter pilot is addressed by
his pilot's assistant system: "Sir, I see an enemy tank column below.
Your orders, please."  The pilot: "When you see something like that,
don't bother me, destroy the bastards and record the action.  That's
all."  The system answers: "Yes, sir!" and the plane's rockets fly
earthward.
.    This pilot's assistant system is one of three weapons systems
that are expressly described, mainly as a problem for artificial
intelligence, in the Strategic Computing Initiative, a new major
research and development program of the American military.  Over
$600,000,000 are to be spent on this program in the next four or five
years.
.    It isn't my intention to assail or revile military systems at
this point.  I intend this example from the actual practice of
academic artificial intelligence research in America to illustrate the
euphemistic linguistic dissimulation whose effect it is to hinder
thought and, ultimately, to still conscience.
.    I don't know whether it is especially computer science or its
subdiscipline Artificial Intelligence that has such an enormous
affection for euphemism.  We speak so readily of computer systems that
understand, that see, decide, make judgments, and so on, without
ourselves recognizing our own superficiality and immeasurable naivete
with respect to these concepts.  We anesthetize our ability to
evaluate the quality of our work and, what is more important, to
identify and become conscious of its end use.
.    The student mentioned above imagines his work to be about
computer games for children, involving perhaps toy kittens, bears and
balls.  Its actual and intended end use will probably mean that some
day a young man, quite like the student himself -- someone with
parents and possibly a girl friend -- will be set afire by an
exploding missile sent his way by a system shaped by his own research.
The psychological distance between the student's conception of his
work and its actual implications is astronomic.  It is precisely that
enormous distance that makes it possible not to know and not to ask if
one is doing sensible work or contributing to the greater efficiency
of murderous devices.
.    One can't escape this state without asking, again and again:
"What do I actually do?  What is the final application and use of my
work?  Am I content or ashamed to have contributed to this use?"
.    I am reminded in this context of a well known American journalist
who, during a Middle East highjacking, suggested that under certain
circumstances the Israelis shoot ten Arab prisoners and, should the
circumstances not change, shoot ten more the next day, and so on.  He
should not have made this suggestion unless he was prepared to go
personally among the prisoners and look into the eyes of the men, some
of whom would hear him say, "You, you will die today."  He should have
been prepared as well to hold the pistol to the heads of those he
selected and command his own finger to pull the trigger.
.    Just so should we ask ourselves about our own work.  Once we have
abandoned the prettifying of our language, we can begin to speak among
ourselves realistically and in earnest about our work as computer
professionals.
.    "You, colleague of many years, you are working on a machine
consisting of two to the fifteenth and more microprocessors running
simultaneously.  With the help of such a machine one can first
simulate then construct much more efficient, smaller and lighter
hydrogen bombs.  Imagine, for a moment, you were an eyewitness at
Hiroshima in 1945; you saw people stripped of their skin die.  Would
you want to make this happen thousands of times more?  Would you so
torture a single human being with your own hands?  If you would not,
regardless of what end would be served, then you must stop your work."
.    One should ask similar questions with respect to other branches
of computer science, for example, with respect to attempts to make it
possible for computer systems to see.  Progress in this domain will be
used to steer missiles like the Cruise and Pershing ever more
precisely to their targets, where murder will be committed.
.    Many will argue that the computer is merely a tool.  As such it
can be used for good or evil.  In and of itself, it is value free.
Scientists and technicians cannot know how the products of their work
will be applied, whether they will find a good or an evil use.  Hence
scientists and technicians cannot be held responsible for the final
application of their work.
.    That point of view is manifested in the world famous Draper
Laboratory, next door to the MIT building where I work.  Draper is
devoted almost entirely to missile guidance and submarine navigation.
Many of the scientists employed there argue that the systems they work
on can take men to the moon and bring them back, as well as guarantee
that missiles aimed at Moscow will actually hit Moscow, their target.
They cannot know in advance, they say, which of these two or still
other goals their work will serve in the end.  How then can they be
held responsible for all the possible consequences of their work?
.    So it is, on the whole, with computer professionals.  The
doctoral student I mentioned, who wishes to be able to converse with
his computer display, does in fact believe that future applications of
his work will be exclusively in innocent applications like children's
games.  Perhaps his research is not sponsored by the Pentagon's
Strategic Computing Initiative; perhaps he never even heard of SCI.
How then can he be held responsible if his work is put to anti-human
use?
.    Here is where we come to the essence of the matter.  Today we
know with virtual certainty that every scientific and technical result
will, if at all possible, be put to use in military systems.
.    The computer, together with the history of its development, is
perhaps the key example.  But we should also think in this connection
of everything that has to do with flight, or of things atomic, of
communications systems, satellites, space ships, and most of the
scientific achievements of the human genius.  We may then convince
ourselves that in the concrete world in which we live, the burden of
proof rests with those who assert that a specific new development is
immune from the greed of the military.
.    In these circumstances, scientific and technical workers cannot
escape their responsibility to inquire about the end use of their
work.  They must then decide, once they know to what end it will be
used, whether or not they would serve these ends with their own hands.
.    I don't believe the military, in and of itself, to be an evil.
Nor would I assert that the fact that a specific technology that has
been adopted by the military is, on that ground alone, an evil.  In
the present state of the evolution of the sovereign nation-state -- in
other words, in the insane asylum in which we live -- each state needs
a military just as every city needs a fire department.  But no one
pleads for a fire station on every corner, and no one wishes for a
city fire department that makes a side business of committing arson in
the villages adjacent to the city.
.    But we see our entire world, particularly its universities and
science and engineering facilities, being more profoundly militarized
every day.  "Little" wars burn in almost every part of the earth.
(They serve, in part, to test the high tech weapons of the "more
advanced nations.")  More than half of all the earth's scientists and
engineers work more or less directly in military institutions, or in
institutions supported by the military.  That is an evil that must be
resisted.
.    We must also recognize that it is only our already internalized
habit of prettifying our language, in order not to arouse our
conscience, that permits us to speak in terms of weapons and weapons
delivery systems at all, when we are, in fact, discussing atomic
explosives and hydrogen bombs.  Those aren't weapons, they are mass
murder machines and mass murder machine delivery systems.  That is how
we should speak of them: clearly, distinctly, and without evasion.
Once we recognize that a nuclear mass murder machine is nothing other
than an instant Auschwitz -- without railroads or Eichmanns or Dr.
Mengele, but an Auschwitz just the same -- can we continue then to
work on systems that steer these devices to living cities?
.    That is the question I ask.  Each of us must earnestly ask
ourselves such questions and deeply consider the responses we find in
ourselves.  Our answers must finally manifest themselves in our
actions -- concretely, in what we do every day.
.    Probably the most pandemic mental illness of our time is the
almost universally held belief that the individual is powerless.  This
self-fulfilling delusion will surely be offered as a counter argument
to my theses.  I demand, do I not, that a whole profession refuse to
participate in the murderous insanity of our time.  "That cannot be
effective," I can already hear it said," That is plainly impossible.
After all, if I don't do it, someone else will."  First, and on the
most elementary level, "If I don't do it, someone else will" cannot
serve as a basis of moral behavior.  Every crime imaginable can be
justified with those words.  For example: If I don't steal the
sleeping drunk's money, someone else will.  But it is not at all
trivial to ask after the meaning of effectiveness in the present
context.  Surely, effectiveness is not a binary matter, an either/or
matter.  To be sure, if what I say here were to induce a strike on the
part of all scientists with respect to weapons work, that would have
to be counted as effective.  But there are many much more modest
measures of effectiveness.
.    I think it was George Orwell who once wrote, "The highest duty of
intellectuals in these times is to speak the simplest truths in the
simplest possible words."  For me that means, first of all, to
articulate the absurdity of our work in my actions, my writings and
with my voice.  I hope thereby to stir my students, my colleagues,
everyone to whom I can speak directly.  I hope to encourage those who
have already begun to think similarly, and to be encouraged by them,
and possibly rouse others out of their slumber.  Courage, like fear is
catching.
.    Even the most modest success in such attempts has to be counted
as effective.  Beyond that, in speaking as I do, I put what I discuss
here on the public agenda and contribute to its legitimation.  These
are modest goals that can surely be reached.
.    But, finally, I want to address such larger goals as, for
example:

.    Ridding the world of nuclear mass murder devices and perhaps also
of nuclear power generators.

.    So reordering the world that it becomes impossible ever again to
convince workers of one country that it is a necessity of life that
they feed their families on the flesh and the blood and the tears of
people of other countries.  (That is, unfortunately, the fate of many
workers today, and not only those who earn their daily bread in
armaments factories, but equally those of us whose daily work is to
sharpen high tech weapons.)

.    So reordering the world that every human being has available to
him or herself all material goods necessary for living in dignity.  (I
have often heard well-meaning people say that, if we apply technology,
especially computer and communications technology wisely, we may reach
this goal in perhaps 50 to 100 years.  But we can reach it sooner, and
without waiting for technological advances.  For the obstacle is not
the absence of technology, it is the absence of political will.)

.    I once heard Elie Wiesel say: "We must believe the impossible is
possible."  I understood that in two different ways.  First, had we
been able to believe that "the land of the poets and the thinkers"
could give birth to human extermination factories, we might not have
had to experience Bergen Belsen.  The impossible horror proved
possible and became reality.
.    But there is a more hopeful interpretation.  It seemed impossible
in the America of only 150 years ago ever to abolish the slavery of
the black people.  The entire economy of America's south was built on
cotton.  Cotton could neither be planted nor harvested, it was
believed, without the unpaid toil of thousands of human beings out of
whose wretchedness the plantation master could squeeze his profit.
Nevertheless, at first only a few farseeing men and women, dreamers
all, in Massachusetts, later many more citizens, came to believe the
impossible was possible, that the slaves could be freed and slavery
ended.
.    The impossible goals I mention here are possible, just as it is
possible that we will destroy the human race.  I alone can neither
achieve the one nor prevent the other.  But neither can it be done
without me, without us.
.    I have no right to demand anything from my colleagues.  But they
must know that we have the power either to increase the efficiency of
the mass murder instruments we have and thereby make the murder of our
children more likely, or to bring the present insanity to a halt, so
that we and our children have a chance to live in human dignity.
.    Let us think about what we actually accomplish in our work, about
how it will be used, and whether we are in the service of life or
death.

Computers for Christ - Chicago