ftp.cse.unsw.edu.au/2014.06.ftp.cse.unsw.edu.au.tar/ftp.cse.unsw.edu.au/pub/doc/misc/lightning/Strike@Hawaii

Internet Message Format | 1992-10-18 | 11.6 KB

From: [e-mail] (Bob Cunningham)
Subject: lightning, PACXs and computers
Date: 23 Jul 85 21:30:14 GMT
Organization: Hawaii Institute of Geophysics

Had serious problems from a lightning strike last week.  Now that most
everything is working again, though I'd pass on some details.  Those of you
in some parts of the U.S. might not find this sort of thing so unusual, but
if you think it Couldn't Happen Here, read on ...

Thunderstorms occur only a few times a year out here in the Hawaiian
Islands, and lightning tends to strike in relatively harmless places.  Last
Wednesday at about 1500 a small but intense little storm was accompanied by
a relatively intense lightning strike on the the eastern end of the
University of Hawaii campus.  Eyewitness reports of exactly where the
lightning struck are numerous and contradictory, observers up to five miles
away were mightily impressed by the huge bolt of lightning and very loud
thunder.

There was no apparent effect on the power lines, outside of the possible
"jiggling" of a few cycles.  The effect on data communications lines was
more impressive.

Briefly, all of the on-campus computer facilities with data communications
lines extending beyond their immediate buildings (typically RS232
3-or-4-wire leased phone circuits or similar) suffered burned out
ports and burned out terminals, notably:

    several VAX780s at the Center for Cultural Interchange between
    East & West (an on-campus "think tank" type organization) lost
    multiplexer (port) boards and distribution panels.

    a VAX785 in the Information & Computing Sciences Dept. lost
    multiplexer boards & distribution, and appears to have suffered
    memory and/or cpu damage.

    several VAX780s in the Management Systems Office (administrative
    computing) lost multiplexer boards & distribution, and may have
    more extensive damage.

    the High Energy Physics group lost multiplexer boards & panels
    and a tape drive on their VAX780.

    a VAX750 in the Marine Sciences Building kept on operating,
    oblivious to the fact that more than 8 of its ports had just burned
    out.

    an H800 at the Hawaii Institute of Geophysics lost virtually all of
    its DMACP (port) daughter and mother boards.

    All three of the larger port selectors on campus (1 large Gandalf
    at the UH Computing Center, 1 smaller Gandalf at the East/West
    Center, and 1 Micom 600/2 at the HIG) suffered a large number of
    burned out line boards, along with some port boards.

None of the computers which crashed (all of the above, except the VAX750),
suffered disc crashes.

For the most part, most of the damage appears in obviously-burned-out RS232
driver circuitry, and continues back into logic on the various port
interface (multiplex) boards.

An as-yet-uncounted number of terminals (minimum estimate: several hundred)
were damaged.  Typically, the RS232 driver chips (most often 1488) were
burned out.  In about 30% of the terminals I've personally checked so far,
damage spreads further into the terminal logic board, and sometimes all the
way to the power supply.

Most of the affected terminals were on relatively long lines (more than
200' or so), however many terminals with much shorter lines -- even some
in the same room as their computer -- suffered damage.  Usually, these were
attached to port/multiplexer cards along with one or more long-line
terminals.

Microcomputers (such as IBM PCs) chugged right along without noticing the
lightning strike -- unless they had an out-of-building asynch connection,
in which case they suffered damage like terminals.

In many cases (notably the IBM3081D, DEC20 and HP3000/64 at the UH
Computing Center), port selectors protected individual machines almost
completely.

I don't have an accurate monetary estimate of the damage, but
it will certainly run over $100,000.

Field engineering response from Digital was quite good, but handicapped by
the fact that DEC only carries one "kit" of spares in the islands for each
model of VAX.  Assessing the damage took a day or so (UH has almost
entirely, basic service), some replacement boards arrived quickly, others
not until after the weekend.  Response from other vendors was
similar; Harris pulled parts off its production line
in Florida and burned-in over the weekend for us.  Micom responded
quickly to ERE requests, shipping within 24 hours via Federal
Express.  I don't have details on Gandalf's response yet.

A wide variety of different maintenance agreements were in effect with the
various manufacturers.  Many of the contracts turned out to have "acts of
god" exclusion clauses.  Fortunately, all of the vendors involved took the
attitude that they'd fix now, and worry about who would pay later.
--
Bob Cunningham  {dual|vortex|ihnp4}!islenet!bob
Honolulu, Hawaii


> ... In particular I wonder
> if your communications lines are above or below ground?  What about
> your power lines?  Are they above or below ground? Secondly, do your
> communications lines have any type lightning or surge protection?
> Were any of the affected lines shielded?  Do you have surge protection
> on your power lines?

All of the comm lines are underground, in conduits (not steam tunnels,
which we don't have out here).  In some cases these were leased "4-wire
control circuits" from the local phone company (Hawaiian Telephone Co.,
a GTE subsidiary); in other cases, self-installed.  All power lines are
also underground (separate conduit, usually completely different trenches).
Note that power lines were not affected, nor were regular phone lines (not
a single modem was zapped, and no phone problems of any kind were
reported).  None of the comm lines had lightning or surge protection.  Some
(a few) were shielded -- though I'm not sure how well the shielding was
grounded.  Some power lines have surge protection, some not; in any case
the path was NOT thru the power lines.

Interestingly, some of the affected lines were entirely within a single
building, typically of reinforced concrete (and for ordinary r.f. signals,
the rebar cage usually provides some shielding).  My theory is that -- at
some point -- those comm lines were bundled with other lines passing from
building to building.

> We are in a lightning prone area, and I have been attempting to
> understand the intricacies of lightning protection recently.  I
> confess that the more I hear the more confused I get.  I have
> concentrated on power protection, but your problems seem to have resulted
> from a communications line hit.

That's what the evidence definitely indicates.  Surprised us.

> I wouldn't have thought that such a
> hit would spread to so many machines, unless you had a tremendous
> earth current transient that affected buried lines in a wide area,
> or were unlucky enough to get a hit on some central distribution
> point.

There was no central distribution point -- several completely separate
computer "centers" were involved, none tied together in any way.  I'd say
the ground current transient (or, perhaps several as a charge briefly
"bounded" back and forth from the clouds to the earth and back again until
settling) seems likely.

> Am I wrong in this evaluation?  Is there some other way such
> a strike could get into a widespread communications network?  How
> does one go about protecting such a network from lightning?

I'd like some of those answers myself.  Anyone else with experience care to
comment?
--
Bob Cunningham  {dual|vortex|ihnp4}!islenet!bob
Hawaii Institute of Geophysics Computing Facilities
Honolulu, Hawaii


From: [e-mail] (Bob Cunningham)
Subject: Re: lightning, PACXs and computers (followup)
Date: 11 Aug 85 00:14:22 GMT
Organization: Hawaii Institute of Geophysics

At last count, over 40% of the zapped terminals we had were repairable only by
replacing the 1488 and/or 1489 chips (usually the 1488).  An
industrial-quality solder remover (heater + vacuum pump) is highly
recommended.  The percentage seems to be slightly higher (better) for port
selector boards (Gandalf & Micom).  Needless to say, we've been installing
sockets for those chips where practical.

On the computer side, all the port/multiplexer (DZ or whatever) boards
were just swapped out.  Not sure what percentage just had driver chips
burned out or not.  Subsidiary damage (one tape unit and some memory
boards) was limited to just 2 systems, and my opinion is that not all
the equipment at those locations was solidly tied to a single ground.

Besides the followup articles in this newsgroup, I've received a
considerable number of mail messages (all appreciated, though
I've not had time to send individual replies), falling into
two categories:

    1) similar horror stories
    2) thoughtful advice on lightning protection

Apparently, similar incidents (lightning damage via local data comm lines)
are much more common & widespread than I'd have thought.  Unless you're in
a very unusual location, if you've got comm lines going between buildings,
be warned:  something similar might just happen to you.

While I've received a number of very sensible suggestions on lightning
protection, there doesn't seem to be one single solution we could adopt
in all cases.  Suggestions have ranged from using opto-isolators,
diodes (of various sorts, including MOVs), to using telco-type
spark gap devices, to fiber optics.  Each approach has some good points.

However, the thought of having to install any particular suggestion
on the 800 or so data comm lines around campus which probably should have
protection is a sobering thought.  Schemes cheap in material (e.g. diodes)
look to be rather labor-intensive (if both terminal and computer/port
selector ends both need protection -- which seems optimal).  Schemes
cheap in time (e.g. fiber optics) look a bit expensive in parts
(though the thought of pulling a lot of fiber optic cables to replace
twisted-pair lines is also rather sobering).

So far, I think our best approach here is to stick to the basics.

For starters, going over all of the central grounds.  All Computer
equipment and auxiliary racks within a room should be securely grounded
to a single point with generous-size braided ground straps.
The objective is to minimize any possible ground differences between
computers and their peripherals (including port selectors).
It seems to be a good idea to tie down all incoming terminal
grounds (RS232 pin 7) to that same point -- as the lines come in
(typically on punch blocks).  This should localize damage (typically to the
port selectors).  It also seems reasonable to dedicate whole computer-side
port/multiplexer boards to PACX lines ... no more mixing direct-connect
and port selector terminals.  Typically, the only terminals I want
to leave directly connected will be those in the same building
(preferrably the same wing) as the computer site.  This should also
minimize computer-side damage.

Fortunately, port selector boards are MUCH cheaper to repair/replace
than computer port/multiplexer boards.

Relatively expensive terminals (graphics types, for the most part) should
get some special form of isolation at their end from the RS232 lines
(preferrably something simple that plugs in between the RS232 line
and the terminal).

Cheaper terminals (< $1,000) I think are best left as-is for now.

All NEW inter-building trunks (multiple data comm lines) that I have
any control over will be fiber optic lines (stat muxes at each end,
of course).

We do have some plans for a couple of real LANs around campus.  I
do believe that specs for those will now definitely include some
form of lightning protection.
--
Bob Cunningham  {dual|vortex|ihnp4}!islenet!bob
Hawaii Institute of Geophysics Computing Facilities
Honolulu, Hawaii

--
Dave Horsfall (VK2KFU), Alcatel-STC Australia, [e-mail]
[e-mail], ...munnari!stcns3.stc.OZ.AU!dave
    WordStar 4 - The thinking person's word processor