NetNews Usenet Archive 1992 #27/NN_1992_27.iso/spool/comp/sys/stratus/77

Internet Message Format | 1992-11-19 | 12.4 KB

Path: sparky!uunet!think.com!rpi!bu.edu!transfer.stratus.com!transfer.stratus.com!usenet
From: [e-mail] (Dan Danz)
Newsgroups: comp.sys.stratus
Subject: Stratus Fault Tolerant Architecture
Keywords: FT duplex fault-tolerant Stratus architecture
Message-ID: <[e-mail]>
Date: 20 Nov 92 17:51:44 GMT
Organization: Stratus Computer Inc, Marlboro MA
Lines: 212
NNTP-Posting-Host: bittersprings.az.stratus.com

Recent  discussions  in  this   group about   the   Stratus approach to fault
tolerance include  over-simplified close guesses,  and some downright slanted
information provided by employees of competing vendors who have their own axe
to grind

So I  thought I'd try to  present a brief  explanation  of  the Stratus fault
tolerant (FT) architecture  from the perspective  of a Stratus  engineer (who
else  knows   the product better?).    In  most   cases,  I'll  use   Stratus
terminology;     however,  interested  readers    should  note   that Stratus
architecture systems are also marketed by  business partners such as IBM (who
calls it the System/88), Olivetti (CPS/32), et al.

The basic component of a Stratus  system is a  "module", a complete computer.
Each module is bus-oriented, and contains one or more processor  board pairs,
one  or more memory  board pairs, and  pairs of  I/O controllers connected to
various peripheral  devices, including   disks,  tapes,   and  communications
interfaces.   A module contains  its own  redundant power  supplies,  battery
backup  subsystem, and  various card  cages for  I/O  line adapters  and  I/O
controlling logic.

BASIC MODULE BUS:

The major boards of a Stratus module interface with the  StrataBUS, which has
32 logical slots implemented in various models with varying numbers (6-40) of
physical slots.  Even-numbered slots derive  power  from a subsystem that  is
totally independent of the power subsystem used by  odd-numbered slots of the
bus, allowing a module to survive the complete failure of either of its power
subsystems.  Each board regulates   its own power, thus   it  is  possible to
remove and insert boards  while the bus  is  fully powered and the system  is
operating.  No components  of  the  system need to  be disabled to  remove or
insert a board.

The StrataBUS is a synchronous bus.  The system clocking  function drives all
circuitry in the system  and is distributed  using a single bus  signal.  The
source of this clock is a component that has been engineered for  a very high
Mean Time Between  Failure  (MTBF); nevertheless,  a fully-duplexed redundant
clock option is available.

Nearly every signal on  the bus  is duplicated.  The   bus is  viewed  as two
independent buses, "A" and  "B",  each   with its  own power,  ground,  data,
address, and control lines.  Parity protects the lines of each bus.

Each major system board simultaneously interfaces with both  buses.   A board
drives the same data on both buses and reads data from both buses if they are
both enabled.  (The hardware automatically instructs  all  boards to ignore a
suspect bus.)

SYSTEM BOARDS:

The boards that interface to the StrataBUS have several common features.

First,  they all operate synchronously  at a  simple  multiple of  the common
system frequency.

Second, all boards are  self-checking and  auto-isolating.  The boards  check
themselves for component errors on every clock  tick and  will not place data
on the bus if a  board finds  itself to  be in  error.  This self-checking is
typically performed by  duplicating the logic on  the board and  running both
sets of  logic  independently  but   synchronously.  The outputs   of   these
independent logic networks then run through on-board comparator circuits that
enable/disable the bus drivers.  The self-checking logic automatically causes
a board to "break".  When a  broken board breaks, a red  LED  on the front of
the board lights to identify it.  A  broken board never  drives data onto the
buses.

Third, each board must provide its own power regulation (see below).

Fourth, each board is  self-identifying, providing system software with coded
information describing  the board's type,  the  revision  level of  the board
circuitry and of  the PROM  software on  the board,  and a  limited amount of
repair history.  This information  is critical to the  software  that  places
re-inserted boards on-line.  If the boards are incompatible, the  system does
not accept the new board.

Fifth, all boards that interface to the  StrataBUS must obey  a set of common
interface conventions used by the Stratus maintenance and diagnostic software
for testing, initializing, and  enabling boards in the system.   One specific
requirement  is   that  both  halves   of    a board   must    behave totally
deterministically.   Any logic must progress from  state to  state  with each
clock  tick in a  deterministic manner.  In  particular, "don't care"  states
(bits that can harmlessly assume either a zero or one)  are not allowed since
they might present conflicting values to the comparator logic.

Boards within a Stratus system nearly always occur in pairs.  Such boards are
referred to  as "partners".  Major  boards operate  in two  ways:  either  in
synchronous  lockstep with a  partner board, or  only logically paired with a
partner board.

When   operating  in  lockstep,   proprietary Stratus  hardware  and software
synchronize  both self-checking boards  of the pair;  thus both boards (for a
total of  four sets of logic)  do exactly the  same thing  on any bus  cycle.
This  imposes another   requirement for totally deterministic behavior,  this
time between partners.

Two boards  running in synchronous  lockstep  read the same  values  from the
StrataBUS simultaneously.  For  example, two client partner  boards ask for a
bus cycle at exactly  the same time (on  the same  cycle) by  asserting their
intention to use  the bus.  If the bus  arbitration network grants the bus to
the pair of boards, they  both place the address for  a read or write on  the
bus  at the next  clock  tick.  Two  ticks later, the clients  boards  or the
responding boards - usually memory controllers  - place the  data on the bus.
The data is taken off the bus one tick later.

Boards that interface with devices such as disks,  tapes, and the StrataLINK,
operate in a logically  paired state.   Such boards  are not  synchronized in
lockstep with their partner; rather, they  rely on operating  system software
for an equivalent function.  Nevertheless, the two halves of each  board must
be synchronized and continue to  conform to the self-checking, auto-isolation
paradigm.

Disk mirroring with dedicated disk controllers uses this logical pair method.
Software ensures that mirrored disks contain  duplicate, consistent copies of
data.  This software is completely invisible and  inaccessible to application
code  (and most  operating  system   code) and  therefore  need not   concern
developers.

Certain    pairs  of   boards    perform  "reflexive checking":    interboard
communications to guarantee that they  are synchronized.  If  the  boards get
out of synchronization, one  of them  is forced off-line automatically by the
hardware  so that  the board's signals,  which are  ORed on  the bus,  do not
confuse other boards  in the system.  (For  one processor board  product, the
designers called it the T/A (transaction analysis) function - "I'm okay, your
okay". :-) )

All  memory boards  monitor the bus  for  problems with  the bus transceivers
themselves, a  type of failure that  cannot be  detected by on-board checking
logic.

BOARD FAILURE:

When a component fails, the failing  board automatically drops  out while its
partner continues to run.  The partner does not  "take over" as would be case
with a backup component; instead, it continues the actions  it was performing
synchronously  with  its partner.  One   board rather  than  two is supplying
signals to (and responding to  requests from) the  StrataBUS.  If the  failed
board was not running in lockstep  with a partner,  then the operating system
effectively  hides  the   unavailability    of the   failed board   from  the
application.

When a failed board is off-line, the system recalculates the boards MTBF.  If
the MTBF threshold is exceeded, the board is marked  for replacement, and the
Remote Service Network (RSN)  is invoked to report the  failure to a  Stratus
Customer Assistance Center (CAC); otherwise, the board  performs  a series of
self tests.  If it  fails  these tests, the failure is  not transient and the
replacement  process is initiated.  If the  board passes these self tests, it
is   either re-duplexed with its   partner board (for  lockstep partners)  or
logically brought back into service.

POWER SUBSYSTEM:

Each power supply has an associated  battery backup system that works  in two
modes: it either powers all boards  within the module, or  it powers only the
memory boards.  The system reaction to  powerfail is to  ride through up to 6
seconds of AC power loss.  In 90% to 95% of all power failures, power returns
within a  few  seconds, which  case the  system simply  continues what it was
doing.  After the ride through time expires, the system quiesces  all boards,
saves information needed  to restart them in  main memory, and supplies power
only to the main memory boards.  In this mode, the system is not operational,
but its complete state is preserved.  If power is  restored with  an  hour or
two (depending on the size of the memory and the state of the batteries), the
system   software restores the  system   to its  state  at the time of power
interruption.

ARCHITECTURE TRADEOFFS

The entire Stratus architecture reflects  tradeoffs between simplicity,  ease
of maintenance, lifetime  system costs,   logistics concerns, and  technology
trends on one hands, and slight increases in product cost  on the other hand.
The resulting products clearly justify such tradeoffs.

First, the Stratus solution requires more hardware.  Any truly fault-tolerant
system  requires more  hardware  than a  conventional system,  but  a Stratus
systems needs, in some cases,  four times the  amount of  hardware.  However,
the cost  of   hardware - primarily   logic   chips  -   that   Stratus  must
quadruplicate is low  and is decreasing, becoming a  less significant part of
the total cost  of  a computer  system,  particularly  when measured over the
lifetime of the system.   The components with  the most  significant hardware
costs - mainly on-line memory and peripherals - need only be  duplicated (or,
in   the cases  of   disks, made   redundant  in some    way), and any  truly
fault-tolerant computer has these same requirements.

Second, to  design circuits that use Stratus  concepts,  logic chips  must be
totally deterministic, something  that is becoming less  of a problem as chip
manufacturers become aware of the requirement.

Third, since the logic  circuits depend on  total  synchronization,  anything
that detracts from this can be  a potential problem.   The primary sources of
difficulty in this area are new revisions of chips (new  mask, usually to fix
bugs).   Chips that behave slightly  differently  from earlier  revisions are
harmless to all other architectures, but to the Stratus architecture they can
be  devastating.   If the   differing chips   are  on the  same   board,  the
self-checking logic detects  it and  the board (as we say)  "red lights".  If
the differing  chips are on  separate boards, the cannot  be  synchronized or
cannot  stay synchronized.    Therefore  Stratus must  be  sensitive  to  the
revision level of chips that are placed on boards.

I took most of the above information from "Stratus Technical Report TR-1: The
Stratus Architecture", and I've only touched the surface of it.  There's lots
more  information   (and drawings)  in  the document:   System Design  Goals,
Corporate Product Strategy, Issues  of  Fault Tolerance,  System Architecture
Overview, Recovery Scenarios, Stratus Software, and Service Strategies.  This
marvelous, 20-page, very technical discussion of the architecture was derived
from a  technical paper written by  Steve  Webber, a Stratus  pioneer, and is
available to interested persons from  any  Stratus sales office.  Recommended
reading.

--
L. W. "Dan" Danz     (WA5SKM)     VOS Mail:  [e-mail]
Sr Consulting Software SE         NeXT Mail: [e-mail]
Customer Assistance Center        Voice Mail/Pager: [phone]
Telecommunications Division       Customer Service: [phone]
Stratus Computer, Inc. 4455 E. Camelback #115-A, Phoenix AZ 85018