Internet Info 1997 December/Internet_Info_CD-ROM_Walnut_Creek_December_1997.iso/ietf/95apr/msgway-minutes-95apr.txt

Text File | 1995-05-27 | 12KB | 285 lines

MessageWay BOF (MSGWAY)

Reported by Danny Cohen/Myricom

The MSGWAY BOF, chaired by Danny Cohen, was held on Tuesday, 4 April, at
the 32nd IETF meeting in Danvers, MA. Sixteen people attended.

Danny presented the problem and a proposed approach (see slides).  A
discussion of MsgWay and of the working group followed.


The Problem

While the speed of computing circuits increases with time, the speed of
light is unchanged.  As a result, distances shrink.  For example, the
diameter of Ethernets has shrunk from 2Km to 0.2Km as their speed grew
from 10 to 100Mbps.  Similarly, buses that used to dominate the
inter-board communication (e.g., VME) are useful nowadays mainly for
intra-board communication (e.g., PCI).

Modern computing systems in general, and MPPs (Massively Parallel
Processing systems) in particular, use ``I/O fabric'' (MPP-networks) in
stead of the traditional I/O buses.

Most MPP-networks are built to handle variable length packets, are made
of very short point-to-point FDX links of high performance (high data
rates, low latency, and very low BER), have error detection and flow
control, and use cut-thru (aka ``wormhole'') switches with source
routing.  In spite of these common similarities, each MPP network is
typically an island unto itself, incapable of interoperating with other
MPP-networks.

There is a need to use several homogeneous MPPs, and clusters (or
networks) of workstations, as a single MPP, without losing the high
performance communication native to them.


A Proposed Approach

The interoperability between heterogeneous computing systems should be
handled at Level 3, like IP, not just at the lower levels.

IP, that has successfully served the Internet for over 20 years as the
basic tool for interoperability among heterogeneous computers, is not
appropriate for MsgWay because many tradeoffs were made sacrificing high
performance for generality and scalability.  In addition, IP does not
address individual processors in MPPs.  (However, IPv6 could have been
modified to fix this problem.)

The MsgWay approach is to define a Level 3 protocol that is similar in
its philosophy to IP, but has implementation details geared toward
high-performance, possibly at some cost of generality and wide area
scalability.

MsgWay will be a Level 3 protocol, like IP, which could support IP (by
encapsulation).  The MsgWay protocol will have both an EEP (end-to-end
protocol, like IP) and an RRP (router-to-router protocol, like GGP).

Among the tradeoffs that made IP general, but proved to be deficiencies
for high performance are:


   o Long addresses (32 going on 128 bits)
   o Addressing ``hosts'' only (not individual processors)
   o No support of source routing
   o Need for routers with global knowledge
   o Hierarchical de-muxing
   o No flow control
   o No error detection
   o No fault recovery
   o No support of DMA
   o No support of byte alignment
   o Fields not sorted by need


MsgWay will alleviate these deficiencies by having addresses of 16-bits,
that could be dynamically assigned for sessions.  MsgWay will support
both source routing (for Level 2 forwarding) and Level 3 addressing (for
Level 3 forwarding).  The use of the source routing would allow the
MsgWay switches to operate without any routing knowledge that has to be
loaded to them.  MsgWay will have format to support zero-copy operation
(i.e., direct copy from the network interface into the destination user
area).  MsgWay would have flow control based on the flow control of the
participating networks.  Similarly, MsgWay would have error indication
in trailers, to allow the use of various CRC hardware.  Even though each
participating network may use a different technique for error detection,
MsgWay would have a uniform way to indicate errors.  MsgWay will address
the alignment issue, to allow computers with different chunks (such as
the Paragon's 8B-chunks, RACEway's 4B-chunks, and Myrinet's 2B-chucks)
to efficiently communicate.  In addition, in order to minimize the
wormhole latency, the fields in the MsgWay protocol header will be
sorted by their need (e.g., starting with the destination address that
is always needed).

MsgWay would support dynamic mapping and discovery required for
automatic fault recovery.

Like IP, MsgWay does not define performance figures, connectors,
communication media, address assignment, routing and discovery, APIs,
and so on.  Following the IP philosophy, all these issues will be
defined separately.

MsgWay defines a Level 3 protocol for interoperability of heterogeneous
multi-processors at high performance.



Discussion


A discussion about the proposed MsgWay activity followed the above
presentation.  Several questions were raised by the participants in the
BOF.


   o Why should MsgWay be an IETF activity?

         It is proposed to conduct the MsgWay activity as an IETF
         working group because of the firm belief that interoperability
         should be handled at Level 3 (not just at Levels 1 and 2), and
         because of the recognition that MPP-networks are computer
         communication networks with much in common with the networks
         that the IETF community is dealing with.  MsgWay is a small
         computer network, not an extended computer bus.


   o Why not use IP ``as is'' with slight modifications, as needed for
     high performance?

         It is believed that this is the proposed approach.


   o What about transport level issues, like reliability (a la TCP)?

         It is left for higher level protocols, as/if needed (note that
         this is exactly IP's approach).


   o Must MsgWay hosts use source-route?

         No.  MsgWay will support both Level 2 forwarding (by source
         routes) and Level 3 forwarding (by addresses).


   o Must processors in the same host (say a Paragon) use MsgWay among
     each other?

         No.  They may use their native communication system.  For
         generality the API may look the same but there is no need to
         use MsgWay for internal communication within a system.  This is
         similar to the use of IP between hosts on the same LAN. (Hosts
         on the same ethernet could communicate by raw ethernet packets,
         without IP - but using IP has some advantages.)


   o How is the Source Route handled?

         It is consumed along the way (not an incremented pointer).
         This allows each network along the path to be presented with
         exactly the optimal bit pattern for its use.  Note that this
         requires recomputing the checksum.


   o MTU?

         The maximum packet size will be configured for the entire
         MsgWay (probably not exceeding a few KBytes).  It is assumed
         that each participating network can handle large packets.
         There is no need to legislate that all MsgWay's always have the
         same MTU. It is expected that the mapping process will
         automatically discover the MTU and disseminate it.


   o Interconnection of separate MsgWay-islands?

         MsgWay-islands could be interconnected via IP. They could be
         either (1) interconnected by using IP as a tunnel encapsulating
         MsgWay, or (2) connected by using IP and having the
         MsgWay-islands independent of each other (treating MsgWay as a
         LAN). Once IP is used over WANs the high-performance of MsgWay
         is most likely to be lost.


   o Up to how many stages of source-route make sense (rather than
     addresses)?

         This is a runtime binding.  No need to decide at committee
         time.  Msgway should be able to handle both.



The MSGWAY Working Group

Mailing list information for the MSGWAY group:


      General Discussion:    [e-mail]
      To Subscribe:          [e-mail]
      Archive:               ftp://ftp.isi.edu/msgway/msgway.mail


Danny will work with Frank Kastenholz, one of the Internet Area
co-Directors, on a draft charter for the proposed working group and will
post it on the mailing list.

The MSGWAY Working Group is expected to conduct its work over e-mail, to
meet at IETF meetings, and to possibly have additional meetings between
IETF meetings.

Danny reported that in addition to those who participated in the 32nd
IETF BOF meeting, there are about 20 other people from academia,
government, and industry (see list below) who expressed interest in
participating in defining MsgWay.  Most of them had already participated
in two meetings discussing MsgWay (January 1995 in Utah, and March 1995
in Florida).  Most of these people expressed interest in participating
in the IETF MSGWAY Working Group.  Given that both Jon Postel and Danny
Cohen were already scheduled to be in this IETF BOF meeting, the others
were advised that their presence at this meeting was not necessary.

Among those who participated in the earlier MsgWay meetings are people
from Intel, Mercury and Myricom that are committed to implement and to
demonstrate interoperability among Intel's Paragon, Mercury's RACEway,
and Myricom's Myrinet, using the format that will be adopted for MsgWay
by the MSGWAY Working Group.



                                Academia

 Jon Postel       [e-mail]              USC/ISI
 Tony Skjellum    [e-mail]  Mississippi State University
 Al Davis         [e-mail]             Univ of Utah/CSD
 Barney Maccabe   [e-mail]          UNM/CS + Sandia
 Stu Tewksbury    [e-mail]            West Virginia University
 Andy White       [e-mail]                Los Alamos National Lab

                                Government

 Mike.  Masters   [e-mail] Naval Surface Warfare Center
 Jose L. Munoz    [e-mail]              ARPA/CSTO
 Bob Parker       [e-mail]            ARPA/CSTO

                                 Industry

 Danny Cohen      [e-mail]              Myricom
 Chuck Seitz      [e-mail]              Myricom
 Craig Lund       [e-mail]                Mercury Computer Systems
 Alan L. Pool     [e-mail]                  Mercury Computer Systems
 Bob Graybill     [e-mail] Martin Marietta Laboratories
 Greg Chesson     [e-mail]                Silicon Graphics
 Glenn.  Ladd.    [e-mail]       Hughes
 Lloyd Lewins     [e-mail]     Hughes
 Phil Sementilli  [e-mail]       Hughes Missiles
 Dave Dunning     [e-mail]      Intel SSD
 Paul Pierce      [e-mail]           Intel SSD
 Stephen Wheat    [e-mail]       Intel SSD
 Joe Brewer       [e-mail]          Westinghouse
 Bob Means        [e-mail]                 HNC, Inc.
 Marc Campbell    [e-mail]           Northrop-Grumman


Schedule

It is expected to have a rough draft of the minimal MsgWay protocol by
the 33rd IETF meeting in mid-July.

It is expected that the first interoperability demonstration will take
place no later than October 1995.  Myrinet, RACEway, and Intel's Paragon
are expected to participate in that interoperability demonstration.


Legalities

Frank Kastenholz of FTP Software brought up legal issues.  It was
suggested that Danny should check with Carl Malamud about related
patents that may be in the way of MsgWay.  (Already done.)

It was reported that Myricom has trademarked both MessageWay and Msgway,
for free use by this activity.

By including the slides and this text in the proceedings of the IETF we
are establishing MsgWay prior-art at least for April 95.