Internet Info 1997 December

home *** CD-ROM | disk | FTP | other *** search

/ Internet Info 1997 December / Internet_Info_CD-ROM_Walnut_Creek_December_1997.iso / drafts / draft_ietf_j_p / draft-ietf-pktway-protocol-eep-spec-01.txt < prev next >

Wrap

Text File | 1997-09-24 | 49KB | 1,181 lines

PktWay-WG <01> EEP-spec D R A F T September 1997 Proposed Specification for the End-to-End (EEP) PacketWay Protocol ------------------ <draft-ietf-pktway-protocol-eep-spec-01.txt> Danny Cohen (Myricom) Craig Lund (Mercury Computers) Tony Skjellum (MSU), Thom McMahon (MSU) and Robert George (MSU) PktWay-WG The shorter "PktWay" is used for "PacketWay". This page...................................................1 PktWay at a Glance ("cheat-sheet")..........................2 A note about the PktWay documents...........................3 Notations...................................................4 Overview....................................................5 Introduction................................................7 PktWay EEP Messages........................................10 The PktWay Message Structure...............................10 [1] Optional Sequence of L2RHs and Symbols.................11 [2] EEP Header (16 bytes) (PH).............................14 [3] Optional Header Fields (OH)............................18 [4] Optional Data Block (DB)...............................19 [5] Optional Trailer Fields (OT)...........................19 [6] EEP Trailer (TAIL).....................................19 Appendix-A: A Recommendation for PktWay Address Assignment.20 Appendix-B: Glossary.......................................21 Appendix-C: Acronyms and Abbreviations.....................23 Information about the PktWay activity may be found in the URL: http://www.erc.msstate.edu/PktWay/ Please send your comments re this draft to <Cohen@myri.com>. PktWay-WG <02> EEP-spec PktWay at a Glance ------------------ 2 6 type 24 16 16 PW-Hdr+-+------+-------+--------+---------+--------+--------+--------+--------+ PH1|V| P | Destination-Type | Type-Extension | Packet-Type | +-+-+---++--------------------------+-+------+--------+-----------------+ PH2| E | PL| Data-Length (8B-words) |h| RZ |0 Source-Address | +---+---+--------+--------+---------+-+------+--------+--------+--------+ 4 3 25 1 7 1 23 type = 0xxx Physical Address 10xx L2RH 110x Reserved 1110 Logical Address 1111 Symbols 2 6 2 6 8 8 8 8 8 8 +--------+--------+--------+--------+--------+--------+--------+--------+ L2RH |V| P |11LLLLLL| SR01 | SR02 |........|........|........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ Length 2 6 4 6 8 8 8 8 8 8 +--------+--------+--------+--------+--------+--------+--------+--------+ Symbol|V| P |1011ssss|ssssssss|ssssssss| Length | data |........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ <---- Symbol Type ---> 2 6 8 8 8 8 8 8 8 Opt'l +--------+--------+--------+--------+--------+--------+--------+--------+ hdr |TCtttttt|LLLLLLLL| data |........|........|........|........|........| fields+--------+--------+--------+--------+--------+--------+--------+--------+ T: 0=optional, 1=mandatory; C: 0=more OH-fields follow, 1=last OH-field 8 8 8 8 8 8 8 8 RRP +--------+--------+--------+--------+--------+--------+--------+--------+ Record| RTyp | PL | RL |........|........|........|........| +--------+--------+--------+--------+--------+--------+--------+--------+ RRP-messages: GVL2, L2SR, RDRC, TELL, INFO, HRTO, WRU, GVRT, RTBL; RTyp: ADDR, NAME, CAPA, LADR, SRQR, MTUR, RCVF, RTHD; PktWay-WG <03> Documents A note about the PacketWay Documents ------------------------------------ The PacketWay protocol is defined by a series of documents: * EEP (End-to-End Protocol) * RRP-1 (basic Router-to-Router Protocol) * RRP-2 (dynamic inter-SAN routing) * PktWay enumerations Each of these documents should include the same "PacketWay at a Glance (Cheat=Sheet)", this note, and the Notations page. They should include also (as appendices) a copy of the PacketWay glossary of terms and its acronyms and abbreviations list. The EEP and the RRP documents will be published first as Internet-Drafts and later as Proposed-Standards, Draft-Standards, and Standards. The Enumeration Document will be first published as an "Informational-RFC" and later will be maintained by IANA. The enumeration document may be attached to the EEP/RRP documents, as a matter of convenience. The enumeration is NOT a part of the PktWay standard, just as RFC0739 (the original "Assigned Numbers" RFC) is not a part of RFC0791, that defines IP. Similarly, the EEP-document has "Appendix-A: A Recommendation for PktWay Address Assignment" which is a recommendation only and NOT a part of the PktWay standard, just as IP-address-assignment is not a part of RFC0791, that defines IP. PktWay-WG <04> Notations Notations --------- The shorter "PktWay" is used for "PacketWay". 8B means "8-byte" (64 bits). 0x indicates hexadecimal values, e.g., 0x0100 is 2^8=256(decimal). 0b indicates binary values, e.g., 0b0100 is 4(decimal). xxx indicate a field that is discarded without any checking (e.g., padding). [fff] indicates that fff is an optional field, when appropriate. [exp], in equations, is the integral part, rounded down, of `exp`. e.g., [17/8]=2. All length fields do not include themselves, and therefore may be zero. Lengths are specified either (a) by byte count, implying that some padding bytes may follow to fill 8B-words, or (b) by 8B-word count and PL, the number of trailing padding bytes (with PL between 0 and 7). PktWay-WG <05> EEP-spec Overview -------- PktWay is an open family of specifications for inter-networking high performance System Area Networks (SANs) and high performance Local Area Networks (LANs) into computing clusters. Most modern SANs have much in common, such as high data rates, low message latency and low bit error rates. Such SANs are often packet networks made of point-to-point links with flow control, and utilize source routing. Yet these SANs do not provide heterogeneous networking support, and are subsequently incapable of direct inter-communications with other SANs. PktWay's goal is to "internet" such SANs and high performance LANs. The core PktWay protocol comprises the End-to-End Protocol (EEP) and the Router-to-Router-Protocol (RRP). This document describes (the EEP End-to-End) protocol of PktWay. A companion document ("Specification for the Router-to-Router (RRP) PktWay Protocol") specifies the Router-to-Router protocol of PktWay, including the definition of the format of the RRP packets. Several protocol extensions, which are layered on the core PktWay protocol, have been defined (and implemented). These include dynamic resource and routing discovery, secure PktWay, and multicast PktWay. These protocol extensions will be described in documents to be provided later. Some basic PktWay terminology requires explanation. PktWay interconnects high performance System Area Networks (SANs). Each SAN contains "nodes", which may send and/or receive PktWay packets. At least one node in each SAN is also a PktWay "router", connected to more than one SAN. PktWay's goal is to move data from a source node, (on some arbitrary SAN) to a destination node, (either on the same SAN, or on another SAN). Sources and destinations can be physical entities, such as a processor or a smart memory board, or logical entities, such as a group of cooperating processes or a collection of threads. Sources, destinations, and routers are such nodes. Within each PktWay configuration all nodes have unique 23-bit physical PktWay addresses. A system designer can assign these PktWay addresses manually. Alternatively, the optional PktWay Server Layer provides a way to assign and discover addresses dynamically. Throughout this document "address" always means the 23-bit physical PktWay address. SANs also may have PktWay addresses, which function as SAN identifiers (SAN-IDs). They are also 23-bit physical values, sharing the physical address space with the nodes. These addresses, of SANs and nodes, are unique within each instance of PktWay. PktWay-WG <06> EEP-spec To optimize for performance, PktWay has a data transfer mode that leverages the native message routing schemes used within the SANs. This mode uses a "Planned Transfer" paradigm. During the planning phase, a source node collects information on optimal routes to a destination, expressed in the various native formats of the intervening SANs. A source node later uses this information for low latency transfers to that destination. In PktWay, the transfer phase of a Planned Transfer is called "L2-forwarding". The RRP document demonstrates the use of L2-forwarding. PktWay also supports a more traditional data transfer mode that requires no planning. Such transfers specify the destinations by their addresses only. In PktWay, this more traditional approach is called "L3-forwarding". As a heterogeneous network layer, PktWay packets are transported by the native data-link layer of each SAN. As a result, PktWay packets will be encapsulated with any native routing headers and trailers as required by the local network fabric. PktWay packets may be routed by Level-2 (L2) forwarding, Level-3 (L3) forwarding, or a combination thereof. In L3-forwarding (similar to IP forwarding), the L2-routing through each SAN is determined by an inter-SAN router upon entering that SAN. The router prefixes the packet with an L2 routing header (such as a source route) corresponding to the destination address specified in the packet directing the packet either to its destination or to an intermediate router. It is a task for that router to determine the L2-routing-header corresponding to the given PktWay-address. In L2-forwarding the source prefixes the packet with all the L2-routing headers needed along the entire path to the destination. Each router has only to get the L2-routing-header from the leading L2RH (L2-Routing-Header record) that was provided by the source. PktWay does not provide Segmentation and Reassembly (SAR). Therefore, the length of a packet cannot exceed the minimum MTU (Maximum Transmission Unit) along its path. PktWay does not detect errors. It only gathers error detection information from the SANs and inter-SAN routers that a packet transits. PktWay is big-Endian 8B-word based. Hence, the terms "first bit" and "first byte" are equivalent to MSbit and MSByte. PktWay-WG <07> EEP-spec Introduction ------------ A modern MPP (Massively Parallel Processing system) is a set of processors interconnected by a high performance SAN (System Area Network). Examples are Intel's Paragon and ASCI-red, CRAY's T3D and T3E, and IBM's SP2 and SP3. Most modern SANs have much in common, such as high data rates, low message latency and low bit error rates. Such SANs are often packet networks made of point-to-point links with flow control, and utilize source routing. The problem: When such high performance SANs are interconnected to form a computing cluster, there is no efficient way to "internet" these SANs - to allow each computing node to have high performance communication directly with any other computing node, in any other interconnected SAN. IP is the general solution for "internetting" diverse networks, proven for over 20 years. However, IP was designed for the generality required for Wide Area Networks, without regard to the high performance requirements of tightly coupled systems. In addition, IP is designed to addresses "systems" rather than individual processors in MPPs (as PktWay does). For example, a 9,000 processor system is not expected to be assigned 9,000 IP addresses. The objective of PktWay is to provide high performance communication among all the processors in a cluster of tightly coupled SANs. PktWay borrows heavily from the experience and wisdom of IP, with a few modifications needed for high performance. PktWay sacrifices scalability and generality to improve performance. PktWay is slightly below IP in the OSI Reference Model. It has many Level-3 features, like IP, but also can support IP as if PktWay was a Level-2 protocol. Hence, it is below IP. In addition, PktWay supports Level-2 optimizations (such as source routing). Like IP, PktWay has an End-to-End Protocol (EEP) and a Router-to-Router Protocol (RRP). This document defines the EEP part of the PktWay protocol. Like IP, PktWay uses routers between its SANs. Each PktWay-router is composed of two (or more) half-routers (HRs), each of which is a PktWay node on a SAN. Each PktWay HR is responsible only for routing packets within its own SAN. If a PktWay HR receives a packet whose destination node resides on another SAN, it is the responsibility of the HR to forward the packet towards the HR that is responsible for that SAN. These HRs communicate with each other according to the RRP, over an arbitrary "local" physical link (e.g., PCI or PPP over a serial line). More about the RRP may be found in the companion document ("Specification for the Router-to-Router (RRP) PktWay Protocol", Parts 1 and 2). Unlike IP, the PktWay routers do not have to pop each packet back to Level-3, and are capable of operating entirely at Level-2, if this operation is requested by the communicating hosts. PktWay-WG <08> EEP-spec PktWay allows hosts to construct a source-route built entirely of Level-2 headers, allowing each SAN to exploit the full performance of its native interconnection fabric. These SAN-headers (actually, MAC-headers) are provided by the SANs that will use them, in their native format. PktWay does not define the format of the local routing envelope. Instead, it defines how the encapsulated PktWay packets should be passed between HRs, leaving it up to the local network of each SAN how to properly deliver the packet. PktWay supports resource discovery, by name or capabilities. PktWay's unit of data is 64-bit long (8 bytes). PktWay provides hosts with padding as required for various alignments. PktWay handles the Little vs. Big-Endian issue by providing a field in the EEP header which defines the endianness and the "chunk-size" of the data in the payload (Data Block). The intent is that byte-swapping hardware, if any, could be used to invert the endianness of payloads with uniform data elements (e.g., all the data being 32-bit floating point). Although this approach does not address the problems of transporting a general C-language structures, it does allows the participation of smart memory cards as PktWay nodes, as well as supporting direct memory access (DMA) operations. The PktWay protocol is designed to allow wormhole (or "cut-through") forwarding, in which a router can start forwarding packets after receiving the first four bytes only (that include the PktWay-protocol version, priority, and the destination-type) without waiting for information that may not be needed for the packet forwarding task. This is unlike IP routers that receive the sender address before receiving the destination address, even though the former is not always needed whereas the latter is. PktWay's addresses are short (23 bits) because, unlike IP, PktWay is not designed for global operation. The amount of state that is stored in the HRs per node (type, name, paths, capabilities, etc.) makes it impractical for scalability beyond a few tens (hundreds?) of thousands of nodes, over a (relatively) small number of SANs. PktWay does not support SAR (Segmentation And Reassembly). Instead, it provides means for hosts to discover the minimum transmission unit (MTU) over several alternative paths to any other node. A PktWay packet must never exceed the minimum MTU along all the network hops from the source node to the destination node. Like IP, the PktWay protocol utilizes the native capabilities of its constituent SANs and routers. PktWay does not define how each HR maps the network in the SAN to which it is attached, nor how each HR constructs SAN-headers for each of its hosts. The PktWay protocol also does not define how error-checking is conducted by each SAN (ie, CRC8, CRC32, CRC64, or anything else). Instead, PktWay assumes that these capabilities are native to each SAN, and defines only how these error indications are carried from where they were detected, to the destination node. PktWay-WG <09> EEP-spec Like IP, the PktWay protocol does not define how routes are selected, and which corrective actions should be taken in case of faults. Instead, PktWay provides the information needed by the host nodes for devising routes and detecting and circumventing faults. As a result, PktWay operates in a fashion very similar to that of IP: When hosts are powered up they contact their default routers to register themselves and to inquire about other hosts (by name or node capabilities). If hosts so prefer, they can address their destinations either by any arbitrary name, a PktWay physical address (which is handled like the Level-3 IP-address), or by concatenating a sequence of Level-2 SAN-headers. Although the generation of a sequence of L2 Routing Headers requires more effort to construct initially, PktWay source routing results in considerably lower network latencies, as the packets are allowed to cut-through route through the intervening SAN networks . When inquiring about other hosts, the HR may provide a set of several routing alternatives, each of which may have different characteristics (e.g., MTU, length, and cost). The PktWay protocol leaves it to the source nodes to choose among these alternatives. Among the features that the RRP of the PktWay protocol borrowed from the IP world are the host-redirect, host-not-reachable, and unknown-destination messages. PktWay-WG <10> EEP-spec PktWay EEP Messages ------------------- The PktWay MESSAGE STRUCTURE +--------------------------- PktWay messages have 6 components, including 4 optional ones: [1]: [Optional Sequence of L2-Routing-Headers Records (L2RHs) and Symbols] [2]: EEP Header (16 bytes) (PH) [3]: [Optional Header fields] (OH) [4]: [Optional, Most likely: Data Block] (DB) [5]: [Optional Trailer fields] (OT) [6]: EEP Trailer (8 bytes) (TAIL) Description of Optional Fields +----------------------------- [1]: as explained later, if the 9th+10th bits of a messages are 0b10 then the message starts with an L2RH, but if the 9th through the 12th bits of a message are 0b1111 then this message starts with a "symbol". The other values of these 4 bits indicate the lack of L2RH and symbols and that the message begins with the EEP-header. [3]: if the h-bit in the EEP header [2] is 1 then there are optional header (OH) fields. The sequence of these OH fields is terminated with an OH field marked as being the last one (with C=1). [4]: if DL>0, in the EEP header, zero then a Data Block (DB) is included in this message. [5]: the optional header fields, [3], may indicate that some optional trailer fields are present after the DB, [4]. The order and the formats of the trailer fields are defined by the optional header fields. It is expected that most messages will have Data Blocks (DB), and that most messages will not have Optional Header fields (OH), nor Optional Trailer fields (OT). Leading L2RHs and symbols [1] are consumed by the HRs before reaching the destination which receives only the other components, [2] through [6]. These parts, [2] to [6], constitute the End-to-End Protocol of PktWay. TAIL, the EEP trailer, [6] may be modified along the way to the destination, unlike [2], [3], [4] and [5], which arrive exactly as sent by the source. Each PktWay packet may be first L2-forwarded (zero or more times) before being L3-forwarded (zero or more times). Although PktWay headers and trailers are always in Big Endian order, the byte order of the Data Block is not defined by PktWay. Since all the elements of PktWay (L2RHs, EEP-headers, optional fields, data, and EEP-trailers) are always multiples of 8B-words, it is recommended that PktWay headers (and data) be aligned on 8B-boundaries in the nodes' memory. PktWay-WG <11> EEP-spec [1]: Optional Sequence of L2-Routing-Headers Records (L2RHs) and Symbols +------------------------------------------------------------------------ PktWay messages may start with a mix of L2RHs and symbols. A PktWay source may specify native routes, by placing the native routes before the PktWay Header. The native routes (for all SANs and LANs beyond the initial one) must appear within a sequence of PktWay L2-Routing-Header records (L2RH). In certain situations symbols may be included among the L2RHs. These symbols are used for conveying information to the routers that handle the messages, such as about encryption. A symbol does not specify its destination and is processed (and consumed) by the entity that encounters it. In L2-forwarding each intermediate HR consumes an L2RH and the preceeding symbols (if any). When a packet reaches its destination all of [1] (the Optional Sequence of L2RHs and Symbols) should be consumed. L2-Routing-Headers Records (L2RHs) +--------------------------------- The contents of the L2RH are totally SAN dependent, with the exception of the first 2 bytes that distinguish this record from an EEP-header and also provide the Length (0<L<64) indicating the number of routing bytes of that L2RH (not including these 2 bytes). This distinction (between L2RHs and EEP-headers) is necessary for routers that L2-forward packets starting with L2RHs, but L3-forward packets starting with EEP-headers. Similarly, hosts expect packets to start with EEP-headers (with optionally preceeding symbols), and may discard packets that start with L2RHs. It's up to each SAN to provide padding, as needed, to fill the L2RH words. Each L2RH is defined by the entity that will process it. In addition to routing information per se, it may also include demuxing information such as a local message-type. For example, over Myrinet the L2RH should end with 0x0300 which is the Myrinet-type assigned to PktWay (and possibly some padding, too). The L2RH must contain enough information to allow a router to create any necessary local routing headers and trailers. Although the low-level network implementation is beyond the scope of this document, the native source routing format must be documented in sufficient detail to allow for heterogeneous network interoperability. When a PktWay message is encapsulated inside any native SAN message (Paragon or Myrinet, for example), it's up to that SAN to distinguish between it and its own native packets. This is not a PktWay issue. For example, Myrinet uses its Message-Type to recognize PktWay messages. PktWay-Routers on boundaries between SANs L2-forward packets starting with L2RH or L3-forward packets starting with EEP-headers. L2RH are distinguished from EEP-headers by the value of the first two bits of the Destination-Type field. PktWay-WG <12> EEP-spec L2RH FORMAT: Each L2RH is in the format: +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|10LLLLLL| SR01 | SR02 |........|........|........| xxx | L2RH +--------+--------+--------+--------+--------+--------+--------+--------+ ^^ The first 2 bits are vv=0b00 for the working version of the protocol. They may have other values for experimental versions. The next 6 bits should be all zeroes. The next two bits must be 0b10 to indicate that this is an L2RH record. This 0b10 was chosen to be consistent with the 0b10 of PktWay-addresses, as described in [2] below. The next 6 bits are the byte count (L) of the routing information that starts in the next byte and is followed by as many padding bytes as needed to fill to the next 8B-boundary. L does not include itself, hence it could be between 0 and 63. However, since this record contains some routing bytes, L is greater than 0. The total number of 8B-words in the L2RH is [(L+9)/8] where the square brackets indicate the integer part, rounded down, of the quantity within. Therefore, the number of padding bytes is PL=8*[(L+9)/8]-2-L. L2RH EXAMPLES: An L2RH with an SR with 5 routing bytes: 0b10 L=5 #1 #2 #3 #4 #5 padding +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|10000101| SR01 | SR02 | SR03 | SR04 | SR05 | xxx | L2RH +--------+--------+--------+--------+--------+--------+--------+--------+ ^^ |<---------- routing information ----------->| An L2RH with an SR with 13 routing bytes: 0b10 L=13 #1 #2 #3 #4 #5 #6 +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|10001101| SR01 | SR02 | SR03 | SR04 | SR05 | SR06 | L2RH +--------+--------+--------+--------+--------+--------+--------+--------+ | SR07 | SR08 | SR09 | SR10 | SR11 | SR12 | SR13 | xxx | +--------+--------+--------+--------+--------+--------+--------+--------+ #7 #8 #9 #10 #11 #12 #13 padding PktWay-WG <13> EEP-spec Symbols +------ SYMBOL FORMAT: +------------ Each symbol is in the format: +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|1111ssss|ssssssss|ssssssss| Length | data |........|........|Symbol +--------+--------+--------+--------+--------+--------+--------+--------+ ^^^^<---- Symbol-Type ---> The 5th byte is the byte-count (L) of the data for this field that starts in the next byte, and is padded with as many padding bytes as needed to fill 8B-words. The length (L) does not include itself, hence it is between 0 and 255. The total number of 8B-words in the symbol L2RH is [(L+12)/8] where the square brackets indicate the integer part, rounded down, of the quantity within. Therefore, the number of padding bytes is PL=8*[(L+12)/8]-2-L. Symbols may be mixed among the L2RHs, before the EEP-header. The values of the Symbol-Type field are defined in the PktWay Enumeration document. SYMBOL EXAMPLE: A symbol with 9 data bytes. 0b1111<---- Symbol Type --->L=9 Bytes +--------+--------+--------+--------+--------+--------+--------+--------+ |vv000000|1111ssss|ssssssss|ssssssss|00001001| data1 | data2 | data3 | +--------+--------+--------+--------+--------+--------+--------+--------+ | data4 | data5 | data6 | data7 | data8 | data9 | xxx | xxx | +--------+--------+--------+--------+--------+--------+--------+--------+ PktWay-WG <14> EEP-spec [2]: EEP Header (16 bytes) (PH) +------------------------------- 2 6 24 16 16 +-+------+-------+--------+---------+--------+--------+--------+--------+ |V| P | Destination-Type | Type-Extension | Packet-Type |PH1 +-+-+---++--------------------------+-+------+--------+--------+--------+ | E | PL| Data-Length>=0 (8B-words) |h| RZ |0 Source-Address |PH2 +---+---+--------+--------+---------+-+------+--------+--------+--------+ 4 3 25 1 7 24 These fields are described below: Bytes.bits ----- ---- Version (V) 0.2 Priority (P) 0.6 Destination-Type (DT) 3.0 Packet Type Extension (TE) 2.0 Packet Type (PT) 2.0 Endianness (E) 0.4 Padding Length (PL) 0.3 Data Length (DL) 3.1 Options flag (h) 0.1 Reserved (RZ) 0.7 Source Address (SA) 3.0 Version (V) 2 bits This field is static. Its 2 bits are 0b00 for the working version of the protocol. These bits should have other values for co-existing experimental versions. Priority (P) unsigned integer, 6 bits It is anticipated that some SANs, especially those working in real time, will want to implement priorities. This field supports such usage. All ones is the highest priority, and all zeroes the lowest. Ideally, packets with higher priority should gain access to contested resources before packets with lower priority. Implementations may ignore the Priority field. PktWay-WG <15> EEP-spec Destination-Type (DT) 24 bits The purpose of this field is to specify the header type, as well as the destination of the packet, when applicable. This field may specify: * A physical PktWay address (of 23 bits); * An L2-Routing-Header (L2RH) of a variable length; * A logical address (of 20 bits); or * A symbol (of 20 bits). In addition, it is anticipated that additional types will be needed in the future. A variant of Huffman coding is used to accommodate all these methods for the Destination-Type field. This is done by assigning the MSbit of 0 to physical addresses, 2 MSbits of 0b10 to L2RH, 3 MSbits of 0b110 to future needs, 4 MSbits of 0b1110 to logical addresses, and 4 MSbits of 0b1111 to symbols. This assignment is summarized in the following table: MSbits | Method --------+---------- 0xxx | Physical 10xx | L2RH 110x | Reserved 1110 | Logical 1111 | Symbol A single C-style 16-way switch can dispatch quickly the protocol processor to the right handler required for any of the methods used to specify the destination. The Physical addresses are unique within each instance of PktWay. Nodes should have addresses assigned to them. The method of assigning unique addresses within each PktWay is not specified here. Examples of potentially addressable PktWay nodes include: groups of cooperating processes, an entire MPP, or each of an MPP's many processors or processes. The 0b10xx was chosen for L2RH to be consistent with the 0b10 indication of L2RHs, as described earlier in this document. "Logical Addresses" (e.g., for broadcast and for multicast groups) are also in this address space. The destination-Type is a "Logical Address" if its 4 MSbits are set to 0b1110. PktWay-WG <16> EEP-spec A few Physical-addresses are reserved: 0x000000 Undefined address (illegal where an address is expected, but is allowed in the SA field) 0x7FFFFE ("Hey-You!") This address could be used at power up to address nodes or routers, over point-to-point links. ("If you receive it, it's for you.") 0x7FFFFF (Broadcast) This address is reserved for broadcast operations which may be added in later versions. ("If you receive it, it's for you.") Type Extension (TE) 2 bytes An extension of the following PT field. Logically, the TE should be after the PT. However, the PT is 8B-word aligned, easier to process than the TE which is 2B-aligned, but not 8B-aligned. Since the PT is more frequently used than the TE, it was assigned to the better aligned field. Packet Type (PT) 2 bytes The PT field provides the information needed for efficient de-multiplexing of multiple protocol layers. Whereas traditional protocol layering requires several stages of sequential de-multiplexing, PktWay provides enough information to support a single combined de-multiplexing operation (such as in support of zero copy TCP). Thus, the PT field may indicate, for example, that the data blocks contain IP, SNMP, ATM, Ethernet, or other layered protocols. PT values to support popular parallel programming APIs such as MPI have been defined. The PktWay Enumeration document defines several values for this PT field. The PT field value of "RRP" indicates that message contains commands used in the PktWay Router-to-Router Protocol (RRP). Some PTs will also use the 2 byte Type Extension (TE) field which precedes the PT for passing PT-specific parameters, such as implementation specific de-multiplexing information. RRP messages (as described in the PktWay RRP document) use the TE field to distinguish among the various RRP-messages. Special Packet Types RRP - PktWay's Router/Router protocol (see the RRP document). ERR - Error reporting packet, usually sent to the Source Address (SA, see below) in response to a PktWay message that could not be properly handled, such as "Destination Unknown." The TE indicates the nature of the error (e.g., UNK) as defined in the PktWay Enumeration document. PktWay-WG <17> EEP-spec Endianness (E), 4 bits If the SAN interface of the receiving-node detects Endianness that is different than its own and if the entire Data Block (DB) consists of N-byte fields, then it may activate byte-swapping hardware for N-byte fields, saving much work for the receiving node. The first bit (MSbit) of E, 'e' indicates whether the DB is in Big-Endian order (e=0) or in Little-Endian order (e=1). The next 3 bits could control hardware byte swapping, if any, which assumes that all the data consists of words of the same length. The meaning associated with the values of the 3 LSbits of this field are defined in the PktWay enumeration document. Pad Length (PL) unsigned integer, 3 bits The number of padding bytes that were added at the end of the DB (i.e., from the end of the data to the end of the DB). PL can be between 0 and 7. Data Length (DL) unsigned integer, 25 bits Length, in 8B-words, of the data block, not including the L2RHs, EEP-header, OH, OT, and TAIL, including any optional padding. Hence, the net length of the Data Block is 8*DL-PL bytes. The minimum is zero, and the maximum length is (2^25-1)*8 bytes = ~2^28 = 256 MBytes. Optional Header-Field Flag (h) 1 bit This bit is set to 1 if there are one (or more) optional header (OH) fields following the standard 16-byte EEP-header. Reserved (RZ) 7 bits This field is reserved for future use. Applications should neither use it, nor count on others not to use it. It should be always set to zero (0b0000000). Source Address (SA) 24 bit This field contains the physical address of the packet's original source in the same format as the DT. However, unlike the DT, the SA must be a physical address. Filling in this field is optional. A value of zero means that the SA is not specified. Routers may use this field to identify the sender to which error messages may be returned. PktWay-WG <18> EEP-spec [3]: Optional Header Fields (OH) +-------------------------------- A PktWay-message has Optional Header fields (OH) following the EEP-header, if the Option-Flag (h) is set to 1 in the EEP-header. Each OH is in the format: +--------+--------+--------+--------+--------+--------+--------+--------+ |TCtttttt|LLLLLLLL| data |........|........|........|........|........| OH +--------+--------+--------+--------+--------+--------+--------+--------+ The first byte indicates the optional header field type (OH-TYPE). The first bit, T, of the first byte indicates the processing of this OH-TYPE: T=0: Optional (may drop this field if this OH-TYPE is unknown) T=1: Mandatory (should not process this message if this OH-TYPE is unknown) The second bit, C, of the first byte indicates whether there are more header fields (i.e., whether this is the last field of this message). C=0: More Optional Header fields follow C=1: End of Optional Header fields group (i.e., this is the last OH) The other 6 bits of this byte, tttttt, define application-specific OH-TYPEs. The second byte is the byte-count (L) of the data for this field that starts in the next byte, and is padded with as many padding bytes as needed to fill 8B-words. The length (L) does not include itself, hence it is between 0 and 255. The total number of 8B-words in the symbol L2RH is [(L+9)/8] where the square brackets indicate the integer part, rounded down, of the quantity within. Therefore, the number of padding bytes is PL=8*[(L+9)/8]-2-L. Example: An Optional Header Field (OH) with a mandatory OH-TYPE and 4 data bytes: L=4 #1 #2 #3 #4 padding padding +--------+--------+--------+--------+--------+--------+--------+--------+ |1xtttttt|00000100| data01 | data02 | data03 | data04 | xxx | xxx | OH +--------+--------+--------+--------+--------+--------+--------+--------+ |<------------- value ------------->| PktWay-WG <19> EEP-spec [4]: Optional Data Block (DB) +---------------------------- The DB is free for applications to use in any way. Routers must not modify this field. The DB has DL 8B-words, including optional padding (at the end) of PL bytes. Hence, the number of data bytes is 8*DL-PL. Both DL and PL are specified in the EEP-header. The maximum length of the DB is 8*(2^25-1)B = ~256 MByte. [5]: Optional Trailer Fields (OT) +--------------------------------- A PktWay-message has Optional Trailer fields (OT) if so indicated in an Optional Header field, e.g., an OH field may indicate that a CRC64 is in the OT. An OT may have just the data for an OH defined above (following the EEP header), or be a stand alone, self-defined field in the same format as OH. The OT-fields are in the order defined by the OHs. For example, if an OH-field indicating that a CRC32 is in the OT, is followed by another OH-fields indicating that a CRC64 is in the OT, then the OT with the CRC32 should be followed by the OT with the CRC64. Self defined OT fields must follow OTs defined by the OHs. [6]: EEP Trailer (TAIL) +----------------------- The TAIL consists of only the Error Indication (EI) field which is a single 8B-word. Routers may start forwarding packets toward their destinations before detecting transmission errors (such as in wormhole routing). The EI field provides such routers with a means to append an error indication to the end of a packet. An all zero EI value means that no error was indicated. Any non-zero EI value indicates one or more errors. The packet source will usually initialize the EI field to all zeros. However, as an alternative example, a memory board may create a packet with a non zero EI field (EI=1) that indicates that a parity error was detected by the memory board. Each router does an arithmetic left shift, on the EI field by one bit unless its MSbit is 1. Routers that detect transmission errors also set the LSbit (after the shift) to 1. This provides the ability to identify which routers have indicated errors (if the route is known). PktWay-WG <20> EEP-spec Appendix-A: A Recommendation for PktWay Address Assignment ---------------------------------------------------------- This section of the EEP document is a recommendation only, and not a part of the PktWay standard. Unlike IP addresses, physical PktWay addresses are not globally unique, but must be locally unique within each PktWay configuration. Hence, when SANs that were developed independently are interconnected to form a PktWay, conflicting physical addresses may occur. It is recommended not to attempt to assure local uniqueness of physical addresses by subdividing the global address space (hence, attempting to achieve global uniqueness). Instead, it is recommended that every SAN would have local PktWay addresses, between 1 and the number of its local nodes, and also have a global "bias" to be added to all the addresses in that SAN. Hence, by proper setting of the biases of interconnected SANs, the local uniqueness of PktWay addresses is achieved. The coordination of these biases is left (at least now) for manual (static) out-of-band coordination. The use of such biases simplifies the mapping of physical addresses to their SANs. PktWay-WG <21> Glossary Appendix-B: Glossary --------------------------- (Last update: Aug-24, 1997) Address: A unique designation of a node (actually an interface to that node) or a SAN. Buddy-HR: HRs are "buddies" if they are on the same SAN. Cut-Thru: See wormhole. Destination: The node to which a packet is intended Dynamic-Routing: Routing according to dynamic information (i.e., acquired at run time, rather than pre-set). Endianness: The property of being Big-Endian or Little-Endian (transmission order, etc.) Ethertype: A 16-bit value designating the type of Level-3 packets carried by a Level-2 communication system. HR: Half-Router, the part of a router that handles one network only. L2-Forwarding: Forwarding based on Level-2 (i.e., data-link layer of the ISORM) information, e.g., the native technique of each SAN or LAN. Also called "source routing." L3-Forwarding: Forwarding based on end-to-end Level-3 (i.e., network layer of the ISORM) addresses. Also called "destination routing." Map: The topology of a network. Mapper: A node on a SAN/LAN that has the map and an RT for that network. It is expected that the mapper dynamically updates the map and the RT. Multi-homed Node: A node with more than one network interface, where each interface has another address. Node: Whatever can send and receive packets (e.g., a computer, an MPP, a software process, etc.) Node structure: A C-struct (or equivalent) containing values for some attributes of a node. Planned Transfer: Transfer of information, occurs after an initial phase in which the sender decides which Level-2 route to use for that transfer. PktWay-WG <22> Glossary RCVF: The "Received From" set includes all the physical addresses through which an RT was disseminated, starting with that of the mapper that created that RT. Re-direct-message: A message that tells nodes which HR should be used in order to get to a certain remote address (or range of). Router: The inter-SAN communication device Security Context: A relationship between 2 (or more) nodes that defines how the nodes utilize security services to communicate securely. Source: The node that created a packet. Source-Route: A Level-2 route that is chosen for a packet by its source. Symbol: Data preceeding the EEP header of a PktWay message, interleaving with the L2RHs. Twin-HR: Two HRs are twins if they both are parts of the same inter-SAN router. Wormhole-routing: (aka cut-thru routing) forwarding packets out of switches as soon as possible, without storing that entire packet in the switch (unlike Stop-and-forward). Zero-copy TCP: A TCP system that copies data directly between the user area and the network device, bypassing OS copies. PktWay-WG <23> Acronyms Appendix-C: Acronyms and Abbreviations -------------------------------------- (Last update: August 24, 1997) 0bNNNN The binary number NNNN (e.g., 0b0100 is 4-decimal) 0xNNNN The hexadecimal number NNNN (e.g., 0x0100 is 256-decimal) 8B 8 byte (64 bits) entity ADDR The Address-record of RRP API Application/Program Interface AT Address Type ATM Asynchronous Transmission Mode B Byte (e.g., 4B) b bit (e.g., 32b) BC Byte Count (of parameters) BER Bit Error Rate CAPA The CAPAbility-record of RRP CC Capability Code CSR Common Source-Route DA Destination Address DB Data Block DL Data Length (in 8B words) DSP Digital Signal Processor DT Destination-Type e The MSbit of E E The Endianness field (in the EEP header) EEP End/End Protocol EI Error Indication GP General Purpose GVL2 An RRP message, requesting L2 route to a given destination GVRT An RRP message asking an HR to give its routing tables h Optional header fields flag HR Half Router HRTO An RRP message asking which HR to use for a given destination ID Identification IGMP Internet Group Management Protocol INFO An RRP message providing information about nodes IP The Internet protocol ISORM The ISO Reference Model L Length field (exclusive of itself) L2 Level-2 of the ISORM (Link) L2RH Level-2 Routing Header L2SR Source Route L3 Level-3 of the ISORM (Network) LA Logical Address LADR The Logical-addresses-record of RRP LAN Local Area Network LRT Local Routing Table LSbit Least Significant bit LSbyte Least Significant byte MAC Message Authentication Code / Media Access Control MPI Message Passing Interface MPP Massively Parallel Processing system MSbit Most Significant bit PktWay-WG <24> Acronyms MSbyte Most Significant byte MSU Mississippi State University MTU Maximum Transmission Unit MTUR The MTU-record of RRP M/C Multicast NAME The name-record of RRP NFS Network File Server OH Optional Header field OH-TYPE The Type of an Optional Header field OT Optional Trailer field P The Priority field PAD Padding After Data PBD Padding Before Data PCI The Peripheral Component Interconnect "standard" PH PacketWay Header PL Padding Length (always in bytes) PPP The Point-to-Point Protocol PROM Programmable ROM (Read-Only-Memory) PT Packet Type (2B) PVM Parallel Virtual Machine PW The Myrinet Packet Type assigned to PktWay (PW=0x0300) Q Quality (of a path) RCVF Received-From list, or the Received-From record of RRP RDRC A re-direct message of RRP RH Routing Header RID Record ID RL Record Length (in 8B-words) RRP Router/Router Protocol RT-hd RT (Routing Table) header RT Routing Table RTBL An RRP message proving a Routing Table RTHD The Routing-Table-Header record of RRP RTyp RRP's Record Type RZ The Reserved field (in the EEP header) SA Source Address SAN System Area Network SAN-ID The 24-bit PktWay-address of a SAN SAR Segmentation and Reassembly SN Serial Number SNID SAN-ID SNMP Simple Network Management Protocol SR Source Route (always at Level-2) SRQR The Source-Route-and-Q-record of RRP ST Symbol Type TAIL PacketWay EEP Trailer TE Type Extension (2B) TELL An RRP message requesting information about nodes partially specified UNK Unknown V Version WRU? An RRP message asking its recipient to identify itself XRT External Routing Table xxx A padding byte [end]