Columbia Kermit/kermit.zip/protocol/streaming.msg

Text File | 2020-01-01 | 49KB | 1,158 lines

13-Nov-97 13:22:37-GMT,1089;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id IAA24115;
    Thu, 13 Nov 1997 08:22:36 -0500 (EST)
Date: Thu, 13 Nov 97 8:22:36 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: some strange results
Cc: [e-mail], [e-mail]
Message-ID: <[e-mail]>

watsun to powerpc2    streaming    349525 cps
            non-streaming    524288 cps

powerpc2 to watsun    streaming    524288 cps
            non-streaming    262144 cps

This will have to be looked at with the Sniffer to determine what is
going on.  Although I think it probably has something to do with the
Delayed Acks.

---

powerpc2 is AIX 4.1 running on a 150MHz Personal Power system.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


13-Nov-97 14:06:37-GMT,1185;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id JAA02794;
    Thu, 13 Nov 1997 09:06:36 -0500 (EST)
Date: Thu, 13 Nov 97 9:06:36 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: AIX ckcnet.h
Cc: [e-mail]
Message-ID: <[e-mail]>

Do we have a #define for AIX?

the include files <sys/xti.h> and <sys/tiuser.h> both include defines
for:

#define TCP_NODELAY     0x1
#define TCP_MAXSEG      0x2
#define TCP_KEEPALIVE   0x8

We either want to conditionally include one of the files or just copy
these #defines into ckcnet.h conditionally for AIX.

BTW, ftp transfers between watsun and powerpc2 are:

watsun to powerpc2    binary        >1000cps
            ascii          792cps

powerpc2 to watsun    binary          942cps
            ascii          738cps


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


13-Nov-97 16:12:55-GMT,3354;000000000015
Return-Path: <[e-mail]>
Received: from GRUMPY.USU.EDU (grumpy.usu.edu [129.123.1.86])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id LAA29915;
    Thu, 13 Nov 1997 11:12:54 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 09:12:42 -0600 (MDT)
Date: Thu, 13 Nov 1997 09:12:42 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: fast stuff
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank and Jeff,
    Thinking about the streaming stuff overnight...
    I suspect this is all way off base so far, including that ZD
article (especially that confused piece of writing). Let me expand a
little.
    Kermit ACKs piling up in a reverse channel send buffer do not
cause TCP ACKs to occur less frequently than if no Kermit ACKs were
present. If anything TCP ACKs will occur more frequently to flush a
full MSS to the file sender.
    A 200ms delayed ACK queue can work a couple of ways. One is
to accumulate ACK needs and then five times per second send an ACK.
That is what your MS statements suggest. Well, suppose that's the
case. Then the file sender could, at max, send a whole TCP window
five times per second, 5 x 32KB = 150KB/sec, lots lots less for
smaller windows such as in MS products.
    The other way is to queue one ACK and when another is queued
see that a queue is outstanding and do a formal transmission. Thus
guys standing in line cause the queue to be flushed. This is the
ACK every other packet scheme in most large TCP stacks. There is no
limit on the transmission throughput in this case. Every arriving
segment with data generates an ACK need, every second generates an
ACK transmission. The very first and very last segments *may* experience
the 200ms delay, as well as first segments after serious congestion
recovery. The TCP window is normally larger than two TCP segments, and
thus the two machines go into full max rate streaming.
    There is one more TCP effect I haven't mentioned yet. Window
opening. If the receiver's TCP window closes then in big stacks it
is reported as still closed until it is half empty. Hysterisis. In
lieu of other requests a window opening transmission will be sent by
the receiver when the window suddenly is declared available again.
Clearly, the file transmitter is blocked by a closed window, though it
can probe if it gets anxious (MSK does). MSK's TCP receiver does not hold
down window reports that strongly.
    What I think is happening is CK is committing too many resources
trying to read Kermit ACKs. This is not a TCP problem at all.
    One can easly see if the file receiver is using ACK plan A or
plan B by watching the packet trace timing. One can easily see closed
window blockages the same way. One can also see if TCP is blocked in any
way rather than CK spinning its wheels, as I think it is.
    That ZD article has matters all confused.
    Note, older SUN TCP stacks have a bug in window handling such that
they drive themselves into a corner and nibble at the data stream. I have
a paper on the matter, and SUN has compendium patches which cure it.
    Joe D.

13-Nov-97 16:45:15-GMT,1496;000000000005
Return-Path: <[e-mail]>
Received: from GRUMPY.USU.EDU (grumpy.usu.edu [129.123.1.86])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id LAA11536;
    Thu, 13 Nov 1997 11:45:13 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 09:44:31 -0600 (MDT)
Date: Thu, 13 Nov 1997 09:44:30 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: fast stuff
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank,
    I still suspect it's all in Kermit and the o/s scheduler. Sweet
spots and all that jazz.
    Packet traces will resolve TCP heuristic issues, presuming the
traces reveal timing to millisec resolution. Again, FTP runs without
high level ACKs and thus generates TCP ACKs *less often* than a scheme
which sends back info. This will be plain in the traces. Getting caught
looking for, and/or dealing with, returned information is the most likely
difficulty in this problem.
    Test the returned info situation with streaming CK. Read the bytes
and discard them at as low a level as you can manage. Ditto, but decimate
the look rate. See if the read routine leads to blockages or context
switches that steal time. Try again with Kermit ACKs padded to one MSS.
    Joe D.

13-Nov-97 16:57:33-GMT,2004;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id LAA13603;
    Thu, 13 Nov 1997 11:57:33 -0500 (EST)
Date: Thu, 13 Nov 97 11:57:32 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: Frank da Cruz <[e-mail]>
Cc: Joe Doupnik <[e-mail]>, [e-mail]
Subject: Re: fast stuff
In-Reply-To: Your message of Thu, 13 Nov 97 11:27:45 EST
Message-ID: <[e-mail]>

I threw the Ziff Davis article in because I found it.  It really has
nothing to do with what I have observed with the Sniffer.

Joe, as you say the Delayed Ack can occur on the very last segment.

Here is the scenario that I have seen.  I don't have access to the
Sniffer until Monday and plan to do much more serious documentation of
the behavior at that time.

The TCP Window from the Sender is full or close to it.

The Receiver sends a Kermit ACK (small segment triggering Nagle) with
a TCP ACK piggybacked with a Window size smaller than one half the MSS.
The Sender stops sending data.

The Receiver does not ACK the last received segment and does not send
any more Kermit ACKs because of Nagle and therefore waits to timeout
200ms.

The Sender gets the TCP ACK containing a Kermit ACK and the larger
Window size but doesn't send anything yet.  It waits 200 ms and then
TCP ACKs the last ACK and the data starts to flow again.

---

Again this is not really a problem with the RTT is close to the order
of the delay time but when we are talking orders of magnitude between
the delay time and the RTT we hit a serious bottleneck.

I will send something much more precise next week.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


13-Nov-97 17:03:17-GMT,1727;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id MAA14630;
    Thu, 13 Nov 1997 12:01:17 -0500 (EST)
Date: Thu, 13 Nov 97 12:01:16 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: Joe Doupnik <[e-mail]>
Cc: [e-mail], [e-mail]
Subject: Re: fast stuff
In-Reply-To: Your message of Thu, 13 Nov 1997 09:44:30 -0600 (MDT)
Message-ID: <[e-mail]>

> Frank,
>     I still suspect it's all in Kermit and the o/s scheduler. Sweet
> spots and all that jazz.
>     Packet traces will resolve TCP heuristic issues, presuming the
> traces reveal timing to millisec resolution. Again, FTP runs without
> high level ACKs and thus generates TCP ACKs *less often* than a scheme
> which sends back info. This will be plain in the traces. Getting caught
> looking for, and/or dealing with, returned information is the most likely
> difficulty in this problem.
>     Test the returned info situation with streaming CK. Read the bytes
> and discard them at as low a level as you can manage. Ditto, but decimate
> the look rate. See if the read routine leads to blockages or context
> switches that steal time. Try again with Kermit ACKs padded to one MSS.
>     Joe D.
>

Even so, an ACKless protocol will always be faster than an ACKing
protocol on Ethernet because Ethernet is half duplex.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


13-Nov-97 17:26:11-GMT,3914;000000000005
Return-Path: <[e-mail]>
Received: from barney.usu.edu (barney.usu.edu [129.123.1.89])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id MAA19289
    for <[e-mail]>; Thu, 13 Nov 1997 12:26:10 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 10:25:10 -0600 (MDT)
Date: Thu, 13 Nov 1997 10:25:08 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: fast stuff
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

>Even so, an ACKless protocol will always be faster than an ACKing
>protocol on Ethernet because Ethernet is half duplex.

    Jeff, may I politely suggest "rubbish"? You guys are not
hitting the limits of the wire. That means you aren't pushing 1MB/sec
of user (Kermit) data. Kermit ACK packets are tiny and use very little
wire capacity.
    I have pushed the wire to the limits, with (IPX) ACKs. And I've
done it at both 10Mbps and 100Mbps Ethernets, and the user data rates
involved are 1MB/sec and 10+MB/sec.

    On the other message.

>The TCP Window from the Sender is full or close to it.

    Actually you don't know this. The sender's transmit buffer
is not advertised on the wire. Only receive windows are advertised.

>The Receiver sends a Kermit ACK (small segment triggering Nagle) with
>a TCP ACK piggybacked with a Window size smaller than one half the MSS.
>The Sender stops sending data.

    The file sender is in Kermit sliding windows mode. It presumably
has more Kermit data to send. If it stops it either stopped itself at
Kermit level for Kermit reasons or it found that the TCP stack declined
to accept further data or the TCP stack was blocked from transmitting
(and Nagle's algorithm has long since faded from the scene, forget it).
    The file sender stopped for a reason. It can do so if the receiver
says its window is closed. That window size is in every TCP segment, so
it's visible in the sniffer. If the window did close then the receiver
is falling way behind at the application level. Falling behing reading
is part of what I suggested is really wrong; CK gets stuck with reading,
perhaps because of o/s side effects.

>The Receiver does not ACK the last received segment and does not send
>any more Kermit ACKs because of Nagle and therefore waits to timeout
>200ms.
    TCP itself ACKs TCP data, always. Every TCP transmission will
carry as much user data as the other side's window, moderated by
congestion avoidance, will allow. Nagle's algorithm does not reduce
this at all. That arriving file data will cause those pent-up Kermit
ACKs to be transmitted; this has nothing at all to do with Nagle.
If the Kermit transmitter + its TCP stack decides to stop sending
for its own reasons then things get sluggish.
    Now to your sentence above.
    So a last segment arrives and its TCP ACK is put on the 200ms
timer queue. So what? There are many other previous segments which have
been ACK'd, and Kermit ACKs going out with them. All the sentence refers
to is the very trailing end of the sliding window material, and the rest
of the window has been cleaned up and should be raring to send more data.

>The Sender gets the TCP ACK containing a Kermit ACK and the larger
>Window size but doesn't send anything yet.  It waits 200 ms and then
>TCP ACKs the last ACK and the data starts to flow again.

    So why is the transmitter sitting idle for 200ms? There must
be a reason, such as the transmitter does not have control of the
machine (scheduler, app busy doing something else, bug in the software
such that it has not continued to fill the Kermit sliding window stream,
receiver's window is closed and won't say open until that end catches up).
    Joe D.

13-Nov-97 18:23:03-GMT,2189;000000000005
Return-Path: <[e-mail]>
Received: from barney.usu.edu (barney.usu.edu [129.123.1.89])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id NAA02406
    for <[e-mail]>; Thu, 13 Nov 1997 13:23:02 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 11:22:27 -0600 (MDT)
Date: Thu, 13 Nov 1997 11:22:26 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: fast stuff
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

>I will send you summations of the Sniffer logs when I get them back.

    Rog. That should help.

>The reason the sender stops sending is because the TCP receive window
>is closed.  The Kermit Window is not.

    Now we are getting somewhere. File receiver falls well behind
and tries to catch up. When it does drain most/much/half of its TCP
receive buffer a window opening packet should be emitted. Perhaps it
is not being emitted, leading to the 200ms deadlock.
    Here's the MSK piece of code on the matter (translation below it):

            /* send a window update if crossing a boundry */
    if (s->rdatalen < (TCP_RBUFSIZE / 2) &&
        (s->rdatalen + x) >= TCP_RBUFSIZE / 2)
            tcp_send(s);        /* send opening update */

    Send absolutely (now) a window opening message if the TCP receive
buffer drains across the half-full watermark.
    If a brand X stack put this message on a 200ms wait queue it
should show up in the sniffer logs and delay matters by 200ms. Such
a delay would be a design error.

>C-Kermit does not just blast data, it checks for socket write
>availability with select() before attempting any write.  Same for reads.
>
>The sniffer logs show that this behavior never occurs to FTP streams.
>FTP streams are continuous with one TCP ACK for every four TCP Data
>Segments.

    Ok, so if Kermit reading causes delays then we are close to
the observations.
    I'll wait for the sniffer logs and spare us more speculation
this morning.
    Thanks,
    Joe D.

13-Nov-97 21:28:06-GMT,815;000000000005
Return-Path: <jrd>
Received: (from jrd@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id QAA11625;
    Thu, 13 Nov 1997 16:28:05 -0500 (EST)
Date: Thu, 13 Nov 97 16:28:05 EST
From: "Joe R. Doupnik" <[e-mail]>
To: [e-mail], [e-mail]
Cc: [e-mail]
Subject: Solaris TCP window bugs
Message-ID: <[e-mail]>

Frank and Jeff,
    In ~jrd on watsun is file probing.TCP.ps.Z, the paper by Doug
Comer on shortcomings of Solaris regarding zero TCP windows. It's 10
pages of paper. Sun has subsequently fixed their TCP stack to overcome
this defect (has been observed here in spades with a fellow's Sun
speaking eventually over a phone line to a Win95 machine at home). One
needs to obtain the Sun patch.
    Joe D.

14-Nov-97  2:35:48-GMT,2713;000000000005
Return-Path: <[e-mail]>
Received: from SLEEPY.USU.EDU (sleepy.usu.edu [129.123.1.85])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA02168;
    Thu, 13 Nov 1997 21:35:47 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 19:35:33 -0600 (MDT)
Date: Thu, 13 Nov 1997 19:35:33 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Misc ramblings on streaming tests
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: [e-mail]
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank and Jeff,
    A few observations from here on the streaming stuff.

    My normal Kermit setup, CK on netlab1 to CK on watsun: 25KB/sec.
    Ftp between the same places: 100KB/sec.
    Look at Kermit settings and see 2KB x 4. That's 8KB of windowing
whereas netlab1's TCP stack has 32KB buffers, a 1:4 dilution by Kermit.
    MSK to watsun gives the same transfer rates, and MSK has only
4KB TCP buffers. Again, the Kermit protocol side is the throttle so the
TCP buffer size doesn't make all that much difference in this test.

    Check assumptions later in the day, USU to CU again.
    CKermit 1.5KB x 20 on both ends: 120KB/sec
    Quick ftp: 180KB/sec. Given CK's formatted screen etc that is close.
    Conclusion: on long distances we starve in the Kermit buffer arena
compared to the underlying TCP transport channel. Not much new about that,
but the scaling above does work out. Good old pipe filling.

    MSK check. Bypass decoding and writing to disk. No significant
change when disk is a RAM drive, some change if using a real hard drive.
So MSK's decoder is faster than a SCSI Seagate Barracuda under DOS.

    CK check. Create a Unix RAM drive (memfs). No change for local
transfers. Conclusion: I dunno.

    FTP check. Ftp to adjacent NW file server. 500-700KB/sec either
way. About right when that server is only a 486-33.

    I Had a quick look at the wire while within Windows (ugh), and saw
CK send to MSK with pauses of 60-120 millisec periodically. MSK does the same
when it sends the file, and that is to read disk, construct Kermit packets,
etc. It turns out those pauses are large consumers of time. I won't be
able to repeat this until the weekend when I can steal my student lab to
run clean DOS machines and the wire snoop independently of the file xfer.
    The wermit involve here was the regular test item, not the streaming
edition. To beat FTP on long distance is no problem: just use enough buffering
to fill the pipe. To beat FTP locally don't bother.
        Joe D.

14-Nov-97  2:47:28-GMT,1027;000000000005
Return-Path: <[e-mail]>
Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA03273;
    Thu, 13 Nov 1997 21:47:27 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 19:46:24 -0600 (MDT)
Date: Thu, 13 Nov 1997 19:46:24 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Rambling, part two
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: [e-mail]
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank and Jeff,
    The last of tonight's "putting the numbers in" experiments.

    MSK to watsun. 2KB x 4: 22KB/sec.
            2KB x 20: 45KB/sec.

    Modify MSK to use a 31KB TCP buffer, repeat MSK to watsun.
        2KB x 20: 125KB/sec

    So the large TCP buffer let MSK fill the pipe and that was that.
        Joe D.

14-Nov-97  4:17:30-GMT,2048;000000000005
Return-Path: <[e-mail]>
Received: from barney.usu.edu (barney.usu.edu [129.123.1.89])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id XAA13435;
    Thu, 13 Nov 1997 23:17:29 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 13 Nov 1997 21:17:24 -0600 (MDT)
Date: Thu, 13 Nov 1997 21:17:24 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Streaming, last comment for tonight
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: [e-mail]
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank and Jeff,
    Upon a little reflection it seems that the way to compete with
FTP on local wiring is to join the club. And that means avoid touching
bytes as much as possible.
    A suggestion is to assure an 8-bit clean channel, use regular
Kermit packets for all but the D packet. The D packet has a normal
Kermit framing but the real data is raw block stuff. To further reduce
touching bytes we need to open a second TCP channel which has no Telnet
processor at each end, ftp-data style. Send all the Kermit packets via
that channel. Use a linear checksum to save time, a la TCP, to double
check byte slippage in parsing frames from the stream.
    Channel closing needs finesse to keep it open over files within
a group, and thus lower session setup overhead. 'B' packet ought to do
nicely as a terminator.
        You've probably thought of this too so I will claim zero originality.
    We claim advantage of: Kermit attributes, character set handling,
and similar bells and whistles, but ftp speed for bulk data.
    The only problem I have with this I can't afford 32KB TCP buffers
very easily, but one can be found to absorb file transfer traffic on a
single session at a time. I also have an administrative mess in the code
dealing with two sessions simultaneously, but I presume there's a way.
    Joe D.

14-Nov-97  5:40:56-GMT,1048;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id AAA25610
    for fdc; Fri, 14 Nov 1997 00:40:56 -0500 (EST)
Date: Fri, 14 Nov 97 0:40:56 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: streaming
Message-ID: <[e-mail]>

something that occurred to me.  There was something ringing in my ear,
something Joe said earlier.  He said that when the TCP Window closes
it won't open again until the recv buffer is half empty.  Well,
C-Kermit on Unix only reads at most 4096 bytes at a time in myread().
But the recvbuf for Sun TCP is 24K.  We need to increase the read
buffer of C-Kermit in ckutio.c.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


14-Nov-97  8:27:36-GMT,972;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id DAA25218
    for fdc; Fri, 14 Nov 1997 03:27:35 -0500 (EST)
Date: Fri, 14 Nov 97 3:27:35 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: more streaming
Message-ID: <[e-mail]>

I increased the MYBUFSIZ in ckutio.c to 32678 and with Streaming on I
consistently transmit data from watsun to powerpc2 at 1mb/sec.

With streaming off the data rate is 0.5mb/sec.

---

you might want to compare the TCP RTO detection algorithm with the one
you wrote for Kermit.  RFC 1122.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


16-Nov-97  2:06:51-GMT,1007;000000000005
Return-Path: <[e-mail]>
Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA01060
    for <[e-mail]>; Sat, 15 Nov 1997 21:06:51 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]> for [e-mail]; Sat,
 15 Nov 1997 19:06:48 -0600 (MDT)
Date: Sat, 15 Nov 1997 19:06:48 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Streaming, a possible aid
To: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: [e-mail]
X-VMS-Cc: JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank,
    On netlab1 I built tcpdump (/opt/bin/tcpdump) as an aid in your
streaming tests. It works best in non-promiscous mode (tcpdump -p ...)
but does run in promiscous mode if desired. For the time being I made
it suid so it could be run by non-root (yes, I know).
    Joe D.

20-Nov-97  5:17:01-GMT,972;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id AAA29623
    for fdc; Thu, 20 Nov 1997 00:17:01 -0500 (EST)
Date: Thu, 20 Nov 97 0:17:00 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: streaming notes
Message-ID: <[e-mail]>

-I flag should set 'streaming' to 1

I thhink Streaming mode should be displayed to the user as a 'Message'
and not as a protocol type.  The results of other protocol
negotiations should be displayed as messages as well.  This will be
important for KUI.

Other than that looks really good.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


20-Nov-97 15:50:54-GMT,1132;000000000011
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA02703
    for fdc; Thu, 20 Nov 1997 10:50:54 -0500 (EST)
Date: Thu, 20 Nov 97 10:50:54 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: streaming
Message-ID: <[e-mail]>

I am using the code from last night.  wermit -Ig ckoco3.c with my
modification to turn streaming on when -I is specificied.

The connection is one which drops out periodicly with very long
delays.  Using autodownload server on K95.

K95 reported the file successfully transfered, however, it really only
transfered 70059 out of 517768 bytes and wermit only received about
66000 bytes.

It is not clear why it failed.

I am going to try to reproduce it.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


20-Nov-97 15:55:18-GMT,1223;000000000001
Return-Path: <fdc>
Received: (from fdc@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA03386;
    Thu, 20 Nov 1997 10:55:16 -0500 (EST)
Date: Thu, 20 Nov 97 10:55:16 EST
From: Frank da Cruz <[e-mail]>
To: [e-mail]
Subject: Re: streaming
In-Reply-To: Your message of Thu, 20 Nov 97 10:50:54 EST
Message-ID: <[e-mail]>

> I am using the code from last night.  wermit -Ig ckoco3.c with my
> modification to turn streaming on when -I is specificied.
>
What modification?  Streaming *is* used when -I is specified, as long as
you haven't changed the default SET STREAMING value, which is AUTO.

> The connection is one which drops out periodicly with very long
> delays.  Using autodownload server on K95.
>
> K95 reported the file successfully transfered, however, it really only
> transfered 70059 out of 517768 bytes and wermit only received about
> 66000 bytes.
>
> It is not clear why it failed.
> I am going to try to reproduce it.
>
Well... packet logs, etc.  It might be just a case of not reporting the
failure on the file-transfer display -- that's where I left off last night.
What does "stat" tell you after such a failure.

- Frank

21-Nov-97  2:49:24-GMT,4361;000000000005
Return-Path: <[e-mail]>
Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA00747;
    Thu, 20 Nov 1997 21:49:23 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 20 Nov 1997 19:49:14 -0600 (MDT)
Date: Thu, 20 Nov 1997 19:49:14 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: Streaming - Draft 3
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Frank,
    Thanks for the most recent streaming draft.
    It still bothers me, as you can guess. Let me sketch my concerns
without getting into debate mode.
    ACK's aren't bad, they cost almost nothing in bandwidth except
for the particular situation of half duplex style connections. Such
might occur with modem technology attempting to use the entire bandwidth
one-way to drain transmit buffers, but I don't think this is actually
true even for the 56Kbps schemes on the market.
    Nagle is a red herring. The scenario is a Kermit puts its small
ACK into the TCP buffer and Nagle's algorithm says wait. Incoming packets
carrying any data, as they do for file transfers, mandate a reply from
the receiver and that reply includes all available data up to TCP limits.
Thus incoming file packets cause transmission of Kermit ACKs.
    Of course this brings up the subject of 200ms delayed ACK queues.
I think, subject to correction from experiments, that only one entry can
remain in the queue, and the second entry causes the queue to be flushed.
If this were not true then FTP data would progress at the rate of 5 ACKs/sec,
and we know there are many more per second than that even in the absence
of data in the ACK. If I am correct then the 200ms delay affects the first
and/or last ACK but not the ones in the middle. Yes, every other ACK may
be delayed, but only until the next packet arrival and hence well within
the window jitter budget.
    Putting these together says the file transfer rolls full blast,
as indeed FTP data does. Kermit ACKs would be free payload fitting easily
within the Ethernet padding budget.
        I suspect error correcting modems also use similar triggered ACKs
as automatic flow control. With those ACKs go queued reverse channel data.
I don't know this to be the case, but it seems likely.
    Underlying some of what you are trying to accomplish is avoiding
retaining many packets in retry buffers, enough to fill the pipe round
trip. I can understand that. Why not let the transport provider supply
those buffers, as TCP does already, is what I hear in the proposal. Given
TCP the buffer sizes are typically 32KB and less, some cases of 64KB
(SGI machines) and a very few cases of larger values. That says Kermit
needs matching retry/out-of-order-reception buffering capacity, and when
it does the system streams at full rate, with ACKs. I demonstrated that
last weekend with MSK to CK on watsun, yielding the same rates as FTP.
Thus providing Kermit buffers is not a big deal. Local links have short
pipes and the Kermit buffer count remains small, even for fat pipes.
Long distance links are most often very skinny indeed and Kermit's bit
capacity is still more than that in the pipe. A middle sized pipe,
USU to CU on a quiet day, appears to be less than 32KB capacity and we
can (I have) fill it with ACK-ful Kermits.
    Interestingly, Kermits can readily have larger packet storage
capacity than an underlying transport provider. 32KB TCP windows are
easily exceeded; modems have much less storage. What's the problem?
    Were I to go about this business I would mimic FTP's behavior of
"raw" data in one channel and control in another, and dispense with
packet framing in the data channel. That itself takes away a measurable
overhead. There is the difficulty, however, of serial comms having only
one channel. But on modems I don't think we have a difficulty with even
ordinary buffers (4 window slot style) and hence no need to give up ACKs
and trust the world.
        So I remain dubious of the need to remove Kermit ACKs. The case
for removal just has not been made as far I am aware.
    Joe D.

21-Nov-97  2:59:19-GMT,1804;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id VAA01522;
    Thu, 20 Nov 1997 21:59:17 -0500 (EST)
Date: Thu, 20 Nov 97 21:59:17 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: Joe Doupnik <[e-mail]>
Cc: [e-mail]
Subject: Re: Streaming - Draft 3
In-Reply-To: Your message of Thu, 20 Nov 1997 19:49:14 -0600 (MDT)
Message-ID: <[e-mail]>

Joe:

Kermit ACKs have almost zero cost on long haul connections, but not
when there are extremely short Round Trip Times and there are hard
coded Delayed ACKs.   Don't test from USU to CU, test from between two
machines on the same segment where one of those machines is running
Microsoft TCP/IP and the other is something other than MSK.

FTP does not result in TCP ACKs for every data segment.  When using
FTP one ACK is sent for every four data segments.

The combination of Nagle and Delayed ACKs can be disasterous.  Look at
some of John Nagle's postings on the subject in recent months.  (I
think I have one, if I can find it I will forward it.)

As soon as I can get the Sniffer back I will send you log summaries.
We are having some serious problems with some equipment from Lucent
that is preventing me from being able to borrow it.

In the meantime, the demonstrative performance gains that both Frank
and I have been able to see by implementing Streaming will have to do
as evidence of its worthiness.

-Jeff


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


21-Nov-97  3:24:01-GMT,2874;000000000005
Return-Path: <[e-mail]>
Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87])
    by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id WAA04289
    for <[e-mail]>; Thu, 20 Nov 1997 22:24:00 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556)
 id <[e-mail]>; Thu, 20 Nov 1997 20:23:57 -0600 (MDT)
Date: Thu, 20 Nov 1997 20:23:57 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: Streaming - Draft 3
To: [e-mail]
Cc: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: [e-mail],JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Jeff,
    Thanks for John's comments. I think I have a copy of them in the
archives (use them later in my networking course). I wish they applied
here but I don't think they do because the file transfer keeps forcing
replies, 1:2 or even 1:4, and isn't send-send-wait unless the receive
window is two segments long (and even then window opening triggers a
reply).
    I'm groping here, but from what I gather on your observations
with the MS Windows TCP/IP stack is it has a small TCP buffer. I should
find a way of looking at it, except I never use that item from MS. And,
to complete the picture, I presume the buffer is smaller than the ACK queue
depth. Is that right? Hence the transmitter sends everything and the ACKs
sit in the receiver's 200ms delay queue and there aren't enough ACKs to
force the queue to flush without a timeout. Right? If wrong please help
me understand the environment. That would put them into the 5 ACKs/sec
situation; burst wait, burst wait, etc.
    If this is about right then enlarging the MS TCP buffer should
turn things around. A guess is the buffer is 4KB, about 2.5 full length
local Ethernet packets, and hence under the 4 and go threshold for ACKs.
Then 8KB should be enough to make it roll. We should be able to measure
this effect from the outside, and maybe you have already.
    Maybe there's yet another MS blunder in the picture: excessive
hold down of window opening ACK. Perhaps they sit on each window opening
announcement until NT 5 is ready. Sorry, couldn't resist. But I am
serious about maybe delaying even that piece of bookkeeping.
    To my way of thinking the TCP stack part of things is testable
by regular FTP, as the slowest of the transfer methods. By slow I mean
there is no reverse channel data to stimulate replies quicker than done
by arriving file data.
    I'll stop here to keep the discussion a discussion, and when your
data become available perhaps I will learn a thing or two. My problem,
you see, is I approach these things rationally, what I would have done
in their position, rather than what MS hacks together and declares to
be revealed truth.
    Thanks,
    Joe D.

21-Nov-97  3:30:37-GMT,1340;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA05284;
    Thu, 20 Nov 1997 22:30:35 -0500 (EST)
Date: Thu, 20 Nov 97 22:30:35 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: Joe Doupnik <[e-mail]>
Cc: [e-mail]
Subject: Re: Streaming - Draft 3
In-Reply-To: Your message of Thu, 20 Nov 1997 20:23:57 -0600 (MDT)
Message-ID: <[e-mail]>

Joe:

I thank you for your patience.

The MS TCP/IP stack defaults to a 8K in, 8K out set of buffers.  I
have tried increasing the window size but it does not help
significantly.  It only puts off the eventual problem.

I will get you the logs as soon as I can.  (Before the next NT 5 beta
is shipped.)

In the meantime, if you have Win95 or NT4, the latest cknker.exe (k95
beta) may be found in watsun in ~kermit/os2test/bean/cknker.exe.
You can use that to test the performance differences between Streaming
and non-streaming on Microsoft TCP/IP.

- Jeff


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


21-Nov-97  3:54:13-GMT,2635;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA08664
    for fdc; Thu, 20 Nov 1997 22:54:13 -0500 (EST)
Resent-Message-Id: <[e-mail]>
Received: from mailrelay1.cc.columbia.edu (mailrelay1.cc.columbia.edu
        [128.59.35.143]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP
        id WAA08193 for <[e-mail]>; Thu, 20 Nov 1997
        22:47:47 -0500 (EST)
Received: from barney.usu.edu (barney.usu.edu [129.123.1.89]) by
        mailrelay1.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id WAA14418 for
        <[e-mail]>; Thu, 20 Nov 1997 22:47:46 -0500 (EST)
Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id
        <[e-mail]> for [e-mail]; Thu, 20 Nov
        1997 20:47:38 -0600 (MDT)
Date: Thu, 20 Nov 1997 20:47:38 -0600 (MDT)
From: Joe Doupnik <[e-mail]>
Subject: Re: Streaming - Draft 3
To: [e-mail]
Message-id: <[e-mail]>
X-VMS-To: IN%"[e-mail]"
X-VMS-Cc: JRD
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT
Resent-To: [e-mail]
Resent-Date: Thu, 20 Nov 97 22:54:12 EST
Resent-From: Jeffrey Altman <[e-mail]>

Jeff,
    I have Win95 and NT 4 so we are in luck on that part. I'll wait
for the packet traces as being more economical of my time rather than
testing product to product for half a day initially. Plus I'm out of spare
machines to act as a packet snoop on the same wire.
    Curiously, in class I'm discussing sliding windows flavors. The
weekend lab project is to use my special packet zappers in IP and Kermit
protocols to view how each deals with losses (go back to N vs selective
repeat, etc). Tomorrow introduces pipe measurements, and packet versus
byte counting, etc for general nameless protocols.
    Oh yes, for Frank. Saturday I may have netlab1 up and down a
bit while I sort out some details in a beta lan driver from Intel. I'm
discussing things with the authors of same. Outages will be to regen
the box, hence a few minutes, and I work around existing connections.
About 2/3 of the tries result in receive but can't send situations,
and then I back away and put in a working configuration for awhile.
    Thanks,
    Joe D.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


21-Nov-97  3:55:45-GMT,1043;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA08789;
    Thu, 20 Nov 1997 22:55:42 -0500 (EST)
Date: Thu, 20 Nov 97 22:55:42 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: Joe Doupnik <[e-mail]>
Subject: Re: Streaming - Draft 3
In-Reply-To: Your message of Thu, 20 Nov 1997 20:47:38 -0600 (MDT)
Cc: [e-mail]
Message-ID: <[e-mail]>

I don't expect you to spend half a day performing traces.

Try a transfer once with streaming and once without just to prove that
the performance difference exists.

SET TCP ? may be used to adjust the send and recv buffers if you
decide you want to play.




    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


21-Nov-97 15:17:29-GMT,768;000000000005
Return-Path: <jaltman>
Received: (from jaltman@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA13853
    for fdc; Fri, 21 Nov 1997 10:17:29 -0500 (EST)
Date: Fri, 21 Nov 97 10:17:28 EST
From: Jeffrey Altman <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: streaming
Message-ID: <[e-mail]>

skipped files (ie, because of date) show up in the Error Count on the
full screen display.


    Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2
                 The Kermit Project * Columbia University
       612 West 115th St #716 * New York, NY * 10025 * [phone]
    http://www.columbia.edu/kermit/k95.html * [e-mail]


21-Nov-97 15:34:54-GMT,3420;000000000005
Return-Path: <fdc>
Received: (from fdc@localhost)
    by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA18425;
    Fri, 21 Nov 1997 10:34:44 -0500 (EST)
Date: Fri, 21 Nov 97 10:34:44 EST
From: Frank da Cruz <[e-mail]>
To: Joe Doupnik <[e-mail]>
Cc: Jeffrey Altman <[e-mail]>
Subject: Streaming
Message-ID: <[e-mail]>

Lively correspondence!  Can't turn my back for minute...  Well, as they say,
the proof is in the putting (FTP pun).  There are two ways to approach this;
bottom-up analysis of many kinds of connections and TCP stacks, or top-down
observations of performance.  Both are fine, but you guys seem to have the
former covered nicely, so I'll concentrate on the latter -- I'm mainly
interested in refining the definition, getting it to work as advertised, and
then doing a lot of measurements.  If I can find situations in which streaming
improves matters, then it will not have been a waste of time.  After all, it
is not the user's fault if the underlying transport is implemented poorly or
behaves badly, and even less their responsibility to understand or fix it;
Kermit has always tried to give them a way to overcome any nastiness below,
and this could be just another way.

Perhaps more interestingly, it might be used with IBM mainframe Kermit to
achieve performance improvements that could not be easily gained any other
way.

In any case, it is another knob to turn, among many.  "Kermit as lab" gets
a new piece of equipment:

declare \&s[2] off on
set file display brief
set control unprefix all
set control prefix 0 1 255
set transfer bell off
if open write close write
open write bench.log
for \%w 1 31 1 {
    set window \%w
    for \%p 480 9000 480 {
        for \%i 1 2 1 {
        set streaming \&s[\%i]
            remote set rec pack \%p
            if fail stop 1 {REMOTE SET FAILED: \&s[\%i] \%w \%p}
            echo S=\&s[\%i] W=\%w P=\%p
        send x
            if fail stop 1 {TRANSFER FAILED: \&s[\%i] \%w \%p}
        writeln file -
\flpad(\v(cps),8) -
\flpad(\&s[\%i],4) -
\flpad(\%w,3) -
\flpad(\%p,4)
        }
    }
}
close write

Running this now, on fully loaded network and host computers, just the first
few passes thru the loop (not surprisingly) show streaming to be way faster
than not streaming (on a TCP connection Kermit-to-Kermit, no Telnet server;
x is a 1MB Sparc executable).  Here is the first bit of the log:

     CPS Stream W  Pkt
   33825  off   1  480
  209715   on   1  480   <- six times faster
   65536  off   1  960
  349525   on   1  960   <- 5.3 times faster
   69905  off   1 1440
  209715   on   1 1440   <- 3 times faster
   58254  off   1 1920
  174762   on   1 1920   etc etc..
   95325  off   1 2400
  349525   on   1 2400
   80659  off   1 2880
  349525   on   1 2880
   87381  off   1 3360
  262144   on   1 3360
   69905  off   1 3840
  349525   on   1 3840
   41943  off   1 4320
  524288   on   1 4320   <- 9.5 times faster!
   55188  off   1 4800
  349525   on   1 4800
   58254  off   1 5280
  209715   on   1 5280
   43690  off   1 5760
  349525   on   1 5760
   52428  off   1 6240
  262144   on   1 6240   <- 9.5 times faster...
   27594  off   1 6720

Obviously we need more controlled tests (e.g. on a dedicated network with
unloaded hosts, on serial connections, etc), as well as sniffer results.

But at least we're having some fun for a change :-)

- Frank