NetNews Usenet Archive 1992 #16/NN_1992_16.iso/spool/comp/protocol/nfs/1962

Internet Message Format | 1992-07-27 | 4.6 KB

Path: sparky!uunet!olivea!sgigate!odin!sgi!rhyolite!vjs
From: [e-mail] (Vernon Schryver)
Newsgroups: comp.protocols.nfs
Subject: Re: NFS I/O Ops/seconds
Message-ID: <[e-mail]>
Date: 27 Jul 92 23:20:52 GMT
References: <[e-mail]> <[e-mail]> <[e-mail]>
Organization: Silicon Graphics, Inc.  Mountain View, CA
Lines: 85

In article <[e-mail]>, [e-mail] (Jim Reid) writes:
> Well, considering that I plucked the number out of thin air, this is
> probably not surprising. Maybe your implementation takes more (or
> less) instructions. However, it should take around 1000 instructions
> or so for the typical RISC chip to take an NFS request off the network
> interface hand it off to the NFS code (or push a reply back out). Even
> if the number was twice that amount, it would still be dwarfed by the
> CPU cycles needed to actually *do* the NFS request. And if there was a
> disk I/O (~10ms), a factor of 100 on the number of instructions, could
> be included without much trouble.

Why should it take vastly more than 1000 cycles for the server to
handle a read or write request?  It is easier to look through the disk
cache, allocate a buffer, add it to the controller's queue, take the
interrupt, call iodone(), and so on than it is to do the equivalent for
an ethernet or FDDI packet.  Disks are dirt simple and easy.

You can't count queue-wait-time as the system runs the null jub,
because the server should be expected to have more than one spindle on
which to spend CPU cycles.  And there are all of those lookup's and
getattr's on which to spend CPU cycles.

I don't know how many cycles the "typical RISC chip" requires to
"actually *do" the NFS request," but people here have spent some little
time discovering how many are required on R3K's.

>    If
>     -your system uses 4KByte pages,
>     -if your NFS implementation happens to use 4K blocks,
>     -if your file system happens to like to use 4K buffers,
>    and  -if your FDDI hardware handles UDP checksums,
>    then you can avoid around 1024 cache-missing loads and stores by using
>    "page flipping".  That is the equivalent of 10,000 to 100,000
>    instructions per packet.  (The cache miss penalty for an R4K at
>    50/100MHz is awesome.)
>
> True. Can the SGI kernel really be so dumb as to take loads of cache
> misses when computing UDP checksums? If so, we all know what machines
> don't make good NFS servers...

Are you a student?  Do you understand how CPU caches work?

It is simply impossible to avoid cache misses when computing the UDP
checksum on an incoming packet.  Remember that by definition the
contents of the packet are not in the cache; they just came off the
wire or the disk.  If they are in your cache, you either have an I/O-
coherent cache (e.g. all current Silicon Graphics multiprocsors) or you
have an incredibly stupid, Intel-386 style network or disk driver.

In some systems you can bypass the caches.  Today that is generally a
mistake because a non-cached access costs the same as a cached access
and does not get the advantage of cache-burst-fill which makes some
subsequent accesses cache-hits.  On the other hand, by bypassing the
cache and paying the extra cycles, you do avoid trashing the data
cache.  Read a recent, nice summary of this point as it applies to
bcopy() and bzero() in comp.arch.

Then there is the problem of multi-level caches.  Some current SGI
machines have a MByte or more  of secondary cache, but only a few K of
on-chip cache.  There is simply no way you can avoid primary cache
misses with a small on-chip cache.  And the same considerations for
burst-mode apply to using non-cached accesses.

SGI's raw TCP or UDP FDDI speed is not bad, in part because the network
hardware can do the UDP checksum.  If you also avoid the bcopy(),
the data never sees the CPU and so cannot cause any cache misses.

>    I also don't think saving 1000 instructions is anything to sneeze at.
>    Saving 1000 instructions would produce a large speed up in SGI NFS
>    server.  (We don't yet page-flip.)  Also, remember that 1000
>    instructions is not necessarly even close to 30 usec even at 33 MHz,
>    unless your primary instruction cache is very large.
>
> Hmmm. How big are your instructions? Let's say they are 64 bits long
> (or wide). 1000 of them will fit in 8K. This is not large for a cache
> these days.

I don't know of a current system with instructions generally 64 bits wide.
The current crop of "64-bit" systems use 32-bit wide instructions.

8K is not too small for an on-chip cache.
100MHz is fast for an off-chip cache.


Vernon Schryver,  [e-mail]