ftp.wwiv.com/ftp.wwiv.com.zip/ftp.wwiv.com/pub/MISC/LN2V100.ZIP/TECHNOTE.TXT

Text File | 1998-08-26 | 19KB | 349 lines

┌───────────────────────────────────────────────────────────────────── ┌┬┐┌┬┬┐┌┐
│░ LNET-II 1.00 TECHNICAL NOTES ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │├┤│├┤│││
└───────────────────────────────────────────────────────────────────── └┴┘└┘└┘└┘

╔════════════════╗ LNET-II is written in Microsoft PDS [Professional Development
║ PROGRAM SOURCE ║ System] v7.1, a high-powered variant of QuickBASIC.  The ANSI
╚════════════════╝ decoder is a heavily modified version of Tom Hanlin's ANSI-
print() routine from the BASWIZ library.  LNET-II's screens were designed in
TheDraw and are displayed using TheDraw's "crunched object" format.

The source code is 13,163 lines in fifteen modules occupying 394.1K [not
counting screen objects].  Development was started on 09/27/1996.  Some core
portions, such as the packet pre-scan and item validation algorithms, were
borrowed from the Dead-Fix v2.30 source code.

╔═════════════════════╗ LNET-II is compiled and linked with nine of its fifteen
║ MEMORY REQUIREMENTS ║ modules in overlays.  While this increases the size of
╚═════════════════════╝ the .EXE file, it also significantly reduces the pro-
gram's conventional memory requirements - the basic conventional RAM requirement
is approximately 269K before the program allocates its data space.  If you have
EMS/XMS, the overlays will be loaded into high RAM and called as needed.  If you
have only 640K RAM, or no EMS/XMS driver loaded, overlays will be swapped in and
out of conventional RAM from disk.  Please note, though, that this will slow the
program down somewhat, so EMS/XMS is highly recommended.

LNET-II reads header data for only one item at a time, so even the largest
packets can be handled with very little memory overhead.  Items' locations
within the packet are stored in a temporary index file, eliminating the need to
index the entire packet in memory.  By creating an index, LNET-II is able to
perform extremely fast access of individual items in a packet.  This does,
however, require some HD space overhead - 15 bytes per item are used to create
the index.  A packet's index file is deleted when the packet is closed.  Index
files are named *.LN2 [i.e. DEAD.NET -> DEAD.LN2].  These files are written in
the packet's network directory.

Occasionally, the text of a message might be too large to display.  The PDS
language limits individual strings to 32767 bytes.  LNET-II will never attempt
to display messages larger than 32767 bytes; however, some processes internal to
LNET-II may overrun the program's allocated string space.  [Unlike C, the PDS
language doesn't have the low-level memory control functions necessary to
dynamically allocate string space, although it can do so for array space.  This
limitation is probably one of the few leftovers from the older versions of MS
BASIC.]  When an item is too large to display, LNET-II recovers from the error
by aborting the text display process.  An error message will indicate that the
item cannot be displayed.

When an item's text is being displayed, LNET-II first closes the packet [which
is opened in BINARY mode], reopens it in INPUT mode, counts the number of lines
in the item's text, reads and displays the item one line at a time [using the
BASIC 'LINE INPUT' function], then closes the packet and reopens it in BINARY
mode.  Although this may sound unnecessarily complicated, it actually makes it
easier to handle the item, because it eliminates the need to read the entire


───────────────────┤ LNET-II v1.00 Technical Notes - Page 1 ├───────────────────



message into memory as one big block [which can EASILY overrun the available
string space].  Reading one line at a time makes it far less likely that the
program's string space will be overrun, but there still may be cases when an
individual line of text is too long to handle.  This is most likely to happen
when trying to view binary data, such as DE* encrypted items, and when viewing
very long lines of ANSI data.

The /LOMEM option reduces LNET-II's memory requirements by causing the program
not to load certain data and by disabling the item text display [which can use
quite a bit of memory on large items].  However, LNET-II may still occasionally
exhibit unusual behavior if the amount of available memory is extremely low.  If
you have limited memory and LNET-II operates improperly even with /LOMEM active,
try to make more memory available, then run the program again.

Because the PDS language limits arrays to no more than 32,767 subscripts [indi-
vidual elements of an array's rows or columns], LNET-II cannot handle networks
with more than 32,767 nodes.  However, this is EXTREMELY unlikely to occur at
any time - as we all know, many nets are shrinking in the face of Internet
competition, and those few nets which are growing are not doing so at a rate
which will ever give them this many systems.  LNET-II will recognize node
numbers up to @65535, but it cannot handle more than 32,767 total nodes in a
network.  In the unlikely event that LNET-II needs to handle a network with more
than 32,767 individual nodes, please contact the author [see page 49 of the user
manual LN2V100.DOC] - although it would be somewhat difficult, a patch could
probably be made to allow up to 65,535 nodes in a network.

╔══════════════╗ The largest packet readable by LNET-II is 2,147,483,647 bytes
║ PACKET SIZES ║ [(2^31)-1].  This is because the PDS language uses SIGNED LONG
╚══════════════╝ values for file access.  Since the smallest possible single
WWIV network item is 24 bytes - the length of the network header structure -
this packet could contain up to 89,478,485 items.  The largest possible valid
non-deleted item is 24 + 8192 + 32767 bytes [40,983 total], so this packet could
contain no less than 52,399 non-deleted items.  [See LN2V100.DOC for item
validation specifications.]  A single deleted item can occupy the entire length
of a packet, regardless of the packet's size; these items are not subject to the
WWIV 32K limit.

╔══════════════╗ LNET-II is compiled as a DOS real-mode application.  At this
║ FILE LOCKING ║ time, it has been successfully tested under MS-DOS 6.22,
╚══════════════╝ Windows 3.11, Windows 95 / MS-DOS 7.00, Novell NetWare Lite,
and OS/2 Warp.  The program includes routines which help make it compatible with
multi-instance BBS systems by locking write access [and, in some cases, read
access] of currently open files.  Locks are released when LNET-II closes files.
The availability of file locking is automatically detected via a call to DOS
interrupt service 2Fh.  The packet currently being read is held open and locked.
The same applies to the packet's temporary index file.  Because of this, the
same packet cannot be opened simultaneously by two instances of LNET-II, and no
other process will be able to read/write the open packet until LNET-II closes it
and releases its locks.





───────────────────┤ LNET-II v1.00 Technical Notes - Page 2 ├───────────────────



Files are never held open when they are not needed.  With the exception of the
currently loaded packet and its index file, most files will be open and locked
only for a fraction of a second while they are being read or written.  LNET-II's
file locking is designed to prevent interference with other processes as much as
possible.

When running LNET-II under multi-tasking operating systems with their own
internal file locking services, it is not necessary to have SHARE.EXE loaded.
SHARE.EXE is usually not needed if your BBS is being run as a single-node system
under DOS; it is used mostly for DOS-based LAN environments.  [Note: If you use
WWIV's /I# parameter when loading the BBS, and you are NOT in a multi-tasking
environment, you will need to have SHARE.EXE loaded.  WWIV will insist on it.]

╔══════════════════╗ During development and beta testing, one of the testers
║ THE /NOHI SWITCH ║ reported that his system was locking up immediately after
╚══════════════════╝ LNET-II had displayed the opening logo.  I examined the
source code for a possible cause, and on a hunch, I added the /NOHI command line
switch to disable all detection and use of high-resolution text modes.  [This
tester had also mentioned something about not being able to use high-resolution
text modes in LNET-II.  He is running Windows 95.]  For some reason, which I am
completely at a loss to explain, the video mode detection process caused the
beta tester's system to lock up after it had displayed the opening logo.  By
disabling the video mode detection process and forcing 25-line displays with the
/NOHI switch, his lockups were eliminated.  Go figure.

LNET-II detects high-resolution text modes by attempting to switch the display
into those modes, then checking to see whether an internal error code is gene-
rated.  If there is no error after a mode is tried, LNET-II considers that mode
to be available.  The video mode detection process may cause a very brief period
of visible display blinking just before the opening CMI Software logo is shown
[after the initial loading messages are printed to the DOS screen].  This is a
normal side-effect of switching from one display mode to another; on a VGA
display, up to five different modes may be accessed.

See the user manual LN2V100.DOC for further details on the /NOHI switch.

╔═════════════════════════════╗ Systems using Quarterdeck's VIDRAM might not be
║ RUNNING LNET-II WITH VIDRAM ║ able to use LNET-II's VGA 30-line and 60-line
╚═════════════════════════════╝ item text displays.  This is because it is
necessary to switch the display into graphics mode to get 30/60-line text, but
VIDRAM takes over the display's graphics memory space to increase EMS/XMS memory
and prevents the screen from switching into graphics modes.  If VIDRAM is being
used, LNET-II probably will not auto-detect the 30 or 60-line modes and will
only allow 25, 43, and/or 50-line text modes.  VIDRAM is most likely to be used
on systems running DESQview.  [LNET-II has not yet been tested under DESQview;
perhaps one of you can do that for me.]

╔═══════════════════╗ LNET-II properly recognizes the full range of possible
║ HIGH NODE NUMBERS ║ WWIV node numbers, from @1 to @65535.  It has been brought
╚═══════════════════╝ to my attention that some packet readers do not properly
display node numbers over @32767, probably because they are treating node num-
bers as signed integers.  LNET-II does not have this problem; all node numbers
are treated as UNSIGNED values when they are being displayed.

───────────────────┤ LNET-II v1.00 Technical Notes - Page 3 ├───────────────────



Networks using my 65-region node numbering system might have problems with high
node numbers when using other packet readers or network utilities.  [E-mail me
for details on my node numbering system.  It is currently in use on DigiNet and
will also be used on USLink.]

╔═════════════════════════════╗ Development of LNET-II was assisted by the
║ WWIV NETWORK TECHNICAL DATA ║ WW4NTECH technical manual for WWIV networking,
╚═════════════════════════════╝ written by Midnight Tree Bandit.  This file is
widely available; Midnight Tree Bandit's system is The Many Titled Board at
804-288-8630.  WW4NTECH contains far more detail than is necessary for this
document - I recommend reading WW4NTECH if you really want to get into the nuts-
and-bolts of WWIV networking and packet layout.

I would also like to thank Wayne Bell, JAFO, Deltigar, and Oderus Urungus for
supplying other information used in the development of LNET-II.

╔═════════════╗ The LNET-II documentation uses various terms to refer to packets
║ TERMINOLOGY ║ and their contents.  Basically, they are:
╚═════════════╝

PACKET: A collection of individual messages, or items, formatted into a single
file.  Some packets may be compressed by NETWORK1.EXE, using a subset of the
PKWARE compression library; however, they are not .ZIP files.  [WW4NTECH con-
tains technical data and source code in C for calling PKWARE library functions
in a WWIV packet handler.]  If a packet is compressed, the first item in the
packet will have TOSYS and TOUSER values of 65534 [or -2 if you are reading the
numbers as signed integers].  If the first four bytes of the packet are FE FF FE
FF in hex, the packet is considered to be compressed.  Compressed packets are
usually named Z*.NET.

ITEM: A single message within a packet.  This includes the item's HEADER, its
DESTINATION LIST [if any], and its TEXT SEGMENT [if any].  Every item must have
a HEADER.  Destination lists are optional on most items and prohibited on a few
items [such as e-mails].  Some items do not need a text segment.

HEADER: 20 bytes of data at the start of every item, indicating what kind of
message it is, where it's from, where it's going, when it was created, how large
it is, and whether it is encrypted.  The header contains the following data:

    TOSYS: The destination system of the message.  This will be zero if the item
    has multiple destinations; in this case, the HEADER is immediately followed
    by the DESTINATION LIST [a series of two-byte integer values representing
    node numbers] and the LISTLEN field contains a count of the number of nodes
    which are to receive copies of the message.

    TOUSER: The WWIV user number of the person who is to receive the message.
    Generally relevant only for e-mails and SSM's.

    FROMSYS: The system which originated the message.  In gated e-mails, this
    will be the number of the node which handled the gating of the mail into the
    current network.



───────────────────┤ LNET-II v1.00 Technical Notes - Page 4 ├───────────────────



    FROMUSER: The WWIV user number of the person who wrote the message.
    Sometimes this is #0 to indicate a message from the net software - or from
    another utility - to the sysop, or to indicate a type 7 e-mail-by-name.

    MAIN TYPE: Each item has a number [currently in the range 1 to 27] indi-
    cating the purpose of the item.   For instance, MAIN TYPE 2 is a standard
    e-mail from one user to another on the current network.  The main type is
    set to 65535 when an item is marked to be deleted from a packet.  See
    LN2V100.DOC for a detailed listing of allocated main and minor types.

    MINOR TYPE: Further identifies the purpose of an item.  For instance, MAIN
    TYPE 3 is a post from a sub's host system to its subscribers, and MINOR TYPE
    12345 would indicate that the post belongs to sub type 12345.  Main and
    minor types are often shown together - for instance, 5/10 would be main type
    5, minor type 10.  Minor types are rarely displayed alone.

    LISTLEN: Indicates how many nodes, if any, are in the item's DESTINATION
    LIST.  When ListLen is non-zero, ToSys is zero, and vice-versa.  Some items
    cannot have multiple destinations; their ListLen values must be zero.

    DATEN: A four-byte value which indicates the date and time of an item's
    origin.  It is calculated in seconds since midnight on 01/01/1970.  The
    daten format originated on UNIX systems.

    LENGTH: A four-byte value indicating the size of an item's TEXT SEGMENT.  It
    does not include the size of the DESTINATION LIST.  A deleted item [main
    type 65535] may occupy the entire length of a packet; non-deleted items are
    currently limited per WWIV network specifications to 32,767 bytes of text.

    METHOD: Some items are encrypted and/or require a call to an external
    program in order to handle them.  EN*.EXE - where '*' represents the method
    number - is used to encode an item, and DE*.EXE is used to decode it.  For
    instance, an item with method 123 would be encoded by EN123.EXE and decoded
    by DE123.EXE.  All items may be Method<>0; some require Method<>0.  The
    encoder is called by NETWORK1.EXE when an item having Method<>0 is being
    analyzed and prepared to go out from the originating system.  The decoder is
    called by NETWORK2.EXE while the LOCAL.NET packet is being unpacked.

DESTINATION LIST: If an item is going to more than one node, the HEADER will be
followed by a series of data - two bytes per node - listing all of the nodes
which are to receive copies of the item.  This is also referred to as "stacking"
an item.  When an item is stacked, TOSYS will be 0 and LISTLEN will indicate the
number of nodes in the destination list.

SUBHEADER: Additional data about certain items, used by third-party network
utilities.  These follow the DESTINATION LIST and are seen by the net software
[and by other programs which do not specifically recognize them] as simply the
first 'x' bytes of the TEXT SEGMENT.






───────────────────┤ LNET-II v1.00 Technical Notes - Page 5 ├───────────────────



TEXT SEGMENT: The actual text of the message itself.  This follows the header,
and is preceded by the DESTINATION LIST if the item is going to more than one
node.  Certain types of items do not need text segments, since all data needed
to process them is in the header.  The type 20/0 SUBS.INF 'ping' is a good
example of this.

OPTIONAL LINES: Some messages contain lines which may or may not be shown to the
person reading the message.  These lines will begin with a ^D [ASCII 4] charac-
ter, followed by a digit from 0 to 9.  If the reader's "optional line" setting
is equal to, or less than, the digit in the optional line's prefix, then that
line will be shown to the reader.  [The BBS uses a value of 10 when a user does
not want to see any optional lines; it never displays lines having a prefix of
^D0.  LNET-II can display all optional lines.]  If the reader has "optional
lines" set to 5, then only those optional lines having prefixes of ^D5 through
^D9 will be displayed.  The routing stamps added to posts and e-mails [starting
with NET33] have a prefix of ^D0.  Some mail utilities also use the ^D0 prefix
to write a FIDO-style 'TO:' field.  Some editors write user signatures as
optional lines with ^D9 prefixes, and write the BBS tags starting with ^D8 and
working downward.

DELETED ITEMS: LNET-II usually regards an item as "deleted" when it originally
had main type 65535 OR when it has been marked for deletion with the ALT+D,
CTRL+D, or ALT+T commands.  When items are marked to be deleted by LNET-II, they
do NOT have their main types changed to 65535; rather, LNET-II simply places a
flag in the packet's index file, indicating that the item is to be deleted when
the packet is rewritten.  The search features consider an item to be deleted
only if it is main type 65535 - items marked for deletion are, technically, not
yet deleted.  Items having main type 65535 cannot be undeleted by LNET-II.

NONEXISTENT NODES: LNET-II considers a node to be "nonexistent" when no data for
it is found in the network's BBSDATA.NET file.  It is important to note that
such nodes MIGHT actually exist but that the updates containing their data
simply haven't reached your system yet.  LNET-II can display only the numbers of
nonexistent nodes; it cannot display any BBS details for them.




















───────────────────┤ LNET-II v1.00 Technical Notes - Page 6 ├───────────────────