Dream 42/Amiga_Dream_42.iso/RiscPC/Internet/Connexion/SPak.spk/!NewsBase_docs_Notes

Text File | 1994-11-10 | 5.6 KB | 123 lines

When debatching a file, newsbase first attempts to recognise the type of
batch file by scanning it for valid separators. These consist of lines
beginning with:

#! rmail                mail
^A                      mail (control-A)
From                    mail (including trailing space)
#! rnews                news

If a separator is not found on the first line of the file, newsbase will
'rewind' to the beginning of the file, after attempting recognition. If no
valid separators are found, newsbase tries to guess the batch type by
presence of "Newsgroups:" headers, etc. Headerless parts of the file will be
saved to the groups "Email.postmaster" or "junk" depending on how newsbase
finally recognises it.

From Newsbase 0.43, articles are held in the directory:
group.name.~x.~xN
where N is (article number DIV 75) MOD 75
This now permits many more articles to be held without creating a deep
directory structure.

Article numbers are assigned for valid groups as soon as the
Newsgroups: line has been decoded. A "Xref:" header line is then inserted
into the article buffer, allowing clients to mark crossposts as read in all
the crossposted groups. Saving then follows the pregenerated group/number
list (allowing up to 10 groups).

From Newsbase 0.23, the "Supercedes:" header is respected, and a superceded
article is deleted (if not locked).

The "Expires:" header is handled by encoding the expiry date within a
file's datastamp for articles which require it (the high byte of the 5-byte
datestamp is set to &FF as a flag; the remaining 4 bytes contain the
expiry date as a 1-word integer in the form CEYRMNDY. Files encoded this
way have an apparent datestamp, to the filing system, sometime in the
year 2247).

Expire code is called on a per-group basis, to simplify index update.
Expired article numbers are added to a delete buffer and are removed from
the group index when the buffer fills or a different group is referenced.

The delete buffer is 512 articles deep (ie, 512 deletions before an index
rewrite is required).

When delete buffer is filled, or a deletion for a different group arrives, the
index is updated to remove entries for deleted articles.

Polling is disabled during an index update; an hourglass is displayed while
this occurs. With a medium-speed hard disk and around 1000 index
entries, an index update generally takes around 2-5 seconds.

Article request by message ID is not currently supported as the lookup is
quite slow (the index file must be scanned).

Index files are held in each group's "root directory", eg
news:rec.audio.~index.

The index file format is fixed-length 200-byte records, one record per article.
The record format is:
bytes 0-3 : article number
bytes 4-7 : article date (format CEYRMNDY)
bytes 8-19: reserved for extra information.
bytes 20-n: "Message-ID" field, from article header, null-terminated
bytes n-m : "From" field.
bytes m-199 : "Subject field.

Fields may be truncated if they do not fit within the record space available.
If no message ID is available in the article header, it is replaced with
"<0>", in accordance with RFC977.

Criteria for design of record structure-
 1. Must be easy to access, and fast to add and remove records, even from
    BASIC. Fixed record size avoids doing any significant processing.
 2. Article information is destinied to be passed to a client program via wimp
    messages. The "Article info" message passes the following information:
    article number, from, and subject. The maximum amount of data
    that can be passed in a wimp message is about 230 bytes.

Threading information is not provided for. However, many newsreaders (xrn,
for example) manage a passable substitute by simply sorting on subject. Full
threading will probably require maintenance of a separate threading
database, as the requirements are probably incompatible with those
(simplicity+speed) of the main indexes. This may be added at a later date
but will not be considered now.


This scheme works well with small numbers of newsgroups (150 tested but more
should scale fairly linearly) and, with the new directory structure of
Newsbase 0.43, more than 2000 articles per group. Better memory management
will be needed within NewsBase to deal with many more groups, but that is an
internal implementation issue. See the file "Tuning" for information on how
to tailor newsbase for larger newsfeeds.


The from and subject strings returned by "ListArticles" are length limited
and intended for display (in article catalogues) only. Use the fields
directly from the article headers for generating replies, etc.

The pathname returned by "article get" should be used once and not stored
across calls to Wimp_Poll. It is possible for article files to move without
notification (generally only at debatch or expiry time)

"Article delete" is included to complete the command set, but should
probably not be used in a multiple-reader environment. It will not be
effective against locked articles. One situation where the function might
be useful is to delete personal email - in this situation, deletion will
always be permitted. For other groups, deletion may be disabled as a
newsbase configuration option.

It is the responsibility of the newsbase client to track which articles and
groups a reader is interested in, or has read.

"Article lock" prevents an article from being deleted or expired, regardless
of expiry date.

Mail messages are mistakenly split on lines starting "From ". The extra bit
of message is saved to Email.postmaster (problem mail) with a suitable
subject (no recipient?) rather than into a group "Email" as currently.
Using the ka9q transport, this is only seen with POP (SMTP alters these
lines to ">From "). This can be avoided by using ^A or "#! rmail" separators.