Black Box 4/BlackBox.cdr/database/datamage.arj/DOCS.ZIP/UPDATE.DOC

Text File | 1991-09-23 | 26KB | 476 lines

                           DATAMAGE - Update Listing

Recently  it  occurred to me that a DATAMAGE user who got an upgrade  might  be
hard  pressed  to  determine  if there were any  differences  between  his  new
programs  and his old ones.  The following is a very incomplete listing of  the
changes DATAMAGE has undergone over the years.  I began this listing with 2.6.

DATAMAGE 1.0

I began this package in '85, via the BASIC interpreter.  Every once in a  while
I  run across some code that I suspect was in the  original,  interpreter-based
program.  But not so often, now.

I began uploading compiled versions of it in '87, when I got a modem.  Sometime
in the middle of that year a sysop had the idea of sending the programs to  PC-
SIG.    I had never heard of them, (and now I wish I still hadn't)  but  worked
hard  to  get something  presentable together to send them.  They  liked,  they
published.

The initial version was kind of wimpy, and had several KILLER bugs in it.   The
results  of the article which SIG placed in their magazine were that a  lot  of
people tried it and wrote me about the bugs, and what else it needed to work.

DATAMAGE 2.0

I  fixed all the bugs in 1.0, added date fields, sorting and  record  selection
based  on dates. I had changed compilers three times and, unbeknown to  me,  my
routine for rounding off the ieee floating point numbers no longer worked.

I have bitten the bullet and spent big dollars on the Microsoft Basic  Compiler
6.0b,  and,  since it usually works (for a change) I don't plan to  upgrade  my
compiler again.  I have moved away from BASIC and the BASIC portion of DATAMAGE
will be maintained but not expanded.

With  V:2.0  the  old-style  GDRECNOS.SAD file  was  replaced  by  the  current
CTRLFILE.RAD,  a random-access version of the same thing.  If you have  an  old
V:1.0  file  you'll need a program I have called OLD2NEW.EXE.   Please  mention
this in your registration letter.

DATAMAGE 2.2

The  main  datafile is now open on two iocbs.  The first in  random  mode,  the
second  in binary mode.  There are several places in the programs where I  need
only  one field from a record.  The random mode lacks the ability to  return  a
single  field and, previously, I had to move an entire record from disk to  get
at  just one field.  This change augmented the speed of the record  select  and
calculation routines greatly.

SCREEN  MAKER was my pet C project for quite some time.  It was a little  buggy
in the initial versions of 2.2, of which there were several.  SCREEN MAKER adds
relational processing to DATAMAGE as it will do lookups and updates into up  to
seven  datafiles.   It  also gives the user the  ability  to  originate  custom
screens  complete with an optional report that is printed after the new  record
is written to disk.  As always within DATAMAGE, there is no text file to write.
The  screens and reports are initialized by locating your cursor  and  pressing
return, then answering the prompts.

Much improvement was done on the POWER MAIL program, rendering it able to print
labels for records found via index search or entered from the keyboard as  well
as it's original targets of all records or records in a MARKER file.

The source code was added to the \MAGE\DOCS directory.  The complete BASIC code
for POWER MAIL, the C core library and ISAM demonstrate how to do programs that
share  datafiles  with DATAMAGE, in keeping with my initial goal of  writing  a
HYBRID system instead of just another database.

DATAMAGE - Version 2.6 Mods

BREAKING THE 640K BARRIER - STORAGE FLEXIBILITY:

The  RAM_BASE  and  DISK_BASE programs have been combined,  and  the  new  BASE
program  can  access  files up to 32,000 records.  It retains  the  ability  to
process  counter  fields  in memory, provided the  memory  is  available.   The
program no longer loads the file index into memory.  On a computer with a  hard
disk  this  makes little difference.  With floppies it does.  The few  lose  in
favor of the norm - sound familiar?

The BASE program will prompt the user for the disk to use for temporary  files.
This  will  happen  when insufficient memory is available for  storage  of  the
counters.   I  have loaded files of 14,000 records and the program is  able  to
find the memory to process the counters.

DATAMAGE  can break the 640K barrier by making use of a RAM DISK.  If you  have
an AT with two megs of ram, simply install the RAMDRIV.SYS that came with  your
DOS,  or  use your choice of RAM DISK drivers  operating  in  extended/expanded
memory.   By specifying the RAM DISK for temporary storage you speed things  up
considerably because DATAMAGE gets access to more than 640K.

ABSOLUTE FIELD NUMBERS:

The  key.sad definition file now holds four numbers per field as compared  with
the  previous three.  The DBSEMAKR program will load an old style file and  add
the  new absolute record numbers.  Write the definition into a junk  directory,
then copy the new key.sad file into the directory holding your file.

The  absolute  field  numbers are a unique identifier for the  field.   If  the
datafile  is reorganized they provide way for the macros,  input-output  groups
and  custom  screens  to find the fields they use.   In  previous  versions  of
DATAMAGE reorganizing a file would cause it's macros and screens to fail.

If  a field is moved within the file, whether directly or by  inserting  fields
previous to it, the absolute record number remains unchanged.  You might have a
field  with an absolute number of 10 in position 13 within your file.  You  may
have moved it down three fields, or you may have inserted three fields pervious
to it.  By writing the absolute field number to disk the programs can now  find
any field wherever it may end up.

AUTOMATIC "INDEX" MAINTENANCE:

DATAMAGE has always automatically maintained it's indexes.  Other programs call
MARKER  FILES indexes.  MARKER FILES were, in prior versions, written  whenever
the user wrote one, or ran a macro that wrote one.

The BASE program now supports full automatic maintenance of your MARKER  FILES.
This  is done via the AUTOEXIT macro file.  The program senses the presence  of
an  AUTOEXIT.MAC file.  If a record is written, (entered, updated  or  deleted)
the  AUTOEXIT macro is automatically executed when the user quits the  file  or
ends the program.

The macro executes and uses all features of the BASE program to create  groups,
do calculations and order records, then writes as many MARKER FILES as you need
to maintain.  By executing on file exit these MARKERS are always current.

INITIAL DISK SPACE RESERVATION:

Datafiles are built one record at a time.  Record entry, provided the file  has
no  holes caused by record deletion that can be filled, causes a new record  to
be added to three files:  CTRLFILE.RAD, FILEINDX.RAD and YOURDATA.RAD.  If  you
started with a blank disk and entered a thousand records your drive would be  a
mish-mash  of clusters belonging to the three files.  And the fact  that  there
are  other  processes allocating/freeing disk space between  DATAMAGE  sessions
doesn't help matters a bit.

The DBSEMAKR program can now reserve disk space at the time of file  definition
or  reorganization.  The user enters the number of records for which he  wishes
to reserve space and the program writes blank records into the three files.  If
the  file grows beyond the number of records anticipated the programs begin  to
dynamically allocate extra space, as usual.  The allocation of initial space is
optional, and may be bypassed.

CALLING YOUR RECORDS:

The BASE program will now dial the modem from the record display/update screen.
The  user  simply presses D with the flasher on the phone  number  field.   The
modem  is  initialized  at three hundred baud and  only  the  most  rudimentary
commands are issued to it.  Almost any modem should work and the HAYES  command
set is not required of the modem.

DATAMAGE V: 3.1 MODS:

ANOTHER CHEAP VIDEO TRICK:

Recently  I received another letter or comment with a registration accusing  me
of  being left handed.  That's about the fifth or sixth one.  Usually it  seems
to be all right with the user; they're left handed, too!  But I'm not.

There  are people who have complained about actually having to press  a  number
key  all  the time.  And I understand, after all, those keys  are  SMALL!   And
actually targeting, then hitting one is too hard, I agree.

Then  there  are  the almost 3,000 registered users of  DATAMAGE,  in  general.
These  tough individuals have been put upon to press a number key for  so  long
that they have become used to it.  And any change in as basic an operating mode
as choice selection would undoubtedly cause them trouble.

So the choice selection routines have been slightly modified.  They now display
one  of  the choices in (gag) inverse video.  You may, IF YOU WISH,  press  the
right/left  arrow  keys until your selection is highlighted, then  hit  return.
But you can just hit the number key beside the option, if you think you can cut
it.  And the num lock key will make it right handed, too.

But  that's MY opinion.  DATAMAGE isn't my program anymore, hasn't  been  since
'88 when it was first published.  There are people who use it a LOT more than I
do, and anything I can do to make it better for YOU I'll be more than happy  to
do  so.  Witness the four days I spent converting all the DATAMAGE code to  the
new video trick.

POWER COPY RE-DONE:

The  computer simply ran out of memory to store the record pairs.  Well, if  it
won't  fit  in memory you'll be obliged to write it to disk.  And  that's  just
what  I  did.  POWER COPY now uses a temporary disk file to store a  number  of
record pairings limited to 32,000.

BASE PROGRAM MODS:

RECORD ENTRY:

The speed with which space for a record entered is found has greatly  improved.
And,  as  the program no longer loads the file index into memory files  can  be
accessed much faster.

BINARY SEARCH:

The  binary  search is dependent upon the current group being in order  on  the
target  data.  The binary search is available on the target data only when  the
current  group has been sorted (or a MARKER file has been loaded) in  order  on
the target field.

The binary search doesn't have to look at all the records, or even the  records
up  to the target.  It's not instantaneous, the number of records in  the  file
still affect it's execution time.  But I trust you will find it VERY fast.

The  binary search is offered in the find records (F-5) function if  the  group
has  not  been  placed in order it will beep you.  Then you  simply  enter  the
search  values,  and the routine does it's thing.

The  BINARY SEARCH will find the place in the file where the search data  would
belong.   It places the first two targets, of which there may be up  to  eight,
in  the  BROWSE  display screen and, as your group is in order  on  the  target
field,  it should be easy to see if you found a match.  To display  the  record
just hit return.

Other  programs make binary search available, usually by keeping  bulky  b-tree
style indexes and allowing binary search only upon fields having an  associated
b-tree index.

DATAMAGE  offers binary searches on it's usual array of targets:  Any  field(s)
in  the datafile, the counter fields or the record numbers.  There's  no  sense
copying the others.  You gotta beat them at their own game.

DATAMAGE doesn't need a tree file to do a binary search.  With DATAMAGE you can
just sort on any data, then search it.  The act of sorting the file creates the
BINARY INDEX in memory.  If you enter, update or delete records the index  will
be  maintained  IN MEMORY by placing the new/updated records  in  their  proper
place, and removing the deleted records from the index.

Since  DATAMAGE  doesn't  require  a  disk file to  do  this  it  also  doesn't
automatically  update any files you may choose to keep.  There may be a way  to
auto-maintain  a  MARKER  file for the purpose of BINARY INDEXING  in  a  later
version of DATAMAGE.  For now, just save your index after updating a file.

If  you  forget to save your index no great loss.  Just load your  most  recent
index  file,  use  the APPEND REMAINING RECORDS function  in  the  MARKER  file
function to add the new records, then sort your file.  The sort will go quickly
as  there  is little for it to do.  Most of the records are  already  in  order
thanks to that most recent MARKER file.

There you go, DATAMAGE users:  The "IMPOSSIBLE" has been done, there are "shoot
from the hip" binary searches that require no definition in the datafile and no
bulky,  error-prone  b-tree file.  And a simple, easy, fast way has  been  made
available to update the MARKER file containing the order of the records, should
you consider it necessary to keep one.  You may have your cake and eat it, too.
You may do, and NOT do, whatever you need/wish.

MARKER FILES:

In order to facilitate the recording of the data that the group was ordered  on
when a marker file was written it was necessary to change the marker files.  If
you  are  getting  an upgrade your old markers won't work.   But  it  won't  be
difficult to write new ones.  Sorry, but there was no other way.

The MARKER function now offers three options instead of the original two.   You
may  now  restore the remaining records to the group.  Doing this,  of  course,
destroys  the  content  of the counter fields.  But you end  up  with  all  the
records in the file, mostly in order on whatever sorting scheme you used.

Doing a sort on your choice of parameters will be much quicker after using  the
restore  remaining  records function as the sort will need to move  only  those
records that have been entered or updated since the last sort.  If your purpose
is  to  place a large file into order for the purpose of maintaining  a  MARKER
file this is the way to go.  It is much faster than sorting the entire file.

IMPROVED SORTING:

The BASE program can now sort up to eight levels of data, in any combination of
field types.  All sorting is now accomplished internally.  The crucibles of the
sort routines have been redone and are FAST.

VERSION 3.5 MODS:

SYSTEM MODS:

The maximum length of a string field has been increased from thirty-five to two
hundred fifty characters.  Again, this change was made due to user  complaints.
DATAMAGE  files  are flat.  When you specify the length of a field  that's  how
long  it actually is, regardless of whether or not the space is used.  This  is
one  of  the reasons DATAMAGE can operate so rapidly, and  it's  datafiles  are
accessible to popular programming languages.

DATAMAGE is NOT a word processor.  I encourage you to use the absolute  minimum
space necessary to record your data.   When I started programming a disk  drive
cost 10s of thousands of dollars, and a disk "pack" cost almost a thousand.   I
have seen people FIRED for wasting disk space.  Things have changed a lot since
then,  everything's  cheap  and easy now.  But good  programming  and  datafile
design have not changed, and in my mind at least, wasting resources is a CRIME.

BASE PROGRAM MODS:

The  BASE  program has been improved to accept command line arguments.   As  is
becoming  commonplace,  this  was suggested by  several  registered  users  who
desired the execution of a MACRO either from a batch file on a menuing system.

I would like to take this opportunity to thank my users for their feedback  and
to encourage you to write to me and state your needs and the way in which their
fulfillment would positively augment the DATAMAGE system.  I will not guarantee
to  include your mod in the next version, but I am dedicated to  improving  the
system  in whatever way I think may be useful to my users.  If your  suggestion
would not benefit most users it may be ignored.

The  BASE  program will now respond to command line arguments  by  loading  the
datafile  and executing the macro.  If you start the program by entering:  BASE
C:\MAGE\INVOICES\PAYABLES.MAC   the   program  will  load   the   datafile   in
C:\MAGE\INVOICES and execute the PAYABLES MACRO.

If  the  program  is  given command a line argument  it  will  terminate  after
executing  the  MACRO.  It will return to the MS-DOS command level  instead  of
running the GO.EXE program as it usually does.  If you are running a batch file
it will continue without user intervention.  If you are using a menu system you
will return to it.

BINARY SORTS/SEARCHES:

The  SORT function (F-6) has been slightly modified, as has the  BINARY  search
routine.   Here we have a classic example of functions that  worked  perfectly,
but whose concept was flawed.

The  implementation of the BINARY searches was a large task.  The fact  that  I
was  obliged to do this in BASIC didn't help a bit.  But, the BASE program  was
originally  done in BASIC, and I would prefer to spend my time and work on  the
improvement of the system as opposed to the re-doing of same.

Version  3.1 prompted the user for his choice of ignoring  case/spacing  during
the BINARY searches.  That was nice, but it didn't work.  This is because  that
choice affects the order of the group produced by the sort.  The data I used to
test it was in all-caps; I didn't see it.

The program now prompts you for your choice of ignoring case/spacing during the
SORT  that produces the BINARY indexes.  Your choice is then  displayed  during
the entry of data upon which to BINARY search.  This works.

MARKER FILES:

Unfortunately, this mod has rendered MARKER files produced by previous versions
unusable.  So, if you're getting an update to 3.5, you'll need to re-create all
your MARKER files.  My apologies.

MACRO FILES:

The  above  error has also rendered all MACROS containing a sort  that  targets
string fields unusable.  This is due to the new need to record your choices  of
ignoring case/spacing in the MACRO.  I feel REALLY badly about this!

You  NEED to be certain that you don't attempt to run old MACROS  with  version
3.5.  The program SHOULD catch the error and terminate your MACRO.  I  wouldn't
be willing to bet it will, in this case.  If the sort is the last thing you did
in the MACRO it may go undetected.  If not, the MACRO will halt after the sort,
and that's no good either.

SCREEN MAKER:

This program, included in versions 2.2 through 3.1, died.  Or, was re-born.  At
the  very least, the name has been changed.  It is now APPLICATIONS MAKER,  and
many  new features have been added; many of the limits of the old version  have
been removed.

APPLICATIONS MAKER - DATAMAGE IS NOW A DATABASE!:

The  APPLICATIONS  MAKER program supports the  design/implementation  of  user-
originated processes.  It can be used to make reports, screens, etc.  It  looks
and works just like SCREEN MAKER, but is much more powerful.

I  am  VERY apprehensive of placing the power of APPLICATIONS  MAKER  into  the
hands of my users.  This program is DECEPTIVELY easy, yet it certainly has  the
power  to totally RUIN any datafile it opens.  For this reason I took  lots  of
time  to  document  a  fool-proof way in which you  may  write  and  test  your
applications.   PLEASE  read  the  docs for  this  program!   The  filename  is
APCNMAKR.DOC; it's in the \MAGE\DOCS directory.

APPLICATIONS MAKER is a DANGEROUSLY powerful program; it should be operated  at
the  edit  level  with great care, and only by those with a  knowledge  of  the
DATAMAGE  system and a thorough understanding of the job to be done.  It's  not
that  the program is difficult or "tricky" in it's design or  operation;  QUITE
THE CONTRARY!  It's user-interface is (if I DO say so, myself) SECOND TO  NONE.
And the same can be said of it's power and it's speed.

With ANY program of this type, which can not assess the fitness of it's user to
accomplish  the things that the program can do, mistakes WILL be made.   If  my
SINCERE  advice on the creation and use of a test environment (in the docs)  is
ignored data will be lost; entire datafiles will be TRASHED.

You  have  a  MUCH  better chance  of  successfully  originating  your  desired
application  with APPLICATIONS MAKER than with any other program.  While  doing
so you will be obliged, with this or any other system, to think in a dual mode.

On  the  one hand, you will be concerned with what you need to do and  how  you
will do it; on the other you will need to figure out how to get the software to
do your bidding.  The latter process will interrupt the former.  There's no way
around  that,  until  someone (and it WON'T be me) invents a way  to  have  the
computer read your mind, and then to comply with your wishes.

VERSION 4.0:

The  modifications  that  comprise  version 4.0 were almost  all  done  on  the
APPLICATIONS MAKER program.  A few cosmetic changes were made to BASE, DBSEMAKR
and  POWRMAIL.  There were (and probably still are!) some spelling mistakes,  a
few inconsequential boo-boos in the prompts, etc.  I wish to STRESS, here, that
I appreciate and solicit the help of my users in finding such errors.

APPLICATIONS MAKER:

SCREEN STORAGE:

The former versions of APP MAKER used the computer's memory to store the  input
and report screens.  This resulted in severe limitations on the maximum size of
both  the screens and the applications, as your application is also  stored  in
memory.   The screens are now stored on disk, and their size has  been  greatly
expanded.  There is now lots more room for your application in memory as well.

LOCATE ITEMS:

The  program  now includes LOCATE items that both locate your  viewport  within
your  application and set horizontal boundaries outside which  the  application
will not go until the LOCATE is released.  This gives you the ability to create
applications  that  consist  of multiple screens,  placed  side-by-side.   Your
application  can branch to one of your vertical screens and then work it's  way
down  within  it.  When the sub-task is completed you may return to  the  first
screen and perhaps select another option on yet another screen.

COLOR SCREENS:

This  change  also  allowed  me to implement color  screens.   The  PC's  video
requires  two bytes of storage for each character on your screen; one  for  the
character  and one for the attribute (or color).  When the program  was  making
use of the computer's memory to store the screens the doubling of the necessary
space in order to offer you color screens was out of the question.

You  may  now select, via the use of the arrow keys, both  the  background  and
foreground  color  of  any  item on your screen  and  any  rectangular  portion
thereof.   I do hope that you enjoy your pretty screens as it took me  a  solid
month to do this for you.

STRING STORAGE AND FORMULAS:

The  APP  MAKER program now offers you twenty areas in your  computer's  memory
designed to hold string data.  The string items on the input screen now include
the  option  to  execute a "formula" with which you  may  store,  retrieve  and
concatenate  your  strings.   These string storage areas can  hold  up  to  the
maximum length of a DATAMAGE string field which is 250 characters.

GET RECORD ITEMS:

The scope of the APP MAKER program continues to expand.  The program can now be
used  to  process  records in a datafile without having to find  them  via  the
indexes.  You may process all records in the file in the order that they appear
or the records in a MARKER file via the GET RECORD items.

MARKER FILES:

The APP MAKER program can now load a MARKER FILE (produced by the BASE program)
and  process the records contained therein.  This comes in REAL handy when  you
wish  to  produce a report (or whatever) for only certain records  in  a  file.

Instead of having to test the content of each record and skip the ones that  do
not  conform  to your current criterion you can create a MACRO  with  the  BASE
program  to  quickly  identify the group of records you  wish  to  process  and
perhaps  place this group into some sort of order, then use a LOAD MARKER  item
to direct your application to process only the targeted records, in the desired
order.   This  entire  process can be automated via batch files  or  a  menuing
system, rendering the process independent of user interaction.

DEBUGGING FACILITIES:

The  DEBUG  items can be placed within your application or activated  from  the
edit  mode.  These items allow you to view the current content of your  numeric
or  string storage at any time during the execution of your application.   Your
variables can now be named and the names you assign to them are saved with your
application.  The DEBUG items/option provides you with a window that lists  all
of your variables, their names/purposes, and current content.

4.0 - WRAP UP:

Version 4.0 has undergone MANY changes, only the most significant of which have
been  documented  above.  I am committed to producing a system that  will  fill
your  needs with speed and ease unmatched by any other data program.  I  would,
again, solicit your feedback so that I may improve and expand the DATAMAGE.