Variable Length Fields

   The enhancements to memo-fields

   FlexFile addresses the four major problems that plague
   Clipper's implementation of memo-fields.

   First, FlexFile reuses all unused space.  If your user edits a
   memo (or deletes elements of an array) making it smaller,
   FlexFile is aware of every byte of the excess space and will
   use it later for new data.  This feature totally eliminates
   the file bloat that characterizes Clipper's DBT files.
   (Clipper frequently abandons space no longer needed by
   memo-field data, and once abandoned, not even packing the file
   can regain the space.)

   Second, FlexFile does not allocate space in blocks.  Instead,
   each item stored requires only the space for its own length
   plus a six byte header that FlexFile uses for management.
   This is in great contrast to Clipper's memo-fields that
   require a fixed block size of 512 bytes.  This feature alone
   will save disk space at an average of 256 bytes per record.

   Third, the size of FlexFile's Variable Length Field (VLF)
   files are limited only by your disk space.  This is in
   contrast to Clipper's size limit on a DBT file of 16M.

   Finally, FlexFile can store any valid Clipper data.  This
   includes arrays, character strings, dates, numerics and
   logicals.  In addition, the content of character data is not
   limited as it is in Clipper.  It is, therefore, possible to
   store SAVESCREEN() variables or any other binary data that you
   use.




   FlexFile's Extended Features

   Through FlexFile's ability to save an array to a field, your
   DBF's can become three dimensional (actually multi-dimensional
   with Clipper 5.0).  Looking at it another way, each record can
   now have a variable number of fields.  This new feature allows
   far greater flexibility in designing a database system.

   Lets take the simple example of storing a telephone number.
   In a customer data base, next to the name and address we store
   the telephone number.  But is one telephone number enough?
   Most businesses require at least a voice line and a fax line.
   Then again, there is that one guy who has got two voice lines,
   a fax line, a data line, and a whole bunch more at his house.

   So, do we build in five or ten telephone fields in the
   customer file?  Certainly this situation would not warrant a
   second file and an index with a relation. Usually, we tell our
   client (very nicely) to live with a maximum of two.

   With FlexFile, however, the answer is simple.  Save an array
   of telephone numbers.  Those customers who have one or two
   telephone numbers use only the space for one or two telephone
   numbers.  Those that have ten can have ten. There are no
   indexes or relations to maintain and you can extend this
   concept into those perplexing data item which your client
   defines as requiring "usually two or three, but sometimes
   twenty or more".

   FlexFile also allows the programmer to store any valid Clipper
   data type to a field.  At first look, this alternative seems
   less than impressive.  Why would you want to store a logical
   value, which takes only one byte per record in a DBF file, by
   using a six byte pointer-field in the DBF to point to an
   object which requires a six byte header in the DBV.  The
   advantage does not come from saving any one type, but rather,
   from being able to store different types to the same field.
   This is particularly advantageous in data driven applications.
   For example, if you give your clients user defined fields,
   they can change the type of a field and you won't have to
   worry about carrying extra fields for each type "Just in
   Case".




   USEing DBV files

   The use of FlexFile's DBV files was designed to parallel the
   use of DBF type files.  Specifically, the system is made up of
   work areas that are used to manage the VLF files in exactly
   the same manner as Clipper uses work areas to manage DBF
   files.  This method avoids the need to keep track of
   "handles", and allows the use of aliases.

   For example, you can V_SELECT(1) and then V_USE() a file in
   area one, and then V_SELECT(0) the next available work area
   and V_USE() another file there.  It should be noted that
   FlexFile's work areas are mutually exclusive from Clippers DBF
   work areas.  For example, you can V_SELECT(1) and V_USE() a
   file in VLF area one, and simultaneously, SELECT 1 and USE a
   file in DBF area one with no conflicts.

   The FlexFile system offers an unlimited number of work areas,
   however, you must decide what your maximum number of work
   areas will be before you open the first DBV type file.  The
   reason for this is to allow the system to create one
   contiguous block of memory for the management of the files
   instead of fragmenting memory with little blocks each time you
   open a new file.  At any point that you have all DBV files
   closed you can readjust this maximum number of work areas by
   using the V_FILES() function.


   Creating DBV files

   The simplest thing to do in FlexFile is to create a DBV type
   file.  Simply V_USE() a file that does not exist and you will
   have a DBV file open and ready to go.


   How to Manage Your Data with FlexFile

   Another great use for FlexFile is to hold data that is not
   repetitive and structured;  the DBF file structure is not at
   all accommodating in this area.  For example, almost every
   Clipper application requires system data such as a color
   scheme, default directories, default drive for temporary
   files, printer definitions, and other randomly sized
   non-repetitive data.  Storing this data is now a trivial
   matter.

   In this case, the design could be as simple as a two field DBF
   file with an index, and one FlexFile VLF file.  The DBF
   structure might look like:

       Structure for database: C:\FF\SYSTEM.DBF
       Number of data records:      20
       Date of last update   : 10/05/90
       Field  Field Name  Type       Width    Dec
           1  VAR_NAME    Character     10
           2  VLF_FLD     Character      6
       ** Total **                      17

   The key field <var_name> will hold the name of a Clipper
   variable that you use; the six byte character field <vlf_fld>
   FlexFile will use to hold a pointer to <var_name>'s value.
   Following from the above description, start-up code for an
   application might look like:

       // Declare the system wide public variables that you use.
       PUBLICaColors,;       // Array variable for colors
             cDefaultDir,;   // A user defined default directory
             dYearEnd        // A user defined year end date.

       //  Open the DBF, its index, and the VLF
       USE System.dbf INDEX System.ntx
       V_USE( "System.dbv" )

       //  Seek your colors array and retrieve it.
       //  Note that this Clipper 5.0 syntax for arrays is slightly
       //  different than the S'87 syntax.  See the V_FILLARR() function.
       SEEK "aColors"
       aColors = V_ARETRIEVE( vlf_fld )

       //  Seek the record that holds a "Default" directory.
       SEEK "cDefaultDir"
       cDefaultDir = V_RETRIEVE( vlf_fld )

       //  Dates can be stored as easily as anything else.
       SEEK "dYearEnd"
       dYearEnd = V_RETRIEVE( vlf_fld )

   Of course, for this code to work you must have previously save
   data to these records.  The next section on FlexFile's
   pointers gives an in depth view into storing the data,
   retrieving it and replacing old data with new.



   FlexFile's Pointers


   The key to maintaining secure data with FlexFile is fully
   understanding the pointers that FlexFile uses to access data.
   Clipper's memo-fields maintains a similar set of pointers but
   Clipper hides them from the programmer as well as the user.
   If you try to assign a Clipper memo-field pointer directly to
   a variable as in:

   var = system->mem_fld      // Clipper memo-field


   Clipper intercepts the call and automatically goes to the DBT
   file and fetches the data that <mem_fld> points to.  With
   FlexFile, this assignment would put the pointer into <var>
   instead of the data.  So, the equivalent code using flexfile
   is:

       var = V_RETRIEVE( system->vlf_fld )    // FlexFile VLF


   What we are doing here is passing the V_RETRIEVE() function a
   pointer to the data and V_RETRIEVE() is kindly passing us back
   data that was stored there.

   The same principal applies to saving data... with a twist.
   When Clipper saves data to a memo-field, it allocates disk
   space in 512 byte chunks and sticks a pointer to the allocated
   space in the memo-field of the DBF file.  For example,

       // Clipper hogs 512 bytes with this one.
       REPLACE system->mem_fld WITH "This will take 512 bytes"

   Clipper allocates 512 bytes of disk space, puts the 24 bytes
   of data in the front of the new space, and puts a pointer to
   the block in <mem_fld>.  If, without moving the record
   pointer, we say

       // Clipper abandons the first mess and requires 1024 more.
       REPLACE system->mem_fld WITH SPACE( 513 )

   then Clipper will abandon the first 512 byte block
   (permanently), and allocate 1024 bytes for the new data.  The
   old pointer to the 512 byte block is lost and the file space
   is gone forever.

   FlexFile is smarter than this.  Using FlexFile, we would save
   the 24 byte string with:


       // FlexFile neatly allocates 30 bytes of disk space.
       REPLACE system->vlf_fld WITH ;
          V_REPLACE( "This will take  30 bytes", system->vlf_fld )

   If we follow the pointer in this example, we can see what
   FlexFile is going to do.  The first thing that we notice is
   that we not only pass to the V_REPLACE() function the new
   data, but also, we give it the old pointer.

   Because this function will operate from right to left, first
   it will look at the right most <vlf_fld> to see if it is
   pointing to any old data.  If it is not, then the function
   proceeds to look through an internal table for other vacated
   space. We will assume for this example that this is the first
   V_REPLACE(). Therefore, there is no available space that has
   already been release by other data, so the new string is
   appended to the end of the file.

   To do this FlexFile will allocate 30 bytes of disk space, 24
   bytes for the data and six bytes to keep some information that
   it uses for management.

   Now we continue the example by replacing the thirty bytes with
   513 bytes of spaces:

       // FlexFile releases 30 bytes and stores 519.
       REPLACE system->vlf_fld WITH ;
             V_REPLACE( SPACE( 513 ), system->vlf_fld )

   This time, FlexFile sees that the old pointer has something in
   it, and so releases it.  Then it seeks internally for an
   available 519 byte space to put the new information.  There is
   none, so it tags this at the end of the file as well.

   There are two very important advantages you can see from this
   example.  First, only the size of the data was required to
   keep the data, and second, the next piece of data that you
   store that is thirty bytes or less will be stored in the
   vacated space.

   The thirty byte space that is available brings up an important
   point.  We have run tests where one VLF file undergoes
   millions of V_REPLACE()s.  The data ranged in random sizes
   from one byte to ten thousand bytes.  The amount of vacant
   space tended to level off at about 20% of the fully pack
   file's size and at that point the VLF file stops growing,
   period.