[Intel Navigation Header]

    StorageExpress(TM) System: NetWare Sparse File Notes

    
    

    STORAGEEXPRESS(TM) SYSTEM: NETWARE SPARSE FILE NOTES

    
       NetWare sparse file - what is it?  What are the symptoms?
    
    
    
       The following explanations were taken from the Novell Network Support
    
       Encyclopedia
    
    
    
       Version 94-7  (November 1994)
    
       FYI:  Sparse Files on NetWare 3.x
    
    
    
       DISCLAIMER The origin of this information may be internal or external to
    
       Novell.  Novell makes every effort within its means to verify this
    
       information.  However, the information provided in this document is FOR
    
       YOUR INFORMATION only.  Novell makes no explicit or implied claims to
    
       the validity of this information.
    
    
    
       TITLE:  Sparse Files on NetWare 3.x
    
       DOCUMENT ID#:  FYI.A.1031
    
       DATE:  28FEB92
    
       PRODUCT:  Btrieve
    
       PRODUCT VERSION:  ALL
    
       SUPERSEDES:  NA
    
       SYMPTOM:  Btrieve file size larger than the disk volume size
    
    
    
       ISSUE/PROBLEM
    
    
    
       A problem has been seen where a Btrieve file appears to be many times
    
       larger than the actual hard disk size.  A DIR on this file showed the
    
       file size to be in excess of 400 megabytes, even though the hard disk
    
       was only 150 megabytes in size.  This occured under NetWare 386 version
    
       3.11 using the Btrieve NLM version 5.15 patched up to #49.
    
    
    
       This phenomenon occurs when NetWare creates a sparse file, which causes
    
       a DIR to show a size much larger than what was actually allocated.
    
    
    
       The Btrieve application had been running successfully for several months
    
       with no problems, but recently several new workstations had been
    
       installed on the ethernet network.  Also, several different shell
    
       versions were being used, up to but not including the latest version
    
       (3.22a).  What caused the Btrieve file to become a sparse file was not
    
       determined, but a Butil -Recover followed by a Butil -Load successfully
    
       rebuilt the file, after which, the sparse file was deleted.
    
    
    
       Sparse file
    
    
    
       A file with at least one empty block. (NetWare won't write any block
    
       that is completely empty.)
    
    
    
       Databases often create sparse files.
    
    
    
       For example, suppose the disk allocation block size for volume VOL1: is
    
       4 KB. Also suppose that a database opens a new file, seeks out the
    
       1,048,576th byte, writes five bytes, and closes the file.
    
    
    
       An inefficient operating system would save the entire file to disk.
    
    
    
       The file would be comprised of 256 zero-filled disk allocation blocks
    
       (the first 1 MB) and one more disk allocation block with five bytes of
    
       data and 4,091 zeros. This method would waste 1 MB of disk space.
    
    
    
       However, NetWare writes only the last block to disk, saving time and
    
       disk space.
    
    
    
       Sparse files aren't limited to large files. If a two-block file is
    
       created and the first block is empty, the operating system treats the
    
       file as a sparse file.
    
    
    
       If a program tries to read from a file's empty blocks, the operating
    
       system generates and returns zeros.
    
    
    
       The NetWare NCOPY command doesn't write to sparse files automatically.
    
       NCOPY has a /f option that forces the operating system to write to
    
       sparse files.
    
    
    
       NETWARE OVERVIEW
    
    
    
       Sparse files
    
    
    
       NetWare 386 also supports sparse files.  Databases sometimes create
    
       sparse files.  The following is an example.  Suppose the disk allocation
    
       block size for volume VOL1 is 4KB.  Also suppose that a database opens a
    
       new file, seeks out to the 1,048,576th byte, writes 5 bytes of data, and
    
       closes the file.  An inefficient server would save this entire file to
    
       disk, 256 zero-filled disk allocation blocks (the first 1MB) and then
    
       one more disk allocation block with 5 bytes of data and 4,091 zeroes.
    
    
    
       This method wastes 1MB of disk space.  The NetWare server does not write
    
       the initial, zero-filled megabyte to disk.  It writes only the last
    
       block to disk.  If a program tries to read from the file's first
    
       megabyte, the server generates and returns zeroes.
    
    
    
       582/Rev1.0                           19
    
    
    
       FYI: NetWare and Sparse Files
    
    
    
       DISCLAIMER The origin of this information may be internal or external to
    
       Novell.  Novell makes every effort within its means to verify this
    
       information.  However, the information provided in this document is FOR
    
       YOUR INFORMATION only.  Novell makes no explicit or implied claims to
    
       the validity of this information.
    
    
    
       TITLE:  NetWare and Sparse Files
    
       DOCUMENT ID#:  FYI.P.9423
    
       DATE:  26JUL91
    
       PRODUCT:  NetWare
    
       PRODUCT VERSION:  3.10, 3.11, 2.2
    
       SUPERSEDES:  NA
    
       SYMPTOM:  NA
    
    
    
       ISSUE/PROBLEM
    
    
    
       Below is an excerpt from the NetWare v3.11 Concepts manual concerning
    
       sparse files.
    
    
    
       Extracted from v3.11 Concepts manual
    
    
    
       Sparse file
    
    
    
       Any file that has at least one empty block.  NetWare will not write any
    
       block that is completely empty.   Databases often create sparse files.
    
    
    
       Example:
    
    
    
       Suppose that the disk allocation block size for volume VOL1 is 4KB. Also
    
       suppose that a database opens a new file, seeks out the 1,048,576th
    
       byte, writes 5 bytes, and closes the file.  An inefficient operating
    
       system would save the entire file to disk. The file would be composed of
    
       256 zero-filled disk allocation blocks (the first 1MB) and one more disk
    
       allocation block with 5 bytes of data and 4,091 zeroes.  This method
    
       would waste 1MB of disk space.  NetWare will write only the last block
    
       to disk, thus saving time and disk space.
    
    
    
       Sparse files are not limited to large files. If a 2-block file is
    
       created and the first block is empty, the operating system treats the
    
       file as a sparse file.  If a program tries to read from a file's empty
    
       blocks, the operating system generates and returns zeroes.
    
    
    
       NCOPY doesn't write to sparse files automatically.   It has a /f option
    
       that forces the operating system to write to sparse files.
    
    
    
       The following is more information concerning sparse files:
    
    
    
       1. The minimum sparse file will contain at least two blocks.  It will
    
       contain, as a minimum, the first block of the file and the last block. 
    
       This is the case even if only the last block in the file had data
    
       written to it.
    
    
    
       2. To detect sparse files you need to use NDIR.  NDIR will report the
    
       number of bytes stored in a number of files and the number of bytes that
    
       are allocated in a number of blocks.  If the number of bytes stored in
    
       files is greater than the number of bytes allocated in blocks, the file
    
       is a sparse file.
    
    
    
       3. When using COPY or XCOPY commands to copy sparse files ALL bytes in
    
       the file will be copied to the destination, therefore, the sparse nature
    
       of the file will be lost.  DOS commands do not preserve the nature of
    
       sparse files when copied because doing so would require that they be
    
       aware of how NetWare stores files.  A DOS copy simply opens the source
    
       and destination files, and reads the source, writing it to the
    
       destination until it reaches EOF, after which it closes both files.  The
    
       NetWare shell intercepts the DOS open, read, write, and close calls in
    
       this process to determine if the calls are for network drives.  If the
    
       files being copied are on network drives, (which is the case with the
    
       sparse files), the shell handles the call and passes the necessary data
    
       back to the program.
    
    
    
       NetWare 386 Theory of Operations manual describes the implementation of
    
    
    
       sparse files on page 16-17, and as it mentions, a block of zeros will be
    
       returned when an application attempts to read from the beginning of the
    
       file (which is not stored).
    
    
    
       Because of this, the copy command will read from the sparse file and
    
       read a block of zeros, which are then explicitly written to the
    
       destination file...  thus is no longer a sparse file.
    
    
    
       4. The NCOPY utility that ships with v3.11 and v2.2 does not copy sparse
    
       files.  The version is 3.56.  It will return "Error copying file" when
    
       you try to copy a sparse file.  The NCOPY utility that comes with 3.10
    
       version 3.07 does work.
    
    
    
       SOLUTION NA
    
    
    
       WHAT ARE THE SYMPTOMS OF A NETWARE SPARSE FILE?
    
    
    
       FYI: Backing Up Sparse Files
    
    
    
       DISCLAIMER The origin of this information may be internal or external to
    
       Novell.  Novell makes every effort within its means to verify this
    
       information.  However, the information provided in this document is FOR
    
       YOUR INFORMATION only.  Novell makes no explicit or implied claims to
    
       the validity of this information.
    
    
    
       TITLE:  Backing Up Sparse Files
    
       DOCUMENT ID#:  FYI.A.3390
    
       DATE:  04FEB93
    
       PRODUCT:  NetWare C Interface DOS
    
       PRODUCT VERSION:  v1.20
    
       SUPERSEDES:  NA
    
       SYMPTOM
    
       NA
    
    
    
       ISSUE/PROBLEM
    
    
    
       Developers are unable to backup sparse files from DOS, OS/2, or Windows
    
       client workstations.
    
    
    
       SOLUTION
    
       Currently, there is no elegant solution to this problem.  Obviously, the
    
       entire file could be read utilizing the standard IO calls.  However, the
    
       file when restored would no longer be a sparse file.
    
    
    
       SYMPTOM
    
       E3072:  Failed to read [file name]'s map ([number] bytes)
    
    
    
       SOLUTION:
    
       ARCserve couldn't read the sparse file's map due to the number of
    
       seqments.
    
    
    
       SYMPTOM:
    
       Restore job fills up the volume then fails.  Error message in activity
    
       log "File XXX FT signature invalid."
    
    
    
       E3096:  Failed to restore sparse file [file name], [number] segments
    
    
    
       CAUSE:
    
       The user backed up a Novell "sparse" file. When the StorageExpress
    
       system performs a backup, it receives the entire physical file with the
    
       0 byte blocks rather than the sparse file.  This can result in an volume
    
       image on tape that is larger than the actual server volume which causes
    
       a volume restore to fail.
    
    
    
       SOLUTION:
    
       Restore to a server with a larger volume and use "NCOPY /Force Sparse"
    
       to transfer files to the original server or submit the restore job with
    
       an exclude for the file listed in the error message activity log.
    
    
    
       SYMPTOM:
    
       Backup is backing up more data then is on the volume.
    
    
    
       Backup hangs or runs for a long time.
    
    
    
       StorageExpress and/or Target Server abends or hangs during backup.
    
    
    
       CAUSE:
    
       The user backed up a Novell "sparse" file. When the StorageExpress
    
       system performs a backup, it receives the entire physical file with the
    
       0 byte blocks rather than the sparse file.
    
    
    
       SOLUTION:
    
       Use NDIR to locate the sparse file and exclude it from the backup.
    
    
    
       There are two ways to locate the file(s) that are causing the backup
    
       problem.  They are:
    
    
    
       Map a drive to the root directory of the server volume.
    
    
    
       Change to the newly mapped drive and type in the command NDIR /SI GR
    
       100000000 /SUB at the prompt.  (This command will find files greater
    
       than 100MB in size.)
    
    
    
       Also try NDIR /SI LE 1 /SUB at the prompt. (This command will find files
    
       less then or equal to 1 Byte in size.)
    
    
    
       or
    
    
    
       Start a custom backup on the server volume that has the long backup
    
       time.
    
    
    
       Use Rconsole to load Monitor on the file server.
    
    
    
       Select "Connection Information" from the Monitor menu.
    
    
    
       Scroll down to the account used by the StorageExpress system and look at
    
       the "open files" screen.
    
    
    
       When a file that you know is not large shows on the open screen for
    
       several minutes, write down its name.
    
    
    
       After the backup is complete, delete and replace the corrupted file.
    
    

    Trademark information