Black Box 4

home *** CD-ROM | disk | FTP | other *** search

/ Black Box 4 / BlackBox.cdr / clipper / clsupbul.arj / WMMTXT.TXT < prev

Wrap

Text File | 1992-06-01 | 18.6 KB | 396 lines

Clipper Support Bulletin #3 Clipper 5.0 runtime memory management Copyright (c) 1991, 1992 Nantucket Corporation. All rights reserved. Version: Clipper 5.0, revisions 1.00 - 1.03 Date: 4th October, 1990 Revised: 21st May, 1992 Status: Active ================================================================================ This Support Bulletin covers the following topics: 1. Virtual Memory Manager (VMM) 1.1 Use of Lim 3.2 Expanded Memory (EMM) 1.2 Disk Swapping 1.3 Swap Space 2. Object Memory (SVOS) 3. Dynamic Overlaying 3.1 Configuration 4. Data File Buffering 5. Segment Locking 6. Fixed-Memory Allocator 7. MEMVAR Tables 8. Symbol Tables 9. The Extend System 9.1 Configuration Note: this document makes several references to the CLIPPER environment variable. For general information on how to configure the runtime environment, refer to the Clipper 5.0 "Programming and Utilities Guide." ================================================================================ 1. Virtual Memory Manager (VMM) "Virtual memory" is a generic term for hardware or software techniques that allow a limited amount of "real" memory to emulate a much larger "virtual" memory. The Clipper 5.0 Virtual Memory Manager (VMM) is implemented in software. The system runs in real mode on any 8086-compatible processor; it does not directly use the protected mode virtual memory capabilities of the Intel 286 and 386 processors. The VMM is a "segmented" memory manager. Virtual memory is allocated in segments, each of which may contain from 1K to 64K of data. Initially, the VMM works somewhat like a normal memory allocator (although it has an extra advantage: most VM segments are movable; the VMM can reorganize them to get maximum utilization of real memory). Unlike a normal memory allocator, however, the VMM does not "run out" when the available real memory can no longer contain all of the allocated segments. Instead, one or more of the least recently used (LRU) segments are "swapped out" into secondary storage (or otherwise removed from real memory, see below) to make room for new segments. Later, if data in a swapped segment must be accessed, it is swapped back in, displacing segments that are not currently needed. The maximum virtual address space of the Clipper 5.0 VMM is 64MB. ---------------------------------------------------------------------------- 1.1 Use of Lim 3.2 Expanded Memory (EMM) By default, the Clipper 5.0 VMM uses LIM 3.2 Expanded Memory (EMM) as secondary storage. EMM provides fast access to swapped segments. Up to 8MB of EMM can be used. Configuration ------------- EMM usage can be controlled using the E setting in the CLIPPER environment variable. Valid settings are from 0 to 8192 (units of 1K). A setting of 0 disables the use of EMM altogether. (Note: disabling EMM is not recommended under Clipper 5.0. Although EMM is not required, it allows increased performance for most applications. In a few cases, limiting EMM usage may be desirable--see the note below under Disk Buffering System.) Example: SET CLIPPER=E2048 Some disk caching programs use EMM to cache disk sectors. Certain cache programs may cause conflicts with the Clipper 5.0 VMM system if they assume that application programs never use EMM. The BADCACHE setting in the CLIPPER environment variable causes the VMM to preserve and restore the state of the EMM "page frame" before and after every EMM access (the EMM page frame is an area in real address space through which EMM data is accessed). This should make Clipper's use of EMM undetectable to any other process using EMM. (Note: on some EMM systems the BADCACHE setting may adversely affect VMM performance. It should only be used if you experience disk or file corruption because of a conflict with a disk cache or other resident software.) Example: SET CLIPPER=BADCACHE ---------------------------------------------------------------------------- 1.2 Disk Swapping If insufficient EMM is available to contain swapped segments, a temporary disk file (the "swap file") is created to hold them. By default, the VMM will use a maximum of 8MB of disk space for the swap file. Disk space is used only as needed. Configuration ------------- The maximum size of the disk swap file can be controlled using the SWAPK setting in the CLIPPER environment variable. Valid settings are from 256 to 64000 (256K to 64MB). The SWAPK setting has no effect on VMM swapping activity; it merely serves to establish an upper limit on the size of the swap file. If the limit is exceeded, the VMM will terminate the application. Example: SET CLIPPER=SWAPK:16000 The disk drive and directory where the VMM swap file is created can be controlled using the SWAPPATH setting in the CLIPPER environment variable. By default, the swap file is created in the current DOS directory. Example: SET CLIPPER=SWAPPATH:"C:\SWAP" ---------------------------------------------------------------------------- 1.3 Swap Space The term "swap space" refers to the region of real memory where the VMM loads virtual memory segments. Configuration ------------- The VMM allocates the swap space from DOS at startup. By default, all available DOS memory is used for swap space. Real memory usage can be restricted using the X setting of the CLIPPER environment variable. The setting specifies an amount of DOS memory (in 1K units) which is to be excluded from the VMM. Valid settings are from 0 to 256. The X setting is provided as a way to reserve DOS memory for other uses (e.g. allocation by memory-resident software). In general, its use is undesirable because limiting the swap space decreases VMM performance. Example: SET CLIPPER=X8 ======================================================================== 2. Object Memory (SVOS) An "object memory" is a special type of memory manager designed to manage complex data values such as character strings and arrays. The Clipper 5.0 object memory is called the Segmented Virtual Object Store (SVOS). SVOS uses virtual memory to store data values, including character strings, arrays, and dynamically created (macro-compiled) code blocks. SVOS provides two important functions beyond the basic capabilities offered by the VMM: 1. Memory compaction: stored values are automatically compacted on an ongoing basis. This eliminates fragmentation of the virtual memory and reduces swapping. 2. Garbage collection: Some Clipper 5.0 values (e.g. arrays) may be referred to by several program variables (or array elements) at the same time. The garbage collection algorithms automatically reclaim space occupied by "unreachable" values (values which are no longer accessible through any variable or array). 3. To minimize delays associated with memory compaction and garbage collection, the SVOS algorithms are dynamically adjusted based on the type, size, and persistence of data being stored, the amount of memory available (real and EMM), and the overall performance of the system. The algorithms attempt to manage memory in such a way that the VMM can swap out infrequently used data, freeing up real address space for time-critical operations. The dynamic nature of the SVOS algorithms makes it difficult to precisely state its maximum capacity. The worst case capacity is in excess of 1MB. The theoretical maximum is in excess of 16MB. ======================================================================== 3. Dynamic Overlaying Clipper 5.0, using the supplied special version of .RTLink, manages compiled Clipper code using a technique called dynamic overlaying. This technique eliminates the need for complicated overlay structures and allows flexible runtime management of compiled code. During linking, .RTLink breaks compiled Clipper modules into fixed- size "pages." These pages are stored either in the executable file or in separate overlay files. The paging architecture eliminates restrictions and memory calculations based on the size of compiled functions or modules. Large modules are broken into multiple pages; small functions are grouped together in a single page. At execution time, the dynamic overlay manager loads pages based on information embedded in the .EXE by the linker. The dynamic pages are loaded into VM segments, allowing the VMM to manage the overlay pages on a competetive basis with other uses of memory. The paging architecture allows the system to discard low-use sections of code even if the code is associated with a pending active function; overlays are not required to "nest" in parallel with function activations. Code pages that are being heavily used are maintained in memory by the VMM's LRU swapping policy. When possible, the VMM will place dynamic overlay pages in EMM, reducing overlay reads. Overlay pages are never written to the VMM disk swap file, however. If a VM segment containing an overlay page is to be removed from memory altogether, it is simply discarded. If it is needed subsequently, it is re-read from the overlay file. ---------------------------------------------------------------------------- 3.1 Configuration In addition to virtual memory, the dynamic overlay manager uses a dedicated area of real memory to cache the most active dynamic overlay pages. The default size of this area ranges from 4K to 16K, depending on the amount of real memory available at startup. For low memory situations, the size of this area can be restricted using the DYNK setting of the CLIPPER environment variable. Valid values are from 4 to 63. (Note: restricting the size of this area may reduce execution speed.) Example: SET CLIPPER=DYNK:4 The dynamic overlay manager will hold overlay pages in virtual memory only as long as it can maintain an active file handle for the disk file containing the pages. If the file is closed, the pages are discarded from virtual memory. The maximum number of file handles used by the dynamic overlay manager can be controlled using the DYNF setting of the CLIPPER environment variable. Valid settings are from 1 to 8. The default setting is 2. For applications which use many separate overlay files, increasing this setting can improve performance. Example: SET CLIPPER=DYNF:4 ======================================================================== 4. Data File Buffering The Clipper 5.0 database drivers use virtual memory segments to contain database and index file buffers. This allows the VMM to manage these buffers on a competetive basis with other uses of memory. As with dynamic overlay segments, the VMM will swap buffer segments into EMM but not to the disk swap file. If a VM segment containing a disk buffer is to be removed from memory altogether, it is written back to the appropriate data file. The file buffering system allocates buffer management tables in fixed memory (see FIXED MEMORY ALLOCATOR below). The actual amount used depends on the combined total of real memory and EMM available at startup. The tables use 14 bytes per 1K of real or Expanded memory (the tables never exceed 8K, however). Because these tables are allocated in fixed memory, they have an effect on the amount of swap space available to the VMM. In some low-memory situations, reducing the amount of available EMM may help to avoid "swap space exhausted" errors. (Note: reducing EMM availability may seriously degrade VMM performance in very low memory situations due to increased disk swapping--see Use of Lim 3.2 Expanded Memory above.) ======================================================================== 5. Segment Locking From time to time, certain VM segments are "locked" by a subsystem. This prevents the VMM from moving or swapping those segments until the subsystem unlocks them. This capability is used sparingly to keep the swap space from becoming cluttered with locked segments. ======================================================================== 6. Fixed-Memory Allocator Clipper 5.0 subsystems use virtual memory to contain code or data that needs to be dynamically managed. Some allocations are not appropriate for VM (e.g. system tables which must remain in the same location throughout execution). A C-style fixed memory allocator is included for this purpose. The fixed memory system allocates one or more VM segments and permanently locks them in real memory. Space within these segments is then parceled out as needed. Since fixed segments are never unlocked, the VMM locates them at the low end of the swap space to prevent them from interfering with other operations. ======================================================================== 7. MEMVAR Tables Clipper 5.0 offers several different storage classes for program variables. Local and static variables are stored in a dedicated area of real memory. Private and public (MEMVAR) variables are stored in VM segments. Since MEMVAR variables are created and destroyed dynamically during execution, the associated VM segments grow dynamically depending on how many MEMVAR variables are in use. For performance reasons, these segments remain locked during most operations (they are unlocked during RUN commands and certain other memory intensive operations). In low memory situations, the MEMVAR tables may have a significant effect on the amount of available swap space (each MEMVAR variable uses 20 bytes in the MEMVAR segments, which grow in increments of 1K or more). MEMVAR variables also require symbol information to be maintained at runtime (see below). Converting private and public variables to local and static variables may reduce memory requirements for some applications. ======================================================================== 8. Symbol Tables Clipper maintains symbol information at runtime in order to support symbolic references to variables and functions. Some symbol information is embedded in the .EXE file (the Clipper 5.0 version of .RTLink eliminates duplication among symbols from compiled Clipper modules). Other symbols are created dynamically at runtime. Memory for symbol tables and related information is allocated using the fixed-memory allocator. The symbols are allocated in blocks and do not fragment memory. However, excessive dynamic symbol creation (e.g. large numbers of MEMVAR variables created via macros) may cause the symbol tables to grow, reducing swap space. ======================================================================== 9. The Extend System The Extend system is a set of functions that are used by C and assembly language programs to interface with Clipper programs. The Extend system includes two memory allocation functions, _xalloc() and _xgrab(), which allow C programs to allocate memory from Clipper. These functions call the fixed-memory allocator (see above). The Extend system also allows C programs to gain access to Clipper data values. The Extend system _parc() function, used to inspect character values passed from Clipper, is defined as returning an address in real memory. Since character values are stored in virtual memory under SVOS (see above), the Extend system automatically locks the appropriate VM segments in response to _parc() requests. The segments are automatically unlocked when the C function returns control to Clipper (they may also be unlocked manually from C, see below). A conflict can occur if a C function locks a VM segment via _parc() and then attempts to allocate a large amount of memory using the Extend allocators. If the segment is locked near the low end of the swap space, the fixed allocator may be unable to lock or expand a fixed segment to satisfy the allocation request. If this occurs, a new fixed segment is created and locked higher in memory. This reduces memory efficiency because the VMM is not allowed to move fixed segments (in severe cases, the VMM may issue a "swap space exhausted" error because the swap space has become too cluttered with fixed segments). To prevent this, the VMM maintains a "safety zone" at the low end of the swap space. _parc() requests are prevented from locking segments within this zone. ---------------------------------------------------------------------------- 9.1 Configuration The size of the Extend "safety zone" can be adjusted using the EXTHEAP setting of the CLIPPER environment variable. The default setting is 8K. Valid settings are from 2 to 256 (units of 1K). Increasing the safety zone allows C programs to allocate large amounts of memory without creating "holes" in the swap space. (Note: the default EXTHEAP setting should be sufficient for most applications; increasing the value too much may cause the VMM to fail when trying to lock a segment in response to a _parc() request.) ---------------------------------------------------------------------------- 9.2 New Extend Function The Clipper 5.0 VMM defines an entry point called _xunlock(). This function can be called from C or assembly language programs to unlock SVOS segments when large amounts of memory are to be allocated after _parc() has been called. CAUTION: calling _xunlock() causes any pointers returned from previous _parc() calls to become invalid. To access a Clipper character value after calling _xunlock(), you must make another call to _parc() to get a new pointer. Nantucket will be publishing an API (Application Program Interface) for the Clipper 5.0 VMM so that C and Assembly language programs can allocate virtual memory directly, reducing demands on swap space. * * *