In my first column for MSJ (October 1993), 1 wrote what I considered a throwaway utility called Below 1MB. This column, which deviates from the usual question and answer format, describes Fix1MB, an update to Below1MB. The purpose of Below1MB was to help you determine which of your own DLLs and drivers were sucking up the precious linear address space below 1MB. Windows[R] 3.x and Windows[R] 95 need a small amount of memory below 1MB for each task it starts up. If it can't allocate the memory, you get a somewhat confusing "insufficient memory to run this program" message, regardless of how much RAM you have installed. When I wrote Below1MB, little did I imagine that it would get noticed outside of programming circles. End users have been using it, and one person even sent me a British general computing magazine with a picture and a short column about Below1MB.
Alas, the low MS-DOS[R] memory shortage is so bad (especially in Windows[R] for Workgroups 3.11) that even end users are grasping at anything that might help, even if they don't understand much of the information provided. Had I known that Below1MB was going to have so broad an audience, I might have written it differently, paying particular attention to reducing details not directly related to the main problem. As you may have guessed by now, I have finally rewritten Below1MB to make it easier to use, and (more importantly) actually help correct the problem. My Fix1MB program should be of interest to programmers for the techniques it uses, as well as end users who are just looking for a quick fix.
The Below One Megabyte Problem
For those of you whose MSJ back issues aren't handy, I'll do a quick overview of the problem in Windows that Fix1MB corrects. Each running program in Windows is a separate task. Each task has a system data structure associated with it known as a task database (or TDB). A task database contains vital information about a task, such as its current directory, its HINSTANCE, and an MSDOS PSP (Program Segment Prefix). The size of each task database is 0x200 bytes (an important number that will come up later.) In Windows 3.x, each task database must be allocated at an address that's below a linear address of 1MB. Why's this? Windows 3.x still uses real-mode MSDOS extensively and many MS-DOS functions need access to a PSP. Since the maximum address accessible in real mode is 1MB (actually, 1MB + 64KB - 16), the PSP stored in each task database must be at a linear address below 1MB.
When Windows starts up, it places any unused MS-DOS memory below 1MB into the global heap. This MS-DOS memory makes up the very lowest portion of the entire Windows global heap. Typically, there may be 300KB in the global heap that comes from this low MS-DOS memory. With 300KB of memory to play with, you should be able to create something like 2400 task databases, which should be plenty even for the most hard-core multitaskers.
The problem with this setup has to do with how Windows allocates global heap memory. Normally when allocating memory, the global heap manager starts its search at the low end of the global heap and stops when it finds the first block big enough. (The exception is when allocating memory for discardable code segments, which causes Windows to start the search at the upper end of the heap. Since the low part of the global heap encompasses the leftover low MS-DOS memory, many global heap allocations end up coming from addresses below 1MB. Unless special precautions are taken, global heap allocation quickly snaps up all the below-1MB memory, making it impossible to create another task database (and hence, start a new task).
Windows keeps the below-1MB memory from quickly being eaten up by making most blocks moveable. Even if a block were originally allocated below 1MB, the global heap manager will move the block up in linear memory to make room for another block that really does need to be below 1MB. While this helps out immensely, there's still the problem of FIXED blocks. The global heap manager is unable to move FIXED blocks, so any blocks that are below 1MB when they're given the FIXED attribute remain there. It is these FIXED blocks that soak up the memory below 1MB and make it impossible to start a new task.
So, how do you get FIXED memory blocks? There are three ways. First, you can allocate memory with GlobalAlloc and give it the GMEM_FIXED flag. Windows ignores the GMEM-FIXED flag when the allocation request comes from EXE code, but honors it when the allocation request comes from a DLL. Second, you can mark your EXE's or DLL's segments with the FIXED attribute. For a now-obsolete reason, some linkers default to making segments FIXED by default. (The solution is to explicitly set the segment attributes in the DEF file.) As with GlobalAlloc, Windows ignores the FIXED attribute when allocating memory for EXE file segments. A segment in a DLL will have the FIXED attribute if the DLL specifies FIXED for that segment.
The third way of getting FIXED memory is to allocate moveable memory and then explicitly give it the FIXED attribute. Both the GlobalFix and GlobalPageLock functions give segments the FIXED attribute. (GlobalPageLock first calls GlobalFix, then pagelocks the segment.) These functions are especially hazardous to the low MS-DOS memory because they attempt to slide the block as far down as possible in memory before they fix it. This can cause blocks that previously were above 1MB to move down into the 1MB area, further compounding the problem.
When you combine the need for task databases to be below 1MB with the global heap allocation strategy, you can see there's going to be trouble in many cases. Many device drivers and system DLLs require FIXED and/or pagelocked memory because they have interrupt handlers or communicate with real-mode MS-DOS-based programs. This is especially true of network drivers. Since most of these drivers load early in the Windows boot process, you can see how a significant amount of the real-mode MSDOS memory required for TDBs can be eaten up. People often tell me that after they start one particular program, they can't start any others. This is almost always because the first program uses some DLLs or drivers that have swallowed up the last bit of memory below 1MB.
In Windows 95, this problem goes away for the most part. Windows 95 changes the way GlobalAlloc works so that (for the most part) FIXED blocks don't get allocated below 1MB. When run under Windows 95, my Below1MB program shows very few FIXED blocks from drivers and DLLs taking up space below 1MB. In addition, task databases don't need to come from below 1MB in Windows 95. However, the PSP has been split out of the Windows 95 task database, and it must still reside below 1MB. To test this out, I slightly modified my original Below1MB program to allocate all the memory in blocks of 0x120 bytes (the size of a PSP in Windows 95). 1 ran this modified Below 1MB under Windows 95 (M8 Preview build) and had it allocate all the memory below 1MB. I then tried to start up a new program. As I suspected, Windows 95 wouldn't start the new task, and gave me an error message about there not being enough memory to start the program.
The Fix1MB Program
To help with the problem of insufficient memory below 1MB, I wrote the Fix1MB program (see Figures 1 and 2). Fix 1MB is really two, two-two programs in one. The first part analyzes the global heap blocks below 1MB and gives you a list of memory blocks that are possibly taking up space unnecessarily. Along with each block, the report names the EXE or DLL that owns the memory block. Armed with this information, you'll at least know the major culprits in stealing your low memory. Based on this knowledge, you might decide to remove a particular device driver or program. You may even be able to press the manufacturer of the offending program or driver to give you an updated version that doesn't use FIXED segments needlessly.
The other part of Fix1MB actually does something about the below 1MB problem. Using sophisticated techniques developed jointly with NASA scientists, it guards the spot where most FIXED memory blocks dive below 1MB and take up residence permanently. By keeping the door to memory below 1MB shut at a critical moment, Fix1MB forces programs and drivers to take their FIXED memory blocks elsewhere. (This is the Windows version of "Not in my back yard.")
If you run the Fix1MB program from within Windows, it prevents all subsequently loaded EYES and DLLs from grabbing FIXED blocks below 1MB (with a few minor exceptions that I'll describe later). While this should help in many cases, your problem may be with drivers and DLLs loaded at startup (such as network drivers). Running Fix1MB normally can't help in this situation. These drivers are already camped out below the 1MB door when Fix1MB starts. However, by loading Fix1MB early enough in the startup sequence, it can usually prevent the offending drivers and DLLs from grabbing significant amounts of FIXED memory below 1MB. Therefore, Fix1MB has an option that allows it to be placed into a strategic spot in the SYSTEM.INI file so that it's loaded fairly early in the startup sequence each time you boot Windows. (This isn't as simple as just putting Fix1MB into the Startup group. I'll describe how it actually works later.) For the SYSTEM.INI load to work, both FIX1MB.EXE and FX1MBDLL.DLL need to be in the same directory. PROCHOOK.DLL should also be in the same directory or somewhere on the path.
Figure 1 shows what Fix1MB looks like. The Potential Free Space field at the
top shows how much memory is available below 1MB. This figure isn't what's
currently free. Rather, it's what would be free if Windows were to pull out
all the stops to move around memory and maximize the space below 1MB.
The list box entitled Potential Memory Hogs is a size-sorted list of memory blocks below 1MB that can't be moved. These blocks are what's taking up space below 1MB. Each line contains a block size, what the block is used for, and the owner of the memory block. Fix1MB doesn't show blocks that you can't do anything about. For instance, KRNL386.EXE (the main system DLL) uses quite a bit of fixed memory below 1MB. However, since you can't do anything about this, why show it? Since filtering out every possible system DLL would be well-nigh impossible, Fix1MB just aims for the biggies (the actual list is in the FIX1MB.C source file). If a block is in the Fix1MB list box, it may be a badly written driver or DLL, it may have legitimate needs for memory below 1MB, or it may be a system DLL that you can't get rid of. To some degree, you'll have to look at the list and make a judgment call about whether you should try and do something about a given line or not. Often, just clearing up the primary offender (if any) may be enough to get you going again.
The Rewalk button causes Fix1MB to redo its analysis of the memory below 1MB. Whenever you restore Fix1MB from an iconized state, it redoes its analysis. However, if programs or DLLs load while Fix1MB is in its noniconized state, Fix1MB's analysis will get out of sync. To get an up to date report, hit the Rewalk button.
The Help button's use should be obvious. The remaining button in Fix1MB is Add FIX1MB to SYSTEM.INI. This causes FIX1MB.DLL to insert FX1MBDLL.DLL into the "drivers=" line in SYSTEM.INI. Once you've done that, subsequent presses of the button result in an error message. The check box at the very bottom changes the way that FIX1MB guards the memory below 1MB. By default, it's unchecked, which gives optimal performance. However, in a few rare cases, the default may not sufficiently guard the low memory, allowing the out-of-memory error to return. If this happens, try checking the box.
Under the Hood of Fix1MB
As I mentioned earlier, Fix1MB is really two programs in one. If you don't care for the heap analysis and my minimalist UI, you can easily extract those portions of the code and recompile it. Still, as it exists now, Fix1MB is pretty small and is helpful in determining what new low-memory-guarding technology has.
The low heap analysis portion of Fix1MB is fairly straightforward, and isn't significantly changed from BELOW1MB.EXE. However, the results Fix1MB shows look quite different, and are much clearer. The WalkHeap function in FIX1MB.C walks the global heap blocks up to the 1MB address limit using the ToolHelp GlobalFirst and GlobalNext functions. As Fix1MB goes through each block, it looks for two things. First, it looks for heap blocks that can't be moved and that the user might have some control over. Second, Fix1MB looks for gaps between the end of one FIXED block and the start of another. The total size of all these gaps is counted as free memory, even if the space is currently occupied by a moveable block. When Windows really does need memory below 1MB (such as for a GlobalDosAlloc call), these moveable blocks are moved upwards in linear memory, removing them from the under-1MB region.
As the WalkHeap routine finds each block, it calls the IsPotentialSpaceHog function that determines the status of the block. If IsPotentialSpaceHog returns TRUE, the block will end up in the Fix1MB list box. IsPotentialSpaceHog immediately throws out free blocks and sentinel blocks (which KRNL386 uses to mark off regions of the global heap). IsPotentialSpaceHog also throws out blocks that don't have the GMEM_FIXED attribute and aren't page locked. (Pagelocked is a superset of the FIXED attribute. Any blocks that make it past these tests are immovable. At this point, IsPotentialSpaceHog looks for a gap between this block and the prior immovable block, and if there is one, adds it to the running total. Before returning TRUE, IsPotentialSpaceHog checks to see if the block is a task database or is owned by a system DLL that the user has no control over. If so, IsPotentialSpaceHog returns FALSE to keep WalkHeap from putting the block into the report list box.
The other subprogram with Fix1MB is a little more complicated. Its basic goal is to allocate all (or most of) the memory below 1MB before the Windows loader or a program tries to allocate FIXED memory. By making no memory free below 1MB, the FIXED memory allocation ends up coming higher up in the address space (remember, in most cases the global heap manager searches for free blocks from the bottom up). After the FIXED memory has been allocated someplace other than below 1MB, Fix1MB frees the memory it allocated previously.
Fix1MB knows when large FIXED memory allocations are about to occur because of the LoadModule routine. LoadModule is the core Windows loader. The majority of allocations of FIXED memory are made within LoadModule when it allocates memory for the code and data segments of EXEs, DLLs, and device drivers. To have a significant effect, Fix1MB merely needs to prevent LoadModule from allocating its FIXED memory from below 1MB. What about other routines that load programs and DLLs? Fix1MB has these bases covered. LoadLibrary and WinExec are both wrappers around a call to LoadModule. The ShellExecute function is a wrapper around WinExec. By watching when LoadModule is called, Fix 1MB can kill four birds with one stone.
Fix1mb monitors when LoadModule is called by using James Finnegan's PROCHOOK.DLL from the January 1994 issue of MSJ. I briefly debated using breakpoints to keep Fix1MB entirely self-contained, but the ease of use of ProcHook won out in the end. Fix1MB installs a hook on LoadModule, and specifies Fix1MBLoadModule in FIX1MB.C as the callback function. Inside Fix1MBLoadModule, the code temporarily unhooks the LoadModule callback, allocates all the memory below 1MB, and then calls the original LoadModule. After LoadModule returns, Fix1MBLoadModule frees all the memory it previously allocated and then reinstalls the LoadModule hook. By calling LoadModule directly, Fix1MB implicitly knows when it's safe to free the allocated memory. This would be harder to determine if the hook simply chained the call onto the original handler. In addition, LoadModule is recursive, so unhooking the callback and calling LoadModule directly lets Fix1MB ignore all the nested calls to LoadModule. This way, Fix1MB only does the allocations on the primary call to LoadModule and doesn't have to do anything for the nested LoadModule invocations.
When allocating all the memory below 1MB, Fix1MB stores the blocks in a linked list. The first WORD of each allocated block contains the handle of the next allocate block. The AllocTiledBelow1MBMemory allocates on memory below 1MB by calling GlobalDosAlloc. To quickly exhaust the memory below 1MB, the function allocates 32KB blocks in a loop until GlobalDosAlloc fails. Then AllocTiledBelow1MBMemory cuts the block size in half and jumps back to the top of the loop. This continues until all the blocks of more than 0x200 bytes have been allocated. (Remember, 0x200 bytes is the size of a task database. After allocating all the blocks of more than 0x200 bytes, the function frees the first block in the list. This gives LoadModule at least one block of sufficient size to create a task database.
It's important to note that this algorithm isn't foolproof. For instance, you could have a scenario where after the allocations, the global heap below 1MB has only one free block, 0x200 bytes in length. If an EXE (or one of its DLLs) has a FIXED segment that's of 0x200 bytes or less, that block will be allocated for the segment, and there won't be any space left over for a new task database. In situations like this, the task creation fails with the "insufficient memory to run this program" message. One solution that usually works in this case is to allocate all low MS-DOS memory in chunks of 0x200 bytes and then free every other block. This should leave plenty of blocks big enough for a task database. Unfortunately, allocating this many small blocks eats up a lot of selectors and noticeably slows program load time. For this reason, this isn't the default allocation scheme. However, you can force Fix1MB to use this method by clicking on the "Worst case mode" check box.
Another hole in the Fix1MB programs is dynamically allocated memory. A program can allocate moveable memory with GlobalAlloc and then GlobalFix or GlobalPageLock it. Since Fix1MB doesn't monitor those calls, KRNL386 could move the block below 1MB when Fix1MB isn't looking. From my observations, memory that's FIXED in this manner is rarely the primary user of memory below 1MB. If you want to close this loophole, you could add additional ProcHook callbacks for GlobalFix and GlobalPageLock, and use the allocation/deallocation scheme that I use for LoadModule.
An interesting problem in writing Fix1MB was what to do about DLLs and drivers that are loaded early in the boot sequence. Since these DLLs come in before Fix1MB could be fired up by PROGMAN's Startup group, a bit of tactical planning was necessary. By studying the module list of Windows (which is kept in module load order), I was able to locate the first DLL loaded via an entry in the SYSTEM.INI file. In my case, it was MMSYSTEM.DLL, which is on the "drivers=" line in SYSTEM.INI's [boot] section. My first approach was to stick FIX1MB.EXE before MMSYSTEM.DLL on the "drivers=" line. Unfortunately, this didn't work. Windows refused to load the executable file (apparently it only wants to load DLLs on that line). My second thought was to put the Fix1MB LoadModule hooking code in a DLL. However, I didn't want to create two versions of the same basic code. Finally, I had the idea of creating a small DLL that would simply WinExec Fix1MB.EXE in the DLL's LibMain. Figuring I had nothing to lose, I gave it a try. To my surprise, it worked! Thus was born FX1MBDLL.DLL.
When loaded via FX1MBDLL.DLL, Fix1MB can significantly increase the amount of memory available below 1MB. In my machine (which has only a couple of device drivers), loading Fix1MB via SYSTEM.INI still saved me roughly 80KB. I'd expect that on more heavily loaded machines, it would save even more. As another test, I fired up WinFax Pro 3.0 (the latest version I have), both with and without Fix1MB running. Without Fix1MB, starting WinFax Pro took 71KB out of my memory below 1MB. With Fix1MB running, WinFax Pro took only 2KB.
What if badly written programs break by using Fix1MB? These problems would be due to a faulty assumption in the other program, not a flaw in Fix1MB. Earlier, I described how FIXED executable file segments often end up being allocated below 1MB. Some programs need memory accessible by MS-DOS (below 1MB) and assume that their FIXED segments will always be below 1MB. While this is usually the case (at least not when Fix1MB isn't running), it's not absolutely required. Since Fix1MB forces most FIXED segments above 1MB, programs that rely on FIXED memory coming from below 1MB are in for a big surprise. The proper way for these applications to allocate their memory would be to use GlobalDosAlloc.
I thought about adding a "Hall of Shame" feature to Fix1MB. The feature would recognize when LoadModule is about to load one of these faulty apps and temporarily disable itself. While a cool idea, I didn't want Fix1MB to get too big. I don't like making accommodations for buggy apps. I'd rather they be flushed out and replaced with working versions. If Fix1MB does cause a certain application to act strangely or crash, try exiting Fix1MB before you start the application in question.
