Read Me About MPFileCopy

1.0b1

This sample demonstrates a) how to copy a folder and its contents using the HFS Plus APIs, and b) how to call the File Manager from MP tasks on Mac OS 9.0 and above. The copy engine preserves long Unicode names, larger than 2 GB forks, and forks beyond the resource and data forks (if present).

This sample requires Mac OS 9.0 or above.

Read the Caveats section for some important caveats.

Packing List

The sample contains the following items:

• ReadMe.html -- This document.
• MPFileCopy.mcp -- A project file for the sample.
• MPFileCopy.c -- Source code to the sample.
• MPFileCopy-PPC -- A compiled version of the above.
• MoreIsBetterParts -- The parts of the DTS sample code library MoreIsBetter that are required to compile this sample.

Using the Sample

To use the sample, simply launch it and first choose a source item and then choose a destination folder. The program will then copy the source items to the destination folder, displaying some console window progress as it goes.

Building the Sample

The sample was built using the standard MoreIsBetter build environment (CodeWarrior Pro 2 compiler) with Universal Interfaces 3.3. You should be able to build the project in CodeWarrior Pro 4 without difficulty as long as you have a late version of Universal Interfaces 3.3 or later. To build the project, select the "C-PPC" target, and choose Make from the Project menu. This will build the MPFileCopy-PPC application.

How it Works

Copying a folder is a surprisingly difficult thing to do, although the HFS Plus APIs make it somewhat easier. I wrestled with the following key issues while working on this sample.

• Efficiency -- As a rule, File Manager calls are expensive, especially when running over a high-latency network link. I wanted to avoid making more File Manager than absolutely necessary. To this end, I was careful to use FSGetCatalogInfoBulk (introduced by the HFS Plus API), and to avoid getting catalogue information more often than necessary.
• Stack Growth -- The copy engine uses a recursive algorithm. I wanted to guarantee that I wouldn't run off the end of my stack, especially in the MP task where I am responsible for explicitly requesting the stack space I need. For details on the solution, see the comments for the kCopyFileTaskStackSize constant in the source.
• Asynchronicity -- While Mac OS 9.0 allows you to make File Manager calls from MP tasks, it doesn't guarantee a good user experience if you do so, specifically on some older machines with synchronous-only I/O buses. See the "True Async Tests" comment in the source for details.
• Transfer Buffer Size -- File Manager is always more efficient when you use large transfers. However, on traditional Mac OS the File Manager is single-threaded. A large transfer on a slow link can take a long time, and thus starve other File Manager clients. See the "Transfer Buffer Size" comment in the source for my take on solving this problem.
• File Sharing Bug -- I found and worked around a bug in File Sharing in Mac OS 9.0. For details, look for the bug number, 2397324, in the source code.
• Multiple Forks -- The code copies all forks of all items it copies, including forks associated with a folder. This significantly complicates the code, especially when dealing with drop folders (see below). I also wanted to avoid the extra overhead associated with multiple forks in the most common case (items that have just a resource and a data fork). Testing this support is also tricky because no current file systems support forks beyond the classic resource and data forks. To test the code, I throw a switch, kSpecialCaseClassicForks, which forces the code to treat the resource and data forks as if they were named forks, which exercises many different code paths.
• Locked Files -- Copying locked files is tricky because you have to create the destination file unlocked, copy the forks, and then lock the destination file.
• Modification Dates -- Copying items without changing their modification dates requires copying the item and then setting the modification date after the copy is finished.
• Marking Items as Busy --While copying items, you should mark them as busy so that the Finder won't process them until you've finished copying. I do this by setting a file's type to a special value (kFirstMagicBusyFiletype) and any items creation date to another special value (kMagicBusyCreationDate). Once I've finished copying the item, I reset these values.
• Drop Folders -- Copying items into drop folders is tricky. Firstly, the rules for drop folders require that you open all the forks of the file before you start writing to any of them. This is tricky when there is just a resource and data fork, but gets really nasty when you have to deal with multiple forks. Also, when copying into a drop folder, you can't change an items file type, locked status, or dates after any fork in the file has content, so many of the techniques described above don't work properly. The upshot is that items copied into a drop folder are always unlocked, have the current time as their modification date and can't be marked as busy. This is the same behaviour as the Finder.

Caveats

This sample still has a number of problems. I shipped with these problems because I thought it was more important to get the sample out the door than to make it perfect. I will attempt to address these in a future release.

• Performance versus Responsiveness -- The current code is neither as fast as I would like it to be, nor does it allow the system to respond well to user actions while a copy to a fast volume from a preemptive task. I can improve the performance by increasing the transfer buffer size, but that makes the responsiveness worse. I'm not sure what the right solution is here.
• Performance versus Finder -- The code currently copies somewhat slower than Finder 9.0. Increasing the transfer buffer size brings me almost to par with the Finder for copying a small number of large files, at the cost of the lowered system responsiveness described above. Also, Finder's copying code beats this code in two important aspects. The upshot is that this code would require significant tuning to match the copying performance of Finder.
  • My current algorithm for copying (copying one item at a time via a recursive descent of the folder hierarchy) is inherently slower than the Finder's algorithm (read items until you fill the transfer buffer, then write items until the transfer buffer is empty) for large numbers of small files on the same volume. This is hard to fix without a complete rewrite of the code.
  • Finder bypasses the File Manager to overlap I/O when copying between a local and an AppleShare volume. The technique for doing this is not documented to third parties, so can't be used in this sample.
• Carbon -- The code compiles for Carbon but I don't have a Carbon-compatible version of MSL with which to link it. Long term, MSL will be available for Carbon, but I wanted to get this "out the door".
• CarbonLib and Synchronous I/O -- I discussed the problems with asynchronicity above, along with my solution. That solution is not possible under Carbon because some of the required APIs are not Carbon-compatible. Under Mac OS X this isn't a problem (all drivers will be asynchronous on Mac OS X), but if you build a Carbon version of this code and run it on traditional Mac OS on some older machines, you will encounter this problem again.
• Destination Names -- The sample does not make the name of the destination item unique. This is tricky code correctly. I have the code lying around (in Pascal, as part of Internet Config) but I haven't had time to integrate it here.

Credits and Version History

If you find any problems with this sample, mail <DTS@apple.com> with "Attn: Quinn" as the first line of your mail and I'll try to fix them up.

1.0d1 (Oct 1999) was an Apple internal release for reviewers.

1.0d2 (Oct 1999) was another Apple internal release for reviewers.

1.0b1 (Nov 1999) was the first shipping version.

Share and Enjoy.

Quinn "The Eskimo!"
Apple Developer Technical Support
Networking, Communications, Hardware

15 Nov 1999