|
Volume Number: | 12 | |||
Issue Number: | 10 | |||
Column Tag: | Rehosting Development Tools |
Hosting Under Metrowerks
by Farooq Butt and Drew Sarkisian, Motorola RISC Software
Introduction
In an earlier article (“SPM, Be My Host”, MacTech v12, No. 6, June 1996) we talked about hosting Motorola’s highly optimizing mcc C/C++ compiler under the Symantec Project Manger. Since that article, the Motorola RISC Software Compilers and Tools Group has created compiler plugins for the popular Metrowerks CodeWarrior IDE. This article shares our experiences creating these plugins. We hope to help others of you out there facing a similar challenge and jump-start your efforts.
Motorola has offered highly optimizing C, C++ and FORTRAN compilers for the MacOS since May 1995. These compilers were initially hosted only in the Macintosh Programmer’s Workshop (MPW) environment, a UNIX-like command-line driven shell. Even though our compilers currently generate the highest performance code in the Mac development arena, we felt the need to provide mcc’s technology in a friendlier wrapper than that afforded by the MPW programming environment. This basic need was the impetus behind our efforts to first port our compiler into the SPM (Symantec Project Manager) and subsequently to the Metrowerks CodeWarrior IDE. For a further description of mcc and its background, please refer to our earlier article.
Project Background
As stated earlier, our CodeWarrior port came after our Symantec port. The grizzled UNIX-heads who had not even known what a resource-fork was prior to porting mcc to the SPM, were quite a bit more savvy the second time around! We decided to not only do a compiler plugin, but also a full preference panel plugin too. This would give the paying public the ability to use the familiar Mac point-and-click interface to send options to mcc instead of using a command-line facsimilie as in the SPM port (the SPM port has also been enhanced with a preference panel).
Nearly all of the work that was done hosting mcc as a CodeWarrior plugin was done at Motorola in Austin, TX. Even though Motorola was one of the very first consumers of their CW8 plugin API, Metrowerks proved to be a supportive and agile partner and provided invaluable technical help all along the way. As of the CW9 release of the API, we feel that Metrowerks’ documentation and examples have progressed to the point that prospective plugin writers should not require any special intervention from Metrowerks.
The Important Issues
We covered these major issues in our last article, but we thought it would be valuable for new readers to re-cap the basic differences between tools that are driven from the command-line and those driven from (Macintosh) IDEs. Some of these differences are obvious and visible to the user of the compiler, while others are of concern only to the engineers who develop and maintain the compiler.
Command line typed by a user . Command-line compilers accept their arguments via a command line typed by a user or generated by a Makefile. On the Mac, these tools are generally MPW tools that take their command lines from an MPW shell. Compilers hosted within IDEs have visual (and more intuitive) interfaces implemented via dialog boxes, preference panels etc.
Allocation and freeing of system resources. Compilers from UNIX (or Mac’s MPW environment) often do a poor job of freeing all of the heap memory they allocate. Instead, they rely on the system (or MPW shell) to reclaim all memory resources when the compiler process terminates. This is a recipe for disaster when running under an IDE: memory will leak until the IDE (or system) crashes. These compilers may also have the bad habit of leaving open file descriptors lying around, causing the IDE to run out of usable file descriptors. Simply stated: command-line based compilation systems often assume that each executable component will be launched by a shell and when its particular job is done, it will be cleaned up after.
Initialization of static data.. Upon each invocation of a command-line compiler, the system loader will initialize all of the compiler’s static variables to either 0 or pre-set values determined at compile-time. A compiler hosted under an IDE may need to remain memory resident, and might have to explicitly re-initialize its own static data between compiles.
ANSI C file I/O functions. Non-Macintosh compilers often use fopen(), fread(), etc. when processing source files for compilation, producing temporary files, and generating resultant object files. Such compilers deal exclusively with ANSI file IO abstractions such as pointers to FILEs, etc. An IDE, on the other hand, typically uses different mechanisms for reading/writing files. A related issue concerns how a compiler is to locate source files and headers files. The most recent copy of a file could be open in an IDE editor window. When the user asks for that file to be compiled, an IDE-hosted compiler has to know that the file in question is in memory, and not on disk, and access it accordingly.
Error messages and warnings. When a programmer presents some syntactically-incorrect program to a command-line compiler, the compiler presents the error messages directly to the terminal (console, window) that processes the command-line. This is not very useful when running under an IDE since there is no “console”, the compiler, therefore, needs a mechanism for communicating errors to the IDE in such a way that the IDE can present the errors using it’s errors/warnings window.
Cooperative multitasking. Something often overlooked when porting software to the Macintosh is the fact that the MacOS is a cooperative multitasking environment. This means that you must find a way to share processing time between your drop-in and the IDE it is running in. If you don’t, the users will have no way of interrupting a long build, or performing other tasks while a compile is taking place. This facet of the MacOS is an interesting challenge when initially porting a compiler.
Metrowerks Codewarrior IDE
CodeWarrior is an open environment and developers can port their own development tools to run under it. Metrowerks not only talks the talk but also walks the walk: all of their own compilers and tools are implemented as plugins using their own API. The fact that Metrowerks would use their own plugin API for their bread and butter products went a long way towards convincing us of its maturity and usefulness.
CodeWarrior plugins come in three flavors: compilers, linkers and preference panels; they are stored in: <CodeWarrior Folder>:Metrowerks CodeWarrior:CodeWarrior Plugins. New plugins are created and stored in this folder under three subfolders named (obviously): Compilers, Preference Panels and Linkers. Every time CodeWarrior is invoked, these folders are searched for new plugins and if any are present, they are added to the list of available plugins shown to the user.
We obviously needed to build a compiler plugin but the first question that came up was that of object file format. CodeWarrior compilers use a proprietary (as yet unpublished) object file format1. In order to better support third-party plugin compilers, Metrowerks does provide a standalone XCOFF Importer plugin which reads and converts any XCOFF object file and associated stabstrings (XCOFF symbolic-debug format) into the Metrowerks proprietary internal object format. Since the object format the mcc compiler emits is XCOFF and we wished to use the standard Metrowerks linker plugin, we needed this importer. This tool is normally supplied as a plugin, but for our purposes, Metrowerks supplied the XCOFF importer to us as a library that could be called from mcc. This turned out to be a smooth and easy to use tool and (short of a few glitches with some ornery stabstrings) was a transparent utilty. Metrowerks will supply the tool as a library to anyone who asks and is, as we write this, considering adding XCOFF support to their linker which will obviate inclusion of the XCOFF Importer library.
Metrowerks also provides the ability to plug in a third-party linker. Since the aforementioned XCOFF converter tool gave us the ability to convert our XCOFF object format to the CodeWarrior internal format, we relied on the standard linker that is supplied with the product. In the future, however, we are considering adding a new plugin linker that would provide some extra functionality (we have discussed the possibility, for example, of having a linker deal with profile-directed feedback).
Finally, Metrowerks, provides the ability to create preference panel plugins. This means that the plugin developer can provide a set of dialog boxes, pull-down menus etc. that are completely customized to their particular product. We decided early on in this project that we would implement a more intuitive interface than a bare command line.
The CodeWarrior Plugin API
Metrowerks provides a “plugin API” for aspiring plugin writers. This API includes all header files and libraries that are required to create each kind of plugin and is, for the most part, a pretty easy way to interface to CodeWarrior’s internals. The documentation about this API is located on the <CodeWarrior Reference CD>:CodeWarrior Documentation:CW Plugin API [doc] folder.
The API itself is located in <CodeWarrior Tools CD>:Other Metrowerks Tools:CW Plugin API and contains the following subfolders:
Sample Plugins: Contains sample plugins for a compiler, a linker and a
preference panel
Test Project: Contains a project that utilizes the sample compiler,
linker and preference panel
Plugin Support: Contains the API headers and libraries
Plugin API Release Notes: The release notes for the current API
The API documentation describes (in not too great detail) how to create a plugin and specifically what resources need to be present in order to have it function properly under the plugin API. We cannot stress too strongly how important it is for plugins to contain exactly the resources in exactly the form specified in the API documentation. A good place to get familiar with these resources is the <CodeWarrior Tools CD>:Other Metrowerks Tools:CW Plugin API:Sample Plugins:Sample Compiler folder which contains the full source and resources for a toy compiler. To get started, we simply copied the Sample Compiler.rsrc and Sample Compiler.c file from the above folder and tailored them for our own purposes. The resources contained in our mcc plugin are detailed in the following listing (for a detailed overview about what these resources are supposed to contain, please consult the API documents):
Listing 1: mcc Plugin Tool’s resources
resource ‘STR ‘ (128, “MCC Name”) { “Motorola MCC “ }; resource ‘STR ‘ (129, “MCC CPU”) { “PowerPC” }; resource ‘STR#’ (128, “MCC Panels”) { { /* array StringArray: 1 elements */ /* [1] */ “MCC Panel” } }; resource ‘STR#’ (11001, “Linker Strings”, purgeable) { { /* array StringArray: 40 elements */ ...error strings deleted }; resource ‘Targ’ (128, “MCC Driver”) { kCurrentResourceVersion, { /* array CPUs: 1 elements */ /* [1] */ PowerPC }, { /* array OperatingSystems: 1 elements */ /* [1] */ MacOS } }; resource ‘EMap’ (128, “MCC Driver”) { kCurrentResourceVersion, { /* array Mappings: 1 elements */ /* [1] */ text, none, dontPrecompile, isLaunchable, notResourceFile, handledByMake, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved } }; resource ‘Flag’ (128, “MCC Driver”) { kCurrentResourceVersion, Compiler { CW8API { generatesCode, doesntGenerateResources, canPreprocess, cantPrecompile, isntPascal, cantImport, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, reserved, CPPLanguage } } }; resource ‘Fmly’ (128, “MCC Driver”) { kCurrentResourceVersion, ‘moto’, “Motorola Preferences” }; };
On the whole, we found the API to be pretty straightforward. The documentation could have been better, especially for those of us coming from a non-Mac environment. We believe seasoned Mac programmers will catch on pretty rapidly. It’s also worth mentioning that Metrowerks has taken pains to maintain upward compatibility with each release of the plugin API. We found that the tools we had written to the CW8 plugin API performed flawlessly under the latest CW9 API. As this API matures, we hope to see what minor annoyances there currently are get ironed out. For example, the current API allows plugin writers to encache and decache header files using API callbacks, a feature we feel is unnecessary and sometimes dangerous to allow API users to do manually.
Restrictions on Plugin Tools
A CodeWarrior plugin acts much like a normal program; however, there are a few restrictions when one uses the CW plugin API:
• Never call the ANSI exit(), assert() or abort() routines from your plugin! This will terminate the CodeWarrior IDE rather than merely terminate your plugin.
• Do not use any console I/O. If you do use console calls via printf() or fprintf(stdout,...) etc. you’ll get one attempt: console I/O via packages such as SIOUX or SIOW wrecks CodeWarrior’s window management after the first session. If you need to dump out informational messages use the CWCompErrorRefMessage() callback to send information to the error window.
• CodeWarrior does not currently support multi-process plugins. This pertains to some tools that consist of two or more separate executables that are called in sequence to perform different stages of the compilation process. Multi-process executables must be transformed into libraries (preferably shared libraries) that contain entry points into the various phases which are called by a master driver.
How Plugins Work
Once you have prepared all the resources you’ll need for a prospective plugin, the next phase is actually coding up the skeletal framework that all CodeWarrior plugins need. The IDE operates on the idea of requests and callbacks. Callbacks cover commands sent from your plugin to the IDE. These generally take the form of API calls such as CWCompErrorMessage() and are detailed in the documentation. Requests, on the other hand, generally cover commands sent to a plugin from the IDE. CodeWarrior expects to be able to give the following requests to any drop-in compiler:
reqInitCompiler: plugin is to perform any needed initialization.
reqTermCompiler: drop-in is to clean up any system resources it may
still have allocated.
reqCompile: plugin is to start compilation.
How does the IDE communicate these requests over to the plugin? Your plugin must have a main() entrypoint which accepts the following argument:
pascal short main(CompilerParameterBlockPtr cpb)
The CompilerParameterBlockPtr argument is the master control structure that everything in the IDE depends on. This is laid out as follows:
Listing 2: The Compiler Parameter Block
/* parameter block - this is passed to the dropin at each request */ typedef struct CompilerLinkerParameterBlock { /*******************************************************/ /* Common to all plugins */ /*******************************************************/ long request; /* [->] requested action */ long version; /* [->] version # of shell’s API */ void *context;/* [->] reserved for use by shell */ void *storage;/* [<->] reserved for use by the dropin*/ FSSpec targetfile;/* [->] FSSpec of current project */ long numfiles;/* [->] number of files in project */ long whichfile; /* [->] number of file being compiled */ long sequenceid;/* [<-] used internally by Metrowerks */ long reservedcommon[8];/* reserved for future extensions */ /*******************************************************/ /* specific to compilers */ /*******************************************************/ FSSpec sourcefile;/* [->] FSSpec of file being compiled*/ Handle sourcehandle;/* [->] handle to source text */ long sourcehandlesize; /* [->] length of source text */ Booleanprecompile;/* [->] are we precompiling? */ Booleanautoprecompile; /* [->] is precompile part of a Make? */ Booleanpreprocess;/* [->] are we preprocessing? */ BooleanSYMinfo; /* [->] is SYM Info enabled for this file? */ Booleancaching_includes;/* [->] is IDE caching include files? */ CWBrowseOptions browseoptions; /* [->] enabled browse information */ Boolean recordbrowseinfo; /* [->] record browse info for this file? */ short browserfileID; /* [->]fileID to use in browse records*/ Handle objectdata;/* [<-] handle to enerated object code, resource data, or preprocessed text*/ Handle browsedata;/* [<-] handle to browse data */ long codesize;/* [<-] size of generated code */ long udatasize; /* [<-] size of uninitialized data*/ long idatasize; /* [<-] size of initialized data*/ long compiledlines; /* [<-] number of lines of source compiled */ short compilechoice; /* [->] used internally by Metrowerks */ Handle symboldata;/* [<-] used internally by Metrowerks */ unsigned long crc; /* [<-] used internally by Metrowerks */ void* browsercontext; /* [<-] used internally */ long reservedcomp[16];/* reserved for future extensions */ /*******************************************************/ /* specific to linkers */ /*******************************************************/ Handle texthandle;/* [<->] handle to disassembly text */ Booleanuptodate;/* [<-] should project be marked up-to-date? */ TargetInfo targetinfo; /* [<-] returned on reqTargetInfo*/ long reservedlink[16]; /* reserved for future extensions */ ... deleted full listing of callbacks... /* version 3 API */ CWCompilerCallbacks*callbacks; }
We thought it would be helpful for readers to actually see how all this plays together. Listing 3 is the actual main() entry point for our compiler. There is not much more to our plugin other than setup/cleanup and a call to the compiler itself. The compiler is implemented as a shared library with one exported entry point, namely this main() routine. We highly recommend this approach since it cleanly separates the system-level housekeeping work from the task of compilation itself. The basic flow of execution under this model is illustrated by figure 1 below.
Figure 1: Execution Flow Under the Codewarrior IDE
Listing 3: mcc Plugin’s main() entrypoint
pascal short main(CompilerParameterBlockPtr cpb) { short result; int argc=0; char *argv[5]; MAINPTR the_entry; /* set up global world (68K only) */ EnterCodeResource(); result = noErr; #ifndef __powerc CWCompErrorMessage(cpb,”68K host not supported”); #else /* dispatch on compiler request */ switch (cpb->request) { case reqInitCompiler: establish_CW_homedir(); /* compiler has just been loaded into memory */ break; case reqTermCompiler: /* Do nothing here */ break; case reqCompile: /* compile a source file */ /* Since the compiler has static data, it must be reloaded and its static data re-initialized every time */ the_entry = Map_In_MCC(); if (the_entry == NULL) { result = 0; CWCompErrorMessage(cpb, “Can’t load mcclib plugin or one of its components”); goto end_of_driver; } else { chdir_to_temporary_folder(); /* The compiler thinks 0 means success */ result = (*the_entry) ( cpb); if (result == 0) result = 1; else result = 0; /* Mark this compiler’s static data as re-initializable */ Unload_MCC_Fragment(); chdir_to_CW_homedir(); } break; default: result = paramErr; break; } #endif end_of_driver: /* return result code & tear down global world (68K only) */ ExitCodeResource(); return (result); }
This listing may seem very simple, but that’s all there is to it! Other than some tricks we had to play with Apple’s help having to do with re-initialization of static data in the compiler (the magic behind the Map_in_MCC() and Unload_MCC_Fragment() calls), there wasn’t much more to getting a simple plugin working. If your compiler/tool is reasonably well-behaved with static data (i.e. there are very few static initializations) you will not have much trouble in making it work as a plugin. However if, like us, you have vast numbers of statically initialized variables, you will have to be somewhat clever in reloading your .data section’s initialized members. Since there are quite a few ways to skin that cat, ranging from reloading the entire compiler every time to setting up private heap zones for initialized data, we’ll leave it as an “exercise for the reader.”
Dealing with Options
mcc is a tremendously configurable product. From optimization-levels to optimization-types to target architectures, there are command-line options that do it all. Since we did not want the task of building dialog boxes for these options interfering with our initial porting efforts, we simply hardwired a good set of command-line options into our initial mcc port and started there. This was achieved by simply preloading the ANSI argc/argv command-line parameters manually in mcc via a routine named setup_argc_argv_pair(). When we did eventually implement a full GUI-based options processing subsystem, we used the very same routine to query the dialog-boxes/panels etc. The example provided in the <CodeWarrior Tools CD>:Other Metrowerks Tools:CW Plugin API:Sample Plugins:Sample Panel folder is a very good place to start looking at what it takes to get a preference-panel working.
System Resources: Heap Memory
After statically initialized data has been dealt with, the issue of what to do with dynamic memory also comes up. Many UNIX compilers are, unfortunately, pretty lax when it comes to dynamic memory management. Many take the approach of “allocate all you want, the OS will clean up after you when you exit().” Since IDE-based compilers never really “exit” the IDE, this laxity is no longer allowable. All dynamic memory must be freed up after use so that, given no other tasks eating up memory, the next call to the plugin has the very same amount of memory available to it has the previous one did. This can be done in several clever ways: private heap zones, very careful programming and “special” malloc()/free() routines, to name but a few. Private heap zones involved dipping our feet into high Apple arcana (not our strongest suit), and careful programming seems to us to be an oxymoron; we decided to go with the specially written versions of malloc()/free()/realloc() etc. together with a routine called at compiler termination time that would deallocate everything in one fell swoop. While this was a bit of a pain to write and debug, it eventually turned out to be a great solution since we can now instrument mcc’s use of dynamic memory using our special hook functions and gather some interesting statistics.
Installing Your Plugin
Once you have written your plugin, you will probably want to give it a spin. Here are some things to bear in mind:
1. Make sure your resources are set up correctly.
2. Make sure that your plugin has a Type of ‘Comp’ and a Creator of ‘CWIE’.
3. Make sure that your plugin’s preference panel does exist and has the same name as mentioned in the plugin’s resources.
4. Copy the plugin into the <CodeWarrior Folder>:Metrowerks CodeWarrior:CodeWarrior Plugins:Compilers folder.
Once this is done, CodeWarrior (upon startup) will include the new plugin in the list of compilers that can be utilized for your project. Choose the Edit menu item, then choose Preferences followed by the Target preferences. The Target preferences item will let you associate a compiler with an extension or a filetype. To pick mcc, we simply change the association of the extension .c from “Metrowerks C/C++” to “Motorola mcc.”
Hints and Gotchas
Before concluding, we’d like to go over some areas that we feel will help the reader get going with the job of getting a functional CodeWarrior plugin. These hints and techniques are in no particular order and do not form an exhaustive list of everything you’ll need, but we feel that they are useful tips.
Filesystem Issues
mcc deals with three types of files when performing a compilation: source files, temporary files, and object files. The CodeWarrior IDE rovides abstractions for dealing with all three types of files that fit in very well with the overall mcc source code.
Sourcefiles
The CodeWarrior IDE locates sourcefiles by the callback CWCompFindIncludeFile().This routine is used for both header files as well as for regular sources. The routine takes 8 parameters in the following manner:
extern pascal OSErr CWCompFindIncludeFile(CompilerParamBlkPtr cpb, ConstStr255Param filename, Boolean fullsearch, Handle* source, long* sourcesize, FSSpec* filespec, Boolean* alreadyincluded, Boolean isprefixfile);
cpb: The compiler parameter block that is used by the IDE
filename: A Pascal string containing the name of the file you are searching for
fullsearch: Do we look in system headers’ path if we cannot locate this file in the local search path?
source: A Handle to the file’s bytes if it is currently open in a text window somewhere
sourcesize: The above Handle’s size
filespec: The FSSpec of the file if found and not open in a text window somewhere
alreadyincluded: Has this file has already been included prior to this time?
isprefixfile: Is this file the prefix file that goes in front of all others?
If you are wondering why there is no mention being made of pathnames above, recall that the IDE has implicit user-modifiable search-paths for both user files and system files. These are applied, based on the value of the fullsearch flag, to every file that the IDE is told to search for. Thus one tells the IDE to get a file “foo” and it does the dirty work of actually locating it.
Temporary Files
Like many other compilers, mcc uses temporary files during the compilation process. In the UNIX environment, these files reside on disk, typically under the directory /tmp, and exist only while the compiler is running. When mcc finishes, it cleans up all temporary files associated with that particular compilation. Under the current release of mcc on CodeWarrior, we decided to store all temporary files in the <System Folder>:Temporary Folder.
Temporary files were created and destroyed by using standard ANSI routines such as fopen(), fclose() , fprintf() and unlink(). Since this was already what mcc had been doing, it just took a little tweaking to get compiler temporaries to appear in the correct place. The Metrowerks ANSI libraries worked without a hitch.
Object Files
Thus far, we’ve been able to open text files for compilation, and manipulate temporary files for use during compilation. How do we manage to hand object files back to the IDE ? Well, given that we had a library version of the XCOFF Importer tool, we simply read the object file into a handle using FSRead() and subsequently hand it to a routine called ImportXCOFF() which translates it into the CodeWarrior proprietary internal object format. As we write this, rumors are afoot that the Metrowerks linker will soon start to digest XCOFF as well as the CodeWarrior object format.
That Funny Business with Carriage Returns
Source file-lines located on UNIX systems are terminated with 0xA, while source-lines on the Mac are terminated by 0xD. In order to work on a Mac, UNIX sourcefiles may need to have their line termination changed from 0xA to 0xD. This is achieved by the API callback CWCompFixTextFileHandle() which takes sourcefile Handle and converts it from UNIX-style termination to Mac-style.
Conclusions
Porting mcc into the CodeWarrior IDE was a useful experience. Motorola increased the visibility of its mcc compiler many-fold in the Macintosh universe. While not trivial, we feel (especially with the latest plugin API) that the task of plugging in a tool into the CodeWarrior IDE is fast becoming pretty straightforward. The mcc compiler was one of the first UNIX type tools to be plugged in and as such provided a stress-test for the plugin API. After a few understandable glitches, we feel the API proved itself as a solid, high-performance substrate upon which to build. We encourage all of you out there with language-tools that could be hosted in the Macintosh environment to “plug ‘em in.”
Credits
First, we would like to heartily thank Fred Peterson and John McEnerney of Metrowerks who helped us through many, many tough areas. Kudos go to Greg Galanos for always being responsive and quick in helping us actually ship this product. Thanks are also due to Ira Ruben and Sassan Hazeghi of Apple Computer for helping us wade through CFM arcana.
On the Motorola side, we’d like to thank Matt Moss for his patient and cheerful Macintosh help whenever we needed it, Scott Osborne for pushing this beast through the testing cycle, Mike Phillip for keeping us focused and the entire Motorola RISC Software Compilers and Tools team for constantly improving mcc and our other development tools.
The PowerPC name and the PowerPC logotype are trademarks of IBM Corp. and are used under license therefrom. All other trademarks and registered trademarks are the property of their respective owners.
1 Metrowerks has informed us that they will make details of their PPC and 68K object file formats available if this is helpful to compiler plugin writers.
- SPREAD THE WORD:
- Slashdot
- Digg
- Del.icio.us
- Newsvine