Date: June 11, 1999
NOTE: In the following document, the string "shelf_x.y" represents the actual pathname of the directory into which you installed SHELF. For example, if you install shelf in /opt/shelf_1.3, then "shelf_x.y/src" represents /opt/shelf_1.3/src on your machine.
Introduction
How do I create a project directory? What should it contain?
What is the control file?
What is the build file?
How do I build the project?
How do I make the project directory part of the build?
How do I make changes to the source?
How do I add new files to the base code?
How do I add a new built-in ELF function?
What if my project depends on other project directories?
How do I access help?
What are all these extra files?
What are all these extra functions?
How do I add help?
Where are the German, French, Japanese versions?
Where do I go for support?
What is a Task?
How does the ELF interpreter work?
What are the do's and don'ts of memory management?
GUI Package Manager & Install program
Conversion away from Applix widgets
Conversion away from Applix threads
Mail client
Better ELF connection to shell
Improve amake
Improve Object Elf (Builder) syntax
Prune the code of Applixware-only functions
You can add functionality to SHELF or edit existing functionality simply by providing the new or changed source code and rebuilding SHELF. The source code is placed in a "project directory."
There is a lot of complex information in this document. Don't panic. If you are developing an ELF or Builder application for distribution to existing SHELF users, you don't have to do any of this stuff -- just develop your software and send the source ELF (.am) or Builder (.ab) file(s) to the users.
If your open-source project uses C files (which therefore must be built by users on the platform which SHELF runs), or extends or alters the behavior of SHELF itself, read on.
All project directories are created in the "delta" directory. The "delta" directory is at the same level as the "src" directory. For example, if you are working on a Matrix Multiplication project you may create the following directory:
% mkdir shelf_x.y/delta/MatrixMultiply
By convention, project names are mixed case with leading uppercase. Please limit project names to 32 characters. The name "Base" is reserved. You should also avoid project names already in use; see the web site under "Projects" for more information.
The project directory must contain the following files:
While your version file must be at the top level of your project directory,
you are free to choose whether or not to organize your project directory into
subdirectories. In general, a good rule of thumb would be to stick with a
single level unless the total file count exceeds 40 files.
When you are ready to publish your project directory, contact info@applix.com. Applix will assign your project an official delta identifier and make it available on the Applix web site as soon as feasible.
The control file provides certain information for the build process, identifying the project and SHELF version information, the build file, and the absolute location of the project directory.
Here is a sample control file:
% cat /applix/delta/MatrixMultiply/MatrixMultiply.ctl
/* MatrixMultiply by John Doe */ /* Contact jdoe@company.com for support */ /* Here is the versioning information for the project */ /* VERSION 12 */ /* USES Base 2.0 */ /* Status message during build */ /* Don't put any comments after this point */ .tell Matrix Multiplication functionality extension project major=AOS_OPENSRC_MAJOR_VERSION minor=AOS_OPENSRC_MINOR_VERSION .locate AOS/opensrc/shelf$(major)_$(minor)/delta/MatrixMultiply .buildfile MatrixMultiply.amk
The first section of the control file consists of whatever comments you wish to include. It is optional.
The second section identifies the version of this project and specifies the version number of the SHELF base code used by the project. It is required. Version sequences begin at 1 and can be integers or floating-point numbers (as a project creator, you can use the fractional portion of version sequence to indicate a minor fix, or stick to integers). The example specifies that this is version 12 of the MatrixMultiply project, requiring (at least) version 2.0 of the SHELF base code.
The third section emits a status message (the string after the .tell command) during the build process. It is optional.
The fourth section locates the absolute project directory for the build process. It is required. Note that the AOS_ flags are defined in the .defs files in shelf_x.y/src/axbuild.
The fifth section identifies the build file for the build process. It is required.
The build file specifies what is to be built, where it should be placed, and the source files that make up the build.
Here is a sample build file:
% cat /applix/delta/MatrixMultiply/MatrixMultiply.amk
/* MatrixMultiply by John Doe */ .component MatrixMultiply .global MatrixMultiply.nop .locate exec mmult src_files=mmult.c mminterface.c obj_files=^resuffix .o $(src_files) compile $(src_files) into $(obj_files) partial_link $(obj_files) into mmult.a final_link mmult.a elfapi.a into mmult .locate elf mmult_.am copy_file mmult_elf.h into mmult_.am nop mmult mmult_.am into mmult.nop
The .component line identifies this project as a component distinct from the rest of SHELF, and is required.
The .global line identifies MatrixMultiply.nop as a target accessible by the rest of the SHELF build. This is discussed in more detail below; it is optional but recommended.
The .locate line indicates that mmult (an executable created in the build) will be located in the "exec" directory (shelf_x.y/exec/[d]axdata). See the .defdir directives in shelf_x.y/src/axbuild/opensrc.ctl for a list of directories; you can add additional directives in your own control file. All final outputs of your build should be placed via a .locate directive.
The next section defines a set of source files and object files. Note the syntax of assignment to a variable: no spaces before or after the equal sign; spaces (not commas) separating multiple items. Access to a variable uses the syntax $(variable_name). The continuation of long lines is done by inserting a backslash (\) as the last character of all but the last line. The example illustrates the use of the resuffix tool to generate a list of objects matching the source files; for a small number of source files, explicitly defining the object files will work fine.
The next section compiles the sources into objects and then links the objects into a library.
The next section links the library and elfapi.a into the executable mmult. Note elfapi.a is a SHELF library that allows SHELF to communicate with mmult. See the ELF Communications Guide (available from Applix) for more information.
The next section creates an ELF "include" file that the project author thinks will be of assistance to users of this project (e.g., it contains ELF structures and flags used in the interface to mmult.) The file is located in "elf" (shelf_x.y/exec/[d]axdata/elf), and is simply copied there. Files should be renamed when copied; the build system cannot deal with two files of the same name in different locations.
The final section uses the nop (no-operation) directive to create a zero-length file mmult.nop "using" mmult and mmult_.am. In order for mmult.nop to be created, the two input files must be successfully created. The user therefore builds mmult.nop to force everything else to be built. Note that the build system implements date-based dependencies in a top-down manner. Thus, if mmult.nop is older than mmult or mmult_.am, it will be created. But before this happens, both mmult and mmult_.am are examined. If either one of these files does not exist, or is older than one of its inputs (e.g., mmult is older than mmult.a or elfapi.a), the item is created, and so forth. By this manner, if you were to edit (for example) mmult.c and build mmult.nop, the following items would be rebuilt in this order: mmult.o, mmult.a, mmult, an "include" file that the project author thinks will be of assistance to users of this project (e.g., it contains ELF structures and flags used in the interface to mmult.nop.)
Unlike the control file, the build file will vary wildly depending on the type of project you are working on. It will probably be helpful to look at a variety of build files, both those shipped with the base SHELF code (look in shelf_x.y/src/axbuild) as well as projects created by others (see www.applixware.org)
As the delta directory may contain multiple project directories, you must indicate to the build logic which directory contains the "current" project. To indicate the current project directory, you must set the environment variable AOS_DELTA_BUILD to the directory name, or pass it as a switch on the amake command line.
% setenv AOS_DELTA_BUILD MatrixMultiply
% shelf_x.y/amake mmult.nop
or
% shelf_x.y/amake -DAOS_DELTA_BUILD=MatrixMultipy mmult.nop
In this example, "MatrixMultiply" is the actual directory name of the project; that is, shelf_x.y/delta/MatrixMultiply contains the project source (directly or under further subdirectories); MatrixMultiply.amk defines mmult.nop as the global target that will cause everything else to build.
Important note: if you intend to distribute the project to current SHELF users, this may be all you have to do. You can now deliver the project directory to these users and have them build the project just as you did.
If you wish to more tightly integrate your project into SHELF, there is a little more work to do. You can modify an amake script to build your project.
To make this happen, you must edit a SHELF build file. Do the following:
nop <some stuff> \ <a long list of more stuff> \ into shelf
Now a build of the shelf target will also build your project. Pass the -icache
flag to indicate to the build procedure that the location of source files
needs to be redetermined. As above, you must set the AOS_DELTA_BUILD
environment variable or pass the AOS_DELTA_BUILD
flag to the build command. This indicates to the build procedure that the opensrc.amk
file in your project directory overrides the file in the base SHELF directory.
By using this method, you can build a customized version of SHELF that integrates your project, and redistribute the entire SHELF to other users.
Don't forget to add a comment to the top of the edited file indicating the date of the change, as required by the LGPL license.
A more complex project than that described above may require the alteration of additional SHELF files. For any given source file, simply copy it to your project directory and make your edits. In your next build, use the -icache command line option in amake. This option causes amake to check the datestamps of the source directories and note when new files come into existence in the project directories.
% cd shelf_x.y/delta/MatrixMultipy
% cp shelf_x.y/src/elf/elio.c .
% emacs elio.c
% shelf_x.y/amake -icache [-DAOS_DELTA_BUILD=<your project>] shelf
Don't forget to add a comment to the top of the edited file indicating the date of the change, as required by the LGPL license.
Again, you can either specify your project directory on the build command line, or as an environment variable. This will cause the edited files in the MatrixMultiply directory to override those in the base SHELF src directories. The build process will report a warning if the version number in the header of the copied file does not match the version number in the file being overridden, such as:
*** elio.c versions don't match File shelf_x.y/delta/MatrixMultiply/elio.c: 100 Overrides shelf_x.y/src/elf/elio.c: 101
As new versions of SHELF are released -- if you intend to continue to add your project directory on top of SHELF -- you must pay attention to such warnings and reapply your edits to the newer versions of such files. In this example, a new function may have been added to elio.c. Unless your edited copy also provides the same function, a problem will occur when SHELF attempts to use that function.
Note that if you remove a file from your project directory, you must use the -cache command line option to inform amake to fully recompute the locations and the dependencies of all files (i.e., find the file in the shelf_x.y/src directory again.)
Note: please do not edit SHELF code "in place" (in the base SHELF src directories) and then redistribute SHELF to others. Such an action will create a version of SHELF that Applix will not be able to support; in addition, your changes will be lost if the users upgrade the base SHELF code. All changes should be made by copying the files into project directories and editing them there.
When you want to add a new file to the build, you will need to edit the opensrc.amk and the dopensrc.amk files. At this time, these files are machine-generated by Applix, so their content is less than pleasant to read. The best bet is to find a file that is of a similar category and replicate its instances.
Using the example above, let's say you want to add a new file, elaudio.c, which contains new sound code. You decide that elio.c seems to be of the same category. You see that elio.c occurs twice in the opensrc.amk file:
compile elio.c into elio.o
compile elio.c into eeio.o with -DFOR_TESTER -DTHIM_STATIC dep elfsyms
The first compilation directive is pretty clear -- it invokes the standard compiler with the standard compilation flags on elio.c to produce elio.o. The second compilation rule is less clear. It specifies that you want to compile elio.c into eeio.o with two new flags. Further investigation reveals that elio.o is used twice -- once to produce a library and once to be linked into elfmain:
archive cppelfmain.o ... elio.o ... into elfapi_cc.a
final_link_c++ elxwins.o ... elio.o ... into elfmain with -laxel -laxdb
The eeio.o file, on the other hand, is only linked into the stand-alone ELF compiler (elfc), which was also used as a stand-alone basic test program in the early days (thus explaining -DFOR_TESTER). The -DTHIM_STATIC switch controls whether to use threaded or non-threaded ("thim_static") calls.
final_link eetest.o ... eeio.o ... into elfc
You determine that there is no need for your code to be part of the stand-alone elf compiler or the general purpose library, but it clearly needs to be part of elfmain. [elf.so]
Add a new line to the opensrc.amk:
compile elaudio.c into elaudio.o
Add elaudio.o to the list used in opensrc.amk to produce elfmain [elf.so].
Now repeat the exercise for dopensrc.amk.
Many ELF built-in functions are routed through one of several dispatching functions. The "king daddy" dispatch function is UTIL$$. This section does NOT discuss how to add to UTIL$$. Instead, this section describes how to add a "core" built-in function.
Let's assume you want to create a function, matrix_multiply@. You will need to edit two files:
When you edit ebnmgen.dat, add
your function name to the end!
(ELFM_MTRX_MULT)MATRIX_MULTIPLY@:MATRIX_MULTIPLY@
The ELF compiler, when processing built-in functions, uses a numeric string index derived from its position in the table. Your function entry should be a logical name in parenthesis, followed by the name of your function twice, with the two instances separated by a colon. The name of the function is entered twice because the ebnmgen.dat was designed to allow for the localization of built-in functions. But Applix opted not to do this -- German programmers use the same function names as American programmers. It currently contains two languages -- English and Gothic. This file will soon become obsolete.
After you have added your function to the end of ebnmgen.dat, add it to the end of the ElfCoreFuncs table in elstatic.c.
{TYPE_C_FUNC,ELFM_MTRX_MULT,AxfMatrixMultiply},
Note that the line is added before the final {0, 0, 0} line. Your called function gets a single Elf array argument, which is the sum of the arguments passed by the calling function. Your function returns a new Elf datum or NULL.
If it is a simple function, just add it to elb.c -- there are already hundreds there!
data_ptr AxfMatrixMult(data_ptr args) { data_ptr retval, m1, m2, size1, size2; retval = NULL; m1 = AnyFromArray(args,0); size1 = ElfbArraySize(m1); /* throws error is m1 not an array */ size2 = ElfbArraySize(m2); /* throws error if m2 not an array */ retval = /* do the work here */ return(retval); }
See the Axf functions in elb.c for MANY examples.
Note that with the core code, we tend to use the typedef data_ptr. Is the same as typdef elfDatum in the published documentation.
Note that the return values must be created. If, for example, you just want to return one of your arguments, you must use the following:
return(AxCopyData(arg));
Try to use reasonable care in allocating memory after calling any functions that may throw an error. As long as you use the task-based allocator, your memory will be automatically freed when the task terminates, but that may take a long time!
As people undertake and complete projects, Applix intends to merge these projects back into the base SHELF code. However, not all projects may be appropriate for such a merger. It is possible that your project will need to use functionality from a project that has not yet been merged. This is easy to do.
First, copy the project directory underneath your delta directory.
Second, edit the control file in your own project directory to reference this other project. For example, suppose your MatrixMultipy project depends on certain functionality in the BasicMath project, which has not been merged into the base SHELF code. The BasicMath source code should be placed in shelf_x.y/delta/BasicMath. Suppose the control file in this directory indicates the BasicMath sequence number is 5. Your Matrix Multiply control file would look something like this:
% cat /applix/delta/MatrixMultiply/MatrixMultiply.ctl
/* MatrixMultiply by John Doe */ /* Contact jdoe@company.com for support */ /* Here is the versioning information for the project */ /* VERSION 12.1 */ /* USES BasicMath 5 */ /* USES Base 2.0 */ /* Status message during build */ /* Don't put any comments after this point */ .tell Matrix Multiplication functionality extension project major=AOS_OPENSRC_MAJOR_VERSION minor=AOS_OPENSRC_MINOR_VERSION .locate AOS/opensrc/shelf$(major)_$(minor)/delta/MatrixMultiply .buildfile MatrixMultiply.amk
This file specifies version 12.1 of the MatrixMultiply project, which requires version 5 of the BasicMath project and version 2.0 of the SHELF base code. The base code version specification should always come last in the version list. If multiple projects are specified, the order of this list becomes important. If foo.c exists in multiple projects, the project specified earlier in the list has priority. For example, given the following version list:
/* VERSION 12.1 */ /* USES AdvancedMath 3.1 */ /* USES BasicMath 5 */ /* USES Base 2.0 */
If there is both a shelf_x.y/delta/AdvancedMath/foo.c and a shelf_x.y/delta/BasicMath/foo.c, the former will be used in preference to the latter. The current project (e.g., MatrixMultiply) always has highest priority; the Base source always has lowest priority.
Some caution must be used when your project depends on other projects. When you override a source file in your project, clearly the functionality in your source file must be a superset of the other project. Similarly, if you depend on multiple projects with "clashing" files, you will have to resolve conflicts with files in your own project directory.
Note that the version file definition is not recursive -- all required deltas must be enumerated. In other words, if the BasicMath project itself requires some other project, you must reference it (in MatrixMultiply.ctl) too.
In Applixware, ELF help uses the standard Applixware help mechanism. Unfortunately, this is based on the Applix word processor, which is not part of the open source release. We are in the process of making all the ELF-related help available in HTML, but we haven't completed the process.
Until the process is completed, you will need Applixware for some of the online help.
If Applixware is already running on your machine, ELF will dynamically connect to it automatically and cause it to display the requested help.
You can specifically direct ELF to start Applixware for help, if it is not already running. To do this, set the helpStartApplixware to "-1" in your profile, $USER/axhome/ax_prof4. If Applixware is not in your $PATH, you must also set the helpApplixwarePath profile.
In general, using Applixware to display the online help results in better performance and a more polished and powerful interface than the HTML scheme.
ELF was developed as part of Applixware. The line between ELF files and Applixware files is not always obvious. Applix did not want to risk releasing an open-source ELF that lacked required functionality, dialogs, or images. Therefore, we included excess files in the build.
As time goes by, these excess files will be removed from the build.
Currently, the Applixware source tree contains over 20,000 source files. The open-source Linux source tree contains about 3,000 files. We are getting there!
ELF was developed as part of Applixware. Many of the "ELF" files included in the open-source release invoke functions associated with the Applix word processor, spreadsheet, or graphics tools. Likewise, as with any mature product, the source tree also contains "dead code" that is never called anymore.
As time goes by, these excess functions will be removed from these files.
Anybody know of a good cross-referencing tool for Linux?!
Applix and ELF have always been Unix-centric. In the early 1990s, Applix started a porting process on pre-release NT systems. At that time, the porting path for Applix required us to use shorter names and smaller words. Developers creating new features need not think small. Please feel free to create filenames of any reasonable length!
Since ELF help is closely tied to the Applixware help mechanisms, you have two choices:
See the code in dialog.els.
We are working on this and expect to have these versions integrated into the open-source release and build logic very soon.
For links to free discussion/support mailing lists, go to www.applixware.org.
In addition to the free public forums, the Applix Professional Services Organization offers, for a fee, consulting services for the creation of ELF/Builder programs. We also expect to offer consulting for developers wishing to enhance the core ELF engine and/or integrate it into their products.
In Applix-ELF parlance, a "task" is what the rest of the world calls a "user thread." Applix threads (tasks) were developed before threading became commonly available. The ELF threads are not preemptive. In other words, an ELF thread "owns" the process until it makes an explicit thread-releasing call. The principal thread releasing calls are delay@(), db_display@(), and the socket communications calls.
The elfmain scheduler changes tasks by swapping stacks. In many other thread implementations, only the stack pointer itself is swapped. Applix chose to perform an actual stack copy/save to allow for an arbitrary stack size -- a feature usually lost on other threadings systems. In reality, as the stack swap tends to occur on small stacks, the performance difference is negligible.
When the elfmain schedular swaps threads, it also copies/saves the value of the global variable Elf.
ELF tasks can be made preemptive by explicitly marking the source code with @@@ TIME_SLICE and @@@ NO_TIME_SLICE markers to signify thread-safe code. The time constant must also be set for the task by calling time_slice@(milliseconds).
For example:
macro my_program non-safe code time_slice@(100) ' time constant 100 ms. = 1/10 sec more non-safe code @@@ TIME_SLICE while true arthmetic-code wend @@@ NO_TIME_SLICE more non-safe code endmacro
Internally, threads are tracked by two numbers. The first, AxInTask is a simple number (1-500) that is used as an index into TaskInfo[]. At the ELF programmer's level, task ids are unique values starting at 1000.
Applix plans to convert to system threads in the future.
The ELF interpreter emulates a straightforward multi-register computer. The major core routine is ElfRun() in elintr.c. This routine is a 1600-line infinite loop that only returns when the top-level function has completed, or an unhandled error is thrown.
The interpreter performs arithmetic using a stack. For example, to add two numbers, the first number is pushed onto the stack. Then, the second number is pushed onto the stack. The ADD operator pops the two numbers, performs the addition and pushes the result onto the stack. [Yes, this could be simplified -- are you volunteering?]
The subroutine call stack is implemented in the heap. Thus, when ELF code makes a subroutine call, a stack frame is allocated using malloc(). The return instruction is simply a stack discard and branch. This heap-based stack system allows ELF interpreters to run on small-stack architectures.
ELF is a dynamic-linking language. This means when ELF first calls a given
function, it calls it by name. A
link-resolving routine (Snapper())
finds the function and overwrites the linkage pointer with the function
address. Thus, this name searching occurs only for the first call to a given
function. When a new elf package
(e.g., foo.elo or myprog.abo)
is loaded, all links are restored to the "unsnapped" state.
Object ELF (Builder) is a very dynamic object language. All methods are resolved at runtime. In Object ELF, objects actually contain their own code and inherit code from other objects. It is possible to add code to an object at runtime. Likewise, it is possible to change the inheritance of an object at runtime.
Be advised that if you add your function to virtually any file that includes aos.h, malloc() and free() will be #defined to task-based alloc and free. This was a poor convention that Applix will eventually move away from. To be safe in any code you write, call TrueMalloc() and TrueFree() rather than malloc() and free().
Memory can be allocated on a per-thread or system basis. Any per-thread memory is automatically freed when the thread terminates. In general, per-thread allocations should be used wherever possible to reduce the likelihood of memory leaks. The principle per-thread allocation calls are TaskAlloc(thread,size), TaskFree(thread,ptr), TaskRealloc(thread,ptr), and so forth. System memory is part of task 0. Thus, TaskAlloc(0,size) performs allocations that are never automatically released.
Red Hat have been successful in establishing their Red Hat Package Manager as the de-facto package installation and management utility for Linux. However, like Sun's pkgadd, RPM yields a less than exciting user experience. It is predominantly command-line driven with many options and switches normally provided at the command line. A lightweight GUI tool called glint ships with Red Hat and provides mainly management and query options about which packages are installed.
We are investigating creating an "InstallShield" application in SHELF which not only provides a GUI front end to get information about which packages are installed, but also provides a compelling installation screen with features such as:
We believe that a fully open source installation process can become the de-facto standard for software installation.
Applixware and ELF use a non-standard widget system. Applix widgets are directly layered on Xlib rather than Xt or any other standard widget model. We made this choice in 1989 before Motif 1.0 was available. Our original plan was to use Motif, but the poor performance and the lack of reliability of the 1.0 Beta and 1.0 FCS versions caused us to move away from them.
At this time, Applix would be very interested in partnering with developers interested in replacing the widget system with GNOME widgets and/or other standard systems.
Applix threading is reliable and works across many platforms. Unfortunately, it limits developer choices for integration into products that use other threading models. We believe for it to become a well-received open-source library, it should integrate into the threading package chosen by the application.
Applix engineers have begun working on a PINE-based mail client that provides a rich user interface via ELF.
While ELF is a good language for building GUI-intensive applications, it is very weak for simple shell programming. The next extension for ELF is to improve its ability to launch programs and perform IO redirection.
amake was written as an internal Applix tool 15 years ago. During that period, it has been "hacked" and polluted.
amake should either be retired or cleaned up. The original author believes the issues it addresses still exist and that an "ideal" version of amake would be very well received for many open source projects. Tell us what you think!
If amake were to be re-written, it would include a much more dynamic model for build rules. Currently, build rules are wired into a table in am_bld.h and implemented in am_rules.c. A publicly useful approach would be dynamic libraries bound to an on-rule invocation.
As the very first step, amake should become fully ANSI-C compatible and use system prototypes from system include files.
Object ELF suffers from some syntax confusion resulting from the over-usage of the "." character. A research project has begun to investigate a less ambiguous syntax for method invocation.
This is a cleanup effort that Applix employees will handle. Anyone who knows of a good cross-referencing tool should contact us!