ftp.robelle3000.ai 2014/2014.06.ftp.robelle3000.ai.tar/ftp.robelle3000.ai/papers/qxport.pro

Text File | 1994-05-03 | 50KB | 1,457 lines

.select(1:49)
.comment    File:     QXPORT
.comment
.comment    Purpose:  Porting Qedit from MPE to HP-UX Paper
.comment
.comment  To print this document, do the following:
.comment
.comment    :run printdoc.pub.robelle;info="QXPORT"
.comment
.com {ifout starts here}
.comment  Choose the output device parameters for this document
.if outfinal
.  if outrecord
.    out(las 30 c1 r+ s7).com Robelle bound version, attached
.  else
.    out(las 30 c1 r- s7).com Robelle bound version
.  endif
.elseif outhelpcomp
.  out(lpt q+ w80).com helpcomp;parm=1
.elseif outa4
.  if outlpt
.    if outrecord
.      out(lpt s7 r+).com A4 paper, $Stdlist/LP/disc, attached
.    else
.      out(lpt s7 r-).com A4 paper, $Stdlist/LP/disc
.    endif
.  elseif outlaser
.    if outrecord
.      if outdouble
.        out(las 30 s8 r+ d+).com A4 paper, LaserJet, attached, duplex
.      else
.        out(las 30 s8 r+).com A4 paper, LaserJet, attached
.      endif
.    else
.      if outdouble
.        out(las 30 s8 r- d+).com A4 paper, LaserJet, duplex
.      else
.        out(las 30 s8 r-).com A4 paper, LaserJet
.      endif
.    endif
.  else.com No other outxxx jcws specified
.    out(lpt s7 r-).com generic: A4 paper, $Stdlist/LP/disc
.  endif
.elseif outtext
.  if outrecord
.    out(lpt s2 u- r+)
.  else
.    out(lpt s2 u-)
.  endif
.else
.  if outlpt
.    if outrecord
.      out(lpt s3 r+).com Letter, $Stdlist/LP/disc, attached
.    else
.      out(lpt s3 r-).com Letter, $Stdlist/LP/disc
.    endif
.  elseif outlaser
.    if outrecord
.      if outdouble
.        out(las 30 s7 r+ d+).com Letter, LaserJet, attached, duplex
.      else
.        out(las 30 s7 r+).com Letter, LaserJet, attached
.      endif
.    else
.      if outdouble
.        out(las 30 s7 r- d+).com Letter, LaserJet, duplex
.      else
.        out(las 30 s7 r-).com Letter, LaserJet
.      endif
.    endif
.  else.com No outxxx jcws specified
.    out(lpt s3 r-).com Letter, generic: $Stdlist/LP/disc
.  endif
.endif
.comment
.comment  Choose the fonts for this document
.if outfinal
.  include final.qlibdata.green
.else
.  include f92286f.qlibdata.robelle
.endif
.comment
.comment  Choose the margins for this document
.comment
.if outhelpcomp
.  mar(r78)
.elseif outa4
.  mar(r62)
.else
.  mar(r65)
.endif
.comment
.comment  Choose the forms for this document
.comment
.comment  Form 1 is for the body of the manual
.comment  Form 2 is for unnumbered pages (end of section)
.comment  Form 3 is for the table of contents (roman numerals)
.comment
.if outtext
.   form(k1 [L8000] )
.   form(k2 [L8000] )
.   form(k3 [L8000] )
.elseif outa4
.   form(k1 [ T #23 S:40 // l58 / #33 "-" pn:1 "-" /]
+           [ S #23 T:40 // l58 / #33 "-" pn:1 "-" /])
.   form(k2 [ // l58 / #33 pr:3 /]).com Roman numerals
.   form(k3 [ // l57 // ])
.else
.   form(k1 [ T #26 S:40 // l55 / #33 "-" pn:1 "-" /]
+           [ S #26 T:40 // l55 / #33 "-" pn:1 "-" /])
.   form(k2 [ // l55 / #33 pr:3 /]).com Roman numerals
.   form(k3 [ // l54 // ])
.endif
.comment
.comment  Option settings:
.comment
.comment   For text output we want a ragged right edge.
.comment
.if outtext
.   opt(j4 p+ b+ r-)
.else
.   opt(j4 p+ b+).com     Okay to insert four blanks between words
.comment                  when justifying, two spaces after period,
.comment                  suppress blank lines at top of page
.endif
.comment
.comment  Other formatting parameters:
.par(f` p5 s1 u3)      .com  Automatic .skip 1.page 5.undent 3
.inp(u~ b@ h\ e& t# f|).com   Underline,Blank,Hyphen,Escape,Tab,Font
.page.count 1          .com  Start page numbering over again.
.form 3

.jump 7
.opt(l- r- f-)
`|3Porting Qedit from MPE to HP-UX|

|1By David J.@Greer|


`|1Abstract|
.opt

Robelle recently ported its Qedit full-screen editor from the MPE
environment to the HP-UX environment. David Greer, manager of
software R&&D at Robelle, headed up the team doing the migration.
This paper describes the problems they faced and the resulting
solutions. Qedit is written in SPL and was ported to HP-UX using
the SPLash! compiler from SRN. The paper covers both the gory
details of how Robelle does cross-\development using SPLash! and
the theoretical problem of how to make UNIX look like MPE. Most
of the Qedit source code is identical for the two systems, but
the MPE file system differs radically and is more complex than
the UNIX file system.

.skip 14
.opt(l- r- f-)
.font 1
Robelle Consulting Ltd.
Unit 201, 15399-102A Ave.
Surrey, B.C. Canada V3R 7K1
Phone:  [phone]
Fax:  [phone]

Email:  david_greer&@robelle.com
.font 0


Copyright Robelle Consulting Ltd. 1994


Permission is granted to reprint this document (but ~not~
for profit), provided that copyright notice is given.


.opt.com Stop centering
.com                         The next FORM is for Contents/Preface:
.form 1
.contents(i+3)
.page.count 1
.jump 1
.opt(l- r- f-)
`|3Porting Qedit from MPE to HP-UX|

|1By David J.@Greer|


`|3Introduction|
.ent `Introduction
.opt
.tit Porting Qedit to HP-UX
.sub Introduction


Our Qedit full-screen text editor was first written in 1977 and
has been continually enhanced since then. We recently ported
Qedit from MPE to HP-UX.
This paper discusses our research strategy and the
technical details of how we actually managed to port approximately
100,000 lines of code that had been written over the last fifteen
years.

`|1Original UNIX Goals|
.ent Original UNIX Goals

For a few years, our customers (especially in Europe) had
started doing development on UNIX machines.  We didn't
know anything about UNIX, but we decided that we had
better start learning.
Our goal was to learn as much as we could about UNIX by seeing how
much of our source code and programs we could convert to
HP's version of UNIX:  HP-UX.

`|1Why HP-UX|
.ent Why HP-UX

We decided on HP-UX for a couple of reasons:
.mar(l+3).par(u3)
`1.#+1It was made by HP, so it was likely to be more familiar to
us than another vendor's UNIX (and at least the Response Center telephone
number would be the same).  Time and again, this proved to be true and
greatly helped our development effort.
`2.#+1Qedit is written almost entirely in SPL/SPLash!.
Because of our compiler technology, we wanted to focus on
the HPPA RISC architecture.
For this reason, we restricted our research to HP's 700 and 800
series of computers (they have HPPA RISC CPUs).
`3.#+1We did
not want to rewrite all of the source code into C
(the usual language of choice for UNIX programs).
We felt that rewriting Qedit into C would be too large an R&&D project
for Robelle. Automated tools for doing the conversion were reported to be
ineffective.
.mar.par

`|1Major R&&D Phases|
.ent Major R&&D Phases

Our HP-UX research was done in different phases.  We
had a short-term goal for each phase and worked towards completing
this goal in a reasonable time frame.  The major phases were as follows:
.mar(l+3).par(u3)
`1.#+1|1Basic Research|.  In this phase we borrowed an HP-UX machine
and investigated HP-UX object-code and run-time libraries.
`2.#+1|1MPE and Robelle Libraries|.  Robelle's software
development is based
on a concept of layers that build on each other.  This phase
involved getting our own machine, writing necessary MPE replacement
routines, and porting our fundamental
subroutine libraries.
`3.#+1|1Rebuild Qedit|.  Once the basic libraries were running we
needed to look at Qedit.  In this phase, we reengineered Qedit so that
it has little knowledge about which computer platform it is
running on.
`4.#+1|1Polish and Release Qedit/UX|.  After the first version of
Qedit was working on HP-UX, we needed to do a series of
enhancements to make Qedit work in a more UNIX-like manner.  We
also needed to complete all the details that turn an R&&D project
into a product.
.mar.par

`|1Personnel|
.ent Personnel

Three people worked on porting Qedit to HP-UX.
.if outtext
.   mar(l+17).par(u17)
.else
.   mar(l+11).par(u11).page 8
.endif
`|1Dave Lo|#+1Dave was the general programmer's helper.  He was
assigned specific research tasks (e.g., how to get the value
of an environment variable in SPLash!).
`|1Robert Green|#+1Bob is the original author and chief architect
of Qedit.  His main job was to keep Dave's and David's spirits up and
to reengineer Qedit so that it didn't depend on MPE any more.
`|1David Greer|#+1I was the overall project leader.  I ported most of
our subroutine libraries, learned everything I could about
programming interfaces to HP-UX, and implemented all of the
Qedit/UX operating system interface.
.mar.par

`|1Development Diaries|
.ent Development Diaries

At Robelle, we keep a diary with summaries of each day's
development effort.  We kept a development diary for HP-UX -- much
of the material in this paper is drawn from it.  It takes work to
keep these diaries up-to-date, but they pay off in the long-term
by saving useful information.

`|1Time and Effort|
.ent Time and Effort

From first inception to delivering a production product took about
20 months.
The work effort is estimated as 19 person-months.
We feel that this is incredibly fast for the amount of
work that needed to be done.  We could never have achieved this
if we had rewritten all of our software into a new language as well
as for a new operating system.

We estimate that a complete rewrite of Qedit into C would have taken
at least twice the time (four years) and three times the
work effort (at least 60 person-months).  A rewrite goes against
our long-held belief in doing things in small incremental steps,
rather than striking out for the big win.

Another disadvantage of rewriting Qedit into C was that we likely
could not have used the C version on Classic 3000s, due to
poor C compiler support on the Classic 3000.  However, we always try to
make available as many Qedit enhancements as possible to all of our
HP 3000 customers.  This is one reason why we have
one set of source code for all versions of Qedit.
.sub Software Development
.page.jump 1
.opt(l- r- f-)
`|3Software Development At Robelle|
.ent `Software Development At Robelle
.opt


To understand how we ported Qedit from MPE to HP-UX, it is
important to understand how we develop software at Robelle. Most
of our development, including Qedit, is done in SPL and SPLash!.
SPL was the original Systems Programming Language for the HP 3000.
The SPL compiler only produces object-code for Classic HP 3000s
(you can also run this code in compatibility-mode on MPE/iX machines).

SPLash! is a compiler from Software Research Northwest
(206-463-3030) that translates SPL source code into
HPPA RISC object-code.  We use SPLash! on MPE/iX to produce the native-mode
version of Qedit.  There is no SPLash! compiler for HP-UX.

SPL has no run-time library (which is similar to how the
C language works).
In Pascal, you would write something to the screen as shown below:

.font 5.page 4
     writeln('Hello, World!');
.font 0

In SPL, there is no language construct that will print a string
on your screen.  Instead, you move the string to a buffer and then
call a library routine to print the string:

.font 5.page 4
     move outbuf := "Hello, World!";

     print(outbuf,-13,0);      ! print is a library routine
.font 0

The print routine displays the thirteen characters on the screen.
Print is a library routine that comes with MPE.

`|1Subroutine Libraries|
.ent Subroutine Libraries

Because SPL does not rely on a language-specific run-time library,
we thought it might be possible to port the SPLash! object-code
to HP-UX running on the HPPA RISC architecture.
The only thing left would be to replace all of the
subroutine libraries on which Qedit depends.  Since Qedit depends on
hundreds of routines, that was still a lot of
potential work.

Qedit is built upon a series of subroutine libraries.  Each set of
libraries depends on other sets of libraries.  The libraries look
something like this:


.opt(l- r- f-)
.page 15.box(c+28 w10 r-0+ h2)
Qedit

.box(c+23 w20 r-0+ h2)
Qedit Library

.box(c+18 w30 r-0+ h2)
Calculator, Qhelp, and Other Libraries

.box(c+13 w40 r-0+ h2)
Low-level Robelle Library

.box(c+8 w50 r-0+ h2)
MPE and Compiler Libraries


|2View of Robelle Libraries|
.opt

While these libraries build on each other, libraries at the upper
levels can call libraries at any lower level.  For example, Qedit
makes many calls directly to the MPE library (e.g., readx, print,
fopen, fread, ...).  Similarly, the calculator makes calls to
both the Robelle low-level library and the MPE library.

`|1Development Summary|
.ent Development Summary

Because we use SPL and SPLash!, we
needed to develop ideas that would let us port object-code to HP-UX.
In order to port Qedit from MPE to HP-UX, we would first need to
port many of our own libraries and investigate how many MPE library
routines were available on HP-UX.
.sub Basic Research
.page.jump 1
.opt(l- r- f-)
`|3Phase One:  Basic Research|
.ent `Phase One:  Basic Research
.opt


We wanted to start our research by doing two things:
.mar(l+3).par(u3)
`1.#+1Learn more about HP-UX (e.g., how to log on!)
`2.#+1Prove whether or not the object-code formats on MPE
and HP-UX were the same.
.mar.par

`|1Get A Machine|
.ent Get A Machine

You can't make an omelet without breaking eggs, and it's hard to
learn about HP-UX without having access to a machine.  Because
we had no idea how successful our initial research would be, we
didn't want to purchase one.

We asked HP if we could rent an HP-UX machine for a week, but HP
doesn't have short-term rentals. Since we've had good relations
with HP/Canada, they arranged for us to have one of their demo
machines for a month. It's a good thing we had the machine that length of
time, because it took us about three weeks to get it up and
running.

We could tell HP-UX was made by HP.  When we first brought the
machine up, one of the first things it did was ask us if our
console was a 2392 or a 700/92.  We can't imagine any non-HP machine
asking this question.

`|1Networking|
.ent Networking

We learned a lot by just installing the borrowed HP-UX machine and
getting it connected to our network.  On our HP 3000 machines, we
had HP's NS software and used Thinlan to connect all of our
machines together.  We lost a lot of time because the demo HP-UX
machine had a jumper switch on the network interface card turned
off.  It took us two weeks to figure out that it was a hardware
problem and not a configuration problem.

By default, HP-UX is
not able to communicate with MPE machines.
On MPE, we are used to having front-end software that handles
configuration (NMMGR being the one for configuring network software
in MPE).  In UNIX, it's common for configuration to be done by
changing scripts (although some common configuration tasks can be
done with HP's SAM tool).
Scripts are similar to MPE/iX command files.
One of the files that is used to
configure networking is called |5/etc/netlinkrc|.  By default, it
contains a line like this:

.font 5.page 4
     /etc/lanconfig lan0 ether
.font 0

For HP-UX to communicate with MPE machines in the Thinlan/NS
environment, this line needs to be:

.font 5.page 4
     /etc/lanconfig lan0 ether ieee
.font 0

Notice the addition of "ieee" to the end of the line.  We also needed
to the following:
.mar(l+3).par(u3)
`1.#+1Fill in the DOMAIN and ORGANIZATION in |5/etc/netlinkrc|.
`2.#+1Add machine names and IP addresses to |5/etc/hosts|.
`3.#+1Reboot our machine for these changes to take effect.
.mar.par

Since our machine had the NS software for HP-UX, we could now use
vt3k to log on to our MPE machines from HP-UX (vt3k provides
virtual terminal access from HP-UX to MPE). After using NMMGR to
add our new HP-UX machine to our MPE configuration, we could also
use dscopy to copy files from MPE to HP-UX and vice versa.

`|1Object-Code Format|
.ent Object-Code Format

Our next major goal was to test our theory that object-code files
were the same on MPE/iX and HP-UX. To show this was the case we
planned to compile a SPLash! program on MPE, copy the object-code
to HP-UX, link the object-code, and run the resulting program.

One of the first programs we tried was written by Randy Medd of
Telamon (510-987-7700). This program calls the HP-UX routines
puts (which writes a string to stdlist) and exit (which
terminates the program). It looked like this:

.font 5.page 5
     $native, nocc, unix
     begin

     equate line'feed = 10;

     byte array buf(0:14);

     double procedure puts(buf);
        virtual byte array buf;
        option native, nocc, external;

     procedure exit(int);
        value   int;
        double  int;
        option native, nocc, external;

     move buf :=  ("Hello, World!",line'feed,0);

     puts(buf);
     exit(0);

     end.
.font 0

`|1Compiling and Linking|
.ent Compiling and Linking

We compiled the above program using SPLash! and copied the
resulting object-code to HP-UX with these commands:

.font 5.page 5
      :splash testsrc,testobj
      :dscopy testobj to /users/david/test.o:daffy[david:pass]
.font 0

The next part took about a week
of work to figure out:  the exact link command to use
in HP-UX in order to have our SPLash! program run.  The
HP-UX command for linking is called |1ld|.  To link our example
program, we used this ld command:

.font 5.page 5
     ld /lib/crt0.o test.o /lib/libc.a
.font 0

This linked the three files |5/lib/crt0.o|, |5test.o|, and
|5/lib/libc.a|. The first file is /lib/crt zero dot oh (don't mix up
the zero and the oh), which
is the C run-time startup library.
The test.o file is our SPLash!
object-code that we generated on MPE. The /lib/libc.a file is one
of the HP-UX run-time libraries.
Like most HP-UX command names and commands, everything must be
in lower case.
The general form of the ld
command is:

.font 5.page 5
     ld /lib/crt0.o objfiles...@/lib/libc.a
        ^                       ^ C library
        C run-time startup
.font 0

`|1Running Our Program|
.ent Running Our Program

By default,
the ld command creates a program file called |a.out|.
To run this program, we only had to type its name (like
using implied run on MPE/iX).
We tried
running the resulting program and got:

.font 5.page 4
     a.out             |0{run our sample program}|
     Hello, World!     |0{result of running a.out}|
.font 0

This was exactly what we expected.
The buffer that we initialized with "Hello, World!" was printed out
on stdlist and the program stopped.
We had now answered one of our most
basic questions -- we could compile programs on MPE, copy the
object-code to HP-UX, link the object-code into a program and run
it on HP-UX.

`|1Run-Time Libraries|
.ent Run-Time Libraries

Once our basic research question was answered, it was time to
start exploring more of HP-UX.  We had no idea how many of the MPE
routines that we needed were available on HP-UX.  We used two
HP-UX tools to search for routines:  man and nm.

`|1Man Pages|
.ent Man Pages

The |1man| pages are similar to MPE's on-line help.
For example, if you want to see if there is a description of
a routine called binary, you do the following:

.font 5.page 4
     man binary
.font 0

Once done, you will see this:

.font 5.page 4
     No manual entry for binary.
.font 0

Another way to search the man pages is to have the man utility
search a keyword index from all the man pages for a string (see below
for how to create the man page index file):

.font 5.page 4
     man -k binary                |0{-k requests a keyword search}|
     .
     .                            |0{many entries printed}|
     .
.font 0

There were many routines with a description that included
the word "binary", but none were similar to the MPE binary routine.
The man page index is in the file |5/usr/lib/whatis|.  You can
rebuild this file by doing:

.font 5.page 5
     catman -w
.font 0

You must be logged on as super user to run catman.  It takes catman
a long time to create the index, so we suggest that you run it in the
background or in a batch job.

`|1Searching Object-Code|
.ent Searching Object-Code

Just because a routine is not documented doesn't mean that it
does not exist.  Many MPE compiler-library routines are available
in HP-UX, but not documented.  The tool to use for finding symbolic
names in object-code is |1nm|.  The nm utility is similar to
the MPE/iX Linkedit Listobj command.
The nm output is similar enough to the Linkedit listobj output
to make MPE users feel at home (another advantage of using HP-UX).
We combined nm with two
other HP-UX tools to search entire directories of files.  For
example, to search the directory |5/usr/lib/| for any files with
a symbolic entry for |5binary| we did the following:

.font 5.page 4
     cd /usr/lib
     nm -r /usr/lib/* &| grep binary &| more
.font 0

If you do this on HP-UX, you get a large number of error messages
(not everything in /usr/lib is an object-code file) and the
following line is found:

.font 5.page 4.mar(r+5)
     libnsfmt.a:binary   &|1073789772&|extern&|data   &|$SHORTDATA$
.font 0.mar

Unfortunately, this is "data". What we needed was "code" or "entry".
Libraries on HP-UX are also stored in the directory /lib, so we
checked those libraries too:

.font 5.page 4
     cd /lib
     nm -r * &| grep binary &| more
.font 0

We did this search and still did not find any routine called binary.
So it appears that there is no binary routine for HP-UX. Since this
routine is used everywhere in our tools, we ended up writing our
own "binary" routine in C (note that the HP-UX routine |5atoi| is
not the same).

`|1Floating-Point Numbers|
.ent Floating-Point Numbers

In MPE, we use the compiler-library routines |5hpinext| and |5hpextin|
to convert floating-point numbers from human-readable form to the
internal IEEE format and vice versa.  These routines have many options
and handle different sizes of floating-point numbers.  Rewriting these
routines would be a lot of work.

Qedit does not use floating-point numbers very often, except in the
calculator.  In order to implement the calculator, we would need the
functionality of the hpinext and hpextin routines.
A search of the man pages revealed no documentation for either
routine, although there were other routines to handle floating-point
conversions.  All of our code assumes that hpinext and
hpextin are available.  So we searched for these routines using nm:

.font 5.page 4
     cd /usr/lib
     nm -r * &| grep hpinext &| more
.font 0

The result of this search was as follows:

.font 5.page 4
     libcl.a:hpinext     &|     13836&|extern&|entry  &|$CODE$
     libcl.sl:hpinext    &|    421568&|extern&|code   &|$CODE$
     libcl.sl:hpinext    &|    421524&|extern&|entry  &|
     libf.a:hpinext      &|     13836&|extern&|entry  &|$CODE$
     libf.sl:hpinext     &|    421568&|extern&|code   &|$CODE$
     libf.sl:hpinext     &|    421524&|extern&|entry  &|
.font 0

Success!  We had found the hpinext routine.
We also found the hpextin routine in the same libraries (using
the same technique).
These routines were located in either the |5libcl.a| or |5libf.a|
library.  Once we started converting the calculator to work
on HP-UX, we had to change our ld command to include one of
these libraries:

.font 5.page 5
     ld /lib/crt0.o test.o /usr/lib/libcl.a /lib/libc.a
                            ^
                            New run-time library
.font 0

This was one example of how familiar HP-UX is to MPE programmers.
It is highly unlikely that any other version of UNIX has the
hpinext or the hpextin routines.  Even if they did, they are unlikely
to have the same parameters and work the same way as the HP version.

`|1Phase One Summary|
.ent Phase One Summary

We succeeded in our basic research goals. We obtained access to
an HP-UX machine (due to the kind loan from HP) and we proved
that SPLash! code compiled on MPE could be linked and executed on
HP-UX. The time that elapsed for this phase was about three months
(February to April). The work time:

.mar(l+11).par(u10).page 8
`Dave Lo#+1One person-month.
`Bob Green#+1Half person-month.
`David Greer#+1One person-month.
.mar.par
.sub MPE and Robelle Libraries
.page.jump 1
.opt(l- r- f-)
`|3Phase Two:  MPE and Robelle Libraries|
.ent `Phase Two:  MPE and Robelle Libraries
.opt


The goals for phase two were:
.mar(l+3).par(u3)
`1.#+1Obtain our own HP-UX machine.
`2.#+1Learn more about HP-UX.
`3.#+1Implement the necessary MPE routines on which we rely.
`4.#+1Start porting the Robelle library to HP-UX.
.mar.par

`|1Obtaining an HP-UX Machine|
.ent Obtaining an HP-UX Machine

After we had returned our demo machine to HP, we started work to
obtain our own HP-UX machine.
We did some investigation and decided to rent a Series 705
workstation from HP, complete with the necessary networking (NS)
and development (C) tools (don't attempt C development with the
"free" C compiler that comes with HP-UX).
Since Bob and I work at home most of
the time, we really planned on using the workstation as a server.
Because HP-UX is a multi-user operating system, we decided that
it would work just fine as a development server.  Even today,
our Series 705 is used as a workstation by one user in the office
and as our primary development machine by those of us doing
HP-UX program development.

`|1Implementing MPE Routines on HP-UX|
.ent Implementing MPE Routines on HP-UX

In phase one, we had already started implementing some common
MPE routines (binary, dbinary, ascii, and dascii were now done).
We continued with other routines such as print, readx, ccode,
and hpsetccode.
Once these routines were completed we could start
porting our most fundamental library source modules.

`|1Porting the Robelle Libraries|
.ent Porting the Robelle Libraries

With a basic set of MPE
routines now available, several of our source modules started working
on HP-UX.
However, we would often try to port a new Robelle library only to find that
we needed to implement additional MPE routines.  So the process
went back-and-forth between porting Robelle source modules and
writing new MPE replacement routines.
In total, we implemented about twenty different MPE routines.

`|1MPE File System Routines|
.ent MPE File System Routines

Things went well, until we had to port our first source module
that needed to access files.  Using sample code written by Stan Sieler
of Allegro Consultants, Inc. (415-369-2303), we investigated
how difficult it would be to implement the MPE fopen, fread, fwrite,
and fclose routines.

The HP-UX file system is simple compared to the MPE file system.
Files are collections of bytes.  There is no implied structure to
any file (although you may deduce the structure by examining the
filename extension or by looking at some of the data in the file).
In MPE, we have fixed-length, variable-length, ascii versus
binary, record-length, blocking-\factor, and many other file attributes.

Our software depends on a number of assertions that are provided by
MPE's file system routines.  For example, if you open a new file
with a filecode of 111 and a record size of 256 words, write some
records to it, close it, open the file again, then ask about its
structure, you expect to get a filecode of 111 and a record size
of 256 words.  In HP-UX, there is no place to store this
information (without using an auxiliary file).

It was at this point that we made a pivotal decision about how we
would port our software.  We would not emulate the MPE file system
routines.  Instead, we would reengineer our products to invoke an
interface layer that would provide file system functionality.  We
would isolate the interface layer for each module that needed file
system access into a separate source file (in object-oriented
terminology this is called encapsulation).

`|1Recognizing Qedit Files|
.ent Recognizing Qedit Files

On MPE, Qedit files are easily recognized as those with a filecode
of 111 and a record length of 256 words.  But how were we going to
recognize Qedit files on HP-UX?  HP-UX doesn't have filecodes or
record lengths.  Bob and I designed a routine that would examine
the first 1024 bytes of an HP-UX file.  By looking at various
relationships in this file, we hoped that we could determine if the
file was a Qedit file or not.

We managed to implement such a routine.  To test its effectiveness,
we used dscopy to copy one Qedit file from MPE to HP-UX.  We wrote
a test program that opened every file on our Series 705 workstation
and checked to see if it was a Qedit file.  Our first
implementations found many HP-UX files that the routine thought were Qedit
files when they were not.  But our final implementation found the
one and only Qedit file that we had copied from MPE.

`|1Porting Qhelp to HP-UX|
.ent Porting Qhelp to HP-UX

Qhelp is the software that provides all of Robelle's on-line help
facilities. Qhelp was nicely isolated into one source module. It
uses a Qedit file to store all of the help text and the help
keywords. We decided that Qhelp would be our first HP-UX module
to use an interface layer to provide file system access on both
MPE and HP-UX.

`|1File System Layer|
.ent File System Layer

The MPE version of Qhelp invoked the MPE file system directly.  It
looked basically like this:


.opt(l- r- f-).page 10
.box(c+23 w20 r-0+ h2)
Qhelp

.box(c+18 w30 r-0+ h2)
MPE File System Library


|2Original Qhelp|
.opt

We reengineered Qhelp to call an interface layer to provide all
file system functionality.  This interface layer would open a file,
let us know whether it was a Qedit file or not, read the file, and
close the file.  Once Qhelp called the interface layer, it would no
longer "know" whether it was running on MPE or on HP-UX.

.page 5
The new Qhelp looked like:


.opt(l- r- f-).page 12
.box(c+23 w20 r-0+ h2)
Qhelp

.box(c+18 w30 r-0+ h2)
Interface Layer

.box(c+13 w40 r-0+ h2)
OS File System Library


|2Revised Qhelp Structure|
.opt

`|1Implementing the Qhelp Interface Layer|
.ent Implementing the Qhelp Interface Layer

In our revised Qhelp diagram, we added an interface layer that is not
operating system specific.  The job of the interface layer is
to hide the details of the underlying operating system.  Each
interface layer has two implementations:  one for MPE and one
for HP-UX.  The MPE interface was written in SPL/SPLash! and the
HP-UX interface was written in C.

A lot of research went into learning how to map parameters in SPL
to parameters in C.  We also had to learn all of the HP-UX file
system calls, how they worked, how to detect and report errors and
so on.  All of this took considerable time, but once we got Qhelp
working we knew that future rewriting of other modules would be
faster.

`|1Phase Two Summary|
.ent Phase Two Summary

During this phase we made a lot of progress. Approximately 20,000
lines of existing SPL source code were ported to HP-UX (e.g., the
calculator and Qhelp modules) and about 1,500 new lines of C code
were written. This provided a strong foundation to begin the port
of Qedit to HP-UX. The time that elapsed for this phase was about
six months (August to January). The work time:

.mar(l+11).par(u10).page 8
`Dave Lo#+1Two person-months.
`Bob Green#+1Half person-month.
`David Greer#+1Three person-months.
.mar.par
.sub Rebuild Qedit
.page.jump 1
.opt(l- r- f-)
`|3Phase Three:  Rebuild Qedit|
.ent `Phase Three:  Rebuild Qedit
.opt


By this point, things were looking pretty good.  We had a number of
useful MPE routines written, a lot of the Robelle library was
running on HP-UX, and we had one module (Qhelp) that did file I/O
using an interface layer.  It was now time for the big show:  porting
the Qedit source code to HP-UX.
This was a much bigger project than what we had completed so far,
since Qedit itself consists of about 75,000 lines of code.

`|1The First Attempt|
.ent The First Attempt

We started by compiling all of Qedit's source modules using SPLash!
and copying the resulting object-code to HP-UX.  Our first
attempt to link Qedit using the
ld command had many errors about missing procedures (e.g., all of the
MPE file system routines).  You can run an HP-UX program with
missing externals, but the program aborts if any missing routine is
called.  To work around this problem, we wrote one source module
that had all of the MPE routines that we had not rewritten for
HP-UX. Each dummy routine returned a failure status. A portion of
this file looked like this:

.font 5.page 4
     integer procedure fopen;
     begin

        fopen := 0;

        hpsetccode(ccl);

     end'proc;   <<fopen>>

     procedure flock      ;begin hpsetccode(ccl); end'proc;
     procedure fpoint     ;begin hpsetccode(ccl); end'proc;
     procedure fread      ;begin hpsetccode(ccl); end'proc;
     procedure freaddir   ;begin hpsetccode(ccl); end'proc;
     procedure freadlabel ;begin hpsetccode(ccl); end'proc;
.font 0

`|1Redo Module|
.ent Redo Module

Qedit supports the Do, Redo, and Listredo commands.  After our first
attempt to link Qedit on HP-UX, we realized that we needed
the redo module to work.
The redo module
also requires a "new" file (these are files that on MPE you
cannot see with Listf or Listftemp).
HP-UX does not have new or temporary files, but there is a solution.
For the invisible scratch files needed by a program, you use the
following algorithm on HP-UX:

    get a temporary filename (use tmpnam or tempnam)
    open the temporary filename
    unlink the filename from the file system

If your program stops for any reason, the scratch file will
disappear and the space the file occupied will be returned to the
HP-UX free space list.

With the redo module working, we were now able to link our first
version of Qedit/UX. However, this version couldn't do much. It
accepted commands, but most failed immediately. For example, the
command:

.font 5.page 4
     /open file
.font 0

would fail because the open command was still calling fopen.  Therefore,
the next step was to completely reengineer Qedit so that no direct
calls were made to MPE routines.  Like Qhelp, we began to reengineer Qedit to
call an interface layer.

`|1Implementing the Qedit Interface Layer|
.ent Implementing the Qedit Interface Layer

Bob and I took about one month to completely reengineer Qedit so
that it called an interface layer for all operating system
services. Bob changed Qedit and wrote the MPE version of the
interface. I wrote the equivalent interface for HP-UX in C. A lot
of our time involved Bob asking me if it was possible to do certain
things in HP-UX (e.g., how to obtain exclusive access to an opened
file in HP-UX). By now, I had ported enough source code to have
a pretty good idea of what was possible and what was not.
Writing the interface layer was not our only problem.  There
were many other parts of Qedit that had to be changed.

`|1Upshifting Filenames|
.ent Upshifting Filenames

Inside Qedit, filenames were always upshifted,
often in several places.  On MPE this
didn't matter, since filenames are not case-sensitive.
On HP-UX, this was now important since filenames are case-sensitive.

Bob had to go through Qedit to find every single location where
filenames were upshifted and remove the upshifting code.  We also
had to cope with a different filename syntax (e.g., |5../init.c| is
a valid HP-UX filename).

While we made these changes for HP-UX, we are now taking advantage
of these same changes for POSIX on MPE.  In MPE/iX 5.0, POSIX
filenames have the same syntax as on HP-UX (they are mixed-case and
can contain directory names).  So our efforts have paid
off twice.

`|1The 32-Bit Alignment Problem|
.ent The 32-Bit Alignment Problem

The HP-UX C compiler prefers to have 32-bit variables aligned
on 32-bit boundaries.  In SPL and SPLash!, it is common to have
32-bit variables aligned on 16-bit boundaries.  The interface
layer for Qedit was much more complicated than any of the other
interfaces that we had written.  Because the interface was more
sophisticated, large data structures had to be
passed between SPLash! and C routines.  We had to carefully
implement these structures so that all 32-bit variables were
aligned on 32-bit boundaries.  We also had to insure that these
structures themselves started on a 32-bit boundary.

You can force SPLash! to allocate arrays on 32-bit boundaries by
making the arrays |5virtual|.  This often has unexpected side-effects
(e.g., you cannot pass a virtual integer array to an option splash
procedure that is only expecting an integer array).  In Qedit,
we ended up doing the following.  We would allocate
one more word of storage than was necessary and then would
have code that looked like this:

.font 5.page 4
     if &@ext'record mod 2 <> 0 then
        &@ext'record := &@ext'record(1); ! assumes integer array
.font 0

This code fragment adjusts the start of the ext'record structure
if it does not start on a 32-bit boundary.  We also had to be
very careful of 32-bit variables that were passed by reference
(these are often status parameters).   It was often very difficult
to detect a 32-bit alignment problem:  due to the nature of SPL
and SPLash!, the problems can come and go.

`|1The End-Of-File Problem|
.ent The End-Of-File Problem

Many parts of Qedit need to know how many records are in a file.
For example, to print the line numbers for a command like:

.font 5.page 4
     /list somefile last-10/
.font 0

requires that Qedit know how many records are in |5somefile|.  MPE
remembers how many records are in a file while HP-UX does not.

Bob realized that rewriting Qedit so that it did not need to know
how many records were in a file would take longer than everything
we had done so far.  This meant that in the HP-UX interface layer
we have to read a file from beginning to end in order to count
the number of lines in it.  We do this every time that we open an
external file in Qedit/UX.  It's not very efficient, but it was
the only way to provide this information to Qedit.

`|1Implementing Full-Screen|
.ent Implementing Full-Screen

By this point we had many of Qedit's line-mode commands working
on HP-UX.  The big question left was what to do about Qedit's
full-screen mode.  On MPE, Qedit uses block-mode to implement
full-screen.
On HP-UX, we didn't know what we were going to do about full-screen.

One Friday night after a long day, my two children came into my
office (at my home).
They wanted to play the computer game "The Playroom".  I soon
had one child on each knee (they were 4 and 2 at the time),
happily playing on my computer.  I was surrounded by
the HP-UX reference manuals, which I had been using all day.
As my children were playing, I started flipping through one of the
manuals.  I wasn't really concentrating, but
suddenly I noticed an entry for |1blmode|.  The opening
description read:

.mar(l+3 r-3)
"This terminal library interface allows support of block mode
transfers with HP terminals."
.mar

By accident, I had found a set of routines that implemented
block-mode. Bob liked the HP-UX block-mode interface enough that
he reengineered the Qedit block-mode interface on MPE to look like the
one on HP-UX. In a few days time, we had full-screen working on
HP-UX.

`|1Phase Three Summary|
.ent Phase Three Summary

Qedit was now running on HP-UX. The process we used helped ensure
that future Qedit enhancements would appear in both versions,
since only one set of source code is used for the MPE and
HP-UX versions of Qedit. Most of Qedit no longer knew which
platform it was running on. The time that elapsed for this phase was
about three months (February to April). The work time:

.mar(l+11).par(u10).page 8
`Dave Lo#+1One person-month.
`Bob Green#+1Two person-months.
`David Greer#+1Two person-months.
.mar.par
.sub Polishing Qedit/UX
.page.jump 1
.opt(l- r- f-)
`|3Phase Four:  Polishing Qedit/UX|
.ent `Phase Four:  Polishing Qedit/UX
.opt


With the basic research complete, we now had to polish Qedit/UX to
turn it into a product.  Up to this point I had to write all of
the C code on HP-UX using the vi editor.   I was certainly pleased
when we had a working version of Qedit/UX that I could start using.
I wasn't sorry to see the end of vi.

`|1Porting the Test Suite|
.ent Porting the Test Suite

For years, we have developed test suites for all of our products.
The Qedit test suite has
a few hundred tests organized into sixty-three batch jobs.  We
knew that in order for Qedit/UX to be reliable, we would have
to have a good portion of our test suite working on HP-UX.

There was no automated way to convert our tests from MPE to HP-UX.
We had to go through each test manually and convert MPE commands
to their equivalent HP-UX commands.  We also had to implement
several of our test tools on HP-UX (e.g., the program that compares
two Qedit files one line at a time looking for differences).

`|1Temporary Files|
.ent Temporary Files

On MPE, Qedit uses temporary files for a scratch file and for
"hold" files.  The hold filename on MPE was chosen so that you
could use the name in commands.  For example,

.font 5.page 5
      /open file1
      /hold 15/20            |0{save lines 15/20 in file "hold"}|
      /open file2
      /add  75=hold          |0{add lines from the "hold" file}|
.font 0

Initial versions of Qedit/UX created files in the current directory
called QEDITSCR, HOLD, and HOLD0 (note the upper case filenames).
The Add command in the previous example would not work, since there
was no file called "hold":  it was "HOLD".

We modified Qedit/UX to create the qeditscr, hold, and hold0 files
in |5/usr/tmp|.  HP-UX system managers expect temporary files to
be created in the /usr/tmp directory (our nightly backup script
removes all files from /usr/tmp more than twenty days old).
We also had to modify Qedit/UX so that when it held lines or when
you wanted to look at a file called "hold", it substituted the
temporary filename.  For example,

.font 5.page 5
      /open file1
      /hold 15/20            |0{save lines in "qholdBAAa23359"}|
      /open file2
      /add  75=hold          |0{add lines from "qholdBAAa23359"}|
.font 0

On HP-UX there really are no temporary files.  HP-UX programs just
create permanent files in a directory considered to be temporary.
The different-looking filename is a result of the HP-UX routine
|5tempnam| which creates a file with a unique name in /usr/tmp.

`|1More HP-UX Features|
.ent More HP-UX Features

Qedit/UX was modified to execute shell commands (a shell is like
the MPE/iX CI.PUB.SYS program). One of the most common HP-UX
commands is ls (like the MPE Listf command). This conflicted with
the Qedit command LSort. In Qedit/UX, we made ls mean list files
and LSO mean sort lines (although the LSOrt command is not yet
implemented).

HP-UX users can have different shells.
We changed Qedit/UX so that when it executes
shell commands it does so with the user's preferred login shell.

We could not obtain command line arguments in our SPLash! programs.
Fortunately, Stan Sieler worked on the problem and modified SPLash!
to make the argument count and a pointer to the argument list
available to SPLash! programs.  HP-UX programs expect options to
be specified by a leading dash.  We changed Qedit/UX to
accept command line arguments such as this:

.font 5.page 4
     qedit -s -cvi file1
.font 0

This example asks Qedit/UX to edit a single file (|5-s|), go directly
into visual-mode (|5-cvi|), and to edit |5file1|.  This also made it
possible to set the HP-UX EDITOR variable, so that any HP-UX
program that invoked an editor would invoke Qedit/UX on behalf of
the user.

`|1Tab Characters|
.ent Tab Characters

One problem that we have not solved adequately is how to handle
tab characters. In MPE, tab characters are used as a short-hand
way of moving the cursor across the screen. The tab character is
always translated into spaces and never stored in files. On
HP-UX, the tab character is also used to move across the screen,
but it is also used as data. Some programs (e.g., make and
sendmail) require that certain lines have tab characters.

In Qedit/UX, we automatically expand tabs to spaces when we text
a file. This is not what HP-UX users expect, especially if you
are editing a Makefile or the sendmail configuration file
(important data gets removed by Qedit/UX). Qedit/UX has a feature
to prevent tabs from turning into spaces, but it is not the
default. If you leave tabs as data and go into
full-screen, you will lose the tab characters that were in the
lines shown on the screen.

There are various technical problems that make it difficult for
Qedit to treat tabs as data. How to handle tab characters is
similar to our number-of-records problem in the file system
interface. It's a fundamental assumption that is different on
HP-UX from MPE. We continue to work on ideas in Qedit/UX to
solve this problem.

`|1Releasing a Product|
.ent Releasing a Product

Besides all of the technical and programming problems, there were
many other issues to deal with.  We had to create a
whole new set of software to make tapes.  I spent two full weeks
rewriting the Qedit documentation so that we could have both a
Qedit/UX user manual and on-line help.  There were data sheets to
write, advertising, training (every technical person at Robelle has
taken the week-long HP-UX introductory course), and pre-release
customers to work with.  It was all worth it when we first
demonstrated Qedit/UX at the San Francisco Interex User Group
Meeting.  We had many excited customers and just a week later we started
shipping Qedit/UX.

`|1Phase Four Summary|
.ent Phase Four Summary

Our final phase took a working R&&D program and turned it into a
new product.  Many changes were made to integrate Qedit/UX into
HP-UX.  The supporting documentation and technical support were created to
provide a high quality product for our customers.
The time that elapsed for this phase was
about five months (May to September). The work time:

.mar(l+11).par(u10).page 8
`Dave Lo#+1Two person-months.
`Bob Green#+1Two person-months.
`David Greer#+1Two person-months.
.mar.par
.sub The Future
.page.jump 1
.opt(l- r- f-)
`|3The Future of Qedit/UX|
.ent `The Future of Qedit/UX
.opt


We have proven that the initial concept of cross-platform development
using SPLash! was feasible. The object-code format on MPE/iX and
HP-UX are the same. It is possible to compile SPLash! code on
MPE, copy it to HP-UX, and then link and run the resulting
program on HP-UX. In total, we ported over 100,000 lines of code.
We could never have rewritten Qedit in the same amount of time.

`|1Full-Screen|
.ent Full-Screen

Our use of the blmode routines enabled us to get full-screen mode
up and running quickly.  Now that Qedit/UX has been released for
several months, it is clear that even on HP-UX many users have decided
to use vt terminals or vt terminal emulators.  Because full-screen
uses HP block-mode, it requires an HP terminal and will not work with
vt terminals.  One major research goal for 1994 is to reengineer
full-screen to use a terminal-independent method of communicating with
the user.

`|1Shell Commands|
.ent Shell Commands

On MPE, users are used to living inside Qedit all day. You cannot
do this in Qedit/UX, because many common shell features are
missing (e.g., setting environment variables). Some of these are
due to limitations of UNIX, but we expect to implement many more
shell features in Qedit/UX. It took us more than ten years to
fully integrate MPE command features into Qedit/MPE, so it isn't
a big surprise that it will take us a year or two to add all the
necessary shell features to Qedit/UX.

`|1MPE Flavor|
.ent MPE Flavor

Qedit was originally written for MPE.  There are many features of
Qedit/UX that retain an MPE flavor.  If we are to expand the
potential market of Qedit to users who have never seen MPE, we will
have to remove most of this MPE feel and introduce more of an
HP-UX feel.  This is a challenge, since we still want Qedit/UX
to look familiar to former Qedit/MPE users.

`|1The End|
.ent The End

Qedit/UX has been both a technical and a marketing
success.  We now have one set of source code that can be used to
generate two different products:  Qedit/MPE and Qedit/UX.  If you
have a lot of SPL/SPLash! code, perhaps our solution will work for
you as well.
.com {toc starts here}
.if outtext
.   page
.else
.   con
.   form 3
.   tit
.   page
.   if outdouble or outfinal
.      align
.   endif
.   count 3
.   com                         The next FORM is for Contents/Preface:
.   form 2
.   jump 6
.   if not outfinal
.     jump 5
.   endif
.   opt(l- r- f-).font 3
Porting Qedit from MPE to HP-UX
.   skip 4
Contents
.   opt.font 0


.   tit |1Contents|
.   con (p+ f. m10 c52 l0 r0)
.   tit
.   page
.   form 3
.   if outdouble or outfinal
.     align
.   endif
.endif    { else of if outtext }
.check(c1 f0 i1 j0 m1 o1 p1)
.com {toc ends here}
.count 50.page.com  Original outline:

.opt(l- r- f-)
`|3Outline|
.opt


Introduction
   We successfully ported Qedit from MPE to HP-UX.
   Target was HP-UX HPPA RISC.
   Time line.
   Goal:  Learn Unix.  Qedit for HP-UX, if possible.

Background
   Qedit is written 99% in SPL/SPLash!
   SPLash! is similar to C -- no run-time library.
   Differences between HP-UX and MPE.
   Software Engineering Model:  show a picture.
   Libraries (MPE & HP-UX).

Object Code
   Object-code format is the same for MPE and HP-UX.
   HP-UX tools for analyzing object-code:  nm.
   Searching libraries -- e.g., floating-point conversion.
   Document the special ld command required by Qedit/UX.
   Time:  one month (includes installation, network problems, ...).

First Phase -- Basic Research
   Replacement routines (ascii, binary, ...).
   Robelle library.
   Dummy routines.
   Problems:  32-bit alignment.
   Time:  six-months.

Second Phase -- Rebuild Qedit
   Qedit no longer "knows" which platform it is running on.
   Redesign Qedit by providing an interface layer.
   Time:  approx 2 months (Bob and David).
   The interface layer talks to the O/S primitives -- Qedit does not.
   Discovered blmode -- MPE-style block-mode.
   There are some assumptions (e.g., end-of-file).

Third Phase -- Integrate Qedit/UX into HP-UX
   Scratch filenames.
   Hold filenames.
   Shell commands via the login shell.
   New command-line options.
   Tab stops and tab characters.
   Time:  4 months.
   Future:  cd command.

Summary
   Was possible because of identical object-code formats.
   Required that Qedit be rebuilt.
   Only have one set of source code.
   The differences between MPE and HP-UX versions is < 1%.