My column about shells and WordStar Release 4 (WS4) in TCJ issue #33ì
prompted more than the usual level of commentary. There were extensiveì
discussions on Z-Node-Central and the Lillipute Z-Node (the official TCJì
bulletin board), and several messages reached me over the ARPA network. Notì
all of the comments were favorable, but I was nevertheless happy to receiveì
them. They helped further clarify my thinking on the very important subjectì
of shells and have spurred me on to prove my points by actually convertingì
WS4 to a ZCPR2-style shell! After a bit of follow-up discussion, I willì
describe how this conversion was accomplished.
Corrections
There were some things I said in the previous column that wereì
factually wrong, and before I do anything else I wish to correct them.
First, I stated that the Z-System code in WS4 was written by someoneì
other than MicroPro. I was wrong. David McCord, who was vice president atì
Echelon at the time WS4 was developed, sent me a message with the facts ofì
this matter. Echelon, through staff like David and through publishedì
materials, educated Peter Mireau of MicroPro on the facilities andì
philosophy of Z-System. Peter did all the actual programming, so the codingì
mistakes were his fault, not Echelon's or David's.
From a broader perspective, however, as I stated in the previousì
column, the real culprit was inadequate testing. Bugs in the code wouldì
have been discovered and conceptual issues clarified had more people in theì
Z community been involved as beta testers. There are so many differentì
styles of using Z-System that it takes a number of testers to uncoverì
problems. Within days after copies of WS4 were delivered to users of my Z¡
Node, I started getting questions about strange behavior exhibited by WS4,ì
behavior that turned out to result from its operation as a shell.
A second mistake in the earlier column was my implication that WS4 doesì
not get its own name from the ZCPR3 external file control block (XFCB). Iì
no longer remember what made me think that this was the case, but Davidì
McCord assured me (and I have now verified for myself) that WS4 does,ì
indeed, get its name from the XFCB when it sets up the shell stack entry.
Finally, one reader reported to me that my WSSHLOFF routine (the oneì
that completely disables shells while WS4 is running and reenables them whenì
WS4 terminated) crashed his system. Unfortunately, a large number ofì
misprints crept into the listings in going from my disk file to the printedì
pages. Most of the typos were obvious, but one was compounded by a doubleì
error. In the WSSHLOFF listing, the value for EXITSUB was printed as 03BVh. ì
The 'V' was obviously a mistake, and clever readers looked at the similarì
listing for WSSHLFIX, where the value was given as 03B3h. This,ì
regrettably, looks correct but was also a typo. The proper value is 03BEh. ì
We hope that Art Carlson will make sure that the listings in this column areì
transcribed accurately (so that all the mistakes will be mine!).
More WS4 Comments
While on the subject of WS4, I would like to add a few further commentsìèabout how it works. Not surprisingly (considering when it was developed),ì
in creating its shell stack entry WS4 does not make use of the facilityì
introduced with ZCPR version 3.3 that allows a program to determine from theì
XFCB not only its name but also the directory from which it was actuallyì
loaded (the user number is at offset 13 and the drive, with A=1, at offsetì
14).
As a result, in order for WS4 to be reinvoked as a shell, the commandì
search path must include the directory in which WS4 is located. I mentionì
this here as a reminder and suggestion to authors of new or updated shellsì
and error handlers that they use this Z33 facility to avoid the requirementì
that the program be on the path and to speed up loading of the program (byì
eliminating any search for it). My WordStar conversion described later addsì
this feature.
With WS4 it is generally necessary that the command search path includeì
WS4's directory for an additional reason. I learned the hard way that whenì
WS4 runs under Z-System, it pays no attention to the drive and user numberì
that WSCHANGE specified as the location for the overlay files; it only usesì
the search path to try to locate them.
This is a problem for me because, as I have explained at length inì
previous columns, I put only my small RAM disk on the path and use ARUNZì
aliases to invoke all programs except the very few that fit on the RAM disk. ì
With this approach, there is no way to get WS4 to find its overlay files. ì
The conversion addresses this problem also.
ZCPR2 vs. ZCPR3 Shells
I would not like to take up again one of the subjects raised in issueì
#33: ZCPR2-style versus ZCPR3-style shells.
First an aside. Shells seem to be a surprisingly emotional issue. Iì
thought my earlier column presented a fairly carefully and calmly reasonedì
discussion of some aspects of shells, including their pros and their cons. ì
Some readers, however, took great offense at my even questioning the currentì
method of implementing shells or of what some people are trying to do withì
them.
One reader went so far as to suggest that I had no business commentingì
on the subject when, by my own admission, there are a number of shells thatì
I have never used. Besides the fact that this is hardly a reasonedì
argument, I would like to make sure that the following facts about shellsì
are fully appreciated.
ZCPR3-style shells are a facility of the command processor. Withoutì
special code in the CPR, there would be no such shells. As the author ofì
the two latest versions of the ZCPR command processor, I think I can speakì
with some authority (though certainly not with infallibility) on theì
subject, since in writing that code I had to consider the issue of shellsì
rather carefully from a rigorous theoretical viewpoint.
ZCPR2-style shells, quite the contrary, are not a facility of theì
command processor; they are a facility of the individual shell programs. ìèTheir functioning depends only on the operation of the multiple command lineì
facility. The command processor does not treat a Z2 shell command anyì
differently than it treats any other command. This is really the key to theì
difference between the two shell implementations.
In the previous column I stated: "...I am coming to the conclusion thatì
only programs like history shells...should be implemented as ZCPR3-styleì
shells. Other programs, like ZFILER and WordStar should use the ZCPR2ì
style." I then invited readers to enlighten me if I was missing someì
important point. I got some responses to this invitation, but no one yetì
has offered me any evidence that I had missed any important point.
One reader reiterated essentially the same difference between Z2 and Z3ì
shells that I attempted to demonstrate with my example in which WordStar wasì
invoked in a multiple command line. Apparently the point bears repeating.
This reader presented the point using a command line like theì
following:
ZFILER;ECHO TESTING
Under ZCPR2, ZFILER would run and present its file display to the user. Ifì
the user generated a command line "CMDLINE" as the result of a macro or inì
response to the prompt after the 'Z' command, a Z2-shell version of ZFILERì
would build the command sequence "CMDLINE;ZFILER" and insert it into theì
multiple command line buffer just before the next command to be executed. ì
This would give:
CMDLINE;ZFILER;ECHO TESTING
The user's command line would run, and then ZFILER would be invoked again. ì
Only on termination of ZFILER would the last command, "ECHO TESTING", beì
performed.
A Z3 shell would respond to the same command line from the user inì
quite a different way. As before, ZFILER would be invoked first. It wouldì
determine from the Z3 message buffer that it had been invoked manually andì
would respond by pushing its own name onto the shell stack. Then it wouldì
terminate. The command processor would then proceed to run "ECHO TESTING". ì
Only after that, once the command line was empty, would ZFILER be reloaded,ì
this time as a shell. Recognizing its shell status, it would now displayì
its screen of file names and do its real work.
The reader who submitted this example, if I understood him correctly,ì
viewed the Z3 behavior as correct and the Z2 behavior as wrong. If you areì
an experienced Z-System user, you will probably recognize in this reader aì
fellow expert (and, indeed, he is). He is so used to ZCPR3 that he noì
longer notices that it is the behavior of the Z3 shell that is trulyì
bizarre!
Consider the following two command lines:
(1) ZFILER;ECHO TESTING
(2) ECHO TESTING;ZFILER
èWe have already analyzed the first one; the second one can safely be left asì
an exercise for the reader. We will simply state the answer that underì
ZCPR3 they will accomplish exactly the same thing! This is hardly a resultì
that conforms to intuition, and I still remember in my early days as a Z¡
Node sysop trying to explain to quite a few users why the second command onì
a VFILER command line executes first!
Under ZCPR2 the result is just what one would expect. In the firstì
case, ZFILER runs first, and ECHO runs only after the user terminates ZFILERì
using its 'X' command. In the second case, ECHO runs first and ZFILERì
second. In other words, with Z2 shells, commands are executed in the orderì
they are entered, a notion that does not require long experience and greatì
expertise to understand and get used to! And it gives the user a greaterì
measure of control.
Mixed Z2 and Z3 Shells
The same reader submitted another interesting example that illustratesì
the confusing behavior that can arise when Z2 and Z3 shells are mixed. Hereì
we assume that WordStar has been implemented as a Z2 shell and ZFILER as aì
Z3 shell. Suppose we use the 'R' command of WordStar to enter the commandì
"ZFILER". WS4, as a Z2 shell, would generate the command line
ZFILER;WS
ZFILER, as a Z3 shell, would install itself on the shell stack and proceedì
to execute "WS". ZFILER would not run in its file maintenance mode untilì
after we terminated WordStar.
This is, admittedly, probably not what one intended, since we mostì
likely entered the ZFILER command with the intention of doing some fileì
maintenance before returning to WordStar. On the other hand, it isì
certainly no more bizarre than what we saw in our earlier example.
If both WS4 and ZFILER were Z3 shells, then the invocation of ZFILERì
from the WS4 'R' command would cause it to become the active shell (the oneì
on the top of the shell stack). The WS4 shell would be pushed down in theì
shell stack, and ZFILER would take control. With a little thought, however,ì
you will see that the same is also true if both ZFILER and WS4 are Z2ì
shells!
The strange behavior with the mixed shells in the above example arisesì
in part because ZFILER was not really being used as a shell in the Z3 sense,ì
namely, as a replacement for the command processor's command-line inputì
routine. It was intended as a file maintenance utility.
Suppose we had entered the command "EASE" (the Z3-type history shell)ì
instead of "ZFILER" from our Z2 version of WordStar. This would establishì
EASE as the current shell and return to WordStar. That behavior would notì
seem strange, because in this case we would be thinking of our EASE commandì
as establishing the shell to be used in place of the command processor theì
next time the command processor needed a new command line. So long asì
WordStar is running, there is no need for EASE to do anything. We expect itì
to take effect only after we are finished using WordStar.
è
Nested Z2 Shells and Recursive Aliases
Although I had once thought that the Z3 shell stack was required inì
order to nest shells, I showed in the earlier column that this is not theì
case. Z2-style shells can, in fact, be nested more flexibly. There is noì
predetermined limit to the nesting depth or to the amount of informationì
that can be passed with each shell command line. The only limit is imposedì
by the length of the multiple command line buffer, just as with the nestingì
of aliases.
With the standard shell stack configuration of 4 32-byte entries, if aì
shell command uses only 16 bytes, 16 bytes are wasted. On the other hand,ì
if a shell command needs 48 bytes to hold its information, it cannot run atì
all under this configuration (NZ-COM can come to the rescue by allowing theì
shell stack configuration to be changed on the fly). With Z2 shells, theseì
problems go away. In 64 bytes of command line, one can have two 32-byteì
shell commands or a combination of one 16-byte shell command and one 48-byteì
shell command (or five 12-byte shell commands).
I did overlook one point when I described putting data for the shellsì
on the command line. In the Z3 shell stack, one can include, after theì
shell command's terminating null, any binary data that one wishes. Thus 256ì
values are possible for each extra byte in the shell stack entry.
In order to carry shell data on the command line, several additionalì
constraints apply. First, the command processor strips the high bits offì
all characters in the command line, so only 128 values are available toì
start with. Secondly, the null character cannot be used because the commandì
processor would interpret this as the end of the command line (that leavesì
127 values). Finally, letters are converted to upper case, thereby makingì
the characters from 'a' to 'z' inaccessible (scratch another 26). Thisì
leaves only 101 possible values out of the original 256. Moreover, extraì
characters are required as flags to signal the program to consider itself asì
having been invoked as a shell (a service provided in ZCPR3 by a flag in theì
message buffer). All of these things reduce the efficiency with which theì
space in the command line buffer can be used compared to the space in theì
shell stack.
One reader pointed out that recursive aliases cannot be used withì
Z2-type shells. This is true...but only if one is using the pseudo¡
recursive alias that I invented. This kind of alias accomplishes a crudeì
approximation to recursion by discarding any pending commands in the commandì
line buffer. This will, indeed, discard any shell reinvocation commands. ì
However, if one uses the logically sound and rigorous recursive aliasì
technique invented by Dreas Nielsen (see my column in issue #28), there isì
no problem. It sometimes pays to do things right!
In fact, it seems to me that the Z2 shell is, in essence, a recursiveì
alias, a program that keeps invoking itself. And this is just what most (ifì
not all) Z3 shells actually do. I am still awaiting an example of somethingì
(good) that a Z3 shell can do that cannot be done in some equivalent wayì
with a Z2 shell or recursive alias.
è
The Real Difference Between
Z2 and Z3 Shells
After much reflection, I think I have finally put my finger on theì
fundamental distinction between Z2 and Z3 shells. It derives from the factsì
I alluded to earlier: that the Z3 shell is a true creature of the commandì
processor and the Z2 shell is not.
Here is an example that will illustrate the point. Suppose the historyì
shell EASE were implemented as a Z2-style shell and that while it isì
running, we issue the command "DIR". EASE will insert into the command lineì
a sequence like the following:
DIR;EASE
DIR will run, and then EASE will be reinvoked. Looks fine! But now supposeì
the user enters the command "IF EXIST FN.FT". EASE will then generate theì
command line
IF EXIST FN.FT;EASE
If the file FN.FT exists, this will again work just fine, but suppose theì
file does not exist. Then the system will enter a false flow state, and theì
EASE command (and perhaps other commands pending in the command line afterì
it) will be flushed by the command processor. The shell function will beì
lost, and any other pending commands will be processed in an unintended way.
For a Z2 shell to function properly in general, all command linesì
inserted by it must result in the same flow state at the end of the commandì
line as at the beginning. With a MENU shell it could be possible for theì
system designer to guarantee this, since he can control which commands areì
generated by the shell. With a history type shell it would be nearlyì
impossible to ensure that this condition would always be met.
The critical feature of shell processing under ZCPR3 is that flowì
processing is suspended during the operation of shells. This allows them toì
run, as they must, even after the user has passed a command that leaves theì
system in a false flow state. The ZCPR33 Users Guide goes into some detailì
on this matter, and had I remembered better what I wrote there, it would notì
have taken me this long to come to the essence of the Z2-vs.-Z3 shell issue.
Some users of ZCPR33 have modified the way the command processor dealsì
with flow control in shell processing. No one has yet convinced me of theì
value of this (the risks are undeniable). It still seems to me that Z2-typeì
shells and recursive aliases can accomplish the same thing, but in aì
logically sound way.
I have extended an invitation to Dreas Nielsen to write a series ofì
columns for TCJ explaining his very powerful shell programs. Since he isì
also one of the people who has made this modification to the CPR, perhaps heì
will also present the other side of this story and explain why it isì
necessary or desirable to treat shells the way he does.
è Remaking WordStar Release 4
When I first received my copy of WordStar 4 and encountered problems ì
with the way it handled shells, I fired up the DSD debugger and tried to ì
figure out how to fix it. After a considerable amount of rummaging about inì
the code (and especially trying to figure out what was going on insideì
WS.OVR), I gave up. Later I tried again...and failed again. In the courseì
of preparing this column, I decided to have one more go at it, and this time ì
things started to click.
The patches I will describe here are preliminary and have not yet been ì
extensively tested. In fact, as I write this, I am the only one who hasì
used them, and you know what I said above about the dangers of a testì
program that does not involve a variety of Z-System users. So, you areì
hereby recruited, if you are willing, to join the test program.
Since I may very well have made some mistakes, and since there areì
further changes that people may want to make (let's hear your suggestions),ì
I will not only give the results; I will describe the process by which theseì
patches have been developed.
Cracking the Code
The first step toward changing the code was figuring out how the virginì
WordStar was doing what it did. In particular, I wanted to locate routinesì
related to Z-System functions, so the first thing I tried was searching forì
all references to address 109h, which contains the address of the Z-Systemì
environment (ENV). Any WS4 feature that made use of a Z-System facilityì
would have to get information from the ENV.
As best I recall, this did not turn up many references and did notì
particularly help (though it was a good idea, and that's why I mention it). ì
In the end, I just started tracing the code from the beginning, figuringì
that WS4 would have to determine fairly early whether it was running underì
Z-System or standard CP/M. This turned out to be correct, and very soon Iì
came to the key Z routine, at address 0AA4h in WS.COM. This routine returnsì
the address and size of a Z-System module specified by an offset passed inì
the E register.
Having discovered this routine, I used DSD to find all references to itì
in WS.COM and WS.OVR. They occur with the following values of E:
E = 9h the command search path (PATH)
E = 15h the named directory register (NDR)
E = 18h the multiple command line buffer (MCL)
E = 1Eh the shell stack (SHL)
E = 22h the message buffer (MSG)
E = 24h the external file control block (XFCB)
Setting up the Shell Stack
The block of code beginning around address 3CBFh in WS.OVR makesì
references to MCL, XFCB, and SHL. I guessed correctly that this had to beìèthe code where WS4 sets up its shell stack entry. (This block of code, byì
the way, is where the shell-pushing mistake occurs for the case where theì
shell stack is currently empty.)
The patch for this part of WS.OVR (see Listing 1) modifies this code. ì
First of all, since WS4 is going to operate as a Z2-type shell, we do notì
want it to do anything with the shell stack. It is easy to disable the codeì
by simply skipping over it, but one has to watch out for subtleties. ì
Indeed, in order for the 'R' command to use the MCL and not chain using theì
greatly inferior CP/M method, WS4 has to think that the shell entry wasì
established successfully.
I noticed that a flag was being set into address 2200h, and I surmisedì
that it is used by WS4 to show that it is running under Z-System. In theì
patch, I set this flag even though the shell stack entry is not being setì
up. I have not examined all references to this flag, and there is a chanceì
that there are additional, more complex effects. If any problems appearì
with the patched version of WordStar, this flag might be involved. For theì
initial attempt at fixing WS4, I just took the easiest course of action, andì
so far it appears to have worked.
It seemed foolish to waste space in WS.OVR by doing nothing more thanì
setting the flag and jumping to where the original code resumed (60AAh). ì
Instead, I have used the space to compute the command line necessary toì
reinvoke WordStar. The code gets not only the name by which WordStar wasì
invoked but also the drive and user number from which it was loaded. Aì
command line of the form ";DUU:WSNAME" is generated.
There is one extra step in this part of the patch. When running as aì
Z3 shell, WS4 knows from the command status flag in the message buffer whenì
it was invoked as a shell so it can put up the press-any-key message beforeì
clearing the screen and resuming operation. As a Z2 shell, WS4 cannot useì
this facility. Instead, a signal has to be included in the command tail. ì
For reasons that I will not go into in full detail, I chose for this signalì
a comma at the very end of the tail. Very briefly, the comma is a handyì
character because it is not parsed into the default file control blocks,ì
where a program could confuse it with a file name.
The final reinvocation command line, with its terminating null, takesì
the form
;DUU:WSNAME ,<0>
Since I could not be sure that this section of overlay code would persist inì
memory until the command would be used, I store it at the top of the WS4'sì
user patch area (MORPAT).
The Initialization and Termination Patches
Having made the above modification, we must make two others in order toì
remain consistent. First, we must modify the WS4 initialization code inì
which it determines whether or not it was invoked as a shell. This is theì
patch at address 1A2Fh in WS.COM. The patch calculates the address of theì
last character passed in the command tail and checks to see if it was aìècomma. If not, it proceeds with normal operation of the program.
If there is a comma, the shell-wait message must be displayed until theì
user presses a key. But one must also remove the comma from the commandì
tail to ensure that WordStar not think it has been passed a file name. Atì
present I do this by replacing the comma with a space. This is notì
rigorous, but it seems to work, since WS4 is apparently not confused by aì
tail consisting only of spaces (unfortunately, a number of programs are).
Since WS4 no longer pushes its name onto the shell stack, we must alsoì
prevent it from popping the shell stack when it terminates. This is theì
patch at address 13CEh in WS.COM. This is the easiest patch of all, sinceì
we simply have to skip some code. As an additional benefit, this frees upì
about 40 bytes of space that we use for some of our other patch code.
Fixing the 'R' Command
Now we come to the main item in this set of patches -- the code thatì
makes the 'R' command work as a ZCPR2-type shell. The new code here is muchì
more complex than what it replaces, and we can only fit part of it at theì
original location 67B2h in WS.OVR. We put what we can there and continueì
with the rest in the MORPAT area in WS.COM.
The basic strategy is to take the command line entered by the user inì
response to the 'R' command prompt, append the WS reinvocation commandì
(including its semicolon separator), and append any remaining command lineì
pending in the multiple command line buffer (if there is one, it will beginì
with a semicolon also). If there is enough room for the result in the MCL,ì
then it is moved there and chained to. If not, a warning message isì
displayed on the screen until a key is pressed, and the user command isì
ignored.
To implement this strategy, I chose the simplest method I could thinkì
of. Since the 'R' command operates from the WordStar no-file menu, theì
entire WS edit buffer is available as a scratch area. I picked an arbitraryì
address of A000h for a buffer in which to build the new command line. ì
Again, rigorous code calculate the address. My code is a quick-and-dirtyì
solution.
Finding the Overlay Files
As I noted earlier, with my style of operation, WS4 had trouble findingì
its overlays. To solve that problem, the patch includes an optional sectionì
to install an internal search path for the overlay files. This patch isì
installed at address 0F5Fh in WS.COM, where it replaces a call for theì
location of the Z-System path with a call to a routine that returns theì
address and size of an internal path. In Listing 1 the internal path hasì
the single element B4:, the directory in which I keep my WordStar programì
files. You can put any values you want here.
Installing the Patches
è It is not possible to install the patches in WS.OVR using MLOAD or aì
debugger, because the OVR file is too large to load entirely into memory. ì
ZPATCH, on the other hand, can handle the job splendidly. ZPATCH assumes anì
offset of 0000 for a file of type OVR, while the addresses in the listingì
are those shown when the file (as much as can fit) is read into memory underì
a debugger. To make things consistent, you should use the ZPATCH 'O'ì
command to set the offset to 100.
Key in the new data carefully, checking from time to time that theì
address is still correct. Also, be careful not to go beyond a recordì
boundary while in ZPATCH. It wraps from the end of the record back to theì
beginning of that record without warning (this really gave me grief until Iì
caught on to the problem). When you get to the end of the current record,ì
write it out (^W), advance to the next record (>), and reenter edit modeì
(E). Then you can resume entering data.
The attached listing was made with a specially configured version ofì
the SLR Z80ASM assembler. Normally, I have it display addresses in logicalì
order for easier interpretation. For hand keying of a patch, however, it isì
far more convenient to have the bytes of a word in physical order. Justì
watch out when reading the displayed symbol values. They, too, are storedì
in byte-reversed format.
It is possible to use MLOAD to install just the patches for WS.COM. ì
Simply delete the parts of WSPAT.Z80 that refer to patches in WS.OVR andì
assemble the remaining code to a HEX file.
Enjoy playing with (and using) this different (improved) version ofì
WordStar 4, and let me know what you think and what further suggestions youì
