Pipes for NetRexx and Java Version 0.50 May 30, 1999
Copyright (c) E. J. Tomlinson <tomlins@cam.org>, 1997. All rights reserved This code is Beta and will contain bugs. Use it at your own risk.$ Permission to use Pipes for NetRexx and Java for noncommercial applications is granted. Please contact me for commercial uses.
Pipes for Netrexx and Java is PROVIDED AS IS AND COMES WITH NO WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED. IN NO EVENT WILL THE COPYRIGHT HOLDER BE LIABLE FOR ANY DAMAGES RESULTING FROM THE USE OF THIS SOFTWARE.
Parts of njPipes are Distributed under the GNU GENERAL PUBLIC LICENSE with NO WARANTEES of ANY kind. See CopyLeft.Txt for the licence and information on using, copying, modifing, and distributing this program.
Not withstanding the above disclaimers, I am very interested in suggestions, problems, bugs, and questions.
njPipes now can extract pipes from java code! You can now use njPipes without knowing NetRexx, though it still must be installed.
Select this link to download the latest version of Pipes for NetRexx and Java.
Select this link to download a previous version of Pipes for NetRexx and Java.
Thanks to Jeff Hennick and René Jansen for their important contributions.
| Member of the NetRexx Ring | Join the Ring |
|---|---|
|
[Next Site]
[Previous Site]
[Back 2 Sites]
[Next 5 Sites] [List All] [Random Site] | |
1. About this Program
2. Installation and Requirements
3. The Pipe Compiler
a. Parameters
b. Errors
c. Debugging
4. Stages
a. What can be piped
b. Supplied stages
c. Processing Ranges
$ d. Processing DelimitedStrings
$ e. Methods For Stages
5. Examples
a. Using pipes with NetRexx and Java
b. Using NetRexx stem variables in stages
c. Using callpipe and addpipe
d. Connectors - pipes as stages
6. Deadlocks
7. Known Bugs
8. Fixes and Changes
9. Futures
10. A (semi)Formal Definition
11. Commercial Use
12. Trademarks Etc.
IBM Java Developer Website
The IBM Java Developer Website is dedicated to supporting the Java
developer. It consists of FREE on-line Java courses, articles, jCentral
search engine and frequently asked questions as well as other information
to support and promote Java. Come Visit!
Here are some stats on the njPipes page
Pipes for NetRexx and Java implements a simple but very powerful piping
methodology based on the one developed by John P. Hartmann of IBM. A more
complete and much faster implementation of pipes can be found in the various
Pipeline products from IBM.
Information about Pipes, including many interesting papers, can be found at:
http://pucc.princeton.edu/~pipeline/
CMS pipes are extremely useful. Both NetRexx and Java look to have bright
futures. I missed pipes when working in NetRexx so I wrote these classes to
make my life easier.
The major difference from CMS is that njpipes are compiled into java .class
files and then run. This has implications that change how callpipe and
addpipe can work - dynamic pipes are not practical. This will remain a
limitation unless some hardy soul wants to write a very fast pipe to
java .class file compiler. Currently pipes get converted to NetRexx,
translated to java, and are compiled by javac.
This implementation uses a thread for each stage. The main pipe also uses a
thread to monitor the stages for deadlock detection. Stages are dispatched
using Java's wait and notifyAll methods. Data is not moved though the pipe.
Its always referred via object pointers. As the objects pass through the pipe
only pointers are changed. This implementation could be, and probably
should be, improved upon. Its not all bad though. If you are lucky and
have multiprocessing hardware, (and a Java that can use it) Pipes for
NetRexx and Java takes advantage of it. On a dual processor system, simple
pipes are 30 to 50% faster using native vs green threads.
To make it easier for an improved pipes implementation to get off the
ground, I have included the source of a NetRexx Interface class in the pipes
directory. Its my hope that any new implementation will use this class as a
base. All the methods needed by stages should be in this class. This way
implementers are free to change how pipes work, what they look like when
compiled, how they are compiled etc - but a simple recompile should produce
working stages. If you think there are missing methods please let me know.
To run Pipes for NetRexx and Java you need both Java and NetRexx runtime
support installed. To write your own pipes or stages you need compilers for
both Java and NetRexx. The OS2 JDK release for Java 1.1.7A and NetRexx 1.148
where used to develop this code. NetRexx 1.132 is required to use version
025 and above as inner classes are used, any Java from 1.1.1 on should work.
The installation has changed as of release 025. The core classes for pipes
and stages have been moved into njpipesC.jar. This file may be added to
your CLASSPATH.
To install create an njpipes directory, cd into it and unzip the njpipes.zip.
Then unzip the versioned njpipes file, unzip njpipesXXX.zip. This will build
the directory structure with with examples, docuementation, the njpipesC.jar,
a pipe.bat and pipe.cmd. If you unzipped on a drive other than C: or you
placed njpipes in a subdirectory, edit the .bat (W95, NT) or the .cmd (OS/2)
file to point to your njpipes.jar and you can start pipeing.
Alternately you can add njpipesC.jar to your CLASSPATH. If you anticipate
modifing any of the built in stages, you will want to add a reference to
the njpipes directory before the njpipesC.jar in the CLASSPATH. This will
make java see any class files in the njpipes subdirectories in before the
classes in the jar.
For NT the following variables or their equivalent need to be defined in the
Control Panel, System, Environment page as user variables. This assumes
you are using the sun JDK with the performance pack applied.
CLASSPATH = c:\jdk1.1.6\lib\NetRexxC.zip;c:\njpipes;c:\njpipes\njpipesC.jar;c:\NetRexx;.;c:\jdk1.1.6\lib\classes.zip
JAVA_COMPILER = symcjit
PATH = C:\jdk1.1.6\bin;.;
To test your installation CD into the njpipes directory and type:
java njp "(test) literal arg() ! dup 999 ! count words ! console"
The first time you use the pipe command in a new directory it will
create a default pipes.cnf file for you. The double quotes are not
really necessary in this case. Using them will avoid grief as shells
sometimes do very unexpected things, when least expected, to arguements.
If you have added the njpipesC.jar to your CLASSPATH you can also use:
java njp "(test) literal arg() ! dup 999 ! count words ! console"
You should see a message that the pipe compiler is processing your pipe and
soon after that messages from the NetRexx compiler as it processes the pipe.
To run the pipe type:
java test some words
The pipe should then output:
2000
| If you have had njPipes working and it fails after a reinstall, it may be
| due to source you have recompiled. Its a good idea to delete any .class
| files you have created in the pipes, stages or tests directories. Also
| check the readme. There maybe requirements to recompile some of your
| pipes. The stageExit method, now used quite a bit, is a two edged sword.
| On one hand it generates much faster pipes, on the other hand, it often
| requires pipes to be recompiled.
Back to Table of Contents
You can use the compiler to compile simple pipes directly like the test
pipe in section 2. You can also compile from files. Pipes for NetRexx and
$ Java use the suffix .njp for pipes. You can see examples of pipes in the
examples directory.
The compiler is invoked with the command:
java njp <arguments>
Where <arguments> are:
<compopt>|<fileid> {<compopt>|<fileid>} - compile the files listed
| ( <name> { <compopt>} ) <pipe> - compile the <pipe> name
Where <compopt> is:
-gen - keep the .nrx source of pipes following
-nogen - delete the .nrx source. This is the default
-<option> - where <option> is a valid NetRexx option
And <fileid> is:
<name>.nrx - any NetRexx file
<name>.njp - any pipe or NetRexx or Java souce using pipes
<name>.grp - a file containing a list of <compopt>s and/or
<fileid>s. lines can be commented using '--'
The <options> are:
runaway <milliseconds> - Interval to use for deadlock checking. This
or only detects true pipe deadlocks, not stages
stall <milliseconds> that are looping or hung. (Should this be
changed?)
package <name> - insert a NetRexx package statement
import <name> - insert a NetRexx import statement
config <name> - override the default config file name (see below)
sep <char> - override the default stage separator '!'
end <char> - override the default end of pipe separator '?'
cont <char> - override the default line continuation character ','
debug <flag> - debugging options
cache <Hashtable> - a Hashtable to help clue in the JIT...
When the pipe is contained in a file the options must be set there.
When a pipe is read from a file, text on the same line following a '-- ',
will be treated as a comment.
When a file called 'pipes.cnf' exists in a directory the compiler will
include it after the initial comment. You can use this to put stages in
packages or to setup standard imports so all your stages are found. You can
see examples of this the the stages, examples and tests directories.
See section 10 for a definition of a <pipe>.
Errors issued by the compiler. Stages can issue additional messages.
Error - name of pipe is missing
Error - runaway time must be numeric
Error - stall time must be numeric
Error - debug level must be numeric, found 'lvl'
Error - cache must be followed by a valid symbol, found 'work'
Error - append or prefix stages cannot be labeled 'stg.word(1)'
Error - missing stage/pipe after 'stg.word(1)'
Error - missing range and/or stage/pipe after 'stg.word(1)'
Error - specs has only one output stream, not requires two
Error - pipe definition 'stg' must be terminated by 'pend'
Error - pipe definition 'stg' must define a pipe
Error - Label 'label' already used
Error - Label 'label' must not be numeric
Error - pipe 'stg' cannot have a label
Error - connectors must be named
Error - Label 'label' not defined
Error - Use a nop stage between 'label' definition a second use
Error - expected a colon after 'stg.word(1)'
Error - pipe as stage definition 'stg' is missing a period
Error - 'parms' incorrect. Use: {({class} {size})} {A|D} {target}
Error - need to use a nop stage between 's[c[i,j-1]]' 'ssep' 's[c[i,j]]'
Error - problem reading group rc='r'
Error - 'w' is unreconized
Warning - could not delete 'fileid'
Error - 'key' is only valid in a class
Warning - Possible netrexx exit in 'arg()' at 'l'
Error - pipe name and parms must be on same line as 'key'
Error - Pipe name missing for 'key'
Error - Body of 'wp' is empty
Error - *: connectors not implemented. Use *in: or *out:
Error - Connector 'w1' should start with *in or *out
Error - Missing colon at 'w1'
Error - Connect 'w1' cannot contain a period
Error - cannot connect 'in' to an input stream with 'key'
Error - Pipe fragment 'sub' needs atleast one 'sep'
Error - cannot connect 'out' to an output stream with 'key'
Error - A object name cannot contain spaces, found: 'a'
Debugging pipes
To find out what is happening in pipeline you have three tools. First,
you can set a debug flag when you compile the pipe. The bits you set
in the flag control what it does:
1 - Show all pipes starting
2 - Show all pipes ending
4 - Show all stages starting
8 - Show all stages stopping
16 - Show all Commit requests
32 - Show all Commit completions
64 - Show StageErrors raised via stage's Error(int,String) method. The
stage class uses Error for all its StageError signals.
128 - Show the arguement that each stage is recieving. Handy since
shells have a habbit of doing unexpected thing to arguements.
(try: java findtext exit *.nrx vs java findtext "exit *.nrx")
- If there is a need I can easily add more events - just ask.
To create a flag to see all stages starting and stopping you would
add 8+4 and use:
java njp (apipe debug 12) ...
The second option is to use the invoke the dump() method in a stage.
This dumps the status of the pipe using the same format you see when
$ a pipe deadlocks. Using dump() does not normally cause a pipe to
$ terminate. Once in a while dump() will generate an exception. This
$ happens since dump() does not use protect or syncronize so it does not
$ stall.
When all else fails, you can recompile all the stages used by the pipe
with trace results enabled. As of NetRexx 1.140 trace displays the
context of the trace output. With this, it is possible to see exactly
what the stages are doing. If this does not help send me the pipe and
I will see if I cannot figure it out - you may have found a njPipes
bug.
One nasty error I have encountered is coding an exit instead of an
exit(rc). An exit alone will end the pipe very abruptly - the compiler
should now warn about this...
Back to Table of Contents
Pipes for NetRexx and Java moves objects through pipes using stages. This
implies that almost anything can be piped. There are a few things to
remember. Many of the supplied stages expect Rexx Objects and will fail
when they do not get them. I wonder if anyone will come up with a use for
piping pipes?
Standard Stages currently available can be found in the stages directory.
Not all options available on the equivalent CMS stages have been implemented
and error checking is not always complete.
Here are a list of stages you can use now:
c abbrev
c append -- the appended stage can be started at any time
c array
c arraya
c arrayr
c arrayw
c between
c buffer
c change (rexx)
c casei
c chop
command (rexx) -- examples work in OS2, NT & Solaris not in W95
compare (rexx)
c console (rexx)
c copy
c count (both) -- lines (objects), words, chars, max, & min (rexx)
dam
deblock
c deal
dict
dicta
dictr
dictw
disk
diska
diskr
diskslow
diskw
c drop
c dup
elastic -- Implemented with two threads...
c fanin
faninany
c fanout
c fblock
file
filea
filer
fileslow
filew
c frlabel
frtarget
gate
getfiles
getovers
getstems
c find (rexx)
hash
hasha
hashr
hashw
c hole
c insert
c inside
c join (rexx)
c joincont
c juxtapose
c literal
c locate (rexx)
c lookup (rexx)
not -- A stage modifier. Useage: not <stage>
Swaps output streams zero and one of <stage>
Usually used with a selection stage like find or locate
noEofBack
nop -- see section 10
c notinside
c nfind
over -- read all subscripts of a stem (like loop over)
c outside
c pad
c pick
prefix
c reverse (rexx)
rexx
c serialize
specs -- massage the data. Selections via start.length
-- start-end, start-* and -from_end;-from_end. These
-- selections can also be prefixed with 'word'. The
-- conversion functions upper, lower, b2x, d2x, x2b and x2d
-- are implemented as are format options left, right and
-- center. The stop, select, read, readstop, pad and
-- write functions work too.
-- The number, TODclock and many other conversion
-- functions are not implemented.
c split (rexx)
stem
stema
stemr
stemw
String
c strfind
c strnfind
c strfrlabel
c strliteral
c strfrlabel
sort -- sort objects: sort {({<Class>} {<size>} {A|D} {IRange}
Where <Class> is a sortClass. The default is: Rexx
and <size> is the maximin number of objects, 10000 is the default.
sortClass -- interface to allow other object type to be sorted
sortRexx -- the default sortClass implementation
sqlselect -- query SQL via JDBC (a submitted stage)
c take
tcpclient
tcpdata
tcplisten
c timestamp
c tokenize
c tolabel
totarget
timer -- time pipes
| c unique
var -- this can only read vars
vector
vectora
vectorr
vectorw
c xlate -- translate data (a submitted stage)
c zone
Stages prefixed by 'c' should work very much like CMS stages.
$ (Rexx) means the stage requires rexx objects as input
The methods you can use in stages are documented here.
$ Ranges
The interface to rexx ranges is in the pipes\IRange and pipes\utils.
From an Email from Jeff Hennick here is how IRange works:
$ Iranges in njpipes are a superset of those in CMS pipes. CMS supports
$ only simple character ranges in parenthisis, njpipes alows any combinations,
$ including SUBSTRING ... OF ..., etc., and -n;-m as a single range.
A Rexx string is scanned for a inputrange word or a parenthesized group
of inputranges and modifiers. An IRange object is created for each
range, and returned as an array. The 0-element contains information
about the entire group: number of elements, and the rest of the input
string after the inputrange(s) have been removed. Usually, the input
string is part of a stage argument.
Input ranges can be specified by character, word, or fields; as singles,
or consecutive positions. They may be referenced to either the left or
right end of the string. They may be subranges of a range. The
character to define words and fields may be specified.
The IRanges are then used to select a sub-string of any Rexx string.
Additional information is then also available: the sub-string's
position in the string, the contents of the string before and after the
sub-string, etc. Usually, the string is the input record read by the stage.
Examples:
$ instring = 'Word 4-7' -- usually, this string is part of the stage argument
-- or --
$ instring = '(20.4 word 4-7 fieldseparator - f3 words-4.2 -40.8) some more data'
ir = IRange[]
ir = GetIRanges(instring) -- parses the string into range objects
-- these can be used now
$ numberOfRanges = ir[0].Number() -- "5" for the above instring
$ extra = ir[0].Residue() -- "some more data" for the above
$ -- usually, the dataString would be the input string
$ -- from peekto(), inside the main loop of a stage
$ dataString = '-NetRexx for Java -Create Java code faster -For easier internet programming'
-- a single sub-string can be picked:
$ sub_string = ir[2].selIRange(dataString)
-- "-Create Java code faster" -- word 4-7
$ -- and a series of additional information is
$ -- also available. For example:
starting = ir[2].StartCol() -- "19"
len = ir[2].Length() -- "24"
last = ir[2].EndCol() -- "42"
before = ir[2].Prefix() -- "-NetRexx for Java "
after = ir[2].Residue() -- " -For easier internet programming"
$ Delimited Strings in Stage Arguments
$
$ To aid in identifying and using full CMS-type delimited strings a type and methods
$ are provided in the pipes package.
$
$ Where a delimited string may be at the beginning of a string, use:
$
$ arg_string = '/The NetRexx Language/ by M. F. Cowlishaw' -- Maybe
$
$ ds = DString(arg_string) -- ds is a DString object
$ delimstring = ds.String() -- "The NetRexx Language"
$ rest = ds.Residue() -- " by M. F. Cowlishaw"
$
$ If no delimited string is found at the beginning of the input string, the String()
$ method returns "", and the full string is returned by Residue(). If the Residue is
$ not needed, use:
$
$ delimstring = DString(arg_string).String
Back to Table of Contents
Writing your own pipes or stage is simple. Take a look at the source of the
supplied stages in the stages directory. Here are some more examples. The
first shows how to use a pipe in a NetRexx program:
-----------------------------------------------------------------------
-- examples\testpipe.njp
-- to compile use: pipe testpipe.njp
-- or: java njp testpipe.njp
-- to execute use: java testpipe some text
import pipes.
import stages.
class testpipe extends Object
method testpipe(avar=Rexx)
F = Rexx 'abase'
T = Rexx 1
F[0]=5
F[1]=222
F[2]=3333
F[3]=1111
F[4]=55
F[5]=444
pipe (apipe stall 1000 )
stem F ! sort ! prefix literal {avar} ! console ! stem T
loop i=1 to T[0]
say 'T['i']='T[i]
end
method main(a=String[]) static
testpipe(Rexx(a))
-----------------------------------------------------------------------
A couple things can be inferred from this example. First its simple to
pass rexx variables to pipes using STEM. Also look at the phrase {avar}.
It passes the Rexx variable's value to the stage at runtime. In CMS the pipe
would be quoted and you would unquote sections to get a similiar effect.
Another thing to note is that the pipe extraction program is fairly smart.
It detects when pipes takes several lines. As long as there are stages,
or the current line ends with a stagesep or stageend character, or the next
line starts with a stagesep or stageend character. It gets added to the
pipe.
$ The arg(), arg(rexx) or arg(null) methods get the arguments passed to a
stage or pipe. To get the complete rexx string of an arguement use arg().
To get the nth word of a rexx arguement use arg(n). When using pipes in
netrexx or java code you can use arg('name') to get the named arguement.
If the class of the arguement is not rexx use arg(null) to get the object.
In .njp files you can use {avar} phrase actually just shorthand for
arg('avar').
The following example shows what has to be done in a
stage to access the rexx variables passed by VAR, STEM and OVER. The real
over stage is a bit more complete.
-----------------------------------------------------------------------
-- over.nrx
package stages
import pipes.
class over extends stage final
method run() public
a = getRexx(arg())
loop i over a
output(a[i])
catch StageError
rc = rc()
end
exit(rc*(rc<>12))
-----------------------------------------------------------------------
The getRexx method is passed the name of a string by the pipe. In the
previous example it would be passed A and would return an Object pointer
to A in testpipe.
If you wish to replace a stream this can be done using
connectors. For example look at the following fragment:
-----------------------------------------------------------------------
-- examples\calltest.njp
pipe (callt1) literal test ! calltest {} ! console
import pipes.
class calltest extends stage final
method run() public
do
a = arg()
callpipe (cp1) gen {a} ! *out0:
loop forever
line = peekto()
output(line)
readto()
end
catch StageError
rc = rc()
end
exit(rc*(rc<>12))
-----------------------------------------------------------------------
$ Running the callt1 pipe with an argument of 10 would pass the 10 to
calltest via {} and arg(). Then cp1's gen stage would be passed 'a'
which is set to 10. Since gen generate numbers in sequence, the console
stage of callt1 would get the numbers from 1 to 10. Now cp1 ends and
calltest's output stream is restored and calltest unblocks and reads the
the literal's data 'test' and passes it to console.
The use of {} only works when compiling from .njp files. It will not
work from the command line.
The njpipes compiler reconizes connectors as labels
with the following forms:
*in:
*inN:
*out:
*outN
When N is a whole number, the connector connects input or output
stream N of the stage with the connector.
When the label *in or *out, the connector connects the stages's current
input or output stream with the connector. This is used instead of *: due
to the way the compiler/preprocessor works.
If you do not want the stage to wait for the called pipe to complete you
can use addpipe. Here is an example.
-----------------------------------------------------------------------
-- similar to examples\addtest.njp
a = 100
b = 'some text for literal'
addpipe (linktest) literal {b} ! dup {a} ! *in0:
loop forever
line = Rexx readto()
catch StageError
end
-----------------------------------------------------------------------
readto() will get 'some text for literal' one hundred times.
| A quick aside. When writing stages remember that njPipes moves objects
| through pipes. Use 'value = peekto()' instead of 'value = rexx peekto()'
| when ever possible. Some of the supplied stages pass objects with
| classes other than rexx and forcing rexx will cause classCastExceptions.
| If a stage needs a rexx object try using the rexx stage modifier to attempt
| to convert the object. Feel free to expand this stage, but please
| send me the updated version.
DISABLED The following example shows how to use a pipe as a stage:
DISABLED
DISABLED -----------------------------------------------------------------------
DISABLED
DISABLED (linkpipe) gen arg() ! *out:
DISABLED
DISABLED (linktest)
DISABLED genpipe= linkpipe(100) ?
DISABLED .genpipe.out !
DISABLED locate /0/ !
DISABLED console
DISABLED
DISABLED -----------------------------------------------------------------------
DISABLED
DISABLED The syntax of the above pipe is a bit different. It works like this. Pipes
DISABLED are special cases of stages. A pipe that is intended to be used in place of
DISABLED a stage must define connectors. When a pipe is used as a stage the
DISABLED convention is: The instance of the pipe must be defined (genpipe=), when
DISABLED the instance is used the syntax is: {connector}.pipeinstance.{connector}
DISABLED
DISABLED This is done since, in this implementation, pipes must be precompiled.
| Serious stage writers will probably want to take a good look at the methods
| defined in pipes\utils.nrx. There you will find various methods for parsing
| ranges. You will also find the stub for the stageExit compiler exit. It can
| be used to produce 'on the fly' code at compile time. You can also use it
| to change the topology of the unprocessed part of the pipe. The major use
| is to allow implementations of stages like prefix, append or zone. Its also
| used to produce better performing stages, for an example see specs.
| The compiler also queries the rexxArg() and stageArg() methods. If your
$| stage expects objects of class Rexx as arguments rexxArg() should return
| the number of variables expected. If your stage expects a stage for an
$| argument, stageArg() should return the word position of the stage.
To get a good idea of what can be done with pipes look at the tasktest pipe
in the examples directory. It, using code from Melinda Varins 'Cramming
for the Journeyman Plumber Exam' paper, implements the shell of a
multitasking server - using about eight stages. The file examples\tcptask.njp
contains an example of this technique being used.
Back to Table of Contents
Pipes for NetRexx and Java detects deadlocks and outputs information to allow
you to fix the problem. Consider the following session:
-----------------------------------------------------------------------
[D:\njpipes]java njp (deadlock) literal test ! a: fanin ! console ! a:
Pipes for NetRexx and Java version 0.33
Copyright (c) E. J. Tomlinson , 1998. All rights reserved.
Building pipe deadlock
NetRexx portable processor, version 1.140
Copyright (c) IBM Corporation, 1998. All rights reserved.
Program deadlock.nrx
Compilation of 'deadlock.nrx' successful
[D:\njpipes\examples]java -nojit deadlock
test
Deadlocked in deadlock
Dumping deadlock Monitored by deadlock
Flag units digit: 1=wait out, 2=wait in, 4=wait any, 8=wait commit
: 10=pending autocommit, 20=pending sever
literal_1
Running rc=0 commit=-1 Flag=201
-> out 0 fanin_2 1 test
fanin_2
Running rc=0 commit=-1 Flag=101
-> in 0 literal_1 1 test
in 1 console_3 0 test
-> out 0 console_3 1 test
console_3
Running rc=0 commit=-1 Flag=101
-> in 0 fanin_2 1 test
-> out 0 fanin_2 0 test
Dumped Pipe deadlock Flag 40F rc=16
RC=16
-----------------------------------------------------------------------
We can see that there are three stages Running. None have any return
codes set. The Flags tell us that all the stages are waiting for an
output to complete. The '->' show which stream is selected. From this
we can see console_3 is trying to output to fanin_2. Unfortunately fanin_2
is waiting for output on stream 0 to complete, it cannot read the data
waiting on in stream 1. Hence the stall.
When a stream has data being output, there is a boolean flag following the
name of the stage the stream is connected to. This tracks the peek state of
the object. For an output stream, true means the following stage has peeked
at the value. With input streams, the current stage has seen the value when
its true.
When a stage is multithreaded, like elastic, you can get flags of 3 or 5.
This means that threads are waiting on output and read, or output and any.
When using multithreaded stages, only one thread should use output unless it
is serialized using protected or syncronized blocks.
When a stage has a pending sever or autocommit flag bits are set too.
Back to Table of Contents
I expect problems with add/callpipe. The cases tested in addtest1 thru
addtest8, calltest1 thru calltest3, siptest, tcpexample, tcptask and
spawntest, all work ok but other types of connections may or may not work
correctly. Good news is that the last few tests (tcptask, spawntest) did
not break anything. The testcases also work on a dual processor system.
The command stage can fail when more that one instance is used at the same
time. This is due to a Java OS/2 problem with unnamed pipes. Also passing
an invalid command can terminate your program... There are also problems
with command under W95. Command should be rewritten to use a thread for
each output stream and it should handle cases where syserr does not exist.
While this is not strictly speaking a bug, it can cause unexpected results.
Often using a JIT can slow down pipes. This is because java, and hense
the JIT, likes to garbage collect objects quickly. Pipes load lots of
classes, the JIT compiles them, the pipe ends and java garbage collects
the objects and the JIT loses its code. In release 027 I added code to
help callpipe and addpipe used in stages. In release 029 the cache option
was introduced so you can pass a Hashtable from your main routine to pipes
to be used for the object cache. This way calling a pipe many times will
not end up causing the JIT to work too hard.
| Sun's JIT for NT and W9x will sometimes raise a NullPointerException when
| none should occur. The fix is to use the -nojit option when running the
| pipe or to upgrade to the, prerelease, JIT available at Sun's javasoft
| site. In general, when something wierd happens with java, try again with
| the JIT off.
Please Report bugs to: tomlins@cam.org
Back to Table of Contents
0.50 - Released May 30, 1999
- Fixed a stall occuring when interruped threads, with the interrupt
caught by ThreadPool, were reused.
- Fixed a thread safety problem in ELASTIC
- Improved the timeout options in TCPDATA and TCPCLIENT, they also
byte[] instead of strings. This was done since converting to and
from strings sometimes scrambles binary data (more research on
encodings...)
- Changed DELBLOCK it now handles byte[] to help keep tcpdata and
tcpclient efficent. The EOF option was broken, its fixed now.
- Changed DISKR, DISKW and DISKA to handle byte[] when using streams.
- Added INSERT which handles byte[]. This should be used instead of
SPECS to add LF or CR .
- Changes SERIALIZE to use byte[].
0.49 - Released May 21, 1999
- compiled with 1.2.1 and netrexx 1.148
- Added preliminary support added to .njp compiler for files containing
java source! See the (some what messy) java samples in vectort1.njp,
overtest.njp and addtest4.njp
- Added code to generate a dummy .nrx file containing the public class
in a .java file. This allows NetRexx to compile class that depend on
the java source.
- Modified sort to accept arguements in the same order as CMS
- Fixed rc logic in drop stage
- Fixed shortcut code for {n} where n is numeric.
0.48 - Released May 16, 1999
- Fixed a (nasty) bug involving reusing pipe objects.
- Added the reuse() method to the stage class. To use it override
it in your stage. It was added so there was a foolproof way to
reset a stage when its pipe object is reused. (doSetup is intended
for use with dynamic arguements in call or added pipes)
- Added the cont option and defaulted it to comma.
- fixed return code logic in some stages and in selectInput/Output
- Added the Emsg methods
- Added arguement debug option (128)
- There are no more final methods
- Much improved error reporting from stages via new Emsg method
0.47 - Released Jan 3, 1999
- recompiled with 1.1.7A and netrexx 1.148
- UNIQUE repaired?
- Added stages to acess java objects easily
VECTOR, VECTORR, VECTORW, VECTORA for java.util.vector
ARRAY, ARRAYR, ARRAYW, ARRAYA for Object[]
HASH, HASHR, HASHW, HASHA for java.util.Hashtable
DICT, DICTR, DICTW, DICTA for java.util.Dictionary
The hash stages mostly map directly to DICT stages. The exception
is HASHW which uses the clear() method of Hashtable.
- Modified LITERAL to be able to put any object into a pipe
- Modified pipe package to store arguements in a hashtable instead of
a rexx stem - arguements can now be of any class. Use the arg(null)
method to get an object arguement.
0.46a - Released Oct 14, 1998
- recompiled with 1.1.7
- TCPLISTEN now supports an input stream to be used to pace accepts
0.46 - Released Sept 20, 1998
- COMMAND, CHANGE, FILE, LOCATE, DROP, LOOKUP, TCPCLIENT, TCPLISTEN
SQLSELECT, CONSOLE, TCPDATA, NOEOFBACK improved.
- Jeff improved the testing process with the addition of the COMPARE
stage, he also upgraded many of the tests.
- Added the buildtests pipe, it builds a test script to be run with:
test > output < console.data
- Unexpected exceptions should no longer hang pipes
0.45 - Released Sept 9, 1998
* Recompile all your stages. To fix a commit problem I had to
change the _stage interface class...
- tcpclient restart problems with oneresp active fixed.
- commit now returns the current return code of the pipe.
- fixed minor errors in tcpclient and diska.
0.44 - Released Sept 8, 1998
* a recompile of pipes using STEM is required
- smart DISK, FILE and STEM stages now exist.
- Made to and from synonoms for in and out in REXX and STRING stages.
- Added stream option to DISKR and DISKW to read raw streams.
- Added DISKSLOW and SERIALIZE stages.
- Now DISK, DISKR, DISKW, DISKA and DISKSLOW have FILE synonyms.
- Deadlock detection improvements.
- TCPDATA & TCPCLIENT optimized once again.
- selectAnyInput could deadlock - fixed.
- interrupting a pipe now kills it - use this with care (ie. kill -9)
- Pseudo methods njpRC() and njpObject() are reconized by the pipes
compiler and return the pipe's RC or object respectivily.
0.43 - Released August 30, 1998
- Fixed deadlock dection to see commit deadlocks.
- Added rest of code to handle improved StageError logic.
- Added stage templates (template*.nrx) in the njpipes directory.
- Added a debug flag (64) to trace all StageError rasied by the
stage class.
0.42 - Not released
* A recompile of pipes using TCPCLIENT, TCPDATA is required.
* A recompile of pipes using REXX, STRING, ZONE, CASEI is recommended.
- Updated the comments in _stage to reflect the possible StageError
and return codes that can be issued.
- Added the DEBLOCK stage and reworked TCPDATA, TCPCLIENT & GATE.
- Improved eofReport processing and added a new option 'either' that
will trigger a StageError when any stream, input or output, severs.
- Fixed variable subsitution so multiple variables passed to a stage
will work.
- Added the ability to pass thru arguements to callpipe and addpipe.
- Fixed a problem with some StageExits requiring stage_reset methods.
- Added a function to utils to help assign smarter name to classes
generated by StageExits.
- Added counter method to stage. use to count external waits so
deadlock/stall detection is not fooled.
0.41 - Released August 23, 1998
* removed OBJ2REXX, OBJ2STRING stages, use REXX and STRING stage
modifiers.
* pipes using TCPDATA, TCPCLIENT & LOOKUP should be recompiled
- exhanced REXX stage modifier via an object2rexx improvement in
pipes/utils.nrx
- optimized ThreadPool startup times. No setName and only use
setPriority when its required.
- made it possible to optimized stage startup time when arguements
are static. See TCPDATA, TCPCLIENT & LOOKUP
- faq.txt enhanced
0.40 - Released August 14, 1998
* All pipes MUST be recompiled. Old pipe class files will stall.
- OBJ2REXX is depreciated and will be removed, use the REXX stage.
- added REXX and STRING stages to convert objects entering and leaving
a stage to rexx or string. Inorder to avoid nasty class conflicts,
REXX and STRING are implemented in _rexx and _string. The compler
adds the '_' when necessary (any stage can use this feature).
- fixed an intermitant stall in callpipe (was completing too fast :-)
- fixed a stall occuring between shortStreams and COMMAND
- optimized pipe startup time in pipe.class and via the compiler.
- optimized rc, commit, deadlock, threadpool code
0.39 - Released August 9, 1998
- WAIT_COMMIT and WAIT_ANY are now used in the call/addpipe logic
- callpipe was not notifiing its pipe when ending leading to an
very intermitant hang.
0.38 - Released August 3, 1998
* All your stages must be recompiled. Recompile your pipes to
exploit the pipe & thread pool performance improvements.
- fixed and optimized commit logic.
- implement a pool for pipes to decrease overhead.
- implement a pool for threads to decrease overhead.
- compiler fix to proprogate return codes from stageExits (thanks Jeff).
- signal StageError('... in all stageExits modified to
signal StageError(13,'Error - 'pInfo' - ...
- UNIQUE stage added by Jeff. It exploits stageExit.
- COMMAND stage was not starting its threads correctly.
- SORTs in different pipes could corrupt each other. Thanks René
0.37 - Released July 25, 1998
* A recompile of pipes using SORT is required
- added NOEOFBACK, TOTARGET and FRTARGET.
- removed a protected method from dump(), added arg() to the dump
- upgraded SORT, sortRexx to exploit IRange and stageExit, optimized
use, and factored the sort algorthym out of sort/sortRexx.
- multiple sort stages no longer try to share static variables...
- the compiler just uses the stage name (not args) when naming stages
0.36 - Released July 19, 1998
* A recompile of ALL pipes with stages using IRANGE is required.
(CHANGE, DEAL, JOINCONT, LOCATE, LOOKUP, PICK, XLATE & ZONE)
* pipes using NFIND, NLOCATE, STRNFIND or SORT also need to be
recompiled
- Added BuildIRangeExit and other methods to an updated IRange
class. Using 'zone range stage ...' will be faster than
'stage range ...' when the range consists of n.c or n-c (s).
- NFIND, NLOCATE, STRNFIND implemented via stageExit and NOT
- Fixed bugs in, JUXTAPOSE, FIND, STRFIND, SORT, COMMAND, CHANGE
- The compiler was not calling stageExit in the correct order when
several calls were needed to build the stage. (zone w1 nfind..)
0.35 - Released July 16, 1998
- Jeff Hennick pointed out a bugglet that effected LOOKUP, ZONE and
PICK that could occur with complex ranges, I found another bug in
strliteral
- Jeff Hennick updated this doc with information on IRange and DString
0.35 - Released July 15, 1998
* A recompile of ALL pipes using ZONE, TCPCLIENT, TCPDATA, PREFIX
and APPEND is required.
- prefix and append can now be labeled, tcpclient and tcpdata
now use a stage, instead of a pipe, to group data.
- added compiler support for negitive stream numbers. This is
intended to be used by stageExit. See append, prefix, tcpdata
and tcpclient.
- Redefined rexxArg() and stageArg() to simplify the compiler.
- selection stages are no longer defined as final.
- SelectInput(0) and selectOutput(0) are always called by the
stage implementation so they can be overridden...
- Reimplemented ZONE using stageExit, added CASEI using the same
technique. In theory NOT could be done the same way but, to
avoid some recursion problems NOT is staying in the compiler.
- StageExit modified to allow it to pass back another stage to
call. see ZONE, CASEI and NOT.
0.34 - Released July 11, 1998
- minor reportEOF(any) logic fix
- improved command stage, threads used to process stdout and stderr.
added zone, pad, lookup, pick, upgraded juxtapose, fixed bugs in
specs & buffer.
- added pad option to setIRange method
0.33 - Released July 5, 1998
- added rexxArg() and stageArg() methods to utils.nrx for use by the
$ compiler to query stages about what they expect their arguments to
contain. This allowed the compiler to be simplified.
$ - locate now handles null arguments correctly. literals now include
leading blanks. Thanks for pointing out the problem René.
- René Jansen contributed the timestamp stage.
- logic modified to stop output() from getting an EOF when the output
object has been peeked. The peek status is also displayed by the
dump() method and hense by deadlocks.
- minor specs bug fixes (next.n and nextw.n output specs now work)
- modified the compiler to invoke stageExit(rexx,rexx) method. This
allows stages to generate code and/or change the pipe topology. See
specs, append, prefix, change and xnop, in the stages directory.
- modified StageError in preparation for usage changes.
- removed the Range class - Jeff's code is better and anything that
could be done with Range can be done using stageExit.
- Jeff fixed bugs in change and join and added:
fblock joincont notinside outside
inside
0.32 - Released June 20, 1998
Jeff updated these stages adding a few new ones too:
abbrev between split locate
nlocate strnfind strfind nfind
find chop
- minor docuementation updates
- the Range class is depreciated and will be removed. Use the
replacements Jeff created (see pipes\utils.nrx and stages\).
0.31 - Released June 17, 1998
- modified count, drop, take and deal to handle non rexx objects
when possible
0.31 - Released June 16, 1998
- improved eofReport(ANY) logic to trigger when waiting on output
and a different output stream severs.
- factored the source for utils.class out of stages so there is
a class to add (probably static) shared methods for all stages
- fixed a deadlock that occured between shortStreams and exit
(severInput)
- Jeff Hennick updated many stages to work at CMS or near CMS levels.
append deal join strfrlabel xlate
buffer drop literal strliteral
change fanin locate strtolabel
console fanout split take
count frlabel strfind tokenize
All of Jeff's changes are GNUed. See CopyLeft.txt in the njpipes
directory.
0.30 - Released May 24, 1998
- fixed logic in core classes to post all pending severs and not
clear them too early either, this corrects a problem seen on
Multiprocessor machines.
0.29 - www page update (docuemention) May 20
- deadlock section updated
- installation verification example corrected!
0.29 - Released May 17, 1998
- added obj2rexx stage, tolabel stage courtesy of Chuck Moore.
- enhanced change to support a single range
- Added setJITCache(Hashtable) method to pipes. This can be used
to build a global object cache in programs calling pipes. The name
of the Hasttable is passed to pipe/callpipe/addpipe via a cache
parameter.
- Added support for reportEof options. This support is not too
well tested - some good testcases are needed.
0.28 - Released May 9, 1998
- Enhanced parsing in specs (word2.1 would work, word 2.1 would not)
- Fixed COPY for a NT jit bug, fixed locate so NOT LOCATE would
work, updated LITERAL not to use more than one exit(rc)
- Fixed a compiler problem that would hit multistreamed pipes using
append or prefix.
- Any options not consumed by njp are passed on to nrc
and java. Mainly for use from the command line, use with care
in .njp files...
- Fixed shortStreams() so it works correctly when shorting streams
in a stage with multiple streams.
- Tested all 8 addpipe forms and fixed runtime to work with all
test cases
- modified filternjp to accept *in and *out without additional labels
- reenabled stop() in exit code...
- added gate, dam, tokenize, juxtapose and courtesy of Chuck Moore,
frlabel stages
0.27 - Released May 3, 1998
- Automated the generation of in/outStream calls. For this to work
the labels need to be of the form *in0: or *out0: where the '0' is
replaced by the input or output stream to connect to.
- Fixed compiler/filter problems with stema
- Tighted range checking code in specs, fixed problem with delimited
ranges. Specs was compiling the NetRexx EXIT command...
- Fixed a problem where output was not see that objects were
consumed when using sipping pipes...
- Fixed a problem where severing an output stream did not cause the
stages stacked on the node's outlist to see the sever
- Fixed a problem where the stage issuing a callpipe was not seeing
the called pipe end
- Added debug option to pipes compiler
- Repaired commit and added commit levels to dump() method
- Fixed problems with callpipe servering several outputs, unstacking
the saved stream was selecting it...
- Modified tcpclient and tcpdata to use a secondary thread to
recieve the tcpip inputs.
- Now keep a referenced object for each pipe/stage so the JIT does
not throw away its work and call/addpipes in loops work faster.
- in/outStreamState now return -1 when autocommit is enabled and
the stream is unused.
0.26 - Released April 26, 1998
- Added selection methods to compiler (see getRange in section 4 and
the locate stage an example#
- Added the specs stage. The compiler builds a stage to process the
specs, reducing overhead.
- Added tcp/ip stages
- Fixed problems with severs using addpipe
0.25 - Optimized some stages using jinsight from www.alphaworks.ibm.com.
This more that doubled the speed of some stages.
- fixed bugs in fanin, diskw
- Added netrexx filters to extract pipes, extended the functions
of .njp files (multiple pipes in a file and .njp files can now
contain netrexx code with pipe/callpipe/addpipe)
- fixed a timing bug in deadlock detection.
- xlate and sqlselect stages contributed by René Jansen added
0.24 - Release Feb 98
- modified the compiler so the syntax of pipes from the command line
is the same as pipes from .njp files
- added the sort stage, the sortClass interface and the sortRexx
example implementation
- added the timer stage
0.23 - fixed minor compiler errors (20 Dec 97)
- not stage modifier added.
- errors in this page corrected, NT install information added.
- modified diskr/diskw to use Buffered Streams.
0.22 - second public release
0.21 - enabled auto commit, stages start at a commit level of -2 and
commit to a level of -1 at the first readto, peekto or output.
nocommit disables the auto commit. This feature has not been
completely tested (yet).
- fixed compiler not to call netrexx if one of its pipes deadlocks
0.20 - Upgraded to May version of the NetRexx compiler (Thanks Mike!)
this changed the compiler interface. NetRexx from May 10 or
later is now required.
- nocommit added to _stages, though its a nop for now
- modified the compiler class to use the May 10th NetRexx compiler
0.19 - initial public release (4 May 97)
Planned for the next month or three (pending BUGS! and my time)
0.xx - make pipe objects java beans.
0.xx - implement the new StageError/Emsg code in all stages
0.xx - rewrite the stage class to use notifyAll/wait to pass data and
interupt to pass flow control info (ie. severs...). This will
result in an implementation that is easier to understand and
maintain.
Back to Table of Contents
<pipe> = {<pipestages>} <stage> { <sep>|<end> <stage> }
<pipestages> = <defstage> {<end> <defstage>} <end>
<defstage> = <pipeinstance>= <pipeclass>({<arguments})
<sep> = stage seprator (!)
<end> = end of pipe character (?)
<stage> = {<pipeasstage> {<sep> nop <sep> <pipeasstage>} }
| {<label>} { not } <stageclass> {<arguments>}
<pipeasstage> = {<connector>}.<pipeinstance>.{<connector>}
<label> = <labelname>: | *<connector>:
$ <arguments> = a string of characters
| arg({<index>})
<stageclass> = the name of a stage .class file
<pipeclass> = the name of a pipe .class file
<pipeinstance> = a non numeric string of characters
<labelname> = a non numeric string of characters
<connector> = {in | out}{a string of characters}
<index> = Index of a rexx argument (see calling pipes from NetRexx)
Back to Table of Contents
Commercal restrictions are as follows:
1. Contact information is required, an email address is prefered.
There are several reasons for this. The first is that if I stop working on it
I will make the complete source available and would like to notify sites that
may be depending on the product. A second reason is that if ever a better
implementation of the core classes is developed, or a serious bug is corrected
I will notify Commerical users. Your email address will not be redistributed
without permission.
2. You are free to redistribute Pipes for Netrexx and Java but cannot charge for
it. You can include it, or subsets of it, in products you charge for but must
reference my site (http://www.cam.org/~tomlins/njpipes.html).
Back to Table of Contents
Java is a trademark of Sun Microsystems
Windows NT is a trademark of MicroSoft
The following terms, are trademarks or registered
trademarks of IBM Corporation in the United States and/or other countries:
IBM
OS/2
NetRexx
CMS
Back to Table of Contents