Primitive Tokens

   Token                Token Type      Flags

   identifier           IDENT            b e
   integer-literal      INTLIT           b e
   real-literal         REALLIT          b e
   string-literal       STRINGLIT        b e
   cset-literal         CSETLIT          b e
   end-of-file          EOFX

Reserved Words

   Token                Token Type      Flags

   break                BREAK            b e
   by                   BY
   case                 CASE             b
   create               CREATE           b
   default              DEFAULT          b
   do                   DO
   else                 ELSE
   end                  END              b
   every                EVERY            b
   fail                 FAIL             b e
   global               GLOBAL
   if                   IF               b
   initial              INITIAL          b
   invocable            INVOCABLE
   link                 LINK
   local                LOCAL            b
   next                 NEXT             b e
   not                  NOT              b
   of                   OF
   procedure            PROCEDURE
   record               RECORD
   repeat               REPEAT           b
   return               RETURN           b e
   static               STATIC           b
   suspend              SUSPEND          b e
   then                 THEN
   to                   TO
   until                UNTIL            b
   while                WHILE            b

``Operator'' tokens not used in standard unary/binary syntax, see op.txt.

   Token                Token Type      Flags

   (                    LPAREN           b
   )                    RPAREN             e
   +:                   PCOLON
   ,                    COMMA
   -:                   MCOLON
   :                    COLON
   ;                    SEMICOL
   [                    LBRACK           b
   ]                    RBRACK             e
   {                    LBRACE           b
   }                    RBRACE             e

tokens starting with $ are alternate spellings for some tokens

   $(                   LBRACE           b
   $)                   RBRACE           e
   $<                   LBRACK           b
   $>                   RBRACK           e

Explanation of Flags

   b    indicates that the token may begin an expression.
   e    indicates that the token may end an expression.

   These two flags are used for semicolon insertion.  If a line
   ends with an "e" token, and the next token is a "b" token,
   a semicolon is inserted between the two tokens.