Carbon


TokenBlock

Header: Script.h

struct TokenBlock {
    Ptr source; 
    SInt32 sourceLength; 
    Ptr tokenList; 
    SInt32 tokenLength; 
    SInt32 tokenCount; 
    Ptr stringList; 
    SInt32 stringLength; 
    SInt32 stringCount; 
    Boolean doString; 
    Boolean doAppend; 
    Boolean doAlphanumeric; 
    Boolean doNest; 
    ScriptTokenType leftDelims[2]; 
    ScriptTokenType rightDelims[2]; 
    ScriptTokenType leftComment[4]; 
    ScriptTokenType rightComment[4]; 
    ScriptTokenType escapeCode; 
    ScriptTokenType decimalCode; 
    Handle itlResource; 
    SInt32 reserved[8];
};
typedef TokenBlock TokenBlockPtr;

Field descriptions

source

A pointer to a stream of characters.

sourceLength

The length of the source stream.

tokenList

A pointer to an array of tokens.

tokenLength

The maximum length of TokenList

tokenCount

The number of tokens generated by the tokenizer.

stringList

A pointer to a stream of identifiers.

stringLength

The length of the string list.

stringCount

The number of bytes currently used.

doString

A Boolean value indicating, on input, whether the IntlTokenize function should create a Pascal string representing the contents of each token it generates.

doAppend

A Boolean value indicating, on input, whether to append tokens and strings it generates to the current token list and string list.

doAlphanumeric

A Boolean value indicating, on input, whether to interpret numeric characters as alphabetic when mixed with alphabetic characters.

doNest

A Boolean value indicating, on input, whether to allow nested comments (to any depth of nesting).

leftDelims

An array of two integers, each of which contains the token code of the symbol that may be used as an opening delimiter for a quoted literal. If only one opening delimiter is needed, the other must be specified to be delimPad.

rightDelims

An array of two integers, each of which contains the token code of the symbol that may be used as the matching closing delimiter for the corresponding opening delimiter in the leftDelims field.

leftComment

An array of two pairs of integers, each pair of which contains codes for the two token types that may be used as opening delimiters for comments.

rightComment

An array of two pairs of integers, each pair of which contains codes for the two token types that may be used as closing delimiters for comments.

escapeCode

A single integer that contains the token code for the symbol that may be an escape character within a quoted literal.

decimalCode

A single integer that contains the token type of the symbol to be used for a decimal point

itlResource

A handle to the tokens ('itl4') resource of the script system under which the source text was created.

reserved

An 8-byte array of type LongInt.On input, this must be set to 0.

The token block structure is a parameter block used to pass information to the IntlTokenize function and to retrieve results from it.


© 2000 Apple Computer, Inc. (Last Updated 6/30/2000)