Carbon


TokenRec

Header: Script.h

struct TokenRec {
    ScriptTokenType theToken; 
    Ptr position; 
    SInt32 length; 
    StringPtr stringPosition;
};
typedef TokenRec TokenRecPtr;

Field descriptions

theToken

A numeric code that specifies the type of token (such as whitespace, opening parenthesis, alphabetic or numeric sequence) described by this token structure. Constants for all defined token codes are listed in “Token Codes”.

position

A pointer to the first character in the source text that caused this particular token to be generated.

length

The length, in bytes, of the source text that caused this particular token to be generated.

stringPosition

If doString = TRUE, a pointer to a null-terminated Pascal string, padded if necessary so that its total number of bytes (length byte + text + null byte + padding) is even. If doString = FALSE, this field is NULL.

The value in the length byte of the null-terminated Pascal string does not include either the terminating zero byte or the possible additional padding byte. There may be as many as two additional bytes beyond the specified length.

The token structure holds the results of the conversion of a sequence of characters to a token by the IntlTokenize function. When it analyzes text, IntlTokenize generates a token list, which is a sequence of token structures.


© 2000 Apple Computer, Inc. (Last Updated 6/30/2000)