1 Extended Characters

A number of languages use character sets that are larger than the range of values of type char. Japanese and Chinese are probably the most familiar examples.

The GNU C library includes support for two mechanisms for dealing with extended character sets: multibyte characters and wide characters. This chapter describes how to use these mechanisms, and the functions for converting between them.

The behavior of the functions in this chapter is affected by the current locale for character classification—the LC_CTYPE category; see @ref{Locale Categories}. This choice of locale selects which multibyte code is used, and also controls the meanings and characteristics of wide character codes.

1.1 Introduction to Extended Characters

You can represent extended characters in either of two ways:

• As multibyte characters which can be embedded in an ordinary string, an array of char objects. Their advantage is that many programs and operating systems can handle occasional multibyte characters scattered among ordinary ASCII characters, without any change.
• As wide characters, which are like ordinary characters except that they occupy more bits. The wide character data type, wchar_t, has a range large enough to hold extended character codes as well as old-fashioned ASCII codes.

  An advantage of wide characters is that each character is a single data object, just like ordinary ASCII characters. There are a few disadvantages:

  • Each existing program must be modified and recompiled to make it use wide characters.
  • Files of wide characters cannot be read by programs that expect ordinary characters.

Typically, you use the multibyte character representation as part of the external program interface, such as reading or writing text to files. However, it’s usually easier to perform internal manipulations on strings containing extended characters on arrays of wchar_t objects, since the uniform representation makes most editing operations easier. If you do use multibyte characters for files and wide characters for internal operations, you need to convert between them when you read and write data.

If your system supports extended characters, then it supports them both as multibyte characters and as wide characters. The library includes functions you can use to convert between the two representations. These functions are described in this chapter.

1.2 Locales and Extended Characters

A computer system can support more than one multibyte character code, and more than one wide character code. The user controls the choice of codes through the current locale for character classification (@pxref{Locales}). Each locale specifies a particular multibyte character code and a particular wide character code. The choice of locale influences the behavior of the conversion functions in the library.

Some locales support neither wide characters nor nontrivial multibyte characters. In these locales, the library conversion functions still work, even though what they do is basically trivial.

If you select a new locale for character classification, the internal shift state maintained by these functions can become confused, so it’s not a good idea to change the locale while you are in the middle of processing a string.

1.3 Multibyte Characters

In the ordinary ASCII code, a sequence of characters is a sequence of bytes, and each character is one byte. This is very simple, but allows for only 256 distinct characters.

In a multibyte character code, a sequence of characters is a sequence of bytes, but each character may occupy one or more consecutive bytes of the sequence.

There are many different ways of designing a multibyte character code; different systems use different codes. To specify a particular code means designating the basic byte sequences—those which represent a single character—and what characters they stand for. A code that a computer can actually use must have a finite number of these basic sequences, and typically none of them is more than a few characters long.

These sequences need not all have the same length. In fact, many of them are just one byte long. Because the basic ASCII characters in the range from 0 to 0177 are so important, they stand for themselves in all multibyte character codes. That is to say, a byte whose value is 0 through 0177 is always a character in itself. The characters which are more than one byte must always start with a byte in the range from 0200 through 0377.

The byte value 0 can be used to terminate a string, just as it is often used in a string of ASCII characters.

Specifying the basic byte sequences that represent single characters automatically gives meanings to many longer byte sequences, as more than one character. For example, if the two byte sequence 0205 049 stands for the Greek letter alpha, then 0205 049 065 must stand for an alpha followed by an ‘A’ (ASCII code 065), and 0205 049 0205 049 must stand for two alphas in a row.

If any byte sequence can have more than one meaning as a sequence of characters, then the multibyte code is ambiguous—and no good. The codes that systems actually use are all unambiguous.

In most codes, there are certain sequences of bytes that have no meaning as a character or characters. These are called invalid.

The simplest possible multibyte code is a trivial one:

The basic sequences consist of single bytes.

This particular code is equivalent to not using multibyte characters at all. It has no invalid sequences. But it can handle only 256 different characters.

Here is another possible code which can handle 9376 different characters:

The basic sequences consist of

• single bytes with values in the range 0 through 0237.
• two-byte sequences, in which both of the bytes have values in the range from 0240 through 0377.

This code or a similar one is used on some systems to represent Japanese characters. The invalid sequences are those which consist of an odd number of consecutive bytes in the range from 0240 through 0377.

Here is another multibyte code which can handle more distinct extended characters—in fact, almost thirty million:

The basic sequences consist of

• single bytes with values in the range 0 through 0177.
• sequences of up to four bytes in which the first byte is in the range from 0200 through 0237, and the remaining bytes are in the range from 0240 through 0377.

In this code, any sequence that starts with a byte in the range from 0240 through 0377 is invalid.

And here is another variant which has the advantage that removing the last byte or bytes from a valid character can never produce another valid character. (This property is convenient when you want to search strings for particular characters.)

The basic sequences consist of

• single bytes with values in the range 0 through 0177.
• two-byte sequences in which the first byte is in the range from 0200 through 0207, and the second byte is in the range from 0240 through 0377.
• three-byte sequences in which the first byte is in the range from 0210 through 0217, and the other bytes are in the range from 0240 through 0377.
• four-byte sequences in which the first byte is in the range from 0220 through 0227, and the other bytes are in the range from 0240 through 0377.

The list of invalid sequences for this code is long and not worth stating in full; examples of invalid sequences include 0240 and 0220 0300 065.

The number of possible multibyte codes is astronomical. But a given computer system will support at most a few different codes. (One of these codes may allow for thousands of different characters.) Another computer system may support a completely different code. The library facilities described in this chapter are helpful because they package up the knowledge of the details of a particular computer system’s multibyte code, so your programs need not know them.

You can use special standard macros to find out the maximum possible number of bytes in a character in the currently selected multibyte code with MB_CUR_MAX, and the maximum for any multibyte code supported on your computer with MB_LEN_MAX.

Macro: int MB_LEN_MAX

This is the maximum length of a multibyte character for any supported locale. It is defined in ‘limits.h’.

Macro: int MB_CUR_MAX

This macro expands into a (possibly non-constant) positive integer expression that is the maximum number of bytes in a multibyte character in the current locale. The value is never greater than MB_LEN_MAX.

MB_CUR_MAX is defined in ‘stdlib.h’.

Normally, each basic sequence in a particular character code stands for one character, the same character regardless of context. Some multibyte character codes have a concept of shift state; certain codes, called shift sequences, change to a different shift state, and the meaning of some or all basic sequences varies according to the current shift state. In fact, the set of basic sequences might even be different depending on the current shift state. See section Multibyte Codes Using Shift Sequences, for more information on handling this sort of code.

What happens if you try to pass a string containing multibyte characters to a function that doesn’t know about them? Normally, such a function treats a string as a sequence of bytes, and interprets certain byte values specially; all other byte values are “ordinary”. As long as a multibyte character doesn’t contain any of the special byte values, the function should pass it through as if it were several ordinary characters.

For example, let’s figure out what happens if you use multibyte characters in a file name. The functions such as open and unlink that operate on file names treat the name as a sequence of byte values, with ‘/’ as the only special value. Any other byte values are copied, or compared, in sequence, and all byte values are treated alike. Thus, you may think of the file name as a sequence of bytes or as a string containing multibyte characters; the same behavior makes sense equally either way, provided no multibyte character contains a ‘/’.

1.4 Wide Character Introduction

Wide characters are much simpler than multibyte characters. They are simply characters with more than eight bits, so that they have room for more than 256 distinct codes. The wide character data type, wchar_t, has a range large enough to hold extended character codes as well as old-fashioned ASCII codes.

An advantage of wide characters is that each character is a single data object, just like ordinary ASCII characters. Wide characters also have some disadvantages:

• A program must be modified and recompiled in order to use wide characters at all.
• Files of wide characters cannot be read by programs that expect ordinary characters.

Wide character values 0 through 0177 are always identical in meaning to the ASCII character codes. The wide character value zero is often used to terminate a string of wide characters, just as a single byte with value zero often terminates a string of ordinary characters.

Data Type: wchar_t

This is the “wide character” type, an integer type whose range is large enough to represent all distinct values in any extended character set in the supported locales. @xref{Locales}, for more information about locales. This type is defined in the header file ‘stddef.h’.

If your system supports extended characters, then each extended character has both a wide character code and a corresponding multibyte basic sequence.

In this chapter, the term code is used to refer to a single extended character object to emphasize the distinction from the char data type.

1.5 Conversion of Extended Strings

The mbstowcs function converts a string of multibyte characters to a wide character array. The wcstombs function does the reverse. These functions are declared in the header file ‘stdlib.h’.

In most programs, these functions are the only ones you need for conversion between wide strings and multibyte character strings. But they have limitations. If your data is not null-terminated or is not all in core at once, you probably need to use the low-level conversion functions to convert one character at a time. See section Conversion of Extended Characters One by One.

Function: size_t mbstowcs (wchar_t *wstring, const char *string, size_t size)

The mbstowcs (“multibyte string to wide character string”) function converts the null-terminated string of multibyte characters string to an array of wide character codes, storing not more than size wide characters into the array beginning at wstring. The terminating null character counts towards the size, so if size is less than the actual number of wide characters resulting from string, no terminating null character is stored.

The conversion of characters from string begins in the initial shift state.

If an invalid multibyte character sequence is found, this function returns a value of -1. Otherwise, it returns the number of wide characters stored in the array wstring. This number does not include the terminating null character, which is present if the number is less than size.

Here is an example showing how to convert a string of multibyte characters, allocating enough space for the result.

wchar_t *
mbstowcs_alloc (const char *string)
{
  size_t size = strlen (string) + 1;
  wchar_t *buf = xmalloc (size * sizeof (wchar_t));

  size = mbstowcs (buf, string, size);
  if (size == (size_t) -1)
    return NULL;
  buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
  return buf;
}

Function: size_t wcstombs (char *string, const wchar_t wstring, size_t size)

The wcstombs (“wide character string to multibyte string”) function converts the null-terminated wide character array wstring into a string containing multibyte characters, storing not more than size bytes starting at string, followed by a terminating null character if there is room. The conversion of characters begins in the initial shift state.

The terminating null character counts towards the size, so if size is less than or equal to the number of bytes needed in wstring, no terminating null character is stored.

If a code that does not correspond to a valid multibyte character is found, this function returns a value of -1. Otherwise, the return value is the number of bytes stored in the array string. This number does not include the terminating null character, which is present if the number is less than size.

1.6 Multibyte Character Length

This section describes how to scan a string containing multibyte characters, one character at a time. The difficulty in doing this is to know how many bytes each character contains. Your program can use mblen to find this out.

Function: int mblen (const char *string, size_t size)

The mblen function with a non-null string argument returns the number of bytes that make up the multibyte character beginning at string, never examining more than size bytes. (The idea is to supply for size the number of bytes of data you have in hand.)

The return value of mblen distinguishes three possibilities: the first size bytes at string start with valid multibyte character, they start with an invalid byte sequence or just part of a character, or string points to an empty string (a null character).

For a valid multibyte character, mblen returns the number of bytes in that character (always at least 1, and never more than size). For an invalid byte sequence, mblen returns -1. For an empty string, it returns 0.

If the multibyte character code uses shift characters, then mblen maintains and updates a shift state as it scans. If you call mblen with a null pointer for string, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section Multibyte Codes Using Shift Sequences.

The function mblen is declared in ‘stdlib.h’.

1.7 Conversion of Extended Characters One by One

You can convert multibyte characters one at a time to wide characters with the mbtowc function. The wctomb function does the reverse. These functions are declared in ‘stdlib.h’.

Function: int mbtowc (wchar_t *result, const char *string, size_t size)

The mbtowc (“multibyte to wide character”) function when called with non-null string converts the first multibyte character beginning at string to its corresponding wide character code. It stores the result in *result.

mbtowc never examines more than size bytes. (The idea is to supply for size the number of bytes of data you have in hand.)

mbtowc with non-null string distinguishes three possibilities: the first size bytes at string start with valid multibyte character, they start with an invalid byte sequence or just part of a character, or string points to an empty string (a null character).

For a valid multibyte character, mbtowc converts it to a wide character and stores that in *result, and returns the number of bytes in that character (always at least 1, and never more than size).

For an invalid byte sequence, mbtowc returns -1. For an empty string, it returns 0, also storing 0 in *result.

If the multibyte character code uses shift characters, then mbtowc maintains and updates a shift state as it scans. If you call mbtowc with a null pointer for string, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section Multibyte Codes Using Shift Sequences.

Function: int wctomb (char *string, wchar_t wchar)

The wctomb (“wide character to multibyte”) function converts the wide character code wchar to its corresponding multibyte character sequence, and stores the result in bytes starting at string. At most MB_CUR_MAX characters are stored.

wctomb with non-null string distinguishes three possibilities for wchar: a valid wide character code (one that can be translated to a multibyte character), an invalid code, and 0.

Given a valid code, wctomb converts it to a multibyte character, storing the bytes starting at string. Then it returns the number of bytes in that character (always at least 1, and never more than MB_CUR_MAX).

If wchar is an invalid wide character code, wctomb returns -1. If wchar is 0, it returns 0, also storing 0 in *string.

If the multibyte character code uses shift characters, then wctomb maintains and updates a shift state as it scans. If you call wctomb with a null pointer for string, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section Multibyte Codes Using Shift Sequences.

Calling this function with a wchar argument of zero when string is not null has the side-effect of reinitializing the stored shift state as well as storing the multibyte character 0 and returning 0.

1.8 Character-by-Character Conversion Example

Here is an example that reads multibyte character text from descriptor input and writes the corresponding wide characters to descriptor output. We need to convert characters one by one for this example because mbstowcs is unable to continue past a null character, and cannot cope with an apparently invalid partial character by reading more input.

int
file_mbstowcs (int input, int output)
{
  char buffer[BUFSIZ + MB_LEN_MAX];
  int filled = 0;
  int eof = 0;

  while (!eof)
    {
      int nread;
      int nwrite;
      char *inp = buffer;
      wchar_t outbuf[BUFSIZ];
      wchar_t *outp = outbuf;

      /* Fill up the buffer from the input file.  */
      nread = read (input, buffer + filled, BUFSIZ);
      if (nread < 0)
        {
          perror ("read");
          return 0;
        }
      /* If we reach end of file, make a note to read no more. */
      if (nread == 0)
        eof = 1;

      /* filled is now the number of bytes in buffer. */
      filled += nread;

      /* Convert those bytes to wide characters--as many as we can. */
      while (1)
        {
          int thislen = mbtowc (outp, inp, filled);
          /* Stop converting at invalid character;
             this can mean we have read just the first part
             of a valid character.  */
          if (thislen == -1)
            break;
          /* Treat null character like any other,
             but also reset shift state. */
          if (thislen == 0) {
            thislen = 1;
            mbtowc (NULL, NULL, 0);
          }
          /* Advance past this character. */
          inp += thislen;
          filled -= thislen;
          outp++;
        }

      /* Write the wide characters we just made.  */
      nwrite = write (output, outbuf,
                      (outp - outbuf) * sizeof (wchar_t));
      if (nwrite < 0)
        {
          perror ("write");
          return 0;
        }

      /* See if we have a real invalid character. */
      if ((eof && filled > 0) || filled >= MB_CUR_MAX)
        {
          error ("invalid multibyte character");
          return 0;
        }

      /* If any characters must be carried forward,
         put them at the beginning of buffer. */
      if (filled > 0)
        memcpy (inp, buffer, filled);
      }
    }

  return 1;
}

1.9 Multibyte Codes Using Shift Sequences

In some multibyte character codes, the meaning of any particular byte sequence is not fixed; it depends on what other sequences have come earlier in the same string. Typically there are just a few sequences that can change the meaning of other sequences; these few are called shift sequences and we say that they set the shift state for other sequences that follow.

To illustrate shift state and shift sequences, suppose we decide that the sequence 0200 (just one byte) enters Japanese mode, in which pairs of bytes in the range from 0240 to 0377 are single characters, while 0201 enters Latin-1 mode, in which single bytes in the range from 0240 to 0377 are characters, and interpreted according to the ISO Latin-1 character set. This is a multibyte code which has two alternative shift states (“Japanese mode” and “Latin-1 mode”), and two shift sequences that specify particular shift states.

When the multibyte character code in use has shift states, then mblen, mbtowc and wctomb must maintain and update the current shift state as they scan the string. To make this work properly, you must follow these rules:

• Before starting to scan a string, call the function with a null pointer for the multibyte character address—for example, mblen (NULL, 0). This initializes the shift state to its standard initial value.
• Scan the string one character at a time, in order. Do not “back up” and rescan characters already scanned, and do not intersperse the processing of different strings.

Here is an example of using mblen following these rules:

void
scan_string (char *s)
{
  int length = strlen (s);

  /* Initialize shift state. */
  mblen (NULL, 0);

  while (1)
    {
      int thischar = mblen (s, length);
      /* Deal with end of string and invalid characters. */
      if (thischar == 0)
        break;
      if (thischar == -1)
        {
          error ("invalid multibyte character");
          break;
        }
      /* Advance past this character. */
      s += thischar;
      length -= thischar;
    }
}

The functions mblen, mbtowc and wctomb are not reentrant when using a multibyte code that uses a shift state. However, no other library functions call these functions, so you don’t have to worry that the shift state will be changed mysteriously.

About This Document

This document was generated on March 29, 2022 using texi2html 5.0.

The buttons in the navigation panels have the following meaning:

where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:

• 1. Section One
  • 1.1 Subsection One-One
    • ...
  • 1.2 Subsection One-Two
    • 1.2.1 Subsubsection One-Two-One
    • 1.2.2 Subsubsection One-Two-Two
    • 1.2.3 Subsubsection One-Two-Three     <== Current Position
    • 1.2.4 Subsubsection One-Two-Four
  • 1.3 Subsection One-Three
    • ...
  • 1.4 Subsection One-Four

This document was generated on March 29, 2022 using texi2html 5.0.