home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Columbia Kermit
/
kermit.zip
/
pub
/
public_html
/
accents.save
< prev
next >
Wrap
Text File
|
2020-01-01
|
69KB
|
1,801 lines
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html><head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Interchange of Non-English Computer Text</title>
</head><body bgcolor="#ffffff" text="#000000">
<h2 align="center">Interchange of Non-English Computer Text</h2>
<center>
Frank da Cruz<br>
Columbia University, New York<br>
1994
</center>
<p>
<blockquote>
<i>
This paper was written in 1994 to explain the ideas behind the character-set
conversion features of the Kermit file transfer protocol, which were developed
in the late 1980s and early 1990s to address the problems of meaningful
interchange of text within the installed base of diverse computers using a
multitude of character sets. The problems are less acute now that the
diversity of the 1970s and 80s is all but wiped out and Unicode has become the
standard common universal character set. This document is encoded in Unicode
UTF-8 (properly announced) and requires a font that includes certain
characters that were introduced in Unicode 3.1 and a browser that
(unlike Internet Explorer) is friendly to monospace text. Mozilla and Netscape
work best.
</i>
<table width="100%">
<tr>
<td align="right">—<i>fdc, June 2003</i></td>
</table>
</blockquote>
<hr>
<h3>INTRODUCTION</h3>
Thirty years ago, computers and people communicated using a small
repertoire of symbols, digits, and Roman letters. Often the letters were
only in uppercase, and there were no accents. The language of computing
was exclusively English. Today, we are poised on the brink of a worldwide
computer-based communications revolution, with a single character set
encompassing all the world's writing systems.
<p>
In the intervening decades, we have contrived a vast Babel of mutually
incomprehensible character sets, both proprietary and standard. The new
Universal Character Set, ISO 10646 <a href="#refs">[19]</a>, offers a single
common encoding for all writing systems. But radical and massive changes are
required in data entry and display hardware as well as in computer software
and data files at all levels from operating system to application, and could
therefore take decades to see widespread use. In the meantime, how shall we
survive our Babel?
<p>
The problem is immediately apparent to anyone who tries to transfer
non-English textual data between two different kinds of computers using
conventional methods. "Pâté" on an IBM PC becomes "PEta" on the
Macintosh, and a truckload of pocket-bread is delivered instead of goose
liver.
<p>
This paper presents a simplified and condensed description of the
character-set translation method developed for the Kermit file transfer
protocol. The lessons learned should be useful in any arena where text must
be transmitted meaningfully between unlike computer systems. Familiarity with
the standards process in general, and the US ASCII <a href="#refs">[1]</a>,
ISO 646 <a href="#refs">[14]</a>, and ISO 8859 <a href="#refs">[18]</a>
character set standards is assumed, and with ISO Standards 2022 <a
href="#refs">[15]</a> and 4873 <a href="#refs">[17]</a>, as well as with
proprietary character sets such as IBM PC code pages.
<hr>
<h3>TYPES OF CHARACTER SETS</h3>
Today's coded character sets can be classified along several axes: standard
versus proprietary, 7-bit versus 8-bit, single-byte versus multibyte, and so
on. This paper treats only the character sets used in application-independent
plain-text files, and not the application-specific text representations used in
word processing, publishing, and similar environments that are concerned with
rendering forms such as fonts, style, point size, ligatures, and so forth, and
which sometimes offer character repertoires different from any plain-text
character set.
<p>
<h4>Standard Character Sets</h4>
<p>
Let us define a standard character set as one that is registered in the ISO
<cite>Register of Coded Characters to Be Used with Escape Sequences</cite>
<a href="#refs">[21]</a> under the provisions of ISO Standard 2375 <a
href="#refs">[16]</a>. The Register, which is maintained by the European
Computer Manufacturers Association (ECMA), includes listings of all
ISO-registered character sets and assigns unique registration numbers and
designating escape sequences to each one.
<p>
Standard character sets are subdivided into two major types: graphic and
control. Thus ASCII, which is the USA version of ISO 646, and which most
people think of as a single character set, is really two sets: the ISO 646
32-character control set (ISO registration number 001) and a 94-character
graphics set (ISO registration 006). And, as specified by ISO Standard 4873,
the characters Space and Delete are not part of ASCII <i>per se</i>, but rather
separate components that must always be available in the presence of a
94-character graphics set.
<p>
Similarly, ISO 8859-1 Latin Alphabet 1, which most of us think of as a
coherent 8-bit character set is, in truth, composed of the ISO 646 control set
(registration 001), the ISO 646 USA graphics set (006), the characters Space
and Delete, a second 32-character control set (normally, but not necessarily,
ISO 6429 <a href="#refs">[20]</a>, registration 077), and a 96-character set
known as "The Right-hand Part of Latin Alphabet 1" (registration 100). Each
of these pieces except Space and Delete has its own unique registration number
and designating escape sequence. There is no single, unique identifier for
the 8-bit Latin-1 character set in its entirety.
<p>
<hr>
<blockquote>
<pre>
<ΓöÇΓöÇC0ΓöÇΓöÇ> <ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇGLΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ><ΓöÇΓöÇC1ΓöÇΓöÇ> <ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇGRΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ>
00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15
ΓöîΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓö¼ΓöÇΓöÇΓöÇΓöÉ
00 Γöé ΓöéSP Γöé Γöé Γöé Γöé Γöé
01 Γöé Γö£ΓöÇΓöÇΓöÇΓöÿ Γöé Γö£ΓöÇΓöÇΓöÇΓöÿ Γöé
02 Γöé Γöé Γöé Γöé Γöé
03 Γöé C Γöé ASCII Γöé C Γöé Special Γöé
04 Γöé o Γöé graphics Γöé o Γöé Graphics Γöé
05 Γöé n Γöé Γöé n Γöé Γöé
06 Γöé t Γöé Γöé t Γöé Γöé
07 Γöé r Γöé Γöé r Γöé Γöé
08 Γöé o Γöé Γöé o Γöé Γöé
09 Γöé l Γöé Γöé l Γöé Γöé
10 Γöé s Γöé Γöé s Γöé Γöé
11 Γöé Γöé Γöé Γöé Γöé
12 Γöé Γöé Γöé Γöé Γöé
13 Γöé Γöé Γöé Γöé Γöé
14 Γöé Γöé ΓöîΓöÇΓöÇΓöÇΓöñ Γöé ΓöîΓöÇΓöÇΓöÇΓöñ
15 Γöé Γöé ΓöéDELΓöé Γöé Γöé Γöé
ΓööΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓö┤ΓöÇΓöÇΓöÇΓöÿ
<ΓöÇΓöÇC0ΓöÇΓöÇ> <ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇGLΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ><ΓöÇΓöÇC1ΓöÇΓöÇ> <ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇGRΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ>
</pre>
<p>
<b>Figure 1:</b> Structure of a Standard 8-Bit Single-Byte Character Set
</blockquote>
<hr>
<p>
In practice, most standard character sets have the same structure, which is
illustrated in Figure 1:
<p>
<ol>
<li>The 32-character control set of ISO 646 in columns 0 and 1. This is
called the C0 region.
<p>
<li>The character Space at position 2/0.
<p>
<li>A 94-character graphics set in positions 2/1 through 7/14. This is called
the Graphics Left, or GL, region. For 7-bit character sets, these are the
84 characters of the ISO 646 International Reference Version plus 10
country-specific characters. For 8-bit sets, these are the 94 graphic
characters of US ASCII.
<p>
<li>The character Delete at position 7/15.
<p>
<li>For 8-bit sets, a 32-character "right half" control set in columns 8 and
9. This is the C1 region.
<p>
<li>For 8-bit sets, a 94- or 96-character "right half" graphics set in columns
10 through 15. This is the Graphics Right or GR region.
<p>
<li>In multibyte graphic sets (not illustrated in the figure), each character
is composed of a fixed number of bytes, generally two, with each byte in
the graphic range, i.e. not corresponding to a C0 or C1 control character,
and following either the 94- or 96-character structure. Thus, single-byte
control characters can be mixed with double-byte graphic characters with
no ambiguity.
</ol>
<p>
The rationale for reserving an area for control characters in the right half
of an 8-bit set is that communications devices tend to examine only the
low-order 7 bits of a character when deciding whether it is a control or
graphic character. Placing graphic characters in columns 8 and 9 often
triggers unwanted control functions.
<p>
During the early decades of computing, when 7-bit communication was the
rule, ISO 646 was the predominant method for representing the special
characters of each language, and each European country had its own version of
ISO 646. But this imposed severe limitations on the user. For example, ISO
646 makes it impossible to mix (say) German text and C-language programming
syntax in the same file. The following C program fragment:
<p>
<blockquote>
<pre>
if (~(a[i] | x)) {
printf("Grüße aus Köln\n");
}
</pre>
</blockquote
<p>
can not be encoded in either ASCII or in German ISO 646, because neither set
has all the required characters. If German ISO 646 is used, German special
characters are substituted for the braces, brackets, and bars:
<p>
<blockquote>
<pre>
if (ß(aÄiÜ ö x)) ä
printf("Grüße aus KölnÖn");
├╝
</pre>
</blockquote
<p>
whereas if ASCII is chosen, we see the opposite effect:
<p>
<blockquote>
<pre>
if (~(a[i] | x)) {
printf("Gr}~e aus K|ln\n");
}
</pre>
</blockquote
<p>
Neither result is satisfactory, and matters only deteriorate when
German-language program commentary is to be added.
<p>
To alleviate these problems, many sites are switching to the ISO 8859 Latin
alphabets. However, the ISO 646 versions are still widely used, especially in
electronic mail, a predominantly 7-bit medium.
<h4>Proprietary Character Sets</h4>
Most computers use either US ASCII <a href="#refs">[1]</a> or IBM EBCDIC <a
href="#refs">[13]</a> as their basic character set. But to support entry,
display, printing, and processing of textual data in languages other than
English, computer manufacturers soon recognized the need to extend these basic
sets to allow representation of the accented Roman letters, non-Roman letters,
ideograms, and other symbols used by the world's writing systems.
<p>
Some manufacturers provide ISO 646 national versions, but users suffer with
their limitations. Besides the sacrifice of characters needed for programming,
ISO 646 does not allow mixture of text in different languages, such as Italian,
French, and Norwegian, in the same file. So manufacturers such as Digital and
IBM began to devise 8-bit international character sets for the European
languages, as well as other sets for languages using other writing systems.
Most of these 8-bit sets are capable of representing text in several languages,
allowing a single product to serve, and work compatibly, over a broader market,
for example all of Western Europe.
<p>
Prominent among the proprietary sets are IBM's PC code pages <a
href="#refs">[13]</a>. They resemble an ISO Latin Alphabet by having ASCII in
the left half, but depart from from the standard structure by using all of
columns 8 through 15, including the C1 area, for graphic characters. Thus IBM
PC code pages have (at least) 32 more graphic characters than a standard 8-bit
character set. Major manufacturers including Apple and NeXT follow the IBM
design, but with different character repertoires and encoding. Others,
notably Digital, Hewlett Packard, and Data General, observe the standard
structure, usually with different repertoires and encoding.
<p>
To add to the confusion, we also have IBM's many EBCDIC-based code pages,
whose structure does not follow any national or international standard, as
well as variations on IBM-like mainframes manufactured in Eastern Europe and
the Far East, plus unknown numbers of proprietary sets from other
manufacturers.
<h4>The Current Situation</h4>
Today we are confronted with hundreds of different coded character sets, both
standard and proprietary. These sets differ in important ways:
<p>
<ul>
<li><i>Size:</i> The total code space, 7-bit or 8-bit, single byte or
multibyte.
<li><i>Structure:</i> Standard or nonstandard allocation of control and
graphics areas.
<li><i>Repertoire:</i> The particular selection of characters.
<li><i>Encoding:</i> The particular code values assigned to each character.
</ul>
<p>
Every application-independent plain-text file is encoded in a particular
character set. It is generally not possible to mix character sets within a
plain-text file. Furthermore, a text file generally does not contain any
indication of its character set. Neither, in general, does the host operating
system identify a file's character set, nor indeed, provide any mechanism to do
so.
<p>
Within most applications and computing environments, a certain character
set is simply <i>assumed</i>. When a workstation supports multiple character
sets, or when data must be communicated between unlike computers, there is no
automatic mechanism for software applications to identify a file's character
set, and hence no way to automatically display its characters correctly, nor
to announce the character set to another computer or application during data
transfer.
<p>
This problem has grown over the past decade as computers have become
increasingly interconnected, and so are used increasingly for communication of
text: news, conferencing, file transfer and sharing, and electronic mail. Not
only are text character sets likely to be incompatible, but there are no
universally accepted methods for translation.
<p>
<h4>Character Set Translation</h4>
<p>
Characters, such as the letter A, are represented in the computer, and in
telecommunications, by numeric codes. Different computers use different codes
for the same character. For example, the letter A is code 65 in ASCII, code
193 in EBCDIC, and code 9025 in JIS X 0208.
<p>
The most commonly used translation function is a simple array-indexing
operation. Suppose we are translating from character set <i>A</i> to
character set <i>B</i>, and each set has 256 characters, and the characters in
each set are represented by 8-bit code values in the range 0..255. The
translation is accomplished by a linear array of 256 8-bit elements called a
translation table. The table element at position <i>i</i> contains the
translation from the character in set <i>A</i> whose code is <i>i</i> to the
corresponding character in set <i>B</i>, namely its code in set <i>B</i>. For
example, the 65th element of an ASCII-to-EBCDIC translation table would be the
number 193.
<p>
During the translation process, a particular input character, <i>c_in</i>,
in set <i>A</i> is translated to the output character, <i>c_out</i>, in set
<i>B</i> by an array indexing operation as in this example, written in the
C programming language:
<p>
<p>
<blockquote>
<pre>
unsigned char a_to_b[256] = { ... }; /* Translation table */
unsigned char c_in, c_out; /* Input and output characters */
c_out = a_to_b[c_in]; /* Translation function */
</pre>
</blockquote
<p>
where the <i>c_in</i> variable is used as a subscript to the <i>a_to_b</i>
array, and the notation <tt>{ ... }</tt> stands for the
initialization of the translation table. In practice, the braces contain the
quantities forming the table, in the appropriate order.
<p>
In constructing a translation function between any pair of character sets,
there are three important and often conflicting goals:
<p>
<ol>
<li><i>Invertibility</i> (I): After translating text from <i>A</i> to
<i>B</i>, and then back to <i>A</i>, is the result identical to the original?
<p>
<li><i>Readability</i> (R): After translating text from <i>A</i> to <i>B</i>,
is the result readable?
<p>
<li><i>Consistency</i> (C): Are translations from <i>A</i> to <i>B</i> by
different applications the same?
</ol>
<p>
In attempting to achieve these goals, we must look at the size, structure, and
repertoire of the two character sets. In many cases, the R-versus-I decision
is forced upon us, but in others a choice is possible. For example, consider
translating between two character sets of the same size: Latin-1 and
Latin/Cyrillic. An invertible translation is possible that will not be
readable, and a readable translation is possible that is not invertible. In
cases like this, the best course is to let the user set the translation goal.
<h4>Invertibility</h4>
Invertibility is important in cases where the exact contents of the file is
important, or when the goal of data transfer is not necessarily final usage.
Suppose, for example, you compose a C-language program (in ASCII) on your PC,
transfer it to an IBM mainframe (where it is converted to EBCDIC), work on it
some more, and then transfer it back to your PC. If the character-set
translation from the PC to the mainframe and back is not invertible, you will
likely not be able to compile the program again on your PC without syntax
errors.
<p>
An invertible translation from character set <i>A</i> to character set
<i>B</i> is possible only if <i>A</i> is smaller than or the same size as
<i>B</i>. Similarly, an invertible translation from character set <i>B</i> to
character set <i>A</i> is possible only if <i>B</i> is smaller than or the
same size as <i>A</i>. So it follows that invertibility can be achieved in
both directions only if the two character sets are the same size. The
following discussion applies only to bidirectional invertibility.
<p>
The intersection of the two character sets <i>A</i> and <i>B</i>, written
<i>A</i> ∩ <i>B</i>, is the set of characters, <i>c</i>,
that both sets have in common, that is, all
the characters are members of (Γêê) both <i>A</i> and <i>B</i>:
<p>
<blockquote>
<i>A</i> ∩ <i>B</i> = { <i>c</i> : <i>c</i> ∈ <i>A</i>, <i>c</i> ∈ <i>B</i> }
</blockquote>
<p>
The characters in <i>A</i> that are not in <i>B</i> can be written as:
<p>
<blockquote>
<i>A</i> \ <i>B</i> = { <i>c</i> : <i>c</i> Γêê <i>A</i>, <i>c</i> Γêë <i>B</i> }
</blockquote>
<p>
and the characters in <i>B</i> that are not in <i>A</i> are:
<p>
<blockquote>
<i>B</i> \ <i>A</i> = { <i>c</i> : <i>c</i> Γêê <i>B</i>, <i>c</i> Γêë <i>A</i> }
</blockquote>
<p>
To make an invertible translation table, the characters of
<i>A</i> ∩ <i>B</i>
are paired together: the letter "E" in one set to "E" in the other, "È"
in one set to "È" in the other, and so on. The characters in
<i>A</i> \ <i>B</i> are
paired 1-to-1 with the characters in
<i>B</i> \ <i>A</i> according to <i>some criterion</i>:
readability, consistency, whimsy, or caprice. The exact method for pairing
the leftovers is problematic, and frequently a particular pair makes no sense
at all, for example "L-with-stroke" with "Vulgar fraction 3/4".
<p>
Any 1-to-1 pairing will give an invertible translation, but to achieve the
most useful translation it is necessary to examine all the character sets
involved. To illustrate, Latin Alphabet 1 lacks the OE digraph character but
this character is found in the Digital Multinational character set, the Apple
Quickdraw set, the Hewlett Packard Roman8 set, the Data General International
set, and the NeXT character set, but at different code points in each.
Ideally, the translations for each of these character sets would map OE
digraph into the same Latin-1 code point, so that text translated from (say)
NeXT to Latin-1 and thence to Data General would keep its OE intact. But this
would require an unprecedented degree of cooperation among competing
manufacturers.
<p>
The construction of invertible translations between private and standard
character sets is beyond the scope of the national and international standards
organizations, nor should these translations be made arbitrarily by
programmers. Translation tables (or algorithms) are most appropriately
furnished by the creators or owners of each private character set. This lends
the appropriate "official" air and allows all software developers to use the
same translations, thus promoting interoperability of diverse applications.
In 1990, IBM became one of the few computer manufacturers to take this step
when it published invertible tables between ISO 8859-1 and its code pages 500
and 850 in its Character Data Representation Architecture Registry <a
href="#refs">[13]</a>.
<p>
Other translations, however, are lacking from IBM, for example between its
Cyrillic code pages and the ISO Latin/Cyrillic alphabet; similarly for Hebrew,
Arabic, Greek, and so on. Official invertible translations of any kind seem
to be entirely lacking from most other computer and software makers.
<h4>A Simple Rule</h4>
In the absence of an official translation, a simple procedure can be used to
produce consistent invertible translations across all applications. Let us
assume that <i>P</i> is a private nonstandard character set, and that <i>S</i>
is a standard character set, and that <i>P</i> and <i>S</i> are the same size,
<i>n</i>. Follow these steps to construct the translation table from <i>P</i>
to <i>S</i> as an array, <i>p_to_s</i>, of <i>n</i> elements:
<p>
<ol>
<li>The characters that are common to <i>P</i> and <i>S</i>,
<i>P</i> ∩ <i>S</i>, are mapped together. For each such
character in <i>P</i>, whose code value is <i>i</i>,
<i>p_to_s</i><tt>[</tt><i>i</i><tt>]</tt> takes the corresponding character's
code value from <i>S</i>.
<p>
<li>The members of <i>A</i> \ <i>B</i> and <i>B</i> \ <i>A</i> are paired with
each other in code order.
</ol>
<p>
This procedure guarantees that <i>p_to_s</i> has exactly <i>n</i> unique
elements.
<p>
Step (1) is not always easy. All too frequently, private character sets
are documented only by tables showing the graphic characters, often unclearly
(as when working from a fax of third-generation photocopy), with no names or
other identifiers assigned to the characters. Even when the material is
legible and names are assigned, conventions for graphic representation differ,
and so do the names. Thus, some knowledge of languages, writing systems,
world and corporate cultures, history, and politics is helpful.
<p>
Let's say that character set <i>P</i> consists of four characters, the
letters A, B, C, and D, whose code values are 0, 1, 2, and 3, respectively.
And set <i>S</i> consists of the letters B, X, A, and Y, also with code values
0, 1, 2, and 3, in that order. The letters <i>A</i> and <i>B</i> are common
to both sets. <i>A</i> is represented by code 0 in <i>P</i> and by code 2 in
<i>S</i>, so:
<p>
<blockquote>
<pre>
p_to_s[0] = 2
</pre>
</blockquote
<p>
Similarly for the letter B:
<p>
<blockquote>
<pre>
p_to_s[1] = 0
</pre>
</blockquote
<p>
Positions 2 and 3 of our translation array remain empty, so we assign them in
code order:
<p>
<blockquote>
<pre>
p_to_s[2] = 1
p_to_s[3] = 3
</pre>
</blockquote
<p>
and the translation from <i>P</i> to <i>S</i> is complete. Each element of
the <i>p_to_s</i> array has a unique value.
<p>
To create the reverse translation table from <i>S</i> to <i>P</i>,
<i>s_to_p</i>, we could repeat the process in the reverse direction, or,
equivalently (and more safely), simply turn the <i>p_to_s</i> table "inside
out" by sorting it according to its values. Here is a C language program
fragment that does the job:
<p>
<p>
<blockquote>
<pre>
for (i = 0; i < n; i++)
s_to_p[p_to_s[i]] = i;
</pre>
</blockquote
<p>
This leaves us with the two translation tables:
<p>
<blockquote>
<table border>
<tr>
<th>Index
<th>p_to_s
<th>s_to_p
<tr>
<td align="center">0
<td align="center">2
<td align="center">1
<tr>
<td align="center">1
<td align="center">0
<td align="center">2
<tr>
<td align="center">2
<td align="center">1
<td align="center">0
<tr>
<td align="center">3
<td align="center">3
<td align="center">3
</table>
</blockquote>
<p>
Naturally, such an arbitrary method will please few (hence the foregoing
plea for more-sensible <i>official</i> invertible translations); in this case C
becomes X and vice versa. But <i>vice-versa</i> is exactly what is needed for
invertibility and consistency. Those characters the two sets have in common
are translated readably, and the rest are translated according to the Simple
Rule for <i>consistent</i> invertibility.
<p>
In a more useful application of the Simple Rule, let us construct an
invertible mapping between two real-life, 8-bit, single-byte character sets,
Data General International (DGI) <a href="#refs">[7]</a> and ISO 8859-1 <a
href="#refs">[18]</a>, between which there is no official invertible mapping.
We begin by finding all the characters from DGI that are also in Latin-1 (80
of them) and make the appropriate mappings. We are left with two lists of
sixteen unmatched characters. Applying the Simple Rule, the lists are sorted
in code order and placed side-by-side to obtain the following correspondence:
<p>
<blockquote>
<table border>
<tr>
<td>160
<td>Undefined
<td>160
<td>No-break space
<tr>
<td>175
<td>Double dagger
<td>166
<td>Broken bar
<tr>
<td>179
<td>Trade mark uncircled
<td>173
<td>Soft hyphen
<tr>
<td>180
<td>Florin sign
<td>175
<td>Macron
<tr>
<td>183
<td>Less-than-or-equal sign
<td>184
<td>Cedilla
<tr>
<td>184
<td>Greater-than-or-equal sign
<td>185
<td>Superscript one
<tr>
<td>186
<td>Grave accent
<td>188
<td>Vulgar fraction one quarter
<tr>
<td>191
<td>Up arrow
<td>190
<td>Vulgar fraction three quarters
<tr>
<td>215
<td>Capital OE digraph
<td>208
<td>Capital Icelandic letter Eth
<tr>
<td>220
<td>Undefined
<td>215
<td>Multiplication sign
<tr>
<td>221
<td>Uppercase letter Y diaeresis
<td>221
<td>Capital letter Y with acute accent
<tr>
<td>222
<td>Undefined
<td>222
<td>Capital Icelandic letter Thorn
<tr>
<td>223
<td>Undefined
<td>240
<td>Small Icelandic letter eth
<tr>
<td>247
<td>Small oe digraph
<td>247
<td>Division sign
<tr>
<td>254
<td>Undefined
<td>253
<td>Small letter y with acute accent
<tr>
<td>255
<td>Fill character light
<td>254
<td>Small Icelandic letter thorn
</table>
</blockquote>
<p>
So if <i>P</i> is DGI and <i>S</i> is Latin-1, then
<i>p_to_s</i><tt>[160]</tt> = 160,
<i>p_to_s</i><tt>[175]</tt> = 166, …,
<i>p_to_s</i><tt>[255]</tt> = 254, and the <i>P</i> to <i>S</i>
mapping is complete. The <i>s_to_p</i> array is obtained by exchanging the
index and value of each <i>p_to_s</i> element, as in the program fragment
given above.
<h4>Readability</h4>
Bidirectional invertibility cannot be achieved when the character sets are
different sizes, nor can invertibility be achieved from a larger set to a
smaller set. In such cases, readability becomes the only sensible translation
goal. Even in cases where invertibility is possible, readability might be
preferred for a particular data transfer.
<p>
When translating from a larger set, <i>A</i>, to a smaller one, <i>B</i>,
several different characters in <i>A</i> can be mapped to a single character
in B. For example, the following Latin-1 characters:
<p>
<blockquote>
à á â ã ä å æ
</blockquote>
<p>
might all be mapped to the letter "a" when translating to ASCII. In the
resulting ASCII file, we can't tell where a particular "a" came from, so we
can't reconstruct the original Latin-1 text when translating in the reverse
direction. But the ASCII file is more intelligible than if we had used some
other mapping, such as simply stripping off the high-order bit. Translation
by removing diacritics is useful with Roman-based languages, such as French;
"pâté" becomes "pate" rather than (say) "pbti". Or German:
"Grüße aus Köln" becomes "Gruse aus Koln" instead of
"Gr|_e aus Kvln".
<p>
In German, the words "Gruse" and "Grusse" have entirely different meanings
(we don't want to say "soot" when we mean "greetings"). We can do better.
European languages like German, Swedish, Norwegian, Danish, Icelandic, and
Dutch have rules for converting accented or other special characters into
unadorned ABC's. For example, any German vowel with an umlaut (diaeresis) can
be written without the umlaut and followed by the letter "e". These rules are
specific to each language. So while we can write the German word
"K├╢ln" as "Koeln", we cannot write the English word
"co├╢peration" as "cooeperation".
<p>
Such language rules can not be applied blindly in reverse. For example, if
"oe" were translated back to "├╢", then "Kommandoebene" would become
"Kommand├╢bene" (not a German word), and <tt>AUTOEXEC.BAT</tt> would
become <tt>AUTÖXEC.BAT</tt> (a PC file that you don't want to rename!).
<p>
Construction of a readable translation between two entirely different
alphabets, such as Cyrillic and Roman, is called transliteration. The
specific transliteration rules must take into account not only the alphabets
themselves, but also what languages they represent. For example, the surname
of a former leader of the former USSR,
КРУЩЕВ,
is transliterated into Roman letters as "Khrushchev" in English, but into
"Khruschtschew" in German.
<p>
Newspapers and magazines, libraries, immigrant bureaus, and other
organizations have their own standard procedures for transliterating "foreign"
writing systems. Not just in "ASCII-speaking" lands, but everywhere: Russian
names are written in Arabic newspapers, Hebrew names in Greek journals,
English names on Chinese passports, Korean publications in Vietnamese library
catalogs. But these standards are not widely known. When a standard can be
found, use it. If not, look harder.
<hr>
<h3>CHARACTER-SET TRANSLATION IN THE KERMIT FILE TRANSFER PROTOCOL</h3>
The Kermit File Transfer Protocol was developed at Columbia University to
allow the transfer of both text and binary files among all types of personal
computers, minicomputers, and mainframes, in both the 7-bit and 8-bit
communication environments. Kermit is a layered, point-to-point,
transport-independent, error-correcting packet protocol described in detail
elsewhere <a href="#refs">[5]</a>.
<p>
Transfer of text files between unlike computers requires conversion of both
record format and character set at the presentation layer. For example, a
document composed under the UNIX operating system using the ASCII character
set with lines separated by imbedded Linefeed characters, upon transfer to an
IBM mainframe, is converted to EBCDIC encoding and a mainframe-specific
variable- or fixed-length record format. Kermit accomplishes this conversion
with another Simple Rule: during file transfer, the character set used for
text files is ASCII, and the record format is stream, with records (lines)
delimited by Carriage Return and Linefeed.
<p>
Thus, it is the responsibility of each Kermit program to convert between
the text character sets and record formats of its own computer and the
standard Kermit format. This means that no Kermit program needs to know the
specific codes and formats of any kind of computer except its own, and it
forms the basis of Kermit's strategy for converting between different
character sets. This idea is known as a "common intermediate representation,"
and it lies at the heart of any presentation-layer protocol <a
href="#refs">[26]</a>.
<p>
By the mid 1980s, Kermit had become a de facto standard for file transfer.
Kermit software programs had been written for almost every kind of computer in
existence. But the Kermit protocol lacked a formal and consistent means for
exchanging text that contained non-Roman or accented Roman characters. Files
could be transferred, but the results would be gibberish unless the receiving
computer supported the same character set as the sender.
<p>
At first, this problem was remedied by pre- or postprocessing. But this
approach places an unreasonable burden on the user. Not only must extra steps
be taken, but a suitable translation utility must be found for every pair of
character sets. More subtly, translation utilities (for example, between an
IBM code page and a Macintosh character set) are constructed in an ad-hoc
manner, with no guarantee of consistency from one utility to another.
<p>
The problem is compounded by the rapid proliferation of proprietary,
national, and international standard character sets. By the late 1980s, there
were many encodings for each major writing system, a problem pointed out by
attendees at international conference sessions on Kermit in Europe and Japan
<a href="#refs">[10]</a>. A consistent approach to character-set translation
had become an urgent matter.
<h4>Basic Design Principles</h4>
How can we enable meaningful exchange of text between any two computers? The
obvious approach is to require each data-transfer application to understand
every character set in existence. This works adequately when the number of
sets is small and stable, but quickly becomes unwieldy and unmanageable as the
number increases. If the number of character sets is <i>n</i>, the number of
translations is:
<p>
<blockquote id="eqn1">
<table width="100%">
<tr>
<td><pre>
ΓÄ¢ <i>n</i> ΓÄ₧ <i>n</i>!
ΓÄ£ Γă = ——————————— = <i>n</i> × (<i>n</i> - 1)
ΓÄ¥ 2 ΓÄá 2! × (<i>n</i>-2)!
</pre>
<td align="right">(1)</td>
</table>
</blockquote>
<p>
If we have two character sets, <i>A</i> and <i>B</i>, we need two
translations, one from A to <i>B</i> and one from <i>B</i> to A. If we have
three sets — <i>A</i>, <i>B</i>, and <i>C</i> — we need
3 × 2 = 6 translations: <i>AB</i>, <i>BA</i>,
<i>AC</i>, <i>CA</i>, <i>BC</i>, and <i>CB</i>. And so on.
<p>
Now consider that in 1990, IBM alone listed 276 different coded
character-set identifiers in its registry <a href="#refs">[13]</a>. If we
wanted translations between every pair of IBM character sets, there would be
75,900 of them! Add in all the other sets from all the other companies to
appreciate the magnitude of the problem.
<p>
By using a standard intermediate representation for each type of character
set (Roman, Cyrillic, Hebrew, Japanese, etc), we eliminate the need for any
particular computer to know about the character sets used by any other kind of
computer. Kermit's common intermediate character set, previously always
ASCII, is now allowed to be any of a small number of character sets. The set
used during a particular file transfer is called the transfer character set
(TCS).
<p>
The character set of the file that is being sent or received is called the
file character set (FCS). The sender translates from its local codes (the
FCS) to the standard ones (the TCS), and the receiver translates from TCS
codes to its own FCS, as shown in Figure 2.
<p>
<hr>
<blockquote>
<pre>
COMPUTER A COMPUTER B
ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ
Γöé ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ Γöé Γöé ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ Γöé
Γöé Γöé Translation Γöé Γöé Transfer Γöé Γöé Translation Γöé Γöé
Γöé Γöé Function: ΓöéΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ>Γöé Function: Γöé Γöé
Γöé Γöé FCS to TCS Γöé Γöé Character Set Γöé Γöé TCS to FCS Γöé Γöé
Γöé ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ Γöé Γöé ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ Γöé
Γöé ^ Γöé Γöé Γöé Γöé
Γöé Γöé Γöé Γöé v Γöé
Γöé Kermit Program Γöé Γöé Kermit Program Γöé
Γöé SEND Γöé Γöé RECEIVE Γöé
ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ
^ Γöé
Γöé v
ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ ΓöîΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÉ
Γöé Local File Γöé Γöé Local File Γöé
Γöé Character Set A Γöé Γöé Character Set B Γöé
ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ ΓööΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÿ
</pre>
<b>Figure 2:</b> File Transfer Character-Set Translation
</blockquote>
<hr>
<p>
For a particular file and transfer character set combination, a Kermit
program has one translation function for sending files and another for
receiving them. Theoretically, all combinations of file and transfer
character set are allowed. Thus the number of translation functions,
<i>f</i>, is given by:
<p>
<blockquote id="eqn2">
<table width="100%">
<tr>
<td align="left"><i>f</i> = <i>tcs</i> × <i>fcs</i> × 2
<td align="right">(2)
</table>
</blockquote>
<p>
That is, one function in each direction, for each combination of TCS and FCS.
While this number is significantly lower than the number of pairs of all
character sets (<a href="#eqn1">Equation 1</a>), we still want to reduce it to
conserve computer memory and cut down on user confusion.
<h4>The Transfer Character Set</h4>
<p>
Generally, we want to support all the file character sets used on a particular
computer, so the way to keep the total number of translation functions small
is to minimize the number of transfer character sets that can handle the given
selection of file character sets. This is done, in part, by restricting the
set of possible transfer character sets to a small number according to the
following rules:
<p>
<ol>
<li>The transfer character set must be a national or international standard
character set registered with the ISO, or a combination of such sets.
This means that its structure is consistent with other standard character
sets and that it has a unique identifier. Furthermore, it means that the
character set is well known, its specification is readily available, and
the characters have names.
<p>
<li>US ASCII <a href="#refs">[1]</a> is included for compatibility with the
original Kermit protocol and with unextended Kermit programs.
<p>
<li>The ISO 8859 Latin Alphabets <a href="#refs">[18]</a> are included.
<p>
<li>The ISO-registered Chinese <a href="#refs">[4]</a>, Japanese <a
href="#refs">[22, 23, 24]</a>, and Korean <a href="#refs">[25]</a> sets are
included. These are usually used in conjunction with one or more single-byte
sets that provide control characters and single-width ASCII or ISO 646
graphics.
<p>
<li>Additional sets, such as (for example) Vietnamese VSCII <a
href="#refs">[8, 28]</a>, can be included if they are registered with the ISO,
as long as they are not proper subsets of any of those already included.
<p>
<li>All else being equal, a simple and compact representation is preferred.
</ol>
<p>
The national versions of ISO 646 <a href="#refs">[14]</a> (other than US
ASCII) are not included because of Rule 5; these sets are covered adequately
by the ISO Latin alphabets. CCITT (ITU-T) T.61 <a href="#refs">[2]</a>, which
represents accented characters exclusively by composition, is not included for
reasons 1, 5, and 6.
<p>
Table 1 lists the transfer character sets presently allowed by the Kermit
protocol. The Kermit Name allows uniform reference to these sets by Kermit
software users. The requirement for ISO registration provides for unique and
incontestible identifiers for Kermit's transfer character sets. The Kermit
Designator is the means by which the sending Kermit program informs the
receiver of the transfer character set, a key part of the presentation
protocol. As noted earlier, however, ISO standards do not provide a single
designator for a complete character set, but rather separate designators for
its pieces. Thus Latin-1 is designated as "I6/100", meaning that the left
half (G0) is ASCII and the right half (G1) is "the Right-hand Part of Latin
Alphabet 1." The C0 and C1 control regions are not explicitly designated.
The C0 region is assumed to be the normal ASCII and ISO 646 control set, with
format effectors used to delimit records and so on. The C1 set is assumed to
be ISO 6429 to allow the use of character-set shifting functions such as SS2
and SS3 <a href="#refs">[15]</a>, for example in Japanese EUC.
<p>
In the Kermit Designator, the initial letter "I" indicates ISO registration
numbers for character sets, leaving open the possibility for other
registration authorities. Japanese EUC (Extended UNIX Code) is a special
case, having three parts, chosen in preference to JIS X 0208 alone to allow
the commonly used mixture of single-width and double-width characters. The
registration numbers are listed in G0/G1/G2 order, so SS2 is required to shift
between Kanji (ISO 87) and Katakana (ISO 13) in accordance with ISO 2022.
<p>
This notation is used in preference to, say, the name of the standard
itself (for example "ISO8859-1") because the same character set can be defined
by more than one standard (for example, Latin-1 and ECMA 94), and one standard
can specify more than one character set (e.g. ISO 646).
<p>
Using a standard international character set as the TCS, it is possible to
transfer text written in a language other than English between unlike
computers, and it is usually also possible to transfer text containing a
mixture of languages. For example, text in Latin Alphabet 1 might contain a
mixture of Italian, Norwegian, French, German, English, and Icelandic.
<p>
A particular Kermit program need not incorporate all the defined transfer
character sets. In many cases, a single 8-bit set will suffice, such as
LATIN1 for Western Europe, LATIN2 for Eastern European languages with
Roman-based writing systems, CYRILLIC for Russia, and so on. Thus <a
href="#eqn2">Equation 2</a> generally results in a comfortably small number.
For example, an IBM PC that supports five Roman-alphabet code pages for
Western European languages plus a Cyrillic code page can be used with two
transfer character sets, LATIN1 and CYRILLIC for a total of 24 translation
functions.
<p>
When a language is representable in more than one set from Table 1, as are
English, German, Finnish, Turkish, Greek, Russian, etc., the character set
highest on the list that adequately represents the language should be used.
For example, ASCII should be used for English. Within the ISO 8859 family,
lower-numbered sets that contain all the characters of interest are preferred
to higher-numbered sets containing the same characters.
<p>
This guideline maximizes the chance that any two particular Kermit programs
will interoperate. For example, LATIN1 would be chosen for French, German,
Italian, Spanish, Danish, Dutch, Swedish, etc; LATIN3 for Turkish;
JAPANESE-EUC for Japanese text that includes Kanji characters, KATAKANA for
Japanese text that includes only Roman and Katakana characters, and so on.
<p>
If a file containing a mixture of languages, say English, Finnish, and
Latvian, must be transferred, the user must find a transfer character set that
can adequately represent all three languages, in this case Latin Alphabet 4.
For a mixture of Norwegian and Turkish, Latin-5 must be used, and so on.
<p>
The user can employ this flexibility to achieve useful effects. For
example, since there is no requirement that a Cyrillic file be transferred
using a Cyrillic transfer character set, the user can transliterate between
Cyrillic and Roman characters as part of the file transfer process.
<h4>The Translation Function</h4>
<p>
A typical Kermit program contains an <i>m</i> × <i>n</i> × 2
matrix of translation functions, where <i>m</i> is the number of supported
file character sets, <i>n</i> is the number of supported transfer character
sets (including the <i>transparent</i> set, which indicates that no
translation is to be done), and there are two tables for each combination, one
for sending and one for receiving. The translation function is selected when
the user identifies the file and transfer character sets and then sends or
receives a file.
<p>
The normal behavior of a particular function can be altered in several
ways. The user can override its default translation goal, and when the goal
is readability (as it must be when translating from a larger to a smaller
set), language-specific rules can be invoked.
<p>
The function itself can work by any combination of algorithm, translation
table, exception list, and shameless tricks. For example, an invertible
translation between IBM Code Page 437 and Latin Alphabet 1 would be a simple
indexing operation into a table, but a translation from Japanese EUC to the PC
"Shift JIS" code is normally accomplished by a tableless algorithm.
Translation from Latin-1 to ASCII with German language rules could be done
with a combination of table accesses and exception lists. To accomplish the
desired translation, each Kermit program needs to know:
<p>
<ul>
<li>The local file character set.
<li>The transfer character set to be used.
<li>The translation goal, invertibility or readability.
<li>For readable translations, optionally, the language and a corresponding
set of language-specific rules.
</ul>
<p>
In most situations, some or all of these are implicit, and no particular
efforts are required. To illustrate, suppose that you have an IBM PC on your
desk, and the PC is connected to a Hewlett Packard (HP) timesharing computer.
The PC uses IBM code page 850 and the timesharing computer uses the HP Roman8
Set. In that case, your PC's file character set is always CP850, the
timesharing computer's file character set is always HP Roman8, and the
transfer character set is always Latin-1. These items can be set in your
Kermit profiles, and the appropriate translations will always occur
automatically.
<p>
On the other hand, suppose you must occasionally write some text in German
and send it from your PC to another computer that supports only ASCII. In
this case you would override your Kermit profiles by specifying a transfer
character set of ASCII, which automatically activates the readability goal,
and you might also choose to elect language-specific rules for German.
<p>
<blockquote>
<b>Table 1:</b> Kermit Transfer Character Sets
<table border>
<tr>
<th>Kermit Name</th>
<th>Standard</th>
<th>ISO Registration Number</th>
<th>Kermit Designator</th>
<th>Languages</th>
<tr>
<td>ASCII
<td>ANSI X3.4
<td align="center">6
<td align="center">(none)
<td>English, Latin
<tr>
<td>LATIN1
<td>ISO 8859-1
<td align="center">100
<td align="center">I6/100
<td>Danish, Dutch, English, Faeroese, Finnish, French, German, Icelandic,
Irish, Italian, Norwegian, Portuguese, Spanish, and Swedish.
<tr>
<td>LATIN2
<td>ISO 8859-2
<td align="center">101
<td align="center">I6/101
<td>Albanian, Czech, English, German, Hungarian, Polish, Romanian, Croatian,
Slovak, and Slovene.
<tr>
<td>LATIN3
<td>ISO 8859-3
<td align="center">109
<td align="center">I6/109
<td>Afrikaans, Catalan, Dutch, English, Esperanto, French, Galician, German,
Italian, Maltese, Spanish, and Turkish.
<tr>
<td>LATIN4
<td>ISO 8859-4
<td align="center">110
<td align="center">I6/110
<td>Danish, English, Estonian, Finnish, German, Greenlandic, Sami, Latvian,
Lithuanian, Norwegian, and Swedish.
<tr>
<td>LATIN5
<td>ISO 8859-9
<td align="center">148
<td align="center">I6/148
<td>Danish, Dutch, English, Faeroese, Finnish, French, German, Irish, Italian,
Norwegian Portuguese, Spanish, Swedish, and Turkish.
<tr>
<td>CYRILLIC
<td>ISO 8859-5
<td align="center">144
<td align="center">I6/144
<td>Bulgarian, Belorussian, English, Macedonian, Russian, Serbocroatian
(Serbian), Ukrainian.
<tr>
<td>ARABIC
<td>ISO 8859-6
<td align="center">127
<td align="center">I6/127
<td>Arabic
<tr>
<td>GREEK
<td>ISO 8859-7
<td align="center">126
<td align="center">I6/126
<td>Greek
<tr>
<td>HEBREW
<td>ISO 8859-8
<td align="center">138
<td align="center">I6/138
<td>Hebrew
<tr>
<td>KATAKANA
<td>JIS X 0201
<td align="center">14, 13
<td align="center">I14/13
<td>Japanese (Roman and Katakana)
<tr>
<td>JAPANESE-EUC
<td>JIS X 0201,<br>JIS X 0208
<td align="center">14, 13, 87
<td align="center">I14/87/13
<td> Japanese (Roman, Katakana, Hiragana, Kanji), English, Greek, Russian.
<tr>
<td>CHINESE
<td>CS GB 2312-80
<td align="center">58
<td align="center">I55/58
<td>Chinese (Roman, Phonetic, and Hanzi), Japanese (Roman, Katakana,
Hiragana), English, Greek, Russian
<tr>
<td>KOREAN
<td>KS C 5601
<td align="center">149
<td align="center">I6/149
<td>Korean (Hangul, Hanja), Japanese (Roman, Katakana, Hiragana), Greek,
Russian, English, and others
<tr>
<td>VIETNAMESE
<td>TCVN 5712
<td align="center">180
<td align="center">I6/180
<td>Vietnamese
</table>
</blockquote>
<p>
<blockquote>
<i>
UPDATE: After this paper was written, Unicode UCS-2 (Big Endian) and UTF-8
were added as transfer character sets. The designator for UCS-2 is I162; the
designator for UTF-8 is I190. UTF-8 can be used to convert between any two
known character sets, but the overhead can be higher than using a
script-specific transfer character set. For example, using ISO 8859-5
as the TCS for Cyrillic text transfer does not expand the data at all, whereas
UTF-8 doubles its size.
</i>
</blockquote>
<h4>Examples</h4>
Let's look at a few of many possible translation scenarios. Each one presents
its own set of problems requiring decisions by the creator of the translation
function, or by the user.
<p>
<ol>
<li>From a 7-bit set to a different 7-bit set, e.g. from the Spanish version
of ISO 646 to ASCII (or vice versa). The two sets do not contain the same
characters. Here we must choose between readability (R) and invertibility
(I). To achieve readability in the Spanish-to-ASCII direction, we strip
diacritical marks (n-tilde becomes simply n, and so on). To achieve
invertibility, we make no translation at all.
<p>
<li>From a 7-bit set to an 8-bit set. The 7-bit sets are usually ASCII or an
ISO 646 national version. Often, all the characters from the 7-bit set
are also present in the 8-bit set, and there is no R-versus-I conflict.
For example: ASCII (and most ISO 646 national variants) to Latin-1 —
here we satisfy both R and I. In other cases we must choose between R and
I. For example: the ISO 646 Italian national variant to ISO Latin /
Arabic: here we either remove the accents for readability, or map the
accented characters into right-half characters for invertibility.
<p>
<li>From an 8-bit set to another 8-bit set. A common case is converting
between one of the corporate "extended ASCII" sets (Digital, IBM, HP,
Apple, NeXT, Data General) and ISO Latin-1. The two sets share a large
percentage of common characters. How do we handle the characters that
differ? Again, we must choose between R and I. To complicate matters, the
IBM, Apple, and NeXT sets use the forbidden C1 control-character area for
graphic characters. To create an invertible translation in the absence
of an official corporate standard, we use the Simple Rule.
<p>
<li>From an 8-bit set to a 7-bit set. For example, from Latin-1 to ASCII or
to an ISO 646 national set. Here we are forced to accept a large amount
of information loss. We cannot possibly achieve invertibility, so we aim
for maximum readability, for example by removing diacritics or invoking
language-specific rules.
<p>
<li>From a single-byte character set to a multibyte character set. Most
multibyte character sets include ASCII and sometimes several other alphabets
(such as Greek and Cyrillic). Here we translate each character into its
equivalent, if it has one. When it doesn't, we must choose between R and
I. For example, "Ö" is not found in JIS X 0208 so it can be mapped to
"O" for readability, or some unique value (preferably one unassigned in JIS X
0208) for invertibility.
<p>
<li>From a multibyte set to a single-byte set, for example Japanese JIS X
0208 into Latin-1 (or Latin/Cyrillic, Latin/Greek, or even ASCII). An
invertible translation is clearly impossible. A readable translation
would require rendering Kanji ideograms phonetically or translating them
into an entirely different language, clearly beyond the scope of a
character-set conversion scheme.
<p>
<li>From one national multibyte set to another. These sets are for Chinese,
Japanese, and Korean, and have a very large number of characters —
ideograms, ASCII graphics, Greek, and Cyrillic characters — in common.
They also have large blocks of unassigned character positions, so the
characters they do not share in common (such as the Chinese phonetic
symbols that are absent from the standard Japanese set) can be assigned
to these areas to preserve invertibility.
</ol>
<p>
No two programmers are likely make the same decisions and this will lead to
inconsistent translations (unless the Simple Rule is followed). This
emphasizes the need for officially published translations between the private
and standard sets. And as this list suggests, we also need translations
between some of the standard sets themselves, for example Chinese and Korean.
This need is addressed to some extent by the Unicode books <a
href="#refs">[27]</a>, which include the mappings from various character sets
to the Unicode set.
<h4>Performance</h4>
Character-set translation in itself does not affect the performance of the
Kermit file transfer protocol to any significant degree. The introduction of
per-character translation introduces an extra table access or function call
but the extra work is usually minimal. In general, the bottlenecks are
elsewhere.
<p>
One of the strong points of the Kermit protocol is its ability to transfer
8-bit data in the 7-bit communication environment. This is done using a
single shift, or prefixing, technique in which each 8-bit character is
stripped of its 8th bit and then prefixed by a special shift-indicating
character. This results in negligible overhead for English and Western
European text (such as French, German, Italian).
<p>
But for text in "right-handed" languages like Russian, Greek, Hebrew, and
Arabic, where text characters come predominantly from the right half of the
character set, single shifts can result in up to 80% overhead. The situation
is even worse for Japanese EUC, in which all Kanji bytes have their 8th bits
set to 1, resulting in transmission overhead of 100% for pure Kanji text on a
7-bit connection.
<p>
Because 7-bit communication is still prevalent, Kermit's support for Greek,
Cyrillic, Hebrew, Arabic, and Japanese text file transfer calls for a more
efficient technique. This was accomplished by adding a locking shift
mechanism to the Kermit protocol, allowing sequences of 8-bit characters to be
transmitted in 7-bit form with shifting overhead applying to entire 8-bit
sequences, rather than to each 8-bit character. Isolated 8-bit characters can
still be transmitted using single shifts. These methods are very similar to
those of ISO 2022, but without the risk of "loss-of-state" due to corruption
or loss of the shift characters, and with the addition of the "single-shift-1"
mechanism lacking from ISO 2022. A combination of single and locking shifts
can achieve maximum efficiency by using a lookahead technique. A detailed
specification is given elsewhere <a href="#refs">[12]</a>.
<p>
The addition of locking shifts to the Kermit protocol increases the
transfer efficiency on 7-bit connections for typical Cyrillic text by about
50% and for typical Kanji text by more than 90%, bringing these transfers to
within the efficiency range of normal 7-bit ASCII transfers on the same
connections.
<hr>
<h3>CONCLUSIONS AND RECOMMENDATIONS</h3>
File transfer character-set translation is an optional feature for Kermit
programs, and is designed to interoperate (with, of course, no claim to
correct translation) with Kermit programs that do not support it. As of this
writing, translation of (at least) Roman, Cyrillic, Hebrew, and Japanese text
is supported by MS-DOS Kermit for the IBM PC <a href="#refs">[9]</a>, IBM
mainframe Kermit for VM, MVS, and CICS <a href="#refs">[3]</a>, and C-Kermit
for UNIX, VMS, OS/2, and other operating systems <a href="#refs">[6]</a>.
Three basic commands were added to these programs to select the file and
transfer character sets and any desired language-specific translation rules.
Locking shifts are used automatically in the 7-bit communication environment
to prevent Kermit from discriminating against "right-handed" character sets.
Kermit programs equipped with the new translation features have become popular
in Europe, Latin America, the former Soviet Union, and Japan. Work is in
progress to add further translation capabilities for other parts of the world.
<p>
Space has not permitted discussion of the details of the Kermit protocol,
the forms of the commands, translation negotiation and refusal mechanisms,
unilateral and local translation features, automatic matching of file and
transfer character sets, character sets in terminal emulation, and numerous
other issues. These will be covered in a future edition of reference <a
href="#refs">[5]</a>.
<p>
In the discussions that resulted in the character-set translation
extension to Kermit, the most fundamental lesson we have learned is that if
existing standards can solve a particular problem, they should be used
instead of inventing new techniques to solve the same problem. Applying this
lesson to Kermit file transfer results in the following conclusions:
<p>
<ol>
<li>Only ISO-registered standard character sets should be used for
interchange. This eliminates the need for any computer to support any
character sets except its own and the corresponding well-known standards.
<p>
<li>The sender should identify the transfer character set to the receiver
using a standard notation such as its ISO registration number. This
eliminates the need for setting up separate and redundant registration
authoritities for character-set identifiers.
<p>
<li>Translations should be invertible, readable, and consistent. When all
three of these goals cannot be achieved by a single translation, the user
should be able to choose the translation goal.
</ol>
<p>
These principles can be applied to any form of textual data interchange,
including electronic mail, network file systems, terminal emulation, virtual
terminal service, distributed databases, remote procedure calls, cutting and
pasting among object-oriented applications, and so on.
<p>
The translation process itself, however, remains ill-defined. It is hoped
that the industry and the standards organizations will take the following
steps:
<p>
<ol>
<li>Standard character sets should be used in preference to private character
sets.
<p>
<li>Owners of private character sets should publish official invertible
translations to ISO-registered standard sets. In the absence of official
translations, a simple procedure such as the one presented in this paper
should be used to achieve consistent invertible translations across all
applications.
<p>
<li>Standards organizations are encouraged to consider publishing
translations between different standard character sets, such as the
Japanese, Chinese, and Korean sets, as well as readable transliterations
among different alphabetic writing systems, such as Roman, Greek, Hebrew,
Arabic, and Cyrillic.
<p>
<li>Operating system designers should consider tagging plain-text files with
character-set identifiers, like the Kermit tags listed in Table 1, to
allow applications software to determine a file's character set
automatically. When standard character sets are used, their tags should
be consistent across different operating systems.
</ol>
<p>
Ten or twenty years from now, perhaps all the computers, as well as all the
display, entry, printing, and telcommunication devices of the world will use
one universal character set, and the issues discussed in this paper will be
irrelevant. On the other hand, perhaps the accumulated and ever-growing
installed base of existing hardware, software, and electronic information will
prove too massive for conversion and the universal character set will be just
one more character set on the list.
<h4>ACKNOWLEDGEMENTS</h4>
My deepest thanks to Christine M. Gianone of Columbia University for inspiring
the work and ideas described in this paper, and for her key contributions
thereto. Special thanks also to the others who played prominent roles in the
design and development of Kermit's character-set translation capabilities: Joe
R. Doupnik of Utah State University, John Chandler of the Harvard /
Smithsonian Center for Astrophysics, Hirofumi Fujii of the Japan National
Laboratory of High Energy Physics in Tokyo, John Klensin of the United Nations
University, Andre Pirard of the University of Liege in Belgium, Johan van
Wingen of the Netherlands and numerous ISO committees, Gisbert W. Selke of the
Wissenschaftliches Institut der Ortskrankenkassen in Bonn, Germany, and
Konstantin Vinogradov of the International Centre for Scientific and Technical
Information (ICSTI) in Moscow. Grateful acknowledgements also to Juri
Gornostaev and A. Butrimenko of ICSTI for hosting the First International
Kermit Conference in Moscow <a href="#refs">[11]</a> in Spring 1989, where the
ideas in this paper received their first public hearing. Thanks also to the
many participants in the ISO8859, UNICODE, and ISO10646 network discussion
groups for valuable information and insights.
<h4>ABOUT THE AUTHOR</h4>
FRANK DA CRUZ is Manager of Communication Software Development at Columbia
University, author of the book <cite>Kermit, A File Transfer Protocol</cite>,
co-author (with Christine M. Gianone) of the book <cite>Using C-Kermit</cite>,
leader of the team that developed the Kermit protocol, and principal author of
several Kermit software programs including C-Kermit.
Present address: Columbia University, 612 West 115th Street, New York, NY
10025, USA; Email: fdc@columbia.edu.
<h4><a name="refs">REFERENCES</a></h4>
<ol>
<li><cite>ANSI X3.4-1986, Code for Information Interchange</cite>. American
National Standards Institute, 1986. The ASCII specification; the US version
of ISO 646.
<p>
<li><cite>CCITT Recommendation T.61, Character Repertoire and Coded Character
Sets for the International Teletex Service</cite>. CCITT, Geneva, 1980
(amended 1984).
<p>
<li>Chandler, John. <cite>IBM System/370 Kermit User's Guide</cite>. Columbia
University Academic Information Systems, 1993. Available in separate versions
for VM/CMS, MVS/TSO, and CICS.
<p>
<li><cite>Chinese Standard GB 2312-80, Coded Chinese Graphic Character Set for
Information Interchange</cite>. China Association for Standardization,
Beijing, 1980.
<p>
<li>da Cruz, Frank. <cite>Kermit, A File Transfer Protocol</cite>. Digital
Press, Bedford, MA, 1987.
<p>
<li>da Cruz, Frank and Christine Gianone. <cite>Using C-Kermit</cite>.
Digital Press, Burlington, MA, 1993. EY-J896E-DP. Also available in German.
<i>UPDATE: 2nd Edition published 1997</i>.
<p>
<li>Data General. <cite>Programming the Display Terminal: Models D217, D413,
and D463</cite>. Data General, Westboro, MA, 1991. 014-002111-00.
<p>
<li>─Éß╗ù, James, Ngô Thanh Nhàn, Hoang Nguyen.
"A proposal for Vietnamese character encoding standards in a unified text
processing framework". <cite>Computer Standards & Interfaces</cite> 14
(1992).
<p>
<li>Gianone, Christine M. <cite>Using MS-DOS Kermit</cite>. Digital Press,
Burlington, MA, 1992. EY-H893E-DP, Also available in French and German
editions.
<p>
<li>Gianone, Christine M. "Have Kermit, Will Travel". <cite>Kermit
News</cite> 3, 1 (June 1988).
<p>
<li>Gianone, Christine M. "Mission to Moscow". <cite>Kermit News</cite>,
4 (June 1990).
<p>
<li>Gianone, Christine M. and Frank da Cruz. <cite>A Locking Shift Mechanism
for the Kermit File Transfer Protocol</cite>. Columbia University, 1991.
<p>
<li><cite>IBM Character Data Representation Architecture, Level 1
Registry</cite>. IBM Canada Ltd., National Language Technical Centre,
Ontario, 1990. SC09-1391-00.
<p>
<li><cite>ISO Standard 646, 7-Bit Coded Character Set for Information
Processing Interchange</cite>. Second edition, International Organization for
Standardization, 1983. Also available as ECMA-6, and similar to CCITT T.50.
<p>
<li><cite>ISO International Standard 2022, Information processing — ISO 7-bit
and 8-bit coded character sets — Code extension techniques</cite>. Third
edition, International Organization for Standardization, 1986. Also available
as ECMA-35.
<p>
<li><cite>ISO International Standard ISO 2375, Information processing —
Procedure for Registration of Escape Sequences</cite>. International
Organization for Standardization, 1985.
<p>
<li><cite>ISO International Standard 4873, Information processing — ISO 8-bit
code for information interchange — Structure and rules for
implementation</cite>. Second edition, International Organization for
Standardization, 1986. Also available as ECMA-43.
<p>
<li><cite>ISO International Standard 8859 Parts 1 through 9, Information
Processing — 8-Bit Single-Byte Coded Graphic Character Sets</cite>.
International Organization for Standardization, 1987-. ISO 8859-1 through -4
are the Latin Alphabets 1 through 4, also available as ECMA-94. ISO 8859-5 is
the Latin/Cyrillic Alphabet (ECMA 113).
<p>
<li><cite>ISO/IEC 10646-1, International Standard 10646, Information
Technology — Universal Multiple-Octet Coded Character Set (UCS)</cite>.
ISO/IEC JTC1, 1993.
<p>
<li><cite>ISO International Standard 6429, Information processing — C1
Control Character Set of ISO 6429</cite>. International Organization for
Standardization, 1983.
<p>
<li><cite>ISO International Register of Coded Characters to Be Used with
Escape Sequences</cite>. European Computer Manufacturers Association (ECMA),
1990, updated periodically.
<p>
<li><cite>JIS X 0201, The Japanese Katakana and Roman Set of
Characters</cite>. Japan Industrial Standards Committee, 1969.
<p>
<li><cite>JIS X 0208, The Japanese Graphic Character Set for Information
Interchange</cite>. Japan Industrial Standards Committee, 1983.
<p>
<li><cite>JIS X 0212, Supplementary Japanese Graphic Character Set for
Information Interchange</cite>. Japan National Committee on ISO/IEC JTC1/SC2,
1991.
<p>
<li><cite>Korean Standard KS C 5601-1987, Korean Graphic Character Set for
Information Interchange</cite>. Korea Bureau of Standards, 1987.
<p>
<li>Padlipsky, M. A. <cite>The Elements of Networking Style</cite>. Prentice
Hall, 1985.
<p>
<li>The Unicode Consortium. <cite>The Unicode Standard, Worldwide Character
Encoding, Version 1.0</cite>. Addison-Wesley Publishing Company, Volume 1,
1991; Volume 2, 1992.
<p>
<li>Viet Nam General Department for Standardization. <cite>Vietnamese National
Standard TCVN 5712, 8-bit Vietnamese Standard Code for Information Interchange
(VSCII)</cite>. Viet Nam State Committee for Sciences, 1993.
</ol>
<p>
(End)
</body></html>