3. Locale setup
3.1 Files & the kernel
You can now already use any Unicode characters in file names. No kernel or file utilities need modifications. This is because file names in the kernel can be anything not containing a null byte, and '/' is used to delimit subdirectories. When encoded using UTF-8, non-ASCII characters will never be encoded using null bytes or slashes. All that happens is that file and directory names occupy more bytes than they contain characters. For example, a filename consisting of five greek characters will appear to the kernel as a 10-byte filename. The kernel does not know (and does not need to know) that these bytes are displayed as greek.
This is the general theory, as long as your files stay inside Linux. On filesystems which are used from other operating systems, you have mount options to control conversion of filenames to/from UTF-8:
- The "vfat" filesystems has a mount option "utf8". See file:/usr/src/linux/Documentation/filesystems/vfat.txt. When you give an "iocharset" mount option different from the default (which is "iso8859-1"), the results with and without "utf8" are not consistent. Therefore I don't recommend the "iocharset" mount option.
- The "msdos", "umsdos" filesystems have the same mount option, but it appears to have no effect.
- The "iso9660" filesystem has a mount option "utf8". See file:/usr/src/linux/Documentation/filesystems/isofs.txt.
- Since Linux 2.2.x kernels, the "ntfs" filesystem has a mount option "utf8". See file:/usr/src/linux/Documentation/filesystems/ntfs.txt.
3.2 Upgrading the C library
glibc-2.2 supports multibyte locales, in particular UTF-8 locales. But glibc-2.1.x and earlier C libraries do not support it. Therefore you need to upgrade to glibc-2.2. Upgrading from glibc-2.1.x is riskless, because glibc-2.2 is binary compatible with glibc-2.1.x (at least on i386 platforms, and except for IPv6). Nevertheless, I recommend to have a bootable rescue disk handy in case something goes wrong.
Prepare the kernel sources. You must have them unpacked and configured. /usr/src/linux/include/linux/autoconf.h must exist. Building the kernel is not needed.
Retrieve the glibc sources ftp://ftp.gnu.org/pub/gnu/glibc/, su to root, then unpack, build and install it:
# unset LD_PRELOAD
# unset LD_LIBRARY_PATH
# tar xvfz glibc-2.2.tar.gz
# tar xvfz glibc-linuxthreads-2.2.tar.gz -C glibc-2.2
# mkdir glibc-2.2-build
# cd glibc-2.2-build
# ../glibc-2.2/configure --prefix=/usr --with-headers=/usr/src/linux/include --enable-add-ons
# make
# make check
# make info
# LC_ALL=C make install
# make localedata/install-locales
Upgrading from glibc versions earlier than 2.1.x cannot be done this way; consider first installing a Linux distribution based on glibc-2.1.x, and then upgrading to glibc-2.2 as described above.
Note that if -- for any reason -- you want to rebuild GCC after having installed glibc-2.2, you need to first apply this patch gcc-glibc-2.2-compat.diff to the GCC sources.
3.3 General data conversion
You will need a program to convert your locally (probably ISO-8859-1) encoded texts to UTF-8. (The alternative would be to keep using texts in different encodings on the same machine; this is not fun in the long run.) One such program is `iconv', which comes with glibc-2.2. Simply use
$ iconv --from-code=ISO-8859-1 --to-code=UTF-8 < old_file > new_file
Here are two handy shell scripts, called "i2u" i2u.sh (for ISO to UTF conversion) and "u2i" u2i.sh (for UTF to ISO conversion). Adapt according to your current 8-bit character set.
If you don't have glibc-2.2 and iconv installed, you can use GNU recode 3.6 instead. "i2u" i2u_recode.sh is "recode ISO-8859-1..UTF-8", and "u2i" u2i_recode.sh is "recode UTF-8..ISO-8859-1". ftp://ftp.gnu.org/pub/gnu/recode/recode-3.6.tar.gz
Or you can also use CLISP instead. Here are "i2u" i2u.lisp and "u2i" u2i.lisp written in Lisp. Note: You need a CLISP version from July 1999 or newer. ftp://clisp.cons.org/pub/lisp/clisp/source/clispsrc.tar.gz.
Other data conversion programs, less powerful than GNU recode, are `trans' ftp://ftp.informatik.uni-erlangen.de/pub/doc/ISO/charsets/trans113.tar.gz, `tcs' from the Plan9 operating system ftp://ftp.informatik.uni-erlangen.de/pub/doc/ISO/charsets/tcs.tar.gz, and `utrans'/`uhtrans'/`hutrans' ftp://ftp.cdrom.com/pub/FreeBSD/distfiles/i18ntools-1.0.tar.gz by G. Adam Stanislav <adam@whizkidtech.net>.
For the repeated conversion of files to UTF-8 from different character sets, a semi-automatic tool can be used: to-utf8 presents the non-ASCII parts of a file to the user, lets him decide about the file's original character set, and then converts the file to UTF-8.
3.4 Locale environment variables
You may have the following environment variables set, containing locale names:
- LANGUAGE
override for LC_MESSAGES, used by GNU gettext only
- LC_ALL
override for all other LC_* variables
- LC_CTYPE, LC_MESSAGES, LC_COLLATE, LC_NUMERIC, LC_MONETARY, LC_TIME
individual variables for: character types and encoding, natural language messages, sorting rules, number formatting, money amount formatting, date and time display
- LANG
default value for all LC_* variables
man 7 locale' for a detailed description.)
Each of the LC_* and LANG variables can contain a locale name of the following form:
language[_territory[.codeset]][@modifier]
where language is an ISO 639 language code (lower case), territory is an ISO 3166 country code (upper case), codeset denotes a character set, and modifier stands for other particular attributes (for example indicating a particular language dialect, or a nonstandard orthography).
LANGUAGE can contain several locale names, separated by colons.
In order to tell your system and all applications that you are using UTF-8, you need to add a codeset suffix of UTF-8 to your locale names. For example, if you were using
LC_CTYPE=de_DE
LC_CTYPE=de_DE.UTF-8
You do not need to change your LANGUAGE environment variable. GNU gettext in glibc-2.2 has the ability to convert translations to the right encoding.
3.5 Creating the locale support files
You create using localedef the support files for each UTF-8 locale
you intend to use, for example:
$ localedef -v -c -i de_DE -f UTF-8 de_DE.UTF-8
You typically don't need to create locales named "de" or "fr" without country suffix, because these locales are normally only used by the LANGUAGE variable and not by the LC_* variables, and LANGUAGE is only used as an override for LC_MESSAGES.
Next Previous Contents