NAME

ConvCS,  CSconverter − Character set conversion suite

SYNOPSIS

include "convcs.m";
convcs := load ConvCS ConvCS->PATH;
 CSconverter: module {
       init: fn(): string;
       threadsafe: fn(): int;
       btos: fn(b: array of byte, eod: int): string;
       stob: fn(s: string): array of byte;
};
 init: fn(csfile: string): string;
getconv: fn(csname: string): (CSconverter, string);
 

DESCRIPTION

The ­ConvCS suite is a collection of ­CSconverter modules for converting various standard coded character sets (CCS) and character encoding schemes (CES) to and from the Limbo string representation (Unicode runes). 

The ­ConvCS module provides an entry point to the suite, mapping character-set names and aliases to their associated ­CSconverter implementation. 

The ConvCS module

init(csfile)
­Init should be called once to initialise the internal state of ConvCS.  The ­csfile argument specifies the path of the character-set converter mapping file.  If this argument is nil, the default mapping file, /lib/charsets, is used. 

getconv(csname)
­Getconv returns an initialised ­CSconverter module, ready for converting the requested character-set csname.

The return value is a tuple, holding the module reference and an error string.  If any errors were encountered in locating, loading or initialising the requested converter, the module reference will be nil and the string will contain an explanation.  If the converter was successfully located and initialised, the string will be nil. 

The character-set name, csname, is normalised by mapping all upper-case latin1 characters to lower-case before comparison with character-set names and aliases from the ­charsets file. 

Using a converter

The ­CSconverter module returned by ­getconv is already initialised and is ready to start the conversion.  Conversions can be made on a individual basis, or in a ‘streamed’ mode. 

converter->btos(b, eod)

Convert raw byte codes of the character-set encoding to a Limbo string. 
The argument ­b is an array of byte, the raw byte codes to be converted. 

A non-zero value for the ­eod argument indicates that the data in ­b is the last of the current conversion.  A zero value is used when streaming data through the converter.  When converting multi-byte character encoding schemes, a valid character encoding may be split across multiple calls to btos().  When in streaming mode, only the completed characters are returned and any incomplete encoding sequence is cached to see if the next call to ­btos() will complete it. 

Upon completing a ‘streamed’ conversion, ­btos() should be called with a non-zero value of eod.  This must be done before commencing another conversion as a pending incomplete encoding could invalidate the start of the new conversion. 

converter->stob(s)

Convert a Limbo string to the raw byte codes of the character-set encoding.  The return value is an array of byte, containing the conversion data. 

Conversion errors

When using converter->btos() to convert data to Limbo strings, any byte sequences that are not valid for the specific character encoding scheme will be converted to the Unicode error character 16rFFFD. 

When using converter->stob() to convert Limbo strings, any Unicode characters that can not be mapped into the character-set will normally be substituted by the US-ASCII code for ‘?’.  Note that this may be inappropriate for certain conversions, such converters will use a suitable error character for their particular character-set and encoding scheme. 

Charset file format

The file ­/lib/charsets provides the mapping between character-set names and their implementation modules, and is used by ­getconv to find the Dis module corresponding to csname.

Each line holds fields separated by white-space; the first field gives the full path of the implementation module, which must implement the ­CSconverter module interface; subsequent fields give the standard names and commonly used aliases of the character-set supported by that module. 

Blank lines, and lines beginning with the ‘#’ character are ignored. 

Implementing a new converter

To implement a new CSconverter, you must implement the ­btos and ­stob functions as described above.  You must also implement the following functions:

­CSconverter->init()
This function is called by ­ConvCS when the converter is first loaded.  This function performs any required initialisation, such as loading required data files or support modules.  The return value is a string.  A nil return value indicates successful initialisation.  If the converter cannot initialise itself properly, it should report the reason via the returned string. 

­CSconverter->threadsafe()
­CsConv calls this function to determine if the converter is thread-safe.  The converter is considered thread safe if overlapping calls to ­btos from separate threads cannot result in incorrect conversions.  This is usually only the case for simple table-based conversions of single-byte character encoding schemes.  If the converter needs to maintain global state for a ‘streamed’ conversion then it cannot be thread-safe. 

The ­ConvCS module maintains a cache of thread-safe converters in order to reduce the run-time overhead of repeatedly loading and initialising the same converter module. 

For simple table-based converters, standard implementations of the above functions are given in the file /appl/lib/convcs/tblconv.b, for inclusion in the specific converter implementation. 

The ­tblconv code requires a ­CHARSET string constant, used for error reporting and a ­cstab array of 256 integers, for mapping character-set byte codes to unicode code-points.  See ­/appl/lib/convcs/iso8859-1.b for an example of its use. 

FILES

­/lib/charsets
The default mapping between character-set names and their implementation modules.

SOURCE

­/appl/lib/convcs/convcs.b
Implementation of the ­ConvCS module. 

/appl/lib/convcs/CSconverter
Character-set specific CSconverter implementations.

­/appl/lib/convcs/tblconv.b
Generic table-based CSconverter code.

BUGS

Most implementations of ­CSconverter->stob() require the converter module to create a large unicode conversion lookup table.  If you use this facility, try not to keep the module reference for any longer than necessary so as to free up memory as soon as possible. 

However, if the module is cached by ­ConvCS the memory will not be released until the last reference to ­ConvCS is dropped. 

Plan 9  —  June 29, 2000