Unicode source files

[Simon Marlow]

On 04 May 2005 15:57, Bulat Ziganshin wrote:

> it is true what to support unicode source files only StringBuffer
> implementation must be changed?

It depends whether you want to support several different encodings, or
just UTF-8.  If we only want to support UTF-8, then we can keep the
StringBuffer in UTF-8 and also FastStrings.  (or you could re-encode the
other encodings into UTF-8).

> if so, then task can be simplified by
> converting any files read by hGetStringBuffer to UTF-32 (PackedString)
> representation and putting in memory array in this form. After this,
> we must change indexing of ByteArray to indexing of Array Int Char,
> and somewhat replace call to mkFastSubStringBA#.

This is the other alternative.  It uses rather more memory, but that
might not be an issue.

The other thing that needs to be changed is the lexer, to be able to
recognise classes of Unicode characters (i.e. upper/lower case for
identifiers, symbol characters. etc.).  The code recently added to the
libraries can be used for this, I believe.  

The question is what Alex should see for a unicode character: Alex
currently assumes that characters are in the range 0-255 (you need a
fixed range in order to generate the lexer tables).  One possibility is
to map all Unicode upper-case characters to a single character code for
Alex, and similarly for the other classes of character.

> btw, why in FastString module unicode strings are saved as [Int], not
> as String itself?

Probably for reasons that are no longer relevant.  When we changed Char
from 8 to 32 bits, we still had to compile GHC with older versions of
itself that only supported 8-bit Chars.

Cheers,
	Simon