Unicode again

[Kent Karlsson]

> -----Original Message-----
> From: haskell-admin@haskell.org [mailto:haskell-admin@haskell.org]On
> Behalf Of Wolfgang Jeltsch
> Sent: den 5 januari 2002 13:04
> To: The Haskell Mailing List
> Subject: Unicode again
> 
> 
> Hello,
> there was some discussion about Unicode and the Char type 
> some time ago. At
> the moment I'm writing some Haskell code dealing with XML. 
> The problem is
> that there seems to be no consensus concerning Char so that 
> it is difficult
> for me to deal with the XML unicode issues appropriately. Is there any
> option that is very likely to get into the Haskell report? 

I don't have any opinion on what is likely for Haskell here, but...

> According to my
> memory the following were more or less propagated:
>     (1) range '\x0000' to '\xFFFF'; UTF-16 is used
>     (2) range '\x000000' to '\x10FFFF'; chars denote codepoints
>     (3) range '\x00000000' to '\x7FFFFFFF'; chars denote codepoints


My suggestion here is (2a) range '\x0' to '\xD7FF' union '\xE000' to '\x10FFFF';
chars denote codepoints; the excluded subrange is for "surrogate" codes, they
are excluded from UTF-8 and UTF-32, and must occur in proper pairs in UTF-16.



> GHC 5 seems to implement the second variant; Hugs still uses 
> the poor range
> of '\x00' to '\xFF'. What does nhc98 do?
> My opinion is that using (1) is very, very bad. The name 

Not really.  Most processes on text must take context into account.
Even such a seemingly simple thing as counting what most users think
of as characters (or splitting strings at legitimate points) must take
context into account.  E.g. <a, combining ring above> is one character
in the view of the user (though not in the view of the character processing
programmer).  Everyone that is serious about Unicode and where efficiency
is also of concern(!) target UTF-16 (MacOS, Windows, Epoc, Java, Oracle, ...).
That does not necessarily mean that Haskell should follow suit.  

> "Char" suggests
> that values of the type are characters (glyphs). Well, even 

Characters and glyphs are very different concepts.  I will not go into
detail here, just note that there is not a 1-1 relationship between
characters and glyphs even in a single font for many scripts.

> when using (2)
> or (3) Char does not denote characters but codepoints, but 
> this is closer
> to denoting chars than (1).

Vary marginally.  But when looking at individual characters
(in the Unicode/10646 sense) UTF-32 is better.

> And memory usage shouldn't be an issue - a

For Haskell, I agree.

> concept beeing much 

s/much/marginally (for strings)/ 

> better is IMO worth the higher memory 
> use. I prefer (3)
> over (2) because there is the possibility of expansion of the Unicode
> character set in the future. 

Well, it is going in the other direction.  With amendment 1 to 10646-1:2000
the limit at 10FFFF is strengthened also in 10646, even though it is not
yet as absolute as in Unicode.

	/kent k


> Another solution would be to 
> specify that the
> upper bound of Char is initially '\x10FFFF' and shall be 
> adapted as Unicode
> evolves.
> Also, as already said by another person, we should introduce a package
> dealing with encoding and decoding of character strings to/from octet
> streams. The encoding used with character I/O must be specified.
> Any comments?
> 
> Wolfgang
> 
> _______________________________________________
> Haskell mailing list
> Haskell@haskell.org
> http://www.haskell.org/mailman/listinfo/haskell