[Haskell-cafe] Re: String vs ByteString

[John Meacham]

On Tue, Aug 17, 2010 at 03:21:32PM +0200, Daniel Peebles wrote:
> Sounds to me like we need a lazy Data.Text variation that allows UTF-8 and
> UTF-16 "segments" in it list of strict text elements :) Then big chunks of
> western text will be encoded efficiently, and same with CJK! Not sure what
> to do about strict Data.Text though :)

If space is really a concern, there should be a varient that uses LZO or
some other fast compression algorithm that allows concatination as the
back end. 

<ranty thing to follow>
That said, there is never a reason to use UTF-16, it is a vestigial
remanent from the brief period when it was thought 16 bits would be
enough for the unicode standard, any defense of it nowadays is after the
fact justification for having accidentally standardized on it back in
the day. When people chose to use the 16 bit representation, it was
because they wanted a one-to-one mapping between codepoints and units of
computation, which has many advantages. However, this is no longer true,
if the one-to-one mapping is important then nowadays you use ucs-4,
otherwise, you use utf8. If space is very important then you work with
compressed text. In practice a mix of the two is fairly ideal.

        John

-- 
John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/