[Haskell-cafe] Re: String vs ByteString

[Ketil Malde]

Michael Snoyman <michael at snoyman.com> writes:

> As far as space usage, you are correct that CJK data will take up more
> memory in UTF-8 than UTF-16. 

With the danger of sounding ... alphabetist? as well as belaboring a
point I agree is irrelevant (the storage format):

I'd point out that it seems at least as unfair to optimize for CJK at
the cost of Western languages.  UTF-16 uses two bytes for (most) CJK
ideograms, and (all, I think) characters in Western and other phonetic
scripts.  UTF-8 uses one to two bytes for a lot of Western alphabets,
but three for CJK ideograms.

Now, CJK has about 20K ideograms, which is almost 15 bits per ideogram,
while an ASCII letter is about six bits.  Thus, the information density
of CJK and ASCII is about equal for UTF-8, 5/8 vs 6/8 - compared to
15/16 vs 6/16 for UTF-16.  In other words a given document translated
between Chinese and English should occupy roughly the same space in
UTF-8, but be 2.5 times longer in English for UTF-16.

-k
-- 
If I haven't seen further, it is by standing in the footprints of giants