[Xmonad] XSelection.hs - Unicode problems

[Mats Jansborg]

Andrea Rossato <mailing_list at istitutocolli.org> writes:

> I hope Mats Jansborg is still reading the mailing list: hopefully he
> could give us some direction.

I am, sorry for the late reply. Although I'm not sure I'm comfortable
being appointed some kind of x11 unicode authority; I'm completely new
to low-level X11 programming and what knowledge I have can be had more
quickly and reliably by reading the X11 manual, the ICCCM and Keith
Packard's paper at http://keithp.com/~keithp/talks/selection.ps.

Never the less, since you ask I'll try to answer as well as I can.

> On Sat, Aug 25, 2007 at 05:43:53AM -0400, Gwern Branwen wrote:
>> However, while it works fine for your basic bread and butter ASCII
>> characters, I noticed that it does terrible things to more exotic
>> phrases involving Unicode, such as "Henri Poincaré". I borrowed
>> some code from utf-string, and that improved it a little bit -
>> "Henri Poincar�" now becomes "Henri Poincarý" which is still
>> better than "Henri Poincar\245" or whatever.
>> 
>
> Well, it is not better: Poincar\245 is right, the other is wrong,
> since part of the character has been truncated.

> The problem is not Haskell, it's me (and you..;-): actually

I'm afraid it's a little of both :)

Henri Poincaré is not quite "exotic" enough for X to have any problems
with it, for this simple case it is actually all Haskell's fault. Henri
Poincaré is completely representable in ISO-8859-1 which is what you get
when you ask for the selection encoded as STRING.

The problem is that most Haskell functions that deal with String and
interface with the operating system are broken, they behave as though
they had types involving [Word8] instead of [Char] where only the least
significant byte is used. As a workaround the programmer must manually
convert the Haskell String to the locale encoding, represented as a
Haskell [Char] using only the low byte of each Char. To do this properly
you need iconv or something similar such as
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/encoding-0.2.

To support only utf-8 locales is perhaps a little better than supporting
only ASCII or ISO-8859-1, but still fundamentally wrong in my opinion.

The other problem is that you ask for the selection in the STRING
encoding. This limits its usefulness immensely, as STRING is unlikely to
support many of the characters in the user's locale. X11 and ICCCM
defines a method of negotiating the format of the selection. Both
UTF8_STRING (which is not in ICCCM but supported by most modern
applications) and COMPOUND_TEXT should imho be preferred to STRING which
can be used as fallback encoding if neither of the two first two are
available. I've previously posted an example of how to read compound text
properties in my patch to the NamedWindow module, although that method
is not portable either since it uses withCWString from Foreign.C.String
which works correctly only if __STDC_ISO_10646__ is defined (i.e. if
wchar_t is UTF-32).

> If I test hxsel with the first 3 Cyrillic characters of this page:
> http://gorgias.mine.nu/unicode.php
> I get:
> \u041f\u0440\u0438
> which is the correct answer. The problem is: how can I convert this
> unicode characters into something that can be printed?

Yes, since it is impossible to represent Cyrillic characters in
ISO-8859-1, Firefox (or whatever browser you use to provide the
selection) makes its best effort and renders it as the string ['\\',
'0', '4', '1', ... ]. The proper solution is not to try to interpret
this string as an application is free to do whatever it wants with
unrepresentable characters (including omitting them or replacing them
with e.g. '?'). The solution is to ask for the selection in an encoding
where these characters are representable, convert it to Haskell String,
convert that String to the locale C encoding and pass it to the
operating system. In simple applications you can sometimes skip the
middle step and ask for the selection in the locale encoding and pass
that directly back to the os. This is very convenient when it works but
it breaks down as soon as you need to interpret the data as a string of
characters, for instance because you want to prepend "/bin/sh -c" to it.

In addition, not having anything to do with unicode, from my quick read
through the ICCCM it appears the requestor (you) is responsible for
deleting the target property, and that currentTime should not be used in
the xConvertSelection request. Note also that the window is not
destroyed in the case where the selection is properly converted and that
you do not check that the conversion has succeeded before starting the
transfer of the property.

/Mats