patch applied (ghc-6.12/packages/base): Strip any Byte Order Mark (BOM) from the front of decoded streams.

[Duncan Coutts]

On Sat, 2009-10-03 at 04:50 -0700, Ian Lynagh wrote:
> Wed Sep 30 01:42:29 PDT 2009  Ben.Lippmeier at anu.edu.au
>   * Strip any Byte Order Mark (BOM) from the front of decoded streams.
>   Ignore-this: d0d0c3ae87b31d71ef1627c8e1786445
>   When decoding to UTF-32, Solaris iconv inserts a BOM at the front
>   of the stream, but Linux iconv doesn't. 
> 
>     M ./GHC/IO/Handle/Internals.hs -6 +27

I agree with Simon that this is not the correct fix.

As Simon suspected, Solaris iconv does indeed insert a BOM if you ask to
convert into UTF-32. Arguably this is actually the correct behaviour.
Also, as Simon suspected, it does not insert a BOM if you ask to convert
into UTF-32BE or LE.

So the better solution is to ask iconv for UTF-32BE or UTF-32LE
depending on the host byte order. This should also work correctly on
Linux so there doesn't need to be Solaris #ifdeffery (just host order
#ifdeffery which is needed anyway).

Demo: (Solaris iconv on big endian CPU)

echo foo | iconv -f UTF-8 -t UTF-32 | hexdump -c

0000000  \0  \0 376 377  \0  \0  \0   f  \0  \0  \0   o  \0  \0  \0   o
0000010  \0  \0  \0  \n

echo foo | iconv -f UTF-8 -t UTF-32BE | hexdump -c
0000000  \0  \0  \0   f  \0  \0  \0   o  \0  \0  \0   o  \0  \0  \0  \n

The ambiguity with "UTF-32" is whether you are asking for UTF-32 in the
host byte order, or if you are asking for UTF-32 suitable for external
storage (which should use the BOM). The GNU iconv takes the former
interpretation while Solaris iconv takes the latter.

Duncan