H98 Text IO

[Simon Marlow]

Duncan Coutts wrote:
>>From the H98 report:
> 
>         All I/O functions defined here are character oriented. [...]
>         These functions cannot be used portably for binary I/O.
>         
>         In the following, recall that String is a synonym for [Char]
>         (Section 6.1.2).
> 
> So ordinary text Handles are for text, not binary. Char is of course a
> Unicode code point.
> 
> The crucial question of course is what encoding of text to use. For the
> H98 IO functions we cannot set it as a parameter, we have to pick a
> sensible default. Currently different implementations disagree on that
> default. Hugs has for some time used the current locale on posix systems
> (and I'm guessing the current code page on windows). GHC has always used
> the Latin-1 encoding.
> 
> These days, most operating systems use a locale/codepage encoding that
> covers full the Unicode range. So on hugs we get the benefit of that but
> on GHC we do not.
> 
> This is endlessly surprising for beginners. They do
> putStrLn "αβγδεζηθικλ"
> and it comes out on their terminal as junk.
> 
> It also causes problems for serious programs, see for example the recent
> hand-wringing on cabal-devel.
> 
> So here is a concrete proposal:
> 
>       * Haskell98 file IO should always use UTF-8.
>       * Haskell98 IO to terminals should use the current locale
>         encoding.

While I support Duncan's proposal (we discussed it on IRC), I thought I 
should point out some of the ramifications of this, and the alternatives.

If everything that is not a terminal uses UTF-8 by default, then shell 
commands may behave in an unexpected way, e.g. for a Haskell program "prog",

   prog | cat

will output in UTF-8, and if your locale encoding is something other than 
UTF-8 you'll see junk.  Similarly,

   prog >file; cat file

will give the same (wrong) result.

So some alternatives that fix this are

   1. all text I/O is in the locale encoding (what C and Hugs do)

   2. stdin/stdout/stderr and terminals are always in the locale
      encoding, everything else is UTF-8

   3. everything is UTF-8

(1) has the advantage of being easy to understand, but causes problems when 
you want to move a file created on one system to another system, or share 
files between users.  The programmer in this case has to anticipate the 
problem and set an encoding (and we're not proposing to provide a way to 
specify encodings, yet, so openBinaryFile and a separate UTF-8 step would
be required).

(2) has a sort of "do what I want" feel, and will almost certanly cause
confusion in some cases, simply because it's an aribtrary choice.

(3) is easy to understand, but does the wrong thing for people who have
a locale encoding other than UTF-8.

Duncan's proposal occupies a useful point: text that we know to be 
ephemeral, because it is being sent to a terminal, is definitely sent in 
the user's default encoding.  Text that might be persistent or might be 
crossing a locale-boundary is always written in UTF-8, which is good for 
interchange and portability, the catch is that sometimes we identify a 
Handle as persistent when it is really ephemeral.

Note that sensible people who set their locale to UTF-8 are not affected by 
any of this - and that includes most new installations of Linux these days, 
I believe.

Cheers,
	Simon