[Haskell-cafe] Ready for testing: Unicode support for Handle I/O

[Simon Marlow]

John Goerzen wrote:
> Duncan Coutts wrote:
>> Sorry, I think we've been talking at cross purposes.
> 
> I think so.
> 
>>> There always has to be *some* conversion from a 32-bit Char to the
>>> system's selection, right?
>> Yes. In text mode there is always some conversion going on. Internally
>> there is a byte buffer and a char buffer (ie UTF32).
>>
>>> What exactly do we have to do to avoid the penalty?
>> The penalty we're talking about here is not the cost of converting bytes
>> to characters, it's in switching which encoding the Handle is using. For
>> example you might read some HTTP headers in ASCII and then switch the
>> Handle encoding to UTF8 to read some XML.
> 
> Simon referenced a 30% penalty.  Are you saying that if we read from a
> Handle using the same encoding that we used when we first opened it,
> that we won't see any slowdown vs. the system in 6.10?

No, there's a fixed 30% penalty for hGetContents/readFile/hPutStr in the 
new library, regardless of what encoding you're using.  Presumably if 
you're using a complex encoding there will be an extra penalty imposed by 
iconv.

The cost is mostly in decoding (or copying, in the case of latin1) bytes 
from the byte buffer into characters in the character buffer.  Previously 
there was only a byte buffer.

I was surprised at the slowdown too, so I looked into it.  As it turns out, 
hGetContents and hPutStr are actually quite well optimised already: 
virtually all the allocation is accounted for by the [Char], and we have 
good tight inner loops, so the cost of shuffling between the byte buffer 
and the char buffer is quite noticeable.

We could add a special-case for the latin1 encoding and eliminate the 
intermediate char buffer, but that would add significant complexity to the 
code, and it's not the right way to go about it.  If you want binary data 
and speed, then hGetBuf/hPutBuf (perhaps via bytestring) should be as fast 
as before.

Cheers,
	Simon