[Haskell-cafe] Re: String vs ByteString

[Donn Cave]

Quoth John Millikin <jmillikin at gmail.com>,

> Ruby, which has an enormous Japanese userbase, solved the problem by
> essentially defining Text = (Encoding, ByteString), and then
> re-implementing text logic for each encoding. This allows very
> efficient operation with every possible encoding, at the cost of
> increased complexity (caching decoded characters, multi-byte handling,
> etc).

Ruby actually comes from the CJK world in a way, doesn't it?

Even if efficient per-encoding manipulation is a tough nut to crack,
it at least avoids the fixed cost of bulk decoding, so an application
designer doesn't need to  think about the pay-off for a correct text
approach vs. `binary'/ASCII, and the language/library designer doesn't
need to think about whether genome data is a representative case etc.

If Haskell had the development resources to make something like this
work, would it actually take the form of a Haskell-level type like
that - data Text = (Encoding, ByteString)?  I mean, I know that's
just a very clear and convenient way to express it for the purposes
of the present discussion, and actual design is a little premature -
... but, I think you could argue that from the Haskell level,
`Text' should be a single type, if the encoding differences aren't
semantically interesting.

	Donn Cave, donn at avvanta.com