Raw filenames vs locales

[Ian Lynagh]

Hi all,

I'd like to propose some changes to the IO library to fix some problems
(IMO) with hugs' recent closer adherence to the Haskell 98 report. I
believe this is orthogonal to the proposed new IO library stuff that's
been discussed before.

The rest of this mail is in 3 parts. First I describe the problem, then
a proposed solution, and finally some comments on backwards
compatibility.


===========
The problem
===========

With it's closer adherence to the Haskell 98 report, it is no longer
possible with hugs to manipulate files using the standard IO functions
if the filenames are not representable in your locale. To demonstrate
the problem consider this:

--------------------------------------------------
touch `printf "1\xA4"`
echo '
    import System.Directory (getDirectoryContents)
    import Data.Char (ord)
    main = do xs <- getDirectoryContents "."
              print (map (map ord) xs)
 ' > foo.hs
for locale in en_GB en_GB.ISO-8859-15 en_GB.UTF-8
do  
    echo "==============================="
    echo "Doing $locale"
    LC_ALL=$locale ../runhugs foo.hs
done
--------------------------------------------------

Here we create a file whose filename if 1\xA4. \xA4 is a
"currency sign" in ISO-8859-1, "euro sign" in ISO-8859-15 and not a
valid character in UTF-8. We then print the results of
getDirectoryContents, converting the Chars to Ints so we can see what's
going on.

The result is this:

===============================
Doing en_GB
[[46],[46,46],[49,164],[102,111,111,46,104,115]]
===============================
Doing en_GB.ISO-8859-15
[[46],[46,46],[49,8364],[102,111,111,46,104,115]]
===============================
Doing en_GB.UTF-8
[[46],[46,46],[49,65533],[102,111,111,46,104,115]]

The third file is the interesting one. We have:
ISO-8859-1:  164   = U+A4   = "currency sign"
ISO-8859-15: 8364  = U+20AC = "euro sign"
UTF-8:       65533 = U+FFFD = "replacement character"

"replacement character" is "used to replace an incoming character whose
value is unknown or unrepresentable in Unicode".


=================
Proposed solution
=================

My suggestion is essentially that we change all functions using the
FilePath type to instead use FilePath a => a.

[ By jumping through hoops I think this could be done H98-compatibly,
  but for simplicity I'll ignore that for now. I'm not sure if it's a
  problem for any impl anyway? ]

I imagine the class would look something like

class FilePath a where
    to_filename :: a -> IO FileName
    from_filename :: FileName -> IO a

from_free_filename :: FileName -> IO a
from_free_filename f = do x <- from_filename f
                          free f
                          return x

with_filename :: FilePath a => a -> (FileName -> IO b) -> IO b
with_filename x f = do x' <- to_filename x
                       res <- f x'
                       free_filename x'
                       return res

We would then have

  System.IO.Impl.getDirContents :: FileName -> [FileName]
  System.IO.getDirContents :: FilePath a => a -> [a] -- Could be more general
  System.IO.getDirContents x = do ys <- with_filename x Impl.getDirContents
                                  mapM from_free_filename ys

On Unix systems FileName would be a Ptr Word8.
My knowledge of Windows isn't great, but I think there it would be an
array of 16-bit values?

We would have instances of FilePath for String and [Word8] to solve the
immediate problem. String would be the current behaviour, but [Word8]
would be converted to a FileName unchanged. On Windows it would probably
be necessary to throw an exception if a [Word8] is passed which is not
valid utf8.

It would also be nice to have a FileName instance, to avoid unnecessary
conversions. A Ptr Word8 instance would also be handy for things like
darcs' FastPackedString module to be able to use efficiently (without
taking a round trip via a lazy list).

=======================
Backwards compatibility
=======================

I haven't done any research into it, but I hope that a lot of the time
this will not be an issue as the impl will be able to infer the type
String is being used, either by a string literal, the fact it is
putStrLn'd, there is a type signature saying it is a String, etc.

The Haskell 98 modules like IO could re-export the functions with their
types restricted to what they are now. This would give us complete
backwards compatibility to Haskell 98.

It is certainly possible for there to be ambiguities in programs that
use the hierarchial libraries, however. Possible solutions are:

* Tell people to add type sigs to fix it.

* Define the new stuff in System.IO.Impl in a package iobase.
  The oldio package would then contain System.IO which re-exports all
  the functions with the old types, and the io package would do the same
  with the new types.
  Unfortunately i don't think this would work if you have some libraries
  compiled against the io package you don't want to use. I think this
  might be an argument that the package system is not being flexible
  enough.


That's all I've got.
Comments welcomed!


Thanks
Ian