[Haskell-cafe] Network.HTTP+ByteStrings Interface--Or: How to shepherd handles and go with the flow at the same time?

[Jules Bean]

I've been having something of a discussion on #haskell about this but
I had to go off-line and, in any case, it's a complicated issue, and I
may be able to be more clear in an email.

The key point under discussion was what kind of interface the HTTP
library should expose: synchronous, asynchronous? Lazy, strict?

As someone just pointed out, "You don't like lazy IO, do you?". Well,
that's a fair characterisation. I think unsafe lazy IO is a very very
cute hack, and I'm in awe of some of the performance results which
have been achieved, but I think the disadvantages are underestimated.

Of course, there is a potential ambiguity in the phrase 'lazy IO'. You
might interpret 'lazy IO' quite reasonably to refer any programming
style in which the IO is performed 'as needed' by the rest of the
program. So, to be clear, I'm not raising a warning flag about that
practice in general, which is a very important programming
technique. I'm raising a bit of a warning flag over the particular
practice of achieving this in a way which conceals IO inside thunks
which have no IO in their types: i.e. using unsafeInterleaveIO or even
unsafePerformIO.

Why is this a bad idea? Normally evaluating a haskell expression can
have no side-effects. This is important because, in a lazy language,
you never quite know[*] when something's going to be evaluated. Or if
it will. Side-effects, on the other hand, are important things (like
launching nuclear missiles) and it's rather nice to be precise about
when they happen. One particular kind of side effect which is slightly
less cataclysmic (only slightly) is the throwing of an exception. If
pure code, which is normally guaranteed to "at worst" fail to
terminate can suddenly throw an exception from somewhere deep in its
midst, then it's extremely hard for your program to work out how far
it has got, and what it has done, and what it hasn't done, and what it
should do to recover. On the other hand, no failure may occur, but the
data may never be processed, meaning that the IO is never 'finished'
and valuable system resources are locked up forever. (Consider a naive
program which reads only the first 1000 bytes of an XML document
before getting an unrecoverable parse failure. The socket will never
be closed, and system resources will be consumed permanently.)

Trivial programs may be perfectly content to simply bail out if an
exception is thrown. That's very sensible behaviour for a small
'pluggable' application (most of the various unix command line
utilities all bail out silently or nearly silently on SIGPIPE, for
example). However this is not acceptable behaviour in a complex
program, clearly. There may be resources which need to be released,
there may be data which needs saving, there may be reconstruction to
be attempted on whatever it was that 'broke'.

Error handling and recovery is hard. Always has been. One of the
things that simplifies such issues is knowing "where" exceptions can
occur. It greatly simplifies them. In haskell they can only occur in
the IO monad, and they can only occur in rather specific ways: in most
cases, thrown by particular IO primitives; they can also be thrown
'To' you by other threads, but as the programmer, that's your
problem!.

Ok. Five paragraphs of advocacy is plenty. If anyone is still reading
now, then they must be either really interested in this problem, or
really bored. Either way, it's good to have you with me! These issues
are explained rather more elegantly by Oleg in [1].


So, where does that leave the HTTP library? Well here are the kinds of
interface I can imagine. I'm deliberately ignoring all the stuff about
request headers, request content, and imagining that this is all about
URL -> ByteString. Here are the options that occur to me:

A. Strict, synchronous GET
    sSynGET :: URL -> IO ByteString

    Quite simply blocks the thread until the whole data has
    arrived. Throws some kind of exception on failure, presumably. This
    is a simple primitive, appropriate for relatively small files
    (files which fit comfortably in your memory) and simple
    programs. It's also great for programs which want to take their own
    control over the degree of asynchrony; they can just fork as many
    threads as they choose to GET with.

B. Strict, asynchronous GET
    sAsynGET :: URL -> IO (MVar ByteString)

    Download the entire data, but do it in a separate thread. Give me
    an MVar so I can synchronise on the data's arrival in whichever way
    suits my program best. Suitable for small files which fit
    conveniently in memory. Very easy to implement in terms of forkIO
    and sSynGET so really it's a convenience function.

C. Strict, synchronous, GET-to-file
    sSynFileGET :: URL -> FilePath -> IO ()
D. Strict, asynchronous, GET-to-file
    sAsynFileGET :: URL -> FilePath -> IO (MVar ())

    Download the entire data to a local file. This means that it
    doesn't matter if the data is far bigger than local memory, it can
    still be done efficiently. [Note that this doesn't mean it must use
    lazy getContents magic under the hood. It could easily use strict
    hGet in reasonable sized chunks and write them out straightaway.]
    The only difference between the two variants is that one keeps
    control until completion, the other gives you an MVar which you can
    block on if/when you choose.  This method is appropriate for
    clients which need extremely large data, and don't mind waiting for
    it to finish before they start processing. It is also appropriate
    for clients which want random access to large data requests (using
    underlying hSeek-based random file IO, once the file has downloaded).

E,F. Progressive GET
    pSynGET :: URL -> ((Bool,ByteString) -> IO ()) -> IO ()
    pAsynGET :: URL -> ((Bool,ByteString) -> IO ()) -> IO (MVar ())

    (This is a particular simple case of Oleg's iteratees, I
    think) Download the data at whatever speed is convenient. As data
    arrives, feed it to the 'callback' provided. The ByteString is the
    new chunk of data, the 'Bool' is just supposed to indicate whether
    or not this is the final chunk. You can imagine slight
    variations. Note that the library promises not to retain any
    references to the ByteString chunks, so if the callback processes
    them and then discards them they are eligible for garbage
    collection. If the callback wishes to accumulate them, it can quite
    easily 'append' the strict chunks into a lazy bytestring, which is
    an efficient operation. This is suitable for applications which
    wish to do something like progressive display of a pJPEG, or
    compute a summary function, or perhaps even display an animation,
    although that last would normally need a little more structure to
    try to guarantee the data rate.

    Incidentally there are more complex options than (Bool,Bytestring)
    -> IO ().  A simple and obvious change is to add a return
    value. Another is a 'state monad by hand', as in (Bool,Bytestring)
    -> s -> s, and change the final return value of the type to IO s,
    which allows the callback to accumulate summary information and
    still be written as pure code. Other options allow the 'callback'
    to request early termination, by layering in an 'Either' type in
    there. Another more sophisticated option, I think, is the higher
    rank

    MonadTrans t => URL ->
    	      	   ((forall m. Monad m) => (Bool,ByteString) -> t m)
		   -> t IO ()

    ...which, unless I've made a mistake, allows you to write in 'any
    monad which can be expressed as a transformer', by transforming it
    over IO, but still contains the implicit promise that the
    'callback' does no IO. For example t = StateT reduces to the
    earlier s -> s example, in effect, with a slightly different data
    layout.

    Another couple of refinements to the above are that in practice you
    normally want to 'guarantee' your callback only runs on a chunk
    size of at least X (in some cases 'exactly X'), or you want to
    guarantee it's called at least every Y seconds. Neither of these
    are hard to add.

Given these three pairs of options, what need is there for an unsafe
lazy GET?  What niche does it fill that is not equally well filled by
one of these?

Program conciseness, perhaps. The kind of haskell oneliner whose
performance makes us so (justly) proud. In isolation, though I don't
find that a convincing argument; not with the disadvantages taken also
into account.

The strongest argument then is that you have a 'stream processing'
function, that is written 'naively' on [Word8] or Lazy ByteString, and
wants to run as data is available, yet without wasting space. I'm
inclined to feel that, if you really want to be able to run over 650M
files, and you want run as data is available, then you in practice
want to be able to give feedback to the rest of your application on
your progress so far; I.e, L.Bytestring -> a is actually too simple a
type anyway.

I'm interested to know what opinions other people have on this,
whether I've made any serious logic mistakes or just overlooked
another approach which has advantages. Having spent quite a while this
evening thinking this over, I just don't see the convincing case for
the unsafe lazy approach, and I see plenty of problems with it...

Cheers,

Jules


* Well, OK. If you're smart, then you know. Because you're smart, and
   you thought about it carefully. But most of the time, you don't need
   to know. And if you produce data (a pure structure) which is then
   consumed by a library - or vice versa - then you don't know unless
   you read their code.

[1] http://www.haskell.org/pipermail/haskell-cafe/2007-March/023073.html