[Haskell-cafe] ANNOUNCE: enumerator 0.4.8

[James Cook]

On Mar 28, 2011, at 12:54 AM, wren ng thornton wrote:

> On 3/27/11 9:58 PM, John Millikin wrote:
>> Resending is slightly more complex -- if the other end can say  
>> "resend that
>> last chunk", then it should be easy enough, but "resend the last 2  
>> bytes of
>> that chunk you sent 5 minutes ago" would be much harder. What is  
>> your use
>> case?
>
> This does highlight one of the restrictions I've lamented about the  
> iteratee framework. Namely that the current versions I've seen place  
> unnecessary limitations on the communication the iteratee is allowed  
> to give to the enumerator/enumeratees above it. This is often  
> conflated with the iteratee throwing an error/exception, which is  
> wrong because we should distinguish between bad program states and  
> argument passing. Moreover the type system doesn't capture the kinds  
> of communication iteratees assume of their enumerators/enumeratees,  
> nor the kinds of communication supported by the enumerators/ 
> enumeratees, which means that failure to hook them up in the right  
> (non-typechecked) way /does/ constitute an error.
>
> The one example that tends to be supported is the iteratee  
> requesting that the enumerator/enumeratees seek to a given position  
> in a file. Which is a good example, but it's not the only one.  
> Requesting the resending of chunks is another good example. But  
> there's no limit to the reasonable kinds of communication an  
> iteratee could want.
>
> In an ideal framework the producers, transformers, and consumers of  
> stream data would have a type parameter indicating the up-stream  
> communication they support or require (in addition to the type  
> parameters for stream type, result type, and side-effect type). That  
> way clients can just define an ADT for their communication protocol,  
> and be done with it. There may still be issues with the Expression  
> Problem, but at least those are pushed out of the stream processing  
> framework itself which really shouldn't care about the types of  
> communication used.

It's somewhat outdated and underdeveloped (I was writing for myself so  
I never really bothered finishing it), but I wrote an exploration of  
iteratee semantics[1] a while back in which I specified an iteratee as  
a monad-transformer stack involving, at its core, the "PromptT" or  
"ProgramT" monad transformers (as far as I know, the same could be  
done with the "Coroutine" monad).  I personally found that  
construction far more lucid than the usual ad-hoc view, and it also  
makes it very clear how the model can be trivially extended to support  
additional operations such as these.

Based on what I learned while writing that (and on the similarity  
between coroutines and the concepts I used), I strongly agree with  
Mario Blažević, suggestion to look at his monad-coroutine library as  
a way of understanding where they fit in some larger design space.  I  
would even go so far as to suggest that something like it could be  
considered as either a replacement for iteratees or as the underlying  
implementation of an iteratee library, because the concept not only  
subsumes iteratees and enumerators, but also delegates control to code  
that can be independent of both rather than simply reversing the  
"conventional" iterator concept.  I believe it also subsumes iterators  
and whatever their corresponding parts are called.

As he mentions, his implementation does not come with all the  
"plumbing", but I think it would be worthwhile to create that  
plumbing, because either coroutines or "operational monads" may very  
well be the basis needed to develop a "grand unified theory" of  
composable stream processing.  If nothing else, the isomorphisms in  
his coroutine-enumerator[2] and coroutine-iteratee[3] packages seem to  
give a much more direct and useful iteratee semantics than I've seen  
given anywhere else, and at the same time they are much more readily  
extended to cover additional operations.

-- James

1. https://github.com/mokus0/junkbox/tree/master/Papers/HighLevelIteratees
2. http://hackage.haskell.org/package/coroutine-enumerator
3. http://hackage.haskell.org/package/coroutine-iteratee