[Haskell-cafe] ANNOUNCE: pipes-core 0.0.1

[Mario Blažević]

On 12-03-10 05:16 AM, Paolo Capriotti wrote:
> On Sat, Mar 10, 2012 at 4:21 AM, Mario Blažević<blamario at acanac.net>  wrote:
>>     I like your design, it seems to strike a good balance between elegance
>> and practicality. The only thing missing for the latter is a deeper support
>> for chunking. Of course, that would probably destroy some of the elegance
>> [1]. I don't think that problem has been solved in any of the
>> enumerator/iteratee/pipe/wire/conduit libraries so far.
> Chunking is supported but not by primitive constructs. The way you
> implement chunked streams is to simply use some form of "container"
> representing a chunk as your input/output type.
>
> Of course, that means that the abstraction is now operating at the
> level of chunks instead of elements, which may be inconvenient, but I
> doubt that there exists a way to "lift" element operations to chunks
> in an efficient and general way.
>
> Another issue is how to deal with unconsumed input. For that, there is
> a ChunkPipe type (in pipes-extra) with a specialized monad instance
> that threads unconsumed input along. You can see an example of
> ChunkPipe in action in this prototype http server by Jeremy Shaw:
> http://src.seereason.com/pipes-http-2/pipes-http-2/. Note that this is
> based on a old version of pipes-core, however.

     The only sane way I've found to deal with chunks is to move the 
responsibility into the glue logic, which would be your (>+>) and (>>) 
combinators. The upstream argument of (>+>) could then produce chunks of 
any size it finds suitable, while the downstream argument would specify 
exactly how much input it needs without having to worry about the 
upstream chunk boundaries. How these requests are phrased is an open 
question. I've developed the incremental-parser package specifically for 
this purpose. Other approaches are possible, but I'm convinced that 
chunking should not be left to individual components. The chunk type 
probably shouldn't be reflected even in their types.


>>     Did you consider adding some stream-splitting and merging pipes, like
>> those in the SCC package [2] or those described in the last Monad.Reader
>> issue [3]? Your arrow-like combinators seem well thought out, but they
>> don't go very far.
> I'm not sure why you say that they don't go very far. I looked at
> Splitter and Join in Monad.Reader 19, and they seem equivalent to
> 'splitP' and 'joinP' in pipes-core.


     The main purpose of the Splitter type is to act as a conditional, 
sending each input item *either* into the left or into the right output, 
marking is as either true or false. Your splitterP sends each item to 
*both* left and right output. It's a tee, not a split.

     If your point is that the Splitter a m r type is isomorphic to Pipe 
a (Either a a) m r, that is true. There is a benefit to the abstraction, 
though. Once you introduce chunking into the picture, however, the 
Splitter type can be changed under the hood to send an entire chunk to 
its left or right output. The corresponding efficient chunked Pipe type 
would be Pipe [a] (Either [a] [a]) m r, which is not at all the same as 
Pipe [a] [Either a a] m r -- if you have to pack every single item of 
the input chunk into an Either value, you've lost all performance 
benefits of chunking. The former type is efficient but I'm not sure if 
it would allow you to abstract the chunking logic out of the individual 
components.

> There shouldn't be any problem implementing all the other combinators
> there in terms of monoidal primitives (e.g. 'not' is just 'swap').

     I agree, with the chunking reservation above. Anyway, consider 
adding the Boolean combinators to the library. I find them quite intuitive.