[Haskell-cafe] Conduit and pipelined protocol processing using a threadpool

[Nicolas Trangez]

On Wed, 2012-11-28 at 09:17 +0200, Michael Snoyman wrote:
> 
> 
> 
> On Tue, Nov 27, 2012 at 7:25 PM, Nicolas Trangez
> <nicolas at incubaid.com> wrote:
>         Michael,
>         
>         On Tue, 2012-11-27 at 17:14 +0200, Michael Snoyman wrote:
>         > I think the stm-conduit package[1] may be helpful for this
>         use case.
>         > Each time you get a new command, you can fork a thread and
>         give it the
>         > TBMChan to write to, and you can use sourceTBMChan to get a
>         source to
>         > send to the client.
>         
>         
>         That's +- what I had in mind. I did find stm-conduit before
>         and did try
>         to get the thing working using it, but these attempts failed.
>         
>         I attached an example which might clarify what I intend to do.
>         I'm aware
>         it contains several potential bugs (leaking threads etc), but
>         that's
>         beside the question ;-)
>         
>         If only I could figure out what to put on the 3 lines of
>         comment I left
>         in there...
>         
>         Thanks for your help,
>         
>         Nicolas
>         
> 
> 
> The issue is that you're trying to put everything into a single
> Conduit, which forces reading and writing to occur in a single thread
> of execution. Since you want your writing to be triggered by a
> separate event (data being available on the Chan), you're running into
> limitations.
> 
> 
> The reason network-conduit provides a Source for the incoming data and
> a Sink for outgoing data is specifically to address your use case. You
> want to take the data from the Source and put it into the Chan in one
> thread, and take the data from the other Chan and put it into the Sink
> in a separate thread. Something like:
> 
> 
> myApp appdata = do
>     chan1 <- ...
>     chan2 <- ...
>     replicateM_ 5 $ forkIO $ worker chan1 chan2
>     forkIO $ appSource appdata $$ sinkTBMChan chan1
>     sourceTBMChan chan2 $$ appSink appdata
> 
> 
> You'll also want to make sure to close chan1 and chan2 to make sure
> that your threads stop running.

Thanks, I +- figured that out last night. Only thing left is managing
the handshake before forking off the workers, if possible (otherwise
things become very messy IMHO), but ResumableSources etc might be of use
here.

Maybe there's a library somewhere in here...

Thanks a bunch,

Nicolas