[Haskell-cafe] Conduit and pipelined protocol processing using a threadpool

[Michael Snoyman]

On Tue, Nov 27, 2012 at 7:25 PM, Nicolas Trangez <nicolas at incubaid.com>wrote:

> Michael,
>
> On Tue, 2012-11-27 at 17:14 +0200, Michael Snoyman wrote:
> > I think the stm-conduit package[1] may be helpful for this use case.
> > Each time you get a new command, you can fork a thread and give it the
> > TBMChan to write to, and you can use sourceTBMChan to get a source to
> > send to the client.
>
> That's +- what I had in mind. I did find stm-conduit before and did try
> to get the thing working using it, but these attempts failed.
>
> I attached an example which might clarify what I intend to do. I'm aware
> it contains several potential bugs (leaking threads etc), but that's
> beside the question ;-)
>
> If only I could figure out what to put on the 3 lines of comment I left
> in there...
>
> Thanks for your help,
>
> Nicolas
>
>
The issue is that you're trying to put everything into a single Conduit,
which forces reading and writing to occur in a single thread of execution.
Since you want your writing to be triggered by a separate event (data being
available on the Chan), you're running into limitations.

The reason network-conduit provides a Source for the incoming data and a
Sink for outgoing data is specifically to address your use case. You want
to take the data from the Source and put it into the Chan in one thread,
and take the data from the other Chan and put it into the Sink in a
separate thread. Something like:

myApp appdata = do
    chan1 <- ...
    chan2 <- ...
    replicateM_ 5 $ forkIO $ worker chan1 chan2
    forkIO $ appSource appdata $$ sinkTBMChan chan1
    sourceTBMChan chan2 $$ appSink appdata

You'll also want to make sure to close chan1 and chan2 to make sure that
your threads stop running.

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20121128/993eb2ba/attachment.htm>