[Haskell-cafe] conduit/pipes/streaming and prompt cleanup

[Evan Laforge]

Hi, thanks for the response!

> I think you're seeing the fact that conduit no longer has finalizers, see:
>
> https://www.snoyman.com/blog/2018/01/drop-conduits-finalizers
>
> Though I don't think your example proves the point. You know that the code
> following `takeC 3` will never `await`, but there's no way for the conduit
> library to know that. Instead, if you wanted to demonstrate the limitation of
> removing finalizers you'd need to rewrite your code to this:

The way I was looking at it, the problem is just that: takeC will never await
again, but doesn't tell anyone about that, so conduit can't know it.  This is
why I was thinking of giving it such a way, by notifying upstream explicitly.
I can see how that's an unsatisfying solution, because it means that all
functions that cut off the input have to say so explicitly, and forgetting to
do that in any one of them will lead to a leak.  The bracket pattern encodes
the scope of the open file into the shape of the expression, and if I can fit
my program into that shape then I get automatic finalization, which is nicer.

The question then is can I fit it into that shape?  If not then it's at least a
data point that you were asking for on the blog post.  If I can, but only for
conduit and not for streaming, then it's a demonstration of a difference of
expressiveness between the two, which is interesting.

The complicating factor is that I'm streaming multiple overlapping files, which
are all demanded by a single sink.  So it's not the concatenation that (>>)
does.  E.g.

mainLoop (events :: [(Int, FilePath)]) = loop 0 [] events
    where
    loop now running events = do
        let (toStart, notYet) = span ((<=now) . fst) events
        starting <- mapM startStream toStart
        chunks <- mapMaybeM await (running ++ starting)
        yield (mix chunks)
        loop (now+n) (running ++ starting) notYet
    startStream = takeC 3 . fileLines

There's some missing stuff to drop streams that ran out, and stop the loop once
events and streams run out, but you get the idea.  I can't figure out where the
(>>) would go that would terminate one of those streams, or even what
"terminate" actually means, if it's not something that takeC does explicitly.