<div dir="ltr"><div>I tried to tackle this problem in my "streaming-bracketed" library <a href="http://hackage.haskell.org/package/streaming-bracketed">http://hackage.haskell.org/package/streaming-bracketed</a>, by using a "decorator" that wraps regular streams, as opposed to having a resource-aware base monad.</div><div><br></div><div>When lifting take-like functions to the decorator, new deallocation actions are inserted.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 2:53 PM <<a href="mailto:haskell-cafe-request@haskell.org">haskell-cafe-request@haskell.org</a>> wrote:<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm using the 'streaming' library and realized it doesn't close files in a<br>
timely way for the way I'm using it, and in fact can't, due to how the library<br>
works. I know conduit has put a lot of thought into closing resources in a<br>
timely way, so I did an experiment to see what it does, but as far as I can<br>
tell, conduit has the same problem. Maybe I'm doing it wrong?<br>
<br>
The situation is that I'm opening multiple files and mixing their output.<br>
I want to close the inputs as soon as I'm done with them. But I can get<br>
done with them earlier than the end of the file. Since all of these libraries<br>
are based on pulling from downstream, if you don't pull all the way to the end,<br>
the close at the end doesn't happen, and has to wait until runResourceT<br>
returns, which is too late. I remember long ago reading Oleg's original<br>
iteratee paper, and it seems like he called out this problem with pull-based<br>
iterators, that the iterator doesn't know when its caller is done with it, so<br>
it can't close files on time.<br>
<br>
Here's a conduit version that I think illustrates the situation:<br>
<br>
import qualified Conduit as C<br>
import Conduit ((.|))<br>
import qualified Control.Monad.Trans as Trans<br>
import qualified System.IO as IO<br>
<br>
main :: IO ()<br>
main = C.runResourceT $ C.runConduit pipe<br>
<br>
pipe :: C.ConduitM a c (C.ResourceT IO) ()<br>
pipe = fileLines "TODO" .| (C.takeC 3 >> C.yield "***")<br>
.| C.mapM_C (Trans.liftIO . putStrLn)<br>
<br>
fileLines :: C.MonadResource m => FilePath -> C.ConduitT i String m ()<br>
fileLines fname = C.bracketP (IO.openFile fname IO.ReadMode) close<br>
handleLines<br>
<br>
handleLines :: Trans.MonadIO m => IO.Handle -> C.ConduitT i String m ()<br>
handleLines hdl = loop<br>
where<br>
loop = do<br>
eof <- Trans.liftIO $ IO.hIsEOF hdl<br>
if eof then return () else do<br>
line <- Trans.liftIO $ IO.hGetLine hdl<br>
C.yield line<br>
loop<br>
<br>
close :: IO.Handle -> IO ()<br>
close hdl = IO.hClose hdl >> putStrLn "=== close"<br>
<br>
This prints the first three lines of TOOD, then ***, and then "=== close",<br>
where the close should go before the ***s.<br>
<br>
As far as I can see, conduit can't do this any more than 'streaming' can,<br>
because 'C.takeC' is just some awaits and yields, with no indication that<br>
the final await is more special than any other await.<br>
<br>
I think what would be necessary to solve this is that combinators like 'take'<br>
have to be able to tell the stream to close, and that has to propagate back up<br>
to each producer that has registered a cleanup. Of course this renders the<br>
stream invalid, so it's not safe to have any other references to the stream<br>
around, but I guess streams are stateful in general so that's always true.<br>
Maybe I could accumulate the finalizers in the stream data type and have<br>
combinators like 'take' call it as soon as they've taken their last. What<br>
I actually wound up doing was make a 'takeClose' that also takes a 'close'<br>
action to run when its done with the stream. It's not exactly general but I'm<br>
not writing a library so I don't need general.<br>
<br>
Is there some kind of standard or built-in solution for this situation?<br>
I know others have given a lot more thought to streaming than I have,<br>
so surely this issue has come up.<br>
<br>
I know there is a lot of talk about "prompt finalisation" and streams vs.<br>
pipes vs. conduit, and talk about brackets and whatnot, but despite reading<br>
various documents (<a href="https://hackage.haskell.org/package/streaming-with" rel="noreferrer" target="_blank">https://hackage.haskell.org/package/streaming-with</a>,<br>
<a href="http://www.haskellforall.com/2013/01/pipes-safe-10-resource-management-and.html" rel="noreferrer" target="_blank">http://www.haskellforall.com/2013/01/pipes-safe-10-resource-management-and.html</a>,<br>
etc.) I still don't really understand what they're talking about.<br>
It seems like they're really about reliable cleanup when there are<br>
exceptions, not really about prompt cleanup. Certainly pipes-safe doesn't do<br>
prompt cleanup, at least not the kind I'm talking about.<br>
<br><br>
</blockquote></div></div>