[Haskell-cafe] Space leak in WAI 3.0 application

[Thomas Koster]

On Tue, Nov 18, 2014 at 4:59 PM, Thomas Koster <tkoster at gmail.com> wrote:
> -- | This version has a space leak.
> zerosAppA :: Application
> zerosAppA _req respond =
>   withZeros 100000000 $ \ largeLBS ->
>     respond $ responseStream status200 [] $ \ write _flush ->
>       write $ fromLazyByteString largeLBS
>
> -- | This version does not have a space leak.
> zerosAppB :: Application
> zerosAppB _req respond =
>  respond $ responseStream status200 [] $ \ write _flush ->
>    withZeros 100000000 $ \ largeLBS ->
>      write $ fromLazyByteString largeLBS

On 20 November 2014 05:30, Gregory Collins <greg at gregorycollins.net> wrote:
>> Why does version A not process the LBS in constant space?
>
> The lazy bytestring is let-lifted out of the function so that subsequent
> calls reuse the same heap value.
>
>> What in version A is preventing the GC from collecting the LBS chunks
>> after they have been fed to Warp?
>
> The value is re-used (and the closure holds a reference) so the GC can't
> collect it.
>
>> What is it about version B that permits the LBS chunks to be collected?
>
> The allocation is performed underneath the lambda in version B and so you
> get a fresh copy every time.

Thanks for your reply.

My coding style makes heavy use of let to keep lines succinct so am I
going to have to be more careful about the order of lets in case I
accidentally introduce unwanted sharing?

I was able to re-arrange the leaky version A into version B in my
example, but I may not be able to do this in my real application. What
if I need the first few bytes of "largeLBS" to determine the response
status code or headers? This is only possible in version A.

Or will replacing the lazy bytestring with a streaming abstraction
like pipes or conduit make the problem go away in both versions?

Thanks,
--
Thomas Koster