[Haskell-cafe] Space leak in WAI 3.0 application

[Thomas Koster]

On Tue, Nov 18, 2014 at 4:59 PM, Thomas Koster <tkoster at gmail.com> wrote:
> -- | This version has a space leak.
> zerosAppA :: Application
> zerosAppA _req respond =
>   withZeros 100000000 $ \ largeLBS ->
>     respond $ responseStream status200 [] $ \ write _flush ->
>       write $ fromLazyByteString largeLBS
>
> -- | This version does not have a space leak.
> zerosAppB :: Application
> zerosAppB _req respond =
>  respond $ responseStream status200 [] $ \ write _flush ->
>    withZeros 100000000 $ \ largeLBS ->
>      write $ fromLazyByteString largeLBS

On 20 November 2014 16:59, Bardur Arantsson <spam at scientician.net> wrote:
> I would recommend avoiding lazy I/O altogether and using
> "responseStream" instead. This will let you decide exactly what to write
> and when.

I am already using responseStream. It's just that in my examples and
my real application, the 'octets' to be used for the response entity
("largeLBS" from my examples) are provided to the callback as a lazy
bytestring built from lazy I/O, until I get my head around pipes and
conduit.

>> Or will replacing the lazy bytestring with a streaming abstraction
>> like pipes or conduit make the problem go away in both versions?
>
> Not magically, but both of those can certainly be used to avoid the
> problem. You'll still have to arrange it so that you don't close over
> a huge (lazy) byte string.

OK, but could I use pipes or conduit (no lazy I/O) to remove the space
leak in version A without transforming it into version B (which
doesn't have the space leak anyway)?

That is, can I switch out the lazy bytestring in version A with a pipe
or conduit so that I can start streaming the zeros *outside* of
responseStream in order to determine the HTTP status code and headers
(which is only possible in version A) and continue to stream them
*inside* of responseStream for the rest of the response entity,
without incurring a space leak?

If the answer is "yes" then that's that and I will go and learn pipes
and/or conduit right away.

> If what you're trying to do is simple, then I'd just recommend writing
> your responses using "responseStream" and writing your "genereate
> output" function in terms of the "write" and "flush" callbacks you get
> access to write "responseStream". It's pretty easy when you get the hang
> of it.

...except that the HTTP status code and headers are arguments to
responseStream and therefore I cannot determine them from within the
callback passed as responseStream's third argument. This is why some
of my 'long stream' needs to be accessed before invoking
responseStream and why my real application looks like the leaky
version A.

Thanks,
--
Thomas Koster