[Haskell-beginners] HTTP Download -> Save File - non-strict

[Magnus Therning]

On Sat, Aug 10, 2013 at 05:16:58PM -0700, Dan Krol wrote:
> Hi,
> 
> I'm working on an rss file getter. I was wondering if I could get
> some help getting files to download and save without holding the
> entire file in memory in between. I chose Conduit's version of
> SimpleHttp only because it was recommended, and it was the quickest
> thing I could get to work correctly because I was eager to get
> started on this project, so I'd be happy to switch.
> 
> Here's where I define the download and save functions:
> 
> https://github.com/orblivion/feedGetter/blob/master/rss.hs#L107
> 
> And here's where I use them, getting multiple at a time with async:
> 
> https://github.com/orblivion/feedGetter/blob/master/rss.hs#L208
> 
> What happens when I run this is that it outputs that it's "Getting"
> the file, waits a while (presumably to download the whole thing),
> then says it's "Saving". And I checked the file system, it's not
> there during the pause. I'm not entirely sure why. Is it my choice
> of libraries, or the way I'm using them? Perhaps something to do
> with async? I just tried content <- simpleHttp "http://google.com"
> in ghci, and it does pause for a second, so I'm guessing this is
> strict from the getgo. But I've done almost no I/O before.
> 
> Is there a straightforward, canonical option? It seems like there
> perhaps should be. But if it comes down to using pipes or conduit,
> what the heck I'll try it out, I'd like to learn pipes eventually.

Michael is very good with documenting his packages, this is what I
found in the docs for http-conduit (http://is.gd/WkDb7G):

  Note: Even though this function returns a lazy bytestring, it does
  not utilize lazy I/O, and therefore the entire response body will
  live in memory. If you want constant memory usage, you'll need to
  use the conduit package and http directly.

/M

-- 
Magnus Therning                      OpenPGP: 0xAB4DFBA4 
email: magnus at therning.org   jabber: magnus at therning.org
twitter: magthe               http://therning.org/magnus

I invented the term Object-Oriented, and I can tell you I did not have
C++ in mind.
     -- Alan Kay
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 230 bytes
Desc: not available
URL: <http://www.haskell.org/pipermail/beginners/attachments/20130811/25aaf61e/attachment.pgp>