cabal-install: Conduits/IO concurrency

[Ben Millwood]

I'm not at all an authority on this, but one thing to keep in mind is
that it's probably a good idea to keep the transitive package
dependency list of cabal-install as small as possible, otherwise it's
a nuisance to install. In particular, cabal-install is in the Haskell
Platform, so any packages it depends on would have to be distributed
with the HP as well. I doubt the conduit package is stable enough yet
to manage that.

You could possibly introduce some degree of concurrency into the use
of HTTP via the stream hooks interface:
http://hackage.haskell.org/packages/archive/HTTP/4000.2.3/doc/html/Network-TCP.html#t:StreamHooks

On Sun, Apr 15, 2012 at 12:43 AM, Eyal Lotem <eyal.lotem at gmail.com> wrote:
> Hey,
>
> Currently it seems that cabal-install lacks IO concurrency in its various
> operations (i.e: download packages concurrently to their compilation).
>
> Specifically, I found it to be a major bottleneck during "cabal update"
> operations. The download, decompression, and use of the decompressed index
> are all serially executed, and all take quite a bit of time, and could
> probably be executed in parallel easily.
>
> To make "cabal update" execute the download and decompression concurrently,
> it would be nice to use a streaming decompressor on a stream download.
>
> The approach I had in mind was replacing the use Network.HTTP and zlib with
> use of the conduits counterparts.
>
> Additionally, use of conduits can probably ease addition of status reports
> (X% completed) by injecting progress reporters in the conduit pipeline.
> "cabal update" progress could be less opaque that way.
>
>
> My question is: Would patches that add conduit use and demonstrate
> performance benefits be welcome?
>
> I don't want to embark on this adventure if there's a good reason to avoid
> it in the first place.
>
> Eyal
>
>
> _______________________________________________
> Libraries mailing list
> Libraries at haskell.org
> http://www.haskell.org/mailman/listinfo/libraries
>