[Haskell-cafe] Performance Tuning & darcs (a real shootout?)
dagit at eecs.oregonstate.edu
Mon Jan 23 23:37:55 EST 2006
On Jan 23, 2006, at 3:33 AM, Bulat Ziganshin wrote:
> what you mean? afaik, there is no standard FastPackedString
> implementation, but there is some library that with minimal
> modifications used in darcs, jhc and many other apps
I considered the version at Don Stewart's web site to be the
standard, perhaps that was silly of me.
> JD> still reports large amounts of memory used. Does ghc use mmap to
> JD> allocate memory instead of malloc?
> afaik, the answer is yes. look at ghc-6.4.1-src\ghc\rts\MBlock.c
I've been doing the bulk of my testing in linux on i386, and if I'm
reading the ifdefs correctly that is mmap'd anonymously. So at least
I would expect Ian's memory tool to report the memory allocations
correctly on that platform.
> JD> I'm looking for advice or help in optimizing darcs in this
> case. I
> JD> guess this could be viewed as a challenge for people that felt
> JD> the micro benchmarks of the shootout were unfair to Haskell.
> Can we
> JD> demonstrate that Haskell provides good performance in the real-
> JD> when working with large files? Ideally, darcs could easily
> work with
> JD> a patch that is 10GB in size using only a few megs of ram if
> need be
> JD> and doing so in about the time it takes read the file once or
> JD> and gzip it.
> it seems that Ian just used this as memory/time-efficient alternative
> for hGetContents. reading from memory-mapped file may be done as pure
> computation if the whole file is mapped. is this used in darcs?
I'm not sure, I have looked at the code but I can't tell. I think
that was the point with the mmap'd files. There are several layers
of abstraction at work here. Slurpies, PackedStrings, (custom) Lazy
Reader monad, and maybe some others.
> if not, then we can map small block at the time or even just use
> hGetBuf - at least, for first version. what you will say about this?
I'd expect it to be horribly slow, but if it gives guarantees about
memory usage then I bet we could tune it up and make it tolerable.
> i'll be happy to help, but i'm not ready to study the darcs code. i'm
> ready to solve more concrete problems - such as providing new
> input mechanism with given external interface or optimizing something
> rather small
That reminds me, you wrote FastIO right? I haven't looked at FastIO,
do you think it would apply here?
More information about the Haskell-Cafe