[Haskell-cafe] Performance Tuning & darcs (a real shootout?)

[Jason Dagit]

On Jan 23, 2006, at 3:33 AM, Bulat Ziganshin wrote:

> what you mean? afaik, there is no standard FastPackedString
> implementation, but there is some library that with minimal
> modifications used in darcs, jhc and many other apps
>

I considered the version at Don Stewart's web site to be the  
standard, perhaps that was silly of me.

> JD> still reports large amounts of memory used.  Does ghc use mmap to
> JD> allocate memory instead of malloc?
>
> afaik, the answer is yes. look at ghc-6.4.1-src\ghc\rts\MBlock.c

I've been doing the bulk of my testing in linux on i386, and if I'm  
reading the ifdefs correctly that is mmap'd anonymously.  So at least  
I would expect Ian's memory tool to report the memory allocations  
correctly on that platform.

>
> JD> I'm looking for advice or help in optimizing darcs in this  
> case.  I
> JD> guess this could be viewed as a challenge for people that felt  
> like
> JD> the micro benchmarks of the shootout were unfair to Haskell.   
> Can we
> JD> demonstrate that Haskell provides good performance in the real- 
> world
> JD> when working with large files?  Ideally, darcs could easily  
> work with
> JD> a patch that is 10GB in size using only a few megs of ram if  
> need be
> JD> and doing so in about the time it takes read the file once or  
> twice
> JD> and gzip it.
>
> it seems that Ian just used this as memory/time-efficient alternative
> for hGetContents. reading from memory-mapped file may be done as pure
> computation if the whole file is mapped. is this used in darcs?

I'm not sure, I have looked at the code but I can't tell.  I think  
that was the point with the mmap'd files.  There are several layers  
of abstraction at work here.  Slurpies, PackedStrings, (custom) Lazy  
Reader monad, and maybe some others.

>
> if not, then we can map small block at the time or even just use
> hGetBuf - at least, for first version. what you will say about this?

I'd expect it to be horribly slow, but if it gives guarantees about  
memory usage then I bet we could tune it up and make it tolerable.

> i'll be happy to help, but i'm not ready to study the darcs code. i'm
> ready to solve more concrete problems - such as providing new
> input mechanism with given external interface or optimizing something
> rather small

That reminds me, you wrote FastIO right?  I haven't looked at FastIO,  
do you think it would apply here?

Thanks,
Jason