[Haskell-cafe] 1G strings in Haskell

[Donald Bruce Stewart]

dagit:
> On 4/19/06, Donald Bruce Stewart <dons at cse.unsw.edu.au> wrote:
> > Question:
> >     Can I manipulate 1G strings in Haskell?
> >
> >
> > Failed due to memory exhaustion.
> > Almost made it though, just need a tad more ram than I had.
> >
> >     filter          !
> >     unlines         !
> >     unwords         !
> >     reverse         !           -- copy
> >     cons            !           -- copy
> >     snoc            !           -- involves a copy
> >     ++              !           -- can't concat two 1G strings on this box
> 
> I would say given the nature of the experiment that you want to feed
> these functions with values that lead to 1 GB results.  At least, that
> makes sense in the case of ++, but maybe not the others.

Yes, good point. Previous experiments with 500M strings indicated that
these all worked fine:

Size of test data: 531416k

Fine:
    filter          7.264   
    unwords         1.715
    reverse         1.980
    cons            0.976   
    snoc            0.689  
    ++              2.845   

Still wouldn't work at 0.5G
    unlines         !
    concat          !
  
> Did the machine have any available swap?  If so, why was it not used? 

Swap was used, 1G more, but ulimits kicked in a few times (and I'm not
an admin on the box). This was a bit frustrating, but the 0.5G results
show what is possible.

> The strings had to be in memory all at once to not skew the benchmark
> by including I/O time?

The strings were all in memory, yes, with some deepSeq tricks.

-- Don