[Haskell-cafe] Re: Mining Twitter data in Haskell and Clojure
ketil at malde.org
Tue Jun 15 02:50:04 EDT 2010
braver <deliverable at gmail.com> writes:
> In fact, the tag cafe2, when run on the full dataset, gets stuck at 11
> days, with RAM slowly getting into 50 GB
One tip might be to limit available heap memory by using +RTS -M2G (or
whatever your real memory is). If (as seems likely) the RAM usage leads to
thrashing (the symptoms being 'top' showing substantially less than 100%
CPU usage, and a less responsive system), limiting heap will cause your
program to fail faster, which is always an advantage when debugging.
Unless you actually expect the working set to be fifty gig, that is.
If I haven't seen further, it is by standing in the footprints of giants
More information about the Haskell-Cafe