[Haskell-cafe] Mining Twitter data in Haskell and Clojure

[Ron de Bruijn]

Op 14-06-10 07:00, braver schreef:
> I'm computing a communication graph from Twitter data and then scan it
> daily to allocate social capital to nodes behaving  in a good karmic
> manner.  The graph is culled from 100 million tweets and has about 3
> million nodes. First I wrote the simulation of the 35 days of data in
> Clojure and then translated it into Haskell with great help from the
> glorious #haskell folks.  I had to add -A5G -K5G to make it work.  It
> does 10 days OK hovering at 57 GB of RAM; I include profiling of that
> in sc10days.prof.
>
> At first the Haskell executable goes faster than Clojure, not by an
> order of magnitude, but by 2-3 times per day simulated.  (Clojure also
> fits well in its 32 GB JVM with compressed references.)  However,
> Haskell gets stuck after a while, and for good.  Clearly I'm not doing
> Haskell optimally here, and would appreciate optimization advice.
> Here's the code:
>
> http://github.com/alexy/husky
>
> The data and problem description is in
>
> http://github.com/alexy/husky/blob/master/Haskell-vs-Clojure-Twitter.md
>
> -- also referred from the main README.md.
>
> The main is in sc.hs, and the algorithm is in SocRun.hs.  The original
> Clojure is in socrun.clj.  This is a continuation of active Twitter
> research and the results will be published, and I'd really like to
> make Haskell work at this scale and beyond!  The seq's sprinkled
> already did no good.  I ran under ghc 6.10 with -O2 with or without -
> fvia-C, with no difference in stallling, and am working to bring 6.12
> to bear now.
>
> Cheers,
> Alexy
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
If you just want to optimize it and not compare exactly equal idiomatic code, 
you should stop using functional data structures and use a structure that fits 
your problem (the ST monad has been designed for that in Haskell), because 
compilers do not detect single-threaded usage and rewrite all your code to 
something mutable and thereby avoid O(log n) costs.

In practice it is probably easier to write the whole thing against the parallel 
Boost Graph Library (a C++ library), since that library provides the 
abstractions that you would want. If you go this path, it will probably end up 
being 10-100 times faster.

-- 

Best Regards,
Ron de Bruijn,