[Haskell-cafe] How can I improve the pipes's performance with a huge file?

Tom Ellis tom-lists-haskell-cafe-2013 at jaguarpaw.co.uk
Fri Nov 14 13:51:20 UTC 2014

On Fri, Nov 14, 2014 at 09:14:17PM +0800, zhangjun.julian wrote:
> Dear Tom
> I change Map to Strict,it be little fast when test with 18M rows, but it hanged again with 40M rows.
> Do you have any other advice?

Dear Zhangjun Julian,

Perhaps too much of the output string is being kept around when it is
printed.  I would try

    mapM_  (\((x,y), i) -> hPutStrLn writeHadle (show x ++ "," ++ show y ++ "," ++ show i))
           (DM.toList rt)

instead of

    hPutStr writeHadle $ mapToString  rt

Apart from that, I don't have any other ideas.  Can you determine whether
the large memory usage comes from the Pipe or from the printing of the


More information about the Haskell-Cafe mailing list