[Haskell-cafe] How can I improve the pipes's performance with a huge file?

[Tom Ellis]

On Fri, Nov 14, 2014 at 09:14:17PM +0800, zhangjun.julian wrote:
> Dear Tom
> 
> I change Map to Strict，it be little fast when test with 18M rows, but it hanged again with 40M rows.
> 
> Do you have any other advice?

Dear Zhangjun Julian,

Perhaps too much of the output string is being kept around when it is
printed.  I would try

    mapM_  (\((x,y), i) -> hPutStrLn writeHadle (show x ++ "," ++ show y ++ "," ++ show i))
           (DM.toList rt)

instead of

    hPutStr writeHadle $ mapToString  rt

Apart from that, I don't have any other ideas.  Can you determine whether
the large memory usage comes from the Pipe or from the printing of the
result?

Tom