[Haskell-cafe] How can I improve the pipes's performance with a huge file?

KC kc1956 at gmail.com
Fri Nov 14 16:01:25 UTC 2014


Are you compiling?

Just recently I had someone complain that Haskell wouldn't handle large
files but he was using the interpreter.

After compiling his problem vanished.

--
--

Sent from an expensive device which will be obsolete in a few months! :D

Casey

On Nov 14, 2014 5:54 AM, "Tom Ellis" <
tom-lists-haskell-cafe-2013 at jaguarpaw.co.uk> wrote:

> On Fri, Nov 14, 2014 at 09:14:17PM +0800, zhangjun.julian wrote:
> > Dear Tom
> >
> > I change Map to Strict,it be little fast when test with 18M rows, but it
> hanged again with 40M rows.
> >
> > Do you have any other advice?
>
> Dear Zhangjun Julian,
>
> Perhaps too much of the output string is being kept around when it is
> printed.  I would try
>
>     mapM_  (\((x,y), i) -> hPutStrLn writeHadle (show x ++ "," ++ show y
> ++ "," ++ show i))
>            (DM.toList rt)
>
> instead of
>
>     hPutStr writeHadle $ mapToString  rt
>
> Apart from that, I don't have any other ideas.  Can you determine whether
> the large memory usage comes from the Pipe or from the printing of the
> result?
>
> Tom
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20141114/c0624d4e/attachment.html>


More information about the Haskell-Cafe mailing list