[Haskell-beginners] Diagnosing : Large memory usage + low CPU

[Edward Z. Yang]

Hello Hugo,

Can you do a heap profile (+RTS -hT, or maybe use one of the other
options if you've got a profiling copy lying around)?  Try using
smaller data if it's taking too long; usually the profile will still
look the same, unless it's a particular type of input that is triggering
bad behavior.

There is not enough detail in your code for me to use my psychic
debugging skills, unfortunately.

Edward

Excerpts from Hugo Ferreira's message of Wed Nov 30 09:23:53 -0500 2011:
> Hello,
> 
> On 11/29/2011 10:57 PM, Stephen Tetley wrote:
> > Hi Hugo
> >
> > What is a POSTags and how big do you expect it to be?
> >
> >
> 
> type Token = String
> type Tag = String
> 
> type NGramTag = (Token, Tag, Tag)
> 
> type POSTags = Z.Zipper NGramTag
> 
> > Generally I'd recommend you first try to calculate the size of your
> > data rather than try to strictify things, see Johan Tibell's very
> > useful posts:
> >
> >
> > http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html
> > http://blog.johantibell.com/2011/06/computing-size-of-hashmap.html
> >
> 
> According to size in String I am expecting a maximum of 50 Mega.
> Profiling (after a painful 80 minutes) shows:
> 
> total alloc = 20,350,382,592 bytes
> 
> Way too much.
> 
> > Once you know the size of your data - you can decide if it is too big
> > to comfortably work with in memory. If it is too big you need to make
> > sure you're are streaming[*] it rather than forcing it into memory.
> >
> > If POSTags is large, I'd be very concerned about the top line of
> > updateState - reversing lists (or sorting them) simply doesn't play
> > well with streaming.
> >
> 
> The zipper does quite a bit of reversing and appending.
> I also need to reverse lists to retain the order of the
> characters (text). I also do sorting but I have eliminated this
> in the tests.
> 
> So my question: how can one "force" the reversing and append?
> Anyone?
> 
> TIA,
> Hugo F.
> 
> >
> > [*] Even in a lazy language like Haskell, streaming data isn't
> > necessarily automatic.
> >
> > _______________________________________________
> > Beginners mailing list
> > Beginners at haskell.org
> > http://www.haskell.org/mailman/listinfo/beginners
> >
>