[Haskell-beginners] Diagnosing : Large memory usage + low CPU

[Stephen Tetley]

Hi Hugo

What is a POSTags and how big do you expect it to be?


Generally I'd recommend you first try to calculate the size of your
data rather than try to strictify things, see Johan Tibell's very
useful posts:


http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html
http://blog.johantibell.com/2011/06/computing-size-of-hashmap.html

Once you know the size of your data - you can decide if it is too big
to comfortably work with in memory. If it is too big you need to make
sure you're are streaming[*] it rather than forcing it into memory.

If POSTags is large, I'd be very concerned about the top line of
updateState - reversing lists (or sorting them) simply doesn't play
well with streaming.


[*] Even in a lazy language like Haskell, streaming data isn't
necessarily automatic.