[Haskell-beginners] Flying Dutchman sailing blind

[Jeroen van Maanen]

A few weeks ago I was greatly helped by members of this list to expose an error that was hiding in my, by now quite extensive, Haskell program. I should still thank Daniel Fisher for his lucid explanation of how an overdose of strictness can prevent a try-expression from wrapping an exception.

Now that the error is gone, I face a new challenge: when I let my program run for a while (it is a computationally intensive task that never stops by design), after fifteen minutes or so the CPU usage drops from nearly 100% to around 15% and a few minutes later the process dies with the message:

  "lexau: memory allocation failed (requested 2097152 bytes)"

The whole thing is a tail biting snail pit of threads that are communicating through MVars. Every thread runs a tail recursive function. (I read in another post that it is not a good idea to use explicit recursion, but when I compared alternatives the fold variants produced even worse stack/heap problems.) The thread that is likely to cause the problem is an optimizer that tries many possible improvements of a complex data structure and incrementally applies the successful ones. I use a strict foldl style in an attempt to limit the memory used by the optimizer. Frankly, however, I have no idea what eats up so much heap space.

Now I have always proudly labeled myself a 'printf programmer', but I am afraid that I am going to need some profiling tool to determine where the problem is. Any suggestions where I should start?

Cheers,  Jeroen

P.S. For an idea of what is living in the snail pit, have a look at:

  http://lexau.svn.sourceforge.net/viewvc/lexau/branches/totem/src/LExAu/Pipeline/Concurrent.hs?view=markup
  http://lexau.svn.sourceforge.net/viewvc/lexau/branches/totem/src/LExAu/Model/HistoryTree.hs?view=markup

(I know, HistoryTree.hs badly needs to be split up into smaller modules.)