ghc releasing memory during compilation

[Bulat Ziganshin]

Hello John,

Thursday, March 16, 2006, 4:00:34 AM, you wrote:

JM> the best solution would be for ghc's garbage collector to periodically
JM> call getrusage(2) and look at the number of page faults and swaps it is
JM> causing and switch to compaction when it sees a jump in that.

just some more thoughts about this:

1. this can be implemented at first time just as user-level routines that
controls ghc's internal variables which controls GC. Simon one time
said how these vars can be used. after thorough testing and may be
even comparison of different strategies this can go into the official
RTS. one my idea was even allowing user-defined GC strategies by
evaluation user-specified routines before each minor gc. i think that
user-defined gc strategies may be a big win both for further
developments in this area and for implementing custom strategies for
large applications. making some simple official that allows to control
these variables, perform minor gc and call some routine before each gc
should be helpful in further development of new algorithms of optimal
GCs usage

2. we can develop even better strategies by combining the 4 types of
major GCs:

1) no major GCs at all
2) copying
3) copying with with returning memory to OS
4) compacting

and switching between these 4 strategies according to external
conditions. when there are enough memory, we can just don't perform
major GCs (i know that this is debatable, but typical way to make
programs faster is to use large -A or -H values). when heap size is
close to size of available physical ram, we should perform at least
copying GC. and when page faults grows, we should use only compacting
algorithm

3.
JM> the best solution would be for ghc's garbage collector to periodically
JM> call getrusage(2) and look at the number of page faults and swaps it is
JM> causing and switch to compaction when it sees a jump in that.

and moreover, we should perform compaction immediately when swapping grows.
imagine for example algorithm that had 80 mb residency and runs on a
machine with 100 mb free. performing compacting GC each time when
memory usage grows to 100 mb should be better strategy than waiting
for 160 mb usage. of course, concrete calculation whether to start GC
or not, is a delicate task, but at least it should be performed on
each minor GC

4. i still don't understand - wgen GHC performs copying GC, whether it
copies data from old pages sequential or not? if it releases old pages
sequentially, we can run copying GC using the same amount of physical ram
as required for compaction gc


-- 
Best regards,
 Bulat                            mailto:Bulat.Ziganshin at gmail.com