Building on limited memory environments (or does -M really work?)

[Ben Gamari]

Saurabh Nanda <saurabhnanda at gmail.com> writes:

>>
>> Did you ever make any progress on this, Saurabh?
>>
>
> We made progress in some sense, by introducing a separate `stack build -j1`
> step in our CI pipeline for compiling packages that are known to use a lot
> of memory.
>
>
>>
>>  * -j just tells GHC to parallelise compilation across modules. This can
>>     increase the maximum heap size needed by the compiler.
>>
>
>
> From the docs, it wasn't very clear to me how -j interacts with -M when
> both the options are passed to the GHC process. Is it the max heap size
> across all build, or per build?
>
The short answer is that they don't interact: -j is a GHC flag whereas
-M is an RTS flag. -M controls the amount of heap that the RTS allows
the mutator (GHC in this case) to allocate. This includes all
threads. GHC when run with -j is just like any other threaded Haskell
program.

>>  * -M is a bit tricky to define. For one, it defines the maximum heap
>>    size beyond which we will terminate. However, we also use it in
>>    garbage collector to make various decisions about GC scheduling. I'll
>>    admit that I'm not terribly familiar with the details here.
>>
>> Note that -M does not guarantee that GHC will find a way to keep your
>> program under the limit that you provide. It merely ensures that the
>> program doesn't exceed the given size, aborting if necessary.
>>
>
>
> Quoting from
> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M
> :
>
> *> The maximum heap size also affects other garbage collection parameters:
> when the amount of live data in the heap exceeds a certain fraction of the
> maximum heap size, compacting collection will be automatically enabled for
> the oldest generation, and the -F parameter will be reduced in order to
> avoid exceeding the maximum heap size.*
>
> It just makes it sound that the RTS is going to tweak the GC algo, and the
> number of time GC is run, to avoid crossing the heap limit. However, I've
> found the GHC process easily consuming more memory than what is specified
> in the -M flag (as reported by top).
>
Yes, as I mentioned we do tweak some things in the GC; however, these
tweaks are really a best-effort attempt to avoid going over the limit.
It's entirely possible that your mutator will be terminated if it wants
to use significantly more than the limit set with -M. There is
relatively little else the RTS can do in this case that wouldn't require
explicit cooperation from the mutator to keep working sets down.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20171108/1b1abeca/attachment.sig>