Issues with the large address space allocator and HPC systems with resource limits

[Ben Gamari]

Luis Pedro Coelho <luispedro at big-data-biology.org> writes:

> Dear GHC devs,
>
> I hope this is the right forum to bring this up.
>
Absolutely, thanks for bringing this up.

> I am the lead developer of NGLess
> (https://github.com/ngless-toolkit/ngless a bioinformatics tool,
> written in Haskell). Several users have complained about not being
> able to easily use NGLess in an academic cluster environment due to
> the fact that it allocates 1TB of address space (e.g.,
> https://groups.google.com/forum/#!topic/ngless/9su2E0EdeCc and I have
> also gotten several private emails on this issue).
>
> In particular, many systems are set up with a limit on the address
> space so that if the job allocates more than the given limit, it is
> immediately killed.
>
> This appears to be the default way to set up SGE, the most widely used
> batch system. Users are dependent on their sysadmins and lack the
> permissions to change these settings easily (and may not always be
> cognizant of the difference between "allocating address space" and
> "allocating memory"). Using ulimit seem to make the issue disappear on
> most, but not all, user setups.
>
Indeed as someone who has had to run on a shared cluster in the past, I
can sympathize. These sorts of restrictions are common and quite
annoying.

Are these address space limits advertised via getrlimit(2)? If so, have
you tried GHC 8.6.1-alpha1? While fixing #14492 I taught GHC to respect
rlimits when allocating its heap, so this might work now.

> I have now built NGLess with a version of GHC that was compiled
> without the large address allocator (using ./configure
> --disable-large-address-space). At least locally, this seems to run
> correctly and solve the issue.
>
> I assume that there are performance or other reasons to use the large
> address space allocator as the default, but, right now, for the
> problem space I am working in, disabling it seems to be a better
> trade-off. In principle, the RTS that is used for GHC and the one that
> is used for the programme being linked do not need to be the same. Is
> there any possibility of making this choice when a programme is linked
> and not when GHC is compiled?
>
Indeed you will take a bit of performance hit by using the one-step
allocator since the check of whether an object resides in the heap
(which is very hot during GC) is a fair bit more complex.

As far as I know the choice of allocator has no effect on code
generation so in principle it should be possible to link the same code
against either RTS. However, the build system looks to be built around
the assumption that the choice is made at configure-time.

I'm sure this could be fixed, but it's not immediately obvious how. I
suppose you could make the two allocators different RTS ways (e.g. like
the distinction between event-logged, debugging, profiled and vanilla
RTSs), but that would double the already large number of ways.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20180704/f924b46e/attachment.sig>