[GHC] #14492: Tiered memory allocation restricts available memory

[GHC]

#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
           Reporter:  unode          |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Runtime        |           Version:  8.0.2
  System                             |
           Keywords:  memory ulimit  |  Operating System:  Linux
       Architecture:  x86_64         |   Type of failure:  Poor/confusing
  (amd64)                            |  error message
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 This was first noticed when running a binary produced with GHC 8.0.2
 through a queueing system (Sun Grid Engine) that limits available memory
 by setting `ulimit -v`.

 When running with **unlimited** address space, the binary reserves 1TB of
 RAM and everything works as expected.
 When running with **limited** address space, reservation happens in powers
 of 2 leading to significant unused memory at higher memory values.

 The tests are summarized in the following table:

 {{{
     Limit   VIRT    Unused  (all in GB)
     8       4       8
     10      8       2
     16      8       8
     32      16      16
     34      32      2
     64      32      32
     66      64      2
     128     64      64
     130     128     2
     256     128     128
     258     256     2
     512     256     256
     514     512     2
     1024    512     512
 }}}

 In the last line, setting `ulimit -v` to 1024GB (1073741824) causes the
 binary to only be able to reserve 512GB.
 If at runtime the software needs more than 512GB it will abort with "out
 of memory" instead of expanding to as close to the specified ulimit as
 possible.

 PS: Is there any way to make the "out of memory" error a little bit more
 informative? Something that informs the user of how much memory was
 already being used and how much more was going to be reserved when the
 failure was seen?

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14492>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler