[GHC] #14492: Tiered memory allocation restricts available memory

[GHC]

#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
        Reporter:  unode             |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Runtime System    |              Version:  8.0.2
      Resolution:                    |             Keywords:  memory ulimit
Operating System:  Linux             |         Architecture:  x86_64
 Type of failure:  Poor/confusing    |  (amd64)
  error message                      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by luispedro):

 The current behaviour is to start at 1TB and divide by 2 until success (or
 failure if it cannot allocate at least 1MB):

 https://github.com/ghc/ghc/blob/25f36bd7ba6899be6c720292528c56bc35e0f089/rts/posix/OSMem.c#L517


 If the memory available is in between two multiples of 2, then this can
 lead to a lot of unused memory. A (not so implausible, I think our systems
 might run something similar) worst-case scenario is running on a system
 with 1T of RAM, but where the sysadmin has wisely chosen to reserve 1GB
 for a monitoring process. Now, a whole 511GB of RAM are wasted!

 Btw, that code mentions https://ghc.haskell.org/trac/ghc/ticket/10877 with
 a previous discussion on this issue.

 A simple solution is to replace `*len /= 2` by `*len -= *len/8`. The max
 number of calls to mmap() goes up from 14 to 67, but wastage is now at
 most 1/8th of the final allocated memory.

 (Another question is whether a GHC compiled programme could access >1TB of
 RAM? We now have a few 4TB machines and more on the way...)

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14492#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler