[GHC] #14959: Heap overflow in optimizer

[GHC]

#14959: Heap overflow in optimizer
-------------------------------------+-------------------------------------
        Reporter:  darchon           |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.4.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by simonpj):

 I see this code in Core
 {{{
 go_a2Rl
   = \ (x_a2Rm :: GHC.Prim.Int#)
       (eta_B1 :: [Integer])
       (eta_X2 :: Integer) ->
       case eta_B1 of {
         [] -> eta_X2;
         : y_a2SA ys_a2SB ->
           let {
             eta_Xn :: Integer
             eta_Xn
               = GHC.Integer.Type.orInteger
                   eta_X2
                   (GHC.Integer.Type.bitInteger x_a2Rm) } in
           case x_a2Rm of wild_XL {
             __DEFAULT -> go_a2Rl (GHC.Prim.+# wild_XL 1#) ys_a2SB eta_Xn;
             9223372036854775807# -> eta_Xn
           }
       }
 }}}
 If we inline `eta_Xn` (which is only used once) we get
 {{{
 ...(case x_a2Rm of wild_XL {
        __DEFAULT -> ...
        9223372036854775807# -> GHC.Integer.Type.orInteger eta_X2
                                   (GHC.Integer.Type.bitInteger x_a2Rm)
    )...
 }}}
 Now GHC sees that in the branch of the case it knows that `x =
 9223372036854775807#`.  So it tries to do constant folding.  But the
 result is a rather big Integer:
 {{{
 Prelude Data.Bits GHC.Exts> bit (I# 3#) :: Integer
 8
 Prelude Data.Bits GHC.Exts> bit (I# 4#) :: Integer
 16
 Prelude Data.Bits GHC.Exts> bit (I# 20#) :: Integer
 1048576
 Prelude Data.Bits GHC.Exts> bit (I# 60#) :: Integer
 1152921504606846976
 Prelude Data.Bits GHC.Exts> bit (I# 200#) :: Integer
 1606938044258990275541962092341162602522202993782792835301376
 }}}
 I'll leave you to imagine how big `bit 9223372036854775807#` is.

 Solution: the `bitInteger` rule should only work if its argument is "small
 enough".  I suggest that "small enough" means "smaller than
 wordSizeInBits", ie x<64 on a 64 bit machine.

 I'm validating a patch.

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14959#comment:6>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler