panic when compiling SHA

[Simon Peyton Jones]

Any time GHC simply falls over, it's a bug.  Even if you give GHC a 100 Gbyte input program and try to compile it on a 1 Gbyte machine, it's a bug if GHC simply falls over with "heap exhausted".  It would be better if it chopped the program into pieces and compiled them one at a time; or failed with a civilised message like "I'm afraid this input program is too big for me to compile".

But some bugs are more pressing than others, and with slender resources we have to concentrate on the ones that affect most people most seriously.  There seems to be consensus that this one falls into the "does not affect many people" category, and it has a workaround.  So I'm fine with leaving it open, but I think the priority is probably low.

Simon

| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Adam
| Wick
| Sent: 07 January 2014 23:57
| To: Ben Lippmeier
| Cc: ghc-devs at haskell.org
| Subject: Re: panic when compiling SHA
| 
| On Jan 7, 2014, at 2:27 AM, Ben Lippmeier <benl at ouroborus.net> wrote:
| > On 07/01/2014, at 9:26 , Adam Wick <awick at galois.com> wrote:
| >
| >>> Not if we just have this one test. I'd be keen to blame excessive
| use of inline pragmas in the SHA library itself, or excessive
| optimisation flags. It's not really a bug in GHC until there are two
| tests that exhibit the same problem.
| >>
| >> The SHA library uses SPECIALIZE, INLINE, and bang patterns in fairly
| standard ways. There's nothing too exotic in there, I just basically
| sprinkled hints in places I thought would be useful, and then backed
| those up with benchmarking.
| >
| > Ahh. It's the "sprinkled hints in places I thought would be useful"
| which is what I'm concerned about. If you just add pragmas without
| understanding their effect on the core program then it'll bite further
| down the line. Did you compare the object code size as well as wall
| clock speedup?
| 
| I understand the pragmas and what they do with my code. I use SPECIALIZE
| twice for two functions. In both functions, it was clearer to write the
| function as (a -> a -> a -> a), but I wanted specialized versions for
| the two versions that were going to be used, in which (a == Word32) or
| (a == Word64). This benchmarked as faster while maintaining code clarity
| and concision. I use INLINE in five places, each of them a SHA step
| function, with the understanding that it would generate ideal code for a
| compiler for the performance-critical parts of the algorithm: straight
| line, single-block code with no conditionals.
| 
| When I did my original performance work, several versions of GHC ago, I
| did indeed consider compile time, runtime performance, and space usage.
| I picked what I thought was a reasonable balance at the time.
| 
| I also just performed an experiment in which I took the SHA library,
| deleted all instances of INLINE and SPECIALIZE, and compiled it with
| HEAD on 32-bit Linux. You get the same crash. So my usage of SPECIALIZE
| and INLINE is beside the point.
| 
| > Sadly, "valid input" isn't a well defined concept in practice. You
| could write a "valid" 10GB Haskell source file that obeyed the Haskell
| standard grammar, but I wouldn't expect that to compile either.
| 
| I would. I'm a little disappointed that ghc-devs does not. I wouldn't
| expect it to compile quickly, but I would expect it to run.
| 
| > You could also write small (< 1k) source programs that trigger
| complexity problems in Hindley-Milner style type inference. You could
| also use compile-time meta programming (like Template Haskell) to
| generate intermediate code that is well formed but much too big to
| compile. The fact that a program obeys a published grammar is not
| sufficient to expect it to compile with a particular implementation
| (sorry to say).
| 
| If I write a broken Template Haskell macro, then yes, I agree. This is
| not the case in this example.
| 
| > Adding an INLINE pragma is akin to using compile-time meta
| programming.
| 
| Is it? I find that a strange point of view. Isn't INLINE just a strong
| hint to the compiler that this function should be inlined? How is using
| INLINE any different from simply manually inserting the code at every
| call site?
| 
| 
| - Adam