From simonpj at microsoft.com Wed Nov 1 08:28:00 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 1 Nov 2017 08:28:00 +0000 Subject: Building the docs, again In-Reply-To: References: Message-ID: It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. Maybe Hadrian will help here? Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Iavor Diatchki Sent: 31 October 2017 21:49 To: ghc-devs at haskell.org Subject: Re: Building the docs, again Hello, never mind, I figured it out---apparently the build targets were disabled for the `quick` flavor, which I was using. It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. -Iavor On Tue, Oct 31, 2017 at 2:26 PM Iavor Diatchki > wrote: Hello, sometime ago, I asked if it is possible to build just the GHC docs, without building the compiler. I was told to just run `make html`. This does not appear to work, the command seems to just start building GHC. What am I doing wrong? Here is the output I see, running the command from the root of the GHC tree. > make html V=0 ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds HC [stage 0] compiler/stage1/build/BufWrite.o HC [stage 0] compiler/stage1/build/Pretty.o compilation IS NOT required HC [stage 0] compiler/stage1/build/PprColour.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Outputable.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Json.o compilation IS NOT required HC [stage 0] compiler/stage1/build/SrcLoc.o compilation IS NOT required HC [stage 0] compiler/stage1/build/BasicTypes.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Unique.o HC [stage 0] compiler/stage1/build/Packages.o-boot compilation IS NOT required HC [stage 0] compiler/stage1/build/FiniteMap.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Name.o-boot (then I interrupted it). Also, in `mk/config.mk` I see this: BUILD_SPHINX_HTML = YES -Iavor -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed Nov 1 08:46:20 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 1 Nov 2017 08:46:20 +0000 Subject: GHCi recompilation avoidance UI In-Reply-To: <7587879.l535S4SiGz@squirrel> References: <7587879.l535S4SiGz@squirrel> Message-ID: I'm lost. * What causes the undesired behaviour in GHC 8.2.1? Is it this? - GHCi wants to load module B with flags F - There is a B.o but the flags differ - So GHCi recompiles B to bytecode * How does (1) help? * Does (2) propose a new command (:load *M vs :load M)? If so, how does it help. * Is (3), what is the spec of the proposed new flags? Is it this? - GHC wants to compile module B with flags F - There is a B.o compiled with flags F' - And B.hs (and its dependencies) have not changed - Then normally we'd recompile F unless F = F' - But with -fignore-optim-changes we would weaken that equality to ignore differences between F and F' that only affected optimisation level (the list of such flags would need to be written out) - And similarly for -fignore-hpc-changes Precision is all! Thanks Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of David | Feuer | Sent: 31 October 2017 15:42 | To: ghc-devs at haskell.org | Subject: GHCi recompilation avoidance UI | | Changes in GHC 8.2.1 lead to a lot of recompilation, because GHCi now | refuses to load optimized code unless -fobject-code (and optimization | flags) are enabled. I propose the following slight modification to | https://ghc.haskell.org/trac/ghc/ticket/13604#comment:48 | | 1. Optimization flags (except -O0) imply -fobject-code. This ensures | that GHC respects optimization flags regardless of --interactive. | | 2. Even when -fobject-code is on, :load *M will load M as bytecode. | This provides the "escape hatch" from -fobject-code that you need to | use debugging features, etc. | | 3. New -fignore-optim-changes and -fignore-hpc-changes (​​Phab:D4123) | flags should enable users to put together object code and bytecode with | diverse optimization levels/options and HPC options while still | updating automatically based on source changes and whether profiling is | enabled. | | Does this seem like a reasonable way forward? | | David Feuer | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.ha | skell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cd533ac396d6a4a6ff68208d52 | 076040e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450613629249637 | &sdata=9duvpS1xcNXXUm%2BOJgWg4jFBCgvT3SqPJMU2k8Rdxj0%3D&reserved=0 From andrey.mokhov at newcastle.ac.uk Wed Nov 1 12:13:36 2017 From: andrey.mokhov at newcastle.ac.uk (Andrey Mokhov) Date: Wed, 1 Nov 2017 12:13:36 +0000 Subject: Building the docs, again In-Reply-To: References: Message-ID: > Maybe Hadrian will help here? Hadrian can build documentation in various formats but the current implementation is very coarse-grain: we either build all possible docs or no docs at all. It would be relatively easy to add more fine-grain documentation targets and modes. If someone could design/describe the requirements in a ticket that would be great. Cheers, Andrey From: Simon Peyton Jones [mailto:simonpj at microsoft.com] Sent: 01 November 2017 08:28 To: Iavor Diatchki ; ghc-devs at haskell.org; Andrey Mokhov Subject: RE: Building the docs, again It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. Maybe Hadrian will help here? Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Iavor Diatchki Sent: 31 October 2017 21:49 To: ghc-devs at haskell.org Subject: Re: Building the docs, again Hello, never mind, I figured it out---apparently the build targets were disabled for the `quick` flavor, which I was using. It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. -Iavor On Tue, Oct 31, 2017 at 2:26 PM Iavor Diatchki > wrote: Hello, sometime ago, I asked if it is possible to build just the GHC docs, without building the compiler. I was told to just run `make html`. This does not appear to work, the command seems to just start building GHC. What am I doing wrong? Here is the output I see, running the command from the root of the GHC tree. > make html V=0 ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds HC [stage 0] compiler/stage1/build/BufWrite.o HC [stage 0] compiler/stage1/build/Pretty.o compilation IS NOT required HC [stage 0] compiler/stage1/build/PprColour.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Outputable.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Json.o compilation IS NOT required HC [stage 0] compiler/stage1/build/SrcLoc.o compilation IS NOT required HC [stage 0] compiler/stage1/build/BasicTypes.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Unique.o HC [stage 0] compiler/stage1/build/Packages.o-boot compilation IS NOT required HC [stage 0] compiler/stage1/build/FiniteMap.o compilation IS NOT required HC [stage 0] compiler/stage1/build/Name.o-boot (then I interrupted it). Also, in `mk/config.mk` I see this: BUILD_SPHINX_HTML = YES -Iavor -------------- next part -------------- An HTML attachment was scrubbed... URL: From qdunkan at gmail.com Wed Nov 1 17:33:02 2017 From: qdunkan at gmail.com (Evan Laforge) Date: Wed, 1 Nov 2017 10:33:02 -0700 Subject: GHCi recompilation avoidance UI In-Reply-To: References: <7587879.l535S4SiGz@squirrel> Message-ID: On Wed, Nov 1, 2017 at 1:46 AM, Simon Peyton Jones via ghc-devs wrote: > I'm lost. > > * What causes the undesired behaviour in GHC 8.2.1? > Is it this? > - GHCi wants to load module B with flags F > - There is a B.o but the flags differ > - So GHCi recompiles B to bytecode > > * How does (1) help? >From my perspective, what happened is that GHC 8.2 started including the -O and -fhpc flags in the hash for recompilation avoidance, so it's started to notice when those particular flags differ. However, ghci has always rejected those two flags, courtesy of DynFlags.makeDynFlagsConsistent. The result is that it's now impossible for ghci to load modules compiled with those flags, since it will never have the right flags. The reason that's a problem is that it's really convenient to run tests (compiled with -fhpc) interactively, and use the GHC API to interpret code when the rest of the system is compiled with -O. That latter usage might be on thin ice according to the comment on makeDynFlagsConsistent, but in practice it's worked fine for a long time. I'll let David answer about how #1 helps, but it looks like maybe ghci could have supported -O just fine with -fobject-code, so it should have been using that to determine whether or not to reject -O, not --interactive. > * Does (2) propose a new command (:load *M vs :load M)? > If so, how does it help. :load *M has always been present, so I'm not sure what is the difference. Maybe it didn't override -fobject-code before. I'll let David answer this and #3. But I think the effect of the flags in #3 would be "go back to <8.2", which is to say, omit -O and -fhpc from the interface hash. So yes, I think your description sounds right. From patrick.doc at ameritech.net Wed Nov 1 18:07:32 2017 From: patrick.doc at ameritech.net (Patrick Dougherty) Date: Wed, 1 Nov 2017 13:07:32 -0500 Subject: Building the docs, again In-Reply-To: References: Message-ID: <33F5CAAC-402C-44E1-B46B-CFBEBCEFCB9B@ameritech.net> I spent some time with the docs this summer, and wrote a fair bit of the Hadrian documentation building code. So maybe I can shed some light. The question is which docs do you want to build? The library documentation relies on the haddock executable, which is one of the last targets in the entire build. Also, IIRC, creating the nice links and references is somewhat tangled up with actually building the libraries. So it's not possible to avoid building GHC if you want the haddocks. On the other hand, you can build just the user's guide using Hadrian. You can build any sphinx html target like this: ../build.sh _build/docs/html/users_guide/index.html Or pdf with: ../build.sh _build/docs/pdfs/users_guide.pdf While not the cleanest, it does work. So, I think the only way you can skip building GHC is using Hadrian for sphinx targets. Until recently, the users guide did depend on building all of GHC. I believe the make-based build just doesn't know about this change? I think I looked into updating it, but determined that it would be easier just to update Hadrian instead. I'm a little foggy on the details at this point. Best, Patrick > On Nov 1, 2017, at 7:13 AM, Andrey Mokhov wrote: > > > Maybe Hadrian will help here? > > Hadrian can build documentation in various formats but the current implementation is very coarse-grain: we either build all possible docs or no docs at all. It would be relatively easy to add more fine-grain documentation targets and modes. If someone could design/describe the requirements in a ticket that would be great. > > Cheers, > Andrey > > > From: Simon Peyton Jones [mailto:simonpj at microsoft.com] > Sent: 01 November 2017 08:28 > To: Iavor Diatchki ; ghc-devs at haskell.org; Andrey Mokhov > Subject: RE: Building the docs, again > > It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. > > Maybe Hadrian will help here? > > Simon > > From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Iavor Diatchki > Sent: 31 October 2017 21:49 > To: ghc-devs at haskell.org > Subject: Re: Building the docs, again > > Hello, > > > > never mind, I figured it out---apparently the build targets were disabled for the `quick` flavor, which I was using. > > > > It is quite confusing that we can be setting the same variable to different values in different places. It would also be quite helpful if we modified the Makefile to say `documentation build target is disabled` or some such, rather then going ahead and building the whole of GHC, only at the end to inform me that it doesn't know what is `html`. > > > > -Iavor > > > > > > > > On Tue, Oct 31, 2017 at 2:26 PM Iavor Diatchki wrote: > > Hello, > > > > sometime ago, I asked if it is possible to build just the GHC docs, without building the compiler. I was told to just run `make html`. This does not appear to work, the command seems to just start building GHC. What am I doing wrong? > > > > Here is the output I see, running the command from the root of the GHC tree. > > > > > make html V=0 > > ===--- building phase 0 > > make --no-print-directory -f ghc.mk phase=0 phase_0_builds > > make[1]: Nothing to be done for 'phase_0_builds'. > > ===--- building phase 1 > > make --no-print-directory -f ghc.mk phase=1 phase_1_builds > > HC [stage 0] compiler/stage1/build/BufWrite.o > > HC [stage 0] compiler/stage1/build/Pretty.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/PprColour.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Outputable.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Json.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/SrcLoc.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/BasicTypes.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Unique.o > > HC [stage 0] compiler/stage1/build/Packages.o-boot > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/FiniteMap.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Name.o-boot > > > > (then I interrupted it). > > > > Also, in `mk/config.mk` I see this: > > BUILD_SPHINX_HTML = YES > > > > -Iavor > > > > > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From iavor.diatchki at gmail.com Wed Nov 1 18:22:24 2017 From: iavor.diatchki at gmail.com (Iavor Diatchki) Date: Wed, 01 Nov 2017 18:22:24 +0000 Subject: Building the docs, again In-Reply-To: <33F5CAAC-402C-44E1-B46B-CFBEBCEFCB9B@ameritech.net> References: <33F5CAAC-402C-44E1-B46B-CFBEBCEFCB9B@ameritech.net> Message-ID: Hello, I was talking about the user manual. I imagine that the situation I was in is quite common---you've just implemented some small feature, and you want to write some documentation about it. While working on the documentation, I want to rebuild the manual quickly, so that I can see what it looks like, and make sure that links etc. So I think it would be very important for Hadrian to have some high-level targets equivalent to the current `make html` etc. rather than having to ask it to rebuild a specific file somewhere in the build tree. The confusion with my previous e-mail was caused because I hadn't realized what happens when building the documentation is disabled, namely: 1. There are multiple places to check if the documentation is actually disabled 2. When the documentation is disabled, the build targets disappear, rather than change their behavior to say "Docs are disabled, I am not going to build them, do X if you want them build". So the issue is more of a usability one. After I sent the e-mail to the list I realized that this all is documented online, so that's nice at least, but---in general---I think it would be nice if we could avoid such surprised, rather than having to document them in a separate place. -Iavor On Wed, Nov 1, 2017 at 11:07 AM Patrick Dougherty wrote: > I spent some time with the docs this summer, and wrote a fair bit of the > Hadrian documentation building code. So maybe I can shed some light. > > The question is which docs do you want to build? The library documentation > relies on the haddock executable, which is one of the last targets in the > entire build. Also, IIRC, creating the nice links and references is > somewhat tangled up with actually building the libraries. So it's not > possible to avoid building GHC if you want the haddocks. > > On the other hand, you can build just the user's guide using Hadrian. You > can build any sphinx html target like this: > > ./build.sh _build/docs/html/users_guide/index.html > > Or pdf with: > > ./build.sh _build/docs/pdfs/users_guide.pdf > > While not the cleanest, it does work. So, I think the only way you can > skip building GHC is using Hadrian for sphinx targets. > > Until recently, the users guide did depend on building all of GHC.. I > believe the make-based build just doesn't know about this change? I think I > looked into updating it, but determined that it would be easier just to > update Hadrian instead. I'm a little foggy on the details at this point. > > Best, > Patrick > > On Nov 1, 2017, at 7:13 AM, Andrey Mokhov > wrote: > > > Maybe Hadrian will help here? > > > > Hadrian can build documentation in various formats but the current > implementation is very coarse-grain: we either build all possible docs or > no docs at all. It would be relatively easy to add more fine-grain > documentation targets and modes. If someone could design/describe the > requirements in a ticket that would be great. > > > > Cheers, > > Andrey > > > > > > *From:* Simon Peyton Jones [mailto:simonpj at microsoft.com > ] > *Sent:* 01 November 2017 08:28 > *To:* Iavor Diatchki ; ghc-devs at haskell.org; > Andrey Mokhov > *Subject:* RE: Building the docs, again > > > > It is quite confusing that we can be setting the same variable to > different values in different places. It would also be quite helpful if > we modified the Makefile to say `documentation build target is disabled` or > some such, rather then going ahead and building the whole of GHC, only at > the end to inform me that it doesn't know what is `html`. > > Maybe Hadrian will help here? > > > > Simon > > > > *From:* ghc-devs [mailto:ghc-devs-bounces at haskell.org > ] *On Behalf Of *Iavor Diatchki > *Sent:* 31 October 2017 21:49 > *To:* ghc-devs at haskell.org > *Subject:* Re: Building the docs, again > > > > Hello, > > > > never mind, I figured it out---apparently the build targets were disabled > for the `quick` flavor, which I was using. > > > > It is quite confusing that we can be setting the same variable to > different values in different places. It would also be quite helpful if > we modified the Makefile to say `documentation build target is disabled` or > some such, rather then going ahead and building the whole of GHC, only at > the end to inform me that it doesn't know what is `html`. > > > > -Iavor > > > > > > > > On Tue, Oct 31, 2017 at 2:26 PM Iavor Diatchki > wrote: > > Hello, > > > > sometime ago, I asked if it is possible to build just the GHC docs, > without building the compiler. I was told to just run `make html`. This > does not appear to work, the command seems to just start building GHC. > What am I doing wrong? > > > > Here is the output I see, running the command from the root of the GHC > tree. > > > > > make html V=0 > > ===--- building phase 0 > > make --no-print-directory -f ghc.mk > > phase=0 phase_0_builds > > make[1]: Nothing to be done for 'phase_0_builds'. > > ===--- building phase 1 > > make --no-print-directory -f ghc.mk > > phase=1 phase_1_builds > > HC [stage 0] compiler/stage1/build/BufWrite.o > > HC [stage 0] compiler/stage1/build/Pretty.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/PprColour.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Outputable.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Json.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/SrcLoc.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/BasicTypes.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Unique.o > > HC [stage 0] compiler/stage1/build/Packages.o-boot > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/FiniteMap.o > > compilation IS NOT required > > HC [stage 0] compiler/stage1/build/Name.o-boot > > > > (then I interrupted it). > > > > Also, in `mk/config.mk > ` > I see this: > > BUILD_SPHINX_HTML = YES > > > > -Iavor > > > > > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Thu Nov 2 08:33:22 2017 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Thu, 2 Nov 2017 09:33:22 +0100 Subject: When does GHC produce unlifted `let` bindings? In-Reply-To: References: Message-ID: Thanks both of you, that makes sense. Re: let-bindings floating more easily, would a check for `exprIsOkForSpeculation scrut` in theory suffice to float out cases? On Mon, Oct 30, 2017 at 10:56 AM, Simon Peyton Jones wrote: > See Note [CoreSyn let/app invariant] in CoreSyn. > > > > Briefly, you can write > > let x::Int# = e in … > > if e is “ok-for-speculation”. See extensive comments in CoreUtils on what > that means. > > > > You could also use case, but let-bindings “float” more easily than cases, > because they are not worried about preserving exception behaviour. > > > > Simon > > > > *From:* ghc-devs [mailto:ghc-devs-bounces at haskell.org] *On Behalf Of *Sebastian > Graf > *Sent:* 29 October 2017 21:07 > *To:* ghc-devs > *Subject:* When does GHC produce unlifted `let` bindings? > > > > Hi folks, > > > > I was debugging a Core-to-Core transform for TEST=spec-inline > and > was wondering (yet again) why GHC produces unlifted `let` bindings in Core > like it seems supposed to be doing > > . > > - Why doesn't this use `case` instead? > - Is there a semantic difference? > - Can `case` be used with unlifted types? > - And if not, why can `let`? > > Unlifted `let` seems much close to `case` than to `let`. Some GHC passes > seem to agree. > > > > > Cheers, > > Sebastian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Nov 2 08:43:15 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 2 Nov 2017 08:43:15 +0000 Subject: When does GHC produce unlifted `let` bindings? In-Reply-To: References: Message-ID: Re: let-bindings floating more easily, would a check for `exprIsOkForSpeculation scrut` in theory suffice to float out cases? Yes; but instead we simply turn such cases into lets. Simon From: Sebastian Graf [mailto:sgraf1337 at gmail.com] Sent: 02 November 2017 08:33 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: When does GHC produce unlifted `let` bindings? Thanks both of you, that makes sense. Re: let-bindings floating more easily, would a check for `exprIsOkForSpeculation scrut` in theory suffice to float out cases? On Mon, Oct 30, 2017 at 10:56 AM, Simon Peyton Jones > wrote: See Note [CoreSyn let/app invariant] in CoreSyn. Briefly, you can write let x::Int# = e in … if e is “ok-for-speculation”. See extensive comments in CoreUtils on what that means. You could also use case, but let-bindings “float” more easily than cases, because they are not worried about preserving exception behaviour. Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Sebastian Graf Sent: 29 October 2017 21:07 To: ghc-devs > Subject: When does GHC produce unlifted `let` bindings? Hi folks, I was debugging a Core-to-Core transform for TEST=spec-inline and was wondering (yet again) why GHC produces unlifted `let` bindings in Core like it seems supposed to be doing. * Why doesn't this use `case` instead? * Is there a semantic difference? * Can `case` be used with unlifted types? * And if not, why can `let`? Unlifted `let` seems much close to `case` than to `let`. Some GHC passes seem to agree. Cheers, Sebastian -------------- next part -------------- An HTML attachment was scrubbed... URL: From marlowsd at gmail.com Thu Nov 2 10:21:07 2017 From: marlowsd at gmail.com (Simon Marlow) Date: Thu, 2 Nov 2017 10:21:07 +0000 Subject: GHCi recompilation avoidance UI In-Reply-To: <7587879.l535S4SiGz@squirrel> References: <7587879.l535S4SiGz@squirrel> Message-ID: On 31 October 2017 at 15:42, David Feuer wrote: > Changes in GHC 8.2.1 lead to a lot of recompilation, because GHCi now > refuses to load optimized > code unless -fobject-code (and optimization flags) are enabled. I propose > the following slight > modification to https://ghc.haskell.org/trac/ghc/ticket/13604#comment:48 > > 1. Optimization flags (except -O0) imply -fobject-code. This ensures that > GHC respects optimization flags regardless of --interactive. > > 2. Even when -fobject-code is on, :load *M will load M as bytecode. This > provides the "escape hatch" from -fobject-code that you need to use > debugging features, etc. > Yes, I think this is probably what we want. I'm not sure how smooth it will be to implement though. > 3. New -fignore-optim-changes and -fignore-hpc-changes (​​Phab:D4123) > flags should enable users to put together object code and bytecode with > diverse optimization levels/options and HPC options while still updating > automatically based on source changes and whether profiling is enabled. > As I mentioned on the diff, I think we'll want at least -fignore-optim-changes to be the default, so that GHCi does the expected thing when you have compiled object files. Cheers Simon > > Does this seem like a reasonable way forward? > > David Feuer > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marlowsd at gmail.com Thu Nov 2 10:27:16 2017 From: marlowsd at gmail.com (Simon Marlow) Date: Thu, 2 Nov 2017 10:27:16 +0000 Subject: Bringing some observable sharing to GHCi In-Reply-To: References: Message-ID: Not being able to :print an exception would be quite annoying. Do TypeReps need to be cyclic? Observable sharing can be done with StableName, but that would be a bit pile of hassle in the printer. Maybe exceptions warrant a special case. Cheers Simon On 29 October 2017 at 20:40, David Feuer wrote: > The :sprint, :print, and :force commands in GHCi fall into infinite > loops when confronted by cyclical data. This bit me hard in > https://phabricator.haskell.org/D4085 because that produces cyclical > TypeReps, which is trouble for (e.g.) the test break011 which tries to > :force a SomeException (which wraps an Exception dictionary, which has > a Typeable constraint). I could try coming up with a fix myself, but > I'm rather curious whether some of the work you (or others) have > already done on observing GHC data graphs could be yanked into GHCi > itself for this purpose. We want > > 1. To be able to display cyclical data in some sensible way. > > > x = "hi" : x > > x `seq` () > > :print x > > should print some useful representation of x. > > 2. To be able to force cyclical data without looping. > > > x = "hi" : x > > :force x > > should print a useful representation of x. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Nov 2 10:37:52 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 2 Nov 2017 10:37:52 +0000 Subject: Bringing some observable sharing to GHCi In-Reply-To: References: Message-ID: Do TypeReps need to be cyclic? No – I think David is working on a non-cyclic version Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon Marlow Sent: 02 November 2017 10:27 To: David Feuer Cc: Joachim Breitner ; ghc-devs Subject: Re: Bringing some observable sharing to GHCi Not being able to :print an exception would be quite annoying. Do TypeReps need to be cyclic? Observable sharing can be done with StableName, but that would be a bit pile of hassle in the printer. Maybe exceptions warrant a special case. Cheers Simon On 29 October 2017 at 20:40, David Feuer > wrote: The :sprint, :print, and :force commands in GHCi fall into infinite loops when confronted by cyclical data. This bit me hard in https://phabricator.haskell.org/D4085 because that produces cyclical TypeReps, which is trouble for (e.g.) the test break011 which tries to :force a SomeException (which wraps an Exception dictionary, which has a Typeable constraint). I could try coming up with a fix myself, but I'm rather curious whether some of the work you (or others) have already done on observing GHC data graphs could be yanked into GHCi itself for this purpose. We want 1. To be able to display cyclical data in some sensible way. > x = "hi" : x > x `seq` () > :print x should print some useful representation of x. 2. To be able to force cyclical data without looping. > x = "hi" : x > :force x should print a useful representation of x. _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan.gl.scott at gmail.com Fri Nov 3 00:18:54 2017 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Thu, 2 Nov 2017 20:18:54 -0400 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 Message-ID: I built all of Stackage with this RC the other night and didn't observe any regressions. Hooray! (Thanks to Doug Wilson for helping me get set up with his https://github.com/duog/build-stackage repo.) Ryan S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Nov 3 08:44:28 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 3 Nov 2017 08:44:28 +0000 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: References: Message-ID: Thank you for doing this! From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Ryan Scott Sent: 03 November 2017 00:19 To: ghc-devs at haskell.org Subject: Re: [ANNOUNCE] GHC 8.2.2 release candidate 2 I built all of Stackage with this RC the other night and didn't observe any regressions. Hooray! (Thanks to Doug Wilson for helping me get set up with his https://github.com/duog/build-stackage repo.) Ryan S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ggreif at gmail.com Fri Nov 3 10:44:22 2017 From: ggreif at gmail.com (Gabor Greif) Date: Fri, 3 Nov 2017 11:44:22 +0100 Subject: [commit: ghc] master: llvmGen: Pass vector arguments in vector registers by default (15f788f) In-Reply-To: <20171103001607.C23CA3A5EA@ghc.haskell.org> References: <20171103001607.C23CA3A5EA@ghc.haskell.org> Message-ID: IIRC there are no more static flags around. So adding ":type: dynamic" each time seems redundant. Could this be automated somehow? Or should we remove ":type:" altogether? Cheers, Gabor On 11/3/17, git at git.haskell.org wrote: > Repository : ssh://git at git.haskell.org/ghc > > On branch : master > Link : > http://ghc.haskell.org/trac/ghc/changeset/15f788f5e5096641245a4f060600a6db9cbc2c4e/ghc > >>--------------------------------------------------------------- > > commit 15f788f5e5096641245a4f060600a6db9cbc2c4e > Author: Ben Gamari > Date: Thu Nov 2 17:28:40 2017 -0400 > > llvmGen: Pass vector arguments in vector registers by default > > Earlier this year Edward Kmett requested [1] that we enable passing of > vector values in vector registers by default. The GHC calling convention > changes have been in LLVM for a number of years now so let's just flip > the switch. > > [1] https://mail.haskell.org/pipermail/ghc-devs/2017-March/013905.html > > Reviewers: austin > > Subscribers: rwbarton, thomie > > Differential Revision: https://phabricator.haskell.org/D4142 > > >>--------------------------------------------------------------- > > 15f788f5e5096641245a4f060600a6db9cbc2c4e > compiler/main/DynFlags.hs | 5 +++-- > docs/users_guide/using-optimisation.rst | 12 ++++++++++++ > 2 files changed, 15 insertions(+), 2 deletions(-) > > diff --git a/compiler/main/DynFlags.hs b/compiler/main/DynFlags.hs > index a421284..825497e 100644 > --- a/compiler/main/DynFlags.hs > +++ b/compiler/main/DynFlags.hs > @@ -3757,7 +3757,7 @@ fFlagsDeps = [ > flagSpec "kill-one-shot" Opt_KillOneShot, > flagSpec "late-dmd-anal" Opt_LateDmdAnal, > flagSpec "liberate-case" Opt_LiberateCase, > - flagHiddenSpec "llvm-pass-vectors-in-regs" > Opt_LlvmPassVectorsInRegisters, > + flagSpec "llvm-pass-vectors-in-regs" > Opt_LlvmPassVectorsInRegisters, > flagHiddenSpec "llvm-tbaa" Opt_LlvmTBAA, > flagHiddenSpec "llvm-fill-undef-with-garbage" > Opt_LlvmFillUndefWithGarbage, > flagSpec "loopification" Opt_Loopification, > @@ -4051,7 +4051,8 @@ defaultFlags settings > Opt_RPath, > Opt_SharedImplib, > Opt_SimplPreInlining, > - Opt_VersionMacros > + Opt_VersionMacros, > + Opt_LlvmPassVectorsInRegisters > ] > > ++ [f | (ns,f) <- optLevelFlags, 0 `elem` ns] > diff --git a/docs/users_guide/using-optimisation.rst > b/docs/users_guide/using-optimisation.rst > index 4714de7..fc958e0 100644 > --- a/docs/users_guide/using-optimisation.rst > +++ b/docs/users_guide/using-optimisation.rst > @@ -493,6 +493,18 @@ by saying ``-fno-wombat``. > self-recursive saturated tail calls into local jumps rather than > function calls. > > +.. ghc-flag:: -fllvm-pass-vectors-in-regs > + :shortdesc: Pass vector value in vector registers for function calls > + :type: dynamic > + :reverse: -fno-llvm-pass-vectors-in-regs > + :category: > + > + :default: on > + > + Instructs GHC to use the platform's native vector registers to pass > vector > + arguments during function calls. As with all vector support, this > requires > + :ghc-flag:`-fllvm`. > + > .. ghc-flag:: -fmax-inline-alloc-size=⟨n⟩ > :shortdesc: *default: 128.* Set the maximum size of inline array > allocations > to ⟨n⟩ bytes (default: 128). > > _______________________________________________ > ghc-commits mailing list > ghc-commits at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-commits > From ben at smart-cactus.org Fri Nov 3 16:31:40 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 03 Nov 2017 12:31:40 -0400 Subject: [commit: ghc] master: llvmGen: Pass vector arguments in vector registers by default (15f788f) In-Reply-To: References: <20171103001607.C23CA3A5EA@ghc.haskell.org> Message-ID: <87po8z8gsz.fsf@ben-laptop.smart-cactus.org> Gabor Greif writes: > IIRC there are no more static flags around. So adding ":type: dynamic" > each time seems redundant. Could this be automated somehow? Or should > we remove ":type:" altogether? > Well, we can't quite remove :type: yet since it still distinguishes "mode" flags (e.g. --make) from "normal" flags. In addition, some flags are marked as `dynamic / :set`, indicating that they can be changed within GHCi with `:set`. That being said, I agree that the :type:'s are needlessly verbose. I see two ways to proceed: * Make :type: default to `dynamic` (or perhaps another, friendlier name) * Represent mode flags with a different directive entirely In either case we might want to add a new boolean option to :ghc-flag: asserting it can be set from within ghci, separating this information from :type:. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From trp at bluewin.ch Sat Nov 4 08:50:33 2017 From: trp at bluewin.ch (Peter Trommler) Date: Sat, 4 Nov 2017 09:50:33 +0100 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> Message-ID: Hi Ben, Building and packaging RC2 on openSUSE works on all openSUSE platforms. I noticed, however, that versions of Cabal and process were downgraded from the respective versions in ghc 8.2.1: Cabal 2.0.0.2 -> 2.0.0.0 process 1.6.1.0 -> 1.6.0.0 Peter > On 31. Oct 2017, at 19:15, Ben Gamari wrote: > > > Hello everyone, > > The GHC team is very pleased to announce the second candidate of the > 8.2.2 release of the Glasgow Haskell Compiler. Source and binary > distributions are available at > > https://downloads.haskell.org/~ghc/8.2.2-rc2/ > > [...] > As always, please report any issues you encounter. > > Happy testing, > > - Ben From ben at well-typed.com Sat Nov 4 22:03:30 2017 From: ben at well-typed.com (Ben Gamari) Date: Sat, 04 Nov 2017 18:03:30 -0400 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> Message-ID: <87h8u98zwt.fsf@ben-laptop.smart-cactus.org> Peter Trommler writes: > Hi Ben, > > Building and packaging RC2 on openSUSE works on all openSUSE platforms. > I noticed, however, that versions of Cabal and process were downgraded > from the respective versions in ghc 8.2.1: > > Cabal 2.0.0.2 -> 2.0.0.0 > process 1.6.1.0 -> 1.6.0.0 > Thanks for pointing this out, Peter. It seems that my source distribution script cleans submodules, but doesn't actually run `git submodule update`, hence the issue. Quite unfortunate and yet another argument for building these artifacts via CI. I suppose at this point we have two options: 1. invalidate the release, cut an -rc3 and try again, or 2. make the issue known but continue testing under the assumption that the submodule issues won't interact with any of the other changes and eventually and move directly to 8.2.2 final Unfortunately at the moment I'm rather deep in debugging a codegen bug, so I don't have the bandwidth to handle either of these at the moment. Cheers, - Ben > Peter >> On 31. Oct 2017, at 19:15, Ben Gamari wrote: >> >> >> Hello everyone, >> >> The GHC team is very pleased to announce the second candidate of the >> 8.2.2 release of the Glasgow Haskell Compiler. Source and binary >> distributions are available at >> >> https://downloads.haskell.org/~ghc/8.2.2-rc2/ >> >> [...] >> As always, please report any issues you encounter. >> >> Happy testing, >> >> - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From svenpanne at gmail.com Sun Nov 5 12:56:53 2017 From: svenpanne at gmail.com (Sven Panne) Date: Sun, 5 Nov 2017 13:56:53 +0100 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> Message-ID: This is not an issue about 8.2.2 per se, but 8.2 changes in general: Recent discussions on Haskell Cafe showed serious problems with Emacs' haskell-mode due to some ad hoc changes like https://phabricator.haskell.org/D3651. Related GitHub issues: https://github.com/haskell/haskell-mode/issues/1553 https://github.com/haskell/haskell-mode/issues/1496 It should be noted that the output of GHC(i) is actually part of GHC's interface, so in this light, there have been some breaking changes, probably unintended, but nevertheless. So my question is: Is there a chance to revert some of these ad hoc changes and/or add some flags to get the old behavior back? I guess that Emacs + haskell-mode is one of the most important IDEs, so it would be a pity to worsen the situation there. I'm quite aware that it is very late in the release cycle, but it would be extremely nice if there was something which can be done. In the future it might make sense to co-operate with the haskell-mode team a bit more, perhaps adding some tests regarding the required output format etc. to GHC's test suite. Cheers, S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From svenpanne at gmail.com Sun Nov 5 20:52:11 2017 From: svenpanne at gmail.com (Sven Panne) Date: Sun, 5 Nov 2017 21:52:11 +0100 Subject: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: <83961DD2-1115-44A0-9E34-B553526B2CE2@gmx.com> References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> <83961DD2-1115-44A0-9E34-B553526B2CE2@gmx.com> Message-ID: 2017-11-05 15:37 GMT+01:00 : > A better approach might be to develop a "machine-readable" output format > which then is kept stable, and can be enabled with a flag. Git has a > similar solution. > Without doubt, this is definitely the better approach, but this is hardly what can be achieved for 8.2.2. Adding some flag to get the old behavior back when wanted *is* achievable. > It would be a shame to avoid changes which make the user experience better > simply because other projects cannot sync their development cycle, > Don't get me wrong: I'm all for improving user experience, but making ad hoc changes without enough thought or even a chance to get the old behavior back is probably not the right way to proceed. All SW lives in some kind of ecosystem, so it should behave well in that. And for Emacs users, the user experience has been made much worse. > especially if those projects are not universally used or required. > This is highly a matter of personal taste: No project is "universally used", so this is tautological statement. The question is: Is a minor cosmetic change really worth breaking things in one of the major IDEs? Cheers, S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qdunkan at gmail.com Sun Nov 5 21:12:39 2017 From: qdunkan at gmail.com (Evan Laforge) Date: Sun, 5 Nov 2017 13:12:39 -0800 Subject: [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> Message-ID: On Sun, Nov 5, 2017 at 4:56 AM, Sven Panne wrote: > This is not an issue about 8.2.2 per se, but 8.2 changes in general: Recent > discussions on Haskell Cafe showed serious problems with Emacs' haskell-mode > due to some ad hoc changes like https://phabricator.haskell.org/D3651. I've long used ghci -v0 to achieve the same effect. When you have a couple hundred modules the "loaded X Y Z" is worse than useless. But it's better to just add -v0, and maybe add more flags for finer control, than change the default and break tools. From simonpj at microsoft.com Mon Nov 6 16:22:56 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 6 Nov 2017 16:22:56 +0000 Subject: GHC build broken Message-ID: Ben: I'm getting this from a clean build on Linux. Any ideas? I'm up to date on master git branch -v data-kind-syntax 4174458178 Add IfacePromotionInfo * master 436b3ef01e Clean up comments about match algorithm a bit. tc-untouchables b737a45391 More simplifications to the constraint solver type-nats 87d2a63b8d [behind 1491] Fix uses of (<=?) as a function to reduce properly. wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing only wip/spj-T13397 43540c8c6b Improve code generation for conditionals wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot simonpj at cam-05-unx:~/code/HEAD/compiler$ Simon libraries/ghc-prim/cbits/atomic.c:311:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 311 | __atomic_store_n((StgWord16 *) x, (StgWord16) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite32': make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.o] Error 1 make[1]: *** Waiting for unfinished jobs.... libraries/ghc-prim/cbits/atomic.c:318:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 318 | __atomic_store_n((StgWord32 *) x, (StgWord32) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite64': libraries/ghc-prim/cbits/atomic.c:325:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 325 | __atomic_store_n((StgWord64 *) x, (StgWord64) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread64': libraries/ghc-prim/cbits/atomic.c:294:1: error: error: control reaches end of non-void function [-Werror=return-type] | 294 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread32': libraries/ghc-prim/cbits/atomic.c:287:1: error: error: control reaches end of non-void function [-Werror=return-type] | 287 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread16': libraries/ghc-prim/cbits/atomic.c:280:1: error: error: control reaches end of non-void function [-Werror=return-type] | 280 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread8': libraries/ghc-prim/cbits/atomic.c:273:1: error: error: control reaches end of non-void function [-Werror=return-type] | 273 | } | ^ cc1: all warnings being treated as errors `gcc' failed in phase `C Compiler'. (Exit code: 1) make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.dyn_o] Error 1 <> make: *** [all] Error 2 simonpj at cam-05-unx:~/5builds/HEAD$ dirs ~/5builds/HEAD ~/5builds/HEAD/testsuite/tests/typecheck/should_fail simonpj at cam-05-unx:~/5builds/HEAD$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Nov 6 16:24:57 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 6 Nov 2017 16:24:57 +0000 Subject: GHC build broken In-Reply-To: References: Message-ID: ps this Linux box is way behind gcc --version gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 Copyright (C) 2011 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. From: Simon Peyton Jones Sent: 06 November 2017 16:23 To: ghc-devs at haskell.org Subject: GHC build broken Ben: I'm getting this from a clean build on Linux. Any ideas? I'm up to date on master git branch -v data-kind-syntax 4174458178 Add IfacePromotionInfo * master 436b3ef01e Clean up comments about match algorithm a bit. tc-untouchables b737a45391 More simplifications to the constraint solver type-nats 87d2a63b8d [behind 1491] Fix uses of (<=?) as a function to reduce properly. wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing only wip/spj-T13397 43540c8c6b Improve code generation for conditionals wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot simonpj at cam-05-unx:~/code/HEAD/compiler$ Simon libraries/ghc-prim/cbits/atomic.c:311:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 311 | __atomic_store_n((StgWord16 *) x, (StgWord16) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite32': make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.o] Error 1 make[1]: *** Waiting for unfinished jobs.... libraries/ghc-prim/cbits/atomic.c:318:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 318 | __atomic_store_n((StgWord32 *) x, (StgWord32) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicwrite64': libraries/ghc-prim/cbits/atomic.c:325:54: error: error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) | 325 | __atomic_store_n((StgWord64 *) x, (StgWord64) val, __ATOMIC_SEQ_CST); | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread64': libraries/ghc-prim/cbits/atomic.c:294:1: error: error: control reaches end of non-void function [-Werror=return-type] | 294 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread32': libraries/ghc-prim/cbits/atomic.c:287:1: error: error: control reaches end of non-void function [-Werror=return-type] | 287 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread16': libraries/ghc-prim/cbits/atomic.c:280:1: error: error: control reaches end of non-void function [-Werror=return-type] | 280 | } | ^ libraries/ghc-prim/cbits/atomic.c: In function 'hs_atomicread8': libraries/ghc-prim/cbits/atomic.c:273:1: error: error: control reaches end of non-void function [-Werror=return-type] | 273 | } | ^ cc1: all warnings being treated as errors `gcc' failed in phase `C Compiler'. (Exit code: 1) make[1]: *** [libraries/ghc-prim/dist-install/build/cbits/atomic.dyn_o] Error 1 <> make: *** [all] Error 2 simonpj at cam-05-unx:~/5builds/HEAD$ dirs ~/5builds/HEAD ~/5builds/HEAD/testsuite/tests/typecheck/should_fail simonpj at cam-05-unx:~/5builds/HEAD$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Nov 6 16:54:37 2017 From: ben at well-typed.com (Ben Gamari) Date: Mon, 06 Nov 2017 11:54:37 -0500 Subject: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> <83961DD2-1115-44A0-9E34-B553526B2CE2@gmx.com> Message-ID: <87y3njl54i.fsf@ben-laptop.smart-cactus.org> Next time something like this arises please do open a ticket. Sven Panne writes: > 2017-11-05 15:37 GMT+01:00 : > >> A better approach might be to develop a "machine-readable" output format >> which then is kept stable, and can be enabled with a flag. Git has a >> similar solution. >> > > Without doubt, this is definitely the better approach, but this is hardly > what can be achieved for 8.2.2. Adding some flag to get the old behavior > back when wanted *is* achievable. > Yes, I have opened a differential adding such a flag. See D4164 [1]. Please bikeshed to taste. In general I would really prefer that we *not* consider GHCi's REPL to be a stable programmatic interface. If haskell-mode needs access to some bit of information from GHC, then please let's discuss how to provide it in a more robust manner. This shouldn't be hard; we just need to have feedback on what the interfaces required should look like. [1] https://phabricator.haskell.org/D4164 > >> It would be a shame to avoid changes which make the user experience better >> simply because other projects cannot sync their development cycle, >> > > Don't get me wrong: I'm all for improving user experience, but making ad > hoc changes without enough thought or even a chance to get the old behavior > back is probably not the right way to proceed. All SW lives in some kind of > ecosystem, so it should behave well in that. And for Emacs users, the user > experience has been made much worse. > The trouble is that GHC has an extremely large surface area, any bit of which might have mechanical consumers. Those who reviewed the patch indeed considered making this change behind a flag (as Francesco's original patch did) but we concluded that this would be pointless as it didn't seem likely that anyone relied on this behavior. It seems that in this case we were wrong That being said, we cannot always preemptively add complexity to the project out of fear that a given change might break a hypothetical mechanical consumer. GHCi is first-and-foremost a REPL for users. When evaluating a change, if we feel it is likely that we will break a mechanical user then we will likely guard the change with a flag. However, if not, we won't. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From svenpanne at gmail.com Mon Nov 6 17:46:24 2017 From: svenpanne at gmail.com (Sven Panne) Date: Mon, 6 Nov 2017 18:46:24 +0100 Subject: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: <87y3njl54i.fsf@ben-laptop.smart-cactus.org> References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> <83961DD2-1115-44A0-9E34-B553526B2CE2@gmx.com> <87y3njl54i.fsf@ben-laptop.smart-cactus.org> Message-ID: 2017-11-06 17:54 GMT+01:00 Ben Gamari : > Next time something like this arises please do open a ticket. > Yep, will do... > Yes, I have opened a differential adding such a flag. See D4164 [1]. > Please bikeshed to taste. > Thanks for the quick fix! > In general I would really prefer that we *not* consider GHCi's REPL to be > a stable programmatic interface. I fully understand that, and that's definitely the way to go. Nevertheless, parsing tool/compiler output is still one of the most used hacks^H^H^H techniques for lots of Emacs modes (and probably other IDEs). Not every project is as open to suggestions and changes as GHC is, so this is often the only way out. > That being said, we cannot always preemptively add complexity to the > project out of fear that a given change might break a hypothetical > mechanical consumer. That's of course not what was proposed. :-) > GHCi is first-and-foremost a REPL for users. > When evaluating a change, if we feel it is likely that we will break a > mechanical user then we will likely guard the change with a flag. > However, if not, we won't. > I think the main problem here was communication. I can't speak for the haskell-mode maintainers, but for my part I didn't notice the problems because I mainly use LTS Stackage and that is still(!) at 8.0.2 (Why? This is definitely part of the whole problem.). I tried the 8.2 series only sparingly and only via the command line, so this is perhaps what others did, too, so the interaction bug went unnoticed for such a long time. Cheers, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Mon Nov 6 18:46:15 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 06 Nov 2017 13:46:15 -0500 Subject: GHC build broken In-Reply-To: References: Message-ID: <87vainkzyg.fsf@ben-laptop.smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Ben: I'm getting this from a clean build on Linux. Any ideas? > I'm up to date on master > > git branch -v > > data-kind-syntax 4174458178 Add IfacePromotionInfo > > * master 436b3ef01e Clean up comments about match algorithm a bit. > > tc-untouchables b737a45391 More simplifications to the constraint solver > > type-nats 87d2a63b8d [behind 1491] Fix uses of (<=?) as a function to reduce properly. > > wip/pattern-synonyms e54d396752 [ahead 6, behind 4569] Comments and tracing only > > wip/spj-T13397 43540c8c6b Improve code generation for conditionals > > wip/spj-early-inline 8515fcfa91 Mark non-recursive join lambdas as one-shot > > simonpj at cam-05-unx:~/code/HEAD/compiler$ > > Simon > > > > libraries/ghc-prim/cbits/atomic.c:311:54: error: > > error: '__ATOMIC_SEQ_CST' undeclared (first use in this function) > This is due to Peter Trommler's D4009 (bd765f4b1332b3d2a7908de3f9ff1d50da0e0b1d). This changes the atomic primops to use new GCC builtins available only in GCC 4.7 and later. This version was released in 2012, so I thought we would be fairly safe requiring it. This is another case where upgrading your Ubuntu installation will help. That being said, we should certainly provide a better error message, perhaps catching this in `configure` instead of failing during the build. We could also provide a fallback in the event that we are compiling with an older gcc, although I'm not sure that this is worth the effort. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From alan.zimm at gmail.com Mon Nov 6 19:04:49 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 6 Nov 2017 21:04:49 +0200 Subject: [Haskell-cafe] [ANNOUNCE] GHC 8.2.2 release candidate 2 In-Reply-To: References: <877evbuqtf.fsf@ben-laptop.smart-cactus.org> <83961DD2-1115-44A0-9E34-B553526B2CE2@gmx.com> <87y3njl54i.fsf@ben-laptop.smart-cactus.org> Message-ID: To me this is just further evidence that we need a first class API for tool makers. Which is a (slow) work in progress. Alan On 6 November 2017 at 19:46, Sven Panne wrote: > 2017-11-06 17:54 GMT+01:00 Ben Gamari : > >> Next time something like this arises please do open a ticket. >> > > Yep, will do... > > >> Yes, I have opened a differential adding such a flag. See D4164 [1]. >> Please bikeshed to taste. >> > > Thanks for the quick fix! > > >> In general I would really prefer that we *not* consider GHCi's REPL to be >> a stable programmatic interface. > > > I fully understand that, and that's definitely the way to go. > Nevertheless, parsing tool/compiler output is still one of the most used > hacks^H^H^H techniques for lots of Emacs modes (and probably other IDEs). > Not every project is as open to suggestions and changes as GHC is, so this > is often the only way out. > > >> That being said, we cannot always preemptively add complexity to the >> project out of fear that a given change might break a hypothetical >> mechanical consumer. > > > That's of course not what was proposed. :-) > > >> GHCi is first-and-foremost a REPL for users. >> When evaluating a change, if we feel it is likely that we will break a >> mechanical user then we will likely guard the change with a flag. >> However, if not, we won't. >> > > I think the main problem here was communication. I can't speak for the > haskell-mode maintainers, but for my part I didn't notice the problems > because I mainly use LTS Stackage and that is still(!) at 8.0.2 (Why? This > is definitely part of the whole problem.). I tried the 8.2 series only > sparingly and only via the command line, so this is perhaps what others > did, too, so the interaction bug went unnoticed for such a long time. > > Cheers, > Sven > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Mon Nov 6 22:27:40 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 06 Nov 2017 17:27:40 -0500 Subject: Building on limited memory environments (or does -M really work?) In-Reply-To: References: Message-ID: <874lq7kppf.fsf@ben-laptop.smart-cactus.org> Saurabh Nanda writes: > Hi, > > We're struggling to get our build pipeline working on CircleCI, which has a > 4GB RAM limit. Here are some project stats to set the context: > > - 1,200+ modules > - 36,315 LoC of Haskell > - On the local machine with -O1 -j the build takes approx 5.2 GB of RAM [1] > > Trying to build on CircleCI with -O1 -j fails consistently, as expected. > However, the behaviour of -M -j -O1 is erratic, at best. Sometimes it > magically works, sometimes it fails. A number of times we've seen GHC take > 3.5 GB (as reported by `top`), even though the limit was set to 2.5GB. Here > are flags that we've tried in various combinations we've tried. None of > them is consistent in building / failing: > > * -O1 > * -j > * +RTS -M2621440000 -RTS > * +RTS -A32m -RTS > * +RTS -n2m -RTS > > What is the **real** behaviour of the -M option? How does it interact with > -j and -A? > Did you ever make any progress on this, Saurabh? The summary is this: * -j just tells GHC to parallelise compilation across modules. This can increase the maximum heap size needed by the compiler. * -A sets the nursery size; to first order the doesn't affect the maximum heap size, but rather is helpful when running parallel programs (e.g. ghc with -j) to minimize the frequency with which we must garbage-collect. * -M is a bit tricky to define. For one, it defines the maximum heap size beyond which we will terminate. However, we also use it in garbage collector to make various decisions about GC scheduling. I'll admit that I'm not terribly familiar with the details here. Note that -M does not guarantee that GHC will find a way to keep your program under the limit that you provide. It merely ensures that the program doesn't exceed the given size, aborting if necessary. At least this is my understanding. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Tue Nov 7 23:00:10 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 7 Nov 2017 23:00:10 +0000 Subject: valid substitutions for holes In-Reply-To: References: Message-ID: I'm not sure why pure and return are reported now when they weren't before. Characterising that would be helpful. But yes, it's an experimental feature, so I don't think you should hold up the train for it. You should talk to the author. Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Richard | Eisenberg | Sent: 27 October 2017 02:43 | To: ghc-devs | Subject: valid substitutions for holes | | Hi all, | | GHC HEAD currently suggests valid in-scope single-identifier | substitutions for typed holes (underscores in terms). Some unrelated work | somehow changed the output, and I'm wondering if this is a change for the | better or for the worse. | | The test case is this: | | > f :: Int -> Int -> Int | > f x y = _ x y | | With my patch, GHC now suggests `pure` and `return` as replacements for _ | (along with the other top contenders of (-), asTypeOf, and const). (You | can also increase the number of suggestions, which will then include (+) | and (*) among others.) It's true that `pure` and `return` fit the bill, | noting that ((->) Int) is an Applicative. But I doubt these suggestions | will help the user. | | I'm not keen to make fixing this "valid suggestions" feature (which I do | really like) a dependency of finishing my patch, but I also don't want | the change to output to go unnoticed. So, I'm asking: when I merge my | patch, should I post a bug asking for perhaps a priority ordering of | suggestions? | | Thanks, | Richard | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.hask | ell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cabcb64569709446adb0808d51d5 | 53ba6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636447174270553152&sda | ta=bGgYpNvuhR7rWnubjsYG5FKfza5gmbP0PPu6bNcJBUs%3D&reserved=0 From simonpj at microsoft.com Tue Nov 7 23:08:09 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 7 Nov 2017 23:08:09 +0000 Subject: tcSubsumes In-Reply-To: <66EEFF7E-040D-45AE-874F-81D8F4AF5F64@cs.brynmawr.edu> References: <66EEFF7E-040D-45AE-874F-81D8F4AF5F64@cs.brynmawr.edu> Message-ID: Richard It's clearly wrong as it is, but I suggest you that you make tcSubsumes use captureConstraints rather than pushLevelAndCaptureConstratints. That's what solveEqualities does, after all. Simon | -----Original Message----- | From: Richard Eisenberg [mailto:rae at cs.brynmawr.edu] | Sent: 26 October 2017 23:12 | To: Simon Peyton Jones | Cc: ghc-devs | Subject: tcSubsumes | | Hi Simon, | | Could you take a look at TcSimplify.tcSubsumes? This relatively new | function pushes the TcLevel when calling tcSubType, but then doesn't set | the TcLevel when calling the solver. Inexplicably, this sometimes works - | - even passing the testsuite. Somehow, my work caused this to fail. I | fixed it by setting the TcLevel before calling the solver. | | Two questions: | | 1. Is this a reasonable fix? | | 2. (optional) How is it possible that this didn't fail drastically often? | | Note (to author / reviewers of this function): my astonishment isn't that | this error crept into the codebase -- I've done similar many many times. | My astonishment is how the testsuite didn't find this error forthwith. | | Thanks, | Richard From saurabhnanda at gmail.com Wed Nov 8 09:11:59 2017 From: saurabhnanda at gmail.com (Saurabh Nanda) Date: Wed, 8 Nov 2017 14:41:59 +0530 Subject: Building on limited memory environments (or does -M really work?) In-Reply-To: <874lq7kppf.fsf@ben-laptop.smart-cactus.org> References: <874lq7kppf.fsf@ben-laptop.smart-cactus.org> Message-ID: > > Did you ever make any progress on this, Saurabh? > We made progress in some sense, by introducing a separate `stack build -j1` step in our CI pipeline for compiling packages that are known to use a lot of memory. > > * -j just tells GHC to parallelise compilation across modules. This can > increase the maximum heap size needed by the compiler. > >From the docs, it wasn't very clear to me how -j interacts with -M when both the options are passed to the GHC process. Is it the max heap size across all build, or per build? > * -M is a bit tricky to define. For one, it defines the maximum heap > size beyond which we will terminate. However, we also use it in > garbage collector to make various decisions about GC scheduling. I'll > admit that I'm not terribly familiar with the details here. > > Note that -M does not guarantee that GHC will find a way to keep your > program under the limit that you provide. It merely ensures that the > program doesn't exceed the given size, aborting if necessary. > Quoting from https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M : *> The maximum heap size also affects other garbage collection parameters: when the amount of live data in the heap exceeds a certain fraction of the maximum heap size, compacting collection will be automatically enabled for the oldest generation, and the -F parameter will be reduced in order to avoid exceeding the maximum heap size.* It just makes it sound that the RTS is going to tweak the GC algo, and the number of time GC is run, to avoid crossing the heap limit. However, I've found the GHC process easily consuming more memory than what is specified in the -M flag (as reported by top). -- Saurabh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Wed Nov 8 15:27:03 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 08 Nov 2017 10:27:03 -0500 Subject: Building on limited memory environments (or does -M really work?) In-Reply-To: References: <874lq7kppf.fsf@ben-laptop.smart-cactus.org> Message-ID: <871sl83i60.fsf@ben-laptop.smart-cactus.org> Saurabh Nanda writes: >> >> Did you ever make any progress on this, Saurabh? >> > > We made progress in some sense, by introducing a separate `stack build -j1` > step in our CI pipeline for compiling packages that are known to use a lot > of memory. > > >> >> * -j just tells GHC to parallelise compilation across modules. This can >> increase the maximum heap size needed by the compiler. >> > > > From the docs, it wasn't very clear to me how -j interacts with -M when > both the options are passed to the GHC process. Is it the max heap size > across all build, or per build? > The short answer is that they don't interact: -j is a GHC flag whereas -M is an RTS flag. -M controls the amount of heap that the RTS allows the mutator (GHC in this case) to allocate. This includes all threads. GHC when run with -j is just like any other threaded Haskell program. >> * -M is a bit tricky to define. For one, it defines the maximum heap >> size beyond which we will terminate. However, we also use it in >> garbage collector to make various decisions about GC scheduling. I'll >> admit that I'm not terribly familiar with the details here. >> >> Note that -M does not guarantee that GHC will find a way to keep your >> program under the limit that you provide. It merely ensures that the >> program doesn't exceed the given size, aborting if necessary. >> > > > Quoting from > https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M > : > > *> The maximum heap size also affects other garbage collection parameters: > when the amount of live data in the heap exceeds a certain fraction of the > maximum heap size, compacting collection will be automatically enabled for > the oldest generation, and the -F parameter will be reduced in order to > avoid exceeding the maximum heap size.* > > It just makes it sound that the RTS is going to tweak the GC algo, and the > number of time GC is run, to avoid crossing the heap limit. However, I've > found the GHC process easily consuming more memory than what is specified > in the -M flag (as reported by top). > Yes, as I mentioned we do tweak some things in the GC; however, these tweaks are really a best-effort attempt to avoid going over the limit. It's entirely possible that your mutator will be terminated if it wants to use significantly more than the limit set with -M. There is relatively little else the RTS can do in this case that wouldn't require explicit cooperation from the mutator to keep working sets down. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From oleg.grenrus at iki.fi Wed Nov 8 18:23:49 2017 From: oleg.grenrus at iki.fi (Oleg Grenrus) Date: Wed, 8 Nov 2017 20:23:49 +0200 Subject: Building on limited memory environments (or does -M really work?) In-Reply-To: <871sl83i60.fsf@ben-laptop.smart-cactus.org> References: <874lq7kppf.fsf@ben-laptop.smart-cactus.org> <871sl83i60.fsf@ben-laptop.smart-cactus.org> Message-ID: <8d17bb63-6707-b281-5b9c-e5aa109576b7@iki.fi> Note that there are two -j flags, one to `stack` and one to `ghc`. I'm not completely sure what `stack`s -j does at the moment, but IIRC it specifies how many *packages* are built in parallel (maybe running tests is counted towards that job limit too). This means that `stack build -j3` may (and most likely will) spawn 3 ghc building different packages, if dependency graph permits. This means that `stack -j3 --ghc-options="+RTS -M1G"` may end up using around 3G of memory. And as Ben already mentioned, GHC's -j affects "module parallellisation", i.e. how many modules to build simultaneously. So there are two `-j`s: stack's affects how many GHCs there are, and GHC's only how many threads there are. First multiplies -M memory usage, later doens't. Another thing to consider, is how both affect performance. And that's not an easy question to answer. To make situation even more interesting, latest cabal's support per-component builds, so instead of "how many parallel packages we build", it's "how many parallel components we build". As component graph is more granular (e.g. tests and executables are mostly independent leafs), there is more parallelization opportunities. I remember reading a discussion mentioning to have `-j N:M` for "N ghc's building M modules", so you can fine-tune that for your multi package project. Also there is https://github.com/haskell/cabal/issues/976 discusses how to make multiple GHC to co-operate with shared module limit. Also by digging a bit of Cabal's issue tracker, there are issues like https://github.com/haskell/cabal/issues/1529. For example linker's memory usage, and how -j affects that. It might affect you, where there is memory hungry package building (GHC eats memory), and also `stack` is linking some executable from other package (also memory hungry). As a final note: you can add per package ghc-options in `stack.yaml` [1] (and cabal.project). At work, for example I have (in `cabal.project`) few packages with     package memory-hungry-package         ghc-options: +RTS -M2G -RTS where other packages are unrestricted. This wins us a bit in time, as for most packages GHC's RTS doesn't need to worry about memory usage. Cheers, Oleg - [1] https://docs.haskellstack.org/en/stable/yaml_configuration/#ghc-options On 08.11.2017 17:27, Ben Gamari wrote: > Saurabh Nanda writes: > >>> Did you ever make any progress on this, Saurabh? >>> >> We made progress in some sense, by introducing a separate `stack build -j1` >> step in our CI pipeline for compiling packages that are known to use a lot >> of memory. >> >> >>> * -j just tells GHC to parallelise compilation across modules. This can >>> increase the maximum heap size needed by the compiler. >>> >> >> From the docs, it wasn't very clear to me how -j interacts with -M when >> both the options are passed to the GHC process. Is it the max heap size >> across all build, or per build? >> > The short answer is that they don't interact: -j is a GHC flag whereas > -M is an RTS flag. -M controls the amount of heap that the RTS allows > the mutator (GHC in this case) to allocate. This includes all > threads. GHC when run with -j is just like any other threaded Haskell > program. > >>> * -M is a bit tricky to define. For one, it defines the maximum heap >>> size beyond which we will terminate. However, we also use it in >>> garbage collector to make various decisions about GC scheduling. I'll >>> admit that I'm not terribly familiar with the details here. >>> >>> Note that -M does not guarantee that GHC will find a way to keep your >>> program under the limit that you provide. It merely ensures that the >>> program doesn't exceed the given size, aborting if necessary. >>> >> >> Quoting from >> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M >> : >> >> *> The maximum heap size also affects other garbage collection parameters: >> when the amount of live data in the heap exceeds a certain fraction of the >> maximum heap size, compacting collection will be automatically enabled for >> the oldest generation, and the -F parameter will be reduced in order to >> avoid exceeding the maximum heap size.* >> >> It just makes it sound that the RTS is going to tweak the GC algo, and the >> number of time GC is run, to avoid crossing the heap limit. However, I've >> found the GHC process easily consuming more memory than what is specified >> in the -M flag (as reported by top). >> > Yes, as I mentioned we do tweak some things in the GC; however, these > tweaks are really a best-effort attempt to avoid going over the limit. > It's entirely possible that your mutator will be terminated if it wants > to use significantly more than the limit set with -M. There is > relatively little else the RTS can do in this case that wouldn't require > explicit cooperation from the mutator to keep working sets down. > > Cheers, > > - Ben > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ben at well-typed.com Wed Nov 8 18:57:07 2017 From: ben at well-typed.com (Ben Gamari) Date: Wed, 08 Nov 2017 13:57:07 -0500 Subject: T14394 output Message-ID: <87y3ng1tvg.fsf@ben-laptop.smart-cactus.org> Hi Simon, It seems that your recent commit 30058b0e45e9 may not be quite right. Specifically, the expected output for T14394 is empty, whereas the test is almost certainly going to produce output. The current output is, pattern Foo :: () => (b ~ a) => a :~~: b -- Defined at :5:1 pattern Bar :: forall k2 k1 (a :: k1) (b :: k2). () => (k2 ~ k1, (b :: k2) ~~ (a :: k1)) => a :~~: b -- Defined at :11:1 pattern Bam :: () => Ord a => a -> a -> (S a, S a) -- Defined at :21:1 Does this look right to you? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mail at nh2.me Wed Nov 8 19:21:14 2017 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Wed, 8 Nov 2017 20:21:14 +0100 Subject: Building on limited memory environments (or does -M really work?) In-Reply-To: References: Message-ID: <8c11b1cb-bbb3-e4ff-deb7-6036f1bec372@nh2.me> Hey Saurabh, from my experience with CircleCI it builds on machines with e.g. 32 cores showing up in htop (but allows you to use way less that that). But ghc sees 32 cores so -j will compile up to 32 modules at the same time (thus using tons of RAM). I solved that by setting -jN to the actual number of cores I bought from CircleCI. Greetings! Niklas On 21/10/17 14:36, Saurabh Nanda wrote: > We're struggling to get our build pipeline working on CircleCI, which > has a 4GB RAM limit. From simonpj at microsoft.com Wed Nov 8 22:06:44 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 8 Nov 2017 22:06:44 +0000 Subject: T14394 output In-Reply-To: <87y3ng1tvg.fsf@ben-laptop.smart-cactus.org> References: <87y3ng1tvg.fsf@ben-laptop.smart-cactus.org> Message-ID: Gah! Yes that's the right output. I'm not sure how I messed that up. Sorry Simon | -----Original Message----- | From: Ben Gamari [mailto:ben at well-typed.com] | Sent: 08 November 2017 18:57 | To: Simon Peyton Jones | Cc: ghc-devs at haskell.org | Subject: T14394 output | | Hi Simon, | | It seems that your recent commit 30058b0e45e9 may not be quite right. | Specifically, the expected output for T14394 is empty, whereas the test | is almost certainly going to produce output. The current output is, | | | pattern Foo :: () => (b ~ a) => a :~~: b | -- Defined at :5:1 | pattern Bar :: forall k2 k1 (a :: k1) (b :: k2). () => (k2 ~ k1, | (b :: k2) ~~ (a | :: k1)) => a :~~: b | -- Defined at :11:1 | pattern Bam :: () => Ord a => a -> a -> (S a, S a) | -- Defined at :21:1 | | | Does this look right to you? | | Cheers, | | - Ben From harendra.kumar at gmail.com Thu Nov 9 17:33:16 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Thu, 9 Nov 2017 23:03:16 +0530 Subject: GHC 8.2 GC time stats going negative Message-ID: Hi, I am trying to use the mutator_cpu_ns and mutator_elapsed_ns from GHC 8.2 RTSStats structure retrieved using getRTSStats API documented here - https://hackage.haskell.org/package/base-4.10.0.0/docs/GHC-Stats.html . I am seeing that the values of these stats retrieved about 30 ms later are lower than the values retrieved earlier. Sometimes even after 100 ms the values decrease from the previous ones. However, as the time duration between two calls increase this becomes less and less likely. Is this an expected behavior or am I using it incorrectly? If it is expected, why does it happen, what is the underlying mechanism that makes it happen? Thanks, harendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu Nov 9 18:19:30 2017 From: ben at well-typed.com (Ben Gamari) Date: Thu, 09 Nov 2017 13:19:30 -0500 Subject: GHC 8.2 GC time stats going negative In-Reply-To: References: Message-ID: <87po8r1fil.fsf@ben-laptop.smart-cactus.org> Harendra Kumar writes: > Hi, > > I am trying to use the mutator_cpu_ns and mutator_elapsed_ns from GHC 8.2 > RTSStats structure retrieved using getRTSStats API documented here - > https://hackage.haskell.org/package/base-4.10.0.0/docs/GHC-Stats.html . > > I am seeing that the values of these stats retrieved about 30 ms later are > lower than the values retrieved earlier. Sometimes even after 100 ms the > values decrease from the previous ones. However, as the time duration > between two calls increase this becomes less and less likely. > > Is this an expected behavior or am I using it incorrectly? If it is > expected, why does it happen, what is the underlying mechanism that makes > it happen? > Hmm, interesting. I don't believe this should happen and I do know of bugs in the RTS's time accounting at exit-time (manifesting in profiling issues, e.g. #14257) however what you describe doesn't seem to fit that description. Do you have a simple repro? Is there anything special about your program (e.g. lots of FFI calls)? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From harendra.kumar at gmail.com Thu Nov 9 18:42:52 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Fri, 10 Nov 2017 00:12:52 +0530 Subject: GHC 8.2 GC time stats going negative In-Reply-To: <87po8r1fil.fsf@ben-laptop.smart-cactus.org> References: <87po8r1fil.fsf@ben-laptop.smart-cactus.org> Message-ID: Hi Ben, Yes, there are FFI calls between the two getRTSStats calls. FFI calls are C APIs to get some OS stats like clock time or cpu time etc. Now I know that this is not expected, let me try to minimize the example and see where it goes. I will try removing the FFI calls as well. -harendra On 9 November 2017 at 23:49, Ben Gamari wrote: > Harendra Kumar writes: > > > Hi, > > > > I am trying to use the mutator_cpu_ns and mutator_elapsed_ns from GHC 8.2 > > RTSStats structure retrieved using getRTSStats API documented here - > > https://hackage.haskell.org/package/base-4.10.0.0/docs/GHC-Stats.html . > > > > I am seeing that the values of these stats retrieved about 30 ms later > are > > lower than the values retrieved earlier. Sometimes even after 100 ms the > > values decrease from the previous ones. However, as the time duration > > between two calls increase this becomes less and less likely. > > > > Is this an expected behavior or am I using it incorrectly? If it is > > expected, why does it happen, what is the underlying mechanism that makes > > it happen? > > > Hmm, interesting. I don't believe this should happen and I do know of > bugs in the RTS's time accounting at exit-time (manifesting in profiling > issues, e.g. #14257) however what you describe doesn't seem to fit that > description. Do you have a simple repro? Is there anything special about > your program (e.g. lots of FFI calls)? > > Cheers, > > - Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: From harendra.kumar at gmail.com Thu Nov 9 21:18:23 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Fri, 10 Nov 2017 02:48:23 +0530 Subject: GHC 8.2 GC time stats going negative In-Reply-To: References: <87po8r1fil.fsf@ben-laptop.smart-cactus.org> Message-ID: Ben, I could reproduce it even after removing all FFI calls. I created a ghc trac ticket, you can find it here: https://ghc.haskell.org/trac/ghc/ticket/14445. Let me know if any more information is needed or if explanation about the code/repo is required. Thanks, Harendra On 10 November 2017 at 00:12, Harendra Kumar wrote: > Hi Ben, > > Yes, there are FFI calls between the two getRTSStats calls. FFI calls are > C APIs to get some OS stats like clock time or cpu time etc. Now I know > that this is not expected, let me try to minimize the example and see where > it goes. I will try removing the FFI calls as well. > > -harendra > > On 9 November 2017 at 23:49, Ben Gamari wrote: > >> Harendra Kumar writes: >> >> > Hi, >> > >> > I am trying to use the mutator_cpu_ns and mutator_elapsed_ns from GHC >> 8.2 >> > RTSStats structure retrieved using getRTSStats API documented here - >> > https://hackage.haskell.org/package/base-4.10.0.0/docs/GHC-Stats.html . >> > >> > I am seeing that the values of these stats retrieved about 30 ms later >> are >> > lower than the values retrieved earlier. Sometimes even after 100 ms the >> > values decrease from the previous ones. However, as the time duration >> > between two calls increase this becomes less and less likely. >> > >> > Is this an expected behavior or am I using it incorrectly? If it is >> > expected, why does it happen, what is the underlying mechanism that >> makes >> > it happen? >> > >> Hmm, interesting. I don't believe this should happen and I do know of >> bugs in the RTS's time accounting at exit-time (manifesting in profiling >> issues, e.g. #14257) however what you describe doesn't seem to fit that >> description. Do you have a simple repro? Is there anything special about >> your program (e.g. lots of FFI calls)? >> >> Cheers, >> >> - Ben >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu Nov 9 21:58:14 2017 From: ben at well-typed.com (Ben Gamari) Date: Thu, 09 Nov 2017 16:58:14 -0500 Subject: Cutting the GHC 8.4 branch Message-ID: <87efp715e1.fsf@ben-laptop.smart-cactus.org> Hello everyone, As described earlier in the fall, with the GHC 8.4 release we will be trying to hold to a more reliable six-month release cycle. Towards this end we will be generally try to cut major release branches three months before the release date. As GHC 8.4 is slated to be released in February 2018, we are quickly approaching that three-month mark. Consequently I'd like to try cutting a branch next week. There are a few things that need to happen in the interim: * Hadrian needs to be merged (this should be done by the end of the day today) * Tamar's patches bumping our Windows toolchain need to be merged (they are in my merge queue) * A bunch of submodule bumps The remainder of the Trees That Grow work may not be entirely merged by the time we branch but Alan and I will arrange that things are merged after the fact. If you have work that you would like to see merged, please do let me know. If things go according to plan, we will produce our first alpha release shortly after the branch. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From leo at halfaya.org Fri Nov 10 21:12:14 2017 From: leo at halfaya.org (John Leo) Date: Fri, 10 Nov 2017 13:12:14 -0800 Subject: merge issues with new field for FamEqn Message-ID: Hi everyone, Today I'm trying to merge the latest HEAD into my work on the explicit forall proposal, and I'm running into a problem I'd like help with. The changes that are affecting me seem to come from the latest Trees that Grow checkins. I had added a field to FamEqn, namely feqn_bndrs: data FamEqn pass pats rhs = FamEqn { feqn_tycon :: Located (IdP pass) , feqn_bndrs :: Maybe [LHsTyVarBndr pass] -- ^ Explicit user-provided binders , feqn_pats :: pats , feqn_fixity :: LexicalFixity -- ^ Fixity used in the declaration , feqn_rhs :: rhs } -- ^ -- - 'ApiAnnotation.AnnKeywordId' : 'ApiAnnotation.AnnEqual', -- 'ApiAnnotation.AnnForall','ApiAnnotation.AnnDot' -- For details on above see note [Api annotations] in ApiAnnotation deriving instance (DataId pass, Data pats, Data rhs) => Data (FamEqn pass pats rhs) When I try to compile now, I get the following errors that I cannot understand. I assume I need to do something else, but I'm not sure what it is, so I'd appreciate any guidance. compiler/hsSyn/HsDecls.hs:1441:1: error: • Could not deduce (Data (XValBinds pass pass), Data (XXValBindsLR pass pass)) arising from a use of ‘k’ from the context: (DataId pass, Data pats, Data rhs) bound by the instance declaration at compiler/hsSyn/HsDecls.hs:(1441,1)-(1442,46) • In the first argument of ‘k’, namely ‘((z FamEqn `k` a1) `k` a2)’ In the first argument of ‘k’, namely ‘(((z FamEqn `k` a1) `k` a2) `k` a3)’ In the first argument of ‘k’, namely ‘((((z FamEqn `k` a1) `k` a2) `k` a3) `k` a4)’ When typechecking the code for ‘gfoldl’ in a derived instance for ‘Data (FamEqn pass pats rhs)’: To see the code I am typechecking, use -ddump-deriv | 1441 | deriving instance (DataId pass, Data pats, Data rhs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... compiler/hsSyn/HsDecls.hs:1441:1: error: • Could not deduce (Data (XValBinds pass pass), Data (XXValBindsLR pass pass)) arising from a use of ‘k’ from the context: (DataId pass, Data pats, Data rhs) bound by the instance declaration at compiler/hsSyn/HsDecls.hs:(1441,1)-(1442,46) • In the first argument of ‘k’, namely ‘(k (k (z FamEqn)))’ In the first argument of ‘k’, namely ‘(k (k (k (z FamEqn))))’ In the first argument of ‘k’, namely ‘(k (k (k (k (z FamEqn)))))’ When typechecking the code for ‘gunfold’ in a derived instance for ‘Data (FamEqn pass pats rhs)’: To see the code I am typechecking, use -ddump-deriv | 1441 | deriving instance (DataId pass, Data pats, Data rhs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ John -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Sat Nov 11 08:27:09 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Sat, 11 Nov 2017 10:27:09 +0200 Subject: merge issues with new field for FamEqn In-Reply-To: References: Message-ID: Change the deriving to deriving instance (DataIdLR pass pass, Data pats, Data rhs) Unfortunately the LR stuff bleeds through at the moment. Alan On 10 November 2017 at 23:12, John Leo wrote: > Hi everyone, > > Today I'm trying to merge the latest HEAD into my work on the explicit > forall proposal, and I'm running into a problem I'd like help with. The > changes that are affecting me seem to come from the latest Trees that Grow > checkins. > > I had added a field to FamEqn, namely feqn_bndrs: > > data FamEqn pass pats rhs > = FamEqn > { feqn_tycon :: Located (IdP pass) > , feqn_bndrs :: Maybe [LHsTyVarBndr pass] > -- ^ Explicit user-provided binders > , feqn_pats :: pats > , feqn_fixity :: LexicalFixity -- ^ Fixity used in the declaration > , feqn_rhs :: rhs > } > -- ^ > -- - 'ApiAnnotation.AnnKeywordId' : 'ApiAnnotation.AnnEqual', > -- 'ApiAnnotation.AnnForall','ApiAnnotation.AnnDot' > > -- For details on above see note [Api annotations] in ApiAnnotation > deriving instance (DataId pass, Data pats, Data rhs) > => Data (FamEqn pass pats rhs) > > When I try to compile now, I get the following errors that I cannot > understand. I assume I need to do something else, but I'm not sure what it > is, so I'd appreciate any guidance. > > compiler/hsSyn/HsDecls.hs:1441:1: error: > • Could not deduce (Data (XValBinds pass pass), > Data (XXValBindsLR pass pass)) > arising from a use of ‘k’ > from the context: (DataId pass, Data pats, Data rhs) > bound by the instance declaration > at compiler/hsSyn/HsDecls.hs:(1441,1)-(1442,46) > • In the first argument of ‘k’, namely ‘((z FamEqn `k` a1) `k` a2)’ > In the first argument of ‘k’, namely > ‘(((z FamEqn `k` a1) `k` a2) `k` a3)’ > In the first argument of ‘k’, namely > ‘((((z FamEqn `k` a1) `k` a2) `k` a3) `k` a4)’ > When typechecking the code for ‘gfoldl’ > in a derived instance for ‘Data (FamEqn pass pats rhs)’: > To see the code I am typechecking, use -ddump-deriv > | > 1441 | deriving instance (DataId pass, Data pats, Data rhs) > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^... > > compiler/hsSyn/HsDecls.hs:1441:1: error: > • Could not deduce (Data (XValBinds pass pass), > Data (XXValBindsLR pass pass)) > arising from a use of ‘k’ > from the context: (DataId pass, Data pats, Data rhs) > bound by the instance declaration > at compiler/hsSyn/HsDecls.hs:(1441,1)-(1442,46) > • In the first argument of ‘k’, namely ‘(k (k (z FamEqn)))’ > In the first argument of ‘k’, namely ‘(k (k (k (z FamEqn))))’ > In the first argument of ‘k’, namely ‘(k (k (k (k (z FamEqn)))))’ > When typechecking the code for ‘gunfold’ > in a derived instance for ‘Data (FamEqn pass pats rhs)’: > To see the code I am typechecking, use -ddump-deriv > | > 1441 | deriving instance (DataId pass, Data pats, Data rhs) > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > John > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klebinger.andreas at gmx.at Sat Nov 11 10:47:58 2017 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Sat, 11 Nov 2017 11:47:58 +0100 Subject: Asserting vars with different Uniques represent the same Object Message-ID: <5A06D55E.5020302@gmx.at> My core questions are: * Should variables representing the same thing always have the same unique? * If not how can one assert they actually represent the same thing? Working on the pattern matching code I came across this assertion: > ASSERT(tvs1 `equalLength` ex_tvs ) http://git.haskell.org/ghc.git/blob/HEAD:/compiler/deSugar/MatchCon.hs#l125 tvs1 and ex_tvs are both the existentially quantified type variables of a pattern. One gained by taking apart the pattern itself and one by taking apart the ConLike in the pattern. While as far as I can tell they always represent the same Types they don't always compare as equal. Is there any other stable way to compare them? (By name? Something else?). Or is them not being equal a bug to begin with. Follow up question: The whole assert is essentially a unit test as ex_tvs isn't used outside of the assert. Is there a solution to check these invariants in tests instead of the source code? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Sat Nov 11 22:25:02 2017 From: ben at well-typed.com (Ben Gamari) Date: Sat, 11 Nov 2017 17:25:02 -0500 Subject: Asserting vars with different Uniques represent the same Object In-Reply-To: <5A06D55E.5020302@gmx.at> References: <5A06D55E.5020302@gmx.at> Message-ID: <87d14oo3ly.fsf@ben-laptop.smart-cactus.org> Andreas Klebinger writes: > My core questions are: > > * Should variables representing the same thing always have the same unique? > * If not how can one assert they actually represent the same thing? > > Working on the pattern matching code I came across this assertion: > > > ASSERT(tvs1 `equalLength` ex_tvs ) > http://git.haskell.org/ghc.git/blob/HEAD:/compiler/deSugar/MatchCon.hs#l125 > > tvs1 and ex_tvs are both the existentially quantified type variables > of a pattern. One gained by taking apart the pattern itself and one by > taking apart the ConLike in the pattern. > Could you elaborate a bit more here? This sounds like a bug but I'm not sure I completely follow how you came about these tyvars. > Follow up question: > The whole assert is essentially a unit test as ex_tvs isn't used outside > of the assert. > Is there a solution to check these invariants in tests instead of the > source code? If I understand the question correctly: no. GHC contains hundreds of assertions like this which are only enabled when the compiler is compiled with -DDEBUG. The advantage of including them in the source instead of an external test is that it keeps the invariant check right next to the implementation, where it belonds. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mail at joachim-breitner.de Sun Nov 12 03:03:37 2017 From: mail at joachim-breitner.de (Joachim Breitner) Date: Sat, 11 Nov 2017 22:03:37 -0500 Subject: [commit: ghc] master: WIP on combined Step 1 and 3 for Trees That Grow, HsExpr (e3ec2e7) In-Reply-To: <20171111211700.1CCCE3A5EA@ghc.haskell.org> References: <20171111211700.1CCCE3A5EA@ghc.haskell.org> Message-ID: <1510455817.5571.2.camel@joachim-breitner.de> Hi Alan, the combined effect of commit e3ec2e7ae94524ebd111963faf34b84d942265b4 WIP on combined Step 1 and 3 for Trees That Grow, HsExpr and commit 438dd1cbba13d35f3452b4dcef3f94ce9a216905 WIP on Doing a combined Step 1 and 3 for Trees That Grow causes a 15% regression in the time it takes to build GHC, according to https://perf.haskell.org/ghc/#compare/fe6848f544c2a14086bcef388c46f4070c22d287/e3ec2e7ae94524ebd111963faf34b84d942265b4 Is that a known, expected and accepted regression? Joachim -- Joachim Breitner mail at joachim-breitner.de http://www.joachim-breitner.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From ben at smart-cactus.org Sun Nov 12 04:14:41 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Sat, 11 Nov 2017 23:14:41 -0500 Subject: [commit: ghc] master: WIP on combined Step 1 and 3 for Trees That Grow, HsExpr (e3ec2e7) In-Reply-To: <1510455817.5571.2.camel@joachim-breitner.de> References: <20171111211700.1CCCE3A5EA@ghc.haskell.org> <1510455817.5571.2.camel@joachim-breitner.de> Message-ID: On November 11, 2017 10:03:37 PM EST, Joachim Breitner wrote: >Hi Alan, > >the combined effect of > >commit e3ec2e7ae94524ebd111963faf34b84d942265b4 >WIP on combined Step 1 and 3 for Trees That Grow, HsExpr > > >and > >commit 438dd1cbba13d35f3452b4dcef3f94ce9a216905 >WIP on Doing a combined Step 1 and 3 for Trees That Grow > > >causes a 15% regression in the time it takes to build GHC, according to >https://perf.haskell.org/ghc/#compare/fe6848f544c2a14086bcef388c46f4070c22d287/e3ec2e7ae94524ebd111963faf34b84d942265b4 > > >Is that a known, expected and accepted regression? > >Joachim I noted this on D4177 and discussed the effect with Alan. Indeed there is quite a sizeable regression in compilation time but thankfully this is not because GHC itself is slower. Rather, it simply requires more work to compile. I did a set of nofib runs with and without the first TTG patch and found that compiler allocations remained essentially unchanged. A 15% regression in the compilation time of GHC is indeed hard to stomach but Alan had said that much of this will likely disappear in the future. If this is the case then a temporary regression is in my opinion acceptable. Cheers, - Ben From mail at joachim-breitner.de Sun Nov 12 04:25:16 2017 From: mail at joachim-breitner.de (Joachim Breitner) Date: Sat, 11 Nov 2017 23:25:16 -0500 Subject: [commit: ghc] master: WIP on combined Step 1 and 3 for Trees That Grow, HsExpr (e3ec2e7) In-Reply-To: References: <20171111211700.1CCCE3A5EA@ghc.haskell.org> <1510455817.5571.2.camel@joachim-breitner.de> Message-ID: <1510460716.5571.9.camel@joachim-breitner.de> Hi, > A 15% regression in the compilation time of GHC is indeed hard to stomach but Alan had said that much of this will likely disappear in the future. great, that’s all I wanted to hear :-) Joachim -- Joachim Breitner mail at joachim-breitner.de http://www.joachim-breitner.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From harendra.kumar at gmail.com Sun Nov 12 16:18:02 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Sun, 12 Nov 2017 21:48:02 +0530 Subject: Dynamically choosing the main function Message-ID: Hi, GHC allows choosing a main function at link time using the "-main-is" option. I was wondering if there is a possibility to choose the main function at runtime. Or even better, if something equivalent to "ghc -e" is somehow possible in a linked binary executable. If not, are there any plans to achieve something like that in future? Are there any theoretical, practical obstacles to that? Thanks, Harendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From shea at shealevy.com Sun Nov 12 16:28:05 2017 From: shea at shealevy.com (Shea Levy) Date: Sun, 12 Nov 2017 11:28:05 -0500 Subject: Dynamically choosing the main function In-Reply-To: References: Message-ID: <87r2t3v4ve.fsf@xps13.shealevy.com> What would you be able to achieve with this that you couldn't achieve with branching in a fixed custom main function? Thanks, Shea Harendra Kumar writes: > Hi, > > GHC allows choosing a main function at link time using the "-main-is" > option. I was wondering if there is a possibility to choose the main > function at runtime. Or even better, if something equivalent to "ghc -e" > is somehow possible in a linked binary executable. If not, are there any > plans to achieve something like that in future? Are there any theoretical, > practical obstacles to that? > > Thanks, > Harendra > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From harendra.kumar at gmail.com Sun Nov 12 16:54:07 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Sun, 12 Nov 2017 22:24:07 +0530 Subject: Dynamically choosing the main function In-Reply-To: <87r2t3v4ve.fsf@xps13.shealevy.com> References: <87r2t3v4ve.fsf@xps13.shealevy.com> Message-ID: For example, I can easily define entry points for arbitrary functions. At run time I can choose which entry point to call and benchmark that entry point using existing performance benchmarking tools like the very powerful and comprehensive "perf" tool on linux. This provides a very powerful way to analyze the performance for free and in a convenient manner with little instrumentation. I do not want the main entry point to be clobbered for such things. Ideally I would just use a simple annotation for any arbitrary function to convert it into an entry point that can be chosen at runtime. I am just giving a general idea here without going into specifics. -harendra On 12 November 2017 at 21:58, Shea Levy wrote: > What would you be able to achieve with this that you couldn't achieve > with branching in a fixed custom main function? > > Thanks, > Shea > > Harendra Kumar writes: > > > Hi, > > > > GHC allows choosing a main function at link time using the "-main-is" > > option. I was wondering if there is a possibility to choose the main > > function at runtime. Or even better, if something equivalent to "ghc -e" > > is somehow possible in a linked binary executable. If not, are there any > > plans to achieve something like that in future? Are there any > theoretical, > > practical obstacles to that? > > > > Thanks, > > Harendra > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Sun Nov 12 17:48:45 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 12 Nov 2017 12:48:45 -0500 Subject: Dynamically choosing the main function In-Reply-To: References: Message-ID: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> Harendra Kumar writes: > Hi, > > GHC allows choosing a main function at link time using the "-main-is" > option. I was wondering if there is a possibility to choose the main > function at runtime. Or even better, if something equivalent to "ghc -e" > is somehow possible in a linked binary executable. If not, are there any > plans to achieve something like that in future? Are there any theoretical, > practical obstacles to that? > In general it's not easy to get a reference to a an arbitrary function in an object file by it's source name. Unless the symbol is exported the compiler is free to optimize the function or even drop it altogether. If you can guarantee that the functions you are about are exported then in principle you could probably do what you ask from a library. Just `dlopen` the executable (which would need to be compiled as a position independent exectuable), and dlsym the appropriately mangled name. Either way, I don't think think this request is in scope for the compiler. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From allbery.b at gmail.com Sun Nov 12 18:03:52 2017 From: allbery.b at gmail.com (Brandon Allbery) Date: Sun, 12 Nov 2017 13:03:52 -0500 Subject: Dynamically choosing the main function In-Reply-To: References: Message-ID: On Sun, Nov 12, 2017 at 11:18 AM, Harendra Kumar wrote: > GHC allows choosing a main function at link time using the "-main-is" > option. I was wondering if there is a possibility to choose the main > function at runtime. Or even better, if something equivalent to "ghc -e" > is somehow possible in a linked binary executable. If not, are there any > plans to achieve something like that in future? Are there any theoretical, > practical obstacles to that? > -e is just running the compiler's bytecode backend on an expression from the command line instead of a declaration from a file. It's not related to this. I might point out that this is not at all common from any language. A limited (and Unix-specific) mechanism is to have main pick from a list of fixed operating modes based on the program's basename, which can be set in the filesystem by making a hard link to the executable with a different basename. -- brandon s allbery kf8nh sine nomine associates allbery.b at gmail.com ballbery at sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From harendra.kumar at gmail.com Sun Nov 12 18:54:36 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Mon, 13 Nov 2017 00:24:36 +0530 Subject: Dynamically choosing the main function In-Reply-To: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> Message-ID: On 12 November 2017 at 23:18, Ben Gamari wrote: > Harendra Kumar writes: > > > Hi, > > > > GHC allows choosing a main function at link time using the "-main-is" > > option. I was wondering if there is a possibility to choose the main > > function at runtime. Or even better, if something equivalent to "ghc -e" > > is somehow possible in a linked binary executable. If not, are there any > > plans to achieve something like that in future? Are there any > theoretical, > > practical obstacles to that? > > > In general it's not easy to get a reference to a an arbitrary function > in an object file by it's source name. Unless the symbol is exported the > compiler is free to optimize the function or even drop it altogether. > I understand that. I was also wondering if there is way (some sort of annotation, pragma or any other workaround) to keep the symbols around without actually exporting them. Exporting either clobbers the module interface from user point of view or the modules need a wrapper to avoid exporting such symbols in user exposed modules. I would also expect an option to remove the effect of any such annotation so that the production build is not under optimized because of this. This is in fact one of the problems that I have been facing in general. I do not know of a good way to hide symbols from users but use them in tests and benchmarks during dev. What is the GHC recommended way to achieve this? > If you can guarantee that the functions you are about are exported then > in principle you could probably do what you ask from a library. Just > `dlopen` the executable (which would need to be compiled as a position > independent exectuable), and dlsym the appropriately mangled name. > Yeah that is a good option for dynamically linked executables. In fact I have been using dlopen to implement Haskell plugins but did not think of using it for this use case. Thanks for reminding me. In a static executable, does it make sense to make the RTS choose an exported symbol as entry point using an RTS option? -harendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From saurabhnanda at gmail.com Sun Nov 12 18:59:27 2017 From: saurabhnanda at gmail.com (Saurabh Nanda) Date: Mon, 13 Nov 2017 00:29:27 +0530 Subject: Dynamically choosing the main function In-Reply-To: References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> Message-ID: > In fact I have been using dlopen to implement Haskell plugins but did not think of using it for this use case. Absolutely OT. Is this technique different from what facebook/simonmar have recently open-sourced? Please blog about it. -- Saurabh. -harendra _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From harendra.kumar at gmail.com Sun Nov 12 19:57:34 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Mon, 13 Nov 2017 01:27:34 +0530 Subject: Dynamically choosing the main function In-Reply-To: References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> Message-ID: On 13 November 2017 at 00:29, Saurabh Nanda wrote: > > In fact I have been using dlopen to implement Haskell plugins but did > not think of using it for this use case. > > Absolutely OT. Is this technique different from what facebook/simonmar > have recently open-sourced? Please blog about it. > Take a look at this: http://hackage.haskell.org/package/plugins -harendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Sun Nov 12 22:17:03 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 12 Nov 2017 17:17:03 -0500 Subject: Dynamically choosing the main function In-Reply-To: References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> Message-ID: <87efp3m9ba.fsf@ben-laptop.smart-cactus.org> Harendra Kumar writes: > On 12 November 2017 at 23:18, Ben Gamari wrote: > >> In general it's not easy to get a reference to a an arbitrary function >> in an object file by it's source name. Unless the symbol is exported the >> compiler is free to optimize the function or even drop it altogether. >> > > I understand that. I was also wondering if there is way (some sort of > annotation, pragma or any other workaround) to keep the symbols around > without actually exporting them. Exporting either clobbers the module > interface from user point of view or the modules need a wrapper to avoid > exporting such symbols in user exposed modules. I would also expect an > option to remove the effect of any such annotation so that the production > build is not under optimized because of this. This is in fact one of the > problems that I have been facing in general. I do not know of a good way to > hide symbols from users but use them in tests and benchmarks during dev. > What is the GHC recommended way to achieve this? > The usual way of dealing with this is to expose the symbols from a `.Internal` module. >> If you can guarantee that the functions you are about are exported then >> in principle you could probably do what you ask from a library. Just >> `dlopen` the executable (which would need to be compiled as a position >> independent exectuable), and dlsym the appropriately mangled name. >> > > Yeah that is a good option for dynamically linked executables. In fact I > have been using dlopen to implement Haskell plugins but did not think of > using it for this use case. Thanks for reminding me. In a static > executable, does it make sense to make the RTS choose an exported symbol as > entry point using an RTS option? > I don't see much precedent for this in other compilers. Moreover, I am generally not convinced that it should be the compiler's job to provide such a feature when the same end could be reached with far less magic by the user. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From chak at justtesting.org Mon Nov 13 00:01:36 2017 From: chak at justtesting.org (Manuel M T Chakravarty) Date: Mon, 13 Nov 2017 11:01:36 +1100 Subject: [commit: ghc] master: WIP on combined Step 1 and 3 for Trees That Grow, HsExpr (e3ec2e7) In-Reply-To: References: <20171111211700.1CCCE3A5EA@ghc.haskell.org> <1510455817.5571.2.camel@joachim-breitner.de> Message-ID: <55726148-679B-4445-8D57-38CC56170376@justtesting.org> > Ben Gamari : > > On November 11, 2017 10:03:37 PM EST, Joachim Breitner > wrote: >> Hi Alan, >> >> the combined effect of >> >> commit e3ec2e7ae94524ebd111963faf34b84d942265b4 >> WIP on combined Step 1 and 3 for Trees That Grow, HsExpr >> >> >> and >> >> commit 438dd1cbba13d35f3452b4dcef3f94ce9a216905 >> WIP on Doing a combined Step 1 and 3 for Trees That Grow >> >> >> causes a 15% regression in the time it takes to build GHC, according to >> https://perf.haskell.org/ghc/#compare/fe6848f544c2a14086bcef388c46f4070c22d287/e3ec2e7ae94524ebd111963faf34b84d942265b4 >> >> >> Is that a known, expected and accepted regression? >> >> Joachim > > I noted this on D4177 and discussed the effect with Alan. Indeed there is quite a sizeable regression in compilation time but thankfully this is not because GHC itself is slower. Rather, it simply requires more work to compile. I did a set of nofib runs with and without the first TTG patch and found that compiler allocations remained essentially unchanged. > > A 15% regression in the compilation time of GHC is indeed hard to stomach but Alan had said that much of this will likely disappear in the future. If this is the case then a temporary regression is in my opinion acceptable. Hmm, on what grounds does he think that this is going to disappear and how likely is likely? This doesn’t sound convincing TBH. Cheers, Manuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From harendra.kumar at gmail.com Mon Nov 13 07:46:27 2017 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Mon, 13 Nov 2017 13:16:27 +0530 Subject: Dynamically choosing the main function In-Reply-To: <87efp3m9ba.fsf@ben-laptop.smart-cactus.org> References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> <87efp3m9ba.fsf@ben-laptop.smart-cactus.org> Message-ID: On 13 November 2017 at 03:47, Ben Gamari wrote: > Harendra Kumar writes: > > > On 12 November 2017 at 23:18, Ben Gamari wrote: > > > >> In general it's not easy to get a reference to a an arbitrary function > >> in an object file by it's source name. Unless the symbol is exported the > >> compiler is free to optimize the function or even drop it altogether. > >> > > > > I understand that. I was also wondering if there is way (some sort of > > annotation, pragma or any other workaround) to keep the symbols around > > without actually exporting them. Exporting either clobbers the module > > interface from user point of view or the modules need a wrapper to avoid > > exporting such symbols in user exposed modules. I would also expect an > > option to remove the effect of any such annotation so that the production > > build is not under optimized because of this. This is in fact one of the > > problems that I have been facing in general. I do not know of a good way > to > > hide symbols from users but use them in tests and benchmarks during dev. > > What is the GHC recommended way to achieve this? > > > The usual way of dealing with this is to expose the symbols from a > `.Internal` module. > That's what I meant when I referred to a wrapper module above. However, it requires you to create two modules even if one extra symbol is needed for dev purposes, a lot of boilerplate for a simple thing. Also, the symbols are anyway exposed to the users, we just ask the users to not look at those. I am wondering if using a preprocessor (CPP) conditional around the optional exports is a better way then, you will have to build differently but you won't have to create another module and expose extra symbols? -harendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 09:23:36 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 11:23:36 +0200 Subject: Trees that Grow and constraints Message-ID: At the moment the Trees that Grow implementation in GHC master makes use of massive constraint types to provide Data instances for the hsSyn AST. I am trying to remove the need for this, and have hit a problem. The example I have reduced it to is here [1] The essence of the problem is ------------------- data Experiment p = Experiment { e_ext :: (XEOverLit p), e_val :: Int } | XExperiment (XXOverLit p) deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) type family XEOverLit x type family XXOverLit x -- The following line works -- type instance XEOverLit (GhcPass _) = PlaceHolder -- The following 3 lines do noe type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder type instance XXOverLit (GhcPass _) = PlaceHolder ------------------- Where specifying the type instance with a wild card keeps GHC happy (the XXOverLit case), but specifying for each of the three constructors for pass does not (the XEOverLit case) The exact error message is ------------------------------ Experiment.hs:34:1: error: • Could not deduce (Data (XEOverLit (GhcPass p))) arising from a use of ‘k’ from the context: Data (GhcPass p) bound by the instance declaration at Experiment.hs:34:1-69 • In the first argument of ‘k’, namely ‘(z Experiment `k` a1)’ In the expression: ((z Experiment `k` a1) `k` a2) In an equation for ‘gfoldl’: gfoldl k z Experiment a1 a2 = ((z Experiment `k` a1) `k` a2) When typechecking the code for ‘gfoldl’ in a derived instance for ‘Data (Experiment (GhcPass p))’: To see the code I am typechecking, use -ddump-deriv | 34 | deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ------------------------------ Alan [1] http://lpaste.net/360017 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 10:01:28 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 12:01:28 +0200 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: And it seems that -------------------------------------- data Experiment p = Experiment { e_ext :: (XEOverLit p), e_val :: Int } | XExperiment (XXOverLit p) deriving instance (Data GhcPs) => Data (Experiment GhcPs) deriving instance (Data GhcRn) => Data (Experiment GhcRn) deriving instance (Data GhcTc) => Data (Experiment GhcTc) ---------------------------------------- works, but only for GHC 8.2.1, not GHC 8.0.2 Alan On 13 November 2017 at 11:23, Alan & Kim Zimmerman wrote: > At the moment the Trees that Grow implementation in GHC master makes use > of massive constraint types to provide Data instances for the hsSyn AST. > > I am trying to remove the need for this, and have hit a problem. > > The example I have reduced it to is here [1] > > The essence of the problem is > > ------------------- > data Experiment p > = Experiment { > e_ext :: (XEOverLit p), > e_val :: Int } > | XExperiment (XXOverLit p) > deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) > > type family XEOverLit x > type family XXOverLit x > > -- The following line works > -- type instance XEOverLit (GhcPass _) = PlaceHolder > > -- The following 3 lines do noe > type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder > > type instance XXOverLit (GhcPass _) = PlaceHolder > ------------------- > > Where specifying the type instance with a wild card keeps GHC happy (the > XXOverLit case), but specifying for each of the three constructors for pass > does not (the XEOverLit case) > > The exact error message is > > ------------------------------ > Experiment.hs:34:1: error: > • Could not deduce (Data (XEOverLit (GhcPass p))) > arising from a use of ‘k’ > from the context: Data (GhcPass p) > bound by the instance declaration at Experiment.hs:34:1-69 > • In the first argument of ‘k’, namely ‘(z Experiment `k` a1)’ > In the expression: ((z Experiment `k` a1) `k` a2) > In an equation for ‘gfoldl’: > gfoldl k z Experiment a1 a2 = ((z Experiment `k` a1) `k` a2) > When typechecking the code for ‘gfoldl’ > in a derived instance for ‘Data (Experiment (GhcPass p))’: > To see the code I am typechecking, use -ddump-deriv > | > 34 | deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > ------------------------------ > > Alan > > [1] http://lpaste.net/360017 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Nov 13 12:26:00 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 13 Nov 2017 12:26:00 +0000 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: Where specifying the type instance with a wild card keeps GHC happy (the XXOverLit case), but specifying for each of the three constructors for pass does not (the XEOverLit case) Well, of course! The derived data instance looks something like instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) where gfoldl = ….Needs (Data (XEOverLit (GhcPass p)))… How can GHC solve the wanted constraint Data (XEOverlit (GhcPass p)) With three equations, it can’t. With one, it can. So why not use one? Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Alan & Kim Zimmerman Sent: 13 November 2017 09:24 To: ghc-devs at haskell.org Subject: Trees that Grow and constraints At the moment the Trees that Grow implementation in GHC master makes use of massive constraint types to provide Data instances for the hsSyn AST. I am trying to remove the need for this, and have hit a problem. The example I have reduced it to is here [1] The essence of the problem is ------------------- data Experiment p = Experiment { e_ext :: (XEOverLit p), e_val :: Int } | XExperiment (XXOverLit p) deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) type family XEOverLit x type family XXOverLit x -- The following line works -- type instance XEOverLit (GhcPass _) = PlaceHolder -- The following 3 lines do noe type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder type instance XXOverLit (GhcPass _) = PlaceHolder ------------------- Where specifying the type instance with a wild card keeps GHC happy (the XXOverLit case), but specifying for each of the three constructors for pass does not (the XEOverLit case) The exact error message is ------------------------------ Experiment.hs:34:1: error: • Could not deduce (Data (XEOverLit (GhcPass p))) arising from a use of ‘k’ from the context: Data (GhcPass p) bound by the instance declaration at Experiment.hs:34:1-69 • In the first argument of ‘k’, namely ‘(z Experiment `k` a1)’ In the expression: ((z Experiment `k` a1) `k` a2) In an equation for ‘gfoldl’: gfoldl k z Experiment a1 a2 = ((z Experiment `k` a1) `k` a2) When typechecking the code for ‘gfoldl’ in a derived instance for ‘Data (Experiment (GhcPass p))’: To see the code I am typechecking, use -ddump-deriv | 34 | deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ------------------------------ Alan [1] http://lpaste.net/360017 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 12:48:49 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 14:48:49 +0200 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: > So why not use one? > > > > Simon > > If I do instance (Data p) => Data (Experiment p) then GHC does not know that the type instances for type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder apply. Or do I misunderstand your advice? Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 14:05:46 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 16:05:46 +0200 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: Just to clarify, this example is a simplification, in practice we would be applying different type for each type instance e.g. type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder type instance XEOverLit (GhcPass 'Typechecked) = Type (modelling existing PostTc) Alan On 13 November 2017 at 11:23, Alan & Kim Zimmerman wrote: > At the moment the Trees that Grow implementation in GHC master makes use > of massive constraint types to provide Data instances for the hsSyn AST. > > I am trying to remove the need for this, and have hit a problem. > > The example I have reduced it to is here [1] > > The essence of the problem is > > ------------------- > data Experiment p > = Experiment { > e_ext :: (XEOverLit p), > e_val :: Int } > | XExperiment (XXOverLit p) > deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) > > type family XEOverLit x > type family XXOverLit x > > -- The following line works > -- type instance XEOverLit (GhcPass _) = PlaceHolder > > -- The following 3 lines do noe > type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder > > type instance XXOverLit (GhcPass _) = PlaceHolder > ------------------- > > Where specifying the type instance with a wild card keeps GHC happy (the > XXOverLit case), but specifying for each of the three constructors for pass > does not (the XEOverLit case) > > The exact error message is > > ------------------------------ > Experiment.hs:34:1: error: > • Could not deduce (Data (XEOverLit (GhcPass p))) > arising from a use of ‘k’ > from the context: Data (GhcPass p) > bound by the instance declaration at Experiment.hs:34:1-69 > • In the first argument of ‘k’, namely ‘(z Experiment `k` a1)’ > In the expression: ((z Experiment `k` a1) `k` a2) > In an equation for ‘gfoldl’: > gfoldl k z Experiment a1 a2 = ((z Experiment `k` a1) `k` a2) > When typechecking the code for ‘gfoldl’ > in a derived instance for ‘Data (Experiment (GhcPass p))’: > To see the code I am typechecking, use -ddump-deriv > | > 34 | deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > ------------------------------ > > Alan > > [1] http://lpaste.net/360017 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Mon Nov 13 14:35:12 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 13 Nov 2017 09:35:12 -0500 Subject: [commit: ghc] master: WIP on combined Step 1 and 3 for Trees That Grow, HsExpr (e3ec2e7) In-Reply-To: <55726148-679B-4445-8D57-38CC56170376@justtesting.org> References: <20171111211700.1CCCE3A5EA@ghc.haskell.org> <1510455817.5571.2.camel@joachim-breitner.de> <55726148-679B-4445-8D57-38CC56170376@justtesting.org> Message-ID: <87y3nal019.fsf@ben-laptop.smart-cactus.org> Manuel M T Chakravarty writes: >> I noted this on D4177 and discussed the effect with Alan. Indeed there is quite a sizeable regression in compilation time but thankfully this is not because GHC itself is slower. Rather, it simply requires more work to compile. I did a set of nofib runs with and without the first TTG patch and found that compiler allocations remained essentially unchanged. >> >> A 15% regression in the compilation time of GHC is indeed hard to stomach but Alan had said that much of this will likely disappear in the future. If this is the case then a temporary regression is in my opinion acceptable. > > Hmm, on what grounds does he think that this is going to disappear and how likely is likely? This doesn’t sound convincing TBH. > As Alan has pointed out elsewhere, currently the TTG involves some unnecessarily large constraints which very likely inflate typechecking time. This is almost certainly why we are seeing such increases in compilation time of the compiler (and, as importantly, no changes in the performance characteristics of the resulting compiler). Given how consistent the story seems to be, a reduction after the constraints are simplified sounds very likely. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From allbery.b at gmail.com Mon Nov 13 15:50:27 2017 From: allbery.b at gmail.com (Brandon Allbery) Date: Mon, 13 Nov 2017 10:50:27 -0500 Subject: Dynamically choosing the main function In-Reply-To: References: <87o9o7mlqf.fsf@ben-laptop.smart-cactus.org> <87efp3m9ba.fsf@ben-laptop.smart-cactus.org> Message-ID: On Mon, Nov 13, 2017 at 2:46 AM, Harendra Kumar wrote: > Also, the symbols are anyway exposed to the users, we just ask the users > to not look at those. > Only if you built a dynamic executable, or built for debugging. Default static executables are stripped. -- brandon s allbery kf8nh sine nomine associates allbery.b at gmail.com ballbery at sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From sh.najd at gmail.com Mon Nov 13 16:36:14 2017 From: sh.najd at gmail.com (Shayan Najd) Date: Mon, 13 Nov 2017 16:36:14 +0000 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: Isn't the solution always if generic programming makes things complicated, avoid it! Here generic programming is where you define instances parametric on the phase index. Why not defining three instances of the type class, one per each phase? Yes, we get code duplication (which in this case is still not much as we use automatic deriving), but the compilation gets faster (which is the motivation), like [2]. [2] http://lpaste.net/360019 /Shayan On Mon, Nov 13, 2017 at 2:05 PM, Alan & Kim Zimmerman wrote: > Just to clarify, this example is a simplification, in practice we would be > applying different type for each type instance > > e.g. > > type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Typechecked) = Type > > (modelling existing PostTc) > > Alan > > On 13 November 2017 at 11:23, Alan & Kim Zimmerman > wrote: > >> At the moment the Trees that Grow implementation in GHC master makes use >> of massive constraint types to provide Data instances for the hsSyn AST. >> >> I am trying to remove the need for this, and have hit a problem. >> >> The example I have reduced it to is here [1] >> >> The essence of the problem is >> >> ------------------- >> data Experiment p >> = Experiment { >> e_ext :: (XEOverLit p), >> e_val :: Int } >> | XExperiment (XXOverLit p) >> deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) >> >> type family XEOverLit x >> type family XXOverLit x >> >> -- The following line works >> -- type instance XEOverLit (GhcPass _) = PlaceHolder >> >> -- The following 3 lines do noe >> type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder >> type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder >> type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder >> >> type instance XXOverLit (GhcPass _) = PlaceHolder >> ------------------- >> >> Where specifying the type instance with a wild card keeps GHC happy (the >> XXOverLit case), but specifying for each of the three constructors for pass >> does not (the XEOverLit case) >> >> The exact error message is >> >> ------------------------------ >> Experiment.hs:34:1: error: >> • Could not deduce (Data (XEOverLit (GhcPass p))) >> arising from a use of ‘k’ >> from the context: Data (GhcPass p) >> bound by the instance declaration at Experiment.hs:34:1-69 >> • In the first argument of ‘k’, namely ‘(z Experiment `k` a1)’ >> In the expression: ((z Experiment `k` a1) `k` a2) >> In an equation for ‘gfoldl’: >> gfoldl k z Experiment a1 a2 = ((z Experiment `k` a1) `k` a2) >> When typechecking the code for ‘gfoldl’ >> in a derived instance for ‘Data (Experiment (GhcPass p))’: >> To see the code I am typechecking, use -ddump-deriv >> | >> 34 | deriving instance (Data (GhcPass p)) => Data (Experiment (GhcPass p)) >> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> ^^^^^^^^^ >> ------------------------------ >> >> Alan >> >> [1] http://lpaste.net/360017 >> >> >> > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Nov 13 17:08:12 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 13 Nov 2017 17:08:12 +0000 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: Alan (adding Shayan and ghc-devs) OK. It’s hard to keep this straight in email. Take a look at https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/Instances Please edit and improve it. Simon From: Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] Sent: 13 November 2017 13:30 To: Simon Peyton Jones Subject: Re: Trees that Grow and constraints At the moment, in GHC master, we have data HsOverLit p = OverLit { ol_ext :: (XOverLit p), ol_val :: OverLitVal, ol_witness :: HsExpr p} -- Note [Overloaded literal witnesses] | XOverLit (XXOverLit p) deriving instance (DataIdLR p p) => Data (HsOverLit p) And in HsExtension.hs we have an ever-growing constraint type defining DataIdLR I am trying to remove the need for that. In the Experiment.hs file, I found that using data Experiment p = Experiment { e_ext :: (XEOverLit p), e_val :: Int } | XExperiment (XXOverLit p) deriving instance Data (GhcPass 'Parsed ) => Data (Experiment (GhcPass 'Parsed )) deriving instance Data (GhcPass 'Renamed ) => Data (Experiment (GhcPass 'Renamed )) deriving instance Data (GhcPass 'Typechecked) => Data (Experiment (GhcPass 'Typechecked)) will compile using GHC 8.2.1, but not with GHC 8.0.2 Alan On 13 November 2017 at 15:13, Simon Peyton Jones > wrote: And it looks like it could work, when bootstrapping from GHC 8.2.1, but this is still a long time away. I’m sorry, I still don’t know what “it” is that “could work”. Can you be more precise. I think you must be suggesting some alternative to my code below? Simon From: Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] Sent: 13 November 2017 13:09 To: Simon Peyton Jones > Subject: Re: Trees that Grow and constraints Yes. If we can solve this, it means we can get rid of the DataId and ForallXXX constraints defined in hsSyn/HsExtension.hs And also move the type family definitions to the files where they are used. I suspect that typechecking those constraint sets is adding to the GHC compilation time, significantly And it looks like it could work, when bootstrapping from GHC 8.2.1, but this is still a long time away. Alan On 13 November 2017 at 15:05, Simon Peyton Jones > wrote: That’s not a problem you are trying to solve – it’s just some code 😊. Let me hazard a guess. You want a derived instance for Data (OverLit (GhcPass p)) But to do that of course we’ll need Data (XEOverLit (GhcPass p)). We can’t possibly have that at the time of writing the instance declaration, because p is universally quantified. So we are stuck with instance (Data (XEOverLit (GhcPass p))) => Data (OverLit (GhcPass p)) where… and the context gets a constraint for every extension field in OverLit, which is painful. So we have a solution bit contexts, but it’s painful. Is that the problem you are trying to solve? Simon From: Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] Sent: 13 November 2017 13:01 To: Simon Peyton Jones > Subject: Re: Trees that Grow and constraints Sorry, I simplified the problem. The actual one I am trying to solve will be the general case, so e.g. type instance XEOverLit (GhcPass 'Parsed ) = RdrName type instance XEOverLit (GhcPass 'Renamed ) = Name type instance XEOverLit (GhcPass 'Typechecked) = Id or more likely, modelling the existing PostTc and PostRn usages. And I suppose looking into using them in a single definition might work Alan On 13 November 2017 at 14:55, Simon Peyton Jones > wrote: Or do I misunderstand your advice? Well, you said Where specifying the type instance with a wild card keeps GHC happy (the XXOverLit case), but specifying for each of the three constructors for pass does not (the XEOverLit case) My question is: why not just use the wildcard, in that case? Do you want to re-state the problem you are trying to solve? Simon From: Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] Sent: 13 November 2017 12:49 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: Trees that Grow and constraints So why not use one? Simon If I do instance (Data p) => Data (Experiment p) then GHC does not know that the type instances for type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder apply. Or do I misunderstand your advice? Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 17:34:31 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 19:34:31 +0200 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: Will do Alan On 13 November 2017 at 19:08, Simon Peyton Jones wrote: > Alan (adding Shayan and ghc-devs) > > > > OK. It’s hard to keep this straight in email. Take a look at > > https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/Instances > > > > Please edit and improve it. > > > > Simon > > > > > > *From:* Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] > *Sent:* 13 November 2017 13:30 > > *To:* Simon Peyton Jones > *Subject:* Re: Trees that Grow and constraints > > > > At the moment, in GHC master, we have > > data HsOverLit p > > = OverLit { > > ol_ext :: (XOverLit p), > > ol_val :: OverLitVal, > > ol_witness :: HsExpr p} -- Note [Overloaded literal witnesses] > > > > | XOverLit > > (XXOverLit p) > > deriving instance (DataIdLR p p) => Data (HsOverLit p) > > > > And in HsExtension.hs we have an ever-growing constraint type defining DataIdLR > > I am trying to remove the need for that. > > In the Experiment.hs file, I found that using > > data Experiment p > = Experiment { > e_ext :: (XEOverLit p), > e_val :: Int } > | XExperiment (XXOverLit p) > deriving instance Data (GhcPass 'Parsed ) => Data (Experiment (GhcPass 'Parsed )) > deriving instance Data (GhcPass 'Renamed ) => Data (Experiment (GhcPass 'Renamed )) > deriving instance Data (GhcPass 'Typechecked) => Data (Experiment (GhcPass 'Typechecked)) > > will compile using GHC 8.2.1, but not with GHC 8.0.2 > > > > Alan > > > > On 13 November 2017 at 15:13, Simon Peyton Jones > wrote: > > And it looks like it could work, when bootstrapping from GHC 8.2.1, but > this is still a long time away. > > > > I’m sorry, I still don’t know what “it” is that “could work”. Can you be > more precise. I think you must be suggesting some alternative to my code > below? > > > > Simon > > > > *From:* Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] > *Sent:* 13 November 2017 13:09 > > > *To:* Simon Peyton Jones > *Subject:* Re: Trees that Grow and constraints > > > > Yes. > > > > If we can solve this, it means we can get rid of the DataId and ForallXXX > constraints defined in hsSyn/HsExtension.hs > > > > And also move the type family definitions to the files where they are used. > > > > I suspect that typechecking those constraint sets is adding to the GHC > compilation time, significantly > > > > And it looks like it could work, when bootstrapping from GHC 8.2.1, but > this is still a long time away. > > > > Alan > > > > On 13 November 2017 at 15:05, Simon Peyton Jones > wrote: > > That’s not a problem you are trying to solve – it’s just some code 😊. > > > > Let me hazard a guess. You want a derived instance for > > > > Data (OverLit (GhcPass p)) > > > > But to do that of course we’ll need Data (XEOverLit (GhcPass p)). > > > > We can’t possibly have that at the time of writing the instance > declaration, because p is universally quantified. So we are stuck with > > > > instance (Data (XEOverLit (GhcPass p))) => Data (OverLit > (GhcPass p)) where… > > > > and the context gets a constraint for every extension field in OverLit, > which is painful. > > > > So we have a solution bit contexts, but it’s painful. > > > > Is that the problem you are trying to solve? > > > > Simon > > > > *From:* Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] > *Sent:* 13 November 2017 13:01 > *To:* Simon Peyton Jones > > *Subject:* Re: Trees that Grow and constraints > > > > Sorry, I simplified the problem. > > The actual one I am trying to solve will be the general case, so e.g. > > type instance XEOverLit (GhcPass 'Parsed ) = RdrName > type instance XEOverLit (GhcPass 'Renamed ) = Name > type instance XEOverLit (GhcPass 'Typechecked) = Id > > or more likely, modelling the existing PostTc and PostRn usages. > > And I suppose looking into using them in a single definition might work > > Alan > > > > > > On 13 November 2017 at 14:55, Simon Peyton Jones > wrote: > > Or do I misunderstand your advice? > > > > Well, you said > > Where specifying the type instance with a wild card keeps GHC happy (the > XXOverLit case), but specifying for each of the three constructors for pass > does not (the XEOverLit case) > > My question is: why not just use the wildcard, in that case? > > Do you want to re-state the problem you are trying to solve? > > > > Simon > > > > > > *From:* Alan & Kim Zimmerman [mailto:alan.zimm at gmail.com] > *Sent:* 13 November 2017 12:49 > *To:* Simon Peyton Jones > *Cc:* ghc-devs at haskell.org > *Subject:* Re: Trees that Grow and constraints > > > > > > So why not use one? > > > > Simon > > > > If I do > > instance (Data p) => Data (Experiment p) > > then GHC does not know that the type instances for > > > type instance XEOverLit (GhcPass 'Parsed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Renamed ) = PlaceHolder > type instance XEOverLit (GhcPass 'Typechecked) = PlaceHolder > > apply. > > > > Or do I misunderstand your advice? > > > > Alan > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon Nov 13 19:36:34 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 13 Nov 2017 21:36:34 +0200 Subject: Trees that Grow and constraints In-Reply-To: References: Message-ID: I have updated the page, with a bit more detail and an additional plan > OK. It’s hard to keep this straight in email. Take a look at >> >> https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow/Instances >> >> >> >> Please edit and improve it. >> >> Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From douglas.wilson at gmail.com Tue Nov 14 22:34:25 2017 From: douglas.wilson at gmail.com (Douglas Wilson) Date: Wed, 15 Nov 2017 11:34:25 +1300 Subject: New core profiling mode Message-ID: Hi ghc-devs, I've been working on a new mode of adding cost-centres to programs and I'd like to ask some questions and solicit some feedback. The code is here [fn:1], it works, provided one enables -fprof-core on all modules. I've recently been trying to pick some low hanging fruit from ghc compilation performance. A common frustration was in the difference between profiled and non-profiled builds. Often I thought had I found a problem in the profiled build, only to find it was optimized away in the non-profiled build. Several times an issue was tail-calls not happening in profiled builds. To solve this problem I've been working on a new way of inserting cost-centres: adding them to core after simplification (currently at the end of corePrepPgm) rather than adding them to HsSyn before simplification. This makes it harder to map cost-centres into source code (You have to -ddump-prep currently), but in exchange you are profiling the same core program as the non-profilied build. I intend to investigate whether I can use SourceNotes to create SrcSpans for the generated cost-centres to somewhat alleviate the need to inspect dumped core. There are several new flags: -fprof-core: Enables the aforementioned mode. This is mutually exclusive with -fprof-auto etc. -fprof-core-drop-ticks: Non-user ticks are dropped from unfoldings(though I don't know how to do this yet). -fprof-core-tick-binds: ticks are inserted around the RHS of bindings (except top-level unlifted bindings). -fprof-core-tick-cases: ticks are inserted around the scrutinees of cases. -fprof-core-tick-alts: ticks are inserted around Alt expressions (unless there is only one). Some questions: I need to strip (probably only non-user) ticks out of unfoldings before they are substituted into a module that uses -fprof-core. Where is the right place to do this? I need inlining to proceed exactly as if the ticks were not present, however I don't want to strip ticks when the unfoldings are created as other modules may still need them. Is the end of corePrepPgm the right place to insert the cost-centres? I chose it because it can't affect any core optimizations if it's last, but perhaps it could be earlier, or perhaps it needs to act on Stg? Do you have any examples of programs for which existing profiling tools are inadequate due to how cost-centres affect simplification? There is an example in #12893 but something self-contained would be great! Regards, Doug Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed Nov 15 09:46:25 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 15 Nov 2017 09:46:25 +0000 Subject: New core profiling mode In-Reply-To: References: Message-ID: Doug Good stuff. Do you know about ticky-ticky profiling? It is /guaranteed/ not to affect optimisation. Moreover you can mix modules compiled with and without profiling. But it’s very low level. I had not thought of doing /cost-centre/ profiling after optimisation. That’s an interesting thought. You’d still have to compile the entire program with profiling (because runtime representations change). But you’d get the call-stack information etc that cost centres give you, but with a guarantee that optimisation was unaffected. It’s a sort of middle ground. Is the end of corePrepPgm the right place to insert the cost-centres? I chose it because it can't affect any core optimizations if it's last, but perhaps it could be earlier, or perhaps it needs to act on Stg? I’d be inclined to do it on STG, just before codegen. Because it really is a codgen tactic “compile this with cost centres please”. I need to strip (probably only non-user) ticks out of unfoldings before they are substituted into a module that uses -fprof-core. Why? If you are only adding these cost centres after CorePrep, or in STG, they will never show up in unfoldings anyway. Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Douglas Wilson Sent: 14 November 2017 22:34 To: ghc-devs at haskell.org Subject: New core profiling mode Hi ghc-devs, I've been working on a new mode of adding cost-centres to programs and I'd like to ask some questions and solicit some feedback. The code is here [fn:1], it works, provided one enables -fprof-core on all modules. I've recently been trying to pick some low hanging fruit from ghc compilation performance. A common frustration was in the difference between profiled and non-profiled builds. Often I thought had I found a problem in the profiled build, only to find it was optimized away in the non-profiled build. Several times an issue was tail-calls not happening in profiled builds. To solve this problem I've been working on a new way of inserting cost-centres: adding them to core after simplification (currently at the end of corePrepPgm) rather than adding them to HsSyn before simplification. This makes it harder to map cost-centres into source code (You have to -ddump-prep currently), but in exchange you are profiling the same core program as the non-profilied build. I intend to investigate whether I can use SourceNotes to create SrcSpans for the generated cost-centres to somewhat alleviate the need to inspect dumped core. There are several new flags: -fprof-core: Enables the aforementioned mode. This is mutually exclusive with -fprof-auto etc. -fprof-core-drop-ticks: Non-user ticks are dropped from unfoldings(though I don't know how to do this yet). -fprof-core-tick-binds: ticks are inserted around the RHS of bindings (except top-level unlifted bindings). -fprof-core-tick-cases: ticks are inserted around the scrutinees of cases. -fprof-core-tick-alts: ticks are inserted around Alt expressions (unless there is only one). Some questions: I need to strip (probably only non-user) ticks out of unfoldings before they are substituted into a module that uses -fprof-core. Where is the right place to do this? I need inlining to proceed exactly as if the ticks were not present, however I don't want to strip ticks when the unfoldings are created as other modules may still need them. Is the end of corePrepPgm the right place to insert the cost-centres? I chose it because it can't affect any core optimizations if it's last, but perhaps it could be earlier, or perhaps it needs to act on Stg? Do you have any examples of programs for which existing profiling tools are inadequate due to how cost-centres affect simplification? There is an example in #12893 but something self-contained would be great! Regards, Doug Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Wed Nov 15 16:59:52 2017 From: ben at well-typed.com (Ben Gamari) Date: Wed, 15 Nov 2017 11:59:52 -0500 Subject: New core profiling mode In-Reply-To: References: Message-ID: <877eurlbp9.fsf@ben-laptop.smart-cactus.org> Douglas Wilson writes: > Hi ghc-devs, > > I've been working on a new mode of adding cost-centres to programs and I'd > like to ask some questions and solicit some feedback. The code is here > [fn:1], > it works, provided one enables -fprof-core on all modules. > > I've recently been trying to pick some low hanging fruit from ghc > compilation > performance. A common frustration was in the difference between profiled and > non-profiled builds. Often I thought had I found a problem in the profiled > build, only to find it was optimized away in the non-profiled build. Several > times an issue was tail-calls not happening in profiled builds. > > To solve this problem I've been working on a new way of inserting > cost-centres: adding them to core after simplification (currently at > the end of corePrepPgm) rather than adding them to HsSyn before > simplification. This makes it harder to map cost-centres into source > code (You have to -ddump-prep currently), but in > exchange you are profiling the same core program as the non-profilied build. > Sounds quite reasonable to me. > I intend to investigate whether I can use SourceNotes to create > SrcSpans for the generated cost-centres to somewhat alleviate the need > to inspect dumped core. Yes, I think this would be a great idea. I have also wondered in the past whether it would be worthwhile (optionally?) adding a CSS-like name SourceNote (e.g. in addition to a SrcSpan, include the name of the containing binding; for instance "insert.go"). This would make it a bit easier to understand a backtrace at-a-glance. > Do you have any examples of programs for which existing profiling tools are > inadequate due to how cost-centres affect simplification? There is an > example in #12893 but something self-contained would be great! > I'm afraid I don't know of any off-hand. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From andrey.mokhov at newcastle.ac.uk Thu Nov 16 01:07:05 2017 From: andrey.mokhov at newcastle.ac.uk (Andrey Mokhov) Date: Thu, 16 Nov 2017 01:07:05 +0000 Subject: [ANNOUNCE] Hadrian update References: Message-ID: Dear GHC developers, As some of you might have already noticed, Hadrian has finally been merged into the GHC tree. However it's not yet time to celebrate - there are still many issues that need to be addressed before the Make-based build system may retire. Have a look at the README if you'd like a quick try: https://github.com/ghc/ghc/tree/master/hadrian. Here is a quick update on the on-going development: 1) Hadrian can build GHC and can already be used as part of the CI infrastructure. However, the resulting binary does not pass the validation. Zhen Zhang is looking into this, but more help is needed. See https://github.com/snowleopard/hadrian/issues/299. 2) A major refactoring by Moritz Angermann is on the way. Moritz is primarily interested in cross compilation, but to make it work he had to get rid of the ghc-cabal utility, reorganise the build tree, and make numerous other improvements to Hadrian. See https://github.com/snowleopard/hadrian/pull/445. 3) There is currently no support for binary distribution. Ben Gamari is looking into this issue: https://github.com/snowleopard/hadrian/issues/219. 4) Dynamic linking on Windows is not implemented. Tamar Christina has kindly offered help with this: https://github.com/snowleopard/hadrian/issues/343. 5) Hadrian source code is still not fully documented and tested, and generally requires some polishing. I am currently taking care of this when not distracted by urgent bug fixes and will appreciate your help in making Hadrian easier to understand and use. I can't believe that we seem to approach the finish line! It's been a long, tedious but also interesting project. Thank you all for helping us get this far, and I hope we'll celebrate the switch from Make to Hadrian soon. Cheers, Andrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Thu Nov 16 01:22:40 2017 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 16 Nov 2017 09:22:40 +0800 Subject: [ANNOUNCE] Hadrian update In-Reply-To: References: Message-ID: <8005F7A7-AB0A-468C-B599-007A62BC13AB@gmail.com> Congrats Andrey, > 2) A major refactoring by Moritz Angermann is on the way. Moritz is primarily interested in cross compilation, but to make it work he had to get rid of the ghc-cabal utility, reorganise the build tree, and make numerous other improvements to Hadrian. See https://github.com/snowleopard/hadrian/pull/445. I’ll post some more details on this next week. Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at well-typed.com Thu Nov 16 01:31:55 2017 From: david at well-typed.com (David Feuer) Date: Wed, 15 Nov 2017 20:31:55 -0500 Subject: Reasoning backwards with type families Message-ID: <4608716.0D0gEhqDuK@squirrel> Sometimes it woulld be useful to be able to reason backwards about type families. For example, we have type family a && b where 'False && b = 'False 'True && b = b a && 'False = 'False a && 'True = a a && a = a If we know something about either of the arguments, we can choose an equation and learn something about the result. But if we know something about the *result*, GHC doesn't give us any way to learn anything about the arguments. For (&&), the obvious things you'd want are prj1 :: (a && b) ~ 'True => a :~: 'True prj2 :: (a && b) ~ 'True => b :~: 'True flop :: (a && b) ~ 'False => (Not a || Not B) :~: 'True There's nothing inherently impossible about this sort of reasoning. In order for the constraint (a && b) ~ 'True to hold, the type family application *must have reduced* using one of its equations. The only possibilities are 'True && b = b a && 'True = a a && a = a Substituting 'True for each RHS gives 'True && 'True = 'True 'True && 'True = 'True 'True && 'True = 'True So in each case, the first argument is forced to 'True. Similarly, if (a && b) ~ 'False, we look at all the possibilities: 'False && a = 'False 'True && a = a a && 'False = 'False a && 'True = a a && a = a Substituting 'False for each RHS, 'False && a = 'False 'True && 'False = 'False a && 'False = 'False 'False && 'True = 'False 'False && 'False = 'False and we can calculate (Not a || Not b) as 'True for each of these LHSes. Now consider (==), which is recursive: type family (a :: k) == (b :: k) :: Bool where f a == g b = f == g && a == b a == a = 'True _ == _ = 'False We'd really like to know that eqEqual :: (a == b) ~ 'True => a :~: b :: For eqEqual, we can reason thus: if (a == b) ~ 'True, then we obtained that result from one of the equations f a == g b = f == g && a == b a == a = 'True In the base case, a ~ a on the LHS. In the recursive case, the RHS being 'True tells us (using the reasoning for (&&)) that (f == g) ~ True and that (a == b) ~ True. Inductively, we conclude that f :~: g and a :~: b. Shifting to the LHS, we conclude that f a ~ g b. I wouldn't necessarily expect GHC to be able to work something like this out on its own. But it seems like there should be some way to allow the user to guide it to a proof. A tantalizing feature is that we're essentially reasoning about execution traces, which are necessarily finite. It at least *seems* that this should mean that the induction is well-founded. Now why should anyone care about this sort of business? Because the runtime representation of, say, (a == b && c /= d) :~: 'True is smaller than the runtime representation of (a :~: b, c :~: d -> Void). So it would be nice to be able to recover the latter from the former. Can we do it? David From simonpj at microsoft.com Thu Nov 16 13:08:32 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 16 Nov 2017 13:08:32 +0000 Subject: GHC on Ubuntu Message-ID: The GHC distribution for Ubuntu appears to be 7.10, very old. Do we (or they) bring on new versions? https://packages.ubuntu.com/search?keywords=ghc Here we use the LTS versions of Ubuntu, which seems stuck on 7.10. Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Thu Nov 16 13:36:41 2017 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 16 Nov 2017 21:36:41 +0800 Subject: GHC on Ubuntu In-Reply-To: References: Message-ID: <827573A6-4D64-4A1E-AF5A-FA9CC95D353E@gmail.com> Hi Simon, Herbert provides quite a few GHC builds at https://launchpad.net/~hvr/+archive/ubuntu/ghc Looking at https://wiki.ubuntu.com/Releases, makes me believe LTS might be Xenial or Trusty. For both seem to be packages up to 8.2.2 and even HEAD available. Hope that helps. Cheers, Moritz > On Nov 16, 2017, at 9:08 PM, Simon Peyton Jones via ghc-devs wrote: > > The GHC distribution for Ubuntu appears to be 7.10, very old. Do we (or they) bring on new versions? > > https://packages.ubuntu.com/search?keywords=ghc > > Here we use the LTS versions of Ubuntu, which seems stuck on 7.10. > > Simon > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From hvriedel at gmail.com Fri Nov 17 13:08:11 2017 From: hvriedel at gmail.com (Herbert Valerio Riedel) Date: Fri, 17 Nov 2017 14:08:11 +0100 Subject: Observation on Hadrian's relative performance re current buildsystem Message-ID: <87a7zloxxg.fsf@gmail.com> Hello GHC devs, I took the opportunity to give Hadrian a test-run to see whether it could live up to the big promise of delivering a "more scalable, faster" system than the current GNU Make based system. Unfortunately, my preliminary results don't back this claim, and actually make Hadrian appear to be significantly slower. ---- Here's the summary of the results: | Hadrian | GNU Make | +------------------------+-----------+-----------+ | Compiling `hadrian` | 5m25s | 0 | | (one-time setup) | | | +------------------------+-----------+-----------+ | build "all" at -j8 | 38m | 33m | +------------------------+-----------+-----------+ | no-op build at -j8 | 10.977s | 3.258s | +------------------------+-----------+-----------+ | "clean" | 21s | 51s | +------------------------+-----------+-----------+ So, Hadrian is ~5 minutes than GNU Make (or even ~10 minutes if you also count the one-time setup cost). And what I personally consider a bit annoying is that it's ~3 times slower detecting; i.e. you have to wait 11s for Hadrian to detect there's nothing to be done which compared to GNU Make (which currently needlessly re-runs Sphinx; so it could be even faster!) is very noticeable to me. There's a silver-lining though, deleting files is the part which is a lot more costly in the GNU Make system currently since artifacts are spread over several (scroll to the end of this email) subfolders there. Whereas Hadrian did something we should have done for the GNU Make system as well (and probably would have done sooner or later anyway in order to support the srcdir!=buildir scheme that people are used to from GNU Autotools projects); Hadrian places build-artifacts into a few top-level folders, and so cleaning up is trivial and requires to unlink only a few folders from the filesystem. ---- At the very least, I'd expect Hadrian to be as fast as the GNU Make system (and ideally beat it, not the least as this was besides maintainability its big promise), but so far it doesn't seem to deliver that promise for me. It could easily be that I'm comparing apples to oranges here or that I've otherwise overlooked something, so let me describe how I came to this conclusion: I tried this on an reasonably idle Linux workstation with an Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz CPU, and with 32GiB RAM (i.e. the filesystem content was well-cached into memory; NB: a ghc source tree + compiled artifacts takes up about 4GiB on the filesystem). I've started from a fresh Git clone, i.e. git clone --recursive git://git.haskell.org/ghc.git followed by ./boot ./configure At this point, we're at the common point from which both the Hadrian and the GNU Make build-system would start diverging: For the Hadrian build-system, we need to pay for a one-time setup, since we need to build the `hadrian` executable (which requires to build the in-tree lib:Cabal as an in-place library): $ time ./hadrian/build.sh --help ... real 5m25.992s user 6m19.196s sys 0m6.079s I'm not too worried about this part, as there's a few tricks by which we could likely bring that down to about 2 minutes or so, and we mostly pay this setup-cost, when lib:Cabal and/or `hadrian` changes and requires to be recompiled. Now, after having made sure that the `hadrian` executable is fresh, I started the actual build: $ time ./hadrian/build.sh -j8 ... shakeArgsWith 0.000s 0% Function shake 0.178s 0% Database read 0.000s 0% With database 0.000s 0% Running rules 2338.398s 99% ========================= Pool finished (5261 threads, 8 max) 0.002s 0% Lint checking 0.111s 0% Total 2338.690s 100% Build completed in 38:59m real 38m59.626s user 219m7.421s sys 11m7.584s then I immediately re-issued the same command to test how long it takes to perform a no-op build: $ time ./hadrian/build.sh -j8 Up to date Up to date shakeArgsWith 0.000s 0% Function shake 0.183s 1% Database read 0.144s 1% With database 0.241s 2% Running rules 9.379s 93% ========================= Pool finished (4165 threads, 8 max) 0.004s 0% Lint checking 0.101s 1% Total 10.051s 100% Build completed in 0:11m real 0m10.977s user 0m19.399s sys 0m2.443s Same without -j8: $ time ./hadrian/build.sh Up to date Up to date shakeArgsWith 0.000s 0% Function shake 0.175s 2% Database read 0.123s 1% With database 0.197s 2% Running rules 7.710s 92% ========================= Pool finished (1 threads, 1 max) 0.003s 0% Lint checking 0.084s 1% Total 8.293s 100% Build completed in 0:09m real 0m9.196s user 0m8.656s sys 0m0.724s And finally clean it again: $ sync $ time ./hadrian/build.sh clean Up to date Up to date | Remove directory _build/stage0 | Remove directory _build/stage1 | Remove directory _build/stage2 | Remove directory _build/stage3 | Remove directory inplace/bin | Remove directory inplace/lib | Remove directory sdistprep | Remove Hadrian files... | Remove directory _build/generated | Done. shakeArgsWith 0.000s 0% Function shake 0.175s 0% Database read 0.114s 0% With database 0.205s 0% Running rules 20.037s 97% ======================== Pool finished (1 threads, 1 max) 0.002s 0% Lint checking 0.006s 0% Total 20.540s 100% Build completed in 0:21m real 0m21.426s user 0m1.415s sys 0m1.045s ---- Running the full build via our rusty GNU Make system: $ time make V=0 -j8 real 33m30.801s user 157m49.520s sys 6m49.289s A no-op build: $ time make V=0 -j8 ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all make -C utils/haddock/doc html SPHINX_BUILD=/usr/bin/sphinx-build /usr/bin/sphinx-build -b html . .build-html Running Sphinx v1.3.6 loading translations [en]... done loading pickled environment... done building [mo]: targets for 0 po files that are out of date building [html]: targets for 0 source files that are out of date updating environment: 0 added, 0 changed, 0 removed looking for now-outdated files... none found no targets are out of date. build succeeded. cp -R utils/haddock/doc/.build-html utils/haddock/doc/haddock real 0m3.258s user 0m3.134s sys 0m0.283s And finally `make clean`: $ sync $ time make clean make --no-print-directory -f ghc.mk clean CLEANING=YES "rm" -rf inplace/bin inplace/lib "rm" -rf utils/touchy/dist "rm" -rf inplace/lib/bin/touchy "rm" -rf utils/unlit/dist "rm" -rf inplace/lib/bin/unlit "rm" -rf utils/unlit/dist-install "rm" -rf utils/hp2ps/dist "rm" -rf inplace/bin/hp2ps "rm" -rf inplace/lib/bin/hp2ps "rm" -rf utils/hp2ps/dist-install "rm" -rf driver/split/dist inplace/lib/bin/ghc-split "rm" -rf utils/genprimopcode/dist "rm" -rf inplace/bin/genprimopcode "rm" -rf libffi/build libffi/stamp.ffi.static-shared.configure libffi/stamp.ffi.static-shared.build libffi/stamp.ffi.static-shared.install libffi/dist-install "rm" -rf utils/deriveConstants/dist "rm" -rf inplace/bin/deriveConstants "rm" -rf includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h "rm" -rf rts/dist "rm" -rf bindisttest/"install dir" bindisttest/HelloWorld bindisttest/HelloWorld.o bindisttest/HelloWorld.hi bindisttest/output "rm" -rf utils/genapply/dist "rm" -rf inplace/bin/genapply "rm" -rf libraries/integer-gmp/include/ghc-gmp.h libraries/integer-gmp/gmp/config.mk libraries/integer-gmp/gmp/libgmp.a libraries/integer-gmp/gmp/gmp.h libraries/integer-gmp/gmp/gmpbuild libraries/integer-gmp/gmp/gmp-6.1.2 "rm" -rf libraries/integer-gmp/gmp/objs "rm" -rf libraries/integer-gmp/gmp/gmpbuild "rm" -rf utils/haddock/dist "rm" -rf inplace/bin/haddock "rm" -rf inplace/lib/bin/haddock "rm" -rf compiler/stage1 "rm" -rf compiler/stage2 "rm" -rf compiler/stage3 "rm" -rf utils/hsc2hs/dist "rm" -rf inplace/bin/hsc2hs "rm" -rf inplace/lib/bin/hsc2hs "rm" -rf utils/hsc2hs/dist-install "rm" -rf utils/ghc-pkg/dist "rm" -rf inplace/bin/ghc-pkg "rm" -rf inplace/lib/bin/ghc-pkg "rm" -rf utils/ghc-pkg/dist-install "rm" -rf utils/ghctags/dist-install "rm" -rf inplace/bin/ghctags "rm" -rf utils/check-api-annotations/dist-install "rm" -rf inplace/bin/check-api-annotations "rm" -rf utils/check-ppr/dist-install "rm" -rf inplace/bin/check-ppr "rm" -rf utils/ghc-cabal/dist bootstrapping "rm" -rf utils/ghc-cabal/dist-install "rm" -rf utils/hpc/dist-install "rm" -rf inplace/bin/hpc "rm" -rf inplace/lib/bin/hpc "rm" -rf utils/runghc/dist-install "rm" -rf inplace/bin/runghc "rm" -rf inplace/lib/bin/runghc "rm" -rf ghc/stage1 "rm" -rf inplace/bin/ghc-stage1 "rm" -rf inplace/lib/bin/ghc-stage1 "rm" -rf ghc/stage2 "rm" -rf inplace/bin/ghc-stage2 "rm" -rf inplace/lib/bin/ghc-stage2 "rm" -rf ghc/stage3 "rm" -rf docs/users_guide/.doctrees-html/ docs/users_guide/.doctrees-pdf/ docs/users_guide/build-html/ docs/users_guide/build-pdf/ docs/users_guide/users_guide.pdf "rm" -rf docs/users_guide/.doctrees-man/ docs/users_guide/build-man/ "rm" -rf utils/count_lines/dist inplace/bin/count_lines "rm" -rf utils/compare_sizes/dist-install "rm" -rf iserv/stage2 "rm" -rf inplace/lib/bin/ghc-iserv "rm" -rf iserv/stage2_p "rm" -rf inplace/lib/bin/ghc-iserv-prof "rm" -rf iserv/stage2_dyn "rm" -rf inplace/lib/bin/ghc-iserv-dyn "rm" -f libraries/integer-gmp/include/HsIntegerGmp.h libraries/base/include/EventConfig.h mk/config.mk.old mk/project.mk.old compiler/ghc.cabal.old includes/GHCConstants.h includes/DerivedConstants.h includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h utils/ghc-pkg/Version.hs compiler/prelude/primops.txt "rm" -rf includes/dist-derivedconstants "rm" -rf inplace/bin "rm" -rf inplace/lib "rm" -rf libraries/bootstrapping.conf "rm" -f mk/are-validating.mk "rm" -rf libraries/ghc-boot-th/dist-install "rm" -rf libraries/ghc-boot/dist-install "rm" -rf libraries/ghci/dist-install "rm" -rf libraries/base/dist-install "rm" -rf libraries/ghc-prim/dist-install "rm" -rf libraries/integer-gmp/dist-install "rm" -rf libraries/integer-simple/dist-install "rm" -rf libraries/template-haskell/dist-install "rm" -rf libraries/array/dist-install "rm" -rf libraries/binary/dist-install "rm" -rf libraries/bytestring/dist-install "rm" -rf libraries/Cabal/Cabal/dist-install "rm" -rf libraries/ghc-compact/dist-install "rm" -rf libraries/containers/dist-install "rm" -rf libraries/deepseq/dist-install "rm" -rf libraries/directory/dist-install "rm" -rf libraries/filepath/dist-install "rm" -rf libraries/haskeline/dist-install "rm" -rf libraries/hpc/dist-install "rm" -rf libraries/mtl/dist-install "rm" -rf libraries/parsec/dist-install "rm" -rf libraries/pretty/dist-install "rm" -rf libraries/process/dist-install "rm" -rf libraries/terminfo/dist-install "rm" -rf libraries/text/dist-install "rm" -rf libraries/time/dist-install "rm" -rf libraries/transformers/dist-install "rm" -rf libraries/unix/dist-install "rm" -rf libraries/Win32/dist-install "rm" -rf libraries/xhtml/dist-install "rm" -rf libraries/parallel/dist-install "rm" -rf libraries/stm/dist-install "rm" -rf libraries/random/dist-install "rm" -rf libraries/primitive/dist-install "rm" -rf libraries/vector/dist-install "rm" -rf libraries/dph/dph-base/dist-install "rm" -rf libraries/dph/dph-prim-interface/dist-install "rm" -rf libraries/dph/dph-prim-seq/dist-install "rm" -rf libraries/dph/dph-prim-par/dist-install "rm" -rf libraries/dph/dph-lifted-base/dist-install "rm" -rf libraries/dph/dph-lifted-boxed/dist-install "rm" -rf libraries/dph/dph-lifted-copy/dist-install "rm" -rf libraries/dph/dph-lifted-vseg/dist-install "rm" -rf libraries/ghc-boot-th/dist-boot "rm" -rf libraries/ghc-boot/dist-boot "rm" -rf libraries/ghci/dist-boot "rm" -rf libraries/base/dist-boot "rm" -rf libraries/ghc-prim/dist-boot "rm" -rf libraries/integer-gmp/dist-boot "rm" -rf libraries/integer-simple/dist-boot "rm" -rf libraries/template-haskell/dist-boot "rm" -rf libraries/array/dist-boot "rm" -rf libraries/binary/dist-boot "rm" -rf libraries/bytestring/dist-boot "rm" -rf libraries/Cabal/Cabal/dist-boot "rm" -rf libraries/ghc-compact/dist-boot "rm" -rf libraries/containers/dist-boot "rm" -rf libraries/deepseq/dist-boot "rm" -rf libraries/directory/dist-boot "rm" -rf libraries/filepath/dist-boot "rm" -rf libraries/haskeline/dist-boot "rm" -rf libraries/hpc/dist-boot "rm" -rf libraries/mtl/dist-boot "rm" -rf libraries/parsec/dist-boot "rm" -rf libraries/pretty/dist-boot "rm" -rf libraries/process/dist-boot "rm" -rf libraries/terminfo/dist-boot "rm" -rf libraries/text/dist-boot "rm" -rf libraries/time/dist-boot "rm" -rf libraries/transformers/dist-boot "rm" -rf libraries/unix/dist-boot "rm" -rf libraries/Win32/dist-boot "rm" -rf libraries/xhtml/dist-boot "rm" -rf libraries/parallel/dist-boot "rm" -rf libraries/stm/dist-boot "rm" -rf libraries/random/dist-boot "rm" -rf libraries/primitive/dist-boot "rm" -rf libraries/vector/dist-boot "rm" -rf libraries/dph/dph-base/dist-boot "rm" -rf libraries/dph/dph-prim-interface/dist-boot "rm" -rf libraries/dph/dph-prim-seq/dist-boot "rm" -rf libraries/dph/dph-prim-par/dist-boot "rm" -rf libraries/dph/dph-lifted-base/dist-boot "rm" -rf libraries/dph/dph-lifted-boxed/dist-boot "rm" -rf libraries/dph/dph-lifted-copy/dist-boot "rm" -rf libraries/dph/dph-lifted-vseg/dist-boot "rm" -rf libraries/ghc-boot-th/dist libraries/ghc-boot/dist libraries/ghci/dist libraries/base/dist libraries/ghc-prim/dist libraries/integer-gmp/dist libraries/integer-simple/dist libraries/template-haskell/dist libraries/array/dist libraries/binary/dist libraries/bytestring/dist libraries/Cabal/Cabal/dist libraries/ghc-compact/dist libraries/containers/dist libraries/deepseq/dist libraries/directory/dist libraries/filepath/dist libraries/haskeline/dist libraries/hpc/dist libraries/mtl/dist libraries/parsec/dist libraries/pretty/dist libraries/process/dist libraries/terminfo/dist libraries/text/dist libraries/time/dist libraries/transformers/dist libraries/unix/dist libraries/Win32/dist libraries/xhtml/dist libraries/parallel/dist libraries/stm/dist libraries/random/dist libraries/primitive/dist libraries/vector/dist libraries/dph/dph-base/dist libraries/dph/dph-prim-interface/dist libraries/dph/dph-prim-seq/dist libraries/dph/dph-prim-par/dist libraries/dph/dph-lifted-base/dist libraries/dph/dph-lifted-boxed/dist libraries/dph/dph-lifted-copy/dist libraries/dph/dph-lifted-vseg/dist "rm" -f libraries/base/base.buildinfo libraries/integer-gmp/integer-gmp.buildinfo libraries/terminfo/terminfo.buildinfo libraries/unix/unix.buildinfo "rm" -f libraries/ghc-boot-th/config.log libraries/ghc-boot/config.log libraries/ghci/config.log libraries/base/config.log libraries/ghc-prim/config.log libraries/integer-gmp/config.log libraries/integer-simple/config.log libraries/template-haskell/config.log libraries/array/config.log libraries/binary/config.log libraries/bytestring/config.log libraries/Cabal/Cabal/config.log libraries/ghc-compact/config.log libraries/containers/config.log libraries/deepseq/config.log libraries/directory/config.log libraries/filepath/config.log libraries/haskeline/config.log libraries/hpc/config.log libraries/mtl/config.log libraries/parsec/config.log libraries/pretty/config.log libraries/process/config.log libraries/terminfo/config.log libraries/text/config.log libraries/time/config.log libraries/transformers/config.log libraries/unix/config.log libraries/Win32/config.log libraries/xhtml/config.log libraries/parallel/config.log libraries/stm/config.log libraries/random/config.log libraries/primitive/config.log libraries/vector/config.log libraries/dph/dph-base/config.log libraries/dph/dph-prim-interface/config.log libraries/dph/dph-prim-seq/config.log libraries/dph/dph-prim-par/config.log libraries/dph/dph-lifted-base/config.log libraries/dph/dph-lifted-boxed/config.log libraries/dph/dph-lifted-copy/config.log libraries/dph/dph-lifted-vseg/config.log "rm" -f libraries/ghc-boot-th/config.status libraries/ghc-boot/config.status libraries/ghci/config.status libraries/base/config.status libraries/ghc-prim/config.status libraries/integer-gmp/config.status libraries/integer-simple/config.status libraries/template-haskell/config.status libraries/array/config.status libraries/binary/config.status libraries/bytestring/config.status libraries/Cabal/Cabal/config.status libraries/ghc-compact/config.status libraries/containers/config.status libraries/deepseq/config.status libraries/directory/config.status libraries/filepath/config.status libraries/haskeline/config.status libraries/hpc/config.status libraries/mtl/config.status libraries/parsec/config.status libraries/pretty/config.status libraries/process/config.status libraries/terminfo/config.status libraries/text/config.status libraries/time/config.status libraries/transformers/config.status libraries/unix/config.status libraries/Win32/config.status libraries/xhtml/config.status libraries/parallel/config.status libraries/stm/config.status libraries/random/config.status libraries/primitive/config.status libraries/vector/config.status libraries/dph/dph-base/config.status libraries/dph/dph-prim-interface/config.status libraries/dph/dph-prim-seq/config.status libraries/dph/dph-prim-par/config.status libraries/dph/dph-lifted-base/config.status libraries/dph/dph-lifted-boxed/config.status libraries/dph/dph-lifted-copy/config.status libraries/dph/dph-lifted-vseg/config.status "rm" -f libraries/base/include/HsBaseConfig.h libraries/process/include/HsProcessConfig.h libraries/unix/include/HsUnixConfig.h "rm" -rf libraries/dist-haddock "rm" -rf bindistprep/ test ! -d testsuite || make -C testsuite clean make[1]: Entering directory '/stuff3/work/GHC2/ghc/testsuite' make -C ./timeout clean make[2]: Entering directory '/stuff3/work/GHC2/ghc/testsuite/timeout' test ! -f Setup || ./Setup clean rm -f -rf install-inplace rm -f -f calibrate.out rm -f -f Setup Setup.exe Setup.hi Setup.o make[2]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite/timeout' rm -f -f mk/*.o rm -f -f mk/*.hi rm -f -f mk/ghcconfig*.mk rm -f -f mk/ghc-config mk/ghc-config.exe rm -f -f driver/*.pyc make[1]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite' real 0m50.990s user 0m0.496s sys 0m1.582s -- From simonpj at microsoft.com Fri Nov 17 13:23:37 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 17 Nov 2017 13:23:37 +0000 Subject: Observation on Hadrian's relative performance re current buildsystem In-Reply-To: <87a7zloxxg.fsf@gmail.com> References: <87a7zloxxg.fsf@gmail.com> Message-ID: Urk! I expected Hadrian to be faster because it has more accurate dependencies. Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of | Herbert Valerio Riedel | Sent: 17 November 2017 13:08 | To: ghc-devs at haskell.org | Subject: Observation on Hadrian's relative performance re current | buildsystem | | Hello GHC devs, | | I took the opportunity to give Hadrian a test-run to see whether it | could live up to the big promise of delivering a "more scalable, | faster" | system than the current GNU Make based system. Unfortunately, my | preliminary results don't back this claim, and actually make Hadrian | appear to be significantly slower. | | ---- | | Here's the summary of the results: | | | Hadrian | GNU Make | | +------------------------+-----------+-----------+ | | Compiling `hadrian` | 5m25s | 0 | | | (one-time setup) | | | | +------------------------+-----------+-----------+ | | build "all" at -j8 | 38m | 33m | | +------------------------+-----------+-----------+ | | no-op build at -j8 | 10.977s | 3.258s | | +------------------------+-----------+-----------+ | | "clean" | 21s | 51s | | +------------------------+-----------+-----------+ | | | So, Hadrian is ~5 minutes than GNU Make (or even ~10 minutes if you | also count the one-time setup cost). | | And what I personally consider a bit annoying is that it's ~3 times | slower detecting; i.e. you have to wait 11s for Hadrian to detect | there's nothing to be done which compared to GNU Make (which currently | needlessly re-runs Sphinx; so it could be even faster!) is very | noticeable to me. | | There's a silver-lining though, deleting files is the part which is a | lot more costly in the GNU Make system currently since artifacts are | spread over several (scroll to the end of this email) subfolders | there. Whereas Hadrian did something we should have done for the GNU | Make system as well (and probably would have done sooner or later | anyway in order to support the srcdir!=buildir scheme that people are | used to from GNU Autotools projects); Hadrian places build-artifacts | into a few top-level folders, and so cleaning up is trivial and | requires to unlink only a few folders from the filesystem. | | ---- | | At the very least, I'd expect Hadrian to be as fast as the GNU Make | system (and ideally beat it, not the least as this was besides | maintainability its big promise), but so far it doesn't seem to | deliver that promise for me. | | It could easily be that I'm comparing apples to oranges here or that | I've otherwise overlooked something, so let me describe how I came to | this conclusion: | | I tried this on an reasonably idle Linux workstation with an Intel(R) | Core(TM) i7-3770 CPU @ 3.40GHz CPU, and with 32GiB RAM (i.e. the | filesystem content was well-cached into memory; NB: a ghc source tree | + compiled artifacts takes up about 4GiB on the filesystem). | | I've started from a fresh Git clone, i.e. | | git clone --recursive git://git.haskell.org/ghc.git | | followed by | | ./boot | ./configure | | At this point, we're at the common point from which both the Hadrian | and the GNU Make build-system would start diverging: | | For the Hadrian build-system, we need to pay for a one-time setup, | since we need to build the `hadrian` executable (which requires to | build the in-tree lib:Cabal as an in-place library): | | | $ time ./hadrian/build.sh --help | | ... | | real 5m25.992s | user 6m19.196s | sys 0m6.079s | | | I'm not too worried about this part, as there's a few tricks by which | we could likely bring that down to about 2 minutes or so, and we | mostly pay this setup-cost, when lib:Cabal and/or `hadrian` changes | and requires to be recompiled. | | | Now, after having made sure that the `hadrian` executable is fresh, I | started the actual build: | | $ time ./hadrian/build.sh -j8 | | ... | | shakeArgsWith 0.000s 0% | Function shake 0.178s 0% | Database read 0.000s 0% | With database 0.000s 0% | Running rules 2338.398s 99% | ========================= | Pool finished (5261 threads, 8 max) 0.002s 0% | Lint checking 0.111s 0% | Total 2338.690s 100% | Build completed in 38:59m | | real 38m59.626s | user 219m7.421s | sys 11m7.584s | | | then I immediately re-issued the same command to test how long it | takes to perform a no-op build: | | | $ time ./hadrian/build.sh -j8 | | Up to date | Up to date | shakeArgsWith 0.000s 0% | Function shake 0.183s 1% | Database read 0.144s 1% | With database 0.241s 2% | Running rules 9.379s 93% | ========================= | Pool finished (4165 threads, 8 max) 0.004s 0% | Lint checking 0.101s 1% | Total 10.051s 100% | Build completed in 0:11m | | | real 0m10.977s | user 0m19.399s | sys 0m2.443s | | | Same without -j8: | | $ time ./hadrian/build.sh | Up to date | Up to date | shakeArgsWith 0.000s 0% | Function shake 0.175s 2% | Database read 0.123s 1% | With database 0.197s 2% | Running rules 7.710s 92% | ========================= | Pool finished (1 threads, 1 max) 0.003s 0% | Lint checking 0.084s 1% | Total 8.293s 100% | Build completed in 0:09m | | | real 0m9.196s | user 0m8.656s | sys 0m0.724s | | | And finally clean it again: | | $ sync | $ time ./hadrian/build.sh clean | Up to date | Up to date | | Remove directory _build/stage0 | | Remove directory _build/stage1 | | Remove directory _build/stage2 | | Remove directory _build/stage3 | | Remove directory inplace/bin | | Remove directory inplace/lib | | Remove directory sdistprep | | Remove Hadrian files... | | Remove directory _build/generated | | Done. | shakeArgsWith 0.000s 0% | Function shake 0.175s 0% | Database read 0.114s 0% | With database 0.205s 0% | Running rules 20.037s 97% | ======================== | Pool finished (1 threads, 1 max) 0.002s 0% | Lint checking 0.006s 0% | Total 20.540s 100% | Build completed in 0:21m | | | real 0m21.426s | user 0m1.415s | sys 0m1.045s | | | | ---- | | Running the full build via our rusty GNU Make system: | | $ time make V=0 -j8 | real 33m30.801s | user 157m49.520s | sys 6m49.289s | | A no-op build: | | $ time make V=0 -j8 | ===--- building phase 0 | make --no-print-directory -f ghc.mk phase=0 phase_0_builds | make[1]: Nothing to be done for 'phase_0_builds'. | ===--- building phase 1 | make --no-print-directory -f ghc.mk phase=1 phase_1_builds | make[1]: Nothing to be done for 'phase_1_builds'. | ===--- building final phase | make --no-print-directory -f ghc.mk phase=final all | make -C utils/haddock/doc html SPHINX_BUILD=/usr/bin/sphinx-build | /usr/bin/sphinx-build -b html . .build-html | Running Sphinx v1.3.6 | loading translations [en]... done | loading pickled environment... done | building [mo]: targets for 0 po files that are out of date | building [html]: targets for 0 source files that are out of date | updating environment: 0 added, 0 changed, 0 removed | looking for now-outdated files... none found | no targets are out of date. | build succeeded. | cp -R utils/haddock/doc/.build-html utils/haddock/doc/haddock | | real 0m3.258s | user 0m3.134s | sys 0m0.283s | | And finally `make clean`: | | $ sync | $ time make clean | make --no-print-directory -f ghc.mk clean CLEANING=YES | "rm" -rf inplace/bin inplace/lib | "rm" -rf utils/touchy/dist | "rm" -rf inplace/lib/bin/touchy | "rm" -rf utils/unlit/dist | "rm" -rf inplace/lib/bin/unlit | "rm" -rf utils/unlit/dist-install | "rm" -rf utils/hp2ps/dist | "rm" -rf inplace/bin/hp2ps | "rm" -rf inplace/lib/bin/hp2ps | "rm" -rf utils/hp2ps/dist-install | "rm" -rf driver/split/dist inplace/lib/bin/ghc-split | "rm" -rf utils/genprimopcode/dist | "rm" -rf inplace/bin/genprimopcode | "rm" -rf libffi/build libffi/stamp.ffi.static-shared.configure | libffi/stamp.ffi.static-shared.build libffi/stamp.ffi.static- | shared.install libffi/dist-install | "rm" -rf utils/deriveConstants/dist | "rm" -rf inplace/bin/deriveConstants | "rm" -rf includes/ghcautoconf.h includes/ghcplatform.h | includes/ghcversion.h | "rm" -rf rts/dist | "rm" -rf bindisttest/"install dir" bindisttest/HelloWorld | bindisttest/HelloWorld.o bindisttest/HelloWorld.hi bindisttest/output | "rm" -rf utils/genapply/dist | "rm" -rf inplace/bin/genapply | "rm" -rf libraries/integer-gmp/include/ghc-gmp.h | libraries/integer-gmp/gmp/config.mk libraries/integer-gmp/gmp/libgmp.a | libraries/integer-gmp/gmp/gmp.h libraries/integer-gmp/gmp/gmpbuild | libraries/integer-gmp/gmp/gmp-6.1.2 | "rm" -rf libraries/integer-gmp/gmp/objs | "rm" -rf libraries/integer-gmp/gmp/gmpbuild | "rm" -rf utils/haddock/dist | "rm" -rf inplace/bin/haddock | "rm" -rf inplace/lib/bin/haddock | "rm" -rf compiler/stage1 | "rm" -rf compiler/stage2 | "rm" -rf compiler/stage3 | "rm" -rf utils/hsc2hs/dist | "rm" -rf inplace/bin/hsc2hs | "rm" -rf inplace/lib/bin/hsc2hs | "rm" -rf utils/hsc2hs/dist-install | "rm" -rf utils/ghc-pkg/dist | "rm" -rf inplace/bin/ghc-pkg | "rm" -rf inplace/lib/bin/ghc-pkg | "rm" -rf utils/ghc-pkg/dist-install | "rm" -rf utils/ghctags/dist-install | "rm" -rf inplace/bin/ghctags | "rm" -rf utils/check-api-annotations/dist-install | "rm" -rf inplace/bin/check-api-annotations | "rm" -rf utils/check-ppr/dist-install | "rm" -rf inplace/bin/check-ppr | "rm" -rf utils/ghc-cabal/dist bootstrapping | "rm" -rf utils/ghc-cabal/dist-install | "rm" -rf utils/hpc/dist-install | "rm" -rf inplace/bin/hpc | "rm" -rf inplace/lib/bin/hpc | "rm" -rf utils/runghc/dist-install | "rm" -rf inplace/bin/runghc | "rm" -rf inplace/lib/bin/runghc | "rm" -rf ghc/stage1 | "rm" -rf inplace/bin/ghc-stage1 | "rm" -rf inplace/lib/bin/ghc-stage1 | "rm" -rf ghc/stage2 | "rm" -rf inplace/bin/ghc-stage2 | "rm" -rf inplace/lib/bin/ghc-stage2 | "rm" -rf ghc/stage3 | "rm" -rf docs/users_guide/.doctrees-html/ | docs/users_guide/.doctrees-pdf/ docs/users_guide/build-html/ | docs/users_guide/build-pdf/ docs/users_guide/users_guide.pdf | "rm" -rf docs/users_guide/.doctrees-man/ docs/users_guide/build- | man/ | "rm" -rf utils/count_lines/dist inplace/bin/count_lines | "rm" -rf utils/compare_sizes/dist-install | "rm" -rf iserv/stage2 | "rm" -rf inplace/lib/bin/ghc-iserv | "rm" -rf iserv/stage2_p | "rm" -rf inplace/lib/bin/ghc-iserv-prof | "rm" -rf iserv/stage2_dyn | "rm" -rf inplace/lib/bin/ghc-iserv-dyn | "rm" -f libraries/integer-gmp/include/HsIntegerGmp.h | libraries/base/include/EventConfig.h mk/config.mk.old | mk/project.mk.old compiler/ghc.cabal.old includes/GHCConstants.h | includes/DerivedConstants.h includes/ghcautoconf.h | includes/ghcplatform.h includes/ghcversion.h utils/ghc-pkg/Version.hs | compiler/prelude/primops.txt | "rm" -rf includes/dist-derivedconstants | "rm" -rf inplace/bin | "rm" -rf inplace/lib | "rm" -rf libraries/bootstrapping.conf | "rm" -f mk/are-validating.mk | "rm" -rf libraries/ghc-boot-th/dist-install | "rm" -rf libraries/ghc-boot/dist-install | "rm" -rf libraries/ghci/dist-install | "rm" -rf libraries/base/dist-install | "rm" -rf libraries/ghc-prim/dist-install | "rm" -rf libraries/integer-gmp/dist-install | "rm" -rf libraries/integer-simple/dist-install | "rm" -rf libraries/template-haskell/dist-install | "rm" -rf libraries/array/dist-install | "rm" -rf libraries/binary/dist-install | "rm" -rf libraries/bytestring/dist-install | "rm" -rf libraries/Cabal/Cabal/dist-install | "rm" -rf libraries/ghc-compact/dist-install | "rm" -rf libraries/containers/dist-install | "rm" -rf libraries/deepseq/dist-install | "rm" -rf libraries/directory/dist-install | "rm" -rf libraries/filepath/dist-install | "rm" -rf libraries/haskeline/dist-install | "rm" -rf libraries/hpc/dist-install | "rm" -rf libraries/mtl/dist-install | "rm" -rf libraries/parsec/dist-install | "rm" -rf libraries/pretty/dist-install | "rm" -rf libraries/process/dist-install | "rm" -rf libraries/terminfo/dist-install | "rm" -rf libraries/text/dist-install | "rm" -rf libraries/time/dist-install | "rm" -rf libraries/transformers/dist-install | "rm" -rf libraries/unix/dist-install | "rm" -rf libraries/Win32/dist-install | "rm" -rf libraries/xhtml/dist-install | "rm" -rf libraries/parallel/dist-install | "rm" -rf libraries/stm/dist-install | "rm" -rf libraries/random/dist-install | "rm" -rf libraries/primitive/dist-install | "rm" -rf libraries/vector/dist-install | "rm" -rf libraries/dph/dph-base/dist-install | "rm" -rf libraries/dph/dph-prim-interface/dist-install | "rm" -rf libraries/dph/dph-prim-seq/dist-install | "rm" -rf libraries/dph/dph-prim-par/dist-install | "rm" -rf libraries/dph/dph-lifted-base/dist-install | "rm" -rf libraries/dph/dph-lifted-boxed/dist-install | "rm" -rf libraries/dph/dph-lifted-copy/dist-install | "rm" -rf libraries/dph/dph-lifted-vseg/dist-install | "rm" -rf libraries/ghc-boot-th/dist-boot | "rm" -rf libraries/ghc-boot/dist-boot | "rm" -rf libraries/ghci/dist-boot | "rm" -rf libraries/base/dist-boot | "rm" -rf libraries/ghc-prim/dist-boot | "rm" -rf libraries/integer-gmp/dist-boot | "rm" -rf libraries/integer-simple/dist-boot | "rm" -rf libraries/template-haskell/dist-boot | "rm" -rf libraries/array/dist-boot | "rm" -rf libraries/binary/dist-boot | "rm" -rf libraries/bytestring/dist-boot | "rm" -rf libraries/Cabal/Cabal/dist-boot | "rm" -rf libraries/ghc-compact/dist-boot | "rm" -rf libraries/containers/dist-boot | "rm" -rf libraries/deepseq/dist-boot | "rm" -rf libraries/directory/dist-boot | "rm" -rf libraries/filepath/dist-boot | "rm" -rf libraries/haskeline/dist-boot | "rm" -rf libraries/hpc/dist-boot | "rm" -rf libraries/mtl/dist-boot | "rm" -rf libraries/parsec/dist-boot | "rm" -rf libraries/pretty/dist-boot | "rm" -rf libraries/process/dist-boot | "rm" -rf libraries/terminfo/dist-boot | "rm" -rf libraries/text/dist-boot | "rm" -rf libraries/time/dist-boot | "rm" -rf libraries/transformers/dist-boot | "rm" -rf libraries/unix/dist-boot | "rm" -rf libraries/Win32/dist-boot | "rm" -rf libraries/xhtml/dist-boot | "rm" -rf libraries/parallel/dist-boot | "rm" -rf libraries/stm/dist-boot | "rm" -rf libraries/random/dist-boot | "rm" -rf libraries/primitive/dist-boot | "rm" -rf libraries/vector/dist-boot | "rm" -rf libraries/dph/dph-base/dist-boot | "rm" -rf libraries/dph/dph-prim-interface/dist-boot | "rm" -rf libraries/dph/dph-prim-seq/dist-boot | "rm" -rf libraries/dph/dph-prim-par/dist-boot | "rm" -rf libraries/dph/dph-lifted-base/dist-boot | "rm" -rf libraries/dph/dph-lifted-boxed/dist-boot | "rm" -rf libraries/dph/dph-lifted-copy/dist-boot | "rm" -rf libraries/dph/dph-lifted-vseg/dist-boot | "rm" -rf libraries/ghc-boot-th/dist libraries/ghc-boot/dist | libraries/ghci/dist libraries/base/dist libraries/ghc-prim/dist | libraries/integer-gmp/dist libraries/integer-simple/dist | libraries/template-haskell/dist libraries/array/dist | libraries/binary/dist libraries/bytestring/dist | libraries/Cabal/Cabal/dist libraries/ghc-compact/dist | libraries/containers/dist libraries/deepseq/dist | libraries/directory/dist libraries/filepath/dist | libraries/haskeline/dist libraries/hpc/dist libraries/mtl/dist | libraries/parsec/dist libraries/pretty/dist libraries/process/dist | libraries/terminfo/dist libraries/text/dist libraries/time/dist | libraries/transformers/dist libraries/unix/dist libraries/Win32/dist | libraries/xhtml/dist libraries/parallel/dist libraries/stm/dist | libraries/random/dist libraries/primitive/dist libraries/vector/dist | libraries/dph/dph-base/dist libraries/dph/dph-prim-interface/dist | libraries/dph/dph-prim-seq/dist libraries/dph/dph-prim-par/dist | libraries/dph/dph-lifted-base/dist libraries/dph/dph-lifted- | boxed/dist libraries/dph/dph-lifted-copy/dist libraries/dph/dph- | lifted-vseg/dist | "rm" -f libraries/base/base.buildinfo libraries/integer- | gmp/integer-gmp.buildinfo libraries/terminfo/terminfo.buildinfo | libraries/unix/unix.buildinfo | "rm" -f libraries/ghc-boot-th/config.log libraries/ghc- | boot/config.log libraries/ghci/config.log libraries/base/config.log | libraries/ghc-prim/config.log libraries/integer-gmp/config.log | libraries/integer-simple/config.log libraries/template- | haskell/config.log libraries/array/config.log | libraries/binary/config.log libraries/bytestring/config.log | libraries/Cabal/Cabal/config.log libraries/ghc-compact/config.log | libraries/containers/config.log libraries/deepseq/config.log | libraries/directory/config.log libraries/filepath/config.log | libraries/haskeline/config.log libraries/hpc/config.log | libraries/mtl/config.log libraries/parsec/config.log | libraries/pretty/config.log libraries/process/config.log | libraries/terminfo/config.log libraries/text/config.log | libraries/time/config.log libraries/transformers/config.log | libraries/unix/config.log libraries/Win32/config.log | libraries/xhtml/config.log libraries/parallel/config.log | libraries/stm/config.log libraries/random/config.log | libraries/primitive/config.log libraries/vector/config.log | libraries/dph/dph-base/config.log libraries/dph/dph-prim- | interface/config.log libraries/dph/dph-prim-seq/config.log | libraries/dph/dph-prim-par/config.log libraries/dph/dph-lifted- | base/config.log libraries/dph/dph-lifted-boxed/config.log | libraries/dph/dph-lifted-copy/config.log libraries/dph/dph-lifted- | vseg/config.log | "rm" -f libraries/ghc-boot-th/config.status libraries/ghc- | boot/config.status libraries/ghci/config.status | libraries/base/config.status libraries/ghc-prim/config.status | libraries/integer-gmp/config.status libraries/integer- | simple/config.status libraries/template-haskell/config.status | libraries/array/config.status libraries/binary/config.status | libraries/bytestring/config.status | libraries/Cabal/Cabal/config.status libraries/ghc- | compact/config.status libraries/containers/config.status | libraries/deepseq/config.status libraries/directory/config.status | libraries/filepath/config.status libraries/haskeline/config.status | libraries/hpc/config.status libraries/mtl/config.status | libraries/parsec/config.status libraries/pretty/config.status | libraries/process/config.status libraries/terminfo/config.status | libraries/text/config.status libraries/time/config.status | libraries/transformers/config.status libraries/unix/config.status | libraries/Win32/config.status libraries/xhtml/config.status | libraries/parallel/config.status libraries/stm/config.status | libraries/random/config.status libraries/primitive/config.status | libraries/vector/config.status libraries/dph/dph-base/config.status | libraries/dph/dph-prim-interface/config.status libraries/dph/dph- | prim-seq/config.status libraries/dph/dph-prim-par/config.status | libraries/dph/dph-lifted-base/config.status libraries/dph/dph-lifted- | boxed/config.status libraries/dph/dph-lifted-copy/config.status | libraries/dph/dph-lifted-vseg/config.status | "rm" -f libraries/base/include/HsBaseConfig.h | libraries/process/include/HsProcessConfig.h | libraries/unix/include/HsUnixConfig.h | "rm" -rf libraries/dist-haddock | "rm" -rf bindistprep/ | test ! -d testsuite || make -C testsuite clean | make[1]: Entering directory '/stuff3/work/GHC2/ghc/testsuite' | make -C ./timeout clean | make[2]: Entering directory | '/stuff3/work/GHC2/ghc/testsuite/timeout' | test ! -f Setup || ./Setup clean | rm -f -rf install-inplace | rm -f -f calibrate.out | rm -f -f Setup Setup.exe Setup.hi Setup.o | make[2]: Leaving directory | '/stuff3/work/GHC2/ghc/testsuite/timeout' | rm -f -f mk/*.o | rm -f -f mk/*.hi | rm -f -f mk/ghcconfig*.mk | rm -f -f mk/ghc-config mk/ghc-config.exe | rm -f -f driver/*.pyc | make[1]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite' | | real 0m50.990s | user 0m0.496s | sys 0m1.582s | | | | -- | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h | askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cec692486529544de389e08d5 | 2dbc7432%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6364652097984492 | 46&sdata=I1TY6c2l59jou87OVjqScDvct%2FaygyM4HRuSirCSp5w%3D&reserved=0 From andrey.mokhov at newcastle.ac.uk Fri Nov 17 14:04:04 2017 From: andrey.mokhov at newcastle.ac.uk (Andrey Mokhov) Date: Fri, 17 Nov 2017 14:04:04 +0000 Subject: Observation on Hadrian's relative performance re current References: Message-ID: Hi Herbert, Thanks for the careful performance experiment! Can you please put all the details into the issue tracker so they don't get lost? We have a couple of performance issues open, and we know that there are performance bugs in Hadrian leading to too sequential build (a recent example is #464), but we haven't had time to optimise Hadrian yet. In any case it's good to know where we stand right now. I hope you'll be willing to repeat your experiment once these bugs are fixed. Cheers, Andrey From sheard at pdx.edu Fri Nov 17 18:05:10 2017 From: sheard at pdx.edu (Tim Sheard) Date: Fri, 17 Nov 2017 13:05:10 -0500 Subject: True multi stage Haskell Message-ID: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> After many years of hoping someone else would do this, I would like to make GHC into a true multi-stage programming language. Here is how I thought I might approach this. 1) Use the GHC as a library module. 2) Use the LLVM backend. I have no experience with either of these tools. Lets start simple, How would I write functions like compile :: String -> IO PtrToLLVMCode  -- where the string is a small Haskell program. llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode jumpTo:: PtrToMachineCode -> IO ans   -- where ans is the "type" of the string. Any thoughts on how to get started? What papers to read, examples to look at? I'd love to move to some more disciplined input type, a sort of (mono) typed program representation (with similar complexity) to Template Haskell Exp type. where (Exp t) is a data structure representing a Haskell program of type t. All offers of advice accepted. I'm trying to get started soon, and good advice about what to avoid is especially welcome.  If any one wanted to help with this, that would be great. Tim Sheard From david at well-typed.com Fri Nov 17 18:26:07 2017 From: david at well-typed.com (David Feuer) Date: Fri, 17 Nov 2017 13:26:07 -0500 Subject: True multi stage Haskell Message-ID: <20171117175613.92F23BC58E@haskell.org> I don't know anything myself, but if you're doing anything remotely related to Haskell and macro systems, you should probably read up on what Alexis King (AKA lexi-lambda) has been up to lately, and maybe get in touch with her. David FeuerWell-Typed, LLP -------- Original message --------From: Tim Sheard Date: 11/17/17 1:05 PM (GMT-05:00) To: ghc-devs at haskell.org Subject: True multi stage Haskell After many years of hoping someone else would do this, I would like to make GHC into a true multi-stage programming language. Here is how I thought I might approach this. 1) Use the GHC as a library module. 2) Use the LLVM backend. I have no experience with either of these tools. Lets start simple, How would I write functions like compile :: String -> IO PtrToLLVMCode  -- where the string is a small Haskell program. llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode jumpTo:: PtrToMachineCode -> IO ans   -- where ans is the "type" of the string. Any thoughts on how to get started? What papers to read, examples to look at? I'd love to move to some more disciplined input type, a sort of (mono) typed program representation (with similar complexity) to Template Haskell Exp type. where (Exp t) is a data structure representing a Haskell program of type t. All offers of advice accepted. I'm trying to get started soon, and good advice about what to avoid is especially welcome.  If any one wanted to help with this, that would be great. Tim Sheard _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Fri Nov 17 18:41:43 2017 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 17 Nov 2017 18:41:43 +0000 Subject: True multi stage Haskell In-Reply-To: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> References: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> Message-ID: Hi Tim, When you say "multi-stage programming language" do you have a specific calculus in mind? I think this can mean lots of different things to different people. This is a topic I have been interested in recently. Cheers, Matt On Fri, Nov 17, 2017 at 6:05 PM, Tim Sheard wrote: > After many years of hoping someone else would do this, I would like to > make GHC into a true multi-stage programming language. Here is how I > thought I might approach this. > > 1) Use the GHC as a library module. > 2) Use the LLVM backend. > > I have no experience with either of these tools. > Lets start simple, How would I write functions like > > compile :: String -> IO PtrToLLVMCode -- where the string is a small > Haskell program. > llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode > jumpTo:: PtrToMachineCode -> IO ans -- where ans is the "type" of the > string. > > > Any thoughts on how to get started? What papers to read, examples to look > at? > > I'd love to move to some more disciplined input type, a sort of (mono) typed > program > representation (with similar complexity) to Template Haskell Exp type. > > where (Exp t) is a data structure representing a Haskell program of type t. > > All offers of advice accepted. I'm trying to get started soon, and good > advice > about what to avoid is especially welcome. If any one wanted to help with > this, > that would be great. > > Tim Sheard > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Fri Nov 17 18:46:02 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 17 Nov 2017 13:46:02 -0500 Subject: Observation on Hadrian's relative performance re current buildsystem In-Reply-To: References: <87a7zloxxg.fsf@gmail.com> Message-ID: <874lpskal7.fsf@ben-laptop.smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Urk! I expected Hadrian to be faster because it has more accurate dependencies. > While this is just speculation, this might actually be one of the reasons why the no-op case is slower: in make's case the dependency graph is mostly static whereas in Hadrian the build system needs to discover dependencies dynamically. Either way, thanks for this characterization, Herbert! Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From iavor.diatchki at gmail.com Fri Nov 17 19:57:20 2017 From: iavor.diatchki at gmail.com (Iavor Diatchki) Date: Fri, 17 Nov 2017 19:57:20 +0000 Subject: True multi stage Haskell In-Reply-To: References: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> Message-ID: Hello Tim, I don't know the answers to your questions exactly, but I've played around with something that I think might be related, so I thought I'd share. The example code is here: https://github.com/GaloisInc/galua/blob/master/galua-jit/src/Galua/Micro/Compile.hs This module contains a function that compiles (at runtime) a Haskell module represented as a String to the value of whatever `main` happened to be defined in this module. Actually, in this case the expression is taken as a `Doc` , which is the result of pretty printing something, but this doc is simply "show"-ed to make a string (see `modText`). The function works in roughly two steps: 1. Compile the module: 1.1. Setup some compiler flags, this has to do with the dependencies of the module (i.e., what packages it depends on, etc) 1.2. Define a "target": this is what GHC will be compiling. There is a bit of weirdness here: GHC can take the input to compile either from a text file, or from an in-memory buffer. The second is what we want here. Unfortunately, for some reason GHC still insists on there being a file around (its contents is completely irrelevant) I think it just want to check when it was last modified :-) This is why we create an empty file. This is something that should probably be fixed in GHC. 1.3 Load all the targets: this is what will actually compile the module 2. Get the value of `main` in this module: this is done in the `Succeed` branch of the case, where we create an expression `M.main` and tell GHC to evaluate it. Depending on how you need things to be setup, just the second step might be enough (i.e., call `compileExpr`). You'd still need to do enough to tell GHC in what context it will be compiling the expression though. I am not an expert on this topic, but I'd be happy to try to answer more questions if I can. -Iavor On Fri, Nov 17, 2017 at 10:42 AM Matthew Pickering < matthewtpickering at gmail.com> wrote: > Hi Tim, > > When you say "multi-stage programming language" do you have a specific > calculus in mind? I think this can mean lots of different things to > different people. > > This is a topic I have been interested in recently. > > Cheers, > > Matt > > On Fri, Nov 17, 2017 at 6:05 PM, Tim Sheard wrote: > > After many years of hoping someone else would do this, I would like to > > make GHC into a true multi-stage programming language. Here is how I > > thought I might approach this. > > > > 1) Use the GHC as a library module. > > 2) Use the LLVM backend. > > > > I have no experience with either of these tools. > > Lets start simple, How would I write functions like > > > > compile :: String -> IO PtrToLLVMCode -- where the string is a small > > Haskell program. > > llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode > > jumpTo:: PtrToMachineCode -> IO ans -- where ans is the "type" of the > > string. > > > > > > Any thoughts on how to get started? What papers to read, examples to look > > at? > > > > I'd love to move to some more disciplined input type, a sort of (mono) > typed > > program > > representation (with similar complexity) to Template Haskell Exp type. > > > > where (Exp t) is a data structure representing a Haskell program of type > t. > > > > All offers of advice accepted. I'm trying to get started soon, and good > > advice > > about what to avoid is especially welcome. If any one wanted to help > with > > this, > > that would be great. > > > > Tim Sheard > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sh.najd at gmail.com Fri Nov 17 20:04:54 2017 From: sh.najd at gmail.com (Shayan Najd) Date: Fri, 17 Nov 2017 20:04:54 +0000 Subject: True multi stage Haskell In-Reply-To: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> References: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> Message-ID: Hi Tim, About using GHC as a library module, have you see the ongoing work on "native metaprogramming" in GHC at https://ghc.haskell.org/trac/ghc/wiki/NativeMetaprogramming and a child project at https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow There, online code generation is not of the highest priority, rather we are trying to open up the compiler from top to down: making available GHC's AST and then the parser and the typechecker have priority over the other functionalities down the pipeline (like the code generator). > If any one wanted to help with this, that would be great. I am interested to hear more about the approach you have in mind, and would be happy to help. Yours, Shayan On Fri, Nov 17, 2017 at 6:05 PM, Tim Sheard wrote: > After many years of hoping someone else would do this, I would like to > make GHC into a true multi-stage programming language. Here is how I > thought I might approach this. > > 1) Use the GHC as a library module. > 2) Use the LLVM backend. > > I have no experience with either of these tools. > Lets start simple, How would I write functions like > > compile :: String -> IO PtrToLLVMCode -- where the string is a small > Haskell program. > llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode > jumpTo:: PtrToMachineCode -> IO ans -- where ans is the "type" of the > string. > > > Any thoughts on how to get started? What papers to read, examples to look > at? > > I'd love to move to some more disciplined input type, a sort of (mono) > typed program > representation (with similar complexity) to Template Haskell Exp type. > > where (Exp t) is a data structure representing a Haskell program of type t. > > All offers of advice accepted. I'm trying to get started soon, and good > advice > about what to avoid is especially welcome. If any one wanted to help with > this, > that would be great. > > Tim Sheard > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From malcolm.wallace at me.com Sat Nov 18 11:09:28 2017 From: malcolm.wallace at me.com (Malcolm Wallace) Date: Sat, 18 Nov 2017 11:09:28 +0000 Subject: Observation on Hadrian's relative performance re current buildsystem In-Reply-To: <874lpskal7.fsf@ben-laptop.smart-cactus.org> References: <87a7zloxxg.fsf@gmail.com> <874lpskal7.fsf@ben-laptop.smart-cactus.org> Message-ID: On 17 Nov 2017, at 18:46, Ben Gamari wrote: > Simon Peyton Jones via ghc-devs writes: > >> Urk! I expected Hadrian to be faster because it has more accurate dependencies. >> > While this is just speculation, this might actually be one of the > reasons why the no-op case is slower: in make's case the dependency > graph is mostly static whereas in Hadrian the build system needs to > discover dependencies dynamically. > > Either way, thanks for this characterization, Herbert! But surely the timing for a full build from scratch is not the most important thing to compare? In my work environment, full builds are extremely rare; the common case is an incremental build after pulling changes from upstream. Is this something you can measure? Regards, Malcolm From hvriedel at gmail.com Sat Nov 18 11:25:25 2017 From: hvriedel at gmail.com (Herbert Valerio Riedel) Date: Sat, 18 Nov 2017 12:25:25 +0100 Subject: Observation on Hadrian's relative performance re current buildsystem In-Reply-To: (Malcolm Wallace's message of "Sat, 18 Nov 2017 11:09:28 +0000") References: <87a7zloxxg.fsf@gmail.com> <874lpskal7.fsf@ben-laptop.smart-cactus.org> Message-ID: <87bmjzzv4q.fsf@gmail.com> Hi Malcolm, On 2017-11-18 at 11:09:28 +0000, Malcolm Wallace wrote: [...] > But surely the timing for a full build from scratch is not the most > important thing to compare? In my work environment, full builds are > extremely rare; the common case is an incremental build after pulling > changes from upstream. Is this something you can measure? If Hadrian is more exact about the dependency tracking, I'd expect something close to a full rebuild when I `git pull` from GHC HEAD, since the Git commit and the snapshot versioning we infer from that, pervasively transcends most most artifacts of GHC, and in general you should boot & configure everytime you `git pull` unless you're sure it won't matter. You'd have to suppress/mask this logic if you want to avoid full rebuilds. Also I'm not sure how Hadrian tracks itself as a dependency (NB: The GNU Make system doesn't); When I used Shake myself, I remember that meta-depending on the rules per se wasn't a trivial thing to do; and the simplest way was to introduce very coarse (either manual or by hashing) global versioning over all rules, which would invalidate the full build. But it's been some time since I did that, so I may be wrong here. However, what I think would be a more relevant benchmark matching the usual GHC developer workflow, would be to see how well Hadrian manages to minimize the work needed to rebuild GHC after editing a source file in GHC's source-tree without changing the Git commit. As that's what matters most to me when I'm actively working on a GHC patch. I'll try to measure/compare this with the next GHC patch I hack on. From ggreif at gmail.com Sat Nov 18 12:41:38 2017 From: ggreif at gmail.com (Gabor Greif) Date: Sat, 18 Nov 2017 13:41:38 +0100 Subject: [commit: ghc] master: Adds -ghc-version flag to ghc. (12a7444) In-Reply-To: <20171118030026.4966D3A5EA@ghc.haskell.org> References: <20171118030026.4966D3A5EA@ghc.haskell.org> Message-ID: Hmm, when seeing "-ghc-version" I first thought why a new flag for querying the version of GHC? But the flag has a different purpose! I'd suggest to rename it to "-ghcversion-file" or similar. Just my two cents... Cheers, Gabor On 11/18/17, git at git.haskell.org wrote: > Repository : ssh://git at git.haskell.org/ghc > > On branch : master > Link : > http://ghc.haskell.org/trac/ghc/changeset/12a7444463184e9eddbe7b7251a0ee1e976f4d75/ghc > >>--------------------------------------------------------------- > > commit 12a7444463184e9eddbe7b7251a0ee1e976f4d75 > Author: Moritz Angermann > Date: Fri Nov 17 19:29:25 2017 +0800 > > Adds -ghc-version flag to ghc. > > Summary: > When building the rts with ghc (e.g. using ghc as a c compiler), ghc's > "Value Add"[1] is, it includes adding `-include /path/to/ghcversion.h`. > For > this it looksup the rts package in the package database, which--if > empty--fails. Thus to allow compiling C files with GHC, we add the > `-ghc-version` flag, which takes the path to the `ghcversion.h` file. > > A `-no-ghc-version` flag was omitted, as at that point it becomes > questionable why one would use ghc to compile c if one doesn't > any of the added value. > > -- > > [1] from `compiler/main/DriverPipeline.hs` > > -- add package include paths even if we're just compiling .c > > -- files; this is the Value Add(TM) that using ghc instead of > > -- gcc gives you :) > > Reviewers: bgamari, geekosaur, austin > > Reviewed By: bgamari > > Subscribers: rwbarton, thomie > > Differential Revision: https://phabricator.haskell.org/D4135 > > >>--------------------------------------------------------------- > > 12a7444463184e9eddbe7b7251a0ee1e976f4d75 > compiler/main/DriverPipeline.hs | 11 ++++++++--- > compiler/main/DynFlags.hs | 7 +++++++ > docs/users_guide/using.rst | 23 +++++++++++++++++++++++ > 3 files changed, 38 insertions(+), 3 deletions(-) > > diff --git a/compiler/main/DriverPipeline.hs > b/compiler/main/DriverPipeline.hs > index fab7fad..4f7bfbd 100644 > --- a/compiler/main/DriverPipeline.hs > +++ b/compiler/main/DriverPipeline.hs > @@ -2204,11 +2204,16 @@ touchObjectFile dflags path = do > -- | Find out path to @ghcversion.h@ file > getGhcVersionPathName :: DynFlags -> IO FilePath > getGhcVersionPathName dflags = do > - dirs <- getPackageIncludePath dflags [toInstalledUnitId rtsUnitId] > + candidates <- case ghcVersion dflags of > + Just path -> return [path] > + Nothing -> (map ( "ghcversion.h")) <$> > + (getPackageIncludePath dflags [toInstalledUnitId rtsUnitId]) > > - found <- filterM doesFileExist (map ( "ghcversion.h") dirs) > + found <- filterM doesFileExist candidates > case found of > - [] -> throwGhcExceptionIO (InstallationError ("ghcversion.h > missing")) > + [] -> throwGhcExceptionIO (InstallationError > + ("ghcversion.h missing; tried: " > + ++ intercalate ", " candidates)) > (x:_) -> return x > > -- Note [-fPIC for assembler] > diff --git a/compiler/main/DynFlags.hs b/compiler/main/DynFlags.hs > index 5888acc..04ac635 100644 > --- a/compiler/main/DynFlags.hs > +++ b/compiler/main/DynFlags.hs > @@ -534,6 +534,7 @@ data GeneralFlag > | Opt_ExternalInterpreter > | Opt_OptimalApplicativeDo > | Opt_VersionMacros > + | Opt_GhcVersion > | Opt_WholeArchiveHsLibs > > -- PreInlining is on by default. The option is there just to see how > @@ -917,6 +918,7 @@ data DynFlags = DynFlags { > flushOut :: FlushOut, > flushErr :: FlushErr, > > + ghcVersion :: Maybe FilePath, > haddockOptions :: Maybe String, > > -- | GHCi scripts specified by -ghci-script, in reverse order > @@ -1682,6 +1684,7 @@ defaultDynFlags mySettings myLlvmTargets = > filesToClean = panic "defaultDynFlags: No filesToClean", > dirsToClean = panic "defaultDynFlags: No dirsToClean", > generatedDumps = panic "defaultDynFlags: No generatedDumps", > + ghcVersion = Nothing, > haddockOptions = Nothing, > dumpFlags = EnumSet.empty, > generalFlags = EnumSet.fromList (defaultFlags mySettings), > @@ -2339,6 +2342,9 @@ addDepSuffix s d = d { depSuffixes = s : depSuffixes d > } > > addCmdlineFramework f d = d { cmdlineFrameworks = f : cmdlineFrameworks d} > > +addGhcVersion :: FilePath -> DynFlags -> DynFlags > +addGhcVersion f d = d { ghcVersion = Just f } > + > addHaddockOpts f d = d { haddockOptions = Just f} > > addGhciScript f d = d { ghciScripts = f : ghciScripts d} > @@ -2866,6 +2872,7 @@ dynamic_flags_deps = [ > , make_ord_flag defGhcFlag "no-rtsopts-suggestions" > (noArg (\d -> d {rtsOptsSuggestions = False})) > > + , make_ord_flag defGhcFlag "ghc-version" (hasArg addGhcVersion) > , make_ord_flag defGhcFlag "main-is" (SepArg setMainIs) > , make_ord_flag defGhcFlag "haddock" (NoArg (setGeneralFlag > Opt_Haddock)) > , make_ord_flag defGhcFlag "haddock-opts" (hasArg addHaddockOpts) > diff --git a/docs/users_guide/using.rst b/docs/users_guide/using.rst > index a963ead..e3fa741 100644 > --- a/docs/users_guide/using.rst > +++ b/docs/users_guide/using.rst > @@ -1064,3 +1064,26 @@ Some flags only make sense for particular target > platforms. > and later). The :ref:`LLVM backend ` will also use > SSE4.2 if your processor supports it but detects this automatically > so no flag is required. > + > +Miscellaneous flags > +------------------- > + > +.. index:: > + single: miscellaneous flags > + > +Some flags only make sense for a particular use case. > + > +.. ghc-flag:: -ghc-version ⟨path to ghcversion.h⟩ > + :shortdesc: (GHC as a C compiler only) Use this ``ghcversion.h`` file > + :type: dynamic > + :category: misc > + > + When GHC is used to compile C files, GHC adds package include paths and > + includes ``ghcversion.h`` directly. The compiler will lookup the path > for > + the ``ghcversion.h`` file from the ``rts`` package in the package > database. > + In some cases, the compiler's package database does not contain the > ``rts`` > + package, or one wants to specify a specific ``ghcversions.h`` to be > + included. This option can be used to specify the path to the > + ``ghcversions.h`` file to be included. This is primarily intended to be > + used by GHC's build system. > + > > _______________________________________________ > ghc-commits mailing list > ghc-commits at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-commits > From ben at well-typed.com Sat Nov 18 17:32:54 2017 From: ben at well-typed.com (Ben Gamari) Date: Sat, 18 Nov 2017 12:32:54 -0500 Subject: Observation on Hadrian's relative performance re current buildsystem In-Reply-To: <87bmjzzv4q.fsf@gmail.com> References: <87a7zloxxg.fsf@gmail.com> <874lpskal7.fsf@ben-laptop.smart-cactus.org> <87bmjzzv4q.fsf@gmail.com> Message-ID: <87wp2nijb0.fsf@ben-laptop.smart-cactus.org> Herbert Valerio Riedel writes: > Hi Malcolm, > > On 2017-11-18 at 11:09:28 +0000, Malcolm Wallace wrote: > > [...] > >> But surely the timing for a full build from scratch is not the most >> important thing to compare? In my work environment, full builds are >> extremely rare; the common case is an incremental build after pulling >> changes from upstream. Is this something you can measure? > > If Hadrian is more exact about the dependency tracking, I'd expect > something close to a full rebuild when I `git pull` from GHC HEAD, since > the Git commit and the snapshot versioning we infer from that, > pervasively transcends most most artifacts of GHC, and in general you > should boot & configure everytime you `git pull` unless you're sure it > won't matter. > To be honest, I'm hoping that we can make this less necessary in the future. Relatively few changes really require a re-`configure`. In my experience most changes that current demand reconfigure are in .in files. In these cases I often just hack the changes in to the generated files myself. In principle there is no reason why Hadrian couldn't do this as well. I opened #416 to track this some time ago. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mail at joachim-breitner.de Mon Nov 20 01:11:26 2017 From: mail at joachim-breitner.de (Joachim Breitner) Date: Sun, 19 Nov 2017 20:11:26 -0500 Subject: True multi stage Haskell In-Reply-To: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> References: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> Message-ID: <1511140286.23398.1.camel@joachim-breitner.de> Hi Tim, Am Freitag, den 17.11.2017, 13:05 -0500 schrieb Tim Sheard: > After many years of hoping someone else would do this, I would like to > make GHC into a true multi-stage programming language. Here is how I > thought I might approach this. > > 1) Use the GHC as a library module. > 2) Use the LLVM backend. > > I have no experience with either of these tools. > > Lets start simple, How would I write functions like > > compile :: String -> IO PtrToLLVMCode -- where the string is a small > Haskell program. > llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode > jumpTo:: PtrToMachineCode -> IO ans -- where ans is the "type" of the > string. maybe not quite what you are looking for, but veggies, at https://github.com/nomeata/veggies uses the GHC API build a “new” Haskell compiler with its own Core→LLVM pass. Cheers, Joachim -- Joachim Breitner mail at joachim-breitner.de http://www.joachim-breitner.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From simonpj at microsoft.com Mon Nov 20 13:14:25 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 20 Nov 2017 13:14:25 +0000 Subject: Linux failure Message-ID: Hi devs I've got my Linux box upgraded to a recent version of Ubuntu, so I'm no longer stuck on gcc version. But I can't build either. Something strange with /lib/cpp fails sanity check. The log is below. Can anyone help? I'm a bit stalled. This call to "make" follows the same failure when I typed "sh validate -fast". But by carrying on from where it got stuck the log is way shorter. Thanks Simon simonpj at cam-05-unx:~/5builds/HEAD$ make ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all rts/ghc.mk:547: rts/dist/build/.depend-v-dyn-l-debug-thr-thr_debug-thr_l-debug_dyn-thr_dyn-thr_debug_dyn-l_dyn-thr_l_dyn.c_asm: No such file or directory libraries/ghci/ghc.mk:4: libraries/ghci/dist-install/build/.depend-v-dyn.haskell: No such file or directory "rm" -f libffi/stamp.ffi.static.configure "rm" -f libffi/stamp.ffi.static.build "rm" -f libffi/stamp.ffi.static.install "rm" -f libffi/stamp.ffi.static-shared.configure "rm" -f libffi/stamp.ffi.static-shared.build "rm" -f libffi/stamp.ffi.static-shared.install "rm" -rf libffi/build cat libffi-tarballs/libffi*.tar.gz | /bin/gzip -d | { cd libffi && /bin/tar -xf - ; } mv libffi/libffi-* libffi/build cp "/home/simonpj/5builds/HEAD/config.guess" libffi/build/config.guess cp "/home/simonpj/5builds/HEAD/config.sub" libffi/build/config.sub chmod +x libffi/ln # We need to use -MMD rather than -MD, as otherwise we get paths # like c:/... in the dependency files on Windows, and the extra # colons break make mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed "s/-MD/-MMD/" < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # We attempt to specify the installation directory below with --libdir, # but libffi installs into 'toolexeclibdir' instead, which may differ # on systems where gcc has multilib support. Force libffi to use libdir. # (https://sourceware.org/ml/libffi-discuss/2014/msg00016.html) mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's:@toolexeclibdir@:$(libdir):g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # install-sh is used when /usr/bin/install is missing; ensure its # path in libffi's Makefile is correct. See GHC #11109. mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's|@INSTALL@|$(subst ../install-sh,/home/simonpj/5builds/HEAD/install-sh, at INSTALL@)|g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in cd libffi && \ cd build && \ CC=gcc \ CXX=gcc \ LD=ld.gold \ AR=ar \ NM=nm \ RANLIB=ranlib \ CFLAGS="-Wall -Werror -fno-stack-protector -w" \ LDFLAGS=" -w" \ "/bin/bash" ./configure \ --prefix=/home/simonpj/5builds/HEAD/libffi/build/inst \ --libdir=/home/simonpj/5builds/HEAD/libffi/build/inst/lib \ --disable-docs \ --enable-static=yes \ --enable-shared=yes \ --host=x86_64-unknown-linux checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu continue configure in default builddir "./x86_64-unknown-linux-gnu" ....exec /bin/sh .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux gnu" checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gsed... sed checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for x86_64-unknown-linux-strip... no checking for strip... strip checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for x86_64-unknown-linux-gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether we are using the GNU C++ compiler... no checking whether gcc accepts -g... no checking dependency style of gcc... gcc3 checking dependency style of gcc... gcc3 checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... ld.gold checking if the linker (ld.gold) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... nm checking the name lister (nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1635000 checking how to convert x86_64-pc-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop checking for ld.gold option to reload object files... -r checking for x86_64-unknown-linux-objdump... no checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for x86_64-unknown-linux-dlltool... no checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for x86_64-unknown-linux-ar... ar checking for archiver @FILE support... @ checking for x86_64-unknown-linux-strip... strip checking for x86_64-unknown-linux-ranlib... ranlib checking command to parse nm output from gcc object... ok checking for sysroot... no checking for a working dd... /bin/dd checking how to truncate binary pipes... /bin/dd bs=4096 count=1 checking for x86_64-unknown-linux-mt... no checking for mt... mt checking if mt is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... yes checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (ld.gold -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... /lib/cpp configure: error: in `/home/simonpj/5builds/HEAD/libffi/build/x86_64-unknown-linux-gnu': configure: error: C++ preprocessor "/lib/cpp" fails sanity check See `config.log' for more details libffi/ghc.mk:47: recipe for target 'libffi/stamp.ffi.static-shared.configure' failed make[1]: *** [libffi/stamp.ffi.static-shared.configure] Error 1 Makefile:122: recipe for target 'all' failed make: *** [all] Error 2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at well-typed.com Mon Nov 20 14:33:15 2017 From: david at well-typed.com (David Feuer) Date: Mon, 20 Nov 2017 09:33:15 -0500 Subject: Linux failure Message-ID: <20171120140338.27702BC8AA@haskell.org> Digging around online, I suspect that you may not have the GNU C++ compiler installed at all. Take this with a shaker of salt. David FeuerWell-Typed, LLP -------- Original message --------From: Simon Peyton Jones via ghc-devs Date: 11/20/17 8:14 AM (GMT-05:00) To: ghc-devs at haskell.org Subject: Linux failure Hi devs I've got my Linux box upgraded to a recent version of Ubuntu, so I'm no longer stuck on gcc version. But I can't build either.  Something strange with /lib/cpp fails sanity check.  The log is below. Can anyone help? I'm a bit stalled. This call to "make" follows the same failure when I typed "sh validate -fast".   But by carrying on from where it got stuck the log is way shorter. Thanks Simon simonpj at cam-05-unx:~/5builds/HEAD$ make ===--- building phase 0 make --no-print-directory -f ghc.mk phase=0 phase_0_builds make[1]: Nothing to be done for 'phase_0_builds'. ===--- building phase 1 make --no-print-directory -f ghc.mk phase=1 phase_1_builds make[1]: Nothing to be done for 'phase_1_builds'. ===--- building final phase make --no-print-directory -f ghc.mk phase=final all rts/ghc.mk:547: rts/dist/build/.depend-v-dyn-l-debug-thr-thr_debug-thr_l-debug_dyn-thr_dyn-thr_debug_dyn-l_dyn-thr_l_dyn.c_asm: No such file or directory libraries/ghci/ghc.mk:4: libraries/ghci/dist-install/build/.depend-v-dyn.haskell: No such file or directory "rm" -f libffi/stamp.ffi.static.configure "rm" -f libffi/stamp.ffi.static.build "rm" -f libffi/stamp.ffi.static.install "rm" -f libffi/stamp.ffi.static-shared.configure "rm" -f libffi/stamp.ffi.static-shared.build "rm" -f libffi/stamp.ffi.static-shared.install "rm" -rf  libffi/build cat libffi-tarballs/libffi*.tar.gz | /bin/gzip -d | { cd libffi && /bin/tar -xf - ; } mv libffi/libffi-* libffi/build cp "/home/simonpj/5builds/HEAD/config.guess" libffi/build/config.guess cp "/home/simonpj/5builds/HEAD/config.sub"   libffi/build/config.sub chmod +x libffi/ln # We need to use -MMD rather than -MD, as otherwise we get paths # like c:/... in the dependency files on Windows, and the extra # colons break make mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed "s/-MD/-MMD/" < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # We attempt to specify the installation directory below with --libdir, # but libffi installs into 'toolexeclibdir' instead, which may differ # on systems where gcc has multilib support. Force libffi to use libdir. # (https://sourceware.org/ml/libffi-discuss/2014/msg00016.html) mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's:@toolexeclibdir@:$(libdir):g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in # install-sh is used when /usr/bin/install is missing; ensure its # path in libffi's Makefile is correct. See GHC #11109. mv libffi/build/Makefile.in libffi/build/Makefile.in.orig sed 's|@INSTALL@|$(subst ../install-sh,/home/simonpj/5builds/HEAD/install-sh, at INSTALL@)|g' < libffi/build/Makefile.in.orig > libffi/build/Makefile.in cd libffi && \     cd build && \     CC=gcc \     CXX=gcc \     LD=ld.gold \     AR=ar \     NM=nm \     RANLIB=ranlib \         CFLAGS="-Wall -Werror -fno-stack-protector -w" \         LDFLAGS=" -w" \         "/bin/bash" ./configure \           --prefix=/home/simonpj/5builds/HEAD/libffi/build/inst \           --libdir=/home/simonpj/5builds/HEAD/libffi/build/inst/lib \           --disable-docs \           --enable-static=yes \           --enable-shared=yes \           --host=x86_64-unknown-linux checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu continue configure in default builddir "./x86_64-unknown-linux-gnu" ....exec /bin/sh .././configure "--srcdir=.." "--enable-builddir=x86_64-unknown-linux-gnu" "linux gnu" checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gsed... sed checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for x86_64-unknown-linux-strip... no checking for strip... strip checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for x86_64-unknown-linux-gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking whether we are using the GNU C++ compiler... no checking whether gcc accepts -g... no checking dependency style of gcc... gcc3 checking dependency style of gcc... gcc3 checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... ld.gold checking if the linker (ld.gold) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... nm checking the name lister (nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1635000 checking how to convert x86_64-pc-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop checking for ld.gold option to reload object files... -r checking for x86_64-unknown-linux-objdump... no checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for x86_64-unknown-linux-dlltool... no checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for x86_64-unknown-linux-ar... ar checking for archiver @FILE support... @ checking for x86_64-unknown-linux-strip... strip checking for x86_64-unknown-linux-ranlib... ranlib checking command to parse nm output from gcc object... ok checking for sysroot... no checking for a working dd... /bin/dd checking how to truncate binary pipes... /bin/dd bs=4096 count=1 checking for x86_64-unknown-linux-mt... no checking for mt... mt checking if mt is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... yes checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (ld.gold -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... /lib/cpp configure: error: in `/home/simonpj/5builds/HEAD/libffi/build/x86_64-unknown-linux-gnu': configure: error: C++ preprocessor "/lib/cpp" fails sanity check See `config.log' for more details libffi/ghc.mk:47: recipe for target 'libffi/stamp.ffi.static-shared.configure' failed make[1]: *** [libffi/stamp.ffi.static-shared.configure] Error 1 Makefile:122: recipe for target 'all' failed make: *** [all] Error 2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at ara.io Mon Nov 20 21:06:32 2017 From: me at ara.io (Ara Adkins) Date: Mon, 20 Nov 2017 21:06:32 +0000 Subject: Commit Notifications Flood Message-ID: Is anyone else getting an absolute flood of commit emails related to haddock? I remember this happening a little while back with the hadrian merge into the GHC tree. Best, _ara -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Mon Nov 20 22:58:34 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 20 Nov 2017 17:58:34 -0500 Subject: Commit Notifications Flood In-Reply-To: References: Message-ID: <87shd8ftgn.fsf@ben-laptop.smart-cactus.org> Ara Adkins writes: > Is anyone else getting an absolute flood of commit emails related to > haddock? I remember this happening a little while back with the hadrian > merge into the GHC tree. > Indeed it seems that the commit hook is running amuck [1] Herbert, do you know what we can do about this? Cheers, - Ben [1] https://mail.haskell.org/pipermail/ghc-commits/2017-November/thread.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Mon Nov 20 23:01:08 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 20 Nov 2017 23:01:08 +0000 Subject: Commit Notifications Flood In-Reply-To: References: Message-ID: I’m getting them too ☹ From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Ara Adkins Sent: 20 November 2017 21:07 To: GHC developers Subject: Commit Notifications Flood Is anyone else getting an absolute flood of commit emails related to haddock? I remember this happening a little while back with the hadrian merge into the GHC tree. Best, _ara -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at ara.io Mon Nov 20 23:18:56 2017 From: me at ara.io (Ara Adkins) Date: Mon, 20 Nov 2017 23:18:56 +0000 Subject: Commit Notifications Flood In-Reply-To: References: Message-ID: I've put a redirect rule on them for now, but given that this is the second time that it's happened (as far as I know), is it possible to modify the way in which the commit hook sends notifications for new branches with a significant history? I don't really know much about the exact message generation process, but perhaps it is possible to provide a digest when pushing a new branch. Something that lists the branch name and the number of commits, for example. Best, _ara On Mon, Nov 20, 2017 at 11:01 PM, Simon Peyton Jones wrote: > I’m getting them too ☹ > > > > *From:* ghc-devs [mailto:ghc-devs-bounces at haskell.org] *On Behalf Of *Ara > Adkins > *Sent:* 20 November 2017 21:07 > *To:* GHC developers > *Subject:* Commit Notifications Flood > > > > Is anyone else getting an absolute flood of commit emails related to > haddock? I remember this happening a little while back with the hadrian > merge into the GHC tree. > > > > Best, > > _ara > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Tue Nov 21 00:36:26 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 20 Nov 2017 19:36:26 -0500 Subject: Commit Notifications Flood In-Reply-To: References: Message-ID: <87k1ykfoxn.fsf@ben-laptop.smart-cactus.org> Ara Adkins writes: > I've put a redirect rule on them for now, but given that this is the second > time that it's happened (as far as I know), is it possible to modify the > way in which the commit hook sends notifications for new branches with a > significant history? I don't really know much about the exact message > generation process, but perhaps it is possible to provide a digest when > pushing a new branch. Something that lists the branch name and the number > of commits, for example. > Indeed we really should set an upper bound on the number of messages that it will send. I agree that this is quite ridiculous. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From shumovichy at gmail.com Tue Nov 21 17:43:41 2017 From: shumovichy at gmail.com (Yuras Shumovich) Date: Tue, 21 Nov 2017 20:43:41 +0300 Subject: Can strict ST break referential transparency? Message-ID: <1511286221.2233.20.camel@gmail.com> Hello, I was evaluating a possibility that linear types can break referential transparency [1], exactly like lazy ST [2]. But on the way I realized that even strict ST may suffer from the same issue. If ST computation is interrupted by e.g. async exception, runtime will "freeze" it at the point where it was interrupted [3]. So the question: is the "freezed" computation just a normal thunk? Note that the runtime doesn't guarantee that a thunk will be evaluated only once [4]. If the "freezed" thunk captures e.g. STRef, and will be evaluated twice, its effect could become observable from outside, just like in case of lazy ST. I tried to check the theory by stress testing RTS. Unfortunately I immediately discovered a runtime crash [5], which is probably not related to my question. Hope someone will be able to clarify things for me. Thanks, Yuras. [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomment -345553071 [2] https://ghc.haskell.org/trac/ghc/ticket/14497 [3] See section 8 there: https://www.microsoft.com/en-us/research/wp-co ntent/uploads/2016/07/asynch-exns.pdf [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/09 /2005-haskell.pdf [5] https://ghc.haskell.org/trac/ghc/ticket/14497 From shumovichy at gmail.com Tue Nov 21 18:00:52 2017 From: shumovichy at gmail.com (Yuras Shumovich) Date: Tue, 21 Nov 2017 21:00:52 +0300 Subject: Can strict ST break referential transparency? In-Reply-To: <1511286221.2233.20.camel@gmail.com> References: <1511286221.2233.20.camel@gmail.com> Message-ID: <1511287252.2233.21.camel@gmail.com> Sorry, link [2] meant to be https://ghc.haskell.org/trac/ghc/ticket/11760 21-11-2017, Аўт а 20:43 +0300, Yuras Shumovich напісаў: > Hello, > > I was evaluating a possibility that linear types can break > referential > transparency [1], exactly like lazy ST [2]. > > But on the way I realized that even strict ST may suffer from the > same > issue. If ST computation is interrupted by e.g. async exception, > runtime will "freeze" it at the point where it was interrupted [3]. > > So the question: is the "freezed" computation just a normal thunk? > Note > that the runtime doesn't guarantee that a thunk will be evaluated > only > once [4]. If the "freezed" thunk captures e.g. STRef, and will be > evaluated twice, its effect could become observable from outside, > just > like in case of lazy ST. > > I tried to check the theory by stress testing RTS. Unfortunately I > immediately discovered a runtime crash [5], which is probably not > related to my question. > > Hope someone will be able to clarify things for me. > > Thanks, > Yuras. > > [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomme > nt > -345553071 > [2] https://ghc.haskell.org/trac/ghc/ticket/14497 > [3] See section 8 there: https://www.microsoft.com/en-us/research/wp- > co > ntent/uploads/2016/07/asynch-exns.pdf > [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/ > 09 > /2005-haskell.pdf > [5] https://ghc.haskell.org/trac/ghc/ticket/14497 From david at well-typed.com Tue Nov 21 21:49:29 2017 From: david at well-typed.com (David Feuer) Date: Tue, 21 Nov 2017 16:49:29 -0500 Subject: GHCi recompilation avoidance UI In-Reply-To: References: <7587879.l535S4SiGz@squirrel> Message-ID: <5110217.Pvi1NnV8H5@squirrel> I started digging back into this today, particularly considering Simon PJ's view that it's a bit odd for optimization flags to imply -fobject-code (specifically because we could potentially support optimization for the bytecode interpreter some day). I'm left even more lost about exactly what we want. I believe it's fairly clear that, as Simon M wrote, > [W]e'll want at least -fignore-optim-changes to be the default, so that GHCi > does the expected thing when you have compiled object files. Based on Simon PJ's comment, I believe we want to *continue* to discard optimization flags when -fobject-code is not enabled. As for my suggestion in (2), I spent the last couple hours attempting to figure out what would be necessary to allow :load *M to load a module interpreted even when using -fobject-code, but found myself utterly lost in the module loading logic. I see that the IIModule constructor is deeply involved in this, but I haven't been able to figure out where/how that interacts with -fobject-code to determine whether the module will actually be loaded interpreted or compiled. Can someone give me a clue? On Thursday, November 2, 2017 10:21:07 AM EST Simon Marlow wrote: > On 31 October 2017 at 15:42, David Feuer wrote: > > > Changes in GHC 8.2.1 lead to a lot of recompilation, because GHCi now > > refuses to load optimized > > code unless -fobject-code (and optimization flags) are enabled. I propose > > the following slight > > modification to https://ghc.haskell.org/trac/ghc/ticket/13604#comment:48 > > > > 1. Optimization flags (except -O0) imply -fobject-code. This ensures that > > GHC respects optimization flags regardless of --interactive. > > > > 2. Even when -fobject-code is on, :load *M will load M as bytecode. This > > provides the "escape hatch" from -fobject-code that you need to use > > debugging features, etc. > > > > Yes, I think this is probably what we want. I'm not sure how smooth it will > be to implement though. > > > > 3. New -fignore-optim-changes and -fignore-hpc-changes (​​Phab:D4123) > > flags should enable users to put together object code and bytecode with > > diverse optimization levels/options and HPC options while still updating > > automatically based on source changes and whether profiling is enabled. > > > > As I mentioned on the diff, I think we'll want at least > -fignore-optim-changes to be the default, so that GHCi does the expected > thing when you have compiled object files. > > Cheers > Simon From ben at well-typed.com Tue Nov 21 22:00:36 2017 From: ben at well-typed.com (Ben Gamari) Date: Tue, 21 Nov 2017 17:00:36 -0500 Subject: [ANNOUNCE] GHC 8.2.2 released Message-ID: <878tezfg1u.fsf@ben-laptop.smart-cactus.org> =============================================== The Glasgow Haskell Compiler -- version 8.2.2 =============================================== The GHC Team is pleased to announce a new minor release of GHC. This release builds on the performance and stability improvements of 8.2.1, fixing a variety of correctness bugs, improving error messages, and making the compiler more portable. Notable bug-fixes include * A correctness issue resulting in segmentation faults in some FFI-users (#13707, #14346) * A correctness issue resulting in undefined behavior in some programs using STM (#14171) * A bug which may have manifested in segmentation faults in out-of-memory condition (#14329) * clearBit of Natural no longer bottoms (#13203) * A specialisation bug resulting in exponential blowup of compilation time in some specialisation-intensive programs (#14379) * ghc-pkg now works even in environments with misconfigured NFS mounts (#13945) * GHC again supports production of position-independent executables (#13702) * Better error messages around kind mismatches (#11198, #12373, #13530, #13610) A thorough list of the changes in the release can be found in the release notes, https://haskell.org/ghc/docs/8.2.2/html/users_guide/release-8-2-2.html How to get it ~~~~~~~~~~~~~ This release can be downloaded from https://www.haskell.org/ghc/download_ghc_8_2_2.html For older versions see https://www.haskell.org/ghc/ We supply binary builds in the native package format for many platforms, and the source distribution is available from the same place. Background ~~~~~~~~~~ Haskell is a standardized lazy functional programming language. GHC is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating efficient code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces. GHC is distributed under a BSD-style open source license. A wide variety of Haskell related resources (tutorials, libraries, specifications, documentation, compilers, interpreters, references, contact information, links to research groups) are available from the Haskell home page (see below). On-line GHC-related resources ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Relevant URLs on the World-Wide Web: GHC home page https://www.haskell.org/ghc/ GHC developers' home page https://ghc.haskell.org/trac/ghc/ Haskell home page https://www.haskell.org/ Supported Platforms ~~~~~~~~~~~~~~~~~~~ The list of platforms we support, and the people responsible for them, is here: https://ghc.haskell.org/trac/ghc/wiki/Contributors Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform: https://ghc.haskell.org/trac/ghc/wiki/Building Developers ~~~~~~~~~~ We welcome new contributors. Instructions on accessing our source code repository, and getting started with hacking on GHC, are available from the GHC's developer's site: https://ghc.haskell.org/trac/ghc/ Mailing lists ~~~~~~~~~~~~~ We run mailing lists for GHC users and bug reports; to subscribe, use the web interfaces at https://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets There are several other haskell and ghc-related mailing lists on www.haskell.org; for the full list, see https://mail.haskell.org/cgi-bin/mailman/listinfo Many GHC developers hang out on #haskell on IRC: https://www.haskell.org/haskellwiki/IRC_channel Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here: https://www.haskell.org/ghc/reportabug -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Tue Nov 21 22:31:53 2017 From: ben at well-typed.com (Ben Gamari) Date: Tue, 21 Nov 2017 17:31:53 -0500 Subject: [ANNOUNCE] GHC 8.2.2 released In-Reply-To: <878tezfg1u.fsf@ben-laptop.smart-cactus.org> References: <878tezfg1u.fsf@ben-laptop.smart-cactus.org> Message-ID: <8760a3femx.fsf@ben-laptop.smart-cactus.org> Ben Gamari writes: > A thorough list of the changes in the release can be found in the release > notes, > > https://haskell.org/ghc/docs/8.2.2/html/users_guide/release-8-2-2.html > For the record this link was incorrect. It should be https://downloads.haskell.org/~ghc/8.2.2/docs/html/users_guide/8.2.2-notes.html My apologies for the confusion. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Wed Nov 22 03:38:26 2017 From: ben at well-typed.com (Ben Gamari) Date: Tue, 21 Nov 2017 22:38:26 -0500 Subject: GHCi recompilation avoidance UI In-Reply-To: <5110217.Pvi1NnV8H5@squirrel> References: <7587879.l535S4SiGz@squirrel> <5110217.Pvi1NnV8H5@squirrel> Message-ID: <87k1yj7zkl.fsf@ben-laptop.smart-cactus.org> David Feuer writes: > I started digging back into this today, particularly considering Simon PJ's view > that it's a bit odd for optimization flags to imply -fobject-code (specifically > because we could potentially support optimization for the bytecode > interpreter some day). I'm left even more lost about exactly what we want. > I believe it's fairly clear that, as Simon M wrote, > >> [W]e'll want at least -fignore-optim-changes to be the default, so that GHCi >> does the expected thing when you have compiled object files. > > Based on Simon PJ's comment, I believe we want to *continue* to discard > optimization flags when -fobject-code is not enabled. As for my suggestion in (2), > I spent the last couple hours attempting to figure out what would be necessary > to allow :load *M to load a module interpreted even when using -fobject-code, > but found myself utterly lost in the module loading logic. I see that the IIModule > constructor is deeply involved in this, but I haven't been able to figure out > where/how that interacts with -fobject-code to determine whether the module > will actually be loaded interpreted or compiled. Can someone give me a clue? > It seems to me like the place to start would be occurrences of the targetAllowObjCode field of Target. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From marlowsd at gmail.com Wed Nov 22 08:41:16 2017 From: marlowsd at gmail.com (Simon Marlow) Date: Wed, 22 Nov 2017 08:41:16 +0000 Subject: GHCi recompilation avoidance UI In-Reply-To: <5110217.Pvi1NnV8H5@squirrel> References: <7587879.l535S4SiGz@squirrel> <5110217.Pvi1NnV8H5@squirrel> Message-ID: David, Perhaps it would be good to defer changing the behaviour of :load *M (I believe you that it's hard, that code is quite convoluted) and for now just focus on making GHCi able to load compiled object code again, which I think is a much simpler problem? Cheers Simon On 21 November 2017 at 21:49, David Feuer wrote: > I started digging back into this today, particularly considering Simon > PJ's view > that it's a bit odd for optimization flags to imply -fobject-code > (specifically > because we could potentially support optimization for the bytecode > interpreter some day). I'm left even more lost about exactly what we want. > I believe it's fairly clear that, as Simon M wrote, > > > [W]e'll want at least -fignore-optim-changes to be the default, so that > GHCi > > does the expected thing when you have compiled object files. > > Based on Simon PJ's comment, I believe we want to *continue* to discard > optimization flags when -fobject-code is not enabled. As for my suggestion > in (2), > I spent the last couple hours attempting to figure out what would be > necessary > to allow :load *M to load a module interpreted even when using > -fobject-code, > but found myself utterly lost in the module loading logic. I see that the > IIModule > constructor is deeply involved in this, but I haven't been able to figure > out > where/how that interacts with -fobject-code to determine whether the module > will actually be loaded interpreted or compiled. Can someone give me a > clue? > > On Thursday, November 2, 2017 10:21:07 AM EST Simon Marlow wrote: > > On 31 October 2017 at 15:42, David Feuer wrote: > > > > > Changes in GHC 8.2.1 lead to a lot of recompilation, because GHCi now > > > refuses to load optimized > > > code unless -fobject-code (and optimization flags) are enabled. I > propose > > > the following slight > > > modification to https://ghc.haskell.org/trac/ > ghc/ticket/13604#comment:48 > > > > > > 1. Optimization flags (except -O0) imply -fobject-code. This ensures > that > > > GHC respects optimization flags regardless of --interactive. > > > > > > 2. Even when -fobject-code is on, :load *M will load M as bytecode. > This > > > provides the "escape hatch" from -fobject-code that you need to use > > > debugging features, etc. > > > > > > > Yes, I think this is probably what we want. I'm not sure how smooth it > will > > be to implement though. > > > > > > > 3. New -fignore-optim-changes and -fignore-hpc-changes (​​Phab:D4123) > > > flags should enable users to put together object code and bytecode with > > > diverse optimization levels/options and HPC options while still > updating > > > automatically based on source changes and whether profiling is enabled. > > > > > > > As I mentioned on the diff, I think we'll want at least > > -fignore-optim-changes to be the default, so that GHCi does the expected > > thing when you have compiled object files. > > > > Cheers > > Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Wed Nov 22 13:30:11 2017 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Wed, 22 Nov 2017 15:30:11 +0200 Subject: [GHC] #14490: TTG Snags In-Reply-To: <059.1929b3a2405dc60ac99c63c0e0fd7afd@haskell.org> References: <044.f4edb979bc1a87286428a7690bab92e2@haskell.org> <059.1929b3a2405dc60ac99c63c0e0fd7afd@haskell.org> Message-ID: Yes, but it is not feasible before GHC 8.6, due to needing GHC >= 8.2 for bootstrapping. On 22 November 2017 at 15:26, GHC wrote: > #14490: TTG Snags > -------------------------------------+---------------------- > --------------- > Reporter: alanz | Owner: (none) > Type: bug | Status: new > Priority: normal | Milestone: > Component: Compiler | Version: 8.3 > Resolution: | Keywords: > Operating System: Unknown/Multiple | Architecture: > | Unknown/Multiple > Type of failure: None/Unknown | Test Case: > Blocked By: | Blocking: > Related Tickets: #14482 | Differential Rev(s): > Wiki Page: | > ImplementingTreesThatGrow | > -------------------------------------+---------------------- > --------------- > > Comment (by Shayan-Najd): > > > So making a Data instance per concrete version may be the best way to > go. Plan B. > > Regardless of the GHC version we use, the question is whether Plan B > really fixes the build-time problem. Can we test it? > > -- > Ticket URL: > GHC > The Glasgow Haskell Compiler > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at well-typed.com Wed Nov 22 20:37:00 2017 From: david at well-typed.com (David Feuer) Date: Wed, 22 Nov 2017 15:37:00 -0500 Subject: Can strict ST break referential transparency? Message-ID: <20171122200648.6B808BC8B7@haskell.org> If there is indeed a problem, I suspect the right way to fix it is to make sure that no partially evaluated thunk is ever resumed twice. Inter-thread exceptions are presumably rare enough that we don't have to worry *too* much about their cost. David FeuerWell-Typed, LLP -------- Original message --------From: Yuras Shumovich Date: 11/21/17 12:43 PM (GMT-05:00) To: ghc-devs Subject: Can strict ST break referential transparency? Hello, I was evaluating a possibility that linear types can break referential transparency [1], exactly like lazy ST [2]. But on the way I realized that even strict ST may suffer from the same issue. If ST computation is interrupted by e.g. async exception, runtime will "freeze" it at the point where it was interrupted [3]. So the question: is the "freezed" computation just a normal thunk? Note that the runtime doesn't guarantee that a thunk will be evaluated only once [4]. If the "freezed" thunk captures e.g. STRef, and will be evaluated twice, its effect could become observable from outside, just like in case of lazy ST. I tried to check the theory by stress testing RTS. Unfortunately I immediately discovered a runtime crash [5], which is probably not related to my question. Hope someone will be able to clarify things for me. Thanks, Yuras. [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomment -345553071 [2] https://ghc.haskell.org/trac/ghc/ticket/14497 [3] See section 8 there: https://www.microsoft.com/en-us/research/wp-co ntent/uploads/2016/07/asynch-exns.pdf [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/09 /2005-haskell.pdf [5] https://ghc.haskell.org/trac/ghc/ticket/14497 _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From shumovichy at gmail.com Thu Nov 23 12:34:57 2017 From: shumovichy at gmail.com (Yuras Shumovich) Date: Thu, 23 Nov 2017 15:34:57 +0300 Subject: Can strict ST break referential transparency? In-Reply-To: <5a15dff1.090f1c0a.6b315.9e5dSMTPIN_ADDED_MISSING@mx.google.com> References: <5a15dff1.090f1c0a.6b315.9e5dSMTPIN_ADDED_MISSING@mx.google.com> Message-ID: <1511440497.17688.7.camel@gmail.com> 22-11-2017, Срд а 15:37 -0500, David Feuer напісаў: > If there is indeed a problem, I suspect the right way to fix it is to > make sure that no partially evaluated thunk is ever resumed twice. > Inter-thread exceptions are presumably rare enough that we don't have > to worry *too* much about their cost. Sounds reasonable. But probably it already works that way? Though it doesn't seem to be covered in the papers, and looks like nobody knows an answer to my question. It makes me worry a bit. On the other hand, right now resuming frozen thunk usually crashes/hangs as discovered in #14497. Nobody noticed that so far in the wild, so probably the subject is just a dark corner case, not worse the efforts. > > > David FeuerWell-Typed, LLP > -------- Original message --------From: Yuras Shumovich mail.com> Date: 11/21/17 12:43 PM (GMT-05:00) To: ghc-devs s at haskell.org> Subject: Can strict ST break referential > transparency? > > Hello, > > I was evaluating a possibility that linear types can break > referential > transparency [1], exactly like lazy ST [2]. > > But on the way I realized that even strict ST may suffer from the > same > issue. If ST computation is interrupted by e.g. async exception, > runtime will "freeze" it at the point where it was interrupted [3]. > > So the question: is the "freezed" computation just a normal thunk? > Note > that the runtime doesn't guarantee that a thunk will be evaluated > only > once [4]. If the "freezed" thunk captures e.g. STRef, and will be > evaluated twice, its effect could become observable from outside, > just > like in case of lazy ST. > > I tried to check the theory by stress testing RTS. Unfortunately I > immediately discovered a runtime crash [5], which is probably not > related to my question. > > Hope someone will be able to clarify things for me. > > Thanks, > Yuras. > > [1] https://github.com/ghc-proposals/ghc-proposals/pull/91#issuecomme > nt > -345553071 > [2] https://ghc.haskell.org/trac/ghc/ticket/14497 > [3] See section 8 there: https://www.microsoft.com/en-us/research/wp- > co > ntent/uploads/2016/07/asynch-exns.pdf > [4] https://www.microsoft.com/en-us/research/wp-content/uploads/2005/ > 09 > /2005-haskell.pdf > [5] https://ghc.haskell.org/trac/ghc/ticket/14497 > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Thu Nov 23 15:20:07 2017 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 23 Nov 2017 10:20:07 -0500 Subject: Can strict ST break referential transparency? In-Reply-To: <1511286221.2233.20.camel@gmail.com> References: <1511286221.2233.20.camel@gmail.com> Message-ID: <87r2sp6mzj.fsf@ben-laptop.smart-cactus.org> Yuras Shumovich writes: > Hello, > Hello, Sorry for the late reply; this required a bit of reflection. The invariants surrounding the suspension of ST computations is a rather delicate and poorly documented area. I believe the asynchronous exception case which you point out is precisely #13615. The solution there was, as David suggests, ensure that no resulting thunk could be entered more than once by a very strict blackholing protocol. Note that this isn't normal "eager" blackholing protocol, which still might allow multiple entrancy. It's rather a more strict variant, requiring two atomic operations. I can't be certain that there aren't more cases like this, but I suspect not since most asynchronous suspensions where the resulting thunk might "leak" back into the program go through the raiseAsync codepath that was fixed in #13615. Cheers, - ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From shumovichy at gmail.com Thu Nov 23 16:11:37 2017 From: shumovichy at gmail.com (Yuras Shumovich) Date: Thu, 23 Nov 2017 19:11:37 +0300 Subject: Can strict ST break referential transparency? In-Reply-To: <87r2sp6mzj.fsf@ben-laptop.smart-cactus.org> References: <1511286221.2233.20.camel@gmail.com> <87r2sp6mzj.fsf@ben-laptop.smart-cactus.org> Message-ID: <1511453497.17688.10.camel@gmail.com> So my theory is correct, but it is already fixed in 8.2. Nice! Thank you for clarification! Yuras. 23-11-2017, Чцв а 10:20 -0500, Ben Gamari напісаў: > Yuras Shumovich writes: > > > Hello, > > > > Hello, > > Sorry for the late reply; this required a bit of reflection. The > invariants surrounding the suspension of ST computations is a rather > delicate and poorly documented area. > > I believe the asynchronous exception case which you point out is > precisely #13615. The solution there was, as David suggests, ensure > that > no resulting thunk could be entered more than once by a very strict > blackholing protocol. Note that this isn't normal "eager" blackholing > protocol, which still might allow multiple entrancy. It's rather a > more > strict variant, requiring two atomic operations. > > I can't be certain that there aren't more cases like this, but I > suspect > not since most asynchronous suspensions where the resulting thunk > might > "leak" back into the program go through the raiseAsync codepath that > was > fixed in #13615. > > Cheers, > > - ben > From johnw at newartisans.com Thu Nov 23 19:20:34 2017 From: johnw at newartisans.com (John Wiegley) Date: Thu, 23 Nov 2017 11:20:34 -0800 Subject: Unstuck old messages Message-ID: Hello everyone, I was doing some maintenance on the mailman server that hosts this list, and several messages that were stuck in the queue for what appears to be a rather long time have been released. The queue is now empty, so if these messages are no longer relevant, please ignore them. -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2 From simonpj at microsoft.com Thu Nov 23 23:13:26 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 23 Nov 2017 23:13:26 +0000 Subject: [GHC] #14490: TTG Snags In-Reply-To: References: <044.f4edb979bc1a87286428a7690bab92e2@haskell.org> <059.1929b3a2405dc60ac99c63c0e0fd7afd@haskell.org> Message-ID: Yes, but it is not feasible before GHC 8.6, due to needing GHC >= 8.2 for bootstrapping. True. But 8.4 will fork off shortly; so once that is done, we are happy with perf, we can add the change to master in preparation for 8.6. Simon From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Alan & Kim Zimmerman Sent: 22 November 2017 13:30 To: ghc-devs at haskell.org Subject: Re: [GHC] #14490: TTG Snags Yes, but it is not feasible before GHC 8.6, due to needing GHC >= 8.2 for bootstrapping. On 22 November 2017 at 15:26, GHC > wrote: #14490: TTG Snags -------------------------------------+------------------------------------- Reporter: alanz | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #14482 | Differential Rev(s): Wiki Page: | ImplementingTreesThatGrow | -------------------------------------+------------------------------------- Comment (by Shayan-Najd): > So making a Data instance per concrete version may be the best way to go. Plan B. Regardless of the GHC version we use, the question is whether Plan B really fixes the build-time problem. Can we test it? -- Ticket URL: > GHC > The Glasgow Haskell Compiler -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Nov 23 23:19:17 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 23 Nov 2017 23:19:17 +0000 Subject: Can strict ST break referential transparency? In-Reply-To: <87r2sp6mzj.fsf@ben-laptop.smart-cactus.org> References: <1511286221.2233.20.camel@gmail.com> <87r2sp6mzj.fsf@ben-laptop.smart-cactus.org> Message-ID: If the conclusion is that there's a bug here, could someone distil the example into a ticket? Thanks Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Ben | Gamari | Sent: 23 November 2017 15:20 | To: Yuras Shumovich ; ghc-devs | Subject: Re: Can strict ST break referential transparency? | | Yuras Shumovich writes: | | > Hello, | > | Hello, | | Sorry for the late reply; this required a bit of reflection. The | invariants surrounding the suspension of ST computations is a rather | delicate and poorly documented area. | | I believe the asynchronous exception case which you point out is | precisely #13615. The solution there was, as David suggests, ensure that | no resulting thunk could be entered more than once by a very strict | blackholing protocol. Note that this isn't normal "eager" blackholing | protocol, which still might allow multiple entrancy. It's rather a more | strict variant, requiring two atomic operations. | | I can't be certain that there aren't more cases like this, but I suspect | not since most asynchronous suspensions where the resulting thunk might | "leak" back into the program go through the raiseAsync codepath that was | fixed in #13615. | | Cheers, | | - ben From jwlato at gmail.com Mon Nov 27 01:16:51 2017 From: jwlato at gmail.com (John Lato) Date: Mon, 27 Nov 2017 01:16:51 +0000 Subject: True multi stage Haskell In-Reply-To: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> References: <71c84b2f-61d0-e3c2-6936-01b365f26483@pdx.edu> Message-ID: Hi Tim, Several years ago I wrote a proof of concept of one way to implement this in Haskell, http://johnlato.blogspot.com/2012/10/runtime-meta-programming-in-haskell.html . I used TH to automatically lift expressions into a newtype-wrapped ExpQ that could be used for staged evaluation. More than one stage would probably have been tedious though, and a major problem with any non-trivial code was getting all the imports in place. I do think it would have been possible to automatically track imports though. Of course that was 5 years ago, so state of the art has moved on. TH has better facilities for this now, and the ghc API probably has something better too. On Fri, Nov 17, 2017, 10:06 Tim Sheard wrote: > After many years of hoping someone else would do this, I would like to > make GHC into a true multi-stage programming language. Here is how I > thought I might approach this. > > 1) Use the GHC as a library module. > 2) Use the LLVM backend. > > I have no experience with either of these tools. > Lets start simple, How would I write functions like > > compile :: String -> IO PtrToLLVMCode -- where the string is a small > Haskell program. > llvmCompile :: PtrToLLVMCode -> IO PtrToMachineCode > jumpTo:: PtrToMachineCode -> IO ans -- where ans is the "type" of the > string. > > > Any thoughts on how to get started? What papers to read, examples to > look at? > > I'd love to move to some more disciplined input type, a sort of (mono) > typed program > representation (with similar complexity) to Template Haskell Exp type. > > where (Exp t) is a data structure representing a Haskell program of type t. > > All offers of advice accepted. I'm trying to get started soon, and good > advice > about what to avoid is especially welcome. If any one wanted to help > with this, > that would be great. > > Tim Sheard > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Nov 28 04:52:41 2017 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 28 Nov 2017 12:52:41 +0800 Subject: `binary` serializing datatypes deriving Generic wrong on arm (32bit) with GHC HEAD Message-ID: Hi! while trying to make sure cross compilation with Template Haskell works properly with 8.4, I ran into the following situation: When serializing data types, e.g. `Name OccName NameFlavour` in the transmission of Template Haskell Splice results from a 32bit arm device to the x86_64 host ghc. We expect to see: ``` .- 0 (first constructor) .- 1 .- 4 (fifth constructor) .- 4 .- 3 v v v v v \NUL \NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOHf \EOT \NUL \NUL\NUL\NUL\NUL\NUL\NUL\NUL\EOTmain \NUL\NUL\NUL\NUL\NUL\NUL\NUL\ETXTmp '--' '----------------------------------' '---------------------------------' NameSpace PkgName ModName '-------------------------------' '--------------------------------------------------------------------------------' OccName NameG NameSpace PkgName ModName :: NameFlavour '-----------------------------------------------------------------------------------------------------------------------' Name OccName NameFlavour :: Name ``` However, the `NameSpace` on the 32bit arm ends up being 8 bytes. Even though the full `Namespace` data type can be fully serialized in a single byte. The `binary` package tries to compute the size it needs for a generic data type, using the following logic (from binary/src/Data/Binary/Generic.hs): ``` class SumSize f where sumSize :: Tagged f Word64 newtype Tagged (s :: * -> *) b = Tagged {unTagged :: b} instance (SumSize a, SumSize b) => SumSize (a :+: b) where sumSize = Tagged $ unTagged (sumSize :: Tagged a Word64) + unTagged (sumSize :: Tagged b Word64) instance SumSize (C1 c a) where sumSize = Tagged 1 ``` Thus for a simple sum type `data X = A | B` we should get a `sumSize` of 2. The arm32 device however ends up getting 2^33, because `sumSize :: Tagged a Word64` and `sumSize :: Tagged b Word64` each end up being 2^32. With some help from the nice folks in #ghc, I was able to conjure up the following condensed test case: ``` {-# LANGUAGE DeriveGeneric, KindSignatures, PolyKinds, CPP, ScopedTypeVariables, TypeOperators, TypeSynonymInstances, FlexibleInstances #-} {-# OPTIONS_GHC -O2 #-} import GHC.Generics import Data.Word import Debug.Trace data X = A | B deriving (Show, Generic) main :: IO () main = print (sumSize :: Tagged (Rep X)) -- like traceShowId, but allows us to prepend a message. t :: Show a => String -> a -> a #if TRACE t msg x = traceShow (msg ++ show x) x #else t _ = id #endif class SumSize f where sumSize :: Tagged f newtype Tagged (s :: * -> *) = Tagged {unTagged :: WORD} deriving Show instance (SumSize a, SumSize b) => SumSize (a :+: b) where sumSize = t "SumSize (a :+: b): " $ Tagged $ unTagged (t "a :+: b => sumSize :: Tagged a: " $ sumSize :: Tagged a) + unTagged (t "a :+: b => sumSize :: Tagged b: " $ sumSize :: Tagged b) instance SumSize (C1 c a) where sumSize = t "SumSize (C1 c a): " $ Tagged 1 instance SumSize a => SumSize (M1 D c a) where sumSize = t "SumSize (M1 D c a): " $ Tagged . unTagged $ (sumSize :: Tagged a) ``` compiling this with `-DWORD=Word32 -DTRACE=1` yields the correct result (=2), with `-DWORD=Word64 -DTRACE=0` as well. With `-DWORD=Word64 -DTRACE=2` the wrong result (=2^33) Optimization flags seem not to play any role when everything is in a single module (as the test case). As such I have attached the `-ddump-simple -dsuppress-all` files for the Word64 and Word32 with TRACE=1, as well as the diff between the Word32 and Word64 dump. The output with WORD=Word32, TRACE=1 is: ``` "SumSize (C1 c a): Tagged {unTagged = 1}" "a :+: b => sumSize :: Tagged a: Tagged {unTagged = 4294967296}" "a :+: b => sumSize :: Tagged b: Tagged {unTagged = 4294967296}" "SumSize (a :+: b): Tagged {unTagged = 8589934592}" "SumSize (M1 D c a): Tagged {unTagged = 8589934592}" Tagged {unTagged = 8589934592} ``` with WORD=Word64, TRACE=1 is: ``` "SumSize (C1 c a): Tagged {unTagged = 1}" "a :+: b => sumSize :: Tagged a: Tagged {unTagged = 1}" "a :+: b => sumSize :: Tagged b: Tagged {unTagged = 1}" "SumSize (a :+: b): Tagged {unTagged = 2}" "SumSize (M1 D c a): Tagged {unTagged = 2}" Tagged {unTagged = 2} ``` Any help with this would be greatly appreciated! Cheers, Moritz PS: I'm not absolutely sure, but this might also be related to https://ghc.haskell.org/trac/ghc/ticket/13513 -------------- next part -------------- A non-text attachment was scrubbed... Name: Main.word32.dump-simpl Type: application/octet-stream Size: 18358 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Main.word64.dump-simpl Type: application/octet-stream Size: 23750 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Main.dump-simpl.diff Type: application/octet-stream Size: 22950 bytes Desc: not available URL: From moritz.angermann at gmail.com Tue Nov 28 13:45:33 2017 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 28 Nov 2017 21:45:33 +0800 Subject: Word64 from int64_t on 32bit arm seems off. In-Reply-To: References: Message-ID: Given the following sample program: ``` import Data.Word f :: a -> a f x = x {-# NOINLINE f #-} x, y, x', y' :: Word64 x = 1 y = 1 x' = 4294967296 + 2 -- upper:1 + lower:2 y' = 4294967296 + 2 -- upper:1 + lower:2 main :: IO () main = let z = f x + f y z' = f x' + f y' in do { print (z == x + y) ; print z ; print (z' == x' + y') ; print z' } ``` This produces: ``` True 2 True 8589934596 ``` when compiled with `-O0` for me. and ``` False 8589934592 True 8589934596 ``` when compiled with `-O1` for me. Thus, if we start out with two Word64 that fit into the lower byte, we end up with the sum of both in the upper byte (with `-O1`). The difference between -O0 and -O1 is that -O1 goes through the primOps, and as such we end up with code like: ``` %26 = tail call i64 @hs_word64ToInt64(i64 2) %27 = tail call i64 @hs_word64ToInt64(i64 4294967296) %28 = tail call i64 @hs_plusInt64(i64 %27, i64 %26) %29 = tail call i64 @hs_int64ToWord64(i64 %28) ``` after which interestingly the result is correct. However the subsequent invocation of `@base_GHCziWord_W64zh_con_info`, seems to pick the wrong bytes for when reconstructing the Word64. If anyone got any idea, I'd be happy to know. Otherwise I guess I'd have to start adding debug information into the rts? Cheers, Moritz > On Nov 28, 2017, at 12:52 PM, Moritz Angermann wrote: > > Hi! > > while trying to make sure cross compilation with Template Haskell works properly > with 8.4, I ran into the following situation: > > When serializing data types, e.g. `Name OccName NameFlavour` in the transmission > of Template Haskell Splice results from a 32bit arm device to the x86_64 host ghc. > > We expect to see: > ``` > .- 0 (first constructor) .- 1 .- 4 (fifth constructor) .- 4 .- 3 > v v v v v > \NUL \NUL\NUL\NUL\NUL\NUL\NUL\NUL\SOHf \EOT \NUL \NUL\NUL\NUL\NUL\NUL\NUL\NUL\EOTmain \NUL\NUL\NUL\NUL\NUL\NUL\NUL\ETXTmp > '--' '----------------------------------' '---------------------------------' > NameSpace PkgName ModName > '-------------------------------' '--------------------------------------------------------------------------------' > OccName NameG NameSpace PkgName ModName :: NameFlavour > '-----------------------------------------------------------------------------------------------------------------------' > Name OccName NameFlavour :: Name > ``` > > However, the `NameSpace` on the 32bit arm ends up being 8 bytes. Even though the full > `Namespace` data type can be fully serialized in a single byte. > > The `binary` package tries to compute the size it needs for a generic data type, using > the following logic (from binary/src/Data/Binary/Generic.hs): > > ``` > class SumSize f where > sumSize :: Tagged f Word64 > > newtype Tagged (s :: * -> *) b = Tagged {unTagged :: b} > > instance (SumSize a, SumSize b) => SumSize (a :+: b) where > > sumSize = Tagged $ unTagged (sumSize :: Tagged a Word64) + > unTagged (sumSize :: Tagged b Word64) > > instance SumSize (C1 c a) where > sumSize = Tagged 1 > ``` > > > Thus for a simple sum type `data X = A | B` we should get a `sumSize` of 2. > The arm32 device however ends up getting 2^33, because `sumSize :: Tagged a Word64` > and `sumSize :: Tagged b Word64` each end up being 2^32. > > With some help from the nice folks in #ghc, I was able to conjure up the following > condensed test case: > > ``` > {-# LANGUAGE DeriveGeneric, KindSignatures, PolyKinds, CPP, > ScopedTypeVariables, TypeOperators, TypeSynonymInstances, > FlexibleInstances #-} > {-# OPTIONS_GHC -O2 #-} > > import GHC.Generics > > import Data.Word > import Debug.Trace > > data X = A | B deriving (Show, Generic) > > main :: IO () > main = print (sumSize :: Tagged (Rep X)) > > -- like traceShowId, but allows us to prepend a message. > t :: Show a => String -> a -> a > #if TRACE > t msg x = traceShow (msg ++ show x) x > #else > t _ = id > #endif > > class SumSize f where > sumSize :: Tagged f > > newtype Tagged (s :: * -> *) = Tagged {unTagged :: WORD} deriving Show > > instance (SumSize a, SumSize b) => SumSize (a :+: b) where > sumSize = t "SumSize (a :+: b): " $ Tagged $ unTagged (t "a :+: b => sumSize :: Tagged a: " $ sumSize :: Tagged a) + > unTagged (t "a :+: b => sumSize :: Tagged b: " $ sumSize :: Tagged b) > > instance SumSize (C1 c a) where > sumSize = t "SumSize (C1 c a): " $ Tagged 1 > > instance SumSize a => SumSize (M1 D c a) where > sumSize = t "SumSize (M1 D c a): " $ Tagged . unTagged $ (sumSize :: Tagged a) > > ``` > > compiling this with `-DWORD=Word32 -DTRACE=1` yields the correct result (=2), with `-DWORD=Word64 -DTRACE=0` as well. > With `-DWORD=Word64 -DTRACE=2` the wrong result (=2^33) > > Optimization flags seem not to play any role when everything is in a single module (as the test case). > > As such I have attached the `-ddump-simple -dsuppress-all` files for the Word64 and Word32 with TRACE=1, as well > as the diff between the Word32 and Word64 dump. > > The output with WORD=Word32, TRACE=1 is: > ``` > "SumSize (C1 c a): Tagged {unTagged = 1}" > "a :+: b => sumSize :: Tagged a: Tagged {unTagged = 4294967296}" > "a :+: b => sumSize :: Tagged b: Tagged {unTagged = 4294967296}" > "SumSize (a :+: b): Tagged {unTagged = 8589934592}" > "SumSize (M1 D c a): Tagged {unTagged = 8589934592}" > Tagged {unTagged = 8589934592} > ``` > > with WORD=Word64, TRACE=1 is: > ``` > "SumSize (C1 c a): Tagged {unTagged = 1}" > "a :+: b => sumSize :: Tagged a: Tagged {unTagged = 1}" > "a :+: b => sumSize :: Tagged b: Tagged {unTagged = 1}" > "SumSize (a :+: b): Tagged {unTagged = 2}" > "SumSize (M1 D c a): Tagged {unTagged = 2}" > Tagged {unTagged = 2} > ``` > > Any help with this would be greatly appreciated! > > Cheers, > Moritz > > PS: I'm not absolutely sure, but this might also be related to https://ghc.haskell.org/trac/ghc/ticket/13513 > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From nboldi at elte.hu Thu Nov 30 05:42:24 2017 From: nboldi at elte.hu (=?UTF-8?B?TsOpbWV0aCBCb2xkaXpzw6Fy?=) Date: Thu, 30 Nov 2017 14:42:24 +0900 Subject: GHC typecheck API Message-ID: Dear GHC developers, I'm developing a framework for development tools for Haskell. I use the GHC API to parse and typecheck the source files. I recently started to work on a quick-fix (automatic program correction) for Haskell source code (correcting parenthesis problems, like 'putStrLn "xxx" ++ show a' ==> 'putStrLn ("xxx" ++ show a)'). To do that I need to get the typed syntax tree even if the program contains type or rename errors. I could only do this by a nasty hack, adding a new TH module finalizer (tcg_th_modfinalizers) to extract the type checker's state before it fails. Is there a "correct" way to do this? I would not like this refactorings to be unusable if the GHC API changes. By the way, if you know of any similar attempt please let me know. Sincerely, Boldizsár Németh haskelltools.org From simonpj at microsoft.com Thu Nov 30 15:37:00 2017 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 30 Nov 2017 15:37:00 +0000 Subject: GHC typecheck API In-Reply-To: References: Message-ID: This sounds like a good project! For the most part things look good: * Most type checker errors arise from *type constraints*. The type checkder tries to solve these, but returns an elaborated syntax tree (i.e. typechecked, and annotated with types) even if constraint solving fails. * Some renamer errors are like this, notably out-of-scope variables. (They just show up as another constraint.) However there is historical baggage. Back in the beginning, most errors were treated by throwing an exception in the typechecker monad; such exceptions can be caught, so that we can get more than one error from the file, but no syntax tree is returned. Example let f = in If there was an error in we'd throw an exception, catch it at the 'let', give 'f' the type f :: forall a. a and continue to typecheck . The trouble with the exception stuff is that you don't get an elaborated syntax tree. So: I think you can get some of the way today, just by returning the tree anyway even if there is an error to report. But it'd take a bit more work to make more and more errors into things that don't throw an exception. (Look for failTc, failRn in thd code.) I'm not very familiar with the GHC API for this part, but others will be. I'm certain it can be improved, so rather than hacking around what is there already, do propose and implement improvements. Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Németh | Boldizsár | Sent: 30 November 2017 05:42 | To: ghc-devs at haskell.org | Subject: GHC typecheck API | | Dear GHC developers, | | I'm developing a framework for development tools for Haskell. I use the | GHC API to parse and typecheck the source files. I recently started to | work on a quick-fix (automatic program correction) for Haskell source | code (correcting parenthesis problems, like 'putStrLn "xxx" ++ show a' | ==> 'putStrLn ("xxx" ++ show a)'). To do that I need to get the typed | syntax tree even if the program contains type or rename errors. I could | only do this by a nasty hack, adding a new TH module finalizer | (tcg_th_modfinalizers) to extract the type checker's state before it | fails. Is there a "correct" way to do this? I would not like this | refactorings to be unusable if the GHC API changes. | | By the way, if you know of any similar attempt please let me know. | | Sincerely, | Boldizsár Németh | haskelltools.org | | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.hask | ell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7C364b8db55e064415650d08d537b | 5323f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636476173738772433&sda | ta=Kf5rZYwht8ZGBtGNNH6Q44wLIeefxzHn2UfDIdrNNMU%3D&reserved=0