From moritz.angermann at gmail.com Mon Jun 1 09:23:33 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Mon, 1 Jun 2020 17:23:33 +0800 Subject: GHCs dependencies (libraries) and maintenance Message-ID: Hi there! so this comes up periodically, and I think we need to discuss this. This is not related to anything right now, so if you wonder if I'm writing this because of something that just happened that I'm involved and you might have missed something, you probably did not. It came up on the #ghc IRC channel a few day ago. GHC depends on quite a set of libraries, and ships those in releases. When ever a new GHC release is cut, all these dependencies need to be on hackage and have release versions. We do not want to produce a GHC release which depends on in-flight packages. In-flight might happen for example due to GHC having to patch dependencies to make them work with HEAD. Everyone who maintains any kind of software online, knows that maintenance can be a drag, and then life happens, and what not. There are many very responsive maintainers and we all owe them a grate amount of gratitude towards their relentless work, keeping those libraries up to date and responding to questions, patches, ... I therefore would like to float the following idea to make the GHC release processes a bit more reliable. GHCHQ (that is those in charge of producing GHC releases for us all), are made co-maintainers on each library GHC depends on, to guarantee that GHC can move forward in the worst of circumstances. Now I would hope that in almost all cases GHCHQ would never have to maintain any of the dependencies actively, they deal with GHC already, so let's try to keep it that way. However GHCHQ can, after a 14day notification period, exercise the co-maintainance and cut releases (and upload them to hackage), should the maintainer not be able to do so on his own for various reasons. I'd like to see this as an insurance policy for GHC continuous development. The only alternative that I see would be that GHCHQ starts forking dependencies and initiates the hackage maintainer takeover protocol, which will cause additional delays, and incurs an extra burden on the GHC maintainers. I hope we can all agree that libraries that end up being dependencies of GHC should be held in high regards and form the very foundation GHC is built upon. As such it should be an honour to have GHCHQ being a co-maintainer for ones library, as it signifies that importances of the library for the continuous development of GHC. Again I don't expect much to change, except for GHCHQ becoming co-maintainers for libraries GHC depends on. The baseline expectation will remain as it is. However we will have ensured the frictionless development of GHC going forward. Cheers, Moritz From john.ericson at obsidian.systems Mon Jun 1 14:29:17 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Mon, 1 Jun 2020 10:29:17 -0400 Subject: GHCs dependencies (libraries) and maintenance In-Reply-To: References: Message-ID: I completely agree. I would like GHC be able to use *more* of the ecosystem than it does today, and there's no way we can can do that unless we dramatically decrease the pain per dependency. John On 6/1/20 5:23 AM, Moritz Angermann wrote: > Hi there! > > so this comes up periodically, and I think we need to discuss this. This is not > related to anything right now, so if you wonder if I'm writing this because of > something that just happened that I'm involved and you might have missed > something, you probably did not. It came up on the #ghc IRC channel a > few day ago. > > GHC depends on quite a set of libraries, and ships those in releases. When ever > a new GHC release is cut, all these dependencies need to be on hackage and > have release versions. We do not want to produce a GHC release which depends > on in-flight packages. In-flight might happen for example due to GHC having to > patch dependencies to make them work with HEAD. > > Everyone who maintains any kind of software online, knows that maintenance can > be a drag, and then life happens, and what not. There are many very responsive > maintainers and we all owe them a grate amount of gratitude towards their > relentless work, keeping those libraries up to date and responding to questions, > patches, ... > > I therefore would like to float the following idea to make the GHC > release processes > a bit more reliable. GHCHQ (that is those in charge of producing GHC > releases for > us all), are made co-maintainers on each library GHC depends on, to guarantee > that GHC can move forward in the worst of circumstances. Now I would > hope that in > almost all cases GHCHQ would never have to maintain any of the dependencies > actively, they deal with GHC already, so let's try to keep it that > way. However GHCHQ > can, after a 14day notification period, exercise the co-maintainance > and cut releases > (and upload them to hackage), should the maintainer not be able to do > so on his own > for various reasons. > > I'd like to see this as an insurance policy for GHC continuous > development. The only > alternative that I see would be that GHCHQ starts forking dependencies > and initiates > the hackage maintainer takeover protocol, which will cause additional > delays, and > incurs an extra burden on the GHC maintainers. > > I hope we can all agree that libraries that end up being dependencies > of GHC should > be held in high regards and form the very foundation GHC is built > upon. As such it > should be an honour to have GHCHQ being a co-maintainer for ones library, as it > signifies that importances of the library for the continuous development of GHC. > > Again I don't expect much to change, except for GHCHQ becoming co-maintainers > for libraries GHC depends on. The baseline expectation will remain as > it is. However we > will have ensured the frictionless development of GHC going forward. > > Cheers, > Moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From oleg.grenrus at iki.fi Mon Jun 1 14:39:00 2020 From: oleg.grenrus at iki.fi (Oleg Grenrus) Date: Mon, 1 Jun 2020 17:39:00 +0300 Subject: GHCs dependencies (libraries) and maintenance In-Reply-To: References: Message-ID: I want to ask a honest question: Who is GHC HQ? There is a page listing who is on - Haskell.org committee, - on core libraries committee (CLC), - GHC steering committee; - there is a list of named maintainers for core libraries, - and it is relatively easily to deduce who maintains other tools and libraries relevant to GHC (e.g. Cabal or say `shake` for hadrian). But who are in GHC HQ? Neither google nor gitlab.haskell.org search give any hints. --- I think the core issue here is bad or insufficient communication. GHC-8.12 beta release is planned to happen in three months. There are patches still in flight (e.g. for process, Cabal has tracking issue open). I'm sure that maintainers are able to ack on that. Maybe start by requiring these explicit acknowledgments? I also want to remind about first GHC-8.10 schedule posted on this list     October  18 2019:  start of one week freeze in preparation for branching     October  25 2019:  ghc-8.10 branch cut     November 8  2019:  8.10.1-alpha1     November 22 2019:  8.10.1-alpha2     December 6  2019:  8.10.1-alpha3     December 20 2019:  8.10.1-rc1     January  10 2020:  Final 8.10.1 release I haven't seen *any* updates to that. GHC-8.10.1 was released in end of March 2020, *three months later*, text issue was urgent in beginning of February. The plan for GHC-8.12 is to  * Mid-June 2020: Aim to have all major features in the tree  * Late-June 2020: Cut the ghc-8.12 branch  * June - August 2020: 3 alpha releases  * 1 September 2020: beta release  * 25 September 2020: Final 8.12.1 release are we still on track? What I'm trying to say, it is that it is hard to believe that issue is urgent for GHC. It's not the first time (relating to ghc-8.10) when schedules are not been kept. And this is one of reasons why I opened an issue about "Annual release cycle, its structure and long-term-support". - Oleg On 1.6.2020 12.23, Moritz Angermann wrote: > Hi there! > > so this comes up periodically, and I think we need to discuss this. This is not > related to anything right now, so if you wonder if I'm writing this because of > something that just happened that I'm involved and you might have missed > something, you probably did not. It came up on the #ghc IRC channel a > few day ago. > > GHC depends on quite a set of libraries, and ships those in releases. When ever > a new GHC release is cut, all these dependencies need to be on hackage and > have release versions. We do not want to produce a GHC release which depends > on in-flight packages. In-flight might happen for example due to GHC having to > patch dependencies to make them work with HEAD. > > Everyone who maintains any kind of software online, knows that maintenance can > be a drag, and then life happens, and what not. There are many very responsive > maintainers and we all owe them a grate amount of gratitude towards their > relentless work, keeping those libraries up to date and responding to questions, > patches, ... > > I therefore would like to float the following idea to make the GHC > release processes > a bit more reliable. GHCHQ (that is those in charge of producing GHC > releases for > us all), are made co-maintainers on each library GHC depends on, to guarantee > that GHC can move forward in the worst of circumstances. Now I would > hope that in > almost all cases GHCHQ would never have to maintain any of the dependencies > actively, they deal with GHC already, so let's try to keep it that > way. However GHCHQ > can, after a 14day notification period, exercise the co-maintainance > and cut releases > (and upload them to hackage), should the maintainer not be able to do > so on his own > for various reasons. > > I'd like to see this as an insurance policy for GHC continuous > development. The only > alternative that I see would be that GHCHQ starts forking dependencies > and initiates > the hackage maintainer takeover protocol, which will cause additional > delays, and > incurs an extra burden on the GHC maintainers. > > I hope we can all agree that libraries that end up being dependencies > of GHC should > be held in high regards and form the very foundation GHC is built > upon. As such it > should be an honour to have GHCHQ being a co-maintainer for ones library, as it > signifies that importances of the library for the continuous development of GHC. > > Again I don't expect much to change, except for GHCHQ becoming co-maintainers > for libraries GHC depends on. The baseline expectation will remain as > it is. However we > will have ensured the frictionless development of GHC going forward. > > Cheers, > Moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Mon Jun 1 15:57:14 2020 From: ben at well-typed.com (Ben Gamari) Date: Mon, 01 Jun 2020 11:57:14 -0400 Subject: GHCs dependencies (libraries) and maintenance In-Reply-To: References: Message-ID: <87d06ipyng.fsf@smart-cactus.org> Oleg Grenrus writes: > I want to ask a honest question: Who is GHC HQ? There is a page listing > who is on > > - Haskell.org committee, > - on core libraries committee (CLC), > - GHC steering committee; > - there is a list of named maintainers for core libraries, > - and it is relatively easily to deduce who maintains other tools and > libraries relevant to GHC (e.g. Cabal or say `shake` for hadrian). > > But who are in GHC HQ? Neither google nor gitlab.haskell.org search give > any hints. I frankly have never liked the term GHC HQ for this reason and generally avoid using it. I would guess that in the context of Moritz's proposal it means "the GHC release manager", which is currently me. > I think the core issue here is bad or insufficient communication. > GHC-8.12 beta release is planned to happen in three months. There are > patches still in flight (e.g. for process, Cabal has tracking issue > open). I'm sure that maintainers are able to ack on that. Maybe start by > requiring these explicit acknowledgments? > If this is what it would take I would be happy to perform such one-on-one handshakes. However, I will say that most maintainers (thankfully) are very responsive under the current scheme. There are only a few which have extremely variable communications latencies, despite many attempts at reaching them. I'm skeptical that adding more attempts will somehow change this situation. > I also want to remind about first GHC-8.10 schedule posted on this list > >     October  18 2019:  start of one week freeze in preparation for > branching >     October  25 2019:  ghc-8.10 branch cut >     November 8  2019:  8.10.1-alpha1 >     November 22 2019:  8.10.1-alpha2 >     December 6  2019:  8.10.1-alpha3 >     December 20 2019:  8.10.1-rc1 >     January  10 2020:  Final 8.10.1 release > > I haven't seen *any* updates to that. GHC-8.10.1 was released in end of > March 2020, *three months later*, text issue was urgent in beginning of > February. > Frankly, the text issue was urgent well before February. I made several attempts over weeks to get in touch with Herbert to no avail. It was only in February when it was essentially the *only* record. That being said, I will say that there should have been more communication about the state of the release when the initial release date came and went. 8.10.1-rc1 was cut two weeks after the scheduled release date; at this point I expected to be a week or so away from final. However, it then took months to get releases for a few submodules. All of this should have been clearly communicated. > The plan for GHC-8.12 is to > >  * Mid-June 2020: Aim to have all major features in the tree >  * Late-June 2020: Cut the ghc-8.12 branch >  * June - August 2020: 3 alpha releases >  * 1 September 2020: beta release >  * 25 September 2020: Final 8.12.1 release > > are we still on track? > Yes, we are. This is precisely why I am trying to start this process earlier. I currently have the base version bump underway (!3335) and am working on hammering out some remaining Windows issues so we can have a stable fork point. However, in general the problem is that it is incredibly hard to know whether we are *actually* on track schedule-wise when something as small as a bounds bump may take weeks or months to be merged. > What I'm trying to say, it is that it is hard to believe that issue is > urgent for GHC. It's not the first time (relating to ghc-8.10) when > schedules are not been kept. > I would be happy to send bi-weekly release engineering updates if this would help upstreams better understand where we are in the release process. > And this is one of reasons why I opened an issue about "Annual release > cycle, its structure and long-term-support". > I have had an editor with my draft response open for the last week. Apologies that it has taken so long to finish but things have been busy over the last week due to the imminent fork. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Mon Jun 1 16:33:56 2020 From: ben at well-typed.com (Ben Gamari) Date: Mon, 01 Jun 2020 12:33:56 -0400 Subject: GHCs dependencies (libraries) and maintenance In-Reply-To: References: Message-ID: <87a71mpwyb.fsf@smart-cactus.org> Moritz Angermann writes: > Hi there! > > Again I don't expect much to change, except for GHCHQ becoming > co-maintainers for libraries GHC depends on. The baseline expectation > will remain as it is. However we will have ensured the frictionless > development of GHC going forward. > While I think this would indeed help and wouldn't mind seeing this idea implemented, last time it was floated there was quite some resistance from some maintainers. Really, I think the problems with core library maintenance that GHC feels are bigger than GHC. Afterall, if it takes weeks for a patch from the GHC release manager to make its way into a core library, it will almost certainly take the same time, if not longer, for a patch from a non-GHC contributor to be accepted. We see this in the large number of outstanding merge requests in some of the core library repositories. This isn't a problem that GHC can solve; the bus number for many of our core packages is simply too low. I think the way to fix this is to ensure that core libraries have a sufficiently large pool of maintainers that responsiveness is not a question. Of course, fixing this isn't easy. Our core libraries *need* to maintain a high standard of quality and enforcing that standard requires effort on the part of skilled maintainers. Finding maintainers who have the desire, time, and skill to fill this role is hard. That being said, I do feel that currently outside contributors don't feel welcome enough to even offer to help, largely *because* of the rather minimal maintenance that some core packages see. There is something of a chicken-and-egg problem here. I would also like to reiterate that the above doesn't describe all core libraries. Most of our core library maintainers are impressively responsive and this is something for which I am very grateful. Moreover, while I do believe that the discussion above points to a real problem, I want to make it very clear that all of our core library maintainers deserve recognition for the difficult role that they fill: hand-holding contributors, triaging tickets, reviewing merge requests is hard work and the success of our entire ecosystem rests of their efforts. The fact that Haskell has seen such growth in the past years is in large part the direct result of their work which has made Haskell such a joy to use. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From csaba.hruska at gmail.com Wed Jun 3 17:39:57 2020 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Wed, 3 Jun 2020 19:39:57 +0200 Subject: GHC HEAD windows instability Message-ID: Hello, I built GHC HEAD on windows 10, but some time the build process got stopped due to random crash. But when I restart the build process the error disappears. Is this a known issue? Regards, Csaba *Case A:* *compiler\GHC\Driver\Session.hs:285:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Reader.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Reader.hi: hGetBuf: invalid argument (Invalid argument) |285 | import Control.Monad.Trans.Reader | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^compiler\GHC\Driver\Session.hs:286:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Except.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Except.hi: hGetBuf: invalid argument (Invalid argument) |286 | import Control.Monad.Trans.Except | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^compiler\GHC\Driver\Session.hs:288:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\base-4.14.0.0\Data\Ord.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\base-4.14.0.0\Data\Ord.hi: hGetBuf: invalid argument (Invalid argument) |288 | import Data.Ord | ^^^^^^^^^^^^^^^Error when running Shake build system: at action, called at src\Rules.hs:71:19 in main:Rules at need, called at src\Rules.hs:93:5 in main:Rules* Depends on: _build/stage1/lib/package.conf.d/ghc-8.11.0.20200528.conf at need, called at src\Rules\Register.hs:117:5 in main:Rules.Register* Depends on: _build/stage1/compiler/build/libHSghc-8.11.0.20200528.a at need, called at src\Rules\Library.hs:209:5 in main:Rules.Library* Depends on: _build/stage1/compiler/build/GHC/Driver/Session.o at &%>, called at src\Rules\Compile.hs:77:9 in main:Rules.Compile* Depends on: _build/stage1/compiler/build/GHC/Driver/Session.o _build/stage1/compiler/build/GHC/Driver/Session.hi at cmd', called at src\Builder.hs:291:23 in main:Builder at cmd, called at src\Builder.hs:376:8 in main:Builder* Raised the exception:Development.Shake.cmd, system command failedCommand line: _build/stage0/bin/ghc.exe -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db "-package-db _build/stage1/lib/package.conf.d" "-this-unit-id ghc-8.11.0.20200528" "-package-id Win32-2.6.1.0" "-package-id array-0.5.4.0" "-package-id base-4.14.0.0" "-package-id binary-0.8.7.0" "-package-id bytestring-0.10.9.0" "-package-id containers-0.6.2.1" "-package-id deepseq-1.4.4.0" "-package-id directory-1.3.6.0" "-package-id filepath-1.4.2.1" "-package-id ghc-boot-8.11.0.20200528" "-package-id ghc-boot-th-8.11.0.20200528" "-package-id ghc-heap-8.11.0.20200528" "-package-id ghci-8.11.0.20200528" "-package-id hpc-0.6.1.0" "-package-id integer-gmp-1.0.3.0" "-package-id process-1.6.8.2" "-package-id template-haskell-2.17.0.0" "-package-id time-1.9.3" "-package-id transformers-0.5.6.2" -i -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage1\compiler\build -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage1\compiler\build\autogen -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\compiler -Iincludes -I_build/stage1/lib -I_build/stage1/compiler/build -I_build/stage1/compiler/build/. -I_build/stage1/compiler/build/../rts/dist/build -Icompiler/. -Icompiler/../rts/dist/build -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/process-1.6.8.2/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/time-1.9.3/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/Win32-2.6.1.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/bytestring-0.10.9.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/base-4.14.0.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/integer-gmp-1.0.3.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/rts-1.0/include -I_build/stage1/lib -optc-I_build/stage1/lib -optP-include -optP_build/stage1/compiler/build/autogen/cabal_macros.h -optP-DHAVE_INTERNAL_INTERPRETER -optP-DINTEGER_GMP -outputdir _build/stage1/compiler/build -Wnoncanonical-monad-instances -optc-Wno-error=inline -c compiler/GHC/Driver/Session.hs -o _build/stage1/compiler/build/GHC/Driver/Session.o -O0 -H64m -Wall -Wno-name-shadowing -Wnoncanonical-monad-instances -Wnoncanonical-monoid-instances -this-unit-id ghc -XHaskell2010 -XNoImplicitPrelude -ghcversion-file=C:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/ghcversion.h -optc-DTHREADED_RTS -Wno-deprecated-flags -Wcpp-undefExit code: 1Stderr and Stdout:compiler\GHC\Driver\Session.hs:285:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Reader.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Reader.hi: hGetBuf: invalid argument (Invalid argument) |285 | import Control.Monad.Trans.Reader | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^compiler\GHC\Driver\Session.hs:286:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Except.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\transformers-0.5.6.2\Control\Monad\Trans\Except.hi: hGetBuf: invalid argument (Invalid argument) |286 | import Control.Monad.Trans.Except | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^compiler\GHC\Driver\Session.hs:288:1: error: Bad interface file: _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\base-4.14.0.0\Data\Ord.hi _build/stage1/lib\x86_64-windows-ghc-8.11.0.20200528\base-4.14.0.0\Data\Ord.hi: hGetBuf: invalid argument (Invalid argument) |288 | import Data.Ord | ^^^^^^^^^^^^^^^* *Case B:* *Access violation in generated code when writing 0x0 Attempting to reconstruct a stack trace... Frame Code address * 0x3eadb00 0x3063b36 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x2c63b36 * 0x3eadb08 0x2e12c89 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x2a12c89 * 0x3eadb10 0x4 * 0x3eadb18 0x7c411e1 * 0x3eadb20 0x3665e20 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x3265e20 * 0x3eadb28 0x5c2abd0 * 0x3eadb30 0x9ec8010 * 0x3eadb38 0x3fc3c70 * 0x3eadb40 0x5c00320033006d * 0x3eadb48 0x33004d004d0049 * 0x3eadb50 0x4c0044002e0032 * 0x3eadb58 0x4cError when running Shake build system: at action, called at src\Rules.hs:71:19 in main:Rules at need, called at src\Rules.hs:93:5 in main:Rules* Depends on: _build/stage1/lib/package.conf.d/directory-1.3.6.0.conf at apply1, called at src\Development\Shake\Internal\Rules\Oracle.hs:159:32 in shake-0.18.5-JIltN70Z6uA8zMbpBJImj0:Development.Shake.Internal.Rules.Oracle* Depends on: OracleQ (ContextDataKey (Context {stage = Stage1, package = Package {pkgType = Library, pkgName = "directory", pkgPath = "libraries/directory"}, way = v})) at need, called at src\Hadrian\Oracles\Cabal\Rules.hs:53:9 in main:Hadrian.Oracles.Cabal.Rules* Depends on: _build/stage1/libraries/directory/setup-config at need, called at src\Rules\Library.hs:214:18 in main:Rules.Library* Depends on: _build/stage1/libraries/time/build/HStime-1.9.3.o at need, called at src\Rules\Library.hs:165:5 in main:Rules.Library* Depends on: _build/stage1/libraries/time/build/Data/Time/Format/Format/Class.o at &%>, called at src\Rules\Compile.hs:77:9 in main:Rules.Compile* Depends on: _build/stage1/libraries/time/build/Data/Time/Format/Format/Class.o _build/stage1/libraries/time/build/Data/Time/Format/Format/Class.hi at cmd', called at src\Builder.hs:291:23 in main:Builder at cmd, called at src\Builder.hs:376:8 in main:Builder* Raised the exception:Development.Shake.cmd, system command failedCommand line: _build/stage0/bin/ghc.exe -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db "-package-db _build/stage1/lib/package.conf.d" "-this-unit-id time-1.9.3" "-package-id Win32-2.6.1.0" "-package-id base-4.14.0.0" "-package-id deepseq-1.4.4.0" -i -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage1\libraries\time\build -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage1\libraries\time\build\autogen -iC:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\libraries\time\lib -Iincludes -I_build/stage1/lib -I_build/stage1/libraries/time/build -I_build/stage1/libraries/time/build/lib/include -Ilibraries/time/lib/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/Win32-2.6.1.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/bytestring-0.10.9.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/base-4.14.0.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/integer-gmp-1.0.3.0/include -IC:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/x86_64-windows-ghc-8.11.0.20200528/rts-1.0/include -I_build/stage1/lib -optc-I_build/stage1/lib -optP-include -optP_build/stage1/libraries/time/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/time/build -Wnoncanonical-monad-instances -optc-Wno-error=inline -c libraries/time/lib/Data/Time/Format/Format/Class.hs -o _build/stage1/libraries/time/build/Data/Time/Format/Format/Class.o -O0 -H64m -Wall -fwarn-tabs -XHaskell2010 -XRank2Types -XDeriveDataTypeable -XStandaloneDeriving -XCPP -ghcversion-file=C:/Users/IEUser/haskell/ghc-whole-program-compiler-project/ghc-wpc/_build/stage1/lib/ghcversion.h -Wno-deprecated-flagsExit code: 11Stderr and Stdout:Access violation in generated code when writing 0x0 Attempting to reconstruct a stack trace... Frame Code address * 0x3eadb00 0x3063b36 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x2c63b36 * 0x3eadb08 0x2e12c89 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x2a12c89 * 0x3eadb10 0x4 * 0x3eadb18 0x7c411e1 * 0x3eadb20 0x3665e20 C:\Users\IEUser\haskell\ghc-whole-program-compiler-project\ghc-wpc\_build\stage0\bin\ghc.exe+0x3265e20 * 0x3eadb28 0x5c2abd0 * 0x3eadb30 0x9ec8010 * 0x3eadb38 0x3fc3c70 * 0x3eadb40 0x5c00320033006d * 0x3eadb48 0x33004d004d0049 * 0x3eadb50 0x4c0044002e0032 * 0x3eadb58 0x4c* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Wed Jun 3 17:57:24 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 03 Jun 2020 13:57:24 -0400 Subject: GHC HEAD windows instability In-Reply-To: References: Message-ID: <87ftbcnibl.fsf@smart-cactus.org> Csaba Hruska writes: > Hello, > I built GHC HEAD on windows 10, but some time the build process got stopped > due to random crash. But when I restart the build process the error > disappears. Is this a known issue? > Which compiler are you bootstrapping with? The crash is unfortunately quite generic so it's hard to say whether it resembles any of the other open Windows issues. Do open a ticket with the logs. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Thu Jun 4 15:22:41 2020 From: ben at well-typed.com (Ben Gamari) Date: Thu, 04 Jun 2020 11:22:41 -0400 Subject: Change in ~"backport needed" label convention Message-ID: <871rmuonyc.fsf@smart-cactus.org> Hi everyone, Recently with the increased rate of GHC releases we have had to handle more concurrent stable branches. In particular, currently we have GHC-8.8.4 and GHC-8.10.2 releases both in flight. It has become clear that our current scheme for tracking backports, the ~"baackport needed" GitLab label, doesn't accomodate multiple concurrent stable branches particularly well. Consequently, I have created a new set of labels which are specific to each release series. For instance, ~"backport needed:8.10" means that the MR needs to be backported to the ghc-8.10 branch. The old ~"backport needed" label has been removed. Please do use these new labels on merge requests to indicate that backporting is needed. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ryan.gl.scott at gmail.com Mon Jun 8 10:24:41 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Mon, 8 Jun 2020 06:24:41 -0400 Subject: Marge Bot repeatedly failing due to stat decreases Message-ID: Hello, I've been receiving a flurry of e-mails all weekend about Marge Bot failing to validate a batch on CI. Moreover, the failures appear to be consistent each time. From the most recent one [1]: Unexpected stat failures: /builds/ghc/ghc/tmp/ghctest-mel1y5lo/test spaces/testsuite/tests/perf/compiler/T12150.run T12150 [stat decreased from "x86_64-linux-deb9-hadrian" baseline @ HEAD~6] (optasm) /builds/ghc/ghc/tmp/ghctest-mel1y5lo/test spaces/testsuite/tests/perf/compiler/T12234.run T12234 [stat decreased from "x86_64-linux-deb9-hadrian" baseline @ HEAD~6] (optasm) Where T12150(optasm) compile_time/bytes allocated 75982592.000 (baseline @ HEAD~6) 76945688.000 [decreased, -1.3%] T12234(optasm) compile_time/bytes allocated 82149240.000 (baseline @ HEAD~6) 83109896.000 [decreased, -1.2%] Can something be done about this? It's holding one one of my patches. Ryan S. ----- [1] https://gitlab.haskell.org/ghc/ghc/-/jobs/359994 From ben at smart-cactus.org Mon Jun 8 13:37:00 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 08 Jun 2020 09:37:00 -0400 Subject: Marge Bot repeatedly failing due to stat decreases In-Reply-To: References: Message-ID: <873675n0g7.fsf@smart-cactus.org> Ryan Scott writes: > Hello, > > I've been receiving a flurry of e-mails all weekend about Marge Bot > failing to validate a batch on CI. Moreover, the failures appear to be > consistent each time. From the most recent one [1]: > Yesterday I had amended one of Marge's batches to accept the performance changes and marked it for auto-merging. Unfortunately Marge got to it before CI finished it and closed it. I just manually merged it and restarted Marge. Sorry for the delay! Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ryan.gl.scott at gmail.com Mon Jun 8 13:55:59 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Mon, 8 Jun 2020 09:55:59 -0400 Subject: Marge Bot repeatedly failing due to stat decreases In-Reply-To: <873675n0g7.fsf@smart-cactus.org> References: <873675n0g7.fsf@smart-cactus.org> Message-ID: Something strange is still going on, since Marge now just opened an empty merge batch [1]. Ryan S. ----- [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3465 On Mon, Jun 8, 2020 at 9:37 AM Ben Gamari wrote: > > Ryan Scott writes: > > > Hello, > > > > I've been receiving a flurry of e-mails all weekend about Marge Bot > > failing to validate a batch on CI. Moreover, the failures appear to be > > consistent each time. From the most recent one [1]: > > > Yesterday I had amended one of Marge's batches to accept the performance > changes and marked it for auto-merging. Unfortunately Marge got to it > before CI finished it and closed it. > > I just manually merged it and restarted Marge. Sorry for the delay! > > Cheers, > > - Ben From ben at smart-cactus.org Mon Jun 8 14:48:50 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 08 Jun 2020 10:48:50 -0400 Subject: Marge Bot repeatedly failing due to stat decreases In-Reply-To: References: <873675n0g7.fsf@smart-cactus.org> Message-ID: <87zh9dlik1.fsf@smart-cactus.org> Ryan Scott writes: > Something strange is still going on, since Marge now just opened an > empty merge batch [1]. > Indeed; it seems that she was confused by the fact that the merged MRs weren't yet closed. Fixed and restarted. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ryan.gl.scott at gmail.com Mon Jun 8 15:27:29 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Mon, 8 Jun 2020 11:27:29 -0400 Subject: Marge Bot repeatedly failing due to stat decreases In-Reply-To: <87zh9dlik1.fsf@smart-cactus.org> References: <873675n0g7.fsf@smart-cactus.org> <87zh9dlik1.fsf@smart-cactus.org> Message-ID: Thanks Ben! For future reference, is there anything that we as ordinary GHC developers can do to help Marge along when she gets stuck on stat changes? You mentioned that you amended a Marge batch to accept the changes before. Are there instructions on how to do this properly? For instance, which commit(s) in a batch should be amended? Is it better to amend the branch that Marge works on directly, or to update the original MR? Ryan S. From ben at well-typed.com Wed Jun 10 21:13:51 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 10 Jun 2020 17:13:51 -0400 Subject: Merge window logistics and relaxation of submodule policy Message-ID: <87sgf2k4j7.fsf@smart-cactus.org> Hi everyone, At this point we are in the home stretch of the GHC 8.12 preparation phase. The merge queue has progressed a bit more slowly than I would have liked over the last few weeks but nevertheless I think we are converging. Regardless, if you have a patch which you believe should go in to 8.12 and doesn't already bear the %8.12 GitLab milestone, please do let me know. # Submodule policy One matter that is currently holding us up is a few submodules which haven't yet had necessary patches merged upstream. As there hasn't yet been much motion on the part of maintainer, I propose that we temporarily relax our policy that GHC submodules refer to upstream commits. That is, we allow GHC to temporarily track a non-upstream branch, ensuring that the branch is merged upstream before the final GHC release. Are there any objections to this policy relaxation? It will require a bit more care on our part, but I think it is a sensible compromise way to give upstream maintainers the time they need while ensuring that GHC isn't subject to unpredictable schedules. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Thu Jun 11 11:19:03 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 11 Jun 2020 11:19:03 +0000 Subject: Evac.c Message-ID: I'm getting this in HEAD. Should I worry? It says "error"! Simon rts/sm/Evac.c:463:5: error: note: called from here ACQUIRE_SPIN_LOCK(&gen->sync); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | 463 | ACQUIRE_SPIN_LOCK(&gen->sync); | ^ In file included from includes/Rts.h:196:0: error: 0, from rts/sm/Evac.c:15: includes/rts/SpinLock.h:43:20: error: warning: inlining failed in call to 'ACQUIRE_SPIN_LOCK': call is unlikely and code size would grow [-Winline] INLINE_HEADER void ACQUIRE_SPIN_LOCK(SpinLock * p) ^~~~~~~~~~~~~~~~~ | 43 | INLINE_HEADER void ACQUIRE_SPIN_LOCK(SpinLock * p) | ^ rts/sm/Evac.c:515:31: error: note: called from here if (new_gen != gen) { ACQUIRE_SPIN_LOCK(&new_gen->sync); } ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | 515 | if (new_gen != gen) { ACQUIRE_SPIN_LOCK(&new_gen->sync); } | ^ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu Jun 11 16:49:41 2020 From: ben at well-typed.com (Ben Gamari) Date: Thu, 11 Jun 2020 12:49:41 -0400 Subject: Evac.c In-Reply-To: References: Message-ID: <87imfxk0ny.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > I'm getting this in HEAD. Should I worry? It says "error"! > Simon > Indeed we probably ought to fix the error message handling here. The claim that this is an "error" is from GHC, which parses GCC's error output. You see, however, that GCC is merely giving us a warning. Regardless, the issue itself is non-critical. It's merely telling us that we asked for a function to be inlined yet GCC felt it wouldn't be wise to do so. This usually happens in the debug runtime due to additional assertion code. It's safe to ignore. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From lonetiger at gmail.com Thu Jun 11 16:53:04 2020 From: lonetiger at gmail.com (Phyx) Date: Thu, 11 Jun 2020 17:53:04 +0100 Subject: Evac.c In-Reply-To: <87imfxk0ny.fsf@smart-cactus.org> References: <87imfxk0ny.fsf@smart-cactus.org> Message-ID: Hi Ben, Just FYI, gcc 9 added -fdiagnostics-format=json so that may be an option to make this sort of thing easier to handle. Kind regards Tamar Sent from my Mobile On Thu, Jun 11, 2020, 17:50 Ben Gamari wrote: > Simon Peyton Jones via ghc-devs writes: > > > I'm getting this in HEAD. Should I worry? It says "error"! > > Simon > > > Indeed we probably ought to fix the error message handling here. The > claim that this is an "error" is from GHC, which parses GCC's error > output. You see, however, that GCC is merely giving us a warning. > > Regardless, the issue itself is non-critical. It's merely telling us > that we asked for a function to be inlined yet GCC felt it wouldn't be > wise to do so. This usually happens in the debug runtime due to > additional assertion code. It's safe to ignore. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From depsterr at protonmail.com Fri Jun 12 15:59:38 2020 From: depsterr at protonmail.com (depsterr) Date: Fri, 12 Jun 2020 15:59:38 +0000 Subject: libgmp dependency on alpine linux binary release Message-ID: I apologize if this is the wrong mailing list, however I couldn't find another one that seemed fit. On the official binary release page (https://www.haskell.org/ghc/download_ghc_8_10_1.html) under the x86_64 alpine release it states the following: > Unlike our other binary distributions, this links against the`integer-simple`big-integer backend and therefore does not require`libgmp`. However, ldd shows that it DOES link to gmp: libgmp.so.10 => /lib/libgmp.so.10 (0x7f47f4c94000) Additionally, if gmp is not installed then a number of errors appear: Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_tdiv_qr: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_gcd_1: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_add_1: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_andn_n: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpz_gcdext: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_lshift: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_rshift: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_and_n: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpz_powm: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_mod_1: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_cmp: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_add: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_sub_1: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpn_xor_n: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpz_clear: symbol not found Error relocating /usr/bin/../lib/x86_64-linux-ghc-8.10.1/libHSinteger-gmp-1.0.3.0-ghc8.10.1.so: __gmpz_invert: symbol not found Either this build isn't configured properly or the text on the release page needs to be edited to reflect this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From csaba.hruska at gmail.com Fri Jun 12 21:16:44 2020 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Fri, 12 Jun 2020 23:16:44 +0200 Subject: Introducing GHC whole program compiler (GHC-WPC) Message-ID: Hello, I've created a whole program compilation pipeline for GHC via STG. Please read my blog post for the details: Introducing GHC whole program compiler (GHC-WPC) Regards, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From lexi.lambda at gmail.com Sat Jun 13 20:26:29 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Sat, 13 Jun 2020 15:26:29 -0500 Subject: Introducing GHC whole program compiler (GHC-WPC) In-Reply-To: References: Message-ID: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> Hi Csaba, I originally posted this comment on /r/haskell before I saw you also sent this to ghc-devs. I’ve decided to reproduce my comment here as well, since this list probably has a more relevant audience: > I want to start by saying that I think this sounds totally awesome, and I think it’s a fantastic idea. I’m really interested in seeing how this progresses! > > I do wonder if people might find the name a little misleading. “Whole program compilation” usually implies “whole program optimization,” but most of GHC’s key optimizations happen at the Core level, before STG is even generated. (Of course, I’m sure you’re well aware of that, I’m just stating it for the sake of others who might be reading who aren’t aware.) > > This seems much closer in spirit to “link-time optimization” (LTO) as performed by Clang and GCC than whole program compilation. For example, Clang’s LTO works by “linking” LLVM bitcode files instead of fully-compiled native objects. STG is not quite analogous to LLVM IR—GHC’s analog would be Cmm, not STG—but I think that difference is not that significant here: the STG-to-Cmm pass is quite mechanical, and STG is mostly just easier to manipulate. > > tl;dr: Have you considered naming this project GHC-LTO instead of GHC-WPC? Alexis > On Jun 12, 2020, at 16:16, Csaba Hruska wrote: > > Hello, > > I've created a whole program compilation pipeline for GHC via STG. > Please read my blog post for the details: > Introducing GHC whole program compiler (GHC-WPC) > > Regards, > Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From lazybonesxp at gmail.com Sun Jun 14 11:28:07 2020 From: lazybonesxp at gmail.com (Rinat Stryungis) Date: Sun, 14 Jun 2020 14:28:07 +0300 Subject: Receiving type information from environment instead of hardcoding. Message-ID: Hi. I have a question about a possible way of unification of Nat and Natural. I've almost done that, but only in case of using integer-gmp. If I use integer-simple there is a completely different definition of Natural. How I construct now naturalTyCon (to make `naturalTy` to use it instead of `typeNatKind`) : ```naturalTyCon :: TyCon naturalTyCon = pcTyCon naturalTyConName Nothing [] [natSDataCon,natJDataCon] natSDataCon :: DataCon natSDataCon = pcDataCon natSDataConName [] [wordPrimTy] naturalTyCon etc... ``` Now I have to check`DynFlags` in a few places to reimplement `naturalTyCon` in case of using `integer-simple`. Is there a way to avoid hardcoding of `naturalTy`? My colleague said that it would be nice to get `naturalTy` from an environment by something like `lookupTyCon`, but there are many functions whose don't use any environment like functions from `typeNatTyCons` list in `GHC.Builtin.Types.Literals`. Now I am going to use `DynFlags` checking, but it looks like an ugly way... -- Best regards. Rinat Striungis -------------- next part -------------- An HTML attachment was scrubbed... URL: From csaba.hruska at gmail.com Sun Jun 14 12:45:51 2020 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Sun, 14 Jun 2020 14:45:51 +0200 Subject: Introducing GHC whole program compiler (GHC-WPC) In-Reply-To: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> References: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> Message-ID: Hi, I thought about the GHC-LTO project name before, but it would not be an accurate description though. The GHC-WPC in its current state is about exporting STG + linker info for later processing, either feed it back to GHC backend or to a third party pipeline. It depends what the user/researcher wants, the point is that GHC-WPC solves the IR export part of the issue. It is the external stg compiler that implements a (simple) whole program dead function elimination pass that I implemented as a proof of concept to show the new possibilities GHC-WPC opens up. But I plan to do much more optimization with sophisticated dataflow analyses. I.e. I have a fast and working implementation of control flow analysis in souffle/datalog that I plan to use to do more accurate dead code elimination and partial program defunctionalization on the whole program STG IR. In theory I could implement all GRIN optimizations on STG. That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core. I'd like to go even further with experimentation. I can imagine a dependently typed Cmm with a similar type system that ATS ( http://www.ats-lang.org/MYDATA/VsTsVTs-2018-10-28.pdf) has. I definitely would like to make an experiment in the future, to come up with an Idirs2 EDSL for GHC RTS heap operations where the type system would ensure the correctness of pointer arithmetic and heap object manipulation. The purpose of GHC-WPC in this story is to deliver the IR for these stuff. Beside exporting STG IR, the external STG compiler can compile STG via GHC's standard code generator. This makes GHC codegen/RTS available as a backend for programming language developers. I.e. Idris, Agda, Purescript could use GHC/STG/RTS as a backend with all of its cool features. So these are the key parts of my vision about the purpose and development of GHC-WPC. It is meant to be more than a link time optimizer. Cheers, Csaba On Sat, Jun 13, 2020 at 10:26 PM Alexis King wrote: > Hi Csaba, > > I originally posted this comment on /r/haskell > before > I saw you also sent this to ghc-devs. I’ve decided to reproduce my comment > here as well, since this list probably has a more relevant audience: > > I want to start by saying that I think this sounds totally awesome, and I > think it’s a fantastic idea. I’m really interested in seeing how this > progresses! > > I do wonder if people might find the name a little misleading. “Whole > program compilation” usually implies “whole program optimization,” but most > of GHC’s key optimizations happen at the Core level, before STG is even > generated. (Of course, I’m sure you’re well aware of that, I’m just stating > it for the sake of others who might be reading who aren’t aware.) > > This seems much closer in spirit to “link-time optimization” (LTO) as > performed by Clang and GCC than whole program compilation. For example, > Clang’s LTO works by “linking” LLVM bitcode files instead of fully-compiled > native objects. STG is not quite analogous to LLVM IR—GHC’s analog would be > Cmm, not STG—but I think that difference is not that significant here: the > STG-to-Cmm pass is quite mechanical, and STG is mostly just easier to > manipulate. > > tl;dr: Have you considered naming this project GHC-LTO instead of GHC-WPC? > > > Alexis > > On Jun 12, 2020, at 16:16, Csaba Hruska wrote: > > Hello, > > I've created a whole program compilation pipeline for GHC via STG. > Please read my blog post for the details: > Introducing GHC whole program compiler (GHC-WPC) > > > Regards, > Csaba Hruska > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Sun Jun 14 13:43:17 2020 From: cdsmith at gmail.com (Chris Smith) Date: Sun, 14 Jun 2020 09:43:17 -0400 Subject: Looking for a recent error reporting change Message-ID: Hi. In GHC 8.6.5, if I compile a main module that both fails to define `main` and also contains other errors, the other errors are not reported because GHC aborts after realizing that there is no `main`. In GHC 8.10.1, all errors are reported. I'm looking for where this change was made, in the hopes that it's easy to backport to a local branch of GHC 8.6.5. Does anyone know off-hand? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsx at bluewin.ch Sun Jun 14 15:00:22 2020 From: rsx at bluewin.ch (Roland Senn) Date: Sun, 14 Jun 2020 17:00:22 +0200 Subject: Looking for a recent error reporting change In-Reply-To: References: Message-ID: <1592146822.18121.1.camel@bluewin.ch> Hi Chris, Maybe you want to look at MR 1806 (https://gitlab.haskell.org/ghc/ghc/- /merge_requests/1806). There I touched the `check is main exported` stuff. There is also MR !1885 for GHC 8.6.2 (https://gitlab.haskell.org /ghc/ghc/-/merge_requests/1885). However this got never merged! Regards  Roland Am Sonntag, den 14.06.2020, 09:43 -0400 schrieb Chris Smith: > Hi.  In GHC 8.6.5, if I compile a main module that both fails to > define `main` and also contains other errors, the other errors are > not reported because GHC aborts after realizing that there is no > `main`.  In GHC 8.10.1, all errors are reported.  I'm looking for > where this change was made, in the hopes that it's easy to backport > to a local branch of GHC 8.6.5.  Does anyone know off-hand? > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Sun Jun 14 18:06:45 2020 From: ben at well-typed.com (Ben Gamari) Date: Sun, 14 Jun 2020 14:06:45 -0400 Subject: Looking for a recent error reporting change In-Reply-To: <1592146822.18121.1.camel@bluewin.ch> References: <1592146822.18121.1.camel@bluewin.ch> Message-ID: <874krdbjyl.fsf@smart-cactus.org> Roland Senn writes: > Hi Chris, > Maybe you want to look at MR 1806 (https://gitlab.haskell.org/ghc/ghc/- > /merge_requests/1806). There I touched the `check is main exported` > stuff. There is also MR !1885 for GHC 8.6.2 (https://gitlab.haskell.org > /ghc/ghc/-/merge_requests/1885). However this got never merged! > Regards  Roland > Hmm, I'm a bit confused. It looks to me like !1885 was for 8.8 and was merged. Do you mean that we might *also* want to merge for 8.6? FWIW I doubt there will be another release on the 8.6 series. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Sun Jun 14 21:32:35 2020 From: ben at well-typed.com (Ben Gamari) Date: Sun, 14 Jun 2020 17:32:35 -0400 Subject: Receiving type information from environment instead of hardcoding. In-Reply-To: References: Message-ID: <871rmhbafo.fsf@smart-cactus.org> Rinat Stryungis writes: > Hi. I have a question about a possible way of unification of Nat and > Natural. I've almost done that, but only in case of using integer-gmp. > If I use integer-simple there is a completely different definition of > Natural. > > How I construct now naturalTyCon (to make `naturalTy` to use it instead of > `typeNatKind`) : > > ```naturalTyCon :: TyCon > naturalTyCon = pcTyCon naturalTyConName Nothing [] [natSDataCon,natJDataCon] > > natSDataCon :: DataCon > natSDataCon = pcDataCon natSDataConName [] [wordPrimTy] naturalTyCon > > etc... > ``` > Now I have to check`DynFlags` in a few places to reimplement `naturalTyCon` > in case of using `integer-simple`. > > Is there a way to avoid hardcoding of `naturalTy`? > My colleague said that it would be nice to get `naturalTy` from an > environment by something like `lookupTyCon`, > but there are many functions whose don't use any environment like functions > from `typeNatTyCons` list in `GHC.Builtin.Types.Literals`. > > Now I am going to use `DynFlags` checking, but it looks like an ugly way... Note that all of this will be moot in a matter of days. The ghc-bignum patch, which will ship in 8.12, removes integer-simple and uses a consistent number representation across its various supported backends. In light of this, if I were you I would probably just settle for a hack in the meantime. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Mon Jun 15 09:11:23 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Jun 2020 09:11:23 +0000 Subject: Looking for a recent error reporting change In-Reply-To: References: Message-ID: Is there a reason you want to stick with 8.6.5? One way to find what you want is to use git-bisect to find the commit that made the change. I think various people have automated setups for this. Simon From: ghc-devs On Behalf Of Chris Smith Sent: 14 June 2020 14:43 To: ghc-devs at haskell.org Subject: Looking for a recent error reporting change Hi. In GHC 8.6.5, if I compile a main module that both fails to define `main` and also contains other errors, the other errors are not reported because GHC aborts after realizing that there is no `main`. In GHC 8.10.1, all errors are reported. I'm looking for where this change was made, in the hopes that it's easy to backport to a local branch of GHC 8.6.5. Does anyone know off-hand? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lazybonesxp at gmail.com Mon Jun 15 09:28:38 2020 From: lazybonesxp at gmail.com (Rinat Stryungis) Date: Mon, 15 Jun 2020 12:28:38 +0300 Subject: Receiving type information from environment instead of hardcoding. In-Reply-To: <871rmhbafo.fsf@smart-cactus.org> References: <871rmhbafo.fsf@smart-cactus.org> Message-ID: In light of the mentioned patch, I prefer to freeze my activity about the unification of Nat and Natural up to the merging this patch. After that, I am going to rebase my branch and make MR. Thank you, Ben! пн, 15 июн. 2020 г. в 00:32, Ben Gamari : > Rinat Stryungis writes: > > > Hi. I have a question about a possible way of unification of Nat and > > Natural. I've almost done that, but only in case of using integer-gmp. > > If I use integer-simple there is a completely different definition of > > Natural. > > > > How I construct now naturalTyCon (to make `naturalTy` to use it instead > of > > `typeNatKind`) : > > > > ```naturalTyCon :: TyCon > > naturalTyCon = pcTyCon naturalTyConName Nothing [] > [natSDataCon,natJDataCon] > > > > natSDataCon :: DataCon > > natSDataCon = pcDataCon natSDataConName [] [wordPrimTy] naturalTyCon > > > > etc... > > ``` > > Now I have to check`DynFlags` in a few places to reimplement > `naturalTyCon` > > in case of using `integer-simple`. > > > > Is there a way to avoid hardcoding of `naturalTy`? > > My colleague said that it would be nice to get `naturalTy` from an > > environment by something like `lookupTyCon`, > > but there are many functions whose don't use any environment like > functions > > from `typeNatTyCons` list in `GHC.Builtin.Types.Literals`. > > > > Now I am going to use `DynFlags` checking, but it looks like an ugly > way... > > Note that all of this will be moot in a matter of days. The ghc-bignum > patch, which will ship in 8.12, removes integer-simple and uses a > consistent number representation across its various supported backends. > > In light of this, if I were you I would probably just settle for a hack > in the meantime. > > Cheers, > > - Ben > > -- Best regards. Rinat Striungis -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jun 15 09:34:17 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Jun 2020 09:34:17 +0000 Subject: Introducing GHC whole program compiler (GHC-WPC) In-Reply-To: References: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> Message-ID: I’ve always thought that whole-program compilation has the possibility of doing optimisations that are simply inaccessible without the whole program, but been daunted by the engineering challenges of making WPC actually work. So it’s fantastic that you’ve made progress on this. Well done! Questions that come to mind (to be understood in the context of the above enthusiasm): * If you compile a program that depends on (say) lens, you get a lot of code. Dead-code elim will drop lots, perhaps, but you start with everything. So what do memory footprints and compile times look like when you do WPC? Remembering that people often complain about GHC’s footprint when compiling a single module. Also, WPC means that instead of just linking to precompiled libraries, you have to recompile (parts of) them. What does that do to compile times? * I love the 25% reduction in binary size. In fact I’m surprised it isn’t bigger. * Why are you using STG? It’s mostly untyped – or at least much less strongly typed than Core. It has lots of restrictions (like ANF) that Core does not. Indeed I think of STG as a little lily-pad to alight on in the hop from Core to Cmm. Maybe your entire setup would work equally well with Core, provided you can serialise and deserialise it. * Moreover, we *already* have a fast serialiser and deserialiser for Core – the stuff we use for interface files. So maybe you could re-use that … no need for pretty-print and parse. * You say “That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core.” Fair enough, but what I’m missing is the rationale for doing heavy opts on STG rather than Core. * Apart from (a) dead code, and (b) GRIN, do you have ideas in mind for what we could do with WPC? Thanks Simon From: ghc-devs On Behalf Of Csaba Hruska Sent: 14 June 2020 13:46 To: Alexis King Cc: GHC developers Subject: Re: Introducing GHC whole program compiler (GHC-WPC) Hi, I thought about the GHC-LTO project name before, but it would not be an accurate description though. The GHC-WPC in its current state is about exporting STG + linker info for later processing, either feed it back to GHC backend or to a third party pipeline. It depends what the user/researcher wants, the point is that GHC-WPC solves the IR export part of the issue. It is the external stg compiler that implements a (simple) whole program dead function elimination pass that I implemented as a proof of concept to show the new possibilities GHC-WPC opens up. But I plan to do much more optimization with sophisticated dataflow analyses. I.e. I have a fast and working implementation of control flow analysis in souffle/datalog that I plan to use to do more accurate dead code elimination and partial program defunctionalization on the whole program STG IR. In theory I could implement all GRIN optimizations on STG. That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core. I'd like to go even further with experimentation. I can imagine a dependently typed Cmm with a similar type system that ATS (http://www.ats-lang.org/MYDATA/VsTsVTs-2018-10-28.pdf) has. I definitely would like to make an experiment in the future, to come up with an Idirs2 EDSL for GHC RTS heap operations where the type system would ensure the correctness of pointer arithmetic and heap object manipulation. The purpose of GHC-WPC in this story is to deliver the IR for these stuff. Beside exporting STG IR, the external STG compiler can compile STG via GHC's standard code generator. This makes GHC codegen/RTS available as a backend for programming language developers. I.e. Idris, Agda, Purescript could use GHC/STG/RTS as a backend with all of its cool features. So these are the key parts of my vision about the purpose and development of GHC-WPC. It is meant to be more than a link time optimizer. Cheers, Csaba On Sat, Jun 13, 2020 at 10:26 PM Alexis King > wrote: Hi Csaba, I originally posted this comment on /r/haskell before I saw you also sent this to ghc-devs. I’ve decided to reproduce my comment here as well, since this list probably has a more relevant audience: I want to start by saying that I think this sounds totally awesome, and I think it’s a fantastic idea. I’m really interested in seeing how this progresses! I do wonder if people might find the name a little misleading. “Whole program compilation” usually implies “whole program optimization,” but most of GHC’s key optimizations happen at the Core level, before STG is even generated. (Of course, I’m sure you’re well aware of that, I’m just stating it for the sake of others who might be reading who aren’t aware.) This seems much closer in spirit to “link-time optimization” (LTO) as performed by Clang and GCC than whole program compilation. For example, Clang’s LTO works by “linking” LLVM bitcode files instead of fully-compiled native objects. STG is not quite analogous to LLVM IR—GHC’s analog would be Cmm, not STG—but I think that difference is not that significant here: the STG-to-Cmm pass is quite mechanical, and STG is mostly just easier to manipulate. tl;dr: Have you considered naming this project GHC-LTO instead of GHC-WPC? Alexis On Jun 12, 2020, at 16:16, Csaba Hruska > wrote: Hello, I've created a whole program compilation pipeline for GHC via STG. Please read my blog post for the details: Introducing GHC whole program compiler (GHC-WPC) Regards, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jun 15 09:48:59 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Jun 2020 09:48:59 +0000 Subject: Search in GitLab Message-ID: Does anyone know how to search better in GitLab. Currently I'm using the standard GitLab search. I'm searching for "<>" where I intend the quotes meaning exactly that string as usual in a search term. But I get lots of results mentioning loop, without the angle brackets. Moreover I want to sort the results by date or ticket number, and I can't see how to do that. Does Google index our repo? Can I use Google to search it somehow? Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at haskus.fr Mon Jun 15 09:55:49 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Mon, 15 Jun 2020 11:55:49 +0200 Subject: Search in GitLab In-Reply-To: References: Message-ID: <15b65057-90d1-8f4f-e3b0-64c525e6a885@haskus.fr> > Does Google index our repo?  Can I use Google to search it somehow? In google you can type: site:gitlab.haskell.org "<>" Cheers, Sylvain On 15/06/2020 11:48, Simon Peyton Jones via ghc-devs wrote: > > Does anyone know how to search better in GitLab. > > Currently I’m using the standard GitLab search.  I’m searching for > > “<>” > > where I intend the quotes meaning exactly that string as usual in a > search term.  But I get lots of results mentioning loop, without the > angle brackets. > > Moreover I want to sort the results by date or ticket number, and I > can’t see how to do that. > > Does Google index our repo?  Can I use Google to search it somehow? > > Thanks > > Simon > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Mon Jun 15 10:23:35 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Mon, 15 Jun 2020 11:23:35 +0100 Subject: Search in GitLab In-Reply-To: <15b65057-90d1-8f4f-e3b0-64c525e6a885@haskus.fr> References: <15b65057-90d1-8f4f-e3b0-64c525e6a885@haskus.fr> Message-ID: I've used the site:gitlab.haskell.org trick on Google for some time. Recently (last month or so?) I've found that it's much less useful. I can't quantify this, other than the fact that I find myself frequently displeased with the result. It does seem that Google indexes our site less frequently than it used to, but again, I can't provide data. So I'm also unhappy with the current ability to search. There's not a way to request that Google index our site more often, is there? Richard > On Jun 15, 2020, at 10:55 AM, Sylvain Henry wrote: > > > Does Google index our repo? Can I use Google to search it somehow? > > In google you can type: > > site:gitlab.haskell.org "<>" > Cheers, > Sylvain > > > > On 15/06/2020 11:48, Simon Peyton Jones via ghc-devs wrote: >> Does anyone know how to search better in GitLab. >> >> Currently I’m using the standard GitLab search. I’m searching for >> >> “<>” >> >> where I intend the quotes meaning exactly that string as usual in a search term. But I get lots of results mentioning loop, without the angle brackets. >> >> Moreover I want to sort the results by date or ticket number, and I can’t see how to do that. >> >> Does Google index our repo? Can I use Google to search it somehow? >> >> Thanks >> >> Simon >> >> >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Mon Jun 15 13:33:13 2020 From: cdsmith at gmail.com (Chris Smith) Date: Mon, 15 Jun 2020 09:33:13 -0400 Subject: Looking for a recent error reporting change In-Reply-To: References: Message-ID: My reason for sticking with 8.6.5 is that I don't yet trust GHCJS with later GHC versions. Thanks everyone for the pointers and suggestions. I'll look further into this when I have more time. (Alas, it's not my full-time job, so that's probably not until the weekend.) On Mon, Jun 15, 2020 at 5:11 AM Simon Peyton Jones wrote: > Is there a reason you want to stick with 8.6.5? > > > > One way to find what you want is to use git-bisect to find the commit that > made the change. I think various people have automated setups for this. > > > > Simon > > > > *From:* ghc-devs *On Behalf Of *Chris Smith > *Sent:* 14 June 2020 14:43 > *To:* ghc-devs at haskell.org > *Subject:* Looking for a recent error reporting change > > > > Hi. In GHC 8.6.5, if I compile a main module that both fails to define > `main` and also contains other errors, the other errors are not reported > because GHC aborts after realizing that there is no `main`. In GHC 8.10.1, > all errors are reported. I'm looking for where this change was made, in > the hopes that it's easy to backport to a local branch of GHC 8.6.5. Does > anyone know off-hand? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Jun 15 17:50:19 2020 From: ben at well-typed.com (Ben Gamari) Date: Mon, 15 Jun 2020 13:50:19 -0400 Subject: Search in GitLab In-Reply-To: References: Message-ID: <87wo489q20.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Does anyone know how to search better in GitLab. > Currently I'm using the standard GitLab search. I'm searching for > "<>" > where I intend the quotes meaning exactly that string as usual in a search term. But I get lots of results mentioning loop, without the angle brackets. > Moreover I want to sort the results by date or ticket number, and I can't see how to do that. > Does Google index our repo? Can I use Google to search it somehow? Indeed this is a hard query. It appears that the tokenizer used by GitLab (or perhaps Elasticsearch, which is responsible for full-text indexing) considers the angle-brackets to be token delimiters (which I suppose is fair given that usually you see them used to signify less-than/greater-than operators). Google seems to be slightly better, returning only 10 hits. If this isn't sufficient and the query is important I could run a query against the database directly if you would like. Frankly, this makes me wonder whether we should change the output produced for loops. The current error is essentially un-Googleable, as we see here. I know I have personally struggled with this same issue in the past. Richard, I checked on Google Search Console and it appears to be indexed on a nearly-daily basis. I'm a bit surprised that you have been having trouble. Do you have a specific example of a query that you have been disappointed by in the past? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Mon Jun 15 21:04:46 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Mon, 15 Jun 2020 22:04:46 +0100 Subject: Search in GitLab In-Reply-To: <87wo489q20.fsf@smart-cactus.org> References: <87wo489q20.fsf@smart-cactus.org> Message-ID: <4F91CA01-EAE7-4B24-A779-6FCC1A35837B@richarde.dev> > On Jun 15, 2020, at 6:50 PM, Ben Gamari wrote: > > Richard, I checked on Google Search Console and it appears to be > indexed on a nearly-daily basis. I'm a bit surprised that you have been > having trouble. Do you have a specific example of a query that you have > been disappointed by in the past? For example, when I try `site:gitlab.haskell.org/ghc/ghc/issues/ packaging windows`, I don't get a hit for #18104, even going to the third page. Maybe it's not an indexing issue, but something doesn't seem to be working correctly... Thanks, Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From hecate at glitchbra.in Mon Jun 15 21:06:47 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Mon, 15 Jun 2020 23:06:47 +0200 Subject: Search in GitLab In-Reply-To: <87wo489q20.fsf@smart-cactus.org> References: <87wo489q20.fsf@smart-cactus.org> Message-ID: <9a9be0ef-91a1-b391-febd-7a13a098bca5@glitchbra.in> On 15/06/2020 19:50, Ben Gamari wrote: > Frankly, this makes me wonder whether we should change the output > produced for loops. The current error is essentially un-Googleable, as > we see here. I know I have personally struggled with this same issue in > the past. I wholeheartedly agree with this suggestion. Maybe we could even start a little taxonomy of errors by adding an error code to the message that would be more searchable? Something like E5032? From simonpj at microsoft.com Mon Jun 15 21:24:40 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Jun 2020 21:24:40 +0000 Subject: Search in GitLab In-Reply-To: <87wo489q20.fsf@smart-cactus.org> References: <87wo489q20.fsf@smart-cactus.org> Message-ID: | Frankly, this makes me wonder whether we should change the output | produced for loops. The current error is essentially un-Googleable, as | we see here. I know I have personally struggled with this same issue in | the past. I'd be fine with that. <> is pretty cryptic. Simon | -----Original Message----- | From: Ben Gamari | Sent: 15 June 2020 18:50 | To: Simon Peyton Jones ; GHC developers | Subject: Re: Search in GitLab | | Simon Peyton Jones via ghc-devs writes: | | > Does anyone know how to search better in GitLab. | > Currently I'm using the standard GitLab search. I'm searching for | > "<>" | > where I intend the quotes meaning exactly that string as usual in a | search term. But I get lots of results mentioning loop, without the angle | brackets. | > Moreover I want to sort the results by date or ticket number, and I | can't see how to do that. | > Does Google index our repo? Can I use Google to search it somehow? | | Indeed this is a hard query. It appears that the tokenizer used by | GitLab (or perhaps Elasticsearch, which is responsible for full-text | indexing) considers the angle-brackets to be token delimiters (which I | suppose is fair given that usually you see them used to signify | less-than/greater-than operators). | | Google seems to be slightly better, returning only 10 hits. If this | isn't sufficient and the query is important I could run a query against | the database directly if you would like. | | Frankly, this makes me wonder whether we should change the output | produced for loops. The current error is essentially un-Googleable, as | we see here. I know I have personally struggled with this same issue in | the past. | | Richard, I checked on Google Search Console and it appears to be | indexed on a nearly-daily basis. I'm a bit surprised that you have been | having trouble. Do you have a specific example of a query that you have | been disappointed by in the past? | | Cheers, | | - Ben From a.pelenitsyn at gmail.com Mon Jun 15 21:28:39 2020 From: a.pelenitsyn at gmail.com (Artem Pelenitsyn) Date: Mon, 15 Jun 2020 17:28:39 -0400 Subject: Search in GitLab In-Reply-To: <9a9be0ef-91a1-b391-febd-7a13a098bca5@glitchbra.in> References: <87wo489q20.fsf@smart-cactus.org> <9a9be0ef-91a1-b391-febd-7a13a098bca5@glitchbra.in> Message-ID: As a side note, the idea of making a taxonomy of errors with unique tagging has been brought up on ghc-proposals recently, although marked as out-of-scope (maybe rightly so): https://github.com/ghc-proposals/ghc-proposals/pull/325 The ease of searching is among the major motivations behind it. -- Best, Artem On Mon, Jun 15, 2020, 5:07 PM Hécate wrote: > On 15/06/2020 19:50, Ben Gamari wrote: > > Frankly, this makes me wonder whether we should change the output > > produced for loops. The current error is essentially un-Googleable, as > > we see here. I know I have personally struggled with this same issue in > > the past. > > I wholeheartedly agree with this suggestion. Maybe we could even start a > little taxonomy of errors by adding an error code > to the message that would be more searchable? Something like E5032? > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hecate at glitchbra.in Mon Jun 15 21:35:43 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Mon, 15 Jun 2020 23:35:43 +0200 Subject: Search in GitLab In-Reply-To: References: <87wo489q20.fsf@smart-cactus.org> <9a9be0ef-91a1-b391-febd-7a13a098bca5@glitchbra.in> Message-ID: <1e4a8ffa-b574-0afc-ac0a-6eff8a202d07@glitchbra.in> Thank you for referencing the issue, I couldn't find it anymore for some reason. While the technicality of the "errors-as-values" proposal might delay the implementation of such a taxonomy, I think we could totally lay the groundwork and actually work on defining it first. On 15/06/2020 23:28, Artem Pelenitsyn wrote: > As a side note, the idea of making a taxonomy of errors with unique > tagging has been brought up on ghc-proposals recently, although marked > as out-of-scope (maybe rightly so): > https://github.com/ghc-proposals/ghc-proposals/pull/325 > The ease of searching is among the major motivations behind it. > > -- > Best, Artem > > On Mon, Jun 15, 2020, 5:07 PM Hécate > wrote: > > On 15/06/2020 19:50, Ben Gamari wrote: > > Frankly, this makes me wonder whether we should change the output > > produced for loops. The current error is essentially > un-Googleable, as > > we see here. I know I have personally struggled with this same > issue in > > the past. > > I wholeheartedly agree with this suggestion. Maybe we could even > start a > little taxonomy of errors by adding an error code > to the message that would be more searchable? Something like E5032? > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Wed Jun 17 18:10:50 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 17 Jun 2020 14:10:50 -0400 Subject: Linear types performance characterisation Message-ID: <87eeqda7h6.fsf@smart-cactus.org> Hi everyone, Last week I discussed the plan for merging the LinearTypes branch into GHC 8.12 with Arnaud, Richard, Andreas, and Simon. Many thanks to all of them for their respective roles in pushing this patch over the finish line. One thing that we wanted to examine prior to merge is compiler performance across a larger collection of packages. For this I used the head.hackage patch-set, comparing the Linear Types branch with its corresponding base commit in `master`. Here I will describe the methodology used for this comparison and briefly summarize the (happily, quite positive) results. # Methodology I collected total bytes allocated (as reported by the runtime system), elapsed runtime (as reported by the runtime system), and instructions (as reported by `perf stat`) of head.hackage builds in two configurations: * the `opt` configuration * the `noopt` configuration, which passed `--disable-optimisation` to cabal-install These configurations were evaluated on two commits: * `master`: 2b792facab46f7cdd09d12e79499f4e0dcd4293f * `linear-bang`: 481cf412d6e619c0e47960f4c70fb21f19d6996d Unfortunately, the `noopt` configuration appears to be be affected by a few cabal-install bugs [1,2] and consequently some packages may *still* be compiled with optimisation, so take these numbers with a grain of salt. The test environment was a reasonably quiet Ryzen 7 1800X with 32 GBytes of RAM. The test was run by first building the two tested commits in Hadrian's default build flavour. The head.hackage CI driver was then invoked as follows: # Don't parallelize for stable performance measurements export CPUS=1 export USE_NIX=1 export EXTRA_HC_OPTS=-ddump-timings export COLLECT_PERF_STATS=1 mkdir -p runs # master export GHC=/home/ben/ghc/ghc-compare-2/_build/stage1/bin/ghc ./run-ci --cabal-option=--disable-optimisation mv ci/run runs/master-noopt ./run-ci mv ci/run runs/master-opt # linear-bang export GHC=/home/ben/ghc/ghc-compare-1/_build/stage1/bin/ghc ./run-ci --cabal-option=--disable-optimisation mv ci/run runs/linear-noopt ./run-ci mv ci/run runs/linear-opt As we are building all packages (nearly 300 in total) serially, the full run takes quite a while (around 8 hours IIRC). The final run of this test used head.hackage commit e7e5c5cfbfd42c41b1e62d42bb18483a83b78701 (on the `rts-stats` branch). # Results I examined several different metrics of compiler performance * the total_wall_seconds RTS metric gives an picture of overall compilation effort * time reported by -ddump-timings, summed by module, gives a slightly finer-grained measurement of per-module compilation time * the RTS's bytes_allocated metric gives overall compiler allocations * the RTS"s max_bytes_used metric gives a sense of AST size (and potentially the existence of leaks) To cut straight to the chase, the measurements show the following: metric -O0 -O1 ------------------- --------- ---------- total_wall_seconds +0.3% +0.6% total_cpu_seconds +0.3% +0.7% max_bytes_used +4.2% +4.8% GC_cpu_seconds +1.5% +2.1% mut_cpu_seconds no change no change sum(per-module-time) +4.2% +4.2% sum(per-module-alloc) +0.8% +0.8% There are a few things to point out here: the overall change in compiler runtime is thankfully quite reasonable. However, max_bytes_used increases rather considerably. This seems to give rise to an appreciable regression in GC time. It would be interesting to know whether this can be improved with optimisation to data representation. The fact that the cumulative per-module metrics didn't change between -O0 and -O1 indicate to me that there is a methodological problem which needs to be addressed in the test infrastructure. I investigated this a bit and have a hypothesis for what might be going on here; nevertheless, in the interest of publishing these measurements I'm ignoring these measurements for the time being. I have attached the Jupyter notebook that gave rise to these numbers. This gives a finer-grained breakdown of the data including histograms showing the variance of each metric. Perhaps this will be helpful in better understanding the effects. I would be happy to share my run data as well although it is a bit large. All-in-all, the Tweag folks have done a great job in squashing the performance numbers noticed a few weeks ago. The current numbers look quite acceptable for GHC 8.12. Congratulations to Arnaud, Krzysztof, and Richard on landing this feature! I'm very much looking forward to see what the community does with it in the coming years. Cheers, - Ben [1] https://github.com/haskell/cabal/issues/5353 [2] https://github.com/haskell/cabal/issues/3883 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: linear-types-analysis.html.gz Type: application/octet-stream Size: 277382 bytes Desc: not available URL: From chessai1996 at gmail.com Wed Jun 17 18:18:01 2020 From: chessai1996 at gmail.com (chessai) Date: Wed, 17 Jun 2020 11:18:01 -0700 Subject: Linear types performance characterisation In-Reply-To: <87eeqda7h6.fsf@smart-cactus.org> References: <87eeqda7h6.fsf@smart-cactus.org> Message-ID: Great work! I'm very excited to see these perf issues squashed. Thanks to everyone working on this, and also to Ben for such thorough benchmarking work! On Wed, Jun 17, 2020, 11:11 AM Ben Gamari wrote: > Hi everyone, > > Last week I discussed the plan for merging the LinearTypes branch into > GHC 8.12 with Arnaud, Richard, Andreas, and Simon. Many thanks to all of > them for their respective roles in pushing this patch over the finish > line. > > One thing that we wanted to examine prior to merge is compiler > performance across a larger collection of packages. For this I used > the head.hackage patch-set, comparing the Linear Types branch with its > corresponding base commit in `master`. Here I will describe the > methodology used for this comparison and briefly summarize the (happily, > quite positive) results. > > > # Methodology > > I collected total bytes allocated (as reported by the runtime system), > elapsed runtime (as reported by the runtime system), and instructions > (as reported by `perf stat`) of head.hackage builds in two > configurations: > > * the `opt` configuration > * the `noopt` configuration, which passed `--disable-optimisation` to > cabal-install > > These configurations were evaluated on two commits: > > * `master`: 2b792facab46f7cdd09d12e79499f4e0dcd4293f > * `linear-bang`: 481cf412d6e619c0e47960f4c70fb21f19d6996d > > Unfortunately, the `noopt` configuration appears to be be affected by a > few cabal-install bugs [1,2] and consequently some packages may *still* > be compiled with optimisation, so take these numbers with a grain of > salt. > > The test environment was a reasonably quiet Ryzen 7 1800X with 32 GBytes > of RAM. > > The test was run by first building the two tested commits in Hadrian's > default build flavour. The head.hackage CI driver was then invoked as > follows: > > # Don't parallelize for stable performance measurements > export CPUS=1 > export USE_NIX=1 > export EXTRA_HC_OPTS=-ddump-timings > export COLLECT_PERF_STATS=1 > > mkdir -p runs > > # master > export GHC=/home/ben/ghc/ghc-compare-2/_build/stage1/bin/ghc > ./run-ci --cabal-option=--disable-optimisation > mv ci/run runs/master-noopt > ./run-ci > mv ci/run runs/master-opt > > # linear-bang > export GHC=/home/ben/ghc/ghc-compare-1/_build/stage1/bin/ghc > ./run-ci --cabal-option=--disable-optimisation > mv ci/run runs/linear-noopt > ./run-ci > mv ci/run runs/linear-opt > > As we are building all packages (nearly 300 in total) serially, the full > run takes quite a while (around 8 hours IIRC). > > The final run of this test used head.hackage commit > e7e5c5cfbfd42c41b1e62d42bb18483a83b78701 (on the `rts-stats` branch). > > > # Results > > I examined several different metrics of compiler performance > > * the total_wall_seconds RTS metric gives an picture of overall > compilation effort > > * time reported by -ddump-timings, summed by module, gives a slightly > finer-grained measurement of per-module compilation time > > * the RTS's bytes_allocated metric gives overall compiler allocations > > * the RTS"s max_bytes_used metric gives a sense of AST size (and > potentially the existence of leaks) > > To cut straight to the chase, the measurements show the following: > > metric -O0 -O1 > ------------------- --------- ---------- > total_wall_seconds +0.3% +0.6% > total_cpu_seconds +0.3% +0.7% > max_bytes_used +4.2% +4.8% > GC_cpu_seconds +1.5% +2.1% > mut_cpu_seconds no change no change > sum(per-module-time) +4.2% +4.2% > sum(per-module-alloc) +0.8% +0.8% > > There are a few things to point out here: the overall change in compiler > runtime is thankfully quite reasonable. However, max_bytes_used > increases rather considerably. This seems to give rise to an appreciable > regression in GC time. It would be interesting to know whether this can > be improved with optimisation to data representation. > > The fact that the cumulative per-module metrics didn't change between > -O0 and -O1 indicate > to me that there is a methodological problem which needs to be addressed > in the test infrastructure. I investigated this a bit and have a > hypothesis for what might be going on here; nevertheless, in the > interest of publishing these measurements I'm ignoring these > measurements for the time being. > > I have attached the Jupyter notebook that gave rise to these numbers. > This gives a finer-grained breakdown of the data including histograms > showing the variance of each metric. Perhaps this will be helpful in > better understanding the effects. I would be happy to share my run data > as well although it is a bit large. > > All-in-all, the Tweag folks have done a great job in squashing the > performance numbers noticed a few weeks ago. The current numbers look quite > acceptable for GHC 8.12. Congratulations to Arnaud, Krzysztof, and > Richard on landing this feature! I'm very much looking forward to see > what the community does with it in the coming years. > > Cheers, > > - Ben > > > [1] https://github.com/haskell/cabal/issues/5353 > [2] https://github.com/haskell/cabal/issues/3883 > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Jun 19 08:22:23 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 19 Jun 2020 08:22:23 +0000 Subject: Gitlab 503 Message-ID: I'm getting a lot of "Gitlab is unavailable" 503 messages, after editing a wiki page. Does anyone know what is happening? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Jun 19 08:27:42 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 19 Jun 2020 08:27:42 +0000 Subject: Gitlab is stuck Message-ID: Something is badly wrong with gitlab. I'm trying to push a branch. Is some disk full? I am fully stalled. Thanks Simon bash$ git push --force Counting objects: 79, done. Delta compression using up to 20 threads. Compressing objects: 100% (65/65), done. Writing objects: 100% (79/79), 46.21 KiB | 3.55 MiB/s, done. Total 79 (delta 68), reused 16 (delta 14) error: remote unpack failed: unable to create temporary object directory remote: warning: The last gc run reported the following. Please correct the root cause remote: and remove gc.log. remote: Automatic cleanup will not be performed until the file is removed. remote: remote: warning: There are too many unreachable loose objects; run 'git prune' to remove them. remote: To gitlab.haskell.org:ghc/ghc.git ! [remote rejected] wip/T18282 -> wip/T18282 (unpacker error) error: failed to push some refs to 'git at gitlab.haskell.org:ghc/ghc.git' -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at haskus.fr Fri Jun 19 09:42:24 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Fri, 19 Jun 2020 11:42:24 +0200 Subject: Receiving type information from environment instead of hardcoding. In-Reply-To: References: <871rmhbafo.fsf@smart-cactus.org> Message-ID: <875316ba-13f0-1251-32ef-997889fdf990@haskus.fr> FYI ghc-bignum has been merged yesterday. Cheers, Sylvain On 15/06/2020 11:28, Rinat Stryungis wrote: > In light of the mentioned patch, I prefer to freeze my activity about > the unification of Nat and Natural up to the merging this patch. After > that, I am going to rebase my branch and make MR. Thank you, Ben! > > пн, 15 июн. 2020 г. в 00:32, Ben Gamari >: > > Rinat Stryungis > writes: > > > Hi. I have a question about a possible way of unification of Nat and > > Natural. I've almost done that, but only in case of using > integer-gmp. > > If I use integer-simple there is a completely different > definition of > > Natural. > > > > How I construct now naturalTyCon (to make `naturalTy` to use it > instead of > > `typeNatKind`) : > > > > ```naturalTyCon :: TyCon > > naturalTyCon = pcTyCon naturalTyConName Nothing [] > [natSDataCon,natJDataCon] > > > > natSDataCon :: DataCon > > natSDataCon = pcDataCon natSDataConName [] [wordPrimTy] naturalTyCon > > > > etc... > > ``` > > Now I have to check`DynFlags` in a few places to reimplement > `naturalTyCon` > > in case of using `integer-simple`. > > > > Is there a way to avoid hardcoding of `naturalTy`? > > My colleague said that it would be nice to get `naturalTy` from an > > environment by  something like `lookupTyCon`, > > but there are many functions whose don't use any environment > like functions > > from `typeNatTyCons` list in `GHC.Builtin.Types.Literals`. > > > > Now I am going to use `DynFlags` checking, but it looks like an > ugly way... > > Note that all of this will be moot in a matter of days. The ghc-bignum > patch, which will ship in 8.12, removes integer-simple and uses a > consistent number representation across its various supported > backends. > > In light of this, if I were you I would probably just settle for a > hack > in the meantime. > > Cheers, > > - Ben > > > > -- > Best regards. > Rinat Striungis > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Jun 19 13:36:32 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 19 Jun 2020 09:36:32 -0400 Subject: Gitlab is stuck In-Reply-To: References: Message-ID: <87a70z9nz7.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Something is badly wrong with gitlab. I'm trying to push a branch. Is some disk full? > I am fully stalled. I am rebooting the machine; indeed something is amiss here. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Fri Jun 19 14:30:22 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 19 Jun 2020 14:30:22 +0000 Subject: Gitlab is stuck In-Reply-To: <87a70z9nz7.fsf@smart-cactus.org> References: <87a70z9nz7.fsf@smart-cactus.org> Message-ID: All good now? | -----Original Message----- | From: Ben Gamari | Sent: 19 June 2020 14:37 | To: Simon Peyton Jones ; GHC developers | Subject: Re: Gitlab is stuck | | Simon Peyton Jones via ghc-devs writes: | | > Something is badly wrong with gitlab. I'm trying to push a branch. Is | some disk full? | > I am fully stalled. | | I am rebooting the machine; indeed something is amiss here. | | Cheers, | | - Ben From simonpj at microsoft.com Fri Jun 19 14:52:56 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 19 Jun 2020 14:52:56 +0000 Subject: Gitlab corrupt Message-ID: Ben Something is amiss. "corrupt" sounds bad! Simon bash$ git fetch error: garbage at end of loose object '62be92e5be07b34fc25c77e36223268c4a6eadec' fatal: loose object 62be92e5be07b34fc25c77e36223268c4a6eadec (stored in ./objects/62/be92e5be07b34fc25c77e36223268c4a6eadec) is corrupt fatal: The remote end hung up unexpectedly -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Jun 19 15:37:17 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 19 Jun 2020 11:37:17 -0400 Subject: Gitlab corrupt In-Reply-To: References: Message-ID: <874kr79idv.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Ben > Something is amiss. "corrupt" sounds bad! > Simon > Indeed something seems to have gone awry here. I'm currently working on getting to the bottom of it. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Fri Jun 19 15:39:37 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 19 Jun 2020 11:39:37 -0400 Subject: Gitlab is stuck In-Reply-To: References: <87a70z9nz7.fsf@smart-cactus.org> Message-ID: <871rmb9ia0.fsf@smart-cactus.org> Simon Peyton Jones writes: > All good now? > As noted earlier, the safe answer is "no". It appears that our storage provider is having some trouble. I'm so far unsure of whether this is a temporary problem or something more sinister. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From shayne.fletcher at daml.com Fri Jun 19 21:55:01 2020 From: shayne.fletcher at daml.com (Shayne Fletcher) Date: Fri, 19 Jun 2020 17:55:01 -0400 Subject: [ghc-lib] internal error after removal of integer-simple Message-ID: With the recent MR that removes integer-simple in favor of ghc-bignum, I find that I get a runtime failure when I try to use ghc-lib to generate core: ``` # Running: stack --no-terminal exec -- mini-compile examples/mini-compile/test/MiniCompileTest.hs examples/mini-compile/test/MiniCompileTest.hs:66:5: error: * GHC internal error: `One' is not in scope during type checking, but it passed the renamer tcl_env of environment: [628 :-> ATcTyCon TrName :: *, 62b :-> APromotionErr RecDataConPE, 62e :-> APromotionErr RecDataConPE] * In the definition of data constructor `TrNameS' In the data declaration for `TrName' | 66 | = TrNameS Addr# -- Static | ^^^^^^^^^^^^^ mini-compile: GHC internal error: `One' is not in scope during type checking, but it passed the renamer tcl_env of environment: [628 :-> ATcTyCon TrName :: *, 62b :-> APromotionErr RecDataConPE, 62e :-> APromotionErr RecDataConPE] ``` Anyone have any pointers on what is going wrong and what I should be looking at? -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* , creators of *DAML * -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at  http://www.digitalasset.com/emaildisclaimer.html . If you are not the intended recipient, please delete this message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Jun 19 22:26:58 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 19 Jun 2020 18:26:58 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: References: Message-ID: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> On June 19, 2020 5:55:01 PM EDT, Shayne Fletcher via ghc-devs wrote: >With the recent MR that removes integer-simple in favor of ghc-bignum, >I >find that I get a runtime failure when I try to use ghc-lib to generate >core: >``` ># Running: stack --no-terminal exec -- mini-compile >examples/mini-compile/test/MiniCompileTest.hs > >examples/mini-compile/test/MiniCompileTest.hs:66:5: error: > * GHC internal error: `One' is not in scope during type checking, but >it passed the renamer > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > 62b :-> APromotionErr RecDataConPE, > 62e :-> APromotionErr RecDataConPE] > * In the definition of data constructor `TrNameS' > In the data declaration for `TrName' > | >66 | = TrNameS Addr# -- Static > | ^^^^^^^^^^^^^ >mini-compile: GHC internal error: `One' is not in scope during type >checking, but it passed the renamer >tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > 62b :-> APromotionErr RecDataConPE, > 62e :-> APromotionErr RecDataConPE] >``` > >Anyone have any pointers on what is going wrong and what I should be >looking at? I have a hypothesis for what might be happening here. Investigating From shayne.fletcher.50 at gmail.com Sat Jun 20 01:23:34 2020 From: shayne.fletcher.50 at gmail.com (Shayne Fletcher) Date: Fri, 19 Jun 2020 21:23:34 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> References: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> Message-ID: On Fri, Jun 19, 2020 at 6:27 PM Ben Gamari wrote: > On June 19, 2020 5:55:01 PM EDT, Shayne Fletcher via ghc-devs < > ghc-devs at haskell.org> wrote: > >With the recent MR that removes integer-simple in favor of ghc-bignum, > >I > >find that I get a runtime failure when I try to use ghc-lib to generate > >core: > >``` > ># Running: stack --no-terminal exec -- mini-compile > >examples/mini-compile/test/MiniCompileTest.hs > > > >examples/mini-compile/test/MiniCompileTest.hs:66:5: error: > > * GHC internal error: `One' is not in scope during type checking, but > >it passed the renamer > > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > > 62b :-> APromotionErr RecDataConPE, > > 62e :-> APromotionErr RecDataConPE] > > * In the definition of data constructor `TrNameS' > > In the data declaration for `TrName' > > | > >66 | = TrNameS Addr# -- Static > > | ^^^^^^^^^^^^^ > >mini-compile: GHC internal error: `One' is not in scope during type > >checking, but it passed the renamer > >tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > > 62b :-> APromotionErr RecDataConPE, > > 62e :-> APromotionErr RecDataConPE] > >``` > > > >Anyone have any pointers on what is going wrong and what I should be > >looking at? > > I have a hypothesis for what might be happening here. Investigating > Breaks at commit `96aa57878fd6e6a7b92e841a0df8b5255a559c97` ( https://gitlab.haskell.org/ghc/ghc/-/commit/96aa57878fd6e6a7b92e841a0df8b5255a559c97) "Update compiler". -- Shayne Fletcher -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayne.fletcher at daml.com Sat Jun 20 01:24:24 2020 From: shayne.fletcher at daml.com (Shayne Fletcher) Date: Fri, 19 Jun 2020 21:24:24 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: References: Message-ID: On Fri, Jun 19, 2020 at 5:55 PM Shayne Fletcher wrote: > With the recent MR that removes integer-simple in favor of ghc-bignum, I > find that I get a runtime failure when I try to use ghc-lib to generate > core: > ``` > # Running: stack --no-terminal exec -- mini-compile > examples/mini-compile/test/MiniCompileTest.hs > > examples/mini-compile/test/MiniCompileTest.hs:66:5: error: > * GHC internal error: `One' is not in scope during type checking, but > it passed the renamer > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > 62b :-> APromotionErr RecDataConPE, > 62e :-> APromotionErr RecDataConPE] > * In the definition of data constructor `TrNameS' > In the data declaration for `TrName' > | > 66 | = TrNameS Addr# -- Static > | ^^^^^^^^^^^^^ > mini-compile: GHC internal error: `One' is not in scope during type > checking, but it passed the renamer > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, > 62b :-> APromotionErr RecDataConPE, > 62e :-> APromotionErr RecDataConPE] > ``` > > Anyone have any pointers on what is going wrong and what I should be > looking at? > Breaks at commit `96aa57878fd6e6a7b92e841a0df8b5255a559c97` ( https://gitlab.haskell.org/ghc/ghc/-/commit/96aa57878fd6e6a7b92e841a0df8b5255a559c97) "Update compiler". -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* , creators of *DAML * -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at  http://www.digitalasset.com/emaildisclaimer.html . If you are not the intended recipient, please delete this message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at haskus.fr Sat Jun 20 09:43:11 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Sat, 20 Jun 2020 11:43:11 +0200 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: References: Message-ID: <23b64ac0-a2a5-21ae-61e4-87cef65e4ec1@haskus.fr> Hi, I would think it's more related to the linear types patch given that it added ghc-prim:GHC.Types.One (wired-in). Could you open a ticket with a way to reproduce the failure? Thanks, Sylvain On 19/06/2020 23:55, Shayne Fletcher via ghc-devs wrote: > With the recent MR that removes integer-simple in favor of ghc-bignum, > I find that I get a runtime failure when I try to use ghc-lib to > generate core: > ``` > # Running: stack     --no-terminal exec -- mini-compile > examples/mini-compile/test/MiniCompileTest.hs > > examples/mini-compile/test/MiniCompileTest.hs:66:5: error: >     * GHC internal error: `One' is not in scope during type checking, > but it passed the renamer >       tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >                                62b :-> APromotionErr RecDataConPE, >                                62e :-> APromotionErr RecDataConPE] >     * In the definition of data constructor `TrNameS' >       In the data declaration for `TrName' >    | > 66 |   = TrNameS Addr#  -- Static >    |     ^^^^^^^^^^^^^ > mini-compile: GHC internal error: `One' is not in scope during type > checking, but it passed the renamer > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >                          62b :-> APromotionErr RecDataConPE, >                          62e :-> APromotionErr RecDataConPE] > ``` > > Anyone have any pointers on what is going wrong and what I should be > looking at? > > -- > *Shayne Fletcher* > Language Engineer */* +1 917 699 7663 > *Digital Asset* , creators of *DAML > * > > This message, and any attachments, is for the intended recipient(s) > only, may contain information that is privileged, confidential and/or > proprietary and subject to important terms and conditions available at > http://www.digitalasset.com/emaildisclaimer.html > . If you are not the > intended recipient, please delete this message. > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayne.fletcher at daml.com Sat Jun 20 10:56:07 2020 From: shayne.fletcher at daml.com (Shayne Fletcher) Date: Sat, 20 Jun 2020 06:56:07 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: <23b64ac0-a2a5-21ae-61e4-87cef65e4ec1@haskus.fr> References: <23b64ac0-a2a5-21ae-61e4-87cef65e4ec1@haskus.fr> Message-ID: On Sat, Jun 20, 2020 at 5:43 AM Sylvain Henry wrote: > I would think it's more related to the linear types patch given that it > added ghc-prim:GHC.Types.One (wired-in). Could you open a ticket with a way > to reproduce the failure? > That makes a lot of sense Sylvain. Thanks, will do. -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* , creators of *DAML * -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at  http://www.digitalasset.com/emaildisclaimer.html . If you are not the intended recipient, please delete this message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayne.fletcher.50 at gmail.com Sat Jun 20 11:10:41 2020 From: shayne.fletcher.50 at gmail.com (Shayne Fletcher) Date: Sat, 20 Jun 2020 07:10:41 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: References: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> Message-ID: On Fri, Jun 19, 2020 at 9:23 PM Shayne Fletcher < shayne.fletcher.50 at gmail.com> wrote: > > > On Fri, Jun 19, 2020 at 6:27 PM Ben Gamari wrote: > >> On June 19, 2020 5:55:01 PM EDT, Shayne Fletcher via ghc-devs < >> ghc-devs at haskell.org> wrote: >> >With the recent MR that removes integer-simple in favor of ghc-bignum, >> >I >> >find that I get a runtime failure when I try to use ghc-lib to generate >> >core: >> >``` >> ># Running: stack --no-terminal exec -- mini-compile >> >examples/mini-compile/test/MiniCompileTest.hs >> > >> >examples/mini-compile/test/MiniCompileTest.hs:66:5: error: >> > * GHC internal error: `One' is not in scope during type checking, but >> >it passed the renamer >> > tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >> > 62b :-> APromotionErr RecDataConPE, >> > 62e :-> APromotionErr RecDataConPE] >> > * In the definition of data constructor `TrNameS' >> > In the data declaration for `TrName' >> > | >> >66 | = TrNameS Addr# -- Static >> > | ^^^^^^^^^^^^^ >> >mini-compile: GHC internal error: `One' is not in scope during type >> >checking, but it passed the renamer >> >tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >> > 62b :-> APromotionErr RecDataConPE, >> > 62e :-> APromotionErr RecDataConPE] >> >``` >> > >> >Anyone have any pointers on what is going wrong and what I should be >> >looking at? >> >> I have a hypothesis for what might be happening here. Investigating >> > > Breaks at commit `96aa57878fd6e6a7b92e841a0df8b5255a559c97` ( > https://gitlab.haskell.org/ghc/ghc/-/commit/96aa57878fd6e6a7b92e841a0df8b5255a559c97) > "Update compiler". > Actually it seems almost certain to be `40fa237e1daab7a76b9871bb6c50b953a1addf23`, the linear types patch. So my current theory is that it is probably not a bug but instead points to a ghc-prim mismatch. -- Shayne Fletcher -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Sat Jun 20 12:52:13 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sat, 20 Jun 2020 08:52:13 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> References: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> Message-ID: <87lfkh99xm.fsf@smart-cactus.org> Ben Gamari writes: > On June 19, 2020 5:55:01 PM EDT, Shayne Fletcher via ghc-devs wrote: ... >> >>Anyone have any pointers on what is going wrong and what I should be >>looking at? > > I have a hypothesis for what might be happening here. Investigating My initial hypothesis was a unique conflict (as I had to resolve some merge conflicts in this area while merging). However, a cursory look at the relevant implementation didn't reveal any. I had to drop this last night due to another engagement but I will try to resume today. Indeed having a ticket would be helpful. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Sat Jun 20 12:55:23 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sat, 20 Jun 2020 08:55:23 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> References: <857570D6-310B-4059-856E-A8B4F2965F17@smart-cactus.org> Message-ID: <87imfl99s4.fsf@smart-cactus.org> Ben Gamari writes: > On June 19, 2020 5:55:01 PM EDT, Shayne Fletcher via ghc-devs wrote: >>With the recent MR that removes integer-simple in favor of ghc-bignum, >>I >>find that I get a runtime failure when I try to use ghc-lib to generate >>core: >>``` >># Running: stack --no-terminal exec -- mini-compile >>examples/mini-compile/test/MiniCompileTest.hs >> >>examples/mini-compile/test/MiniCompileTest.hs:66:5: error: >> * GHC internal error: `One' is not in scope during type checking, but >>it passed the renamer >> tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >> 62b :-> APromotionErr RecDataConPE, >> 62e :-> APromotionErr RecDataConPE] >> * In the definition of data constructor `TrNameS' >> In the data declaration for `TrName' >> | >>66 | = TrNameS Addr# -- Static >> | ^^^^^^^^^^^^^ >>mini-compile: GHC internal error: `One' is not in scope during type >>checking, but it passed the renamer >>tcl_env of environment: [628 :-> ATcTyCon TrName :: *, >> 62b :-> APromotionErr RecDataConPE, >> 62e :-> APromotionErr RecDataConPE] >>``` >> >>Anyone have any pointers on what is going wrong and what I should be >>looking at? > One additional question: Can you confirm that this is reproducible with a clean build? Specifically, this sort of thing is very likely to occur if you have stale interface files (e.g. build GHC, pull some commits, and do an incremental build). It would be good to exclude this possibility. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From shayne.fletcher at daml.com Sat Jun 20 13:18:18 2020 From: shayne.fletcher at daml.com (Shayne Fletcher) Date: Sat, 20 Jun 2020 09:18:18 -0400 Subject: [ghc-lib] internal error after removal of integer-simple In-Reply-To: References: <23b64ac0-a2a5-21ae-61e4-87cef65e4ec1@haskus.fr> Message-ID: On Sat, Jun 20, 2020 at 6:56 AM Shayne Fletcher wrote: > > > On Sat, Jun 20, 2020 at 5:43 AM Sylvain Henry wrote: > >> I would think it's more related to the linear types patch given that it >> added ghc-prim:GHC.Types.One (wired-in). Could you open a ticket with a way >> to reproduce the failure? >> > > That makes a lot of sense Sylvain. Thanks, will do. > After some scratching around, I've pretty much convinced myself that there isn't an issue here to be raised against GHC upstream. There IS a problem to be solved in how ghc-lib relates with ghc-prim but that problem is firmly on the ghc-lib side. So, there won't be a ticket for now that stems from this. Thanks again for your input Sylvain! -- *Shayne Fletcher* Language Engineer */* +1 917 699 7663 *Digital Asset* , creators of *DAML * -- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at  http://www.digitalasset.com/emaildisclaimer.html . If you are not the intended recipient, please delete this message. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Sat Jun 20 15:51:17 2020 From: ben at well-typed.com (Ben Gamari) Date: Sat, 20 Jun 2020 11:51:17 -0400 Subject: GitLab status Message-ID: <87d05t91n2.fsf@smart-cactus.org> Hi everyone, As you may have noticed, yesterday was marked by rather consistent CI failures. I believe I have now fixed the root cause but do let me know if you see any further issues. Below I have included a brief post-mortem describing the incident and what we plan to do to prevent it recurring in the future. Cheers, - Ben # Post-mortem Early Friday morning our storage provider experienced a hiccup which rendered the volume which backed our GitLab repositories unavailable for a few minutes. The interruption was long enough that the filesystem remounted as read-only. This caused a bit of filesystem damage affecting a handful of objects in the ghc/perf-notes repository. This resulted in CI failures when attempts to `git push` to this repository failed. To address this I started by ensuring we had an up-to-date backup of our data and began sorting through the various observed errors. Once I had established that the storage volume had been interrupted I went looking for additional corruption. A integrity check of all GitLab repositories revealed no further corruption. This isn't terribly surprising given that the perf notes repository sees the most commit traffic of all repositories hosted on GitLab. While it would likely be possible to recover the corrupted perf-notes objects from clones on the CI builders, I deemed this to be not worth the effort given that these commits hold replaceable performance metric data and the last good commit appears to have been produced mere hours prior to the corrupted HEAD commit. Consequently, I rather reverted the ghc-notes repository to the last-known-good commit (8154013bfdce86fedf2863cb96ccbb723f1144f8). # Planned changes for future mitigation While this incident didn't result in any significant data loss, it was nevertheless a significant headache and resulted in CI failing for the better part of a day. Moreover, this isn't the first time that the network block storage backing GitLab has let us down. At this point, I have lost confidence that network block storage can be trusted. For this reason, in the future we will eliminate this failure mode by adjusting our deployment to avoid relying on network block storage for any deployment-critical data. We hope that we will be able to carry out this change in the coming weeks. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Sun Jun 21 04:08:13 2020 From: ben at well-typed.com (Ben Gamari) Date: Sun, 21 Jun 2020 00:08:13 -0400 Subject: GitLab status In-Reply-To: <87d05t91n2.fsf@smart-cactus.org> References: <87d05t91n2.fsf@smart-cactus.org> Message-ID: <87zh8x6oye.fsf@smart-cactus.org> Ben Gamari writes: > Hi everyone, > > As you may have noticed, yesterday was marked by rather consistent CI > failures. I believe I have now fixed the root cause but do let me know > if you see any further issues. > > Below I have included a brief post-mortem describing the incident and > what we plan to do to prevent it recurring in the future. > Hi all, Unfortunately it seems that our provider's block storage issues have once again reared their ugly head. I will be moving GitLab to a new server in the morning. Until then, expect gitlab.haskell.org to be down. Sorry for any inconvenience. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From csaba.hruska at gmail.com Sun Jun 21 13:40:47 2020 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Sun, 21 Jun 2020 15:40:47 +0200 Subject: Introducing GHC whole program compiler (GHC-WPC) In-Reply-To: References: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> Message-ID: Hello, All the questions you have asked are important, but before I answer them I'd like to emphasize that as a research platform it is worth supporting WPC/WPA even with the worst case memory and runtime properties. It would allow researchers to observe Haskell programs with arbitrary precision. Not giving access to the whole program IR simply rules out certain types of research directions. IMO this is an important issue, because some experiment might reveal something that is applicable in practice or might be implemented in the incremental compilation pipeline as well. I find it cool to allow developers to use GHC in an unexpected or unintended way that may lead to something new and valuable. It is common sense that whole (or large chunks) program analysis and compilation is infeasible. I believe that common sense needs to be reevaluated time to time, because the conditions may change. E.g.. Would the existence of huge amounts of RAM with multicore CPUs or GPU or quantum computers change the situation? Not to mention the new research results of the static analysis field. Questions that come to mind (to be understood in the context of the above > enthusiasm): > > - If you compile a program that depends on (say) lens, you get a lot > of code. Dead-code elim will drop lots, perhaps, but you start with > everything. So what do memory footprints and compile times look like when > you do WPC? Remembering that people often complain about GHC’s footprint > when compiling a *single* module. > Also, WPC means that instead of just linking to precompiled libraries, > you have to recompile (parts of) them. What does that do to compile times? > > I deliberately split the WPC pipeline into IR export and import/codegen pieces. GHC-WPC does only the STG IR export, and the External STG compiler could be seen as a separate project that uses GHC-WPC as frontend and GHC as backend. It is important to note that compilation pipeline design is not fixed in this setting. It could implement the regular per module incremental compilation, or could batch compile multiple modules or even the whole program. The IR export part is the same for each case, only the backend driver differs. While it is true that the GHC-WPC project implements the whole program compiler scenario currently, I also plan to extend it to allow incremental compilation with arbitrary sized program clusters. So it would not use the source code structure based module or package boundaries, instead the compilation unit size granularity could be changed at will. Possibly driven by some static or runtime analysis. It would be a free parameter that can be adjusted to get an acceptable compilation time, and as the software and hardware advances the practical compilation unit size would increase. It is also possible to implement incremental whole program compilation. This technique is already used in the Visual C++ compiler: Incremental Whole Program Optimization and Compilation The current whole program compilation pipeline first compiles the project with GHC-WPC. GHC-WPC compiles the target project through the standard GHC backend and generates executable, in addition it exports the Ext-STG IR. Then gen-exe is executed and the whole stg program compilation is done using the following steps: 1. extracting liveness datalog facts from each project module 2. running the whole program liveness analysis 3. per module link time codegen for the whole project cutting out the dead top level functions using the liveness analysis result I designed the pipeline to have a light memory footprint. I intentionally implemented the static analyses in Datalog using the Souffle Datalog compiler. Souffle generates small and efficient parallel (OpenMP) C++ code that stores the in-memory database in specialised data types (Trie/B-tree). Another important design choice was to collect the datalog facts incrementally for static analysis, instead of loading the whole program IR to the memory. The insight is that it is not the complete IR that is needed for whole program analysis, but just a projection of the IR. It is perfectly OK to calculate that projection on a per module basis, then run the static analysis on the collected data. Furthermore it is even possible to store the collected facts in files separately for each module for later reuse. I measured the compilation of pandoc and a simple (hello world like) project. I put the collected data grouped by compilation stages into bullet points with notes: - ext-stg export Currently the serializer uses binary via the generic instance, which is not the fastest method. The store package via TH would be the fastest way of exporting the IR, but it is not in the foundation libraries. Also at the current stage of the project this is not an issue (binary is good enough). - liveness datalog fact generator simple: 2.107545216s pandoc: 18.431660236s - liveness datalog facts size: simple: 11 MB pandoc: 186 MB - liveness analysis (CSV) result size: simple: 92 KB pandoc: 14 MB - liveness analysis memory usage: simple: Maximum resident set size: 15 MB pandoc: Maximum resident set size: 109 MB - liveness analysis run time: simple: 0.118736736s pandoc: 2.322970868s - link time codegen via GHC: simple: 15.683492995s pandoc: 790.070061268s memory usage: around 100 MB (typical GHC module compilation footprint) - gen-exe is the whole stg program compiler driver: simple: Maximum resident set size: 186 MB Elapsed (wall clock) time: 18.63 sec pandoc: Maximum resident set size: 401 MB Elapsed (wall clock) time: 13 min 35 sec GHC-WPC Ext-STG IR serializer - without ext-stg export -O0 from scratch with deps (full world) simple: 5.455 sec pandoc: 8 min 54.363 sec - with ext-stg export -O0 from scratch with deps (full world) simple: 5.576 sec pandoc: 12 min 50.101 sec > - I love the 25% reduction in binary size. In fact I’m surprised it > isn’t bigger. > > I used a simple and fast method for the link time dead code elimination. It calculates the references of the module top level functions transitively, then only the reachable ones are passed to the codegen. So it does not eliminate the local closures, nor the dead case alternatives. Check the analysis source code how simple the implementation is: ext-stg-liveness.dl I also implemented a much more sophisticated control flow analysis, which could eliminate much more code. i.e. - constructor data fields - case alternatives - function parameters - local closures It is future work to integrate this precise CFA into the external STG compiler pipeline. My initial goal was to create the simplest and fastest working whole compiler pipeline for GHC. So I prioritised my plans and tasks accordingly. I'd like to mention that the current pipeline design is a result of iterative development. The first version was written entirely in Haskell. It comprised a points-to analysis and the simple liveness analysis. The Haskell version of these analyses worked fine for small programs, but the memory and runtime properties were bad. Of course I tried to add some strictness annotations, that helped a bit, but it was not good enough. Then I rewrote the points-to analysis in C++, which resulted in a much better memory footprint, but my fixed point evaluator was still naive so it was not fast enough. Also the C++ implementation increased the development and maintenance complexity that I did not like. Luckily I found the Souffle Datalog compiler that just fits my needs. It is super easy to write static analyses in Datalog and it does not compromise in the performance either. I'd like to write all these parts in Haskell but IMO it is not possible yet. Hopefully this will change in the future, maybe LoCal can make a big difference. (http://recurial.com/pldi19main.pdf) > - Why are you using STG? It’s mostly untyped – or at least much less > strongly typed than Core. It has lots of restrictions (like ANF) that Core > does not. Indeed I think of STG as a little lily-pad to alight on in the > hop from Core to Cmm. Maybe your entire setup would work equally well > with Core, provided you can serialise and deserialise it. > > To me every IR layer is an API for the compiler. The long term existence of the different IRs must have a reason. As I understand, the GHC Core is about polymorphic reasoning and STG IR is about operational semantics with explicit notion of evaluation, allocation of closures/data constructors and pattern matching. Thirdly, Cmm is the language of the runtime system that expresses the lowest level memory operations. I chose STG because I am interested in the analysis of the operational semantics of Haskell programs. I see STG IR as an entry/exit point for the GHC pipeline. The actual IR data type does not matter because most researchers or developers will have their own needs regarding the IR conventions. I.e. I'd like to write analyses in Souffle/Datalog but both GHC Core and GHC STG are insufficient for direct datalog translation, because not every expression has a name, neither a unique one. But this is totally fine. It is impossible to design the perfect IR, instead everyone should do the customised conversion for their project. IMO STG is implicitly typed, its type system is the TYPE's RuntimeRep kind parameter that describes the value’s runtime representation. And this is exactly what I'm interested in, because the Core type system is not expressive enough for the optimizations that I'd like to implement. In fact GHC has the unarise pass that flattens unboxed tuples, and it is implemented in STG because it would not type check in Core. Also there are Cmm and LLVM transformations that are semantically valid but not typeable in GHC Core. Ideally it would be great to have a single IR that could express both Core and Cmm properties in a typed way, but it would require a much stronger (probably dependent) type system. I find this idea a promising research direction. Indeed the whole GHC-WPC could support Core also, it would just be an additional export operation at the same place in the GHC and Cabal source code where STG gets exported. The same would be true for the Cmm IR. > - Moreover, we **already** have a fast serialiser and deserialiser for > Core – the stuff we use for interface files. So maybe you could re-use > that … no need for pretty-print and parse. > > Ideally there would be a binary import/export facility and pretty printer for every IR layer. The speed might matter for the main use cases but for the experimental cases it is not an issue at all. Serializers would allow us to use GHC internal components with finer granularity and from arbitrary programming languages. Having a binary IR (de)serializer without a specification or without stability guarantees is still much better than having nothing. > - You say “That would mean a significant conceptual shift in the GHC > compiler pipeline, because heavy optimizations would be introduced at the > low level IRs beside GHC Core.” Fair enough, but what I’m missing is the > *rationale* for doing heavy opts on STG rather than Core. > > The Core type system is not expressive enough to reason about memory layout. Of course STG's RuntimeRep type system cannot describe the shape of the heap either, but at least it can be used as a starting point in an experiment. Haskell and pure functional programming formed my views on how to automate programming tasks, and how to utilise the type system for specific domains. Haskell as a pure FP also shaped how I see compilers. The traditional view is that at the top of the compilation pipeline there is the high level language with a rich and rigorous type system, and during the compilation process (lowering) every lower level IR has a weaker type system that encodes slightly less semantic information. And at the level of assembly language or machine code there is nothing left from the original types and semantic information. So instead of this traditional approach I can imagine a pipeline that preserves high level semantic information during the lowering process. But for this the low-level IRs would require increasingly more expressive type systems. I'd like to explore this idea further and GHC would be a good platform for such an experiment. I also believe that this direction is the future of compilers and programming languages. > > - Apart from (a) dead code, and (b) GRIN, do you have ideas in mind > for what we could do with WPC? > > I have lots of ideas regarding the usage of the whole program IR. - GHC-WPC's exported STG IR could be useful for any backend or codegen project. The existing cross compilation projects like Asterius and GHCJS could use the External-STG as input IR. The benefit of this approach is that the alternative Haskell compiler backends do not need to be supported by Cabal directly instead they would entirely rely on the External STG IR and the exported linker metadata file. - External STG could allow using GHC as frontend and backend, with additional support of custom IR transformations. E.g. maybe the Gibbon language project could compile real-world Haskell programs via Ext-STG. Or maybe Purescript or Idris would add GHC with RTS as a new target. STG's weaker type system is actually beneficial for programming languages that differ from Haskell in evaluation model or type system. Maybe I'll add external Core or external Cmm support also. - The Intel Haskell Research Compiler optimizer part is called Functional Language Research Compiler (FLRC ) and its IR is called MIL. Its source is on github and it still compiles. I'd like to plug the FLRC vectorising and memory layout optimizer into GHC. FLRC's RTS was not optimized for laziness, however it would be interesting to see how it would perform with GHC's fine tuned GC and RTS. - I also plan to implement an STG interpreter with FFI support that could run any Haskell program. With such an interpreter I could implement runtime control flow tracing, or a stop the world debugger that could observe and visualize the runtime heap values. I'd like to track the source origin and lifetime of run-time values. I'd use that information to build a better intuition of the Haskell program’s runtime behaviour. I hope that it will lead to important insights to improve the performance. I'm optimistic with this idea because AFAIK it has not been done yet. Haskell's GC removes the unreachable values from the memory that might be dead for a long time. According to the As Static As Possible Memory Management thesis, reachability is too conservative approximation of the actual liveness. - I’d like to do partial defunctionalization on the STG IR. Regarding the engineering complexity of an optimizing compiler I believe it is the easiest to start with a whole program analysis and compilation. It is easier to construct a static analysis as a data flow analysis, then later with the insights the same analysis could be formulated as a type system. Which might enable compositional analysis, and therefore incremental compilation. I see GHC-WPC as a framework that would allow us to explore these ideas. I often hear from researchers and Haskell experts that whole program compilation does not work. They seem so confident with this claim, and I'd like to understand why. I read several papers on the topic and also checked quite a few compilers’ source code. LLVM, GCC, Visual C++ is doing LTO. Intel Haskell Research Compiler research results, MLton, Boquist's GRIN supports the idea of whole program analysis and compilation. At least it seems reasonable to continue the research in this direction. I wonder if those who concluded against WPA read the same papers and projects. Do you think that I am on the wrong track? Am I chasing unimportant or impossible things? Important papers: - The Intel Labs Haskell Research Compiler http://www.leafpetersen.com/leaf/publications/hs2013/hrc-paper.pdf - Measuring the Haskell Gap http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf - LoCal: A Language for Programs Operating on Serialized Data http://recurial.com/pldi19main.pdf - On fast large-scale program analysis in Datalog https://souffle-lang.github.io/pdf/cc.pdf - ASAP: As Static As Possible memory management https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-908.pdf - Pushdown Control-Flow Analysis for Free https://arxiv.org/abs/1507.03137 Regards, Csaba Hruska On Mon, Jun 15, 2020 at 11:34 AM Simon Peyton Jones wrote: > I’ve always thought that whole-program compilation has the possibility of > doing optimisations that are simply inaccessible without the whole program, > but been daunted by the engineering challenges of making WPC actually > work. So it’s fantastic that you’ve made progress on this. Well done! > > > > Questions that come to mind (to be understood in the context of the above > enthusiasm): > > - If you compile a program that depends on (say) lens, you get a lot > of code. Dead-code elim will drop lots, perhaps, but you start with > everything. So what do memory footprints and compile times look like when > you do WPC? Remembering that people often complain about GHC’s footprint > when compiling a *single* module. > > > > Also, WPC means that instead of just linking to precompiled libraries, you > have to recompile (parts of) them. What does that do to compile times? > > > > - I love the 25% reduction in binary size. In fact I’m surprised it > isn’t bigger. > > > > - Why are you using STG? It’s mostly untyped – or at least much less > strongly typed than Core. It has lots of restrictions (like ANF) that Core > does not. Indeed I think of STG as a little lily-pad to alight on in the > hop from Core to Cmm. Maybe your entire setup would work equally well > with Core, provided you can serialise and deserialise it. > > > > - Moreover, we **already** have a fast serialiser and deserialiser for > Core – the stuff we use for interface files. So maybe you could re-use > that … no need for pretty-print and parse. > > > > - You say “That would mean a significant conceptual shift in the GHC > compiler pipeline, because heavy optimizations would be introduced at the > low level IRs beside GHC Core.” Fair enough, but what I’m missing is the > *rationale* for doing heavy opts on STG rather than Core. > > > > - Apart from (a) dead code, and (b) GRIN, do you have ideas in mind > for what we could do with WPC? > > > > Thanks > > > > Simon > > > > > > > > > > *From:* ghc-devs *On Behalf Of *Csaba > Hruska > *Sent:* 14 June 2020 13:46 > *To:* Alexis King > *Cc:* GHC developers > *Subject:* Re: Introducing GHC whole program compiler (GHC-WPC) > > > > Hi, > > > > I thought about the GHC-LTO project name before, but it would not be an > accurate description though. The GHC-WPC in its current state is about > exporting STG + linker info for later processing, either feed it back to > GHC backend or to a third party pipeline. It depends what the > user/researcher wants, the point is that GHC-WPC solves the IR export part > of the issue. It is the external stg compiler that implements a (simple) > whole program dead function elimination pass that I implemented as a proof > of concept to show the new possibilities GHC-WPC opens up. But I plan to do > much more optimization with sophisticated dataflow analyses. I.e. I have a > fast and working implementation of control flow analysis in souffle/datalog > that I plan to use to do more accurate dead code elimination and partial > program defunctionalization on the whole program STG IR. In theory I could > implement all GRIN optimizations on STG. That would mean a significant > conceptual shift in the GHC compiler pipeline, because heavy optimizations > would be introduced at the low level IRs beside GHC Core. I'd like to go > even further with experimentation. I can imagine a dependently typed Cmm > with a similar type system that ATS ( > http://www.ats-lang.org/MYDATA/VsTsVTs-2018-10-28.pdf > ) > has. I definitely would like to make an experiment in the future, to come > up with an Idirs2 EDSL for GHC RTS heap operations where the type system > would ensure the correctness of pointer arithmetic and heap object > manipulation. The purpose of GHC-WPC in this story is to deliver the IR for > these stuff. > > > > Beside exporting STG IR, the external STG compiler can compile STG via > GHC's standard code generator. This makes GHC codegen/RTS available as a > backend for programming language developers. I.e. Idris, Agda, Purescript > could use GHC/STG/RTS as a backend with all of its cool features. > > > > So these are the key parts of my vision about the purpose and development > of GHC-WPC. It is meant to be more than a link time optimizer. > > > > Cheers, > > Csaba > > > > On Sat, Jun 13, 2020 at 10:26 PM Alexis King > wrote: > > Hi Csaba, > > > > I originally posted this comment on /r/haskell > before > I saw you also sent this to ghc-devs. I’ve decided to reproduce my comment > here as well, since this list probably has a more relevant audience: > > > > I want to start by saying that I think this sounds totally awesome, and I > think it’s a fantastic idea. I’m really interested in seeing how this > progresses! > > > I do wonder if people might find the name a little misleading. “Whole > program compilation” usually implies “whole program optimization,” but most > of GHC’s key optimizations happen at the Core level, before STG is even > generated. (Of course, I’m sure you’re well aware of that, I’m just stating > it for the sake of others who might be reading who aren’t aware.) > > > This seems much closer in spirit to “link-time optimization” (LTO) as > performed by Clang and GCC than whole program compilation. For example, > Clang’s LTO works by “linking” LLVM bitcode files instead of fully-compiled > native objects. STG is not quite analogous to LLVM IR—GHC’s analog would be > Cmm, not STG—but I think that difference is not that significant here: the > STG-to-Cmm pass is quite mechanical, and STG is mostly just easier to > manipulate. > > > tl;dr: Have you considered naming this project GHC-LTO instead of GHC-WPC? > > > > Alexis > > > > On Jun 12, 2020, at 16:16, Csaba Hruska wrote: > > > > Hello, > > > > I've created a whole program compilation pipeline for GHC via STG. > > Please read my blog post for the details: > > Introducing GHC whole program compiler (GHC-WPC) > > > > > Regards, > > Csaba Hruska > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Jun 22 03:52:00 2020 From: ben at well-typed.com (Ben Gamari) Date: Sun, 21 Jun 2020 23:52:00 -0400 Subject: GitLab status In-Reply-To: <87zh8x6oye.fsf@smart-cactus.org> References: <87d05t91n2.fsf@smart-cactus.org> <87zh8x6oye.fsf@smart-cactus.org> Message-ID: <87pn9r7o67.fsf@smart-cactus.org> Ben Gamari writes: > Ben Gamari writes: > >> Hi everyone, >> >> As you may have noticed, yesterday was marked by rather consistent CI >> failures. I believe I have now fixed the root cause but do let me know >> if you see any further issues. >> >> Below I have included a brief post-mortem describing the incident and >> what we plan to do to prevent it recurring in the future. >> > Hi all, > > Unfortunately it seems that our provider's block storage issues have > once again reared their ugly head. I will be moving GitLab to a new > server in the morning. Until then, expect gitlab.haskell.org to be down. > Sorry for any inconvenience. > A brief update: GitLab is now up again. Moreover, it has been upgraded to GitLab 13.0.6, which hopefully bring some load-time performance improvements. However, do be aware that the Docker images are still being transferred and consequently CI jobs may fail for the next few hours. Also, the Darwin CI runners need to be upgraded and will be down until this happens. I am hopeful that this will happen tomorrow. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From fumiexcel at gmail.com Mon Jun 22 10:09:30 2020 From: fumiexcel at gmail.com (Fumiaki Kinoshita) Date: Mon, 22 Jun 2020 19:09:30 +0900 Subject: Merge window logistics and relaxation of submodule policy In-Reply-To: <87sgf2k4j7.fsf@smart-cactus.org> References: <87sgf2k4j7.fsf@smart-cactus.org> Message-ID: I have no authority but it sounds good to me! In particular it would allow me to make https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3174 pass the CI. 2020年6月11日(木) 6:14 Ben Gamari : > Hi everyone, > > At this point we are in the home stretch of the GHC 8.12 preparation > phase. The merge queue has progressed a bit more slowly than I would > have liked over the last few weeks but nevertheless I think we are > converging. > > Regardless, if you have a patch which you believe should go in to 8.12 > and doesn't already bear the %8.12 GitLab milestone, please do let me know. > > > # Submodule policy > > One matter that is currently holding us up is a few submodules which > haven't yet had necessary patches merged upstream. As there hasn't yet > been much motion on the part of maintainer, I propose that we > temporarily relax our policy that GHC submodules refer to upstream > commits. That is, we allow GHC to temporarily track a non-upstream > branch, ensuring that the branch is merged upstream before the final GHC > release. > > Are there any objections to this policy relaxation? It will require a > bit more care on our part, but I think it is a sensible compromise way > to give upstream maintainers the time they need while ensuring that GHC > isn't subject to unpredictable schedules. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan.gl.scott at gmail.com Mon Jun 22 10:34:53 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Mon, 22 Jun 2020 06:34:53 -0400 Subject: GitLab status Message-ID: Just as an FYI, there appears to be some data loss from this migration. For example, I pushed a commit to !3536 [1] that appears to have been dropped, leading to confusion here [2]. To make things worse, I never received an e-mail notification for this comment—I only discovered it by chance. I also opened an issue about FUN, but that appears to have been lost. I could try to resubmit it, but perhaps the data still exists somewhere on servers somewhere? Ryan S. ----- [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3536 [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3536#note_283322 -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jun 22 12:43:14 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 22 Jun 2020 12:43:14 +0000 Subject: GHC/ | GHC doesn't inline small type class method marked as INLINE with profiling enabled (#18372) In-Reply-To: References: Message-ID: I get a 404 for https://gitlab.haskell.org/ghc/ghc/-/issues/18372 You may need to re-submit this… s From: Andrzej Rybczak Sent: 21 June 2020 15:44 To: Simon Peyton Jones Subject: GHC/ | GHC doesn't inline small type class method marked as INLINE with profiling enabled (#18372) Andrzej Rybczak created an issue: Summary When profiling is enabled, GHC doesn't inline small type class methods which prevents further optimizations and specialization from happening. Steps to reproduce When the following module: module Main where import Optics.Core data T = T { _t1 :: Int , _t2 :: Int } t1 :: Lens' T Int t1 = lensVL $ \f s -> (\n -> s { _t1 = n}) <$> f (_t1 s) {-# INLINE t1 #-} t2 :: Lens' T Int t2 = lensVL $ \f s -> (\n -> s { _t2 = n}) <$> f (_t2 s) {-# INLINE t2 #-} t_val :: T t_val = T 1 2 {-# NOINLINE t_val #-} main :: IO () main = putStrLn . show $ view t1 t_val + view t2 t_val is compiled with profiling enabled, here's how core of main looks: main1 = case ((($fStrongForget_$clinear main_l1) (id `cast` )) `cast` ) t_val of { I# x_a2c8 -> case ((($fStrongForget_$clinear main_l2) (id `cast` )) `cast` ) t_val of { I# y_a2cb -> case $wshowSignedInt 0# (+# x_a2c8 y_a2cb) [] of { (# ww5_a277, ww6_a278 #) -> : ww5_a277 ww6_a278 } } } So linear doesn't get inlined even though it's a small function that is even explicitly marked INLINE. This prevents further optimizations from happening and leaves optics-related code in semi-optimized state, making looking at cost centers unreliable (see well-typed/optics#324). When profiling is disabled, everything inlines and optimizes away as expected: main1 = case t_val of { T ds_d25F ds1_d25G -> case ds_d25F of { I# x_a2c7 -> case ds1_d25G of { I# y_a2ca -> case $wshowSignedInt 0# (+# x_a2c7 y_a2ca) [] of { (# ww5_a276, ww6_a277 #) -> : ww5_a276 ww6_a277 } } } } I'm attaching archive with cabal project that contains above module for easy reproduction: prof_test.tar.gz Profiling build was tested with profiling: True in cabal.project.local. Expected behavior GHC should inline class methods marked as INLINE when profiling is enabled. Environment * GHC version used: 8.10.1 Optional: * Operating System: Arch Linux * System Architecture: x86_64 — Reply to this email directly or view it on GitLab. You're receiving this email because of your account on gitlab.haskell.org. If you'd like to receive fewer emails, you can unsubscribe from this thread or adjust your notification settings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jun 22 12:50:27 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 22 Jun 2020 12:50:27 +0000 Subject: Introducing GHC whole program compiler (GHC-WPC) In-Reply-To: References: <34F9E5B3-6BA2-4D0B-9AB1-5090F8AA4668@gmail.com> Message-ID: I often hear from researchers and Haskell experts that whole program compilation does not work. Do you? Not from me I hope. My position is rather: I think there should be things we can get from WPC, but dead-code elim is the only quick win that is obvious to me. I fully expect there are less obvious wins, but I would love to see demonstrated examples. So I don’t think you are on the wrong track! Simon From: Csaba Hruska Sent: 21 June 2020 14:41 To: Simon Peyton Jones Cc: Alexis King ; GHC developers Subject: Re: Introducing GHC whole program compiler (GHC-WPC) Hello, All the questions you have asked are important, but before I answer them I'd like to emphasize that as a research platform it is worth supporting WPC/WPA even with the worst case memory and runtime properties. It would allow researchers to observe Haskell programs with arbitrary precision. Not giving access to the whole program IR simply rules out certain types of research directions. IMO this is an important issue, because some experiment might reveal something that is applicable in practice or might be implemented in the incremental compilation pipeline as well. I find it cool to allow developers to use GHC in an unexpected or unintended way that may lead to something new and valuable. It is common sense that whole (or large chunks) program analysis and compilation is infeasible. I believe that common sense needs to be reevaluated time to time, because the conditions may change. E.g.. Would the existence of huge amounts of RAM with multicore CPUs or GPU or quantum computers change the situation? Not to mention the new research results of the static analysis field. Questions that come to mind (to be understood in the context of the above enthusiasm): * If you compile a program that depends on (say) lens, you get a lot of code. Dead-code elim will drop lots, perhaps, but you start with everything. So what do memory footprints and compile times look like when you do WPC? Remembering that people often complain about GHC’s footprint when compiling a single module. Also, WPC means that instead of just linking to precompiled libraries, you have to recompile (parts of) them. What does that do to compile times? I deliberately split the WPC pipeline into IR export and import/codegen pieces. GHC-WPC does only the STG IR export, and the External STG compiler could be seen as a separate project that uses GHC-WPC as frontend and GHC as backend. It is important to note that compilation pipeline design is not fixed in this setting. It could implement the regular per module incremental compilation, or could batch compile multiple modules or even the whole program. The IR export part is the same for each case, only the backend driver differs. While it is true that the GHC-WPC project implements the whole program compiler scenario currently, I also plan to extend it to allow incremental compilation with arbitrary sized program clusters. So it would not use the source code structure based module or package boundaries, instead the compilation unit size granularity could be changed at will. Possibly driven by some static or runtime analysis. It would be a free parameter that can be adjusted to get an acceptable compilation time, and as the software and hardware advances the practical compilation unit size would increase. It is also possible to implement incremental whole program compilation. This technique is already used in the Visual C++ compiler: Incremental Whole Program Optimization and Compilation The current whole program compilation pipeline first compiles the project with GHC-WPC. GHC-WPC compiles the target project through the standard GHC backend and generates executable, in addition it exports the Ext-STG IR. Then gen-exe is executed and the whole stg program compilation is done using the following steps: 1. extracting liveness datalog facts from each project module 2. running the whole program liveness analysis 3. per module link time codegen for the whole project cutting out the dead top level functions using the liveness analysis result I designed the pipeline to have a light memory footprint. I intentionally implemented the static analyses in Datalog using the Souffle Datalog compiler. Souffle generates small and efficient parallel (OpenMP) C++ code that stores the in-memory database in specialised data types (Trie/B-tree). Another important design choice was to collect the datalog facts incrementally for static analysis, instead of loading the whole program IR to the memory. The insight is that it is not the complete IR that is needed for whole program analysis, but just a projection of the IR. It is perfectly OK to calculate that projection on a per module basis, then run the static analysis on the collected data. Furthermore it is even possible to store the collected facts in files separately for each module for later reuse. I measured the compilation of pandoc and a simple (hello world like) project. I put the collected data grouped by compilation stages into bullet points with notes: * ext-stg export Currently the serializer uses binary via the generic instance, which is not the fastest method. The store package via TH would be the fastest way of exporting the IR, but it is not in the foundation libraries. Also at the current stage of the project this is not an issue (binary is good enough). * liveness datalog fact generator simple: 2.107545216s pandoc: 18.431660236s * liveness datalog facts size: simple: 11 MB pandoc: 186 MB * liveness analysis (CSV) result size: simple: 92 KB pandoc: 14 MB * liveness analysis memory usage: simple: Maximum resident set size: 15 MB pandoc: Maximum resident set size: 109 MB * liveness analysis run time: simple: 0.118736736s pandoc: 2.322970868s * link time codegen via GHC: simple: 15.683492995s pandoc: 790.070061268s memory usage: around 100 MB (typical GHC module compilation footprint) * gen-exe is the whole stg program compiler driver: simple: Maximum resident set size: 186 MB Elapsed (wall clock) time: 18.63 sec pandoc: Maximum resident set size: 401 MB Elapsed (wall clock) time: 13 min 35 sec GHC-WPC Ext-STG IR serializer * without ext-stg export -O0 from scratch with deps (full world) simple: 5.455 sec pandoc: 8 min 54.363 sec * with ext-stg export -O0 from scratch with deps (full world) simple: 5.576 sec pandoc: 12 min 50.101 sec * I love the 25% reduction in binary size. In fact I’m surprised it isn’t bigger. I used a simple and fast method for the link time dead code elimination. It calculates the references of the module top level functions transitively, then only the reachable ones are passed to the codegen. So it does not eliminate the local closures, nor the dead case alternatives. Check the analysis source code how simple the implementation is: ext-stg-liveness.dl I also implemented a much more sophisticated control flow analysis, which could eliminate much more code. i.e. * constructor data fields * case alternatives * function parameters * local closures It is future work to integrate this precise CFA into the external STG compiler pipeline. My initial goal was to create the simplest and fastest working whole compiler pipeline for GHC. So I prioritised my plans and tasks accordingly. I'd like to mention that the current pipeline design is a result of iterative development. The first version was written entirely in Haskell. It comprised a points-to analysis and the simple liveness analysis. The Haskell version of these analyses worked fine for small programs, but the memory and runtime properties were bad. Of course I tried to add some strictness annotations, that helped a bit, but it was not good enough. Then I rewrote the points-to analysis in C++, which resulted in a much better memory footprint, but my fixed point evaluator was still naive so it was not fast enough. Also the C++ implementation increased the development and maintenance complexity that I did not like. Luckily I found the Souffle Datalog compiler that just fits my needs. It is super easy to write static analyses in Datalog and it does not compromise in the performance either. I'd like to write all these parts in Haskell but IMO it is not possible yet. Hopefully this will change in the future, maybe LoCal can make a big difference. (http://recurial.com/pldi19main.pdf) * Why are you using STG? It’s mostly untyped – or at least much less strongly typed than Core. It has lots of restrictions (like ANF) that Core does not. Indeed I think of STG as a little lily-pad to alight on in the hop from Core to Cmm. Maybe your entire setup would work equally well with Core, provided you can serialise and deserialise it. To me every IR layer is an API for the compiler. The long term existence of the different IRs must have a reason. As I understand, the GHC Core is about polymorphic reasoning and STG IR is about operational semantics with explicit notion of evaluation, allocation of closures/data constructors and pattern matching. Thirdly, Cmm is the language of the runtime system that expresses the lowest level memory operations. I chose STG because I am interested in the analysis of the operational semantics of Haskell programs. I see STG IR as an entry/exit point for the GHC pipeline. The actual IR data type does not matter because most researchers or developers will have their own needs regarding the IR conventions. I.e. I'd like to write analyses in Souffle/Datalog but both GHC Core and GHC STG are insufficient for direct datalog translation, because not every expression has a name, neither a unique one. But this is totally fine. It is impossible to design the perfect IR, instead everyone should do the customised conversion for their project. IMO STG is implicitly typed, its type system is the TYPE's RuntimeRep kind parameter that describes the value’s runtime representation. And this is exactly what I'm interested in, because the Core type system is not expressive enough for the optimizations that I'd like to implement. In fact GHC has the unarise pass that flattens unboxed tuples, and it is implemented in STG because it would not type check in Core. Also there are Cmm and LLVM transformations that are semantically valid but not typeable in GHC Core. Ideally it would be great to have a single IR that could express both Core and Cmm properties in a typed way, but it would require a much stronger (probably dependent) type system. I find this idea a promising research direction. Indeed the whole GHC-WPC could support Core also, it would just be an additional export operation at the same place in the GHC and Cabal source code where STG gets exported. The same would be true for the Cmm IR. * Moreover, we *already* have a fast serialiser and deserialiser for Core – the stuff we use for interface files. So maybe you could re-use that … no need for pretty-print and parse. Ideally there would be a binary import/export facility and pretty printer for every IR layer. The speed might matter for the main use cases but for the experimental cases it is not an issue at all. Serializers would allow us to use GHC internal components with finer granularity and from arbitrary programming languages. Having a binary IR (de)serializer without a specification or without stability guarantees is still much better than having nothing. * You say “That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core.” Fair enough, but what I’m missing is the rationale for doing heavy opts on STG rather than Core. The Core type system is not expressive enough to reason about memory layout. Of course STG's RuntimeRep type system cannot describe the shape of the heap either, but at least it can be used as a starting point in an experiment. Haskell and pure functional programming formed my views on how to automate programming tasks, and how to utilise the type system for specific domains. Haskell as a pure FP also shaped how I see compilers. The traditional view is that at the top of the compilation pipeline there is the high level language with a rich and rigorous type system, and during the compilation process (lowering) every lower level IR has a weaker type system that encodes slightly less semantic information. And at the level of assembly language or machine code there is nothing left from the original types and semantic information. So instead of this traditional approach I can imagine a pipeline that preserves high level semantic information during the lowering process. But for this the low-level IRs would require increasingly more expressive type systems. I'd like to explore this idea further and GHC would be a good platform for such an experiment. I also believe that this direction is the future of compilers and programming languages. * Apart from (a) dead code, and (b) GRIN, do you have ideas in mind for what we could do with WPC? I have lots of ideas regarding the usage of the whole program IR. * GHC-WPC's exported STG IR could be useful for any backend or codegen project. The existing cross compilation projects like Asterius and GHCJS could use the External-STG as input IR. The benefit of this approach is that the alternative Haskell compiler backends do not need to be supported by Cabal directly instead they would entirely rely on the External STG IR and the exported linker metadata file. * External STG could allow using GHC as frontend and backend, with additional support of custom IR transformations. E.g. maybe the Gibbon language project could compile real-world Haskell programs via Ext-STG. Or maybe Purescript or Idris would add GHC with RTS as a new target. STG's weaker type system is actually beneficial for programming languages that differ from Haskell in evaluation model or type system. Maybe I'll add external Core or external Cmm support also. * The Intel Haskell Research Compiler optimizer part is called Functional Language Research Compiler (FLRC) and its IR is called MIL. Its source is on github and it still compiles. I'd like to plug the FLRC vectorising and memory layout optimizer into GHC. FLRC's RTS was not optimized for laziness, however it would be interesting to see how it would perform with GHC's fine tuned GC and RTS. * I also plan to implement an STG interpreter with FFI support that could run any Haskell program. With such an interpreter I could implement runtime control flow tracing, or a stop the world debugger that could observe and visualize the runtime heap values. I'd like to track the source origin and lifetime of run-time values. I'd use that information to build a better intuition of the Haskell program’s runtime behaviour. I hope that it will lead to important insights to improve the performance. I'm optimistic with this idea because AFAIK it has not been done yet. Haskell's GC removes the unreachable values from the memory that might be dead for a long time. According to the As Static As Possible Memory Management thesis, reachability is too conservative approximation of the actual liveness. * I’d like to do partial defunctionalization on the STG IR. Regarding the engineering complexity of an optimizing compiler I believe it is the easiest to start with a whole program analysis and compilation. It is easier to construct a static analysis as a data flow analysis, then later with the insights the same analysis could be formulated as a type system. Which might enable compositional analysis, and therefore incremental compilation. I see GHC-WPC as a framework that would allow us to explore these ideas. I often hear from researchers and Haskell experts that whole program compilation does not work. They seem so confident with this claim, and I'd like to understand why. I read several papers on the topic and also checked quite a few compilers’ source code. LLVM, GCC, Visual C++ is doing LTO. Intel Haskell Research Compiler research results, MLton, Boquist's GRIN supports the idea of whole program analysis and compilation. At least it seems reasonable to continue the research in this direction. I wonder if those who concluded against WPA read the same papers and projects. Do you think that I am on the wrong track? Am I chasing unimportant or impossible things? Important papers: * The Intel Labs Haskell Research Compiler http://www.leafpetersen.com/leaf/publications/hs2013/hrc-paper.pdf * Measuring the Haskell Gap http://www.leafpetersen.com/leaf/publications/ifl2013/haskell-gap.pdf * LoCal: A Language for Programs Operating on Serialized Data http://recurial.com/pldi19main.pdf * On fast large-scale program analysis in Datalog https://souffle-lang.github.io/pdf/cc.pdf * ASAP: As Static As Possible memory management https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-908.pdf * Pushdown Control-Flow Analysis for Free https://arxiv.org/abs/1507.03137 Regards, Csaba Hruska On Mon, Jun 15, 2020 at 11:34 AM Simon Peyton Jones > wrote: I’ve always thought that whole-program compilation has the possibility of doing optimisations that are simply inaccessible without the whole program, but been daunted by the engineering challenges of making WPC actually work. So it’s fantastic that you’ve made progress on this. Well done! Questions that come to mind (to be understood in the context of the above enthusiasm): * If you compile a program that depends on (say) lens, you get a lot of code. Dead-code elim will drop lots, perhaps, but you start with everything. So what do memory footprints and compile times look like when you do WPC? Remembering that people often complain about GHC’s footprint when compiling a single module. Also, WPC means that instead of just linking to precompiled libraries, you have to recompile (parts of) them. What does that do to compile times? * I love the 25% reduction in binary size. In fact I’m surprised it isn’t bigger. * Why are you using STG? It’s mostly untyped – or at least much less strongly typed than Core. It has lots of restrictions (like ANF) that Core does not. Indeed I think of STG as a little lily-pad to alight on in the hop from Core to Cmm. Maybe your entire setup would work equally well with Core, provided you can serialise and deserialise it. * Moreover, we *already* have a fast serialiser and deserialiser for Core – the stuff we use for interface files. So maybe you could re-use that … no need for pretty-print and parse. * You say “That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core.” Fair enough, but what I’m missing is the rationale for doing heavy opts on STG rather than Core. * Apart from (a) dead code, and (b) GRIN, do you have ideas in mind for what we could do with WPC? Thanks Simon From: ghc-devs > On Behalf Of Csaba Hruska Sent: 14 June 2020 13:46 To: Alexis King > Cc: GHC developers > Subject: Re: Introducing GHC whole program compiler (GHC-WPC) Hi, I thought about the GHC-LTO project name before, but it would not be an accurate description though. The GHC-WPC in its current state is about exporting STG + linker info for later processing, either feed it back to GHC backend or to a third party pipeline. It depends what the user/researcher wants, the point is that GHC-WPC solves the IR export part of the issue. It is the external stg compiler that implements a (simple) whole program dead function elimination pass that I implemented as a proof of concept to show the new possibilities GHC-WPC opens up. But I plan to do much more optimization with sophisticated dataflow analyses. I.e. I have a fast and working implementation of control flow analysis in souffle/datalog that I plan to use to do more accurate dead code elimination and partial program defunctionalization on the whole program STG IR. In theory I could implement all GRIN optimizations on STG. That would mean a significant conceptual shift in the GHC compiler pipeline, because heavy optimizations would be introduced at the low level IRs beside GHC Core. I'd like to go even further with experimentation. I can imagine a dependently typed Cmm with a similar type system that ATS (http://www.ats-lang.org/MYDATA/VsTsVTs-2018-10-28.pdf) has. I definitely would like to make an experiment in the future, to come up with an Idirs2 EDSL for GHC RTS heap operations where the type system would ensure the correctness of pointer arithmetic and heap object manipulation. The purpose of GHC-WPC in this story is to deliver the IR for these stuff. Beside exporting STG IR, the external STG compiler can compile STG via GHC's standard code generator. This makes GHC codegen/RTS available as a backend for programming language developers. I.e. Idris, Agda, Purescript could use GHC/STG/RTS as a backend with all of its cool features. So these are the key parts of my vision about the purpose and development of GHC-WPC. It is meant to be more than a link time optimizer. Cheers, Csaba On Sat, Jun 13, 2020 at 10:26 PM Alexis King > wrote: Hi Csaba, I originally posted this comment on /r/haskell before I saw you also sent this to ghc-devs. I’ve decided to reproduce my comment here as well, since this list probably has a more relevant audience: I want to start by saying that I think this sounds totally awesome, and I think it’s a fantastic idea. I’m really interested in seeing how this progresses! I do wonder if people might find the name a little misleading. “Whole program compilation” usually implies “whole program optimization,” but most of GHC’s key optimizations happen at the Core level, before STG is even generated. (Of course, I’m sure you’re well aware of that, I’m just stating it for the sake of others who might be reading who aren’t aware.) This seems much closer in spirit to “link-time optimization” (LTO) as performed by Clang and GCC than whole program compilation. For example, Clang’s LTO works by “linking” LLVM bitcode files instead of fully-compiled native objects. STG is not quite analogous to LLVM IR—GHC’s analog would be Cmm, not STG—but I think that difference is not that significant here: the STG-to-Cmm pass is quite mechanical, and STG is mostly just easier to manipulate. tl;dr: Have you considered naming this project GHC-LTO instead of GHC-WPC? Alexis On Jun 12, 2020, at 16:16, Csaba Hruska > wrote: Hello, I've created a whole program compilation pipeline for GHC via STG. Please read my blog post for the details: Introducing GHC whole program compiler (GHC-WPC) Regards, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Mon Jun 22 17:12:04 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 22 Jun 2020 13:12:04 -0400 Subject: GitLab status In-Reply-To: References: Message-ID: <87lfkf6n4u.fsf@smart-cactus.org> Ryan Scott writes: > Just as an FYI, there appears to be some data loss from this migration. For > example, I pushed a commit to !3536 [1] that appears to have been dropped, > leading to confusion here [2]. To make things worse, I never received an > e-mail notification for this comment—I only discovered it by chance. > > I also opened an issue about FUN, but that appears to have been lost. I > could try to resubmit it, but perhaps the data still exists somewhere on > servers somewhere? > The machine was supposed to be unreachable for the duration of the migration. Unfortunately, it appears there was a roughly three-hour-long window where this was not true. I have manually migrated the data that introduced within this window to the new instance. Many apologies for the inconvenience. Otherwise, things should at this point be back to normal. I'm now going to do a sweep of open MRs and restart CI on those which have failed. Also, do note that you will find that gitlab.haskell.org has a new SSH host key. You will need to invalidate the existing host key entry in ~/.ssh/known_hosts. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From klebinger.andreas at gmx.at Tue Jun 23 21:55:00 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Tue, 23 Jun 2020 23:55:00 +0200 Subject: Fixing type synonyms to Uniq(D)FM newtypes Message-ID: There was a discussion about making UniqFM typed for the keys here a while ago. (https://mail.haskell.org/pipermail/ghc-devs/2020-January/018451.html and following) I wrote up an MR for one possible approach here: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3577 It implements solution 2 from that discussion. Just while getting the patch to typecheck I've already seen a number of cases where this increased readability of the code quite a bit so I think it's a good improvement. If there are strong objections to this solution let me know. In that case I'm happy to abandon the patch. If not I will clean it up and get it ready for merging. From george.colpitts at gmail.com Tue Jun 23 22:40:58 2020 From: george.colpitts at gmail.com (George Colpitts) Date: Tue, 23 Jun 2020 19:40:58 -0300 Subject: Fixing type synonyms to Uniq(D)FM newtypes In-Reply-To: References: Message-ID: I read the email thread you refer to but it doesn't seem to explain why you went with solution 2. If you think it worthwhile can you explain here why you chose solution 2? On Tue, Jun 23, 2020 at 6:55 PM Andreas Klebinger wrote: > There was a discussion about making UniqFM typed for the keys here a > while ago. > (https://mail.haskell.org/pipermail/ghc-devs/2020-January/018451.html > and following) > > I wrote up an MR for one possible approach here: > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3577 > > It implements solution 2 from that discussion. > > Just while getting the patch to typecheck I've already seen a number of > cases where this increased > readability of the code quite a bit so I think it's a good improvement. > > If there are strong objections to this solution let me know. In that > case I'm happy to abandon the patch. > If not I will clean it up and get it ready for merging. > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klebinger.andreas at gmx.at Tue Jun 23 23:08:55 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Wed, 24 Jun 2020 01:08:55 +0200 Subject: Fixing type synonyms to Uniq(D)FM newtypes In-Reply-To: References: Message-ID: <4b213f54-24db-2f9c-034f-c24573121554@gmx.at> My main motivation for going with a phantom type over newtypes was that it makes it easier to use in an adhoc fashion without giving up type safety. As a second benefit it seemed a lot easier to implement. Cheers Andreas George Colpitts schrieb am 24.06.2020 um 00:40: > I read the email thread you refer to but it doesn't seem to explain > why you went with solution 2. If you think it worthwhile can you > explain here why you chose solution 2? > > On Tue, Jun 23, 2020 at 6:55 PM Andreas Klebinger > > wrote: > > There was a discussion about making UniqFM typed for the keys here a > while ago. > (https://mail.haskell.org/pipermail/ghc-devs/2020-January/018451.html > and following) > > I wrote up an MR for one possible approach here: > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3577 > > It implements solution 2 from that discussion. > > Just while getting the patch to typecheck I've already seen a > number of > cases where this increased > readability of the code quite a bit so I think it's a good > improvement. > > If there are strong objections to this solution let me know. In that > case I'm happy to abandon the patch. > If not I will clean it up and get it ready for merging. > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Wed Jun 24 20:34:18 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 24 Jun 2020 16:34:18 -0400 Subject: [HIW'20] Third Call for Talks In-Reply-To: <87zha4yjf9.fsf@smart-cactus.org> References: <87zha4yjf9.fsf@smart-cactus.org> Message-ID: <87y2oc5hko.fsf@smart-cactus.org> Hello everyone, Haskell Implementors Workshop is calling for talk proposals. Co-located with ICFP, HiW is an ideal place to describe a Haskell library, a Haskell extension, compiler, works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. The deadline for submissions is next week, July 2nd 2020. Call for Talks ============== The 12th Haskell Implementors’ Workshop is to be held alongside ICFP 2020 this year. It is a forum for people involved in the design and development of Haskell implementations, tools, libraries, and supporting infrastructure, to share their work and discuss future directions and collaborations with others. Talks and/or demos are proposed by submitting an abstract, and selected by a small program committee. There will be no published proceedings. The workshop will be informal and interactive, with open spaces in the timetable and room for ad-hoc discussion, demos and lightning talks. Scope and Target Audience ------------------------- It is important to distinguish the Haskell Implementors’ Workshop from the Haskell Symposium which is also co-located with ICFP 2020. The Haskell Symposium is for the publication of Haskell-related research. In contrast, the Haskell Implementors’ Workshop will have no proceedings – although we will aim to make talk videos, slides and presented data available with the consent of the speakers. The Implementors’ Workshop is an ideal place to describe a Haskell extension, describe works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. Members of the wider Haskell community encouraged to attend the workshop – we need your feedback to keep the Haskell ecosystem thriving. Students working with Haskell are specially encouraged to share their work. The scope covers any of the following topics. There may be some topics that people feel we’ve missed, so by all means submit a proposal even if it doesn’t fit exactly into one of these buckets: - Compilation techniques - Language features and extensions - Type system implementation - Concurrency and parallelism: language design and implementation - Performance, optimization and benchmarking - Virtual machines and run-time systems - Libraries and tools for development or deployment Talks ----- We invite proposals from potential speakers for talks and demonstrations. We are aiming for 20-minute talks with 5 minutes for questions and changeovers. We want to hear from people writing compilers, tools, or libraries, people with cool ideas for directions in which we should take the platform, proposals for new features to be implemented, and half-baked crazy ideas. Please submit a talk title and abstract of no more than 300 words. Submissions can be made via HotCRP at https://icfp-hiw20.hotcrp.com/ until July 2nd (anywhere on earth). We will also have lightning talks session. These have been very well received in recent years, and we aim to increase the time available to them. Lightning talks be ~7mins and are scheduled on the day of the workshop. Suggested topics for lightning talks are to present a single idea, a work-in-progress project, a problem to intrigue and perplex Haskell implementors, or simply to ask for feedback and collaborators. Logistics --------- Due to the on-going COVID-19 situation, ICFP (and, consequently, HIW) will be held remotely this year. However, the organizers are still working hard to provide for a great workshop experience. While we are sad that this year will lack the robust hallway track that is often the highlight of HIW, we believe that this remote workshop presents a unique opportunity to include more of the Haskell community in our discussion and explore new modes of communicating with our colleagues. We hope that you will join us in making this HIW as vibrant as any other. Program Committee ----------------- - Andrey Mokhov (Newcastle University) - Ben Gamari (Well-Typed LLP) - Christian Baaij (QBayLogic) - George Karachalias (Tweag I/O) - Klara Marntirosian (KU Leuven) - Matthew Pickering (Univeristy of Bristol) - Ryan G.L. Scott (Indiana University Bloomington) Best wishes, ~ Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Thu Jun 25 10:41:12 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Thu, 25 Jun 2020 11:41:12 +0100 Subject: IP/key change for gitlab.haskell.org? Message-ID: An innocent `git push` yielded this today: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The ED25519 host key for gitlab.haskell.org has changed, and the key for the corresponding IP address 139.178.85.33 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:/dI7zsBRZNPB+0TqskF7rSaZ/LhQw0cF4c5W+4uMlRo. Please contact your system administrator. Add correct host key in /Users/rae/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /Users/rae/.ssh/known_hosts:21 ED25519 host key for gitlab.haskell.org has changed and you have requested strict checking. Host key verification failed. I know the server had a rough weekend. Is this a natural consequence, or is something fishy going on? Thanks, Richard From simon.jakobi at googlemail.com Thu Jun 25 10:52:14 2020 From: simon.jakobi at googlemail.com (Simon Jakobi) Date: Thu, 25 Jun 2020 12:52:14 +0200 Subject: IP/key change for gitlab.haskell.org? In-Reply-To: References: Message-ID: Hi Richard, Ben had pointed out this issue in https://mail.haskell.org/pipermail/ghc-devs/2020-June/019000.html On my system I used the command ssh-keygen -f "/home/simon/.ssh/known_hosts" -R gitlab.haskell.org to remove the problematic key. The next `git pull` then included a prompt to add the new key. Cheers, Simon Am Do., 25. Juni 2020 um 12:41 Uhr schrieb Richard Eisenberg : > > An innocent `git push` yielded this today: > > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > The ED25519 host key for gitlab.haskell.org has changed, > and the key for the corresponding IP address 139.178.85.33 > is unknown. This could either mean that > DNS SPOOFING is happening or the IP address for the host > and its host key have changed at the same time. > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > Someone could be eavesdropping on you right now (man-in-the-middle attack)! > It is also possible that a host key has just been changed. > The fingerprint for the ED25519 key sent by the remote host is > SHA256:/dI7zsBRZNPB+0TqskF7rSaZ/LhQw0cF4c5W+4uMlRo. > Please contact your system administrator. > Add correct host key in /Users/rae/.ssh/known_hosts to get rid of this message. > Offending ED25519 key in /Users/rae/.ssh/known_hosts:21 > ED25519 host key for gitlab.haskell.org has changed and you have requested strict checking. > Host key verification failed. > > I know the server had a rough weekend. Is this a natural consequence, or is something fishy going on? > > Thanks, > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Thu Jun 25 11:41:40 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 25 Jun 2020 07:41:40 -0400 Subject: IP/key change for gitlab.haskell.org? In-Reply-To: References: Message-ID: <73CDDD21-C12F-4CDD-8163-97A96B46C867@smart-cactus.org> On June 25, 2020 6:41:12 AM EDT, Richard Eisenberg wrote: >An innocent `git push` yielded this today: > >@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ >@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >The ED25519 host key for gitlab.haskell.org has changed, >and the key for the corresponding IP address 139.178.85.33 >is unknown. This could either mean that >DNS SPOOFING is happening or the IP address for the host >and its host key have changed at the same time. >@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ >@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! >Someone could be eavesdropping on you right now (man-in-the-middle >attack)! >It is also possible that a host key has just been changed. >The fingerprint for the ED25519 key sent by the remote host is >SHA256:/dI7zsBRZNPB+0TqskF7rSaZ/LhQw0cF4c5W+4uMlRo. >Please contact your system administrator. >Add correct host key in /Users/rae/.ssh/known_hosts to get rid of this >message. >Offending ED25519 key in /Users/rae/.ssh/known_hosts:21 >ED25519 host key for gitlab.haskell.org has changed and you have >requested strict checking. >Host key verification failed. > >I know the server had a rough weekend. Is this a natural consequence, >or is something fishy going on? > >Thanks, >Richard >_______________________________________________ >ghc-devs mailing list >ghc-devs at haskell.org >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs Yes, as Simon pointed out this is expected. My apologies for the incovenience. Cheers, - Ben From ml at stefansf.de Fri Jun 26 10:13:44 2020 From: ml at stefansf.de (Stefan Schulze Frielinghaus) Date: Fri, 26 Jun 2020 12:13:44 +0200 Subject: IP/key change for gitlab.haskell.org? In-Reply-To: References: Message-ID: <20200626101344.GA28253@localhost.localdomain> Just in case someone else runs into a similar problem as mine which is related to the SSH key change: Removing the old SSH fingerprint from known_hosts file and then running git resulted in an error for me and the culprit seems to be SSH: $ ssh -vvvT git at gitlab.haskell.org ... debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY Connection reset by 139.178.85.33 port 22 If I manually add gitlab.haskell.org,139.178.85.33 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA7ltOZyaULDgxE3Vw6RgQVp+OPKQi79ssUenbhdWy36 to $HOME/.ssh/known_hosts, then SSH works again for me and consequently git also. I retrieved the known_hosts file entry manually via $ ssh-keyscan gitlab.haskell.org >> $HOME/.ssh/known_hosts which interestingly works without any problem. Note, I'm running a stock Fedora 32 without any SSH configuration changes except adding my SSH key. This is still kind of mystic to me and so far I couldn't figure out why this is the case, but at least it solves my problem. Hope this helps in case someone else runs into the same problem. Cheers, Stefan On Thu, Jun 25, 2020 at 11:41:12AM +0100, Richard Eisenberg wrote: > An innocent `git push` yielded this today: > > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > The ED25519 host key for gitlab.haskell.org has changed, > and the key for the corresponding IP address 139.178.85.33 > is unknown. This could either mean that > DNS SPOOFING is happening or the IP address for the host > and its host key have changed at the same time. > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > Someone could be eavesdropping on you right now (man-in-the-middle attack)! > It is also possible that a host key has just been changed. > The fingerprint for the ED25519 key sent by the remote host is > SHA256:/dI7zsBRZNPB+0TqskF7rSaZ/LhQw0cF4c5W+4uMlRo. > Please contact your system administrator. > Add correct host key in /Users/rae/.ssh/known_hosts to get rid of this message. > Offending ED25519 key in /Users/rae/.ssh/known_hosts:21 > ED25519 host key for gitlab.haskell.org has changed and you have requested strict checking. > Host key verification failed. > > I know the server had a rough weekend. Is this a natural consequence, or is something fishy going on? > > Thanks, > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From simonpj at microsoft.com Fri Jun 26 10:29:35 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 26 Jun 2020 10:29:35 +0000 Subject: Perf notes Message-ID: Despite a recent git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf I'm getting lots of perf regressions in HEAD. For example =====> T9203(normal) 1 of 1 [0, 0, 0] ]0;T9203(normal) 1 of 1 [0, 0, 0]cd "T9203.run" && "/home/simonpj/code/HEAD-3/inplace/bin/ghc-stage2" -o T9203 T9203.hs -dcore-lint -dstg-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat -dno-debug-output -O2< cd "T9203.run" && ./T9203 +RTS -V0 -tT9203.stats --machine-readable -RTS < runtime/bytes allocated increased from x86_64-linux-deb9 baseline @ HEAD~28: Expected T9203 (normal) runtime/bytes allocated: 56046952.0 +/-5% Lower bound T9203 (normal) runtime/bytes allocated: 53244604 Upper bound T9203 (normal) runtime/bytes allocated: 58849300 Actual T9203 (normal) runtime/bytes allocated: 108464536 Deviation T9203 (normal) runtime/bytes allocated: 93.5 % *** unexpected stat test failure for T9203(normal) Performance Metrics (test environment: local): T9203(normal) runtime/bytes allocated 108464536.000 (baseline @ HEAD~28) 56046952.000 [increased, 93.5%] What am I doing wrong? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide at well-typed.com Fri Jun 26 11:06:20 2020 From: davide at well-typed.com (David Eichmann) Date: Fri, 26 Jun 2020 12:06:20 +0100 Subject: Perf notes In-Reply-To: References: Message-ID: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> Hi Simon, skip to the list at the bottom for TL;DR Every time I get an email about perf notes my heart sinks a little. Hopefully there isn't a big issues here. First of all, what commit is your branch based on? Have you rebased on a recent master? The output you posted says "...increased from x86_64-linux-deb9 baseline @ HEAD~28". So this means it is using metrics from CI as a baseline (that's the "x86_64-linux-deb9" part), but the baseline is from 28 commits ago (that's the "HEAD~28" part). The baseline seems a bit old. Also looking at gitlab CI, there are a lot of recent commits on master without completed CI runs.  So this might be a matter of waiting for CI to finish, then fetching the CI metrics again. Any way this may help: TL;DR 1. Rebase of the latest master 2. Wait for CI to finish on a more recent commit (see https://gitlab.haskell.org/ghc/ghc/-/commits/master) 3.git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf 4. Re run the tests Alternatively you can generate local metrics 1. Checkout a recent commit to use as the baseline (make sure the working tree is clean) 2. Run the relevant perf tests 3. Checkout your branches HEAD commit again 4. Run the relevant tests again. If that doesn't do it, I can have a closer look. David E On 6/26/20 11:29 AM, Simon Peyton Jones via ghc-devs wrote: > > Despite a recent > > git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git > > > refs/notes/perf:refs/notes/ci/perf > > I’m getting lots of perf regressions in HEAD. For example > > =====> T9203(normal) 1 of 1 [0, 0, 0] > > ]0;T9203(normal) 1 of 1 [0, 0, 0]cd "T9203.run" && > "/home/simonpj/code/HEAD-3/inplace/bin/ghc-stage2" -o T9203 T9203.hs > -dcore-lint -dstg-lint -dcmm-lint -no-user-package-db -rtsopts > -fno-warn-missed-specialisations -fshow-warning-groups > -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat > -dno-debug-output  -O2< > > cd "T9203.run" && ./T9203  +RTS -V0 -tT9203.stats --machine-readable > -RTS < > > runtime/bytes allocated increased from x86_64-linux-deb9 baseline @ > HEAD~28: > >     Expected    T9203 (normal) runtime/bytes allocated: 56046952.0 +/-5% > >     Lower bound T9203 (normal) runtime/bytes allocated:   53244604 > >     Upper bound T9203 (normal) runtime/bytes allocated:   58849300 > >     Actual      T9203 (normal) runtime/bytes allocated:  108464536 > >     Deviation   T9203 (normal) runtime/bytes allocated:       93.5 % > > *** unexpected stat test failure for T9203(normal) > > Performance Metrics (test environment: local): > > T9203(normal) runtime/bytes allocated                     108464536.000 > >                       (baseline @ HEAD~28)                         > 56046952.000  [increased, 93.5%] > > What am I doing wrong? > > Simon > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Jun 26 14:16:50 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 26 Jun 2020 14:16:50 +0000 Subject: Perf notes In-Reply-To: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> References: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> Message-ID: Also looking at gitlab CI, there are a lot of recent commits on master without completed CI runs I thought that wasn't possible. Isn't that what CI is *for*? And in any case, I don't think anyone will have accepted a doubling of bytes-allocated on T9803, in the last 28 commits, without lots of song and dance. Ben do you know what is going on? Simon From: ghc-devs On Behalf Of David Eichmann Sent: 26 June 2020 12:06 To: ghc-devs at haskell.org Subject: Re: Perf notes Hi Simon, skip to the list at the bottom for TL;DR Every time I get an email about perf notes my heart sinks a little. Hopefully there isn't a big issues here. First of all, what commit is your branch based on? Have you rebased on a recent master? The output you posted says "...increased from x86_64-linux-deb9 baseline @ HEAD~28". So this means it is using metrics from CI as a baseline (that's the "x86_64-linux-deb9" part), but the baseline is from 28 commits ago (that's the "HEAD~28" part). The baseline seems a bit old. Also looking at gitlab CI, there are a lot of recent commits on master without completed CI runs. So this might be a matter of waiting for CI to finish, then fetching the CI metrics again. Any way this may help: TL;DR 1. Rebase of the latest master 2. Wait for CI to finish on a more recent commit (see https://gitlab.haskell.org/ghc/ghc/-/commits/master) 3. git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf 4. Re run the tests Alternatively you can generate local metrics 1. Checkout a recent commit to use as the baseline (make sure the working tree is clean) 2. Run the relevant perf tests 3. Checkout your branches HEAD commit again 4. Run the relevant tests again. If that doesn't do it, I can have a closer look. David E On 6/26/20 11:29 AM, Simon Peyton Jones via ghc-devs wrote: Despite a recent git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf I'm getting lots of perf regressions in HEAD. For example =====> T9203(normal) 1 of 1 [0, 0, 0] ]0;T9203(normal) 1 of 1 [0, 0, 0]cd "T9203.run" && "/home/simonpj/code/HEAD-3/inplace/bin/ghc-stage2" -o T9203 T9203.hs -dcore-lint -dstg-lint -dcmm-lint -no-user-package-db -rtsopts -fno-warn-missed-specialisations -fshow-warning-groups -fdiagnostics-color=never -fno-diagnostics-show-caret -Werror=compat -dno-debug-output -O2< cd "T9203.run" && ./T9203 +RTS -V0 -tT9203.stats --machine-readable -RTS < runtime/bytes allocated increased from x86_64-linux-deb9 baseline @ HEAD~28: Expected T9203 (normal) runtime/bytes allocated: 56046952.0 +/-5% Lower bound T9203 (normal) runtime/bytes allocated: 53244604 Upper bound T9203 (normal) runtime/bytes allocated: 58849300 Actual T9203 (normal) runtime/bytes allocated: 108464536 Deviation T9203 (normal) runtime/bytes allocated: 93.5 % *** unexpected stat test failure for T9203(normal) Performance Metrics (test environment: local): T9203(normal) runtime/bytes allocated 108464536.000 (baseline @ HEAD~28) 56046952.000 [increased, 93.5%] What am I doing wrong? Simon _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide at well-typed.com Sat Jun 27 19:38:38 2020 From: davide at well-typed.com (David Eichmann) Date: Sat, 27 Jun 2020 20:38:38 +0100 Subject: Perf notes In-Reply-To: References: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> Message-ID: <96926365-d098-c17c-f94b-1047c5c962fa@well-typed.com> > I thought that wasn’t possible.  Isn’t that what CI is **for**? Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline. Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master. [1] "|fatal: Could not read from remote repository." | https://gitlab.haskell.org/ghc/ghc/-/jobs/378487 -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jun 29 14:08:35 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 29 Jun 2020 14:08:35 +0000 Subject: Perf notes In-Reply-To: <96926365-d098-c17c-f94b-1047c5c962fa@well-typed.com> References: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> <96926365-d098-c17c-f94b-1047c5c962fa@well-typed.com> Message-ID: Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master. Thanks Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline. OK. But they are guaranteed to be 100.0% identical to the ones discovered by CI, aren't they? So it's just an implementation detail whether the numbers you save are gotten from one run, or another identical one. I'm still lost about when I can rely on the perf output of CI and when I can't. I'm really hoping for a simple answer like: * The CI log tells you the comparison between the preceding commit and this one No ifs, no buts. Simple! Incidentally, would it be possible to output a table (in the log) like we get from nofib-analyse. It looks like this Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- boyer -0.3% +5.4% +0.7% +1.0% 0.0% cichelli -0.3% +5.9% -9.9% -9.5% 0.0% compress2 -0.4% +9.6% +7.2% +6.4% 0.0% constraints -0.3% +0.2% -3.0% -3.4% 0.0% cryptarithm2 -0.3% -3.9% -2.2% -2.4% 0.0% gamteb -0.4% +2.5% +2.8% +2.8% 0.0% life -0.3% -2.2% -4.7% -4.9% 0.0% lift -0.3% -0.3% -0.8% -0.5% 0.0% linear -0.3% -0.1% -4.1% -4.5% 0.0% mate -0.2% +1.4% -2.2% -1.9% -14.3% parser -0.3% -2.1% -5.4% -4.6% 0.0% puzzle -0.3% +2.1% -6.6% -6.3% 0.0% simple -0.4% +2.8% -3.4% -3.3% -2.2% veritas -0.1% +0.7% -0.6% -1.1% 0.0% wheel-sieve2 -0.3% -19.2% -24.9% -24.5% -42.9% -------------------------------------------------------------------------------- Min -0.4% -19.2% -24.9% -24.5% -42.9% Max +0.1% +9.6% +7.2% +6.4% +33.3% Geometric Mean -0.3% -0.0% -3.0% -2.9% -0.3% Instantly comprehensible, one line per benchmark. I find I spent quite a lot of time search manually in the log and manually building a table (or excerpts thereof) looking like this. I don't have an opinion about the columns, just wanting a table with one line per benchmark, and a number of columns. Thanks Simon From: David Eichmann Sent: 27 June 2020 20:39 To: Simon Peyton Jones ; ghc-devs at haskell.org Subject: Re: Perf notes > I thought that wasn't possible. Isn't that what CI is *for*? Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline. Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master. [1] "fatal: Could not read from remote repository." https://gitlab.haskell.org/ghc/ghc/-/jobs/378487 -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Mon Jun 29 14:14:55 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Mon, 29 Jun 2020 15:14:55 +0100 Subject: perf regression with TypeFamilies Message-ID: Hi Harendra, I saw your comment on a ghc proposal (https://github.com/ghc-proposals/ghc-proposals/pull/343#issuecomment-650797297) that said you experienced perf regressions with -XTypeFamilies enabled (but no other changes). Are these reproducible in a way that can be shared publicly? And are the regressions in compile times or run times? -XTypeFamilies enables -XMonoLocalBinds, which disables let-generalization on some nested lets. It is thus just barely conceivable that different Core is produced depending on this extension, and that there may be a possibility of performance changes. But this would be unexpected, and something worth investigating. In other words: if you can, do please post a bug -- simply enabling -XTypeFamilies should not slow anything down! Thanks, Richard From harendra.kumar at gmail.com Tue Jun 30 12:55:45 2020 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Tue, 30 Jun 2020 18:25:45 +0530 Subject: perf regression with TypeFamilies In-Reply-To: References: Message-ID: Hi Richard, I am glad that you are interested in it. It is the runtime performance that degrades, and yes it is fully reproducible and publicly shareable, the library is open source so anyone can run these benchmarks. I had postponed the problem raising an issue to investigate it further here https://github.com/composewell/streamly/issues/567 . Since you are interested I will investigate it sooner. I have added some perf numbers in that issue. I was also surprised by it, thinking what can type families possibly do to degrade run time performance like that. I can try compiling one worst affected benchmark code and look at the core-2-core passes to see where it makes the difference. -harendra On Mon, 29 Jun 2020 at 19:45, Richard Eisenberg wrote: > Hi Harendra, > > I saw your comment on a ghc proposal ( > https://github.com/ghc-proposals/ghc-proposals/pull/343#issuecomment-650797297) > that said you experienced perf regressions with -XTypeFamilies enabled (but > no other changes). Are these reproducible in a way that can be shared > publicly? And are the regressions in compile times or run times? > > -XTypeFamilies enables -XMonoLocalBinds, which disables let-generalization > on some nested lets. It is thus just barely conceivable that different Core > is produced depending on this extension, and that there may be a > possibility of performance changes. But this would be unexpected, and > something worth investigating. > > In other words: if you can, do please post a bug -- simply enabling > -XTypeFamilies should not slow anything down! > > Thanks, > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: