From stefansf at linux.ibm.com Mon Feb 1 11:19:02 2021 From: stefansf at linux.ibm.com (Stefan Schulze Frielinghaus) Date: Mon, 1 Feb 2021 12:19:02 +0100 Subject: Storage layout of integral types In-Reply-To: <20210119163403.GA256775@localhost.localdomain> References: <20210119163403.GA256775@localhost.localdomain> Message-ID: <20210201111902.GA5655@localhost.localdomain> On Tue, Jan 19, 2021 at 05:34:04PM +0100, Stefan Schulze Frielinghaus wrote: > Hi all, > > I'm wondering what the supposed storage layout of integral types is. In > particular for integral types with size less than the size of a word. For > example, on a 64bit machine is a 32bit integer supposed to be written as a > whole word and therefore as 64 bits or just as 32bits in the payload of a > closure? > > I'm asking because since commit be5d74ca I see differently aligned integers in > the payload of a closure on a 64bit big-endian machine. For example, in the > following code an Int32 object is created which contains the actual integer in > the high part of the payload (the snippet comes from the add operator > GHC.Int.$fNumInt32_$c+_entry): > > Hp = Hp + 16; > ... > I64[Hp - 8] = GHC.Int.I32#_con_info; > I32[Hp] = _scz7::I32; > > whereas e.g. in function rts_getInt32 the opposite is assumed and the actual > integer is expected in the low part of the payload: > > HsInt32 > rts_getInt32 (HaskellObj p) > { > // See comment above: > // ASSERT(p->header.info == I32zh_con_info || > // p->header.info == I32zh_static_info); > return (HsInt32)(HsInt)(UNTAG_CLOSURE(p)->payload[0]); > } > > The same seems to be the case for the interpreter and foreign calls (case > bci_CCALL) where integral arguments are passed in the low part of a whole word. > > Currently, my intuition is that the payload of a closure for an integral type > with size smaller than WordSize is written as a whole word where the subword is > aligned according to the machines endianness. Can someone confirm this? If > that is indeed true, then rts_getInt32 seems to be correct but not the former. > Otherwise the converse seems to be the case. Interestingly it looks like as if 32bit floats are also only accessed as 32bit values, whereas for characters we have that a 32bit subword is supposed to be accessed as a 64bit word: section ""data" . stg_CHARLIKE_closure" { stg_CHARLIKE_closure: const ghczmprim_GHCziTypes_Czh_con_info; const 0; ... Thus in total what I see is that the payload of a closure for I{8,16,32}# and F# are read/written according to their natural size, respectively (ignoring the inconsistency in rts_getInt* for a moment). In contrast to the payload of a C# closure which is read/written as a 64bit value although the natural size is 32bit only. Can someone confirm these observations? What is the general direction: are subwords supposed to be read/written according to their natural size only? Cheers, Stefan From csaba.hruska at gmail.com Mon Feb 1 21:56:01 2021 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Mon, 1 Feb 2021 22:56:01 +0100 Subject: remAddr# primop inconsistency Message-ID: Hello, I've discovered that the remAddr# primop documentation is inconsistent with the implementation. According to the docs (primops.txt.pp) the remAddr# should be the same as remInt#. > primop AddrRemOp "remAddr#" GenPrimOp Addr# -> Int# -> Int# {Return the remainder when the {\tt Addr\#} arg, treated like an {\tt Int\#}, is divided by the {\tt Int\#} arg.} > > But it is implemented with the mo_wordURem instruction, just like the WordRemOp primop. > GHC/StgToCmm/Prim.hs: > AddrRemOp -> \args -> opTranslate args (mo_wordURem platform) > WordRemOp -> \args -> opTranslate args (mo_wordURem platform) > IntRemOp -> \args -> opTranslate args (mo_wordSRem platform) > Which one is correct, the docs or the implementation? Regards, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Tue Feb 2 15:30:11 2021 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Tue, 2 Feb 2021 15:30:11 +0000 Subject: Inspecting function arguments in GHCi In-Reply-To: <615845bc-571d-15aa-80c8-fce4b03b3bb5@seznam.cz> References: <0bd42dc9-87b7-7d8f-2c22-7b60599cc55e@seznam.cz> <87a6sv1zw0.fsf@smart-cactus.org> <615845bc-571d-15aa-80c8-fce4b03b3bb5@seznam.cz> Message-ID: Hi Andrew, I updated the README for ghc-debug last week but forget to send this mail! https://gitlab.haskell.org/ghc/ghc-debug Cheers, Matt On Wed, Jan 27, 2021 at 10:15 PM Andrew Kvapil wrote: > > Hello Ben, > > Thanks for your suggestions. The decision to adapt GHCi came out of a > discussion with my supervisor and his colleagues. At this point the > entire set of desired capabilities of the work is still unknown, however > we do consider the GHCi-compatible programs to represent a large enough > set for future analysis, and the ease of mapping breakpoints back to > source code is a significant benefit. > > I do plan on using the ghc-heap-view (I assume that's what you meant by > ghc-heap, or is there another library I don't know about?) logic in the > project, although I'm currently more focused on implementing the proper > hook mechanism. I expect that events of deeply nested thunks being > forced will be quite important. The possibility of tracking control > flow via breakpoints/tracepoints also seems appealing. I'm not aware > of any existing solutions which would allow dynamic tracing, although > it's very well possible I didn't look hard enough. > > Regarding ghc-debug, I'm not sure what kinds of trade-offs it offers > compared to the approach I'm currently taking. It looks like it's a > fairly newborn project, do you think it's mature enough for the > proposed use cases? I couldn't find docs online, although I did come > across [0] which Discourse[1] says is related. I've yet to watch the > introduction video. Support for unboxed tuples and other features not > supported by GHCi would of course be nice, although performance is not > a concern. Keeping the relationship between source code spans and > heap objects in the infotables is an intriguing idea. > > > Note that Luite's recent work on refactoring the bytecode generator to > > > produce code from STG is quite relevant here. In particular, you will > > > likely want to look at !4589 [1], which does the work of refactoring > > > Tickish to follow the Trees That Grow pattern. You would likely want to > > > do the same to capture your free variable information. > > > Excellent, I was not aware of this. Thank you! > > > Regards, > Andrew > > > > [0]: https://well-typed.com/blog/2021/01/first-look-at-hi-profiling-mode/ > [1]: > https://discourse.haskell.org/t/an-introduction-to-ghc-debug-precise-memory-analysis-for-haskell-programs/1771 > > > On 26/01/2021 16:28, Ben Gamari wrote: > > Andrew Kvapil writes: > > > >> Hello, > >> > >> I'm interested in inspecting the strictness of functions at runtime > >> and the depth of thunks "in the wild." > > > > Hi Andrew, > > > > Interesting. When I first read your introduction my first thought was to > > rather walk the "normal" heap using ghc-heap or, perhaps, the relatively > > new ghc-debug library [1] rather than introduce this feature in GHCi. > > It's hard to know whether non-GHCi-based approach is viable without > > knowing more about what you are doing, but it potentially brings the > > benefit of generality: your analysis is not limited to programs with can > > be run under GHCi. > > > > Of course, it also brings some challenges: You would need to find a way > > to associate info table symbols with whatever information you need of > > from the Core program and in the presence of simplification tying your > > results back to the source program may not be possible at all. > > > > Cheers, > > > > - Ben > > > > > > [1] https://gitlab.haskell.org/ghc/ghc-debug > > > >> For this reason I'm modifying > >> GHC 8.10.2, essentially to add additional information to breakpoints. > >> I'd like to reuse the logic behind GHCi's :print command > >> (pprintClosureCommand, obtainTermFromId, ...) for which I suppose I > >> need Id's. Those however don't exist for destructuring patterns, such > >> as those in the following equations: > >> > >> last [x] = x > >> last (_:xs) = last xs > >> > >> So I'm wondering where would be a good place in the pipeline to > >> transform patterns like these into at-patterns, to give them Id's. > >> However, the breakpoint logic only looks at the free variables of the > >> right-hand sides and not transitively, which means that e.g. in the > >> following example neither ':print arg1' nor ':print as' works when the > >> interpreter hits a breakpoint in the top level expression on the RHS: > >> > >> qsort arg1@(a:as) = qsort left ++ [a] ++ qsort right > >> where (left, right) = (filter (<=a) as, filter (>a) as) > >> > >> Thus I'd also like to know how to extend the free var logic for > >> Tickish that eventually leads to CgBreakInfo and :print's ability to > >> inspect these bindings at runtime. My goal would be to determine to what > >> extent was a thunk evaluated during function application. > >> > > Note that Luite's recent work on refactoring the bytecode generator to > > produce code from STG is quite relevant here. In particular, you will > > likely want to look at !4589 [1], which does the work of refactoring > > Tickish to follow the Trees That Grow pattern. You would likely want to > > do the same to capture your free variable information. > > > > Cheers, > > > > - Ben > > > > > > [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4589 > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From simonpj at microsoft.com Thu Feb 4 00:03:27 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 4 Feb 2021 00:03:27 +0000 Subject: Happy version downgrade Message-ID: Friends I can't build ghc-9.0 because of this: checking for ghc-pkg matching /opt/ghc/bin/ghc... /opt/ghc/bin/ghc-pkg checking for happy... /home/simonpj/.cabal/bin/happy checking for version of happy... 1.20.0 configure: error: Happy version 1.19 is required to compile GHC. What is the easiest way to fix? I have happy 1.19.8, but it's not called plain "happy" bash$ happy-1.19.8 --version Happy Version 1.19.8 Copyright (c) 1993-1996 Andy Gill, Simon Marlow (c) 1997-2005 Simon Marlow Happy is a Yacc for Haskell, and comes with ABSOLUTELY NO WARRANTY. This program is free software; you can redistribute it and/or modify it under the terms given in the file 'LICENSE' distributed with the Happy sources. Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Thu Feb 4 01:33:52 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Thu, 4 Feb 2021 01:33:52 +0000 Subject: Happy version downgrade In-Reply-To: References: Message-ID: <010f01776aac9aaf-f258f0e8-2144-4124-a260-c72a8cd8897b-000000@us-east-2.amazonses.com> If you say `./configure --help`, you'll get all the flags to ./configure. Toward the bottom, it says that the HAPPY environment variable can contain the path to the happy executable. So try > HAPPY=happy-1.19.8 ./configure and see where you get. You might need the full path to your installed happy-1.19.8. It might also be possible to pass --with-happy=/full/path/to/happy-19.8 as an argument. Richard > On Feb 3, 2021, at 7:03 PM, Simon Peyton Jones via ghc-devs wrote: > > Friends > > I can’t build ghc-9.0 because of this: > > checking for ghc-pkg matching /opt/ghc/bin/ghc... /opt/ghc/bin/ghc-pkg > checking for happy... /home/simonpj/.cabal/bin/happy > checking for version of happy... 1.20.0 > configure: error: Happy version 1.19 is required to compile GHC. > What is the easiest way to fix? I have happy 1.19.8, but it’s not called plain “happy” > > bash$ happy-1.19.8 --version > Happy Version 1.19.8 Copyright (c) 1993-1996 Andy Gill, Simon Marlow (c) 1997-2005 Simon Marlow > Happy is a Yacc for Haskell, and comes with ABSOLUTELY NO WARRANTY. > This program is free software; you can redistribute it and/or modify > it under the terms given in the file 'LICENSE' distributed with > the Happy sources. > Thanks > > Simon > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshmeredith2008 at gmail.com Thu Feb 4 15:36:39 2021 From: joshmeredith2008 at gmail.com (Josh Meredith) Date: Fri, 5 Feb 2021 02:36:39 +1100 Subject: initIface* usage in plugins Message-ID: Hi, I'm trying to use initIfaceLoad and initIfaceLcl to lookup core `Name`s within a plugin. My understanding is that functions such as `lookupIfaceTop` are used for this, but I'm running into an issue that I suspect is caused by some `Name`s being inappropriate for this function, so they fail with the error `Iface id out of scope: ...`. Is there a robust way to select which `Name` lookup function to use based on a core binding/expression? Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu Feb 4 18:02:53 2021 From: ben at well-typed.com (Ben Gamari) Date: Thu, 04 Feb 2021 13:02:53 -0500 Subject: [ANNOUNCE] GHC 9.0.1 released Message-ID: <87wnvnsoc5.fsf@smart-cactus.org> The GHC team is very pleased to announce the availability of GHC 9.0.1. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/9.0.1/ In addition to numerous bug fixes, GHC 9.0.1 will bring a number of new features: * A first cut of the new LinearTypes language extension [2], allowing use of linear function syntax and linear record fields. * A new bignum library, ghc-bignum, improving portability and allowing GHC to be more easily used with integer libraries other than GMP. * Improvements in code generation, resulting in considerable runtime performance improvements in some programs. * Improvements in pattern-match checking, allowing more precise detection of redundant cases and reduced compilation time. * Implementation of the "simplified subsumption" proposal [3] simplifying the type system and paving the way for QuickLook impredicativity in GHC 9.2. * Implementation of the QualifiedDo extension [4], allowing more convenient overloading of `do` syntax. * An experimental new IO manager implementation for Windows platforms, both improving performance and fixing many of the quirks with the old manager built on POSIX-emulation. * Improvements in compilation time. And many more. See the release notes [5] for a full accounting of the changes in this release. As always, feel free to report any issues you encounter via gitlab.haskell.org. Cheers, - Ben [1]: https://gitlab.haskell.org/ghc/ghc/-/wikis/migration/9.0#ghc-prim-07 [2]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0111-linear-types.rst [3]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0287-simplify-subsumption.rst [4]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0216-qualified-do.rst [5]: https://downloads.haskell.org/ghc/9.0.1/docs/html/users_guide/9.0.1-notes.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Thu Feb 4 18:56:07 2021 From: ben at well-typed.com (Ben Gamari) Date: Thu, 04 Feb 2021 13:56:07 -0500 Subject: Plan for GHC 9.2 Message-ID: <87v9b7slvf.fsf@smart-cactus.org> tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, especially if you have something you would like merged for 9.2.1. Hello all, With GHC 9.0.1 at long-last out the door, it is time that we start turning attention to GHC 9.2. I would like to avoid making the mistake made in the 9.0 series in starting the fork in a state that required a significant amount of backporting to be releaseable. Consequently, I want to make sure that we have a fork schedule that is realistic given the things that need to be merged for 9.2. These include: * Update haddock submodule in `master` (Ben) * Bumping bytestring to 0.11 (#19091, Ben) * Finishing the rework of sized integer primops (#19026, John Ericson) * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) * Merge BoxedRep (#17526, Ben) * ARM NCG backend and further stabilize Apple ARM support? (Moritz) * Some form of coercion zapping (Ben, Simon, Richard) * Tag inference analysis and tag check elision (Andreas) If you see something that you would like to see in 9.2.1 please do holler. Otherwise, if you see your name in this list it would be great if you could let me know when you think your project may be in a mergeable state. Ideally we would strive for a schedule like the following: 4 February 2021: We are here ~4 weeks pass 3 March 2021: Release branch forked 1 week passes 10 March 2021: Alpha 1 released 3 weeks pass 31 March 2021: Alpha 2 released 2 weeks pass 14 April 2021: Alpha 3 released 2 weeks pass 28 April 2021: Alpha 4 released 1 week passes 5 May 2021: Beta 1 released 1 week passes 12 May 2021: Release candidate 1 released 2 weeks pass 26 May 2021: Final release This provides ample time for stabilization while avoiding deviation from the usual May release timeframe. However, this would require that we move aggressively to start getting the tree into shape since the fork would be less than four weeks away. I would appreciate contributors' thoughts on the viability of this timeline. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From sgraf1337 at gmail.com Thu Feb 4 19:10:53 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Thu, 4 Feb 2021 20:10:53 +0100 Subject: Plan for GHC 9.2 In-Reply-To: <87v9b7slvf.fsf@smart-cactus.org> References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: Hi Ben, Since part of the changes of https://gitlab.haskell.org/ghc/ghc/-/issues/14422 are already merged into master (e.g. we ignore the "type signature" part of a COMPLETE sig now, because there is nothing to disambiguate), it would be good if we merged the solution outlined in https://gitlab.haskell.org/ghc/ghc/-/issues/14422#note_321645, as that would allow users to switch to a new, better mechanism instead of discovering that COMPLETE signatures seemingly have been ripped of a feature. The problem with that is that it needs a GHC proposal, I think, and that's not written yet. Also I hope to merge some efforts in the CPR area before the fork. But that's quite optional. Cheers, Sebastian Am Do., 4. Feb. 2021 um 19:56 Uhr schrieb Ben Gamari : > > tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, > especially if you have something you would like merged for 9.2.1. > > Hello all, > > With GHC 9.0.1 at long-last out the door, it is time that we start > turning attention to GHC 9.2. I would like to avoid making the mistake > made in the 9.0 series in starting the fork in a state that required a > significant amount of backporting to be releaseable. Consequently, I > want to make sure that we have a fork schedule that is realistic given > the things that need to be merged for 9.2. These include: > > * Update haddock submodule in `master` (Ben) > * Bumping bytestring to 0.11 (#19091, Ben) > * Finishing the rework of sized integer primops (#19026, John Ericson) > * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) > * Merge BoxedRep (#17526, Ben) > * ARM NCG backend and further stabilize Apple ARM support? (Moritz) > * Some form of coercion zapping (Ben, Simon, Richard) > * Tag inference analysis and tag check elision (Andreas) > > If you see something that you would like to see in 9.2.1 please do > holler. Otherwise, if you see your name in this list it would be great > if you could let me know when you think your project may be in a > mergeable state. > > Ideally we would strive for a schedule like the following: > > 4 February 2021: We are here > ~4 weeks pass > 3 March 2021: Release branch forked > 1 week passes > 10 March 2021: Alpha 1 released > 3 weeks pass > 31 March 2021: Alpha 2 released > 2 weeks pass > 14 April 2021: Alpha 3 released > 2 weeks pass > 28 April 2021: Alpha 4 released > 1 week passes > 5 May 2021: Beta 1 released > 1 week passes > 12 May 2021: Release candidate 1 released > 2 weeks pass > 26 May 2021: Final release > > This provides ample time for stabilization while avoiding deviation from > the usual May release timeframe. However, this would require that we > move aggressively to start getting the tree into shape since the fork > would be less than four weeks away. I would appreciate contributors' > thoughts on the viability of this timeline. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at joachim-breitner.de Thu Feb 4 20:15:44 2021 From: mail at joachim-breitner.de (Joachim Breitner) Date: Thu, 04 Feb 2021 21:15:44 +0100 Subject: Plan for GHC 9.2 In-Reply-To: <87v9b7slvf.fsf@smart-cactus.org> References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: <7bd32d112662bbb42f5c036b05b864ecf30dab61.camel@joachim-breitner.de> Hi, Am Donnerstag, den 04.02.2021, 13:56 -0500 schrieb Ben Gamari: > If you see something that you would like to see in 9.2.1 please do > holler. it’s hopefully not big deal technically, but support for GHC2021 would be desirable. There is a MR https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4853 that “just” needs chaising test suite failures when rebasing on latest master (and I’d be grateful if someone more fluent with today’s GHC development than me would take the MR over at this point) Cheers, Joachim -- Joachim Breitner mail at joachim-breitner.de http://www.joachim-breitner.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From mniip at typeable.io Fri Feb 5 00:51:39 2021 From: mniip at typeable.io (Igor Popov) Date: Fri, 5 Feb 2021 03:51:39 +0300 Subject: Newtype over (~#) Message-ID: Hello list! Recently I've had this idea of letting Haskell source handle unboxed equalities (~#) by the means of an unboxed newtype. The idea is pretty straightforward: in Core Constraint is Type and (=>) is (->), so a datatype like data Eq a b = (a ~ b) => Refl could become a newtype because it only has a single "field": (a ~ b). Furthermore now that we have unlifted newtypes, we could write it as a newtype over (~#), of kind TYPE (TupleRep []). Defining such a datatype is of course impossible in source Haskell, but what I came up with is declaring a plugin that magically injects the necessary bits into the interface file. Sounds like it should be straightforward: define a newtype (:~:#) with a constructor whose representation type is: forall k (a :: k) (b :: k). (a ~# b) -> a :~:# b The worker constructor is implemented by a coercion (can even be eta-reduced): axiom N::~:# :: forall {k}. (:~:#) = (~#) Refl# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (v :: a ~# b) -> v `cast` (Sym (N::~:#) _N) _N _N And the wrapper constructor has a Haskell-ish type: $WRefl# :: forall k (b :: k). b :~:# b $WRefl# = \ (@ k) (@ (b :: k)) -> Refl# @ k @ a @ b @~ _N Caveat: we don't actually get to specify the unwrappings ourselves, and we have to rely on mkDataConRep generating the right wrapper based on the types and the EqSpec (because this will have to be done every time the constructor is loaded from an iface). In this case the machinery is not convinced that a wrapper is required, unless we ensure that dataConUserTyVarsArePermuted by fiddling around with ordering of variables. This is a minor issue (which I think I can work around) but could be addressed on the GHC side. I've indeed implemented a plugin that declares these, but we run into a more major issue: the simplifier is not ready for such code! Consider a simple utility function: sym# :: a :~:# b -> b :~:# a sym# Refl# = Refl# This gets compiled into: sym# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (ds :: a :~:# b) -> case ds `cast` N::~:# _N _N _N of co -> $WRefl# @ k @ b `cast` ((:~:#) _N (Sym co))_R which seems valid but then the simplifier incorrectly inlines the unfolding of $WRefl# and arrives at code that is not even well-scoped (-dcore-lint catches this): sym# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (ds :: a :~:# b) -> case ds `cast` N::~:# _N _N _N of co -> v `cast` (Sym (N::~: _N)) _N _N ; ((:~:#) _N _N (Sym co))_R Actually the problem manifests itself even earlier: when creating an unfolding for the wrapper constructor with mkCompulsoryUnfolding we run the unfolding term through simpleOptExpr once before memorizing the unfolding, and this produces an unfolding for $WRefl# that is broken (ill-scoped) in a similar fashion: $WRefl = \ (@ k) (@ (b :: k)) -> v `cast` Sym (N::~:# _N) _N _N And we can verify that the issue is localized here: applying simpleOptExpr to: (\ (v :: a ~# a) -> v `cast` _R) @~ _N results in: v The former term is closed and the latter is not. There is an invariant on mkPrimEqPred (though oddly not on mkReprPrimEqPred) that says that the related types must not be coercions (we're kind of violating this here). I have several questions here: - Is there a good reason for the restriction that equalities should not contain other equalities? - Should this use case be supported? Coercions are almost first class citizens in Core (are they?), makes sense to let source Haskell have a bit of that? - Does it make sense to include this and a few similar types (unboxed Coercion and Dict) as a wired in type packaged with GHC in some form? -- mniip -------------- next part -------------- An HTML attachment was scrubbed... URL: From ekmett at gmail.com Fri Feb 5 02:40:40 2021 From: ekmett at gmail.com (Edward Kmett) Date: Thu, 4 Feb 2021 18:40:40 -0800 Subject: Newtype over (~#) In-Reply-To: References: Message-ID: A similar unlifted constraint style newtype that would be very valuable to me would be to be able to talk about unlifted implicit parameters. type GivenFoo = (?foo :: Int#) (hopefully properly inhabiting TYPE 'IntRep) This would go a long way towards removing the last bit of overhead when using implicit parameters to automatically dispatch *just* the portions of the environment/state/etc. that you need to handle effect systems without incurring unnecessary boxes. Right now when I work with a custom Monad I can of course unbox the argument to by reader or state, but when I move it into an implicit parameter to get it automatically thinned down to just what portions of the environment I actually need, I lose that level of expressiveness. -Edward On Thu, Feb 4, 2021 at 4:52 PM Igor Popov wrote: > Hello list! > > Recently I've had this idea of letting Haskell source handle unboxed > equalities (~#) by the means of an unboxed newtype. The idea is pretty > straightforward: in Core Constraint is Type and (=>) is (->), so a > datatype like > > data Eq a b = (a ~ b) => Refl > > could become a newtype because it only has a single "field": (a ~ b). > Furthermore now that we have unlifted newtypes, we could write it as a > newtype over (~#), of kind TYPE (TupleRep []). > > Defining such a datatype is of course impossible in source Haskell, but > what I came up with is declaring a plugin that magically injects the > necessary bits into the interface file. > > Sounds like it should be straightforward: define a newtype (:~:#) with a > constructor whose representation type is: > > forall k (a :: k) (b :: k). (a ~# b) -> a :~:# b > > The worker constructor is implemented by a coercion (can even be > eta-reduced): > > axiom N::~:# :: forall {k}. (:~:#) = (~#) > Refl# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (v :: a ~# b) -> > v `cast` (Sym (N::~:#) _N) _N _N > > And the wrapper constructor has a Haskell-ish type: > > $WRefl# :: forall k (b :: k). b :~:# b > $WRefl# = \ (@ k) (@ (b :: k)) -> > Refl# @ k @ a @ b @~ _N > > Caveat: we don't actually get to specify the unwrappings ourselves, and > we have to rely on mkDataConRep generating the right wrapper based on > the types and the EqSpec (because this will have to be done every time > the constructor is loaded from an iface). In this case the machinery is > not convinced that a wrapper is required, unless we ensure that > dataConUserTyVarsArePermuted by fiddling around with ordering of > variables. This is a minor issue (which I think I can work around) but > could be addressed on the GHC side. > > I've indeed implemented a plugin that declares these, but we run into a > more major issue: the simplifier is not ready for such code! Consider a > simple utility function: > > sym# :: a :~:# b -> b :~:# a > sym# Refl# = Refl# > > This gets compiled into: > > sym# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (ds :: a :~:# b) -> > case ds `cast` N::~:# _N _N _N of > co -> $WRefl# @ k @ b `cast` ((:~:#) _N (Sym co))_R > > which seems valid but then the simplifier incorrectly inlines the > unfolding of $WRefl# and arrives at code that is not even well-scoped > (-dcore-lint catches this): > > sym# = \ (@ k) (@ (a :: k)) (@ (b :: k)) (ds :: a :~:# b) -> > case ds `cast` N::~:# _N _N _N of > co -> v `cast` (Sym (N::~: _N)) _N _N > ; ((:~:#) _N _N (Sym co))_R > > Actually the problem manifests itself even earlier: when creating an > unfolding for the wrapper constructor with mkCompulsoryUnfolding we run > the unfolding term through simpleOptExpr once before memorizing the > unfolding, and this produces an unfolding for $WRefl# that is broken > (ill-scoped) in a similar fashion: > > $WRefl = \ (@ k) (@ (b :: k)) -> > v `cast` Sym (N::~:# _N) _N _N > > And we can verify that the issue is localized here: applying > simpleOptExpr to: > > (\ (v :: a ~# a) -> v `cast` _R) > @~ _N > > results in: > > v > > The former term is closed and the latter is not. > > There is an invariant on mkPrimEqPred (though oddly not on > mkReprPrimEqPred) that says that the related types must not be > coercions (we're kind of violating this here). > > I have several questions here: > - Is there a good reason for the restriction that equalities should not > contain other equalities? > - Should this use case be supported? Coercions are almost first class > citizens in Core (are they?), makes sense to let source Haskell have a > bit of that? > - Does it make sense to include this and a few similar types (unboxed > Coercion and Dict) as a wired in type packaged with GHC in some form? > > -- mniip > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takenobu.hs at gmail.com Fri Feb 5 12:05:19 2021 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Fri, 5 Feb 2021 21:05:19 +0900 Subject: GHC Reading Guide Message-ID: Dear devs, I've written a simple document about "GHC source reading" for myself and potential newcomers: * https://takenobu-hs.github.io/downloads/haskell_ghc_reading_guide.pdf (https://github.com/takenobu-hs/haskell-ghc-reading-guide) Please teach me if something's wrong. I'll learn and correct them. Happy Haskelling :) Regards, Takenobu From rae at richarde.dev Fri Feb 5 15:08:33 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Fri, 5 Feb 2021 15:08:33 +0000 Subject: GHC Reading Guide In-Reply-To: References: Message-ID: <010f017772bcd304-517d2972-a9af-4486-b168-cb0ee39301c0-000000@us-east-2.amazonses.com> A quick scroll through didn't reveal anything incorrect in my areas of expertise. On the "Dump intermediate languages" slide, you might want to mention these flags: -fprint-explicit-kinds: print out kind applications -fprint-explicit-coercions: print out details of coercions -fprint-typechecker-elaboration: print out extra gubbins the type-checker inserts -fprint-explicit-runtime-reps: don't simplify away RuntimeRep arguments -ddump-ds-preopt: print out the desugared Core before the very first "simple" optimization pass Thanks for writing & sharing! Richard > On Feb 5, 2021, at 7:05 AM, Takenobu Tani wrote: > > Dear devs, > > I've written a simple document about "GHC source reading" for myself > and potential newcomers: > > * https://takenobu-hs.github.io/downloads/haskell_ghc_reading_guide.pdf > (https://github.com/takenobu-hs/haskell-ghc-reading-guide) > > Please teach me if something's wrong. I'll learn and correct them. > > Happy Haskelling :) > > Regards, > Takenobu > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From sgraf1337 at gmail.com Fri Feb 5 15:29:04 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Fri, 5 Feb 2021 16:29:04 +0100 Subject: GHC Reading Guide In-Reply-To: <010f017772bcd304-517d2972-a9af-4486-b168-cb0ee39301c0-000000@us-east-2.amazonses.com> References: <010f017772bcd304-517d2972-a9af-4486-b168-cb0ee39301c0-000000@us-east-2.amazonses.com> Message-ID: Hi Takenobu, thanks for updating that resource! I know that it was helpful to a couple of students of mine to get a big picture of GHC. I don't have anything to add. There are quite a few more -ddump-* flags for the different Core passes, but I don't think it's interesting to list all of them on that slide. Greetings, Sebastian Am Fr., 5. Feb. 2021 um 16:09 Uhr schrieb Richard Eisenberg < rae at richarde.dev>: > A quick scroll through didn't reveal anything incorrect in my areas of > expertise. > > On the "Dump intermediate languages" slide, you might want to mention > these flags: > -fprint-explicit-kinds: print out kind applications > -fprint-explicit-coercions: print out details of coercions > -fprint-typechecker-elaboration: print out extra gubbins the type-checker > inserts > -fprint-explicit-runtime-reps: don't simplify away RuntimeRep arguments > > -ddump-ds-preopt: print out the desugared Core before the very first > "simple" optimization pass > > Thanks for writing & sharing! > Richard > > > On Feb 5, 2021, at 7:05 AM, Takenobu Tani wrote: > > > > Dear devs, > > > > I've written a simple document about "GHC source reading" for myself > > and potential newcomers: > > > > * https://takenobu-hs.github.io/downloads/haskell_ghc_reading_guide.pdf > > (https://github.com/takenobu-hs/haskell-ghc-reading-guide) > > > > Please teach me if something's wrong. I'll learn and correct them. > > > > Happy Haskelling :) > > > > Regards, > > Takenobu > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Fri Feb 5 20:36:37 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Fri, 5 Feb 2021 20:36:37 +0000 Subject: Newtype over (~#) In-Reply-To: References: Message-ID: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> GHC cheats in this area. The problem is that (a ~# b) is a type, because that is terribly, terribly convenient. But it really shouldn't be. The problem is that coercion variables are different from regular variables. So, if we say (v :: a ~# b), we must always be careful: is v a coercion variable or not? If it isn't, then v is useless. You cannot, for instance, use v in a cast. In a number of places, GHC must produce a variable binder of a certain type. It will sometimes examine that type; if the type is an equality type, then the binder will be a coercion variable; otherwise, it will be a normal Id. This is wrong, but again, very convenient. (Wrong because we should always know in advance whether we want a coercion variable or an Id.) I think the problems you're running into are all around this confusion. For example, (\ (v :: a ~# a) -> v `cast` _R) @~ _N is ill-formed, whether v is an Id or a CoVar. If it's an Id, then this is wrong because you have an Id with a coercion type. (The theory of the core language does not prevent such things, but I'm not surprised that GHC would fall over if you tried it.) If v is a CoVar, then its use as the left-hand side of `cast` is wrong. This is all https://gitlab.haskell.org/ghc/ghc/-/issues/17291 . All that said, I think your original idea is a fine one: a newtype wrapper around ~#. It's possible there is just some Id/CoVar confusion in your implementation, and that's what's causing the trouble. I don't see a correct implementation as impossible. Is your implementation available somewhere? I hope this helps! Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From takenobu.hs at gmail.com Sat Feb 6 00:34:43 2021 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Sat, 6 Feb 2021 09:34:43 +0900 Subject: GHC Reading Guide In-Reply-To: References: <010f017772bcd304-517d2972-a9af-4486-b168-cb0ee39301c0-000000@us-east-2.amazonses.com> Message-ID: Hi Richard, Sebastian Thanks for your comments and advice! I'll try to update dump flags' pages :) Thanks a lot, Takenobu On Sat, Feb 6, 2021 at 12:29 AM Sebastian Graf wrote: > > Hi Takenobu, > > thanks for updating that resource! I know that it was helpful to a couple of students of mine to get a big picture of GHC. > > I don't have anything to add. There are quite a few more -ddump-* flags for the different Core passes, but I don't think it's interesting to list all of them on that slide. > > Greetings, > Sebastian > > Am Fr., 5. Feb. 2021 um 16:09 Uhr schrieb Richard Eisenberg : >> >> A quick scroll through didn't reveal anything incorrect in my areas of expertise. >> >> On the "Dump intermediate languages" slide, you might want to mention these flags: >> -fprint-explicit-kinds: print out kind applications >> -fprint-explicit-coercions: print out details of coercions >> -fprint-typechecker-elaboration: print out extra gubbins the type-checker inserts >> -fprint-explicit-runtime-reps: don't simplify away RuntimeRep arguments >> >> -ddump-ds-preopt: print out the desugared Core before the very first "simple" optimization pass >> >> Thanks for writing & sharing! >> Richard >> >> > On Feb 5, 2021, at 7:05 AM, Takenobu Tani wrote: >> > >> > Dear devs, >> > >> > I've written a simple document about "GHC source reading" for myself >> > and potential newcomers: >> > >> > * https://takenobu-hs.github.io/downloads/haskell_ghc_reading_guide.pdf >> > (https://github.com/takenobu-hs/haskell-ghc-reading-guide) >> > >> > Please teach me if something's wrong. I'll learn and correct them. >> > >> > Happy Haskelling :) >> > >> > Regards, >> > Takenobu >> > _______________________________________________ >> > ghc-devs mailing list >> > ghc-devs at haskell.org >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From mniip at typeable.io Sat Feb 6 00:56:16 2021 From: mniip at typeable.io (Igor Popov) Date: Sat, 6 Feb 2021 03:56:16 +0300 Subject: Newtype over (~#) In-Reply-To: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> References: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> Message-ID: > GHC cheats in this area. The problem is that (a ~# b) is a type, because that is terribly, terribly convenient. But it really shouldn't be. > > The problem is that coercion variables are different from regular variables. So, if we say (v :: a ~# b), we must always be careful: is v a coercion variable or not? If it isn't, then v is useless. You cannot, for instance, use v in a cast. I don't really see a problem here. The fact that only a "coercion variable" can be used in a cast should be enforced by the typing rule for cast. That doesn't require having a distinct "syntactic category" of coercion variables, unless I'm missing something. -- mniip From ben at well-typed.com Sat Feb 6 16:04:28 2021 From: ben at well-typed.com (Ben Gamari) Date: Sat, 06 Feb 2021 11:04:28 -0500 Subject: remAddr# primop inconsistency In-Reply-To: References: Message-ID: <875z3541yu.fsf@smart-cactus.org> Csaba Hruska writes: > Hello, > > I've discovered that the remAddr# primop documentation is inconsistent with > the implementation. > According to the docs (primops.txt.pp) the remAddr# should be the same as > remInt#. > >> primop AddrRemOp "remAddr#" GenPrimOp Addr# -> Int# -> Int# {Return the remainder when the {\tt Addr\#} arg, treated like an {\tt Int\#}, is divided by the {\tt Int\#} arg.} >> >> But it is implemented with the mo_wordURem instruction, just like the > WordRemOp primop. > >> GHC/StgToCmm/Prim.hs: >> > AddrRemOp -> \args -> opTranslate args (mo_wordURem platform) >> > WordRemOp -> \args -> opTranslate args (mo_wordURem platform) >> > IntRemOp -> \args -> opTranslate args (mo_wordSRem platform) >> > > Which one is correct, the docs or the implementation? That is a good question. The only user (in the GHC tree) is GHC.Ptr.alignPtr which appears to want the Word semantics (that is, it would break given a negative remainder). Moreover, my mental model of pointers is that for the purposes of arithmetic they are non-negative. Therefore, I belive that the documentation here is wrong. I have opened #19332 to track this. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Sat Feb 6 16:12:53 2021 From: ben at well-typed.com (Ben Gamari) Date: Sat, 06 Feb 2021 11:12:53 -0500 Subject: [ANNOUNCE] GHC 8.10.4 released Message-ID: <874kip41kq.fsf@smart-cactus.org> The GHC team is very pleased to announce the availability of GHC 8.10.4. Source and binary distributions are available at the usual place: https://downloads.haskell.org/ghc/8.10.4/ This is a small bug-fix release, fixing two bugs present in 8.10.3: * Fix a linker hang triggered by dynamic code loading on Windows (#19155) * Fix a crash caused by inappropriate garbage of heap-allocated data reachable from foreign exports (#19149) As always, feel free to report any issues you encounter via [gitlab.haskell.org](https://gitlab.haskell.org/ghc/ghc/-/issues/new). Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From csaba.hruska at gmail.com Sat Feb 6 16:29:10 2021 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Sat, 6 Feb 2021 17:29:10 +0100 Subject: remAddr# primop inconsistency In-Reply-To: <875z3541yu.fsf@smart-cactus.org> References: <875z3541yu.fsf@smart-cactus.org> Message-ID: Great, thanks! On Sat, Feb 6, 2021 at 5:04 PM Ben Gamari wrote: > Csaba Hruska writes: > > > Hello, > > > > I've discovered that the remAddr# primop documentation is inconsistent > with > > the implementation. > > According to the docs (primops.txt.pp) the remAddr# should be the same as > > remInt#. > > > >> primop AddrRemOp "remAddr#" GenPrimOp Addr# -> Int# -> Int# > {Return the remainder when the {\tt Addr\#} arg, treated like an {\tt > Int\#}, is divided by the {\tt Int\#} arg.} > >> > >> But it is implemented with the mo_wordURem instruction, just like the > > WordRemOp primop. > > > >> GHC/StgToCmm/Prim.hs: > >> > > AddrRemOp -> \args -> opTranslate args (mo_wordURem platform) > >> > > WordRemOp -> \args -> opTranslate args (mo_wordURem platform) > >> > > IntRemOp -> \args -> opTranslate args (mo_wordSRem platform) > >> > > > > Which one is correct, the docs or the implementation? > > That is a good question. The only user (in the GHC tree) is > GHC.Ptr.alignPtr which appears to want the Word semantics (that is, it > would break given a negative remainder). Moreover, my mental model of > pointers is that for the purposes of arithmetic they are non-negative. > Therefore, I belive that the documentation here is wrong. > > I have opened #19332 to track this. > > Cheers, > > - Ben > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Sun Feb 7 16:41:15 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Sun, 7 Feb 2021 16:41:15 +0000 Subject: Newtype over (~#) In-Reply-To: References: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> Message-ID: <010f01777d5e6882-b4657fe4-a37f-4d9d-8d7b-8f0f0c0123be-000000@us-east-2.amazonses.com> We do need a separate category, though: otherwise, we could cast by `f x`, where `f` is a non-terminating functions. We erase casts, so the function would never get called. Maybe if we required that casts are always by variables (essentially, A-normalize casts) we could avoid this? But then would we require A-normalization to build coercions from pieces (as opposed to calling functions)? It's unclear. I think the first versions of this idea didn't require the separate syntactic category, but all work for the past decade has. I think there are other ways, but the way GHC handles this now is somewhat poor, because of #17291. Richard > On Feb 5, 2021, at 7:56 PM, Igor Popov wrote: > >> GHC cheats in this area. The problem is that (a ~# b) is a type, because that is terribly, terribly convenient. But it really shouldn't be. >> >> The problem is that coercion variables are different from regular variables. So, if we say (v :: a ~# b), we must always be careful: is v a coercion variable or not? If it isn't, then v is useless. You cannot, for instance, use v in a cast. > > I don't really see a problem here. The fact that only a "coercion > variable" can be used in a cast should be enforced by the typing rule > for cast. That doesn't require having a distinct "syntactic category" > of coercion variables, unless I'm missing something. > > -- mniip From takenobu.hs at gmail.com Mon Feb 8 14:05:10 2021 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Mon, 8 Feb 2021 23:05:10 +0900 Subject: Has ghc-9.0 for windows changed to require installation? Message-ID: Hi devs, The ghc-binary for windows needs to `make install` since ghc-9.0 [1]. Is this an intended change? Previously, ghc-8.10.4 binary for windows [2] doesn't need to `make install`. We only expand the tar-file and then we can execute `bin/ghcii.sh`. [1]: https://downloads.haskell.org/ghc/9.0.1/ghc-9.0.1-x86_64-unknown-mingw32.tar.xz [2]: https://downloads.haskell.org/ghc/8.10.4/ghc-8.10.4-x86_64-unknown-mingw32.tar.xz Regards, Takenobu From moritz.angermann at gmail.com Mon Feb 8 14:42:54 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Mon, 8 Feb 2021 22:42:54 +0800 Subject: Has ghc-9.0 for windows changed to require installation? In-Reply-To: References: Message-ID: Thanks for flagging this. This would be the opposite direction of what I’ve been advocating for. That we get bindists for Linux and macOS that work by simply unpacking them. On Mon, 8 Feb 2021 at 10:05 PM, Takenobu Tani wrote: > Hi devs, > > The ghc-binary for windows needs to `make install` since ghc-9.0 [1]. > Is this an intended change? > > Previously, ghc-8.10.4 binary for windows [2] doesn't need to `make > install`. > We only expand the tar-file and then we can execute `bin/ghcii.sh`. > > [1]: > https://downloads.haskell.org/ghc/9.0.1/ghc-9.0.1-x86_64-unknown-mingw32.tar.xz > [2]: > https://downloads.haskell.org/ghc/8.10.4/ghc-8.10.4-x86_64-unknown-mingw32.tar.xz > > Regards, > Takenobu > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhpetersen at gmail.com Mon Feb 8 16:32:48 2021 From: juhpetersen at gmail.com (Jens Petersen) Date: Tue, 9 Feb 2021 00:32:48 +0800 Subject: [ANNOUNCE] GHC 9.0.1 released In-Reply-To: <87wnvnsoc5.fsf@smart-cactus.org> References: <87wnvnsoc5.fsf@smart-cactus.org> Message-ID: On Fri, 5 Feb 2021 at 02:04, Ben Gamari wrote: > The GHC team is very pleased to announce the availability of GHC 9.0.1. > This is now testable in Fedora with: sudo dnf --enablerepo=updates-testing-modular install ghc:9.0/default Thanks, Jens -------------- next part -------------- An HTML attachment was scrubbed... URL: From karel.gardas at centrum.cz Mon Feb 8 16:56:01 2021 From: karel.gardas at centrum.cz (Karel Gardas) Date: Mon, 8 Feb 2021 17:56:01 +0100 Subject: [ANNOUNCE] GHC 9.0.1 released In-Reply-To: References: <87wnvnsoc5.fsf@smart-cactus.org> Message-ID: Hi Jens, fedora newbie here. Not sure what I'm doing wrong, but it's not working on f33/amd64 here: [karel at localhost ~]$ sudo dnf update [sudo] password for karel: Last metadata expiration check: 0:00:33 ago on Mon 08 Feb 2021 05:53:17 PM CET. Dependencies resolved. Nothing to do. Complete! [karel at localhost ~]$ sudo dnf --enablerepo=updates-testing-modular install ghc:9.0/default Last metadata expiration check: 0:00:49 ago on Mon 08 Feb 2021 05:53:17 PM CET. No match for argument: ghc:9.0/default Error: Unable to find a match: ghc:9.0/default Thanks, Karel On 2/8/21 5:32 PM, Jens Petersen wrote: > On Fri, 5 Feb 2021 at 02:04, Ben Gamari > wrote: > > The GHC team is very pleased to announce the availability of GHC 9.0.1. > > > This is now testable in Fedora with: > > sudo dnf --enablerepo=updates-testing-modular install ghc:9.0/default > > Thanks, Jens > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > From power.walross at gmail.com Mon Feb 8 18:35:12 2021 From: power.walross at gmail.com (Fendor) Date: Mon, 8 Feb 2021 19:35:12 +0100 Subject: Find the ghc binary path from libdir Message-ID: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> Hello everyone! I want to find out the location of the `ghc` (and `ghc-pkg`) executable given the `$libdir` (obtained by `ghc --print-libdir`). Is there a platform independent way of doing that, which we can rely on? E.g. on windows, we found that the ghc executable's location is: * `$libdir/../bin/ghc` For NixOs (or when using ghcup), it works with: * `$libdir/../../bin/ghc` but I worry that this is a just an accident that is not guaranteed by the installation process. I also noticed there seems to be a binary in * `$libdir/bin/ghc` Is one of these ways documented to be guaranteed by the ghc installation process? (P. S. first time using a mailing list, hope I am doing it correctly) Best regards, Fendor From dxld at darkboxed.org Mon Feb 8 18:48:01 2021 From: dxld at darkboxed.org (Daniel =?utf-8?Q?Gr=C3=B6ber?=) Date: Mon, 8 Feb 2021 19:48:01 +0100 Subject: Find the ghc binary path from libdir In-Reply-To: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> References: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> Message-ID: <20210208184801.cs5bmgfbf64cceq3@darkboxed.org> Hi Fendor, On Mon, Feb 08, 2021 at 07:35:12PM +0100, Fendor wrote: > I want to find out the location of the `ghc` (and `ghc-pkg`) executable > given the `$libdir` (obtained by `ghc --print-libdir`). > Is there a platform independent way of doing that, which we can rely on? Are you aware of the `ghc --print-libdir` option? I think you'd be much better off going from ghc binary to libdir instead of the other way around. Could you explain why you're trying to do it your way? > (P. S. first time using a mailing list, hope I am doing it correctly) Seems fine to me :) --Daniel From power.walross at gmail.com Mon Feb 8 19:08:45 2021 From: power.walross at gmail.com (Fendor) Date: Mon, 8 Feb 2021 20:08:45 +0100 Subject: Find the ghc binary path from libdir In-Reply-To: <20210208184801.cs5bmgfbf64cceq3@darkboxed.org> References: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> <20210208184801.cs5bmgfbf64cceq3@darkboxed.org> Message-ID: <3112aaa0-1662-b926-a01a-0cd3dce55786@gmail.com> Hi, Well the purpose is a bit convoluted, but here it goes: Basically, we want to know which `ghc` version `cabal-install` is going to pick for compiling the project in a platform independent way without resorting to linking to `Cabal`. You can execute `cabal exec -- ghc` which launches the correct `ghc` version, but as far as we found out, there is no programmatic way of finding the path to the `ghc` executable itself (`cabal exec -- where ghc` or `cabal exec -- which ghc` fails on windows). What we really want is `cabal-install` to tell us, e.g. with `cabal show-build-info` which is a command not implemented yet[0]. As long as this command is not implemented, we have to monkey patch it in order to satisfy our goals. You can execute `cabal exec -- ghc --print-libdir`, which gives you the `$libdir`, which brings us back to the original question: can we find the `ghc` executable given `$libdir` in a platform independent way? [0] https://github.com/haskell/cabal/pull/6241 Best regards, Fendor On 2/8/21 7:48 PM, Daniel Gröber wrote: > Hi Fendor, > > On Mon, Feb 08, 2021 at 07:35:12PM +0100, Fendor wrote: >> I want to find out the location of the `ghc` (and `ghc-pkg`) executable >> given the `$libdir` (obtained by `ghc --print-libdir`). >> Is there a platform independent way of doing that, which we can rely on? > Are you aware of the `ghc --print-libdir` option? I think you'd be much > better off going from ghc binary to libdir instead of the other way around. > > Could you explain why you're trying to do it your way? > >> (P. S. first time using a mailing list, hope I am doing it correctly) > Seems fine to me :) > > --Daniel From lonetiger at gmail.com Mon Feb 8 19:24:45 2021 From: lonetiger at gmail.com (Phyx) Date: Mon, 8 Feb 2021 19:24:45 +0000 Subject: Has ghc-9.0 for windows changed to require installation? In-Reply-To: References: Message-ID: No, there's no change at all in the portability. This looks like a fallout from switching from Make to Hadrian. ghcii.sh was created as an artifact of make install. Hadrian seems to lack this step. Note that this script is nothing magical, it's just a hack around how signal handlers in Native windows and Cygwin processes are handled. I'm not entirely sure it's still needed with the new I/O manager, however that's not on yet by default.. Anyways support should probably be added to Hadrian. Kind regards, Tamar Sent from my Mobile On Mon, Feb 8, 2021, 14:43 Moritz Angermann wrote: > Thanks for flagging this. This would be the opposite direction of what > I’ve been advocating for. That we get bindists for Linux and macOS that > work by simply unpacking them. > > On Mon, 8 Feb 2021 at 10:05 PM, Takenobu Tani > wrote: > >> Hi devs, >> >> The ghc-binary for windows needs to `make install` since ghc-9.0 [1]. >> Is this an intended change? >> >> Previously, ghc-8.10.4 binary for windows [2] doesn't need to `make >> install`. >> We only expand the tar-file and then we can execute `bin/ghcii.sh`. >> >> [1]: >> https://downloads.haskell.org/ghc/9.0.1/ghc-9.0.1-x86_64-unknown-mingw32.tar.xz >> [2]: >> https://downloads.haskell.org/ghc/8.10.4/ghc-8.10.4-x86_64-unknown-mingw32.tar.xz >> >> Regards, >> Takenobu >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhpetersen at gmail.com Tue Feb 9 01:24:32 2021 From: juhpetersen at gmail.com (Jens Petersen) Date: Tue, 9 Feb 2021 09:24:32 +0800 Subject: [ANNOUNCE] GHC 9.0.1 released In-Reply-To: References: <87wnvnsoc5.fsf@smart-cactus.org> Message-ID: Hi Karel, On Tue, 9 Feb 2021 at 00:56, Karel Gardas wrote: > [karel at localhost ~]$ sudo dnf --enablerepo=updates-testing-modular > install ghc:9.0/default Apologies, I thought I had tested, but I copied without the 'module' command, it should be: $ sudo dnf --enablerepo=updates-testing-modular module install ghc:9.0/default Alternatively you can run, eg: $ sudo dnf --enablerepo=updates-testing-modular module enable ghc:9.0 $ sudo dnf --enablerepo=updates-testing-modular install ghc etc It should go stable in about a week. Thanks, Jens -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Tue Feb 9 02:03:48 2021 From: ben at well-typed.com (Ben Gamari) Date: Mon, 08 Feb 2021 21:03:48 -0500 Subject: Reduction in Windows CI capacity Message-ID: <87r1lq2e0u.fsf@smart-cactus.org> tl;dr. GHC's CI capacity will be a reduced due to a loss of sponsorship, particularly in Windows runner capacity. Help wanted in finding additional capacity. Hi all, For many years Google X has generously donated Google Compute Engine resources to GHC's CI infrastructure. We all owe a debt of gratitude to Google X for providing us with what has undoubtedly amounted to tens of thousands of dollars of computational capacity over the years. I would especially like to thank Greg Steuck, whose advocacy made this donation possible. Of course, organizational priorities understandably change and Google X will be unable to continue their sponsorship in the future. This puts us in a bit of a tricky situation; Google Compute Engine is currently the home of nearly 100 cores worth of CI capacity, including: * roughly 20% of our x86-64/Linux capacity * our only x86-64/FreeBSD runner * all five of our x86-64 Windows runners While the Linux runners are fairly easy to replace, the Windows capacity is a bit harder since Windows cloud capacity is quite expensive (IIRC nearly half of the cost of our Windows GCE instances is put towards the license). In the short term I can cover for some of this lost capacity by bringing up a Windows runner using our generous donation from Packet [1]. However, I am extrmely wary of outspending our welcome on Packet's infrastructure and therefore we will need to accept a small reduction in capacity for a bit while we work out a more sustainable path forward. We will have to see how things go but it may be necessary to disable the Windows jobs on (non-Marge) merge request validation pipelines. I am looking into various options for again reaching our previous capacity, but this is an area where you might be able to help: * Familiarity with Windows licensing. Unfortunately the details of Windows licensing for virtualization purposes are a bit tricky. I suspect that the cheapest way forward is a single Windows Server license on a large machine but if you are familiar with Windows licensing in this setting, please do be in touch. * Providing Windows licenses. If you know of an organization that may be able to donate Windows licenses either in-kind or via financial support, please do be in touch. * Providing Windows cloud instances. If you know of an organization that may be able to donate Windows cloud instances, do holler. As always, we welcome any hardware or cloud instance contributions. Do be in touch if you may be in an position to help out. Cheers, - Ben [1] https://www.packet.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From karel.gardas at centrum.cz Tue Feb 9 09:29:08 2021 From: karel.gardas at centrum.cz (Karel Gardas) Date: Tue, 9 Feb 2021 10:29:08 +0100 Subject: [ANNOUNCE] GHC 9.0.1 released In-Reply-To: References: <87wnvnsoc5.fsf@smart-cactus.org> Message-ID: Hi Jens, tested and this works! Thanks a lot, Karel On 2/9/21 2:24 AM, Jens Petersen wrote: > Hi Karel, > > On Tue, 9 Feb 2021 at 00:56, Karel Gardas > wrote: > > [karel at localhost ~]$ sudo dnf --enablerepo=updates-testing-modular > install ghc:9.0/default > >   > Apologies, I thought I had tested, but I copied without the 'module' > command, it should be: > > $ sudo dnf --enablerepo=updates-testing-modular module install > ghc:9.0/default > > Alternatively you can run, eg: > > $ sudo dnf --enablerepo=updates-testing-modular module enable ghc:9.0 > $ sudo dnf --enablerepo=updates-testing-modular install ghc > > etc > > It should go stable in about a week. > > Thanks, Jens >   From christiaan.baaij at gmail.com Tue Feb 9 10:54:44 2021 From: christiaan.baaij at gmail.com (Christiaan Baaij) Date: Tue, 9 Feb 2021 11:54:44 +0100 Subject: Find the ghc binary path from libdir In-Reply-To: <3112aaa0-1662-b926-a01a-0cd3dce55786@gmail.com> References: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> <20210208184801.cs5bmgfbf64cceq3@darkboxed.org> <3112aaa0-1662-b926-a01a-0cd3dce55786@gmail.com> Message-ID: Calling `ghc --print-lib-dir` is the only way because the `$libdir` is actually provided on the cmdline using the -B flag: https://gitlab.haskell.org/ghc/ghc/-/blob/master/ghc/Main.hs#L110-112 That's why the `ghc` you normally execute is a shell-wrapper around the `ghc` executable applied to a `-B` argument set at installation time. But a user is free to move that `$libdir` dir to any other place after installation and update the `-B` flag accordingly. On Mon, 8 Feb 2021 at 20:09, Fendor wrote: > Hi, > > Well the purpose is a bit convoluted, but here it goes: > > Basically, we want to know which `ghc` version `cabal-install` is going > to pick for compiling the project in a platform independent way without > resorting to linking to `Cabal`. > > You can execute `cabal exec -- ghc` which launches the correct `ghc` > version, but as far as we found out, there is no programmatic way of > finding the path to the `ghc` executable itself (`cabal exec -- where > ghc` or `cabal exec -- which ghc` fails on windows). What we really want > is `cabal-install` to tell us, e.g. with `cabal show-build-info` which > is a command not implemented yet[0]. > > As long as this command is not implemented, we have to monkey patch it > in order to satisfy our goals. You can execute `cabal exec -- ghc > --print-libdir`, which gives you the `$libdir`, which brings us back to > the original question: can we find the `ghc` executable given `$libdir` > in a platform independent way? > > [0] https://github.com/haskell/cabal/pull/6241 > > Best regards, > Fendor > > On 2/8/21 7:48 PM, Daniel Gröber wrote: > > Hi Fendor, > > > > On Mon, Feb 08, 2021 at 07:35:12PM +0100, Fendor wrote: > >> I want to find out the location of the `ghc` (and `ghc-pkg`) executable > >> given the `$libdir` (obtained by `ghc --print-libdir`). > >> Is there a platform independent way of doing that, which we can rely on? > > Are you aware of the `ghc --print-libdir` option? I think you'd be much > > better off going from ghc binary to libdir instead of the other way > around. > > > > Could you explain why you're trying to do it your way? > > > >> (P. S. first time using a mailing list, hope I am doing it correctly) > > Seems fine to me :) > > > > --Daniel > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takenobu.hs at gmail.com Tue Feb 9 12:22:05 2021 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Tue, 9 Feb 2021 21:22:05 +0900 Subject: Has ghc-9.0 for windows changed to require installation? In-Reply-To: References: Message-ID: Hi Moritz, Tamar, Ben, Thank you for the detailed explaining. I understood the current status. Ben has opened this: https://gitlab.haskell.org/ghc/ghc/-/issues/19339 Always thanks a lot, Takenobu On Tue, Feb 9, 2021 at 4:24 AM Phyx wrote: > > No, there's no change at all in the portability. > This looks like a fallout from switching from Make to Hadrian. > > ghcii.sh was created as an artifact of make install. Hadrian seems to lack this step. > > Note that this script is nothing magical, it's just a hack around how signal handlers in Native windows and Cygwin processes are handled. > > I'm not entirely sure it's still needed with the new I/O manager, however that's not on yet by default.. > > Anyways support should probably be added to Hadrian. > > Kind regards, > Tamar > > Sent from my Mobile > > On Mon, Feb 8, 2021, 14:43 Moritz Angermann wrote: >> >> Thanks for flagging this. This would be the opposite direction of what I’ve been advocating for. That we get bindists for Linux and macOS that work by simply unpacking them. >> >> On Mon, 8 Feb 2021 at 10:05 PM, Takenobu Tani wrote: >>> >>> Hi devs, >>> >>> The ghc-binary for windows needs to `make install` since ghc-9.0 [1]. >>> Is this an intended change? >>> >>> Previously, ghc-8.10.4 binary for windows [2] doesn't need to `make install`. >>> We only expand the tar-file and then we can execute `bin/ghcii.sh`. >>> >>> [1]: https://downloads.haskell.org/ghc/9.0.1/ghc-9.0.1-x86_64-unknown-mingw32.tar.xz >>> [2]: https://downloads.haskell.org/ghc/8.10.4/ghc-8.10.4-x86_64-unknown-mingw32.tar.xz >>> >>> Regards, >>> Takenobu >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From andrzej at rybczak.net Tue Feb 9 14:46:08 2021 From: andrzej at rybczak.net (Andrzej Rybczak) Date: Tue, 9 Feb 2021 15:46:08 +0100 Subject: Plan for GHC 9.2 Message-ID: <07ad0a0b-27ed-70ee-2030-e411467c2eef@rybczak.net> Hey Ben, It would be excellent to get NoFieldSelectors extension (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4743) in. IIUC it's almost ready to be merged. From matthewtpickering at gmail.com Tue Feb 9 16:06:24 2021 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Tue, 9 Feb 2021 16:06:24 +0000 Subject: Plan for GHC 9.2 In-Reply-To: <87v9b7slvf.fsf@smart-cactus.org> References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: My patch adding `-finfo-table-map` and `-fdistinct-constructor-tables` is ready to review and should be included in 9.2. https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3469 There are also a one outstanding patches related to ghc-debug. (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4583) Cheers, Matt On Thu, Feb 4, 2021 at 6:56 PM Ben Gamari wrote: > > > tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, > especially if you have something you would like merged for 9.2.1. > > Hello all, > > With GHC 9.0.1 at long-last out the door, it is time that we start > turning attention to GHC 9.2. I would like to avoid making the mistake > made in the 9.0 series in starting the fork in a state that required a > significant amount of backporting to be releaseable. Consequently, I > want to make sure that we have a fork schedule that is realistic given > the things that need to be merged for 9.2. These include: > > * Update haddock submodule in `master` (Ben) > * Bumping bytestring to 0.11 (#19091, Ben) > * Finishing the rework of sized integer primops (#19026, John Ericson) > * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) > * Merge BoxedRep (#17526, Ben) > * ARM NCG backend and further stabilize Apple ARM support? (Moritz) > * Some form of coercion zapping (Ben, Simon, Richard) > * Tag inference analysis and tag check elision (Andreas) > > If you see something that you would like to see in 9.2.1 please do > holler. Otherwise, if you see your name in this list it would be great > if you could let me know when you think your project may be in a > mergeable state. > > Ideally we would strive for a schedule like the following: > > 4 February 2021: We are here > ~4 weeks pass > 3 March 2021: Release branch forked > 1 week passes > 10 March 2021: Alpha 1 released > 3 weeks pass > 31 March 2021: Alpha 2 released > 2 weeks pass > 14 April 2021: Alpha 3 released > 2 weeks pass > 28 April 2021: Alpha 4 released > 1 week passes > 5 May 2021: Beta 1 released > 1 week passes > 12 May 2021: Release candidate 1 released > 2 weeks pass > 26 May 2021: Final release > > This provides ample time for stabilization while avoiding deviation from > the usual May release timeframe. However, this would require that we > move aggressively to start getting the tree into shape since the fork > would be less than four weeks away. I would appreciate contributors' > thoughts on the viability of this timeline. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From juhpetersen at gmail.com Wed Feb 10 09:57:27 2021 From: juhpetersen at gmail.com (Jens Petersen) Date: Wed, 10 Feb 2021 17:57:27 +0800 Subject: [ANNOUNCE] GHC 8.10.4 released In-Reply-To: <874kip41kq.fsf@smart-cactus.org> References: <874kip41kq.fsf@smart-cactus.org> Message-ID: On Sun, 7 Feb 2021 at 00:17, Ben Gamari wrote: > The GHC team is very pleased to announce the availability of GHC 8.10.4. > Thank you! This is now already built for Fedora and can be tested/installed with: dnf --enablerepo=updates-testing-modular module install ghc:8.10/default It should go stable after one week (after which you don't need the --enablerepo option). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mniip at typeable.io Wed Feb 10 10:43:35 2021 From: mniip at typeable.io (Igor Popov) Date: Wed, 10 Feb 2021 13:43:35 +0300 Subject: Newtype over (~#) In-Reply-To: <010f01777d5e6882-b4657fe4-a37f-4d9d-8d7b-8f0f0c0123be-000000@us-east-2.amazonses.com> References: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> <010f01777d5e6882-b4657fe4-a37f-4d9d-8d7b-8f0f0c0123be-000000@us-east-2.amazonses.com> Message-ID: That's a really interesting quirk. That kind of explains why an argument to a cast has to be a CoVar, we want all casts to be guarded by a case-of. But do we still need the restriction that an Id cannot have a coercion type? Either case it seems like we need to be extremely careful with how the wrapper and the matcher works for such a constructor. I was hoping a sledgehammer approach would work where we kind of just force (a ~# b) as a field into a newtype, and the existing machinery magically works with it. The construction is then chock full of questions: axiom N::~:# :: forall k. (:~:#) @k ~R (~#) @k Is this valid? mkPrimEqPred says it doesn't like equalities on either side (I'm assuming this applies to eta-contracted equalities as well). mkReprPrimEqPred doesn't. Who to believe and what are the pitfalls? Refl# :: forall k (a b :: k). a ~# b -> a :~:# b Refl# = \@k @a @b v -> v `cast` Sym (N::~:# k) Is this valid? You say that v has to be bound as a CoVar (current newtype machinery will not do this). Can a CoVar appear in the LHS of a cast? The result is definitely a type and can be bound to an Id, I guess. $WRefl# :: forall k (a :: k). a :~:# a $WRefl# = \@k @a -> Refl# @k @a @a Relatively tame, but however once we inline the definition of Refl# (simpleOptExpr will do this when registering an unfolding), we get: `cast` Sym (N::~:# k) Is this valid? The matcher/boxer (is that the word) has to be carefully constructed as well: we have to desugar case x of Refl# -> e where (x :: a :~:# b) into: case x `cast` N::~:# k of co -> e where co is now a CoVar. Can a cast's result type be a coercion? Can a coercion be the scrutinee of a case-of? (Judging by the code we generate for eq_sel the answer is yes). -- mniip On Sun, Feb 7, 2021 at 7:41 PM Richard Eisenberg wrote: > > We do need a separate category, though: otherwise, we could cast by `f x`, where `f` is a non-terminating functions. We erase casts, so the function would never get called. Maybe if we required that casts are always by variables (essentially, A-normalize casts) we could avoid this? But then would we require A-normalization to build coercions from pieces (as opposed to calling functions)? It's unclear. > > I think the first versions of this idea didn't require the separate syntactic category, but all work for the past decade has. I think there are other ways, but the way GHC handles this now is somewhat poor, because of #17291. > > Richard > > > On Feb 5, 2021, at 7:56 PM, Igor Popov wrote: > > > >> GHC cheats in this area. The problem is that (a ~# b) is a type, because that is terribly, terribly convenient. But it really shouldn't be. > >> > >> The problem is that coercion variables are different from regular variables. So, if we say (v :: a ~# b), we must always be careful: is v a coercion variable or not? If it isn't, then v is useless. You cannot, for instance, use v in a cast. > > > > I don't really see a problem here. The fact that only a "coercion > > variable" can be used in a cast should be enforced by the typing rule > > for cast. That doesn't require having a distinct "syntactic category" > > of coercion variables, unless I'm missing something. > > > > -- mniip > From moritz.angermann at gmail.com Wed Feb 10 13:05:14 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 10 Feb 2021 21:05:14 +0800 Subject: Stop holding hadrian back with backwards compatibility Message-ID: Hi, so we've finally run into a case where we need to bump the rts version. This has a great ripple effect. There is some implicit assumption that rts-1.0 will always be true. Of course that was a lie, but a lie we lived with for a long time. Now, hadrian tries *really* hard to replicate some of the Make based build systems idiosyncrasies, this includes creating versionless symlinks for the rts. E.g. libHSrts -> libHSrts-1.0. There is a great deal of logic just to achieve this, and of course it all crumbles now. I'd therefore like to float and propose the idea that we agree to *not* bother (too?) much with make based build systems backwards compatibility and warts that grew over the years in the make based build system with hadrian going forward. Yes, I can probably fix this, and add even more code to this burning pile of complexity, but why? The next person will assume libHSrts does not need to be versioned and continue with this mess. Let's have Hadrian be a clean cut in some areas (it already is, it does away with the horrible abomination that ghc-cabal is--which only serves the purpose of translating cabal descriptions into make readable files), and not be bogged down by backwards compatibility. This is thus my call for voicing concern or the upkeep of legacy support, or I'll take silence as the collective support of making hadrian *not* be held back by backwards compatibility. (This would mean in this case, that I'd just delete the backwards compat code instead of adding even more to it). I hope we all still want Hadrian to replace Make, if not and we want to keep Make, why are we concerning ourselves with Hadrian in the first place. If we are intending to ditch Make, let's not be held back by it. Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Wed Feb 10 13:27:59 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Wed, 10 Feb 2021 13:27:59 +0000 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: References: Message-ID: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> This sounds very reasonable on the surface, but I don't understand the consequences of this proposal. What are these consequences? Will this break `make`? (It sounds like it won't, given that the change is to Hadrian.) Does this mean horrible things will happen if I use `make` and `hadrian` in the same tree? (I have never done this, other than with hadrian/ghci, which seems to have its own working directory.) Basically: for someone who uses the build system but does not work on it, how does this affect me? (Maybe not at all!) I would explicitly like to endorse the direction of travel toward Hadrian and away from `make`. Richard > On Feb 10, 2021, at 8:05 AM, Moritz Angermann wrote: > > Hi, > > so we've finally run into a case where we need to bump the rts version. This has a great ripple effect. There is some implicit assumption that rts-1.0 will always be true. Of course that was a lie, but a lie we lived with for a long time. > > Now, hadrian tries *really* hard to replicate some of the Make based build systems idiosyncrasies, this includes creating versionless symlinks for the rts. E.g. libHSrts -> libHSrts-1.0. There is a great deal of logic just to achieve this, and of course it all crumbles now. > > I'd therefore like to float and propose the idea that we agree to *not* bother (too?) much with make based build systems backwards compatibility and warts that grew over the years in the make based build system with hadrian going forward. > > Yes, I can probably fix this, and add even more code to this burning pile of complexity, but why? The next person will assume libHSrts does not need to be versioned and continue with this mess. > > Let's have Hadrian be a clean cut in some areas (it already is, it does away with the horrible abomination that ghc-cabal is--which only serves the purpose of translating cabal descriptions into make readable files), and not be bogged down by backwards compatibility. > > This is thus my call for voicing concern or the upkeep of legacy support, or I'll take silence as the collective support of making hadrian *not* be held back by backwards compatibility. (This would mean in this case, that I'd just delete the backwards compat code instead of adding even more to it). > > I hope we all still want Hadrian to replace Make, if not and we want to keep Make, why are we concerning ourselves with Hadrian in the first place. If we are intending to ditch Make, let's not be held back by it. > > Cheers, > Moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From moritz.angermann at gmail.com Wed Feb 10 13:32:29 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 10 Feb 2021 21:32:29 +0800 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> References: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> Message-ID: My understanding of this backwards compat logic is that it's only there to allow you to do stuff like: build with hadrian, and then continue using make with the artifacts (partially) built by hadrian. I think this is a horrible idea in and onto itself, even if I can somewhat see the appeal as a gateway drug; in which you'd slowly have hadrian take over parts that make used to do, and use make for the stuff that doesn't work (yet) in hadrian. However, I don't think the benefit of constraining hadrian to work in the make framework makes much sense. We should be permitted to explore new (and better) solutions, that do not align with how the make based build system did things if it allows for a less complex build system or faster builds or ... Cheers, Moritz On Wed, Feb 10, 2021 at 9:28 PM Richard Eisenberg wrote: > This sounds very reasonable on the surface, but I don't understand the > consequences of this proposal. What are these consequences? Will this break > `make`? (It sounds like it won't, given that the change is to Hadrian.) > Does this mean horrible things will happen if I use `make` and `hadrian` in > the same tree? (I have never done this, other than with hadrian/ghci, which > seems to have its own working directory.) Basically: for someone who uses > the build system but does not work on it, how does this affect me? (Maybe > not at all!) > > I would explicitly like to endorse the direction of travel toward Hadrian > and away from `make`. > > Richard > > > On Feb 10, 2021, at 8:05 AM, Moritz Angermann < > moritz.angermann at gmail.com> wrote: > > > > Hi, > > > > so we've finally run into a case where we need to bump the rts version. > This has a great ripple effect. There is some implicit assumption that > rts-1.0 will always be true. Of course that was a lie, but a lie we lived > with for a long time. > > > > Now, hadrian tries *really* hard to replicate some of the Make based > build systems idiosyncrasies, this includes creating versionless symlinks > for the rts. E.g. libHSrts -> libHSrts-1.0. There is a great deal of > logic just to achieve this, and of course it all crumbles now. > > > > I'd therefore like to float and propose the idea that we agree to *not* > bother (too?) much with make based build systems backwards compatibility > and warts that grew over the years in the make based build system with > hadrian going forward. > > > > Yes, I can probably fix this, and add even more code to this burning > pile of complexity, but why? The next person will assume libHSrts does not > need to be versioned and continue with this mess. > > > > Let's have Hadrian be a clean cut in some areas (it already is, it does > away with the horrible abomination that ghc-cabal is--which only serves the > purpose of translating cabal descriptions into make readable files), and > not be bogged down by backwards compatibility. > > > > This is thus my call for voicing concern or the upkeep of legacy > support, or I'll take silence as the collective support of making hadrian > *not* be held back by backwards compatibility. (This would mean in this > case, that I'd just delete the backwards compat code instead of adding even > more to it). > > > > I hope we all still want Hadrian to replace Make, if not and we want to > keep Make, why are we concerning ourselves with Hadrian in the first place. > If we are intending to ditch Make, let's not be held back by it. > > > > Cheers, > > Moritz > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed Feb 10 13:50:30 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 10 Feb 2021 13:50:30 +0000 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: References: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> Message-ID: build with hadrian, and then continue using make with the artifacts (partially) built by Hadrian I agree this is a non-goal. Simon From: ghc-devs On Behalf Of Moritz Angermann Sent: 10 February 2021 13:32 To: Richard Eisenberg Cc: ghc-devs Subject: Re: Stop holding hadrian back with backwards compatibility My understanding of this backwards compat logic is that it's only there to allow you to do stuff like: build with hadrian, and then continue using make with the artifacts (partially) built by hadrian. I think this is a horrible idea in and onto itself, even if I can somewhat see the appeal as a gateway drug; in which you'd slowly have hadrian take over parts that make used to do, and use make for the stuff that doesn't work (yet) in hadrian. However, I don't think the benefit of constraining hadrian to work in the make framework makes much sense. We should be permitted to explore new (and better) solutions, that do not align with how the make based build system did things if it allows for a less complex build system or faster builds or ... Cheers, Moritz On Wed, Feb 10, 2021 at 9:28 PM Richard Eisenberg > wrote: This sounds very reasonable on the surface, but I don't understand the consequences of this proposal. What are these consequences? Will this break `make`? (It sounds like it won't, given that the change is to Hadrian.) Does this mean horrible things will happen if I use `make` and `hadrian` in the same tree? (I have never done this, other than with hadrian/ghci, which seems to have its own working directory.) Basically: for someone who uses the build system but does not work on it, how does this affect me? (Maybe not at all!) I would explicitly like to endorse the direction of travel toward Hadrian and away from `make`. Richard > On Feb 10, 2021, at 8:05 AM, Moritz Angermann > wrote: > > Hi, > > so we've finally run into a case where we need to bump the rts version. This has a great ripple effect. There is some implicit assumption that rts-1.0 will always be true. Of course that was a lie, but a lie we lived with for a long time. > > Now, hadrian tries *really* hard to replicate some of the Make based build systems idiosyncrasies, this includes creating versionless symlinks for the rts. E.g. libHSrts -> libHSrts-1.0. There is a great deal of logic just to achieve this, and of course it all crumbles now. > > I'd therefore like to float and propose the idea that we agree to *not* bother (too?) much with make based build systems backwards compatibility and warts that grew over the years in the make based build system with hadrian going forward. > > Yes, I can probably fix this, and add even more code to this burning pile of complexity, but why? The next person will assume libHSrts does not need to be versioned and continue with this mess. > > Let's have Hadrian be a clean cut in some areas (it already is, it does away with the horrible abomination that ghc-cabal is--which only serves the purpose of translating cabal descriptions into make readable files), and not be bogged down by backwards compatibility. > > This is thus my call for voicing concern or the upkeep of legacy support, or I'll take silence as the collective support of making hadrian *not* be held back by backwards compatibility. (This would mean in this case, that I'd just delete the backwards compat code instead of adding even more to it). > > I hope we all still want Hadrian to replace Make, if not and we want to keep Make, why are we concerning ourselves with Hadrian in the first place. If we are intending to ditch Make, let's not be held back by it. > > Cheers, > Moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From dxld at darkboxed.org Wed Feb 10 14:12:31 2021 From: dxld at darkboxed.org (Daniel =?utf-8?Q?Gr=C3=B6ber?=) Date: Wed, 10 Feb 2021 15:12:31 +0100 Subject: Find the ghc binary path from libdir In-Reply-To: <3112aaa0-1662-b926-a01a-0cd3dce55786@gmail.com> References: <05e152ec-729b-e990-85b8-a4501dfedfa9@gmail.com> <20210208184801.cs5bmgfbf64cceq3@darkboxed.org> <3112aaa0-1662-b926-a01a-0cd3dce55786@gmail.com> Message-ID: <20210210141231.pwxgyyyku22owlb2@darkboxed.org> Hi Fendor, On Mon, Feb 08, 2021 at 08:08:45PM +0100, Fendor wrote: > Basically, we want to know which `ghc` version `cabal-install` is going to > pick for compiling the project in a platform independent way without > resorting to linking to `Cabal`. For v2-build this can be found out by looking at plan.json for which you only need to link to cabal-plan (or a json lib of your choice). You do need to have logic run `cabal --dry-run` (IIRC) to generate plan.json if it's not there. I have some code for that in cabal-helper, it's not very tricky as long as you're assuming the cabal project isn't in a broken state anyway. > You can execute `cabal exec -- ghc` which launches the correct `ghc` > version, but as far as we found out, there is no programmatic way of finding > the path to the `ghc` executable itself (`cabal exec -- where ghc` or `cabal > exec -- which ghc` fails on windows). Seems to me it would be relatively straightforward to add a "which" mode to whatever executable you're shipping (we're talking about hls, right?) so you can do something like: cabal exec -- $path_to_hls which which then just prints the path to GHC on PATH found using the appropriate library function without running into the problem of not having `which` on windows. --Daniel From rsx at bluewin.ch Wed Feb 10 15:44:29 2021 From: rsx at bluewin.ch (Roland Senn) Date: Wed, 10 Feb 2021 16:44:29 +0100 Subject: Plan for GHC 9.2 Message-ID: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> I hope ticket #19157 will make it in the GHC 9.2 release. In the GHCi debugger it adds the possibility to set ignore counts to breakpoints. The next times the break point is reached the program's execution does not stop. This feature is available in nearly every debugger, but until now not yet in the GHCi debugger. Merge request !4839 is ready for review (and it's NOT rocket science...) From ben at well-typed.com Wed Feb 10 17:40:10 2021 From: ben at well-typed.com (Ben Gamari) Date: Wed, 10 Feb 2021 12:40:10 -0500 Subject: Plan for GHC 9.2 In-Reply-To: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> References: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> Message-ID: <87h7mj3jph.fsf@smart-cactus.org> Roland Senn writes: > I hope ticket #19157 will make it in the GHC 9.2 release. In the GHCi > debugger it adds the possibility to set ignore counts to breakpoints. > The next times the break point is reached the program's > execution does not stop. This feature is available in nearly every > debugger, but until now not yet in the GHCi debugger. > Merge request !4839 is ready for review (and it's NOT rocket > science...) > Indeed, this seems quite reasonable. I don't see any reason why we shouldn't be able to get it in to 9.2.1. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Wed Feb 10 22:58:59 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Wed, 10 Feb 2021 22:58:59 +0000 Subject: Newtype over (~#) In-Reply-To: References: <010f017773e92ca1-03675e83-871d-4445-9581-c6409dfcff80-000000@us-east-2.amazonses.com> <010f01777d5e6882-b4657fe4-a37f-4d9d-8d7b-8f0f0c0123be-000000@us-east-2.amazonses.com> Message-ID: <010f01778e2b5219-b86ceaea-4156-4b12-a4c9-2fa69ef3ad34-000000@us-east-2.amazonses.com> > On Feb 10, 2021, at 5:43 AM, Igor Popov wrote: > > That's a really interesting quirk. That kind of explains why an > argument to a cast has to be a CoVar, we want all casts to be guarded > by a case-of. But do we still need the restriction that an Id cannot > have a coercion type? No, we wouldn't need the restriction if GHC were implemented correctly -- that is, without the mkLocalIdOrCoVar abomination that looks at the type of a binder to determine whether it's a CoVar. > > Either case it seems like we need to be extremely careful with how the > wrapper and the matcher works for such a constructor. I was hoping a > sledgehammer approach would work where we kind of just force (a ~# b) > as a field into a newtype, and the existing machinery magically works > with it. > > The construction is then chock full of questions: > > axiom N::~:# :: forall k. (:~:#) @k ~R (~#) @k > > Is this valid? mkPrimEqPred says it doesn't like equalities on either > side (I'm assuming this applies to eta-contracted equalities as well). I don't see any equalities on either side here. Instead, this is a relationship between equality *types*. That should be just fine; GHC even produces such things already from time to time. I do think you'll need two `@k`s on the right, though. The rule you're worried about forbids constructions like (cv1 ~R cv2), where cv1 and cv2 have been injected into types via CoercionTy. > mkReprPrimEqPred doesn't. Who to believe and what are the pitfalls? mkReprPrimEqPred has the same properties here; the comments should be better. > > Refl# :: forall k (a b :: k). a ~# b -> a :~:# b > Refl# = \@k @a @b v -> v `cast` Sym (N::~:# k) > > Is this valid? You say that v has to be bound as a CoVar (current > newtype machinery will not do this). Can a CoVar appear in the LHS of > a cast? No, it can't. This may be a real blocker. I hadn't thought about this part of the problem. > > > `cast` Sym (N::~:# k) > > Is this valid? No, for the same reasons. This is unfortunate, as I thought this was a good idea. Yet I'm not hopeful that there's an easy way out of this particular trap. Richard From rae at richarde.dev Wed Feb 10 23:00:26 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Wed, 10 Feb 2021 23:00:26 +0000 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: References: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> Message-ID: <010f01778e2ca578-e1689e15-b72e-41c0-afd0-28125367afbe-000000@us-east-2.amazonses.com> > On Feb 10, 2021, at 8:50 AM, Simon Peyton Jones wrote: > > build with hadrian, and then continue using make with the artifacts (partially) built by Hadrian I almost suggested that this had to be the reason for the back-compat design, but I assumed I had to be wrong. I also agree this is a non-goal; I'm quite happy to be forced to pick one or the other and stick with that choice until blasting away all build products. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From lonetiger at gmail.com Thu Feb 11 09:30:22 2021 From: lonetiger at gmail.com (Phyx) Date: Thu, 11 Feb 2021 09:30:22 +0000 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: <010f01778e2ca578-e1689e15-b72e-41c0-afd0-28125367afbe-000000@us-east-2.amazonses.com> References: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> <010f01778e2ca578-e1689e15-b72e-41c0-afd0-28125367afbe-000000@us-east-2.amazonses.com> Message-ID: Hi, Just leaving my two cents feel free to ignore.. > I almost suggested that this had to be the reason for the back-compat design You're right, but not for backwards compat of Hadrian vs Make, but for compat with RTS versions. I could be wrong, but my understanding is the current design in Make is just an artifact of getting something that works on all OSes without much pain, but has proven to be suboptimal in a very important use case (slight detour time): You have to make a choice of which RTS to use at compile time. Which is quite bad. Because it means that you can't swap between two RTS flavors with the same ABI. It also means building presents a problem, you want your compiler at the end of stage1 to use your new rts, not the one of the stage0 compiler. You can't have multiple versions of the RTS in one library, but if you have the full name as a dependency the dynamic loader happily loads you multiple copies. To solve this issue the design was made to not declare the RTS as a dependency on any haskell library. i.e. there's not DT_NEEDED entry for it on ELF operating systems. Which means before you load a Haskell produced dynamic library on Linux you need to LD_PRELOAD an rts. It's clunky, but it works, it allows you to switch between debug and non-debug rts at initialization time. On Windows, this problem was punted, because everything is statically linked. But the problem exists that you can have multiple DLLs with different RTS and ABIs. This is fine as long as the DLLs have no dependencies on each other. Once they do... you have a big problem. This is one of the primary blockers of shared library support on Windows. I.. don't know whatever wacky solution MacOS uses so can't comment there. Now back to the original question about version 1.0, this has nothing to do with Make at all. Make based system only implemented the scheme that was wanted. It's not like any Make system design issues forced this scheme. Now over the years, assumptions that the RTS is always version 1.0 could have krept into the build system. But I don't believe this to have been design, just convenience. Right now, the design only requires you to know the GHC version, which is available in all makefiles. Knowing the RTS version would be difficult, but the point is that in a proper design you don't need to know the version. Almost half a decade ago a plan was made to replace this scheme with one that would work on all OSes and would allow us to solve these issues. The design was made and debated here https://gitlab.haskell.org/ghc/ghc/-/issues/10352 The actual solution isn't as simple as just adding the rts version to the library name or add it only to the build system, in fact this would be the wrong approach as it makes it impossible to observe backwards compatibility between GHC releases. i.e. without it, you'd need to have GHC 9.0.1 installed to run GHC 9.0.1 programs, you can't run using GHC 9.2.x rts if the version changed. Typically ELF based platforms solve this by a combination of SONAME and symbol versioning. Windows solves this by a combination of SxS Assembly versioning or mingw style SONAME. All of which require you to have the same filename for the libraries, but use a different path to disambiguate: lib/ghc-${ver}/rts-1.0/libHSrts-ghc${ver}.so lib/ghc-${ver}/rts-1.0/thr/libHSrts-ghc${ver}.so lib/ghc-${ver}/rts-1.0/debug/libHSrts-ghc${ver}.so lib/ghc-${ver}/rts-1.0/l/libHSrts-ghc${ver}.so lib/ghc-${ver}/rts-1.0/thr_l/libHSrts-ghc${ver}.so for each RTS with the same ABI. profiling libs for instance have a different ABI and can't use this scheme. So what has taken so long to implement this? Well.. time. As it turns out, getting this scheme to work required a lot of foundational work in GHC (Particularly on Windows where dynamic linking design wasn't optimal, but both GHC and the dynamic linker are happy now). On Linux it took a while to get SONAME support in cabal https://github.com/haskell/cabal/issues/4052 so we don't have to hack around in the build system. But anyway this is why the current scheme exists, and why just adding an rts version isn't really sufficient, especially if the name propagates to the shared lib. TL;DR; If we are going to change the build system, we should do it properly. The current scheme exists because GHC does not observe any mechanism to support multiple runtimes with the same ABI and does not really have a backwards compatibility story. Kind Regards, Tamar On Wed, Feb 10, 2021 at 11:00 PM Richard Eisenberg wrote: > > > On Feb 10, 2021, at 8:50 AM, Simon Peyton Jones > wrote: > > build with hadrian, and then continue using make with the artifacts > (partially) built by Hadrian > > > I almost suggested that this had to be the reason for the back-compat > design, but I assumed I had to be wrong. I also agree this is a non-goal; > I'm quite happy to be forced to pick one or the other and stick with that > choice until blasting away all build products. > > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Thu Feb 11 09:48:56 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 11 Feb 2021 17:48:56 +0800 Subject: Stop holding hadrian back with backwards compatibility In-Reply-To: References: <010f01778c208d4b-66cbabcf-600e-47c5-a49f-bb1716a87659-000000@us-east-2.amazonses.com> <010f01778e2ca578-e1689e15-b72e-41c0-afd0-28125367afbe-000000@us-east-2.amazonses.com> Message-ID: Tamar, thanks so much for the backstory and the tickets. I’ll go dig down this path a bit more. Cheers, Moritz On Thu, 11 Feb 2021 at 5:31 PM, Phyx wrote: > Hi, Just leaving my two cents feel free to ignore.. > > > I almost suggested that this had to be the reason for the back-compat > design > > You're right, but not for backwards compat of Hadrian vs Make, but for > compat with RTS versions. > I could be wrong, but my understanding is the current design in Make is > just an artifact of getting something that works on all OSes without much > pain, but has proven to be suboptimal in a very important use case (slight > detour time): > > You have to make a choice of which RTS to use at compile time. Which is > quite bad. Because it means that you can't swap between two RTS flavors > with the same ABI. It also means building presents a problem, you want your > compiler at the end of stage1 to use your new rts, not the one of the > stage0 compiler. > > You can't have multiple versions of the RTS in one library, but if you > have the full name as a dependency the dynamic loader happily loads you > multiple copies. > > To solve this issue the design was made to not declare the RTS as a > dependency on any haskell library. i.e. there's not DT_NEEDED entry for it > on ELF operating systems. Which means before you load a Haskell produced > dynamic library on Linux you need to LD_PRELOAD an rts. It's clunky, but it > works, it allows you to switch between debug and non-debug rts at > initialization time. > > On Windows, this problem was punted, because everything is statically > linked. But the problem exists that you can have multiple DLLs with > different RTS and ABIs. This is fine as long as the DLLs have no > dependencies on each other. Once they do... you have a big problem. This > is one of the primary blockers of shared library support on Windows. > > I.. don't know whatever wacky solution MacOS uses so can't comment there. > > Now back to the original question about version 1.0, this has nothing to > do with Make at all. Make based system only implemented the scheme that was > wanted. It's not like any Make system design issues forced this scheme. Now > over the years, assumptions that the RTS is always version 1.0 could have > krept into the build system. But I don't believe this to have been design, > just convenience. Right now, the design only requires you to know the GHC > version, which is available in all makefiles. Knowing the RTS version > would be difficult, but the point is that in a proper design you don't need > to know the version. > > Almost half a decade ago a plan was made to replace this scheme with one > that would work on all OSes and would allow us to solve these issues. The > design was made and debated here > https://gitlab.haskell.org/ghc/ghc/-/issues/10352 > > The actual solution isn't as simple as just adding the rts version to the > library name or add it only to the build system, in fact this would be the > wrong approach as it makes it impossible to observe backwards compatibility > between GHC releases. > i.e. without it, you'd need to have GHC 9.0.1 installed to run GHC 9.0.1 > programs, you can't run using GHC 9.2.x rts if the version changed. > > Typically ELF based platforms solve this by a combination of SONAME and > symbol versioning. Windows solves this by a combination of SxS Assembly > versioning or mingw style SONAME. > > All of which require you to have the same filename for the libraries, but > use a different path to disambiguate: > > lib/ghc-${ver}/rts-1.0/libHSrts-ghc${ver}.so > > lib/ghc-${ver}/rts-1.0/thr/libHSrts-ghc${ver}.so > > lib/ghc-${ver}/rts-1.0/debug/libHSrts-ghc${ver}.so > > lib/ghc-${ver}/rts-1.0/l/libHSrts-ghc${ver}.so > > lib/ghc-${ver}/rts-1.0/thr_l/libHSrts-ghc${ver}.so > > for each RTS with the same ABI. profiling libs for instance have a > different ABI and can't use this scheme. > > So what has taken so long to implement this? Well.. time. As it turns out, > getting this scheme to work required a lot of foundational work in GHC > (Particularly on Windows where dynamic linking design wasn't optimal, but > both GHC and the dynamic linker are happy now). > > On Linux it took a while to get SONAME support in cabal > https://github.com/haskell/cabal/issues/4052 so we don't have to hack > around in the build system. > > But anyway this is why the current scheme exists, and why just adding an > rts version isn't really sufficient, especially if the name propagates to > the shared lib. > > TL;DR; > > If we are going to change the build system, we should do it properly. > > The current scheme exists because GHC does not observe any mechanism to > support multiple runtimes with the same ABI and does not really have a > backwards compatibility story. > > Kind Regards, > > Tamar > > On Wed, Feb 10, 2021 at 11:00 PM Richard Eisenberg > wrote: > >> >> >> On Feb 10, 2021, at 8:50 AM, Simon Peyton Jones >> wrote: >> >> build with hadrian, and then continue using make with the artifacts >> (partially) built by Hadrian >> >> >> I almost suggested that this had to be the reason for the back-compat >> design, but I assumed I had to be wrong. I also agree this is a non-goal; >> I'm quite happy to be forced to pick one or the other and stick with that >> choice until blasting away all build products. >> >> Richard >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Thu Feb 11 10:25:27 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Thu, 11 Feb 2021 11:25:27 +0100 Subject: Plan for GHC 9.2 In-Reply-To: <87h7mj3jph.fsf@smart-cactus.org> References: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> <87h7mj3jph.fsf@smart-cactus.org> Message-ID: Hi, Since my hopes of finally merging Nested CPR have recently been crushed again, I hope that we can include the implementation of the UnliftedDatatypes extension (proposal , implementation ). It was on ice since it depends on the BoxedRep proposal, but if BoxedRep is going to make it, surely UnliftedDatatypes can make it, too. I expect quite a few bugs, simply because I don't have much code to test it on yet. But I'm very confident that existing code isn't impacted by that, as most of the functionality (CodeGen for unlifted types, most importantly) was already there and I only had to refine a conditional here and there. Cheers, Sebastian Am Mi., 10. Feb. 2021 um 18:42 Uhr schrieb Ben Gamari : > Roland Senn writes: > > > I hope ticket #19157 will make it in the GHC 9.2 release. In the GHCi > > debugger it adds the possibility to set ignore counts to breakpoints. > > The next times the break point is reached the program's > > execution does not stop. This feature is available in nearly every > > debugger, but until now not yet in the GHCi debugger. > > Merge request !4839 is ready for review (and it's NOT rocket > > science...) > > > Indeed, this seems quite reasonable. I don't see any reason why we > shouldn't be able to get it in to 9.2.1. > > Cheers, > > - Ben > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Feb 11 14:28:26 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 11 Feb 2021 14:28:26 +0000 Subject: Plan for GHC 9.2 In-Reply-To: References: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> <87h7mj3jph.fsf@smart-cactus.org> Message-ID: Yes I agree, unlifted data types would be terrific. From: ghc-devs On Behalf Of Sebastian Graf Sent: 11 February 2021 10:25 To: Ben Gamari Cc: ghc-devs Subject: Re: Plan for GHC 9.2 Hi, Since my hopes of finally merging Nested CPR have recently been crushed again, I hope that we can include the implementation of the UnliftedDatatypes extension (proposal, implementation). It was on ice since it depends on the BoxedRep proposal, but if BoxedRep is going to make it, surely UnliftedDatatypes can make it, too. I expect quite a few bugs, simply because I don't have much code to test it on yet. But I'm very confident that existing code isn't impacted by that, as most of the functionality (CodeGen for unlifted types, most importantly) was already there and I only had to refine a conditional here and there. Cheers, Sebastian Am Mi., 10. Feb. 2021 um 18:42 Uhr schrieb Ben Gamari >: Roland Senn > writes: > I hope ticket #19157 will make it in the GHC 9.2 release. In the GHCi > debugger it adds the possibility to set ignore counts to breakpoints. > The next times the break point is reached the program's > execution does not stop. This feature is available in nearly every > debugger, but until now not yet in the GHCi debugger. > Merge request !4839 is ready for review (and it's NOT rocket > science...) > Indeed, this seems quite reasonable. I don't see any reason why we shouldn't be able to get it in to 9.2.1. Cheers, - Ben _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Fri Feb 12 01:07:51 2021 From: ben at well-typed.com (Ben Gamari) Date: Thu, 11 Feb 2021 20:07:51 -0500 Subject: Plan for GHC 9.2 In-Reply-To: References: <3c415670f231bf21a8b3651f1285815e27be859d.camel@bluewin.ch> <87h7mj3jph.fsf@smart-cactus.org> Message-ID: <878s7u2ivy.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Yes I agree, unlifted data types would be terrific. > > From: ghc-devs On Behalf Of Sebastian Graf > Sent: 11 February 2021 10:25 > To: Ben Gamari > Cc: ghc-devs > Subject: Re: Plan for GHC 9.2 > > Hi, > > Since my hopes of finally merging Nested CPR have recently been crushed again, I hope that we can include the implementation of the UnliftedDatatypes extension (proposal, implementation). > It was on ice since it depends on the BoxedRep proposal, but if > BoxedRep is going to make it, surely UnliftedDatatypes can make it, > too. > > I expect quite a few bugs, simply because I don't have much code to > test it on yet. But I'm very confident that existing code isn't > impacted by that, as most of the functionality (CodeGen for unlifted > types, most importantly) was already there and I only had to refine a > conditional here and there. > It would indeed be very exciting to finally have UnliftedDataTypes. If things turn out to be non-trivial then I think we should open to letting it slide for 9.4, but otherwise I am open to merging for 9.2 if a patch appeared. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri Feb 12 18:10:50 2021 From: ben at well-typed.com (Ben Gamari) Date: Fri, 12 Feb 2021 13:10:50 -0500 Subject: Boot libraries for GHC 9.2 Message-ID: <8735y12m3e.fsf@smart-cactus.org> Hi all, I am currently in the process of planning out the GHC 9.2 release. As you know, part of this process is working out which core library versions we will ship. My current plan is the following: package maintainer version --------------- ------------------ ----------------- Cabal Mikhail Glushenkov 3.6.0.0 Win32 Tamar Christina 2.10.0.0 array GHC HQ 0.5.3.0 binary Lennart Kolmodin 0.8.7.0 bytestring Andrew Lelechenko 0.11.0.0 (see GHC #19091) containers David Feuer/CLC? 0.6.4.1 deepseq CLC 1.4.4.0 directory CLC 1.3.6.1 exceptions Ed Kmett 0.10.4 filepath Neil Mitchell? 1.4.2.1 haddock Hécate ? haskeline Judah Jacobson 0.8.1.0 hsc2hs GHC HQ 0.68.7 mtl Ed Kmett 2.2.2 parsec Herbert Valerio Reidel 3.1.14 pretty David Terei? 1.1.3.6 process Michael Snoyman 1.6.10.0 stm GHC HQ 2.5.0.0 terminfo Judah Jacobson 0.4.1.4 text CLC 1.2.4.0 time Ashley Yakeley 1.11.1 transformers Ross Paterson 0.5.6.2 unix Andrew Lelechenko 2.8.0.0? xhtml Chris Dornan 3000.2.2.1 If you disagree with this plan or see a question mark next to a library that you are responsible for, please do be in touch. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Sat Feb 13 02:12:23 2021 From: ben at well-typed.com (Ben Gamari) Date: Fri, 12 Feb 2021 21:12:23 -0500 Subject: Slowing down churn in master prior to 9.2 freeze Message-ID: <87pn141zsr.fsf@smart-cactus.org> Hi all, Since we are in the final weeks of preparation before the 9.2 branch it would be appreciated if we could refrain from pushing work that isn't either a bug-fix or release-critical feature starting next week, especially where the work requires coordination across submodules. At the moment we have quite a backlog in the merge queue, with a number of dependent patches touching submodules. Consequently, I am a tad worried that the three weeks of runway that we currently have may prove to be insufficient unless we try to keep the merge traffic down. I suggest that we set the milestone of non-release-critical patches to %9.4.1 instead of assigning @marge-bot to allow merging to easily proceed after the branch is cut. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Sun Feb 14 23:37:45 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Sun, 14 Feb 2021 23:37:45 +0000 Subject: gitlab.haskell.org certificate expired? Message-ID: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> Hi Ben, It looks like the Let's Encrypt certificate for gitlab.haskell.org has expired, as of about 15 minutes ago. I guess it's time to renew. Thanks, Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From dxld at darkboxed.org Sun Feb 14 23:46:40 2021 From: dxld at darkboxed.org (Daniel =?utf-8?Q?Gr=C3=B6ber?=) Date: Mon, 15 Feb 2021 00:46:40 +0100 Subject: gitlab.haskell.org certificate expired? In-Reply-To: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> References: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> Message-ID: <20210214234640.wpvumq6uknzxveem@darkboxed.org> Hi, indeed looks to be broken, even though my browser still doesn't complain the openssl command sure does: $ openssl s_client -showcerts -verify_return_error -4 -connect gitlab.haskell.org:443 < /dev/null CONNECTED(00000003) depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 verify return:1 depth=0 CN = gitlab.haskell.org verify error:num=10:certificate has expired notAfter=Feb 14 23:21:04 2021 GMT 140217764021376:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915: --- no peer certificate available --- No client certificate CA names sent Server Temp Key: X25519, 253 bits --- SSL handshake has read 2594 bytes and written 317 bytes Verification error: certificate has expired --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 10 (certificate has expired) --- FYI I wrote a super simple monitoring script using faketime+openssl to prevent this sort of thing from happening in case you guys are interested: https://meta.it-syndikat.org/t/tls-monitoring-fur-unsere-infrastruktur/2492 The description is in German unfortunately, but the script itself is commented in English of course ;) We install this as a cron.daily job and use a cron monitoring make sure the script runs, but I suspect if you're not worried about the "it actually ran" part cron's default emails would work just as well. --Daniel On Sun, Feb 14, 2021 at 11:37:45PM +0000, Richard Eisenberg wrote: > Hi Ben, > > It looks like the Let's Encrypt certificate for gitlab.haskell.org has expired, as of about 15 minutes ago. I guess it's time to renew. > > Thanks, > Richard From ben at well-typed.com Mon Feb 15 02:39:47 2021 From: ben at well-typed.com (Ben Gamari) Date: Sun, 14 Feb 2021 21:39:47 -0500 Subject: gitlab.haskell.org certificate expired? In-Reply-To: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> References: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> Message-ID: <87mtw612c2.fsf@smart-cactus.org> Richard Eisenberg writes: > Hi Ben, > > It looks like the Let's Encrypt certificate for gitlab.haskell.org has expired, as of about 15 minutes ago. I guess it's time to renew. > Thanks for the ping. In principle this happens automatically but it seems that we were hit by a NixOS bug [1]. Anyways, I've worked around it for now and things should be back to normal. Cheers, - Ben [1] https://github.com/NixOS/nixpkgs/issues/101445 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Mon Feb 15 02:41:07 2021 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 14 Feb 2021 21:41:07 -0500 Subject: gitlab.haskell.org certificate expired? In-Reply-To: <20210214234640.wpvumq6uknzxveem@darkboxed.org> References: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> <20210214234640.wpvumq6uknzxveem@darkboxed.org> Message-ID: <87k0ra129o.fsf@smart-cactus.org> Daniel Gröber writes: > Hi, > > indeed looks to be broken, even though my browser still doesn't complain > the openssl command sure does: > For the record, the problem was a NixOS bug [1] which resulted in the automated renewal failing. The problem has been worked around for now and should be fixed upstream soon. Cheers, - Ben [1] https://github.com/NixOS/nixpkgs/issues/101445 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From arian.vanputten at gmail.com Mon Feb 15 07:59:44 2021 From: arian.vanputten at gmail.com (Arian van Putten) Date: Mon, 15 Feb 2021 08:59:44 +0100 Subject: gitlab.haskell.org certificate expired? In-Reply-To: <87mtw612c2.fsf@smart-cactus.org> References: <010f0177a2e83e7e-5981ef37-51d2-4658-9fdd-bebe8c9be790-000000@us-east-2.amazonses.com> <87mtw612c2.fsf@smart-cactus.org> Message-ID: Sorry to hear that you hit this bug. We (NixOS acme team) just merged a backport fix for this. Should be in your NixOS channel once hydra picks it up. https://github.com/NixOS/nixpkgs/pull/106857 On Mon, 15 Feb 2021, 03:40 Ben Gamari, wrote: > Richard Eisenberg writes: > > > Hi Ben, > > > > It looks like the Let's Encrypt certificate for gitlab.haskell.org < > http://gitlab.haskell.org/> has expired, as of about 15 minutes ago. I > guess it's time to renew. > > > Thanks for the ping. In principle this happens automatically but it > seems that we were hit by a NixOS bug [1]. Anyways, I've worked around > it for now and things should be back to normal. > > Cheers, > > - Ben > > [1] https://github.com/NixOS/nixpkgs/issues/101445 > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Feb 15 09:27:03 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Feb 2021 09:27:03 +0000 Subject: cep_app traces Message-ID: Ben I'm getting a lot of cpe_app(keepAlive#) 3 cpe_app(keepAlive) trace messages from HEAD. Maybe it's a leftover from tracing when you were developing? Remove? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Feb 15 10:33:28 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 15 Feb 2021 10:33:28 +0000 Subject: Plan for GHC 9.2 In-Reply-To: <87v9b7slvf.fsf@smart-cactus.org> References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: Ben Can we get record dot syntax into 9.2? * Shayne is really nearly there in !4532; he has been working hard and recently. * It depends on my !4981 (was 4722) which fixes some bugs and I'm keen to commit. So, is it ok in principle to pull to trigger on !4981, and hopefully !4532? Simon | -----Original Message----- | From: ghc-devs On Behalf Of Ben Gamari | Sent: 04 February 2021 18:56 | To: GHC developers | Subject: Plan for GHC 9.2 | | | tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, | especially if you have something you would like merged for | 9.2.1. | | Hello all, | | With GHC 9.0.1 at long-last out the door, it is time that we start | turning attention to GHC 9.2. I would like to avoid making the mistake | made in the 9.0 series in starting the fork in a state that required a | significant amount of backporting to be releaseable. Consequently, I | want to make sure that we have a fork schedule that is realistic given | the things that need to be merged for 9.2. These include: | | * Update haddock submodule in `master` (Ben) | * Bumping bytestring to 0.11 (#19091, Ben) | * Finishing the rework of sized integer primops (#19026, John | Ericson) | * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) | * Merge BoxedRep (#17526, Ben) | * ARM NCG backend and further stabilize Apple ARM support? (Moritz) | * Some form of coercion zapping (Ben, Simon, Richard) | * Tag inference analysis and tag check elision (Andreas) | | If you see something that you would like to see in 9.2.1 please do | holler. Otherwise, if you see your name in this list it would be great | if you could let me know when you think your project may be in a | mergeable state. | | Ideally we would strive for a schedule like the following: | | 4 February 2021: We are here | ~4 weeks pass | 3 March 2021: Release branch forked | 1 week passes | 10 March 2021: Alpha 1 released | 3 weeks pass | 31 March 2021: Alpha 2 released | 2 weeks pass | 14 April 2021: Alpha 3 released | 2 weeks pass | 28 April 2021: Alpha 4 released | 1 week passes | 5 May 2021: Beta 1 released | 1 week passes | 12 May 2021: Release candidate 1 released | 2 weeks pass | 26 May 2021: Final release | | This provides ample time for stabilization while avoiding deviation | from the usual May release timeframe. However, this would require that | we move aggressively to start getting the tree into shape since the | fork would be less than four weeks away. I would appreciate | contributors' | thoughts on the viability of this timeline. | | Cheers, | | - Ben From adam at well-typed.com Mon Feb 15 10:42:56 2021 From: adam at well-typed.com (Adam Gundry) Date: Mon, 15 Feb 2021 10:42:56 +0000 Subject: Plan for GHC 9.2 In-Reply-To: References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: <114d408d-a273-42b9-339b-a4337b710256@well-typed.com> [Re-sending from the correct address, apologies!] It would be great to get RecordDotSyntax for selection into 9.2. As I just commented on !4532 [1] there's one awkward point to resolve, which is that 9.2 will probably not have `setField`, on which RecordDotSyntax updates depend. Cheers, Adam [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4532#note_330581) On 15/02/2021 10:33, Simon Peyton Jones via ghc-devs wrote: > Ben > > Can we get record dot syntax into 9.2? > > * Shayne is really nearly there in !4532; he has been working > hard and recently. > * It depends on my !4981 (was 4722) which fixes some bugs and > I'm keen to commit. > > > So, is it ok in principle to pull to trigger on !4981, and hopefully !4532? > > Simon > > | -----Original Message----- > | From: ghc-devs On Behalf Of Ben Gamari > | Sent: 04 February 2021 18:56 > | To: GHC developers > | Subject: Plan for GHC 9.2 > | > | > | tl;dr. Provisional release schedule for 9.2 enclosed. Please discuss, > | especially if you have something you would like merged for > | 9.2.1. > | > | Hello all, > | > | With GHC 9.0.1 at long-last out the door, it is time that we start > | turning attention to GHC 9.2. I would like to avoid making the mistake > | made in the 9.0 series in starting the fork in a state that required a > | significant amount of backporting to be releaseable. Consequently, I > | want to make sure that we have a fork schedule that is realistic given > | the things that need to be merged for 9.2. These include: > | > | * Update haddock submodule in `master` (Ben) > | * Bumping bytestring to 0.11 (#19091, Ben) > | * Finishing the rework of sized integer primops (#19026, John > | Ericson) > | * Merge of ghc-exactprint into GHC? (Alan Zimmerman, Henry) > | * Merge BoxedRep (#17526, Ben) > | * ARM NCG backend and further stabilize Apple ARM support? (Moritz) > | * Some form of coercion zapping (Ben, Simon, Richard) > | * Tag inference analysis and tag check elision (Andreas) > | > | If you see something that you would like to see in 9.2.1 please do > | holler. Otherwise, if you see your name in this list it would be great > | if you could let me know when you think your project may be in a > | mergeable state. > | > | Ideally we would strive for a schedule like the following: > | > | 4 February 2021: We are here > | ~4 weeks pass > | 3 March 2021: Release branch forked > | 1 week passes > | 10 March 2021: Alpha 1 released > | 3 weeks pass > | 31 March 2021: Alpha 2 released > | 2 weeks pass > | 14 April 2021: Alpha 3 released > | 2 weeks pass > | 28 April 2021: Alpha 4 released > | 1 week passes > | 5 May 2021: Beta 1 released > | 1 week passes > | 12 May 2021: Release candidate 1 released > | 2 weeks pass > | 26 May 2021: Final release > | > | This provides ample time for stabilization while avoiding deviation > | from the usual May release timeframe. However, this would require that > | we move aggressively to start getting the tree into shape since the > | fork would be less than four weeks away. I would appreciate > | contributors' > | thoughts on the viability of this timeline. > | > | Cheers, > | > | - Ben -- Adam Gundry, Haskell Consultant Well-Typed LLP, https://www.well-typed.com/ Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England From csaba.hruska at gmail.com Mon Feb 15 15:06:22 2021 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Mon, 15 Feb 2021 16:06:22 +0100 Subject: Weekly show & tell video meeting Message-ID: Hello, Would you be interested in a weekly show & tell video meeting? The topic would be Haskell & compilers in general. Either GHC or non-GHC related. Regards, Csaba -------------- next part -------------- An HTML attachment was scrubbed... URL: From ekmett at gmail.com Mon Feb 15 16:10:59 2021 From: ekmett at gmail.com (Edward Kmett) Date: Mon, 15 Feb 2021 08:10:59 -0800 Subject: Weekly show & tell video meeting In-Reply-To: References: Message-ID: I'd be happy to go with one and see how it goes and plan from there if that works for you. -Edward On Mon, Feb 15, 2021 at 7:06 AM Csaba Hruska wrote: > Hello, > Would you be interested in a weekly show & tell video meeting? > The topic would be Haskell & compilers in general. Either GHC or non-GHC > related. > > Regards, > Csaba > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From csaba.hruska at gmail.com Mon Feb 15 16:18:05 2021 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Mon, 15 Feb 2021 17:18:05 +0100 Subject: Weekly show & tell video meeting In-Reply-To: References: Message-ID: What time works for you? My schedule is flexible. On Mon, Feb 15, 2021 at 5:11 PM Edward Kmett wrote: > I'd be happy to go with one and see how it goes and plan from there if > that works for you. > > -Edward > > On Mon, Feb 15, 2021 at 7:06 AM Csaba Hruska > wrote: > >> Hello, >> Would you be interested in a weekly show & tell video meeting? >> The topic would be Haskell & compilers in general. Either GHC or non-GHC >> related. >> >> Regards, >> Csaba >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Feb 15 16:39:05 2021 From: ben at well-typed.com (Ben Gamari) Date: Mon, 15 Feb 2021 11:39:05 -0500 Subject: Plan for GHC 9.2 In-Reply-To: References: <87v9b7slvf.fsf@smart-cactus.org> Message-ID: <87czx11e1p.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Ben > > Can we get record dot syntax into 9.2? > > * Shayne is really nearly there in !4532; he has been working > hard and recently. Yes, Shayne asked about this last week; I updated the milestone and added it to the milestone highlights [1]. > * It depends on my !4981 (was 4722) which fixes some bugs and > I'm keen to commit. > Alright, let's add it > > So, is it ok in principle to pull to trigger on !4981, and hopefully !4532? Yes, I've added !4981 to the merge queue. !4532 can be merged whenever it is ready. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Mon Feb 15 16:44:02 2021 From: ben at well-typed.com (Ben Gamari) Date: Mon, 15 Feb 2021 11:44:02 -0500 Subject: cep_app traces In-Reply-To: References: Message-ID: <87a6s51dt9.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > Ben > I'm getting a lot of > > cpe_app(keepAlive#) 3 > > cpe_app(keepAlive) > trace messages from HEAD. Maybe it's a leftover from tracing when you were developing? Remove? Oh dear. Yes, it looks like I removed them from the version merged to 9.0 but not from the master version. I'll remove them. Thanks! Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From carter.schonwald at gmail.com Mon Feb 15 17:48:44 2021 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Mon, 15 Feb 2021 12:48:44 -0500 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: <87zh05458o.fsf@sigkill.dk> References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> Message-ID: Ccing ghc devs since that’s a better forum perhaps Crazy theory: this is a regression due the the partial changes to pattern matching coverage checking in 8.10 that finished / landed in ghc 9 Why: Desugaring is when pattern/ case statement translation happens I think? And the only obvious “big thing” is that you have some huge , albeit sane for a compiler, pattern matching I’d first check if the new ghc 9 release doesn’t have that regression in build time that you experienced. And if it does file a ticket. I may be totally wrong, but that seems like a decent likelihood ! On Mon, Feb 15, 2021 at 12:20 PM Troels Henriksen wrote: > Carter Schonwald writes: > > > How big are your data types in the file In question? Do you ghc generics > or > > deriving or template Haskell? Could you share a link to the file in > > question ? > > The file does not define any large data types itself, but it operates on > some fairly large data types (an AST, it's a compiler). But so do many > other modules that work just fine. It uses no generics, nontrivial > deriving, or Template Haskell. It's this file: > > https://github.com/diku-dk/futhark/blob/master/src/Futhark/Pass/ExtractKernels/DistributeNests.hs > > I also found a handful of other modules in my project that are > significantly slower to compile in GHC 8.10, and seemingly also because > of the desugarer, but none of them have any obvious smoking guns like > generics or TH. > > The only commonality I can find is that the affected modules contain > functions with a relatively large typeclass context. I use > ConstraintKinds to make them more concise, and I guess unfolded there > may be 30-40 equality/class constraints in total. Like this: > > type DistLore lore = > ( Bindable lore, > HasSegOp lore, > BinderOps lore, > LetDec lore ~ Type, > ExpDec lore ~ (), > BodyDec lore ~ () > ) > > where the 'Bindable' constraint in particular then implies a further > handful of "~" constraints: > > class > ( ASTLore lore, > FParamInfo lore ~ DeclType, > LParamInfo lore ~ Type, > RetType lore ~ DeclExtType, > BranchType lore ~ ExtType, > SetType (LetDec lore) > ) => > Bindable lore > where > ... > > FParamInfo/LParamInfo/etc are type families. Are such constraints > particularly slow? > > -- > \ Troels > /\ Henriksen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From athas at sigkill.dk Mon Feb 15 18:03:22 2021 From: athas at sigkill.dk (Troels Henriksen) Date: Mon, 15 Feb 2021 19:03:22 +0100 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: (Carter Schonwald's message of "Mon, 15 Feb 2021 12:48:44 -0500") References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> Message-ID: <87tuqd439x.fsf@sigkill.dk> Carter Schonwald writes: > Ccing ghc devs since that’s a better forum perhaps > Crazy theory: > > this is a regression due the the partial changes to pattern matching > coverage checking in 8.10 that finished / landed in ghc 9 > > Why: > Desugaring is when pattern/ case statement translation happens I think? > And the only obvious “big thing” is that you have some huge , albeit sane > for a compiler, pattern matching > > I’d first check if the new ghc 9 release doesn’t have that regression in > build time that you experienced. And if it does file a ticket. > > I may be totally wrong, but that seems like a decent likelihood ! You may be right! Another module that regressed is also mainly characterised by large-but-not-insane case expressions: https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833f39e/src/Futhark/Optimise/Simplify/Rules.hs I'll try to split these modules up a little bit (I should have done so a while ago anyway) and maybe that will make the picture even clearer. -- \ Troels /\ Henriksen From sgraf1337 at gmail.com Mon Feb 15 18:13:00 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Mon, 15 Feb 2021 19:13:00 +0100 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: <87tuqd439x.fsf@sigkill.dk> References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> <87tuqd439x.fsf@sigkill.dk> Message-ID: Hi, I'm not sure I see all the context of the conversation, but it is entirely possible that code with many local constraints regresses the pattern-match checker (which is accounted to Desugaring in the profile emitted by -v2), I'm afraid. That simply has to do with the fact that we now actually care about them, previously they were mostly discarded. I'd be glad if you submitted a relatively isolated reproducer of what is fast with 8.8 and slow with 8.10 (even better 9.0). I hope that things have improved since we fixed https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but not of 8.10. Cheers, Sebastian Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < athas at sigkill.dk>: > Carter Schonwald writes: > > > Ccing ghc devs since that’s a better forum perhaps > > Crazy theory: > > > > this is a regression due the the partial changes to pattern matching > > coverage checking in 8.10 that finished / landed in ghc 9 > > > > Why: > > Desugaring is when pattern/ case statement translation happens I think? > > And the only obvious “big thing” is that you have some huge , albeit sane > > for a compiler, pattern matching > > > > I’d first check if the new ghc 9 release doesn’t have that regression in > > build time that you experienced. And if it does file a ticket. > > > > I may be totally wrong, but that seems like a decent likelihood ! > > You may be right! Another module that regressed is also mainly > characterised by large-but-not-insane case expressions: > > https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833f39e/src/Futhark/Optimise/Simplify/Rules.hs > > I'll try to split these modules up a little bit (I should have done so a > while ago anyway) and maybe that will make the picture even clearer. > > -- > \ Troels > /\ Henriksen > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From athas at sigkill.dk Mon Feb 15 19:10:06 2021 From: athas at sigkill.dk (Troels Henriksen) Date: Mon, 15 Feb 2021 20:10:06 +0100 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: (Sebastian Graf's message of "Mon, 15 Feb 2021 19:13:00 +0100") References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> <87tuqd439x.fsf@sigkill.dk> Message-ID: <87mtw5406p.fsf@sigkill.dk> It is very likely that issue 17386 is the issue. With {-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-} my module(s) compile very quickly. I'll wait and see if GHC 9 does better before I try to create a smaller case (and now I at least have a workaround). Sebastian Graf writes: > Hi, > > I'm not sure I see all the context of the conversation, but it is entirely > possible that code with many local constraints regresses the pattern-match > checker (which is accounted to Desugaring in the profile emitted by -v2), > I'm afraid. That simply has to do with the fact that we now actually care > about them, previously they were mostly discarded. > > I'd be glad if you submitted a relatively isolated reproducer of what is > fast with 8.8 and slow with 8.10 (even better 9.0). > I hope that things have improved since we fixed > https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 but > not of 8.10. > > Cheers, > Sebastian > > Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < > athas at sigkill.dk>: > >> Carter Schonwald writes: >> >> > Ccing ghc devs since that’s a better forum perhaps >> > Crazy theory: >> > >> > this is a regression due the the partial changes to pattern matching >> > coverage checking in 8.10 that finished / landed in ghc 9 >> > >> > Why: >> > Desugaring is when pattern/ case statement translation happens I think? >> > And the only obvious “big thing” is that you have some huge , albeit sane >> > for a compiler, pattern matching >> > >> > I’d first check if the new ghc 9 release doesn’t have that regression in >> > build time that you experienced. And if it does file a ticket. >> > >> > I may be totally wrong, but that seems like a decent likelihood ! >> >> You may be right! Another module that regressed is also mainly >> characterised by large-but-not-insane case expressions: >> >> https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833f39e/src/Futhark/Optimise/Simplify/Rules.hs >> >> I'll try to split these modules up a little bit (I should have done so a >> while ago anyway) and maybe that will make the picture even clearer. >> >> -- >> \ Troels >> /\ Henriksen >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> -- \ Troels /\ Henriksen From carter.schonwald at gmail.com Mon Feb 15 19:21:39 2021 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Mon, 15 Feb 2021 14:21:39 -0500 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: <87mtw5406p.fsf@sigkill.dk> References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> <87tuqd439x.fsf@sigkill.dk> <87mtw5406p.fsf@sigkill.dk> Message-ID: Don’t forget ghc 9 is already out! :) On Mon, Feb 15, 2021 at 2:10 PM Troels Henriksen wrote: > It is very likely that issue 17386 is the issue. With > > {-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns > -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-} > > my module(s) compile very quickly. I'll wait and see if GHC 9 does > better before I try to create a smaller case (and now I at least have a > workaround). > > Sebastian Graf writes: > > > Hi, > > > > I'm not sure I see all the context of the conversation, but it is > entirely > > possible that code with many local constraints regresses the > pattern-match > > checker (which is accounted to Desugaring in the profile emitted by -v2), > > I'm afraid. That simply has to do with the fact that we now actually care > > about them, previously they were mostly discarded. > > > > I'd be glad if you submitted a relatively isolated reproducer of what is > > fast with 8.8 and slow with 8.10 (even better 9.0). > > I hope that things have improved since we fixed > > https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of 9.0 > but > > not of 8.10. > > > > Cheers, > > Sebastian > > > > Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < > > athas at sigkill.dk>: > > > >> Carter Schonwald writes: > >> > >> > Ccing ghc devs since that’s a better forum perhaps > >> > Crazy theory: > >> > > >> > this is a regression due the the partial changes to pattern matching > >> > coverage checking in 8.10 that finished / landed in ghc 9 > >> > > >> > Why: > >> > Desugaring is when pattern/ case statement translation happens I > think? > >> > And the only obvious “big thing” is that you have some huge , albeit > sane > >> > for a compiler, pattern matching > >> > > >> > I’d first check if the new ghc 9 release doesn’t have that regression > in > >> > build time that you experienced. And if it does file a ticket. > >> > > >> > I may be totally wrong, but that seems like a decent likelihood ! > >> > >> You may be right! Another module that regressed is also mainly > >> characterised by large-but-not-insane case expressions: > >> > >> > https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833f39e/src/Futhark/Optimise/Simplify/Rules.hs > >> > >> I'll try to split these modules up a little bit (I should have done so a > >> while ago anyway) and maybe that will make the picture even clearer. > >> > >> -- > >> \ Troels > >> /\ Henriksen > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >> > > -- > \ Troels > /\ Henriksen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allbery.b at gmail.com Mon Feb 15 19:23:14 2021 From: allbery.b at gmail.com (Brandon Allbery) Date: Mon, 15 Feb 2021 14:23:14 -0500 Subject: Pattern matching desugaring regression? Re: [Haskell-cafe] Why does my module take so long to compile? In-Reply-To: References: <87y2fp5pn7.fsf@sigkill.dk> <57fb4de7-d4c6-f08c-d226-18d1572d26b@henning-thielemann.de> <87pn115kmo.fsf@sigkill.dk> <87zh05458o.fsf@sigkill.dk> <87tuqd439x.fsf@sigkill.dk> <87mtw5406p.fsf@sigkill.dk> Message-ID: They already said something about waiting on dependencies to catch up with ghc9, IIRC. On Mon, Feb 15, 2021 at 2:22 PM Carter Schonwald wrote: > Don’t forget ghc 9 is already out! :) > > On Mon, Feb 15, 2021 at 2:10 PM Troels Henriksen wrote: > >> It is very likely that issue 17386 is the issue. With >> >> {-# OPTIONS_GHC -Wno-overlapping-patterns -Wno-incomplete-patterns >> -Wno-incomplete-uni-patterns -Wno-incomplete-record-updates #-} >> >> my module(s) compile very quickly. I'll wait and see if GHC 9 does >> better before I try to create a smaller case (and now I at least have a >> workaround). >> >> Sebastian Graf writes: >> >> > Hi, >> > >> > I'm not sure I see all the context of the conversation, but it is >> entirely >> > possible that code with many local constraints regresses the >> pattern-match >> > checker (which is accounted to Desugaring in the profile emitted by >> -v2), >> > I'm afraid. That simply has to do with the fact that we now actually >> care >> > about them, previously they were mostly discarded. >> > >> > I'd be glad if you submitted a relatively isolated reproducer of what is >> > fast with 8.8 and slow with 8.10 (even better 9.0). >> > I hope that things have improved since we fixed >> > https://gitlab.haskell.org/ghc/ghc/-/issues/17836, which is part of >> 9.0 but >> > not of 8.10. >> > >> > Cheers, >> > Sebastian >> > >> > Am Mo., 15. Feb. 2021 um 19:04 Uhr schrieb Troels Henriksen < >> > athas at sigkill.dk>: >> > >> >> Carter Schonwald writes: >> >> >> >> > Ccing ghc devs since that’s a better forum perhaps >> >> > Crazy theory: >> >> > >> >> > this is a regression due the the partial changes to pattern matching >> >> > coverage checking in 8.10 that finished / landed in ghc 9 >> >> > >> >> > Why: >> >> > Desugaring is when pattern/ case statement translation happens I >> think? >> >> > And the only obvious “big thing” is that you have some huge , albeit >> sane >> >> > for a compiler, pattern matching >> >> > >> >> > I’d first check if the new ghc 9 release doesn’t have that >> regression in >> >> > build time that you experienced. And if it does file a ticket. >> >> > >> >> > I may be totally wrong, but that seems like a decent likelihood ! >> >> >> >> You may be right! Another module that regressed is also mainly >> >> characterised by large-but-not-insane case expressions: >> >> >> >> >> https://github.com/diku-dk/futhark/blob/d0839412bdd11884d75a1494dd5de5191833f39e/src/Futhark/Optimise/Simplify/Rules.hs >> >> >> >> I'll try to split these modules up a little bit (I should have done so >> a >> >> while ago anyway) and maybe that will make the picture even clearer. >> >> >> >> -- >> >> \ Troels >> >> /\ Henriksen >> >> _______________________________________________ >> >> ghc-devs mailing list >> >> ghc-devs at haskell.org >> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> >> -- >> \ Troels >> /\ Henriksen >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -- brandon s allbery kf8nh allbery.b at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From csaba.hruska at gmail.com Tue Feb 16 19:20:29 2021 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Tue, 16 Feb 2021 20:20:29 +0100 Subject: Weekly show & tell video meeting In-Reply-To: References: Message-ID: Hello, The Haskell & Compilers Show & Tell meeting will be on Sunday 19:00 UTC via google meet. https://meet.google.com/qst-sfid-feu Cheers, Csaba On Mon, Feb 15, 2021 at 5:18 PM Csaba Hruska wrote: > What time works for you? > My schedule is flexible. > > On Mon, Feb 15, 2021 at 5:11 PM Edward Kmett wrote: > >> I'd be happy to go with one and see how it goes and plan from there if >> that works for you. >> >> -Edward >> >> On Mon, Feb 15, 2021 at 7:06 AM Csaba Hruska >> wrote: >> >>> Hello, >>> Would you be interested in a weekly show & tell video meeting? >>> The topic would be Haskell & compilers in general. Either GHC or non-GHC >>> related. >>> >>> Regards, >>> Csaba >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Wed Feb 17 08:10:52 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 17 Feb 2021 16:10:52 +0800 Subject: On CI Message-ID: Friends, I've been looking at CI recently again, as I was facing CI turnaround times of 9-12hs; and this just keeps dragging out and making progress hard. The pending pipeline currently has 2 darwin, and 15 windows builds waiting. Windows builds on average take ~220minutes. We have five builders, so we can expect this queue to be done in ~660 minutes assuming perfect scheduling and good performance. That is 11hs! The next windows build can be started in 11hs. Please check my math and tell me I'm wrong! If you submit a MR today, with some luck, you'll be able to know if it will be mergeable some time tomorrow. At which point you can assign it to marge, and marge, if you are lucky and the set of patches she tries to merge together is mergeable, will merge you work into master probably some time on Friday. If a job fails, well you have to start over again. What are our options here? Ben has been pretty clear about not wanting a broken commit for windows to end up in the tree, and I'm there with him. Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Wed Feb 17 10:31:39 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Wed, 17 Feb 2021 11:31:39 +0100 Subject: On CI In-Reply-To: References: Message-ID: Hi Moritz, I, too, had my gripes with CI turnaround times in the past. Here's a somewhat radical proposal: - Run "full-build" stage builds only on Marge MRs. Then we can assign to Marge much earlier, but probably have to do a bit more of (manual) bisecting of spoiled Marge batches. - I hope this gets rid of a bit of the friction of small MRs. I recently caught myself wanting to do a bunch of small, independent, but related changes as part of the same MR, simply because it's such a hassle to post them in individual MRs right now and also because it steals so much CI capacity. - Regular MRs should still have the ability to easily run individual builds of what is now the "full-build" stage, similar to how we can run optional "hackage" builds today. This is probably useful to pin down the reason for a spoiled Marge batch. - The CI capacity we free up can probably be used to run a perf build (such as the fedora release build) on the "build" stage (the one where we currently run stack-hadrian-build and the validate-deb9-hadrian build), in parallel. - If we decide against the latter, a micro-optimisation could be to cache the build artifacts of the "lint-base" build and continue the build in the validate-deb9-hadrian build of the "build" stage. The usefulness of this approach depends on how many MRs cause metric changes on different architectures. Another frustrating aspect is that if you want to merge an n-sized chain of dependent changes individually, you have to - Open an MR for each change (initially the last change will be comprised of n commits) - Review first change, turn pipeline green (A) - Assign to Marge, wait for batch to be merged (B) - Review second change, turn pipeline green - Assign to Marge, wait for batch to be merged - ... and so on ... Note that (A) incurs many context switches for the dev and the latency of *at least* one run of CI. And then (B) incurs the latency of *at least* one full-build, if you're lucky and the batch succeeds. I've recently seen batches that were resubmitted by Marge at least 5 times due to spurious CI failures and timeouts. I think this is a huge factor for latency. Although after (A), I should just pop the the patch off my mental stack, that isn't particularly true, because Marge keeps on reminding me when a stack fails or succeeds, both of which require at least some attention from me: Failed 2 times => Make sure it was spurious, Succeeds => Rebase next change. Maybe we can also learn from other projects like Rust, GCC or clang, which I haven't had a look at yet. Cheers, Sebastian Am Mi., 17. Feb. 2021 um 09:11 Uhr schrieb Moritz Angermann < moritz.angermann at gmail.com>: > Friends, > > I've been looking at CI recently again, as I was facing CI turnaround > times of 9-12hs; and this just keeps dragging out and making progress hard. > > The pending pipeline currently has 2 darwin, and 15 windows builds > waiting. Windows builds on average take ~220minutes. We have five builders, > so we can expect this queue to be done in ~660 minutes assuming perfect > scheduling and good performance. That is 11hs! The next windows build can > be started in 11hs. Please check my math and tell me I'm wrong! > > If you submit a MR today, with some luck, you'll be able to know if it > will be mergeable some time tomorrow. At which point you can assign it to > marge, and marge, if you are lucky and the set of patches she tries to > merge together is mergeable, will merge you work into master probably some > time on Friday. If a job fails, well you have to start over again. > > What are our options here? Ben has been pretty clear about not wanting a > broken commit for windows to end up in the tree, and I'm there with him. > > Cheers, > Moritz > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Wed Feb 17 10:58:34 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 17 Feb 2021 18:58:34 +0800 Subject: On CI In-Reply-To: References: Message-ID: At this point I believe we have ample Linux build capacity. Darwin looks pretty good as well the ~4 M1s we have should in principle also be able to build x86_64-darwin at acceptable speeds. Although on Big Sur only. The aarch64-Linux story is a bit constraint by powerful and fast CI machines but probabaly bearable for the time being. I doubt anyone really looks at those jobs anyway as they are permitted to fail. If aarch64 would become a bottle neck, I’d be inclined to just disable them. With the NCG soon this will likely become much more bearable as wel, even though we might want to run the nightly llvm builds. To be frank, I don’t see 9.2 happening in two weeks with the current CI. If we subtract aarch64-linux and windows builds we could probably do a full run in less than three hours maybe even less. And that is mostly because we have a serialized pipeline. I have discussed some ideas with Ben on prioritizing the first few stages by the faster ci machines to effectively fail fast and provide feedback. But yes. Working on ghc right now is quite painful due to long and unpredictable CI times. Cheers, Moritz On Wed, 17 Feb 2021 at 6:31 PM, Sebastian Graf wrote: > Hi Moritz, > > I, too, had my gripes with CI turnaround times in the past. Here's a > somewhat radical proposal: > > - Run "full-build" stage builds only on Marge MRs. Then we can assign > to Marge much earlier, but probably have to do a bit more of (manual) > bisecting of spoiled Marge batches. > - I hope this gets rid of a bit of the friction of small MRs. I > recently caught myself wanting to do a bunch of small, independent, but > related changes as part of the same MR, simply because it's such a hassle > to post them in individual MRs right now and also because it steals so much > CI capacity. > - Regular MRs should still have the ability to easily run individual > builds of what is now the "full-build" stage, similar to how we can run > optional "hackage" builds today. This is probably useful to pin down the > reason for a spoiled Marge batch. > - The CI capacity we free up can probably be used to run a perf build > (such as the fedora release build) on the "build" stage (the one where we > currently run stack-hadrian-build and the validate-deb9-hadrian build), in > parallel. > - If we decide against the latter, a micro-optimisation could be to > cache the build artifacts of the "lint-base" build and continue the build > in the validate-deb9-hadrian build of the "build" stage. > > The usefulness of this approach depends on how many MRs cause metric > changes on different architectures. > > Another frustrating aspect is that if you want to merge an n-sized chain > of dependent changes individually, you have to > > - Open an MR for each change (initially the last change will be > comprised of n commits) > - Review first change, turn pipeline green (A) > - Assign to Marge, wait for batch to be merged (B) > - Review second change, turn pipeline green > - Assign to Marge, wait for batch to be merged > - ... and so on ... > > Note that (A) incurs many context switches for the dev and the latency of > *at least* one run of CI. > And then (B) incurs the latency of *at least* one full-build, if you're > lucky and the batch succeeds. I've recently seen batches that were > resubmitted by Marge at least 5 times due to spurious CI failures and > timeouts. I think this is a huge factor for latency. > > Although after (A), I should just pop the the patch off my mental stack, > that isn't particularly true, because Marge keeps on reminding me when a > stack fails or succeeds, both of which require at least some attention from > me: Failed 2 times => Make sure it was spurious, Succeeds => Rebase next > change. > > Maybe we can also learn from other projects like Rust, GCC or clang, which > I haven't had a look at yet. > > Cheers, > Sebastian > > Am Mi., 17. Feb. 2021 um 09:11 Uhr schrieb Moritz Angermann < > moritz.angermann at gmail.com>: > >> Friends, >> >> I've been looking at CI recently again, as I was facing CI turnaround >> times of 9-12hs; and this just keeps dragging out and making progress hard. >> >> The pending pipeline currently has 2 darwin, and 15 windows builds >> waiting. Windows builds on average take ~220minutes. We have five builders, >> so we can expect this queue to be done in ~660 minutes assuming perfect >> scheduling and good performance. That is 11hs! The next windows build can >> be started in 11hs. Please check my math and tell me I'm wrong! >> >> If you submit a MR today, with some luck, you'll be able to know if it >> will be mergeable some time tomorrow. At which point you can assign it to >> marge, and marge, if you are lucky and the set of patches she tries to >> merge together is mergeable, will merge you work into master probably some >> time on Friday. If a job fails, well you have to start over again. >> >> What are our options here? Ben has been pretty clear about not wanting a >> broken commit for windows to end up in the tree, and I'm there with him. >> >> Cheers, >> Moritz >> > _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Thu Feb 18 09:29:41 2021 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 18 Feb 2021 17:29:41 +0800 Subject: On CI In-Reply-To: References: Message-ID: I'm glad to report that my math was off. But it was off only because I assumed that we'd successfully build all windows configurations, which we of course don't. Thus some builds fail faster. Sylvain also provided a windows machine temporarily, until it expired. This led to a slew of new windows wibbles. The CI script Ben wrote, and generously used to help set up the new builder, seems to assume an older Git install, and thus a path was broken which thankfully to gitlab led to the brilliant error of just stalling. Next up, because we use msys2's pacman to provision the windows builders, and pacman essentially gives us symbols for packages to install, we ended up getting a newer autoconf onto the new builder (and I assume this will happen with any other builders we add as well). This new autoconf (which I've also ran into on the M1s) doesn't like our configure.ac/aclocal.m4 anymore and barfs; I wasn't able to figure out how to force pacman to install an older version and *not* give it some odd version suffix (which prevents it from working as a drop in replacement). In any case we *must* update our autoconf files. So I guess the time is now. On Wed, Feb 17, 2021 at 6:58 PM Moritz Angermann wrote: > At this point I believe we have ample Linux build capacity. Darwin looks > pretty good as well the ~4 M1s we have should in principle also be able to > build x86_64-darwin at acceptable speeds. Although on Big Sur only. > > The aarch64-Linux story is a bit constraint by powerful and fast CI > machines but probabaly bearable for the time being. I doubt anyone really > looks at those jobs anyway as they are permitted to fail. If aarch64 would > become a bottle neck, I’d be inclined to just disable them. With the NCG > soon this will likely become much more bearable as wel, even though we > might want to run the nightly llvm builds. > > To be frank, I don’t see 9.2 happening in two weeks with the current CI. > > If we subtract aarch64-linux and windows builds we could probably do a > full run in less than three hours maybe even less. And that is mostly > because we have a serialized pipeline. I have discussed some ideas with Ben > on prioritizing the first few stages by the faster ci machines to > effectively fail fast and provide feedback. > > But yes. Working on ghc right now is quite painful due to long and > unpredictable CI times. > > Cheers, > Moritz > > On Wed, 17 Feb 2021 at 6:31 PM, Sebastian Graf > wrote: > >> Hi Moritz, >> >> I, too, had my gripes with CI turnaround times in the past. Here's a >> somewhat radical proposal: >> >> - Run "full-build" stage builds only on Marge MRs. Then we can assign >> to Marge much earlier, but probably have to do a bit more of (manual) >> bisecting of spoiled Marge batches. >> - I hope this gets rid of a bit of the friction of small MRs. I >> recently caught myself wanting to do a bunch of small, independent, but >> related changes as part of the same MR, simply because it's such a hassle >> to post them in individual MRs right now and also because it steals so much >> CI capacity. >> - Regular MRs should still have the ability to easily run individual >> builds of what is now the "full-build" stage, similar to how we can run >> optional "hackage" builds today. This is probably useful to pin down the >> reason for a spoiled Marge batch. >> - The CI capacity we free up can probably be used to run a perf build >> (such as the fedora release build) on the "build" stage (the one where we >> currently run stack-hadrian-build and the validate-deb9-hadrian build), in >> parallel. >> - If we decide against the latter, a micro-optimisation could be to >> cache the build artifacts of the "lint-base" build and continue the build >> in the validate-deb9-hadrian build of the "build" stage. >> >> The usefulness of this approach depends on how many MRs cause metric >> changes on different architectures. >> >> Another frustrating aspect is that if you want to merge an n-sized chain >> of dependent changes individually, you have to >> >> - Open an MR for each change (initially the last change will be >> comprised of n commits) >> - Review first change, turn pipeline green (A) >> - Assign to Marge, wait for batch to be merged (B) >> - Review second change, turn pipeline green >> - Assign to Marge, wait for batch to be merged >> - ... and so on ... >> >> Note that (A) incurs many context switches for the dev and the latency of >> *at least* one run of CI. >> And then (B) incurs the latency of *at least* one full-build, if you're >> lucky and the batch succeeds. I've recently seen batches that were >> resubmitted by Marge at least 5 times due to spurious CI failures and >> timeouts. I think this is a huge factor for latency. >> >> Although after (A), I should just pop the the patch off my mental stack, >> that isn't particularly true, because Marge keeps on reminding me when a >> stack fails or succeeds, both of which require at least some attention from >> me: Failed 2 times => Make sure it was spurious, Succeeds => Rebase next >> change. >> >> Maybe we can also learn from other projects like Rust, GCC or clang, >> which I haven't had a look at yet. >> >> Cheers, >> Sebastian >> >> Am Mi., 17. Feb. 2021 um 09:11 Uhr schrieb Moritz Angermann < >> moritz.angermann at gmail.com>: >> >>> Friends, >>> >>> I've been looking at CI recently again, as I was facing CI turnaround >>> times of 9-12hs; and this just keeps dragging out and making progress hard. >>> >>> The pending pipeline currently has 2 darwin, and 15 windows builds >>> waiting. Windows builds on average take ~220minutes. We have five builders, >>> so we can expect this queue to be done in ~660 minutes assuming perfect >>> scheduling and good performance. That is 11hs! The next windows build can >>> be started in 11hs. Please check my math and tell me I'm wrong! >>> >>> If you submit a MR today, with some luck, you'll be able to know if it >>> will be mergeable some time tomorrow. At which point you can assign it to >>> marge, and marge, if you are lucky and the set of patches she tries to >>> merge together is mergeable, will merge you work into master probably some >>> time on Friday. If a job fails, well you have to start over again. >>> >>> What are our options here? Ben has been pretty clear about not wanting a >>> broken commit for windows to end up in the tree, and I'm there with him. >>> >>> Cheers, >>> Moritz >>> >> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Thu Feb 18 22:34:35 2021 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 18 Feb 2021 17:34:35 -0500 Subject: On CI In-Reply-To: References: Message-ID: <87h7m9yphj.fsf@smart-cactus.org> Apologies for the latency here. This thread has required a fair amount of reflection. Sebastian Graf writes: > Hi Moritz, > > I, too, had my gripes with CI turnaround times in the past. Here's a > somewhat radical proposal: > > - Run "full-build" stage builds only on Marge MRs. Then we can assign to > Marge much earlier, but probably have to do a bit more of (manual) > bisecting of spoiled Marge batches. > - I hope this gets rid of a bit of the friction of small MRs. I > recently caught myself wanting to do a bunch of small, independent, but > related changes as part of the same MR, simply because it's such a hassle > to post them in individual MRs right now and also because it > steals so much CI capacity. > > - Regular MRs should still have the ability to easily run individual > builds of what is now the "full-build" stage, similar to how we can run > optional "hackage" builds today. This is probably useful to pin down the > reason for a spoiled Marge batch. I am torn here. For most of my non-trivial patches I personally don't mind long turnarounds: I walk away and return a day later to see whether anything failed. Spurious failures due to fragile tests make this a bit tiresome, but this is a problem that we are gradually solving (by fixing bugs and marking tests as fragile). However, I agree that small MRs are currently rather painful. On the other hand, diagnosing failed Marge batches is *also* rather tiresome. I am worried that by deferring full validation of MRs we will only exacerbate this problem. Furthermore, I worry that by deferring full validation we run the risk of rather *increasing* the MR turnaround time, since there are entire classes of issues that wouldn't be caught until the MR made it to Marge. Ultimately it's unclear to me whether this proposal would help or hurt. Nevertheless, I am willing to try it. However, if we go this route we should consider what can be done to reduce the incidence of failed Marge batches. One problem that I'm particularly worried about is that of tests with OS-dependent expected output (e.g. `$test_name.stdout-mingw32). I find that people (understandably) forget to update these when updating test output. I suspect that this will be a frequent source of failed Marge batches if we defer full validation. I can see a few ways that would mitigate this: * eliminate platform-dependent output files * introduce a linter that fails if it sees a test with platform-dependent output that doesn't touch all output files * always run the full-build stage on MRs that touch tests with platform-dependent output files Regardless of whether we implement Sebastian's proposal, one smaller measure we could implement to help the problem of small MRs is to introduce some sort of mechanism to mark MRs as "trivial" (e.g. a label or a commit/MR description keyword), which results in the `full-build` being skipped for that MR. Perhaps this would be helpful? > Another frustrating aspect is that if you want to merge an n-sized chain of > dependent changes individually, you have to > > - Open an MR for each change (initially the last change will be > comprised of n commits) > - Review first change, turn pipeline green (A) > - Assign to Marge, wait for batch to be merged (B) > - Review second change, turn pipeline green > - Assign to Marge, wait for batch to be merged > - ... and so on ... > > Note that this (A) incurs many context switches for the dev and the latency of > *at least* one run of CI. > And then (B) incurs the latency of *at least* one full-build, if you're > lucky and the batch succeeds. I've recently seen batches that were > resubmitted by Marge at least 5 times due to spurious CI failures and > timeouts. I think this is a huge factor for latency. > > Although after (A), I should just pop the the patch off my mental stack, > that isn't particularly true, because Marge keeps on reminding me when a > stack fails or succeeds, both of which require at least some attention from > me: Failed 2 times => Make sure it was spurious, Succeeds => Rebase next > change. > > Maybe we can also learn from other projects like Rust, GCC or clang, which > I haven't had a look at yet. > I did a bit of digging on this. * Rust: It appears that Rust's CI scheme is somewhat similar to what you proposed above. They do relatively minimal validation of MRs (e.g. https://github.com/rust-lang/rust/runs/1905017693), with a full-validation for merges (e.g. https://github.com/rust-lang-ci/rust/runs/1925049948). The latter usually takes between 3 and 4 hours, with some jobs taking 5 hours. * GCC: As far as I can tell, gcc doesn't actually have any (functional) continuous integration. Discussions with contributors suggest that some companies that employ contributors might have their own private infrastructure, but I don't believe there is anything public. * LLVM: I can't work out whether/how LLVM validates MRs (their Phabricator instance mentions Buildkite, although it appears to be broken). `master` appears to be minimally checked (only Linux/x86-64) via buildbot (http://lab.llvm.org:8011/#/builders/16/builds/6593). These jobs take between 3 and 4 hours, although it's unclear what one shou * Go: Go's appears to have its own homebrew CI infrastructure (https://build.golang.org/) for comprehensive testing of master it's hard to tell how long these take but it's at least two hours. Code review happens by way of Gerrit with integration with some sort of CI. These runs take between 1 and 3 hours and seem to test a fairly comprehensive set of configurations. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Thu Feb 18 22:40:04 2021 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 18 Feb 2021 17:40:04 -0500 Subject: On CI In-Reply-To: References: Message-ID: <87eehdyp8b.fsf@smart-cactus.org> Moritz Angermann writes: > At this point I believe we have ample Linux build capacity. Darwin looks > pretty good as well the ~4 M1s we have should in principle also be able to > build x86_64-darwin at acceptable speeds. Although on Big Sur only. > > The aarch64-Linux story is a bit constraint by powerful and fast CI > machines but probabaly bearable for the time being. I doubt anyone really > looks at those jobs anyway as they are permitted to fail. For the record, I look at this once in a while to make sure that they haven't broken (and usually pick off one or two failures in the process). > If aarch64 would become a bottle neck, I’d be inclined to just disable > them. With the NCG soon this will likely become much more bearable as > wel, even though we might want to run the nightly llvm builds. > > To be frank, I don’t see 9.2 happening in two weeks with the current CI. > I'm not sure what you mean. Is this in reference to your own 9.2-slated work or the release as a whole? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From john.ericson at obsidian.systems Fri Feb 19 03:18:38 2021 From: john.ericson at obsidian.systems (John Ericson) Date: Thu, 18 Feb 2021 22:18:38 -0500 Subject: On CI In-Reply-To: <87h7m9yphj.fsf@smart-cactus.org> References: <87h7m9yphj.fsf@smart-cactus.org> Message-ID: <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> I am also wary of us to deferring checking whole platforms and what not. I think that's just kicking the can down the road, and will result in more variance and uncertainty. It might be alright for those authoring PRs, but it will make Ben's job keeping the system running even more grueling. Before getting into these complex trade-offs, I think we should focus on the cornerstone issue that CI isn't incremental. 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. 2. We don't cache between jobs. Shake and Make do not enforce dependency soundness, nor cache-correctness when the build plan itself changes, and this had made this hard/impossible to do safely. Naively this only helps with stage 1 and not stage 2, but if we have separate stage 1 and --freeze1 stage 2 builds, both can be incremental. Yes, this is also lossy, but I only see it leading to false failures not false acceptances (if we can also test the stage 1 one), so I consider it safe. MRs that only work with a slow full build because ABI can so indicate. The second, main part is quite hard to tackle, but I strongly believe incrementality is what we need most, and what we should remain focused on. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Feb 19 08:57:26 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 19 Feb 2021 08:57:26 +0000 Subject: On CI In-Reply-To: <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> Message-ID: 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. I don't get this. We have to build GHC before we can test it, don't we? 2 . We don't cache between jobs. This is, I think, the big one. We endlessly build the exact same binaries. There is a problem, though. If we make *any* change in GHC, even a trivial refactoring, its binary will change slightly. So now any caching build system will assume that anything built by that GHC must be rebuilt - we can't use the cached version. That includes all the libraries and the stage2 compiler. So caching can save all the preliminaries (building the initial Cabal, and large chunk of stage1, since they are built with the same bootstrap compiler) but after that we are dead. I don't know any robust way out of this. That small change in the source code of GHC might be trivial refactoring, or it might introduce a critical mis-compilation which we really want to see in its build products. However, for smoke-testing MRs, on every architecture, we could perhaps cut corners. (Leaving Marge to do full diligence.) For example, we could declare that if we have the result of compiling library module X.hs with the stage1 GHC in the last full commit in master, then we can re-use that build product rather than compiling X.hs with the MR's slightly modified stage1 GHC. That *might* be wrong; but it's usually right. Anyway, there are big wins to be had here. Simon From: ghc-devs On Behalf Of John Ericson Sent: 19 February 2021 03:19 To: ghc-devs Subject: Re: On CI I am also wary of us to deferring checking whole platforms and what not. I think that's just kicking the can down the road, and will result in more variance and uncertainty. It might be alright for those authoring PRs, but it will make Ben's job keeping the system running even more grueling. Before getting into these complex trade-offs, I think we should focus on the cornerstone issue that CI isn't incremental. 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. 2. We don't cache between jobs. Shake and Make do not enforce dependency soundness, nor cache-correctness when the build plan itself changes, and this had made this hard/impossible to do safely. Naively this only helps with stage 1 and not stage 2, but if we have separate stage 1 and --freeze1 stage 2 builds, both can be incremental. Yes, this is also lossy, but I only see it leading to false failures not false acceptances (if we can also test the stage 1 one), so I consider it safe. MRs that only work with a slow full build because ABI can so indicate. The second, main part is quite hard to tackle, but I strongly believe incrementality is what we need most, and what we should remain focused on. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Feb 19 10:37:45 2021 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 19 Feb 2021 05:37:45 -0500 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> Message-ID: <871rdcz6kq.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: >> 1. Building and testing happen together. When tests failure >> spuriously, we also have to rebuild GHC in addition to re-running >> the tests. That's pure waste. >> https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more >> or less. > I don't get this. We have to build GHC before we can test it, don't we? >> 2 . We don't cache between jobs. > This is, I think, the big one. We endlessly build the exact same binaries. > There is a problem, though. If we make *any* change in GHC, even a > trivial refactoring, its binary will change slightly. So now any > caching build system will assume that anything built by that GHC must > be rebuilt - we can't use the cached version. That includes all the > libraries and the stage2 compiler. So caching can save all the > preliminaries (building the initial Cabal, and large chunk of stage1, > since they are built with the same bootstrap compiler) but after that > we are dead. > > I don't know any robust way out of this. That small change in the > source code of GHC might be trivial refactoring, or it might introduce > a critical mis-compilation which we really want to see in its build > products. > > However, for smoke-testing MRs, on every architecture, we could > perhaps cut corners. (Leaving Marge to do full diligence.) For > example, we could declare that if we have the result of compiling > library module X.hs with the stage1 GHC in the last full commit in > master, then we can re-use that build product rather than compiling > X.hs with the MR's slightly modified stage1 GHC. That *might* be > wrong; but it's usually right. > The question is: what happens if the it *is* wrong? There are three answers here: a. Allowing the build pipeline to pass despite a build/test failure eliminates most of the benefit of running the job to begin with as allow-failure jobs tend to be ignored. b. Making the pipeline fail leaves the contributor to pick up the pieces of a failure that they may or may not be responsible for, which sounds frustrating indeed. c. Retry the build, but this time from scratch. This is a tantalizing option but carries the risk that we end up doing *more* work than we do now (namely, if all jobs end up running both builds) The only tenable option here in my opinion is (c). It's ugly, but may be viable. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From josefs at fb.com Fri Feb 19 10:40:29 2021 From: josefs at fb.com (Josef Svenningsson) Date: Fri, 19 Feb 2021 10:40:29 +0000 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems>, Message-ID: Doing "optimistic caching" like you suggest sounds very promising. A way to regain more robustness would be as follows. If the build fails while building the libraries or the stage2 compiler, this might be a false negative due to the optimistic caching. Therefore, evict the "optimistic caches" and restart building the libraries. That way we can validate that the build failure was a true build failure and not just due to the aggressive caching scheme. Just my 2p Josef ________________________________ From: ghc-devs on behalf of Simon Peyton Jones via ghc-devs Sent: Friday, February 19, 2021 8:57 AM To: John Ericson ; ghc-devs Subject: RE: On CI 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. I don’t get this. We have to build GHC before we can test it, don’t we? 2 . We don't cache between jobs. This is, I think, the big one. We endlessly build the exact same binaries. There is a problem, though. If we make *any* change in GHC, even a trivial refactoring, its binary will change slightly. So now any caching build system will assume that anything built by that GHC must be rebuilt – we can’t use the cached version. That includes all the libraries and the stage2 compiler. So caching can save all the preliminaries (building the initial Cabal, and large chunk of stage1, since they are built with the same bootstrap compiler) but after that we are dead. I don’t know any robust way out of this. That small change in the source code of GHC might be trivial refactoring, or it might introduce a critical mis-compilation which we really want to see in its build products. However, for smoke-testing MRs, on every architecture, we could perhaps cut corners. (Leaving Marge to do full diligence.) For example, we could declare that if we have the result of compiling library module X.hs with the stage1 GHC in the last full commit in master, then we can re-use that build product rather than compiling X.hs with the MR’s slightly modified stage1 GHC. That *might* be wrong; but it’s usually right. Anyway, there are big wins to be had here. Simon From: ghc-devs On Behalf Of John Ericson Sent: 19 February 2021 03:19 To: ghc-devs Subject: Re: On CI I am also wary of us to deferring checking whole platforms and what not. I think that's just kicking the can down the road, and will result in more variance and uncertainty. It might be alright for those authoring PRs, but it will make Ben's job keeping the system running even more grueling. Before getting into these complex trade-offs, I think we should focus on the cornerstone issue that CI isn't incremental. 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. 2. We don't cache between jobs. Shake and Make do not enforce dependency soundness, nor cache-correctness when the build plan itself changes, and this had made this hard/impossible to do safely. Naively this only helps with stage 1 and not stage 2, but if we have separate stage 1 and --freeze1 stage 2 builds, both can be incremental. Yes, this is also lossy, but I only see it leading to false failures not false acceptances (if we can also test the stage 1 one), so I consider it safe. MRs that only work with a slow full build because ABI can so indicate. The second, main part is quite hard to tackle, but I strongly believe incrementality is what we need most, and what we should remain focused on. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Fri Feb 19 12:20:07 2021 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Fri, 19 Feb 2021 13:20:07 +0100 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> Message-ID: Recompilation avoidance I think in order to cache more in CI, we first have to invest some time in fixing recompilation avoidance in our bootstrapped build system. I just tested on a hadrian perf ticky build: Adding one line of *comment* in the compiler causes - a (pretty slow, yet negligible) rebuild of the stage1 compiler - 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? It doesn't depend in any way on the change I made) - apparent full rebuild the libraries - apparent full rebuild of the stage2 compiler That took 17 minutes, a full build takes ~45minutes. So there definitely is some caching going on, but not nearly as much as there could be. I know there have been great and boring efforts on compiler determinism in the past, but either it's not good enough or our build system needs fixing. I think a good first step to assert would be to make sure that the hash of the stage1 compiler executable doesn't change if I only change a comment. I'm aware there probably is stuff going on, like embedding configure dates in interface files and executables, that would need to go, but if possible this would be a huge improvement. On the other hand, we can simply tack on a [skip ci] to the commit message, as I did for https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975. Variants like [skip tests] or [frontend] could help to identify which tests to run by default. Lean I had a chat with a colleague about how they do CI for Lean. Apparently, CI turnaround time including tests is generally 25 minutes (~15 minutes for the build) for a complete pipeline, testing 6 different OSes and configurations in parallel: https://github.com/leanprover/lean4/actions/workflows/ci.yml They utilise ccache to cache the clang-based C++-backend, so that they only have to re-run the front- and middle-end. In effect, they take advantage of the fact that the "function" clang, in contrast to the "function" stage1 compiler, stays the same. It's hard to achieve that for GHC, where a complete compiler pipeline comes as one big, fused "function": An external tool can never be certain that a change to Parser.y could not affect the CodeGen phase. Inspired by Lean, the following is a bit inconcrete and imaginary, but maybe we could make it so that compiler phases "sign" parts of the interface file with the binary hash of the respective subcomponents of the phase? E.g., if all the object files that influence CodeGen (that will later be linked into the stage1 compiler) result in a hash of 0xdeadbeef before and after the change to Parser.y, we know we can stop recompiling Data.List with the stage1 compiler when we see that the IR passed to CodeGen didn't change, because the last compile did CodeGen with a stage1 compiler with the same hash 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the function CodeGen stayed the same", so we can reuse its cached outputs. Of course, that is utopic without a tool that does the "taint analysis" of which modules in GHC influence CodeGen. Probably just including all the transitive dependencies of GHC.CmmToAsm suffices, but probably that's too crude already. For another example, a change to GHC.Utils.Unique would probably entail a full rebuild of the compiler because it basically affects all compiler phases. There are probably parallels with recompilation avoidance in a language with staged meta-programming. Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via ghc-devs < ghc-devs at haskell.org>: > Doing "optimistic caching" like you suggest sounds very promising. A way > to regain more robustness would be as follows. > If the build fails while building the libraries or the stage2 compiler, > this might be a false negative due to the optimistic caching. Therefore, > evict the "optimistic caches" and restart building the libraries. That way > we can validate that the build failure was a true build failure and not > just due to the aggressive caching scheme. > > Just my 2p > > Josef > > ------------------------------ > *From:* ghc-devs on behalf of Simon Peyton > Jones via ghc-devs > *Sent:* Friday, February 19, 2021 8:57 AM > *To:* John Ericson ; ghc-devs < > ghc-devs at haskell.org> > *Subject:* RE: On CI > > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > > I don’t get this. We have to build GHC before we can test it, don’t we? > > 2 . We don't cache between jobs. > > This is, I think, the big one. We endlessly build the exact same > binaries. > > There is a problem, though. If we make **any** change in GHC, even a > trivial refactoring, its binary will change slightly. So now any caching > build system will assume that anything built by that GHC must be rebuilt – > we can’t use the cached version. That includes all the libraries and the > stage2 compiler. So caching can save all the preliminaries (building the > initial Cabal, and large chunk of stage1, since they are built with the > same bootstrap compiler) but after that we are dead. > > I don’t know any robust way out of this. That small change in the source > code of GHC might be trivial refactoring, or it might introduce a critical > mis-compilation which we really want to see in its build products. > > However, for smoke-testing MRs, on every architecture, we could perhaps > cut corners. (Leaving Marge to do full diligence.) For example, we could > declare that if we have the result of compiling library module X.hs with > the stage1 GHC in the last full commit in master, then we can re-use that > build product rather than compiling X.hs with the MR’s slightly modified > stage1 GHC. That **might** be wrong; but it’s usually right. > > Anyway, there are big wins to be had here. > > Simon > > > > > > > > *From:* ghc-devs *On Behalf Of *John > Ericson > *Sent:* 19 February 2021 03:19 > *To:* ghc-devs > *Subject:* Re: On CI > > > > I am also wary of us to deferring checking whole platforms and what not. I > think that's just kicking the can down the road, and will result in more > variance and uncertainty. It might be alright for those authoring PRs, but > it will make Ben's job keeping the system running even more grueling. > > Before getting into these complex trade-offs, I think we should focus on > the cornerstone issue that CI isn't incremental. > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > 2. We don't cache between jobs. Shake and Make do not enforce > dependency soundness, nor cache-correctness when the build plan itself > changes, and this had made this hard/impossible to do safely. Naively this > only helps with stage 1 and not stage 2, but if we have separate stage 1 > and --freeze1 stage 2 builds, both can be incremental. Yes, this is also > lossy, but I only see it leading to false failures not false acceptances > (if we can also test the stage 1 one), so I consider it safe. MRs that only > work with a slow full build because ABI can so indicate. > > The second, main part is quite hard to tackle, but I strongly believe > incrementality is what we need most, and what we should remain focused on. > > John > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Fri Feb 19 19:42:54 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Fri, 19 Feb 2021 19:42:54 +0000 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> Message-ID: <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> There are some good ideas here, but I want to throw out another one: put all our effort into reducing compile times. There is a loud plea to do this on Discourse , and it would both solve these CI problems and also help everyone else. This isn't to say to stop exploring the ideas here. But since time is mostly fixed, tackling compilation times in general may be the best way out of this. Ben's survey of other projects (thanks!) shows that we're way, way behind in how long our CI takes to run. Richard > On Feb 19, 2021, at 7:20 AM, Sebastian Graf wrote: > > Recompilation avoidance > > I think in order to cache more in CI, we first have to invest some time in fixing recompilation avoidance in our bootstrapped build system. > > I just tested on a hadrian perf ticky build: Adding one line of *comment* in the compiler causes > a (pretty slow, yet negligible) rebuild of the stage1 compiler > 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? It doesn't depend in any way on the change I made) > apparent full rebuild the libraries > apparent full rebuild of the stage2 compiler > That took 17 minutes, a full build takes ~45minutes. So there definitely is some caching going on, but not nearly as much as there could be. > I know there have been great and boring efforts on compiler determinism in the past, but either it's not good enough or our build system needs fixing. > I think a good first step to assert would be to make sure that the hash of the stage1 compiler executable doesn't change if I only change a comment. > I'm aware there probably is stuff going on, like embedding configure dates in interface files and executables, that would need to go, but if possible this would be a huge improvement. > > On the other hand, we can simply tack on a [skip ci] to the commit message, as I did for https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975 . Variants like [skip tests] or [frontend] could help to identify which tests to run by default. > > Lean > > I had a chat with a colleague about how they do CI for Lean. Apparently, CI turnaround time including tests is generally 25 minutes (~15 minutes for the build) for a complete pipeline, testing 6 different OSes and configurations in parallel: https://github.com/leanprover/lean4/actions/workflows/ci.yml > They utilise ccache to cache the clang-based C++-backend, so that they only have to re-run the front- and middle-end. In effect, they take advantage of the fact that the "function" clang, in contrast to the "function" stage1 compiler, stays the same. > It's hard to achieve that for GHC, where a complete compiler pipeline comes as one big, fused "function": An external tool can never be certain that a change to Parser.y could not affect the CodeGen phase. > > Inspired by Lean, the following is a bit inconcrete and imaginary, but maybe we could make it so that compiler phases "sign" parts of the interface file with the binary hash of the respective subcomponents of the phase? > E.g., if all the object files that influence CodeGen (that will later be linked into the stage1 compiler) result in a hash of 0xdeadbeef before and after the change to Parser.y, we know we can stop recompiling Data.List with the stage1 compiler when we see that the IR passed to CodeGen didn't change, because the last compile did CodeGen with a stage1 compiler with the same hash 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the function CodeGen stayed the same", so we can reuse its cached outputs. > Of course, that is utopic without a tool that does the "taint analysis" of which modules in GHC influence CodeGen. Probably just including all the transitive dependencies of GHC.CmmToAsm suffices, but probably that's too crude already. For another example, a change to GHC.Utils.Unique would probably entail a full rebuild of the compiler because it basically affects all compiler phases. > There are probably parallels with recompilation avoidance in a language with staged meta-programming. > > Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via ghc-devs >: > Doing "optimistic caching" like you suggest sounds very promising. A way to regain more robustness would be as follows. > If the build fails while building the libraries or the stage2 compiler, this might be a false negative due to the optimistic caching. Therefore, evict the "optimistic caches" and restart building the libraries. That way we can validate that the build failure was a true build failure and not just due to the aggressive caching scheme. > > Just my 2p > > Josef > > From: ghc-devs > on behalf of Simon Peyton Jones via ghc-devs > > Sent: Friday, February 19, 2021 8:57 AM > To: John Ericson ; ghc-devs > > Subject: RE: On CI > > Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. > I don’t get this. We have to build GHC before we can test it, don’t we? > 2 . We don't cache between jobs. > This is, I think, the big one. We endlessly build the exact same binaries. > There is a problem, though. If we make *any* change in GHC, even a trivial refactoring, its binary will change slightly. So now any caching build system will assume that anything built by that GHC must be rebuilt – we can’t use the cached version. That includes all the libraries and the stage2 compiler. So caching can save all the preliminaries (building the initial Cabal, and large chunk of stage1, since they are built with the same bootstrap compiler) but after that we are dead. > I don’t know any robust way out of this. That small change in the source code of GHC might be trivial refactoring, or it might introduce a critical mis-compilation which we really want to see in its build products. > However, for smoke-testing MRs, on every architecture, we could perhaps cut corners. (Leaving Marge to do full diligence.) For example, we could declare that if we have the result of compiling library module X.hs with the stage1 GHC in the last full commit in master, then we can re-use that build product rather than compiling X.hs with the MR’s slightly modified stage1 GHC. That *might* be wrong; but it’s usually right. > Anyway, there are big wins to be had here. > Simon > > > > From: ghc-devs > On Behalf Of John Ericson > Sent: 19 February 2021 03:19 > To: ghc-devs > > Subject: Re: On CI > > I am also wary of us to deferring checking whole platforms and what not. I think that's just kicking the can down the road, and will result in more variance and uncertainty. It might be alright for those authoring PRs, but it will make Ben's job keeping the system running even more grueling. > > Before getting into these complex trade-offs, I think we should focus on the cornerstone issue that CI isn't incremental. > > Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. > We don't cache between jobs. Shake and Make do not enforce dependency soundness, nor cache-correctness when the build plan itself changes, and this had made this hard/impossible to do safely. Naively this only helps with stage 1 and not stage 2, but if we have separate stage 1 and --freeze1 stage 2 builds, both can be incremental. Yes, this is also lossy, but I only see it leading to false failures not false acceptances (if we can also test the stage 1 one), so I consider it safe. MRs that only work with a slow full build because ABI can so indicate. > The second, main part is quite hard to tackle, but I strongly believe incrementality is what we need most, and what we should remain focused on. > John > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Sun Feb 21 16:24:01 2021 From: ben at well-typed.com (Ben Gamari) Date: Sun, 21 Feb 2021 11:24:01 -0500 Subject: Changes to performance testing? Message-ID: <87im6lxuch.fsf@smart-cactus.org> Hi all, Recently our performance tests have been causing quite some pain. One reason for this is due to our new Darwin runners (see #19025), which (surprisingly) differ significantly in their performance characteristics (perhaps due to running Big Sur or using native tools provided by nix?). However, this is further exacerbated by the fact that there are quite a few people working on compiler performance currently (horray!). This leads to the following failure mode during Marge jobs: 1. Merge request A improves test T1234 by 0.5%, which is within the test's acceptance window and therefore CI passes 2. Merge request B *also* improves test T1234 by another 0.5%, which similarly passes CI 3. Marge tries to merge MRs A and B in a batch but finds that the combined 1% improvement in T1234 is *outside* the acceptance window. Consequently, the batch fails. This is quite painful, especially given that it creates work for those trying to improve GHC (as the saying goes: no good deed goes unpunished). To mitigate this I would suggest that we allow performance test failures in marge-bot pipelines. A slightly weaker variant of this idea would instead only allow performance *improvements*. I suspect the latter would get most of the benefit, while eliminating the possibility that a large regression goes unnoticed. Thoughts? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Sun Feb 21 20:31:46 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Sun, 21 Feb 2021 20:31:46 +0000 Subject: Changes to performance testing? In-Reply-To: <87im6lxuch.fsf@smart-cactus.org> References: <87im6lxuch.fsf@smart-cactus.org> Message-ID: <010f0177c64a7c40-97040c50-92b1-4ad7-841d-8055824a8de2-000000@us-east-2.amazonses.com> > On Feb 21, 2021, at 11:24 AM, Ben Gamari wrote: > > To mitigate this I would suggest that we allow performance test failures > in marge-bot pipelines. A slightly weaker variant of this idea would > instead only allow performance *improvements*. I suspect the latter > would get most of the benefit, while eliminating the possibility that a > large regression goes unnoticed. The value in making performance improvements a test failure is so that patch authors can be informed of what they have done, to make sure it matches expectations. This need can reasonably be satisfied without stopping merging. That is, if Marge can accept performance improvements, while (say) posting to each MR involved that it may have contributed to a performance improvement, then I think we've done our job here. On the other hand, a performance degradation is a bug, just like, say, an error message regression. Even if it's a combination of commits that cause the problem (an actual possibility even for error message regressions), it's still a bug that we need to either fix or accept (balanced out by other improvements). The pain of debugging this scenario might be mitigated if there were a collation of the performance wibbles for each individual commit. This information is, in general, available: each commit passed CI on its own, and so it should be possible to create a little report with its rows being perf tests and its columns being commits or MR #s; each cell in the table would be a percentage regression. If we're lucky, the regression Marge sees will be the sum(*) of the entries in one of the rows -- this means that we have a simple agglomeration of performance degradation. If we're less lucky, the whole will not equal the sum of the parts, and some of the patches interfere. In either case, the table would suggest a likely place to look next. (*) I suppose if we're recording percentages, it wouldn't necessarily be the actual sum, because percentages are a bit funny. But you get my meaning. Pulling this all together: * I'm against the initial proposal of allowing all performance failures by Marge. This will allow bugs to accumulate (in my opinion). * I'm in favor of allowing performance improvements to be accepted by Marge. * To mitigate against the information loss of Marge accepting performance improvements, it would be great if Marge could alert MR authors that a cumulative performance improvement took place. * To mitigate against the annoyance of finding a performance regression in a merge commit that does not appear in any component commit, it would be great if there were a tool to collect performance numbers from a set of commits and present them in a table for further analysis. These "mitigations" might take work. If labor is impossible to produce to complete this work, I'm in favor of simply allowing the performance improvements, maybe also filing a ticket about these potential improvements to the process. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Mon Feb 22 05:53:07 2021 From: john.ericson at obsidian.systems (John Ericson) Date: Mon, 22 Feb 2021 00:53:07 -0500 Subject: On CI In-Reply-To: <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> Message-ID: <6c6a4089-3a50-4555-413d-4e68d366c63f@obsidian.systems> I'm not opposed to some effort going into this, but I would strongly opposite putting all our effort there. Incremental CI can cut multiple hours to < mere minutes, especially with the test suite being embarrassingly parallel. There simply no way optimizations to the compiler independent from sharing a cache between CI runs can get anywhere close to that return on investment. (FWIW, I'm also skeptical that the people complaining about GHC performance know what's hurting them most. For example, after non-incrementality, the next slowest thing is linking, which is...not done by GHC! But all that is a separate conversation.) John On 2/19/21 2:42 PM, Richard Eisenberg wrote: > There are some good ideas here, but I want to throw out another one: > put all our effort into reducing compile times. There is a loud plea > to do this on Discourse > , > and it would both solve these CI problems and also help everyone else. > > This isn't to say to stop exploring the ideas here. But since time is > mostly fixed, tackling compilation times in general may be the best > way out of this. Ben's survey of other projects (thanks!) shows that > we're way, way behind in how long our CI takes to run. > > Richard > >> On Feb 19, 2021, at 7:20 AM, Sebastian Graf > > wrote: >> >> Recompilation avoidance >> >> I think in order to cache more in CI, we first have to invest some >> time in fixing recompilation avoidance in our bootstrapped build system. >> >> I just tested on a hadrian perf ticky build: Adding one line of >> *comment* in the compiler causes >> >> * a (pretty slow, yet negligible) rebuild of the stage1 compiler >> * 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? >> It doesn't depend in any way on the change I made) >> * apparent full rebuild the libraries >> * apparent full rebuild of the stage2 compiler >> >> That took 17 minutes, a full build takes ~45minutes. So there >> definitely is some caching going on, but not nearly as much as there >> could be. >> I know there have been great and boring efforts on compiler >> determinism in the past, but either it's not good enough or our build >> system needs fixing. >> I think a good first step to assert would be to make sure that the >> hash of the stage1 compiler executable doesn't change if I only >> change a comment. >> I'm aware there probably is stuff going on, like embedding configure >> dates in interface files and executables, that would need to go, but >> if possible this would be a huge improvement. >> >> On the other hand, we can simply tack on a [skip ci] to the commit >> message, as I did for >> https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975 >> . Variants >> like [skip tests] or [frontend] could help to identify which tests to >> run by default. >> >> Lean >> >> I had a chat with a colleague about how they do CI for Lean. >> Apparently, CI turnaround time including tests is generally 25 >> minutes (~15 minutes for the build) for a complete pipeline, testing >> 6 different OSes and configurations in parallel: >> https://github.com/leanprover/lean4/actions/workflows/ci.yml >> >> They utilise ccache to cache the clang-based C++-backend, so that >> they only have to re-run the front- and middle-end. In effect, they >> take advantage of the fact that the "function" clang, in contrast to >> the "function" stage1 compiler, stays the same. >> It's hard to achieve that for GHC, where a complete compiler pipeline >> comes as one big, fused "function": An external tool can never be >> certain that a change to Parser.y could not affect the CodeGen phase. >> >> Inspired by Lean, the following is a bit inconcrete and imaginary, >> but maybe we could make it so that compiler phases "sign" parts of >> the interface file with the binary hash of the respective >> subcomponents of the phase? >> E.g., if all the object files that influence CodeGen (that will later >> be linked into the stage1 compiler) result in a hash of 0xdeadbeef >> before and after the change to Parser.y, we know we can stop >> recompiling Data.List with the stage1 compiler when we see that the >> IR passed to CodeGen didn't change, because the last compile did >> CodeGen with a stage1 compiler with the same hash 0xdeadbeef. The >> 0xdeadbeef hash is a proxy for saying "the function CodeGen stayed >> the same", so we can reuse its cached outputs. >> Of course, that is utopic without a tool that does the "taint >> analysis" of which modules in GHC influence CodeGen. Probably just >> including all the transitive dependencies of GHC.CmmToAsm suffices, >> but probably that's too crude already. For another example, a change >> to GHC.Utils.Unique would probably entail a full rebuild of the >> compiler because it basically affects all compiler phases. >> There are probably parallels with recompilation avoidance in a >> language with staged meta-programming. >> >> Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via >> ghc-devs >: >> >> Doing "optimistic caching" like you suggest sounds very >> promising. A way to regain more robustness would be as follows. >> If the build fails while building the libraries or the stage2 >> compiler, this might be a false negative due to the optimistic >> caching. Therefore, evict the "optimistic caches" and restart >> building the libraries. That way we can validate that the build >> failure was a true build failure and not just due to the >> aggressive caching scheme. >> >> Just my 2p >> >> Josef >> >> ------------------------------------------------------------------------ >> *From:* ghc-devs > > on behalf of Simon Peyton >> Jones via ghc-devs > > >> *Sent:* Friday, February 19, 2021 8:57 AM >> *To:* John Ericson > >; ghc-devs >> > >> *Subject:* RE: On CI >> >> 1. Building and testing happen together. When tests failure >> spuriously, we also have to rebuild GHC in addition to >> re-running the tests. That's pure waste. >> https://gitlab.haskell.org/ghc/ghc/-/issues/13897 >> >> tracks this more or less. >> >> I don’t get this.  We have to build GHC before we can test it, >> don’t we? >> 2 .  We don't cache between jobs. >> This is, I think, the big one.   We endlessly build the exact >> same binaries. >> There is a problem, though.  If we make **any** change in GHC, >> even a trivial refactoring, its binary will change slightly.  So >> now any caching build system will assume that anything built by >> that GHC must be rebuilt – we can’t use the cached version.  That >> includes all the libraries and the stage2 compiler.  So caching >> can save all the preliminaries (building the initial Cabal, and >> large chunk of stage1, since they are built with the same >> bootstrap compiler) but after that we are dead. >> I don’t know any robust way out of this.  That small change in >> the source code of GHC might be trivial refactoring, or it might >> introduce a critical mis-compilation which we really want to see >> in its build products. >> However, for smoke-testing MRs, on every architecture, we could >> perhaps cut corners. (Leaving Marge to do full diligence.)  For >> example, we could declare that if we have the result of compiling >> library module X.hs with the stage1 GHC in the last full commit >> in master, then we can re-use that build product rather than >> compiling X.hs with the MR’s slightly modified stage1 GHC.  That >> **might** be wrong; but it’s usually right. >> Anyway, there are big wins to be had here. >> Simon >> >> *From:*ghc-devs > > *On Behalf Of *John Ericson >> *Sent:* 19 February 2021 03:19 >> *To:* ghc-devs > >> *Subject:* Re: On CI >> >> I am also wary of us to deferring checking whole platforms and >> what not. I think that's just kicking the can down the road, and >> will result in more variance and uncertainty. It might be alright >> for those authoring PRs, but it will make Ben's job keeping the >> system running even more grueling. >> >> Before getting into these complex trade-offs, I think we should >> focus on the cornerstone issue that CI isn't incremental. >> >> 1. Building and testing happen together. When tests failure >> spuriously, we also have to rebuild GHC in addition to >> re-running the tests. That's pure waste. >> https://gitlab.haskell.org/ghc/ghc/-/issues/13897 >> >> tracks this more or less. >> 2. We don't cache between jobs. Shake and Make do not enforce >> dependency soundness, nor cache-correctness when the build >> plan itself changes, and this had made this hard/impossible >> to do safely. Naively this only helps with stage 1 and not >> stage 2, but if we have separate stage 1 and --freeze1 stage >> 2 builds, both can be incremental. Yes, this is also lossy, >> but I only see it leading to false failures not false >> acceptances (if we can also test the stage 1 one), so I >> consider it safe. MRs that only work with a slow full build >> because ABI can so indicate. >> >> The second, main part is quite hard to tackle, but I strongly >> believe incrementality is what we need most, and what we should >> remain focused on. >> >> John >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Feb 22 09:04:59 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 22 Feb 2021 09:04:59 +0000 Subject: On CI In-Reply-To: <6c6a4089-3a50-4555-413d-4e68d366c63f@obsidian.systems> References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> <6c6a4089-3a50-4555-413d-4e68d366c63f@obsidian.systems> Message-ID: Incremental CI can cut multiple hours to < mere minutes, especially with the test suite being embarrassingly parallel. There simply no way optimizations to the compiler independent from sharing a cache between CI runs can get anywhere close to that return on investment. I rather agree with this. I don't think there is much low-hanging fruit on compile times, aside from coercion-zapping which we are working on anyway. If we got a 10% reduction in compile time we'd be over the moon, but our users would barely notice. To get truly substantial improvements (a factor of 2 or 10) I think we need to do less compiling - hence incremental CI. Simon From: ghc-devs On Behalf Of John Ericson Sent: 22 February 2021 05:53 To: ghc-devs Subject: Re: On CI I'm not opposed to some effort going into this, but I would strongly opposite putting all our effort there. Incremental CI can cut multiple hours to < mere minutes, especially with the test suite being embarrassingly parallel. There simply no way optimizations to the compiler independent from sharing a cache between CI runs can get anywhere close to that return on investment. (FWIW, I'm also skeptical that the people complaining about GHC performance know what's hurting them most. For example, after non-incrementality, the next slowest thing is linking, which is...not done by GHC! But all that is a separate conversation.) John On 2/19/21 2:42 PM, Richard Eisenberg wrote: There are some good ideas here, but I want to throw out another one: put all our effort into reducing compile times. There is a loud plea to do this on Discourse, and it would both solve these CI problems and also help everyone else. This isn't to say to stop exploring the ideas here. But since time is mostly fixed, tackling compilation times in general may be the best way out of this. Ben's survey of other projects (thanks!) shows that we're way, way behind in how long our CI takes to run. Richard On Feb 19, 2021, at 7:20 AM, Sebastian Graf > wrote: Recompilation avoidance I think in order to cache more in CI, we first have to invest some time in fixing recompilation avoidance in our bootstrapped build system. I just tested on a hadrian perf ticky build: Adding one line of *comment* in the compiler causes * a (pretty slow, yet negligible) rebuild of the stage1 compiler * 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? It doesn't depend in any way on the change I made) * apparent full rebuild the libraries * apparent full rebuild of the stage2 compiler That took 17 minutes, a full build takes ~45minutes. So there definitely is some caching going on, but not nearly as much as there could be. I know there have been great and boring efforts on compiler determinism in the past, but either it's not good enough or our build system needs fixing. I think a good first step to assert would be to make sure that the hash of the stage1 compiler executable doesn't change if I only change a comment. I'm aware there probably is stuff going on, like embedding configure dates in interface files and executables, that would need to go, but if possible this would be a huge improvement. On the other hand, we can simply tack on a [skip ci] to the commit message, as I did for https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975. Variants like [skip tests] or [frontend] could help to identify which tests to run by default. Lean I had a chat with a colleague about how they do CI for Lean. Apparently, CI turnaround time including tests is generally 25 minutes (~15 minutes for the build) for a complete pipeline, testing 6 different OSes and configurations in parallel: https://github.com/leanprover/lean4/actions/workflows/ci.yml They utilise ccache to cache the clang-based C++-backend, so that they only have to re-run the front- and middle-end. In effect, they take advantage of the fact that the "function" clang, in contrast to the "function" stage1 compiler, stays the same. It's hard to achieve that for GHC, where a complete compiler pipeline comes as one big, fused "function": An external tool can never be certain that a change to Parser.y could not affect the CodeGen phase. Inspired by Lean, the following is a bit inconcrete and imaginary, but maybe we could make it so that compiler phases "sign" parts of the interface file with the binary hash of the respective subcomponents of the phase? E.g., if all the object files that influence CodeGen (that will later be linked into the stage1 compiler) result in a hash of 0xdeadbeef before and after the change to Parser.y, we know we can stop recompiling Data.List with the stage1 compiler when we see that the IR passed to CodeGen didn't change, because the last compile did CodeGen with a stage1 compiler with the same hash 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the function CodeGen stayed the same", so we can reuse its cached outputs. Of course, that is utopic without a tool that does the "taint analysis" of which modules in GHC influence CodeGen. Probably just including all the transitive dependencies of GHC.CmmToAsm suffices, but probably that's too crude already. For another example, a change to GHC.Utils.Unique would probably entail a full rebuild of the compiler because it basically affects all compiler phases. There are probably parallels with recompilation avoidance in a language with staged meta-programming. Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via ghc-devs >: Doing "optimistic caching" like you suggest sounds very promising. A way to regain more robustness would be as follows. If the build fails while building the libraries or the stage2 compiler, this might be a false negative due to the optimistic caching. Therefore, evict the "optimistic caches" and restart building the libraries. That way we can validate that the build failure was a true build failure and not just due to the aggressive caching scheme. Just my 2p Josef ________________________________ From: ghc-devs > on behalf of Simon Peyton Jones via ghc-devs > Sent: Friday, February 19, 2021 8:57 AM To: John Ericson >; ghc-devs > Subject: RE: On CI 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. I don't get this. We have to build GHC before we can test it, don't we? 2 . We don't cache between jobs. This is, I think, the big one. We endlessly build the exact same binaries. There is a problem, though. If we make *any* change in GHC, even a trivial refactoring, its binary will change slightly. So now any caching build system will assume that anything built by that GHC must be rebuilt - we can't use the cached version. That includes all the libraries and the stage2 compiler. So caching can save all the preliminaries (building the initial Cabal, and large chunk of stage1, since they are built with the same bootstrap compiler) but after that we are dead. I don't know any robust way out of this. That small change in the source code of GHC might be trivial refactoring, or it might introduce a critical mis-compilation which we really want to see in its build products. However, for smoke-testing MRs, on every architecture, we could perhaps cut corners. (Leaving Marge to do full diligence.) For example, we could declare that if we have the result of compiling library module X.hs with the stage1 GHC in the last full commit in master, then we can re-use that build product rather than compiling X.hs with the MR's slightly modified stage1 GHC. That *might* be wrong; but it's usually right. Anyway, there are big wins to be had here. Simon From: ghc-devs > On Behalf Of John Ericson Sent: 19 February 2021 03:19 To: ghc-devs > Subject: Re: On CI I am also wary of us to deferring checking whole platforms and what not. I think that's just kicking the can down the road, and will result in more variance and uncertainty. It might be alright for those authoring PRs, but it will make Ben's job keeping the system running even more grueling. Before getting into these complex trade-offs, I think we should focus on the cornerstone issue that CI isn't incremental. 1. Building and testing happen together. When tests failure spuriously, we also have to rebuild GHC in addition to re-running the tests. That's pure waste. https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more or less. 2. We don't cache between jobs. Shake and Make do not enforce dependency soundness, nor cache-correctness when the build plan itself changes, and this had made this hard/impossible to do safely. Naively this only helps with stage 1 and not stage 2, but if we have separate stage 1 and --freeze1 stage 2 builds, both can be incremental. Yes, this is also lossy, but I only see it leading to false failures not false acceptances (if we can also test the stage 1 one), so I consider it safe. MRs that only work with a slow full build because ABI can so indicate. The second, main part is quite hard to tackle, but I strongly believe incrementality is what we need most, and what we should remain focused on. John _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaud.spiwack at tweag.io Mon Feb 22 13:42:40 2021 From: arnaud.spiwack at tweag.io (Spiwack, Arnaud) Date: Mon, 22 Feb 2021 14:42:40 +0100 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> <6c6a4089-3a50-4555-413d-4e68d366c63f@obsidian.systems> Message-ID: Let me know if I'm talking nonsense, but I believe that we are building both stages for each architecture and flavour. Do we need to build two stages everywhere? What stops us from building a single stage? And if anything, what can we change to get into a situation where we can? Quite better than reusing build incrementally, is not building at all. On Mon, Feb 22, 2021 at 10:09 AM Simon Peyton Jones via ghc-devs < ghc-devs at haskell.org> wrote: > Incremental CI can cut multiple hours to < mere minutes, especially with > the test suite being embarrassingly parallel. There simply no way > optimizations to the compiler independent from sharing a cache between CI > runs can get anywhere close to that return on investment. > > I rather agree with this. I don’t think there is much low-hanging fruit > on compile times, aside from coercion-zapping which we are working on > anyway. If we got a 10% reduction in compile time we’d be over the moon, > but our users would barely notice. > > > > To get truly substantial improvements (a factor of 2 or 10) I think we > need to do less compiling – hence incremental CI. > > > Simon > > > > *From:* ghc-devs *On Behalf Of *John > Ericson > *Sent:* 22 February 2021 05:53 > *To:* ghc-devs > *Subject:* Re: On CI > > > > I'm not opposed to some effort going into this, but I would strongly > opposite putting all our effort there. Incremental CI can cut multiple > hours to < mere minutes, especially with the test suite being > embarrassingly parallel. There simply no way optimizations to the compiler > independent from sharing a cache between CI runs can get anywhere close to > that return on investment. > > (FWIW, I'm also skeptical that the people complaining about GHC > performance know what's hurting them most. For example, after > non-incrementality, the next slowest thing is linking, which is...not done > by GHC! But all that is a separate conversation.) > > John > > On 2/19/21 2:42 PM, Richard Eisenberg wrote: > > There are some good ideas here, but I want to throw out another one: put > all our effort into reducing compile times. There is a loud plea to do this > on Discourse > , > and it would both solve these CI problems and also help everyone else. > > > > This isn't to say to stop exploring the ideas here. But since time is > mostly fixed, tackling compilation times in general may be the best way out > of this. Ben's survey of other projects (thanks!) shows that we're way, way > behind in how long our CI takes to run. > > > > Richard > > > > On Feb 19, 2021, at 7:20 AM, Sebastian Graf wrote: > > > > Recompilation avoidance > > > > I think in order to cache more in CI, we first have to invest some time in > fixing recompilation avoidance in our bootstrapped build system. > > > > I just tested on a hadrian perf ticky build: Adding one line of *comment* > in the compiler causes > > - a (pretty slow, yet negligible) rebuild of the stage1 compiler > - 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? It > doesn't depend in any way on the change I made) > - apparent full rebuild the libraries > - apparent full rebuild of the stage2 compiler > > That took 17 minutes, a full build takes ~45minutes. So there definitely > is some caching going on, but not nearly as much as there could be. > > I know there have been great and boring efforts on compiler determinism in > the past, but either it's not good enough or our build system needs fixing. > > I think a good first step to assert would be to make sure that the hash of > the stage1 compiler executable doesn't change if I only change a comment. > > I'm aware there probably is stuff going on, like embedding configure dates > in interface files and executables, that would need to go, but if possible > this would be a huge improvement. > > > > On the other hand, we can simply tack on a [skip ci] to the commit > message, as I did for > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975 > . > Variants like [skip tests] or [frontend] could help to identify which tests > to run by default. > > > > Lean > > > > I had a chat with a colleague about how they do CI for Lean. Apparently, > CI turnaround time including tests is generally 25 minutes (~15 minutes for > the build) for a complete pipeline, testing 6 different OSes and > configurations in parallel: > https://github.com/leanprover/lean4/actions/workflows/ci.yml > > > They utilise ccache to cache the clang-based C++-backend, so that they > only have to re-run the front- and middle-end. In effect, they take > advantage of the fact that the "function" clang, in contrast to the > "function" stage1 compiler, stays the same. > > It's hard to achieve that for GHC, where a complete compiler pipeline > comes as one big, fused "function": An external tool can never be certain > that a change to Parser.y could not affect the CodeGen phase. > > > > Inspired by Lean, the following is a bit inconcrete and imaginary, but > maybe we could make it so that compiler phases "sign" parts of the > interface file with the binary hash of the respective subcomponents of the > phase? > > E.g., if all the object files that influence CodeGen (that will later be > linked into the stage1 compiler) result in a hash of 0xdeadbeef before and > after the change to Parser.y, we know we can stop recompiling Data.List > with the stage1 compiler when we see that the IR passed to CodeGen didn't > change, because the last compile did CodeGen with a stage1 compiler with > the same hash 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the > function CodeGen stayed the same", so we can reuse its cached outputs. > > Of course, that is utopic without a tool that does the "taint analysis" of > which modules in GHC influence CodeGen. Probably just including all the > transitive dependencies of GHC.CmmToAsm suffices, but probably that's too > crude already. For another example, a change to GHC.Utils.Unique would > probably entail a full rebuild of the compiler because it basically affects > all compiler phases. > > There are probably parallels with recompilation avoidance in a language > with staged meta-programming. > > > > Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via ghc-devs > : > > Doing "optimistic caching" like you suggest sounds very promising. A way > to regain more robustness would be as follows. > > If the build fails while building the libraries or the stage2 compiler, > this might be a false negative due to the optimistic caching. Therefore, > evict the "optimistic caches" and restart building the libraries. That way > we can validate that the build failure was a true build failure and not > just due to the aggressive caching scheme. > > > > Just my 2p > > > > Josef > > > ------------------------------ > > *From:* ghc-devs on behalf of Simon Peyton > Jones via ghc-devs > *Sent:* Friday, February 19, 2021 8:57 AM > *To:* John Ericson ; ghc-devs < > ghc-devs at haskell.org> > *Subject:* RE: On CI > > > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > > I don’t get this. We have to build GHC before we can test it, don’t we? > > 2 . We don't cache between jobs. > > This is, I think, the big one. We endlessly build the exact same > binaries. > > There is a problem, though. If we make **any** change in GHC, even a > trivial refactoring, its binary will change slightly. So now any caching > build system will assume that anything built by that GHC must be rebuilt – > we can’t use the cached version. That includes all the libraries and the > stage2 compiler. So caching can save all the preliminaries (building the > initial Cabal, and large chunk of stage1, since they are built with the > same bootstrap compiler) but after that we are dead. > > I don’t know any robust way out of this. That small change in the source > code of GHC might be trivial refactoring, or it might introduce a critical > mis-compilation which we really want to see in its build products. > > However, for smoke-testing MRs, on every architecture, we could perhaps > cut corners. (Leaving Marge to do full diligence.) For example, we could > declare that if we have the result of compiling library module X.hs with > the stage1 GHC in the last full commit in master, then we can re-use that > build product rather than compiling X.hs with the MR’s slightly modified > stage1 GHC. That **might** be wrong; but it’s usually right. > > Anyway, there are big wins to be had here. > > Simon > > > > > > > > *From:* ghc-devs *On Behalf Of *John > Ericson > *Sent:* 19 February 2021 03:19 > *To:* ghc-devs > *Subject:* Re: On CI > > > > I am also wary of us to deferring checking whole platforms and what not. I > think that's just kicking the can down the road, and will result in more > variance and uncertainty. It might be alright for those authoring PRs, but > it will make Ben's job keeping the system running even more grueling. > > Before getting into these complex trade-offs, I think we should focus on > the cornerstone issue that CI isn't incremental. > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > 2. We don't cache between jobs. Shake and Make do not enforce > dependency soundness, nor cache-correctness when the build plan itself > changes, and this had made this hard/impossible to do safely. Naively this > only helps with stage 1 and not stage 2, but if we have separate stage 1 > and --freeze1 stage 2 builds, both can be incremental. Yes, this is also > lossy, but I only see it leading to false failures not false acceptances > (if we can also test the stage 1 one), so I consider it safe. MRs that only > work with a slow full build because ABI can so indicate. > > The second, main part is quite hard to tackle, but I strongly believe > incrementality is what we need most, and what we should remain focused on. > > John > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klebinger.andreas at gmx.at Mon Feb 22 13:46:45 2021 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Mon, 22 Feb 2021 14:46:45 +0100 Subject: Changes to performance testing? In-Reply-To: <010f0177c64a7c40-97040c50-92b1-4ad7-841d-8055824a8de2-000000@us-east-2.amazonses.com> References: <87im6lxuch.fsf@smart-cactus.org> <010f0177c64a7c40-97040c50-92b1-4ad7-841d-8055824a8de2-000000@us-east-2.amazonses.com> Message-ID: This seems quite reasonable to me. Not sure about the cost of implementing it (and the feasability of it if/when merge-trains arrive). Andreas Am 21/02/2021 um 21:31 schrieb Richard Eisenberg: > > >> On Feb 21, 2021, at 11:24 AM, Ben Gamari > > wrote: >> >> To mitigate this I would suggest that we allow performance test failures >> in marge-bot pipelines. A slightly weaker variant of this idea would >> instead only allow performance *improvements*. I suspect the latter >> would get most of the benefit, while eliminating the possibility that a >> large regression goes unnoticed. > > The value in making performance improvements a test failure is so that > patch authors can be informed of what they have done, to make sure it > matches expectations. This need can reasonably be satisfied without > stopping merging. That is, if Marge can accept performance > improvements, while (say) posting to each MR involved that it may have > contributed to a performance improvement, then I think we've done our > job here. > > On the other hand, a performance degradation is a bug, just like, say, > an error message regression. Even if it's a combination of commits > that cause the problem (an actual possibility even for error message > regressions), it's still a bug that we need to either fix or accept > (balanced out by other improvements). The pain of debugging this > scenario might be mitigated if there were a collation of the > performance wibbles for each individual commit. This information is, > in general, available: each commit passed CI on its own, and so it > should be possible to create a little report with its rows being perf > tests and its columns being commits or MR #s; each cell in the table > would be a percentage regression. If we're lucky, the regression Marge > sees will be the sum(*) of the entries in one of the rows -- this > means that we have a simple agglomeration of performance degradation. > If we're less lucky, the whole will not equal the sum of the parts, > and some of the patches interfere. In either case, the table would > suggest a likely place to look next. > > (*) I suppose if we're recording percentages, it wouldn't necessarily > be the actual sum, because percentages are a bit funny. But you get my > meaning. > > Pulling this all together: > * I'm against the initial proposal of allowing all performance > failures by Marge. This will allow bugs to accumulate (in my opinion). > * I'm in favor of allowing performance improvements to be accepted by > Marge. > * To mitigate against the information loss of Marge accepting > performance improvements, it would be great if Marge could alert MR > authors that a cumulative performance improvement took place. > * To mitigate against the annoyance of finding a performance > regression in a merge commit that does not appear in any component > commit, it would be great if there were a tool to collect performance > numbers from a set of commits and present them in a table for further > analysis. > > These "mitigations" might take work. If labor is impossible to produce > to complete this work, I'm in favor of simply allowing the performance > improvements, maybe also filing a ticket about these potential > improvements to the process. > > Richard > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Mon Feb 22 17:41:44 2021 From: john.ericson at obsidian.systems (John Ericson) Date: Mon, 22 Feb 2021 12:41:44 -0500 Subject: On CI In-Reply-To: References: <87h7m9yphj.fsf@smart-cactus.org> <19ee5399-9363-2206-7005-ea902289b4ae@obsidian.systems> <010f0177bbd109e4-98654d12-c4d5-442b-a55a-d4228b00b0d3-000000@us-east-2.amazonses.com> <6c6a4089-3a50-4555-413d-4e68d366c63f@obsidian.systems> Message-ID: <5f413890-3a09-9899-b952-0fb3f1e3986b@obsidian.systems> I agree one should be able to get most of the testing value from stage1. And the tooling team at IOHK has done some work in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3652 to allow a stage 1 compiler to be tested. That's a very important first step! But TH and GHCi require either iserv (external interpreter) or an compiler whose own ABI and the outputted ABI match for the internal interpreter, and ideally we should test both. I think doing a --freeze1 stage2 build *in addition* to the stage1 build would work in the majority of cases, and that would allow us to incrementally build and test both. Remember that iserv uses the ghc library, and needs to be ABI comparable with the stage1 compiler that is using it, so it is less a panacea than it might seem like for ABI changes vs mere cross compilation. I opened https://github.com/ghc-proposals/ghc-proposals/issues/162 for an ABI-agnostic interpreter that would allow stage1 alone to do GHCi and TH a third away unconditionally. This would also allow TH to safely be used in GHC itself, but for the purposes of this discussion, it's nice to make testing more reliable without the --freeze1 stage 2 gamble. Bottom line is, yes, building stage 2 from a freshly-built stage 1 will invalidate any cache, and so we should avoid that. John On 2/22/21 8:42 AM, Spiwack, Arnaud wrote: > Let me know if I'm talking nonsense, but I believe that we are > building both stages for each architecture and flavour. Do we need to > build two stages everywhere? What stops us from building a single > stage? And if anything, what can we change to get into a situation > where we can? > > Quite better than reusing build incrementally, is not building at all. > > On Mon, Feb 22, 2021 at 10:09 AM Simon Peyton Jones via ghc-devs > > wrote: > > Incremental CI can cut multiple hours to < mere minutes, > especially with the test suite being embarrassingly parallel. > There simply no way optimizations to the compiler independent from > sharing a cache between CI runs can get anywhere close to that > return on investment. > > I rather agree with this.  I don’t think there is much low-hanging > fruit on compile times, aside from coercion-zapping which we are > working on anyway.  If we got a 10% reduction in compile time we’d > be over the moon, but our users would barely notice. > > To get truly substantial improvements (a factor of 2 or 10) I > think we need to do less compiling – hence incremental CI. > > > Simon > > *From:*ghc-devs > *On Behalf Of *John Ericson > *Sent:* 22 February 2021 05:53 > *To:* ghc-devs > > *Subject:* Re: On CI > > I'm not opposed to some effort going into this, but I would > strongly opposite putting all our effort there. Incremental CI can > cut multiple hours to < mere minutes, especially with the test > suite being embarrassingly parallel. There simply no way > optimizations to the compiler independent from sharing a cache > between CI runs can get anywhere close to that return on investment. > > (FWIW, I'm also skeptical that the people complaining about GHC > performance know what's hurting them most. For example, after > non-incrementality, the next slowest thing is linking, which > is...not done by GHC! But all that is a separate conversation.) > > John > > On 2/19/21 2:42 PM, Richard Eisenberg wrote: > > There are some good ideas here, but I want to throw out > another one: put all our effort into reducing compile times. > There is a loud plea to do this on Discourse > , > and it would both solve these CI problems and also help > everyone else. > > This isn't to say to stop exploring the ideas here. But since > time is mostly fixed, tackling compilation times in general > may be the best way out of this. Ben's survey of other > projects (thanks!) shows that we're way, way behind in how > long our CI takes to run. > > Richard > > > > On Feb 19, 2021, at 7:20 AM, Sebastian Graf > > wrote: > > Recompilation avoidance > > I think in order to cache more in CI, we first have to > invest some time in fixing recompilation avoidance in our > bootstrapped build system. > > I just tested on a hadrian perf ticky build: Adding one > line of *comment* in the compiler causes > > * a (pretty slow, yet negligible) rebuild of the stage1 > compiler > * 2 minutes of RTS rebuilding (Why do we have to rebuild > the RTS? It doesn't depend in any way on the change I > made) > * apparent full rebuild the libraries > * apparent full rebuild of the stage2 compiler > > That took 17 minutes, a full build takes ~45minutes. So > there definitely is some caching going on, but not nearly > as much as there could be. > > I know there have been great and boring efforts on > compiler determinism in the past, but either it's not good > enough or our build system needs fixing. > > I think a good first step to assert would be to make sure > that the hash of the stage1 compiler executable doesn't > change if I only change a comment. > > I'm aware there probably is stuff going on, like embedding > configure dates in interface files and executables, that > would need to go, but if possible this would be a huge > improvement. > > On the other hand, we can simply tack on a [skip ci] to > the commit message, as I did for > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975 > . > Variants like [skip tests] or [frontend] could help to > identify which tests to run by default. > > Lean > > I had a chat with a colleague about how they do CI for > Lean. Apparently, CI turnaround time including tests is > generally 25 minutes (~15 minutes for the build) for a > complete pipeline, testing 6 different OSes and > configurations in parallel: > https://github.com/leanprover/lean4/actions/workflows/ci.yml > > > They utilise ccache to cache the clang-based C++-backend, > so that they only have to re-run the front- and > middle-end. In effect, they take advantage of the fact > that the "function" clang, in contrast to the "function" > stage1 compiler, stays the same. > > It's hard to achieve that for GHC, where a complete > compiler pipeline comes as one big, fused "function": An > external tool can never be certain that a change to > Parser.y could not affect the CodeGen phase. > > Inspired by Lean, the following is a bit inconcrete and > imaginary, but maybe we could make it so that compiler > phases "sign" parts of the interface file with the binary > hash of the respective subcomponents of the phase? > > E.g., if all the object files that influence CodeGen (that > will later be linked into the stage1 compiler) result in a > hash of 0xdeadbeef before and after the change to > Parser.y, we know we can stop recompiling Data.List with > the stage1 compiler when we see that the IR passed to > CodeGen didn't change, because the last compile did > CodeGen with a stage1 compiler with the same hash > 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the > function CodeGen stayed the same", so we can reuse its > cached outputs. > > Of course, that is utopic without a tool that does the > "taint analysis" of which modules in GHC influence > CodeGen. Probably just including all the transitive > dependencies of GHC.CmmToAsm suffices, but probably that's > too crude already. For another example, a change to > GHC.Utils.Unique would probably entail a full rebuild of > the compiler because it basically affects all compiler phases. > > There are probably parallels with recompilation avoidance > in a language with staged meta-programming. > > Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef > Svenningsson via ghc-devs >: > > Doing "optimistic caching" like you suggest sounds > very promising. A way to regain more robustness would > be as follows. > > If the build fails while building the libraries or the > stage2 compiler, this might be a false negative due to > the optimistic caching. Therefore, evict the > "optimistic caches" and restart building the > libraries. That way we can validate that the build > failure was a true build failure and not just due to > the aggressive caching scheme. > > Just my 2p > > Josef > > ------------------------------------------------------------------------ > > *From:* ghc-devs > on behalf of > Simon Peyton Jones via ghc-devs > > *Sent:* Friday, February 19, 2021 8:57 AM > *To:* John Ericson >; ghc-devs > > > *Subject:* RE: On CI > > 1. Building and testing happen together. When tests > failure spuriously, we also have to rebuild GHC in > addition to re-running the tests. That's pure > waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > > I don’t get this.  We have to build GHC before we can > test it, don’t we? > > 2 .  We don't cache between jobs. > > This is, I think, the big one.   We endlessly build > the exact same binaries. > > There is a problem, though.  If we make **any** change > in GHC, even a trivial refactoring, its binary will > change slightly.  So now any caching build system will > assume that anything built by that GHC must be rebuilt > – we can’t use the cached version.  That includes all > the libraries and the stage2 compiler.  So caching can > save all the preliminaries (building the initial > Cabal, and large chunk of stage1, since they are built > with the same bootstrap compiler) but after that we > are dead. > > I don’t know any robust way out of this. That small > change in the source code of GHC might be trivial > refactoring, or it might introduce a critical > mis-compilation which we really want to see in its > build products. > > However, for smoke-testing MRs, on every architecture, > we could perhaps cut corners.  (Leaving Marge to do > full diligence.)  For example, we could declare that > if we have the result of compiling library module X.hs > with the stage1 GHC in the last full commit in master, > then we can re-use that build product rather than > compiling X.hs with the MR’s slightly modified stage1 > GHC.  That **might** be wrong; but it’s usually right. > > Anyway, there are big wins to be had here. > > Simon > > *From:*ghc-devs > *On Behalf Of > *John Ericson > *Sent:* 19 February 2021 03:19 > *To:* ghc-devs > > *Subject:* Re: On CI > > I am also wary of us to deferring checking whole > platforms and what not. I think that's just kicking > the can down the road, and will result in more > variance and uncertainty. It might be alright for > those authoring PRs, but it will make Ben's job > keeping the system running even more grueling. > > Before getting into these complex trade-offs, I think > we should focus on the cornerstone issue that CI isn't > incremental. > > 1. Building and testing happen together. When tests > failure spuriously, we also have to rebuild GHC in > addition to re-running the tests. That's pure > waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > tracks this more or less. > 2. We don't cache between jobs. Shake and Make do not > enforce dependency soundness, nor > cache-correctness when the build plan itself > changes, and this had made this hard/impossible to > do safely. Naively this only helps with stage 1 > and not stage 2, but if we have separate stage 1 > and --freeze1 stage 2 builds, both can be > incremental. Yes, this is also lossy, but I only > see it leading to false failures not false > acceptances (if we can also test the stage 1 one), > so I consider it safe. MRs that only work with a > slow full build because ABI can so indicate. > > The second, main part is quite hard to tackle, but I > strongly believe incrementality is what we need most, > and what we should remain focused on. > > John > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Feb 23 16:14:55 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 23 Feb 2021 16:14:55 +0000 Subject: MR template text Message-ID: I often see MRs in my inbox that say Thank you for your contribution to GHC! Please take a few moments to verify that your commits fulfill the following: [ ] are either individually buildable or squashed This is because the author hasn't changed the Description of the MR, but rather has left the template text unchanged. As a way to "nudge" authors to give reviewers more information, I suggest replacing the template text with the draft below. Does anyone have any views, for or against? Simon Proposed new template PLEASE REPLACE ALL OF THIS TEXT with a description of your merge request, including * A description of what the Merge Request does. For a single-commit MR, a copy of the commit message is often perfect. * A reference (e.g. #19415) to the ticket that led to this MR, and that describes the problem that this MR solves. Almost all MRs need a ticket, except the tiniest changes (e.g. code formatting) - A ticket describes a *problem* - A merge request describes a *solution* to that problem. While you are encouraged to write a good MR Description, it's not compulsory. You could just be putting up the MR to share with a colleague, for example. But if you want (a) to get code reviews from others, or (b) to land the patch in GHC, please do follow these guidelines. For general style guidance see https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style For completeness, the current template is Thank you for your contribution to GHC! Please take a few moments to verify that your commits fulfill the following: * [ ] are either individually buildable or squashed * [ ] have commit messages which describe *what they do* (referring to [Notes][notes] and tickets using `#NNNN` syntax when appropriate) * [ ] have added source comments describing your change. For larger changes you likely should add a [Note][notes] and cross-reference it from the relevant places. * [ ] add a [testcase to the testsuite](https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/adding). * [ ] replace this message with a description motivating your change If you have any questions don't hesitate to open your merge request and inquire in a comment. If your patch isn't quite done yet please do add prefix your MR title with `WIP:`. [notes]: https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style#comments-in-the-source-code -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Tue Feb 23 19:21:28 2021 From: john.ericson at obsidian.systems (John Ericson) Date: Tue, 23 Feb 2021 14:21:28 -0500 Subject: Changes to performance testing? In-Reply-To: References: <87im6lxuch.fsf@smart-cactus.org> <010f0177c64a7c40-97040c50-92b1-4ad7-841d-8055824a8de2-000000@us-east-2.amazonses.com> Message-ID: <261c30d7-c26a-35e6-8a25-a43a0fc16dec@obsidian.systems> If we make the changes proposed in the "On CI" thread, I think we solve most of this for free. If a perf test fails, resubmitting should not need to rebuild GHC because it is cached. I'd say at that point, it's not even worth making Marge bot auto-accept extra improvements because restarting with new perf windows should be so easy. John On 2/22/21 8:46 AM, Andreas Klebinger wrote: > > This seems quite reasonable to me. > Not sure about the cost of implementing it (and the feasability of it > if/when merge-trains arrive). > > Andreas > > Am 21/02/2021 um 21:31 schrieb Richard Eisenberg: >> >> >>> On Feb 21, 2021, at 11:24 AM, Ben Gamari >> > wrote: >>> >>> To mitigate this I would suggest that we allow performance test failures >>> in marge-bot pipelines. A slightly weaker variant of this idea would >>> instead only allow performance *improvements*. I suspect the latter >>> would get most of the benefit, while eliminating the possibility that a >>> large regression goes unnoticed. >> >> The value in making performance improvements a test failure is so >> that patch authors can be informed of what they have done, to make >> sure it matches expectations. This need can reasonably be satisfied >> without stopping merging. That is, if Marge can accept performance >> improvements, while (say) posting to each MR involved that it may >> have contributed to a performance improvement, then I think we've >> done our job here. >> >> On the other hand, a performance degradation is a bug, just like, >> say, an error message regression. Even if it's a combination of >> commits that cause the problem (an actual possibility even for error >> message regressions), it's still a bug that we need to either fix or >> accept (balanced out by other improvements). The pain of debugging >> this scenario might be mitigated if there were a collation of the >> performance wibbles for each individual commit. This information is, >> in general, available: each commit passed CI on its own, and so it >> should be possible to create a little report with its rows being perf >> tests and its columns being commits or MR #s; each cell in the table >> would be a percentage regression. If we're lucky, the regression >> Marge sees will be the sum(*) of the entries in one of the rows -- >> this means that we have a simple agglomeration of performance >> degradation. If we're less lucky, the whole will not equal the sum of >> the parts, and some of the patches interfere. In either case, the >> table would suggest a likely place to look next. >> >> (*) I suppose if we're recording percentages, it wouldn't necessarily >> be the actual sum, because percentages are a bit funny. But you get >> my meaning. >> >> Pulling this all together: >> * I'm against the initial proposal of allowing all performance >> failures by Marge. This will allow bugs to accumulate (in my opinion). >> * I'm in favor of allowing performance improvements to be accepted by >> Marge. >> * To mitigate against the information loss of Marge accepting >> performance improvements, it would be great if Marge could alert MR >> authors that a cumulative performance improvement took place. >> * To mitigate against the annoyance of finding a performance >> regression in a merge commit that does not appear in any component >> commit, it would be great if there were a tool to collect performance >> numbers from a set of commits and present them in a table for further >> analysis. >> >> These "mitigations" might take work. If labor is impossible to >> produce to complete this work, I'm in favor of simply allowing the >> performance improvements, maybe also filing a ticket about these >> potential improvements to the process. >> >> Richard >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Tue Feb 23 20:29:51 2021 From: ben at well-typed.com (Ben Gamari) Date: Tue, 23 Feb 2021 15:29:51 -0500 Subject: Changes to performance testing? In-Reply-To: <87im6lxuch.fsf@smart-cactus.org> References: <87im6lxuch.fsf@smart-cactus.org> Message-ID: <875z2iy1c4.fsf@smart-cactus.org> Ben Gamari writes: > Hi all, > > Recently our performance tests have been causing quite some pain. One > reason for this is due to our new Darwin runners (see #19025), which > (surprisingly) differ significantly in their performance characteristics > (perhaps due to running Big Sur or using native tools provided by nix?). > > However, this is further exacerbated by the fact that there are quite a > few people working on compiler performance currently (horray!). This > leads to the following failure mode during Marge jobs: > > 1. Merge request A improves test T1234 by 0.5%, which is within the > test's acceptance window and therefore CI passes > > 2. Merge request B *also* improves test T1234 by another 0.5%, which > similarly passes CI > > 3. Marge tries to merge MRs A and B in a batch but finds that the > combined 1% improvement in T1234 is *outside* the acceptance window. > Consequently, the batch fails. > > This is quite painful, especially given that it creates work for those > trying to improve GHC (as the saying goes: no good deed goes > unpunished). > > To mitigate this I would suggest that we allow performance test failures > in marge-bot pipelines. A slightly weaker variant of this idea would > instead only allow performance *improvements*. I suspect the latter > would get most of the benefit, while eliminating the possibility that a > large regression goes unnoticed. > To get things un-stuck I have disabled the affected tests on Darwin for the time being. I hope we will be able to reenable these tests when we have migrated fully to the new runners although only time will tell. I will try to rebase the open MRs that are currently failing only due to spurious performance failures but please do feel free to hit rebase yourself if I miss any. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Tue Feb 23 21:19:46 2021 From: rae at richarde.dev (Richard Eisenberg) Date: Tue, 23 Feb 2021 21:19:46 +0000 Subject: MR template text In-Reply-To: References: Message-ID: <010f0177d0c32912-c5356ebe-b390-4992-993c-60d58074ba98-000000@us-east-2.amazonses.com> I'm happy with this direction, though I don't think we should expect that even capital letters will actually make people notice the text. One small change to suggest, below: > On Feb 23, 2021, at 11:14 AM, Simon Peyton Jones via ghc-devs wrote: > > Proposed new template > > PLEASE REPLACE ALL OF THIS TEXT with a description of your merge request, including > > * A description of what the Merge Request does. For a single-commit MR, a copy of the > commit message is often perfect. > > * A reference (e.g. #19415) to the ticket that led to this MR, and that describes the > problem that this MR solves. Almost all MRs need a ticket, except the tiniest > changes (e.g. code formatting) > - A ticket describes a *problem* > - A merge request describes a *solution* to that problem. > > While you are encouraged to write a good MR Description, it’s not compulsory. > You could just be putting up the MR to share with a colleague, for example. > > But if you want (a) to get code reviews from others, or > (b) to land the patch in GHC, > please do follow these guidelines. If you are not ready for wide review, please start the MR name with the prefix "WIP:", for "work in progress". > > For general style guidance see > https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Tue Feb 23 21:45:25 2021 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Tue, 23 Feb 2021 21:45:25 +0000 Subject: MR template text In-Reply-To: <010f0177d0c32912-c5356ebe-b390-4992-993c-60d58074ba98-000000@us-east-2.amazonses.com> References: <010f0177d0c32912-c5356ebe-b390-4992-993c-60d58074ba98-000000@us-east-2.amazonses.com> Message-ID: I'm not sure if it would work here, but I have seen some issue templates, e.g. for haskell-language-server on github[1], use comment markup in the template. e.g. This is not rendered on the web or email views, but is seen when editing. Alan [1] https://github.com/haskell/haskell-language-server/issues/new On Tue, 23 Feb 2021 at 21:21, Richard Eisenberg wrote: > I'm happy with this direction, though I don't think we should expect that > even capital letters will actually make people notice the text. One small > change to suggest, below: > > On Feb 23, 2021, at 11:14 AM, Simon Peyton Jones via ghc-devs < > ghc-devs at haskell.org> wrote: > > *Proposed new template* > PLEASE REPLACE ALL OF THIS TEXT with a description of your merge request, > including > > * A description of what the Merge Request does. For a single-commit MR, a > copy of the > commit message is often perfect. > > * A reference (e.g. #19415) to the ticket that led to this MR, and that > describes the > problem that this MR solves. Almost all MRs need a ticket, except the > tiniest > changes (e.g. code formatting) > - A ticket describes a *problem* > - A merge request describes a *solution* to that problem. > > While you are encouraged to write a good MR Description, it’s not > compulsory. > You could just be putting up the MR to share with a colleague, for example. > > But if you want (a) to get code reviews from others, or > (b) to land the patch in GHC, > please do follow these guidelines. > > > If you are not ready for wide review, please start the MR name with the > prefix "WIP:", for "work in progress". > > > For general style guidance see > https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style > > > > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julian at leviston.net Tue Feb 23 21:59:27 2021 From: julian at leviston.net (Julian Leviston) Date: Wed, 24 Feb 2021 08:59:27 +1100 Subject: MR template text In-Reply-To: References: Message-ID: <954CAB94-A582-453D-8B02-BDE779293CC9@leviston.net> Hi Simon, list et al, I’ve only contributed a couple of times, but I personally found the checklist invaluable to guide me (and remind me of) what needed to be done in total. In addition, giving folks a checklist that they can actually check off gives us a common set of agreed upon things that’s needed in an MR right in the MR, which is nice to folks. I wonder if we could reword it to say it’s still a work in progress or words to that effect at the top, and make the system not allow MRs to be built and/or merged unless they edit that text away, as well as have a bot inform them of why this is? :) I like the idea of the system guiding us through the process. Regards, Julian Would it be possible to get our tooling (a bot?) to nudge us if we haven’t changed it? > On 24 Feb 2021, at 3:14 am, Simon Peyton Jones via ghc-devs wrote: > > I often see MRs in my inbox that say > > Thank you for your contribution to GHC! > Please take a few moments to verify that your commits fulfill the following: > [ ] are either individually buildable or squashed > > > This is because the author hasn’t changed the Description of the MR, but rather has left the template text unchanged. > > As a way to “nudge” authors to give reviewers more information, I suggest replacing the template text with the draft below. Does anyone have any views, for or against? > > Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Feb 23 22:16:29 2021 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 23 Feb 2021 22:16:29 +0000 Subject: MR template text In-Reply-To: <954CAB94-A582-453D-8B02-BDE779293CC9@leviston.net> References: <954CAB94-A582-453D-8B02-BDE779293CC9@leviston.net> Message-ID: Thanks Julian I am by definition the wrong person to judge (or even write) text like this. Could you possibly have a go at editing the draft I sent so that you think it has the right tone and content? The current one is not working well. Your draft will almost certainly be better than mine. Simon From: Julian Leviston Sent: 23 February 2021 21:59 To: Simon Peyton Jones Cc: ghc-devs Subject: Re: MR template text Hi Simon, list et al, I’ve only contributed a couple of times, but I personally found the checklist invaluable to guide me (and remind me of) what needed to be done in total. In addition, giving folks a checklist that they can actually check off gives us a common set of agreed upon things that’s needed in an MR right in the MR, which is nice to folks. I wonder if we could reword it to say it’s still a work in progress or words to that effect at the top, and make the system not allow MRs to be built and/or merged unless they edit that text away, as well as have a bot inform them of why this is? :) I like the idea of the system guiding us through the process. Regards, Julian Would it be possible to get our tooling (a bot?) to nudge us if we haven’t changed it? On 24 Feb 2021, at 3:14 am, Simon Peyton Jones via ghc-devs > wrote: I often see MRs in my inbox that say Thank you for your contribution to GHC! Please take a few moments to verify that your commits fulfill the following: [ ] are either individually buildable or squashed This is because the author hasn’t changed the Description of the MR, but rather has left the template text unchanged. As a way to “nudge” authors to give reviewers more information, I suggest replacing the template text with the draft below. Does anyone have any views, for or against? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From julian at leviston.net Wed Feb 24 01:38:23 2021 From: julian at leviston.net (Julian Leviston) Date: Wed, 24 Feb 2021 12:38:23 +1100 Subject: MR template text In-Reply-To: References: <954CAB94-A582-453D-8B02-BDE779293CC9@leviston.net> Message-ID: <28ACD9B6-2ABF-4447-840C-4DFDB060F6A2@leviston.net> I gave it a shot at your behest, but I find your writing far more eloquent than mine, so it could no doubt deal with some editing. This is a work in progress Merge Request! If you want (a) to get code reviews from others, or (b) to land the patch in GHC, please do follow these guidelines. * [ ] (you can do this last) please replace this entire notice and checklist template with the following: - a description of what the Merge Request does. For single-commit MRs, the commit message is often perfect - A reference (e.g. #19415) to the ticket that led to this MR, and that describes the problem that this MR solves. Almost all MRs need a ticket, except the tiniest changes (e.g. code formatting) - A ticket describes a *problem* - A merge request describes a *solution* to that problem. * [ ] commits need to be either individually buildable or squashed * [ ] commits need to have commit messages which describe *what they do* (referring to [Notes][notes] and tickets using `#NNNN` syntax when appropriate) * [ ] add source comments describing your change. For larger changes you likely should add a [Note][notes] and cross-reference it from the relevant places. * [ ] add a [testcase to the testsuite](https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/adding). If you have any questions don't hesitate to open your merge request and inquire in a comment. If your patch isn't quite done yet please do add a `WIP:` prefix to your MR title. [notes]: For general style guidance and information on notes see https://gitlab.haskell.org/ghc/ghc/wikis/commentary/coding-style > On 24 Feb 2021, at 9:16 am, Simon Peyton Jones wrote: > > Thanks Julian > > I am by definition the wrong person to judge (or even write) text like this. > > Could you possibly have a go at editing the draft I sent so that you think it has the right tone and content? The current one is not working well. Your draft will almost certainly be better than mine. > > Simon > > From: Julian Leviston > Sent: 23 February 2021 21:59 > To: Simon Peyton Jones > Cc: ghc-devs > Subject: Re: MR template text > > > Hi Simon, list et al, > > I’ve only contributed a couple of times, but I personally found the checklist invaluable to guide me (and remind me of) what needed to be done in total. In addition, giving folks a checklist that they can actually check off gives us a common set of agreed upon things that’s needed in an MR right in the MR, which is nice to folks. > > I wonder if we could reword it to say it’s still a work in progress or words to that effect at the top, and make the system not allow MRs to be built and/or merged unless they edit that text away, as well as have a bot inform them of why this is? :) I like the idea of the system guiding us through the process. > > Regards, > Julian > > Would it be possible to get our tooling (a bot?) to nudge us if we haven’t changed it? > > On 24 Feb 2021, at 3:14 am, Simon Peyton Jones via ghc-devs > wrote: > > I often see MRs in my inbox that say > > Thank you for your contribution to GHC! > Please take a few moments to verify that your commits fulfill the following: > [ ] are either individually buildable or squashed > > > This is because the author hasn’t changed the Description of the MR, but rather has left the template text unchanged. > > As a way to “nudge” authors to give reviewers more information, I suggest replacing the template text with the draft below. Does anyone have any views, for or against? > > Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Feb 26 20:34:21 2021 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 26 Feb 2021 15:34:21 -0500 Subject: initIface* usage in plugins In-Reply-To: References: Message-ID: <87o8g6wotx.fsf@smart-cactus.org> Josh Meredith writes: > Hi, > I'm trying to use initIfaceLoad and initIfaceLcl to lookup core `Name`s > within a plugin. My understanding is that functions such as > `lookupIfaceTop` are used for this, but I'm running into an issue that I > suspect is caused by some `Name`s being inappropriate for this function, so > they fail with the error `Iface id out of scope: ...`. > > Is there a robust way to select which `Name` lookup function to use based > on a core binding/expression? > Josh, did you ever get an answer here? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From joshmeredith2008 at gmail.com Sat Feb 27 13:15:56 2021 From: joshmeredith2008 at gmail.com (Joshmeredith2008) Date: Sun, 28 Feb 2021 00:15:56 +1100 Subject: initIface* usage in plugins In-Reply-To: <87o8g6wotx.fsf@smart-cactus.org> Message-ID: Hi Ben, Yes, I have a solution. For reference, I was unaware that method instances are named with local variables in expression ASTs, while instances and other top-level bindings are - so the solution is to add those Ids to the environment before loading the expression, which isn’t necessary for other top-level bindings that are sourced from the CgGuts cg_binds. Cheers, Josh > > On Feb 27, 2021 at 7:34 am, wrote: > > > Josh Meredith writes: > Hi, > I'm trying to use initIfaceLoad and initIfaceLcl to lookup core `Name`s > within a plugin. My understanding is that functions such as > `lookupIfaceTop` are used for this, but I'm running into an issue that I > suspect is caused by some `Name`s being inappropriate for this function, so > they fail with the error `Iface id out of scope: ...`. > > Is there a robust way to select which `Name` lookup function to use based > on a core binding/expression? > Josh, did you ever get an answer here? Cheers, - Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: