From ben at smart-cactus.org Mon Oct 2 15:05:40 2023 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 02 Oct 2023 11:05:40 -0400 Subject: Migrating GHC's real-time communications to Matrix? (#24010) Message-ID: <87pm1xjc8w.fsf@smart-cactus.org> Hello all, GHC's user and contributor documentation currently directs users to #ghc on irc.libera.chat for discussion of GHC development. However, for a variety of reasons the center-of-mass of discussion has been gradually shifting towards Matrix (#GHC:matrix.org). Given that Matrix appears to be both more active and more accessible to newcomers, I suggest that we recognize this shift in our documentation. If you agree or disagree with this proposal please leave a :thumbsup: or :thumbsdown: on [#24010] and feel free to leave a comment explaining your feelings. Cheers, - Ben [#24010]: https://gitlab.haskell.org/ghc/ghc/-/issues/24010 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From bryan at haskell.foundation Thu Oct 5 11:29:53 2023 From: bryan at haskell.foundation (Bryan Richter) Date: Thu, 5 Oct 2023 14:29:53 +0300 Subject: Darwin runners fixed on master Message-ID: Hi all, A reinstall of Darwin runners exposed https://gitlab.haskell.org/ghc/ghc/-/issues/24055, which shows up as a failure to fetch perf notes. The fix has landed on master so you may want to rebase to pick it up. -Bryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Oct 9 22:18:24 2023 From: ben at well-typed.com (Ben Gamari) Date: Mon, 09 Oct 2023 18:18:24 -0400 Subject: [ANNOUNCE] GHC 9.8.1 is now available Message-ID: <87ttqzignm.fsf@smart-cactus.org> The GHC developers are very pleased to announce the availability of the final release of GHC 9.8.1. Binary distributions, source distributions, and documentation are available at downloads.haskell.org: https://downloads.haskell.org/ghc/9.8.1 GHC 9.8 brings a number of new features and improvements, including: * Preliminary support the `TypeAbstractions` language extension, allowing types to be bound in type declarations [TypeAbstractions]. * Support for the `ExtendedLiterals` extension, providing syntax for non-word-sized numeric literals in the surface language [extended-literals] * Improved rewrite rule matching behavior, allowing limited matching of higher-order patterns * Better support for user-defined warnings by way of the `WARNING` pragma [warnings] * The introduction of the new `GHC.TypeError.Unsatisfiable` constraint, allowing more predictable user-defined type errors [unsatisfiable] * Implementation of the export deprecation proposal, allowing module exports to be marked with `DEPRECATE` pragmas [deprecated-exports] * The addition of build semaphore support for parallel compilation; with coming support in `cabal-install` this will allow better use of parallelism in multi-package builds [jsem] * More efficient representation of info table provenance information, reducing binary sizes by over 50% in some cases when `-finfo-table-map` is in use A full accounting of changes can be found in the [release notes][]. Plans for GHC's future releases can be found on the [GHC Wiki][status]. We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release. Happy compiling, ~ Ben [TypeAbstractions]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0425-decl-invis-binders.rst [extended-literals]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0451-sized-literals.rst [unsatisfiable]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0433-unsatisfiable.rst [warnings]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0541-warning-pragmas-with-categories.rst [deprecated-exports]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0134-deprecating-exports-proposal.rst [jsem]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0540-jsem.rst [release notes]: https://downloads.haskell.org/ghc/9.8.1/docs/users_guide/9.8.1-notes.html [status]: https://gitlab.haskell.org/ghc/ghc/-/wikis/GHC-status -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From facundo.dominguez at tweag.io Fri Oct 13 20:19:57 2023 From: facundo.dominguez at tweag.io (=?UTF-8?Q?Facundo_Dom=C3=ADnguez?=) Date: Fri, 13 Oct 2023 17:19:57 -0300 Subject: Proposal for communication with GHC API users Message-ID: Dear GHC devs, Your feedback is appreciated in a new proposal to streamline communication with the authors of the GHC API as GHC evolves [1]. The proposal aims to set a process to collect detailed information on how users depend on the GHC API, and how GHC developers think these dependencies should be adjusted when there are breaking changes. This proposal is related to the endeavor to stabilize the GHC API [2]. Although, rather than trying to define a stable API, it sets a process to deal with the current state of affairs where users depend on many unstable parts. Looking forward to your thoughts! Facundo [1] https://gitlab.haskell.org/ghc/ghc/-/issues/24096 [2] https://discourse.haskell.org/t/charting-a-course-toward-a-stable-api-for-ghc/7646 -- All views and opinions expressed in this email message are the personal opinions of the author and do not represent those of the organization or its customers. From facundo.dominguez at tweag.io Fri Oct 13 20:23:40 2023 From: facundo.dominguez at tweag.io (=?UTF-8?Q?Facundo_Dom=C3=ADnguez?=) Date: Fri, 13 Oct 2023 17:23:40 -0300 Subject: Proposal for communication with GHC API users In-Reply-To: References: Message-ID: > Your feedback is appreciated in a new proposal to streamline > communication with the authors of the GHC API as GHC evolves [1]. Errata: - communication with the authors of the GHC API + communication with the *users* of the GHC API Cheers! On Fri, Oct 13, 2023 at 5:19 PM Facundo Domínguez wrote: > > Dear GHC devs, > > Your feedback is appreciated in a new proposal to streamline > communication with the authors of the GHC API as GHC evolves [1]. > > The proposal aims to set a process to collect detailed information on > how users depend on the GHC API, and how GHC developers think these > dependencies should be adjusted when there are breaking changes. > > This proposal is related to the endeavor to stabilize the GHC API [2]. > Although, rather than trying to define a stable API, it sets a process > to deal with the current state of affairs where users depend on many > unstable parts. > > Looking forward to your thoughts! > Facundo > > [1] https://gitlab.haskell.org/ghc/ghc/-/issues/24096 > [2] https://discourse.haskell.org/t/charting-a-course-toward-a-stable-api-for-ghc/7646 > > -- > All views and opinions expressed in this email message are the > personal opinions of the author and do not represent those of the > organization or its customers. -- All views and opinions expressed in this email message are the personal opinions of the author and do not represent those of the organization or its customers. From bryan at haskell.foundation Mon Oct 16 12:56:31 2023 From: bryan at haskell.foundation (Bryan Richter) Date: Mon, 16 Oct 2023 15:56:31 +0300 Subject: How do you keep tabs on commits that fix issues? In-Reply-To: <871qehknq7.fsf@smart-cactus.org> References: <871qehknq7.fsf@smart-cactus.org> Message-ID: Thanks Ben. I've copied that verbatim to https://gitlab.haskell.org/ghc/ghc/-/wikis/contributing#working-conventions :) On Fri, 29 Sept 2023 at 18:23, Ben Gamari wrote: > Bryan Richter via ghc-devs writes: > > > I am not sure of the best ways for checking if a certain issue has been > > fixed on a certain release. My past ways of using git run into certain > > problems: > > > > The commit (or commits!) that fix an issue get rewritten once by Marge as > > they are rebased onto master, and then potentially a second time as they > > are cherry-picked onto release branches. So just following the original > > commits doesn't work. > > > > If a commit mentions the issue it fixes, you might get some clues as to > > where it has ended up from GitLab. But those clues are often drowning in > > irrelevant mentions: each failed Marge batch, for instance, of which > there > > can be many. > > > > The only other thing I can think to do is look at the original merge > > request, pluck out the commit messages, and use git to search for commits > > by commit message and check each one for which branches contain it. But > > then I also need to know the context of the fix to know whether I should > > also be looking for other, logically related commits, and repeat the > dance. > > (Sometimes fixes are only partially applied to certain releases, > > exacerbating the need for knowing the context.) This seems like a > mechanism > > that can't rely on trusting the author of the original set of patches > > (which may be your past self) and instead requires a deep understanding > to > > be brought to bear every time you would want to double check the > situation. > > So it's not very scalable and I wouldn't expect many people to be able to > > do it. > > > > Are there better mechanisms already available? As I've said before, I am > > used to a different git workflow and I'm still learning how to use the > one > > used by GHC. I'd like to know how others handle it. > > > In general Marge leaves a helpful comment with the final SHA on each > pull request they merge to master. If I want to know whether an issue is > fixed in a stable branch, I will typically: > > 1. find the issue, which should have a reference to the merge requests > which fixed it > 2. from the merge request, find the final commit SHA from Marge's > comment > 3. look for references to that commit SHA in the appropriate branch. > > (3) works because in general we cherry-pick backports with the `-x` > flag, which causes a "(cherry-picked from ...)` comment to be appended > to the commit message. Moreover, reverts should also mention the commit > being reverted. > > Does this help? > > Cheers, > > - Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.peytonjones at gmail.com Tue Oct 17 10:08:02 2023 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Tue, 17 Oct 2023 11:08:02 +0100 Subject: Reinstallable - base Message-ID: Dear GHC devs Given the now-agreed split between ghc-internal and base , what stands in the way of a "reinstallable base"? Specifically, suppose that - GHC 9.8 comes out with base-4.9 - The CLC decides to make some change to `base`, so we get base-4.10 - Then GHC 9.10 comes out with base-4.10 I think we'd all like it if someone could use GHC 9.10 to compile a library L that depends on base-4.9 and either L doesn't work at all with base-4.10, or L's dependency bounds have not yet been adjusted to allow base-4.10. We'd like to have a version of `base`, say `base-4.9.1` that has the exact same API as `base-4.9` but works with GHC 9.10. Today, GHC 9.10 comes with a specific version of base, *and you can't change it*. The original reason for that was, I recall, that GHC knows the precise place where (say) the type Int is declared, and it'll get very confused if that data type definition moves around. But now we have `ghc-internal`, all these "things that GHC magically knows" are in `ghc-internal`, not `base`. *Hence my question: what (now) stops us making `base` behave like any other library*? That would be a big step forward, because it would mean that a newer GHC could compile old libraries against their old dependencies. (Some changes would still be difficult. If, for example, we removed Monad and replaced it with classes Mo1 and Mo2, it might be hard to simulate the old `base` with a shim. But getting 99% of the way there would still be fantastic.) Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From list at johnericson.me Tue Oct 17 14:45:34 2023 From: list at johnericson.me (John Ericson) Date: Tue, 17 Oct 2023 10:45:34 -0400 Subject: Reinstallable - base In-Reply-To: References: Message-ID: I think that's exactly right. Furthermore in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11400 the approach Ben has started going with is moving everything to ghc-internal to start, so this should be even easier --- surely we can reinstall an empty (except for reexports) library without the world falling down?! :) I would love us to commit, day 0 of the split, to the principle that that base is now and forever more reinstallable. Then, as we move things back to base (untangling base's and ghc-internals's implementations), we'll be committing to not regressing on this. For example, if we want to move something back that GHC does in fact refer to, then we have to create the facility to make this not interfere with reinstallablilty before we move it. (I would eventually like things like NonEmpty that GHC only barely knows about, and which are perfectly standard bits of Haskell, to live in base not ghc-internals. Agda's and Rust's putting annotations on things to let the compiler know it should use them, rather than baking in names/locations into the compiler, is very good prior art that this can work.) John On Tue, Oct 17, 2023, at 6:08 AM, Simon Peyton Jones wrote: > Dear GHC devs > > Given the now-agreed split between ghc-internal and base , what stands in the way of a "reinstallable base"? > > Specifically, suppose that > • GHC 9.8 comes out with base-4.9 > • The CLC decides to make some change to `base`, so we get base-4.10 > • Then GHC 9.10 comes out with base-4.10 > I think we'd all like it if someone could use GHC 9.10 to compile a library L that depends on base-4.9 and either L doesn't work at all with base-4.10, or L's dependency bounds have not yet been adjusted to allow base-4.10. > > We'd like to have a version of `base`, say `base-4.9.1` that has the exact same API as `base-4.9` but works with GHC 9.10. > > Today, GHC 9.10 comes with a specific version of base, *and you can't change it*. The original reason for that was, I recall, that GHC knows the precise place where (say) the type Int is declared, and it'll get very confused if that data type definition moves around. > > But now we have `ghc-internal`, all these "things that GHC magically knows" are in `ghc-internal`, not `base`. > > *Hence my question: what (now) stops us making `base` behave like any other library*? That would be a big step forward, because it would mean that a newer GHC could compile old libraries against their old dependencies. > > (Some changes would still be difficult. If, for example, we removed Monad and replaced it with classes Mo1 and Mo2, it might be hard to simulate the old `base` with a shim. But getting 99% of the way there would still be fantastic.) > > Simon > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at well-typed.com Tue Oct 17 15:54:41 2023 From: adam at well-typed.com (Adam Gundry) Date: Tue, 17 Oct 2023 16:54:41 +0100 Subject: Reinstallable - base In-Reply-To: References: Message-ID: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Hi Simon, Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal. The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers. Adam On 17/10/2023 11:08, Simon Peyton Jones wrote: > Dear GHC devs > > Given the now-agreed split between ghc-internal and base > , what > stands in the way of a "reinstallable base"? > > Specifically, suppose that > > * GHC 9.8 comes out with base-4.9 > * The CLC decides to make some change to `base`, so we get base-4.10 > * Then GHC 9.10 comes out with base-4.10 > > I think we'd all like it if someone could use GHC 9.10 to compile a > library L that depends on base-4.9 and either L doesn't work at all with > base-4.10, or L's dependency bounds have not yet been adjusted to allow > base-4.10. > > We'd like to have a version of `base`, say `base-4.9.1` that has the > exact same API as `base-4.9` but works with GHC 9.10. > > Today, GHC 9.10 comes with a specific version of base, /and you can't > change it/. The original reason for that was, I recall, that GHC knows > the precise place where (say) the type Int is declared, and it'll get > very confused if that data type definition moves around. > > But now we have `ghc-internal`, all these "things that GHC magically > knows" are in `ghc-internal`, not `base`. > > *Hence my question: what (now) stops us making `base` behave like any > other library*?  That would be a big step forward, because it would mean > that a newer GHC could compile old libraries against their old dependencies. > > (Some changes would still be difficult.  If, for example, we removed > Monad and replaced it with classes Mo1 and Mo2, it might be hard to > simulate the old `base` with a shim.  But getting 99% of the way there > would still be fantastic.) > > Simon -- Adam Gundry, Haskell Consultant Well-Typed LLP, https://www.well-typed.com/ Registered in England & Wales, OC335890 27 Old Gloucester Street, London WC1N 3AX, England From ietf-dane at dukhovni.org Tue Oct 17 16:54:17 2023 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 17 Oct 2023 12:54:17 -0400 Subject: Reinstallable - base In-Reply-To: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote: > Thanks for starting this discussion, it would be good to see progress in > this direction. As it happens I was discussing this question with Ben and > Matt over dinner last night, and unfortunately they explained to me that it > is more difficult than I naively hoped, even once wired-in and known-key > things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will be > compiled against a particular version of base. Then when Template Haskell is > used (with the internal interpreter), code will be dynamically loaded into a > process that already has symbols for ghc's version of base, which means it > is not safe for the code to depend on a different version of base. This is > rather like the situation with TH and cross-compilers. To avoid that problem, GHC's own dependency on "base" could be indirect via a shared object with versioned symbol names and a version-specific SONAME (possibly even a private to GHC SONAME and private symbol version names). Say "libbase.so.4.19.1". The dependency on "base" in the TemplatHaskell generated code would then also need to be dynamic, allowing the two versions of base to coexist without conflict, both in turn depdent on a common version of the GHC internal libraries. This would of course somewhat complicate binary distributions, but that should be manageable. Perhaps there are less invasive (more clever) solutions? -- Viktor From howard.b.golden at gmail.com Tue Oct 17 19:19:28 2023 From: howard.b.golden at gmail.com (Howard B. Golden) Date: Tue, 17 Oct 2023 12:19:28 -0700 Subject: Reinstallable - base In-Reply-To: References: Message-ID: My _very_ naive question is whether Template Haskell should have all the features of the latest language? Perhaps it could instead be limited to a simpler subset so that it could become a bolt-on to the rest of the compiler and it could be compiled and implemented independently using an older compiler and libraries. Howard > On Oct 17, 2023, at 9:54 AM, Viktor Dukhovni wrote: > > On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote: > >> Thanks for starting this discussion, it would be good to see progress in >> this direction. As it happens I was discussing this question with Ben and >> Matt over dinner last night, and unfortunately they explained to me that it >> is more difficult than I naively hoped, even once wired-in and known-key >> things are moved to ghc-internal. >> >> The difficulty is that, as a normal Haskell library, ghc itself will be >> compiled against a particular version of base. Then when Template Haskell is >> used (with the internal interpreter), code will be dynamically loaded into a >> process that already has symbols for ghc's version of base, which means it >> is not safe for the code to depend on a different version of base. This is >> rather like the situation with TH and cross-compilers. > > To avoid that problem, GHC's own dependency on "base" could be indirect > via a shared object with versioned symbol names and a version-specific > SONAME (possibly even a private to GHC SONAME and private symbol version > names). Say "libbase.so.4.19.1". > > The dependency on "base" in the TemplatHaskell generated code would then > also need to be dynamic, allowing the two versions of base to coexist > without conflict, both in turn depdent on a common version of the GHC > internal libraries. > > This would of course somewhat complicate binary distributions, but that > should be manageable. Perhaps there are less invasive (more clever) > solutions? > > -- > Viktor > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From simon.peytonjones at gmail.com Tue Oct 17 20:16:56 2023 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Tue, 17 Oct 2023 21:16:56 +0100 Subject: Reinstallable - base In-Reply-To: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: (Meta-question: on reflection, would this discussion perhaps be better on a ticket? But where? GHC's repo? Or HF's?) The difficulty is that, as a normal Haskell library, ghc itself will be > compiled against a particular verson of base. Then when Template Haskell is > used (with the internal interpreter), code will be dynamically loaded into > a process that already has symbols for ghc's version of base, which means > it is not safe for the code to depend on a different version of base. I'm not understanding the difficulty yet. Let's say that - An old library mylib (which uses TH) depends on base-4.7. - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. - At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API. At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note the the ghc-9.10 binary includes a compiled form of `base-4.9`. - That produces compiled object files, such as, mylib:M.o. - To run TH we need to link them with the running binary - So we need to link the compiled `base-4.7.1` as well. No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10 So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`. I think that's already supported: each has a distinct "installed package id". What am I missing? Simon On Tue, 17 Oct 2023 at 16:54, Adam Gundry wrote: > Hi Simon, > > Thanks for starting this discussion, it would be good to see progress in > this direction. As it happens I was discussing this question with Ben > and Matt over dinner last night, and unfortunately they explained to me > that it is more difficult than I naively hoped, even once wired-in and > known-key things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will be > compiled against a particular version of base. Then when Template > Haskell is used (with the internal interpreter), code will be > dynamically loaded into a process that already has symbols for ghc's > version of base, which means it is not safe for the code to depend on a > different version of base. This is rather like the situation with TH and > cross-compilers. > > Adam > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: > > Dear GHC devs > > > > Given the now-agreed split between ghc-internal and base > > , what > > stands in the way of a "reinstallable base"? > > > > Specifically, suppose that > > > > * GHC 9.8 comes out with base-4.9 > > * The CLC decides to make some change to `base`, so we get base-4.10 > > * Then GHC 9.10 comes out with base-4.10 > > > > I think we'd all like it if someone could use GHC 9.10 to compile a > > library L that depends on base-4.9 and either L doesn't work at all with > > base-4.10, or L's dependency bounds have not yet been adjusted to allow > > base-4.10. > > > > We'd like to have a version of `base`, say `base-4.9.1` that has the > > exact same API as `base-4.9` but works with GHC 9.10. > > > > Today, GHC 9.10 comes with a specific version of base, /and you can't > > change it/. The original reason for that was, I recall, that GHC knows > > the precise place where (say) the type Int is declared, and it'll get > > very confused if that data type definition moves around. > > > > But now we have `ghc-internal`, all these "things that GHC magically > > knows" are in `ghc-internal`, not `base`. > > > > *Hence my question: what (now) stops us making `base` behave like any > > other library*? That would be a big step forward, because it would mean > > that a newer GHC could compile old libraries against their old > dependencies. > > > > (Some changes would still be difficult. If, for example, we removed > > Monad and replaced it with classes Mo1 and Mo2, it might be hard to > > simulate the old `base` with a shim. But getting 99% of the way there > > would still be fantastic.) > > > > Simon > > -- > Adam Gundry, Haskell Consultant > Well-Typed LLP, https://www.well-typed.com/ > > Registered in England & Wales, OC335890 > 27 Old Gloucester Street, London WC1N 3AX, England > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Wed Oct 18 02:30:54 2023 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 18 Oct 2023 10:30:54 +0800 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: Something I haven’t gotten around to but only preliminary experiments with is dynamically built iserv binaries. Using -fexternal-interpreter can decouple the symbols the interpreter sees and those the compiler sees (They can even be of different architectures). iserv could be linked against the base the project wants to use, whereas GHC itself could use a different base. I’m not sure this covers everything, but it covers at least the case where we don’t need to load two different packages into the same process. Wrt to TH, I’m a bit behind on reading all the prior work to solve this, but conceptually I still believe template-haskell itself should not expose the internal ast, but only a combinator API to it. Regarding DSO’s: let’s please not make the existence of DSO a hard dependency. There are platforms for which we don’t have DSO capabilities, and where we are forced to use the in-memory loader and linker. On Wed, 18 Oct 2023 at 4:17 AM, Simon Peyton Jones < simon.peytonjones at gmail.com> wrote: > (Meta-question: on reflection, would this discussion perhaps be better on > a ticket? But where? GHC's repo? Or HF's?) > > The difficulty is that, as a normal Haskell library, ghc itself will be >> compiled against a particular verson of base. Then when Template Haskell is >> used (with the internal interpreter), code will be dynamically loaded into >> a process that already has symbols for ghc's version of base, which means >> it is not safe for the code to depend on a different version of base. > > > I'm not understanding the difficulty yet. > > Let's say that > > - An old library mylib (which uses TH) depends on base-4.7. > - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, > which in turn depends on ghc-internal-9.10. > - At the same time, though, we release base-4.7.1, which depends on > ghc-internal-9.10, and exposes the base-4.7 API. > > At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note > the the ghc-9.10 binary includes a compiled form of `base-4.9`. > > - That produces compiled object files, such as, mylib:M.o. > - To run TH we need to link them with the running binary > - So we need to link the compiled `base-4.7.1` as well. No problem: > it contains very little code; it is mostly a shim for ghc-internal-9.10 > > So the only thing we need is the ability to have a single linked binary > that includes (the compiled form for) two different versions/instantiations > of `base`. I think that's already supported: each has a distinct > "installed package id". > > What am I missing? > > Simon > > > > On Tue, 17 Oct 2023 at 16:54, Adam Gundry wrote: > >> Hi Simon, >> >> Thanks for starting this discussion, it would be good to see progress in >> this direction. As it happens I was discussing this question with Ben >> and Matt over dinner last night, and unfortunately they explained to me >> that it is more difficult than I naively hoped, even once wired-in and >> known-key things are moved to ghc-internal. >> >> The difficulty is that, as a normal Haskell library, ghc itself will be >> compiled against a particular version of base. Then when Template >> Haskell is used (with the internal interpreter), code will be >> dynamically loaded into a process that already has symbols for ghc's >> version of base, which means it is not safe for the code to depend on a >> different version of base. This is rather like the situation with TH and >> cross-compilers. >> >> Adam >> >> >> >> On 17/10/2023 11:08, Simon Peyton Jones wrote: >> > Dear GHC devs >> > >> > Given the now-agreed split between ghc-internal and base >> > , what >> > stands in the way of a "reinstallable base"? >> > >> > Specifically, suppose that >> > >> > * GHC 9.8 comes out with base-4.9 >> > * The CLC decides to make some change to `base`, so we get base-4.10 >> > * Then GHC 9.10 comes out with base-4.10 >> > >> > I think we'd all like it if someone could use GHC 9.10 to compile a >> > library L that depends on base-4.9 and either L doesn't work at all >> with >> > base-4.10, or L's dependency bounds have not yet been adjusted to allow >> > base-4.10. >> > >> > We'd like to have a version of `base`, say `base-4.9.1` that has the >> > exact same API as `base-4.9` but works with GHC 9.10. >> > >> > Today, GHC 9.10 comes with a specific version of base, /and you can't >> > change it/. The original reason for that was, I recall, that GHC knows >> > the precise place where (say) the type Int is declared, and it'll get >> > very confused if that data type definition moves around. >> > >> > But now we have `ghc-internal`, all these "things that GHC magically >> > knows" are in `ghc-internal`, not `base`. >> > >> > *Hence my question: what (now) stops us making `base` behave like any >> > other library*? That would be a big step forward, because it would >> mean >> > that a newer GHC could compile old libraries against their old >> dependencies. >> > >> > (Some changes would still be difficult. If, for example, we removed >> > Monad and replaced it with classes Mo1 and Mo2, it might be hard to >> > simulate the old `base` with a shim. But getting 99% of the way there >> > would still be fantastic.) >> > >> > Simon >> >> -- >> Adam Gundry, Haskell Consultant >> Well-Typed LLP, https://www.well-typed.com/ >> >> Registered in England & Wales, OC335890 >> 27 Old Gloucester Street, London WC1N 3AX, England >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg.grenrus at iki.fi Thu Oct 19 11:31:17 2023 From: oleg.grenrus at iki.fi (Oleg Grenrus) Date: Thu, 19 Oct 2023 14:31:17 +0300 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: iserv doesn't solve issues with GHC plugins (e.g. type-checker plugins)? As far as I understand, if there is a library with a type-checker plugin, then that library is forced to use whatever dependencies `ghc` library is using? I guess that is the fair restriction: such libraries wouldn't be able to use non-bundled `base` On 18.10.2023 5.30, Moritz Angermann wrote: > Something I haven’t gotten around to but only preliminary experiments > with is dynamically built iserv binaries. > > Using -fexternal-interpreter can decouple the symbols the interpreter > sees and those the compiler sees (They can even be of different > architectures). iserv could be linked against the base the project > wants to use, whereas GHC itself could use a different base. I’m not > sure this covers everything, but it covers at least the case where we > don’t need to load two different packages into the same process. > > Wrt to TH, I’m a bit behind on reading all the prior work to solve > this, but conceptually I still believe template-haskell itself should > not expose the internal ast, but only a combinator API to it. > > Regarding DSO’s: let’s please not make the existence of DSO a hard > dependency. There are platforms for which we don’t have DSO > capabilities, and where we are forced to use the in-memory loader and > linker. > > On Wed, 18 Oct 2023 at 4:17 AM, Simon Peyton Jones > wrote: > > (Meta-question: on reflection, would this discussion perhaps be > better on a ticket? But where?  GHC's repo?  Or HF's?) > > The difficulty is that, as a normal Haskell library, ghc > itself will be compiled against a particular verson of base. > Then when Template Haskell is used (with the internal > interpreter), code will be dynamically loaded into a process > that already has symbols for ghc's version of base, which > means it is not safe for the code to depend on a different > version of base. > > > I'm not understanding the difficulty yet. > > Let's say that > > * An old library mylib (which uses TH) depends on base-4.7. > * A new GHC, say GHC 9.10, depends on a newer version of > base-4.9, which in turn depends on ghc-internal-9.10. > * At the same time, though, we release base-4.7.1, which depends > on ghc-internal-9.10, and exposes the base-4.7 API. > > At this point we use ghc-9.10 to compile L, against base-4.7.1. > (Note the the ghc-9.10 binary includes a compiled form of `base-4.9`. > > * That produces compiled object files, such as, mylib:M.o. > * To run TH we need to link them with the running binary > * So we need to link the compiled `base-4.7.1` as well.  No > problem: it contains very little code; it is mostly a shim for > ghc-internal-9.10 > > So the only thing we need is the ability to have a single linked > binary that includes (the compiled form for) two different > versions/instantiations of `base`.   I think that's already > supported: each has a distinct "installed package id". > > What am I missing? > > Simon > > > > On Tue, 17 Oct 2023 at 16:54, Adam Gundry wrote: > > Hi Simon, > > Thanks for starting this discussion, it would be good to see > progress in > this direction. As it happens I was discussing this question > with Ben > and Matt over dinner last night, and unfortunately they > explained to me > that it is more difficult than I naively hoped, even once > wired-in and > known-key things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc > itself will be > compiled against a particular version of base. Then when Template > Haskell is used (with the internal interpreter), code will be > dynamically loaded into a process that already has symbols for > ghc's > version of base, which means it is not safe for the code to > depend on a > different version of base. This is rather like the situation > with TH and > cross-compilers. > > Adam > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: > > Dear GHC devs > > > > Given the now-agreed split between ghc-internal and base > > > , > what > > stands in the way of a "reinstallable base"? > > > > Specifically, suppose that > > > >   * GHC 9.8 comes out with base-4.9 > >   * The CLC decides to make some change to `base`, so we get > base-4.10 > >   * Then GHC 9.10 comes out with base-4.10 > > > > I think we'd all like it if someone could use GHC 9.10 to > compile a > > library L that depends on base-4.9 and either L doesn't work > at all with > > base-4.10, or L's dependency bounds have not yet been > adjusted to allow > > base-4.10. > > > > We'd like to have a version of `base`, say `base-4.9.1` that > has the > > exact same API as `base-4.9` but works with GHC 9.10. > > > > Today, GHC 9.10 comes with a specific version of base, /and > you can't > > change it/. The original reason for that was, I recall, that > GHC knows > > the precise place where (say) the type Int is declared, and > it'll get > > very confused if that data type definition moves around. > > > > But now we have `ghc-internal`, all these "things that GHC > magically > > knows" are in `ghc-internal`, not `base`. > > > > *Hence my question: what (now) stops us making `base` behave > like any > > other library*?  That would be a big step forward, because > it would mean > > that a newer GHC could compile old libraries against their > old dependencies. > > > > (Some changes would still be difficult.  If, for example, we > removed > > Monad and replaced it with classes Mo1 and Mo2, it might be > hard to > > simulate the old `base` with a shim.  But getting 99% of the > way there > > would still be fantastic.) > > > > Simon > > -- > Adam Gundry, Haskell Consultant > Well-Typed LLP, https://www.well-typed.com/ > > Registered in England & Wales, OC335890 > 27 Old Gloucester Street, London WC1N 3AX, England > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From oleg.grenrus at iki.fi Thu Oct 19 11:47:15 2023 From: oleg.grenrus at iki.fi (Oleg Grenrus) Date: Thu, 19 Oct 2023 14:47:15 +0300 Subject: Reinstallable - base In-Reply-To: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> For what it worth, `template-haskell` itself depends on a `base`. So if `base` if different base is used, different `template-haskell` is to be used. In my opinion is not *too unfair* to require that if you actually splice in (i.e. the code not only provides template-haskell combinators to create/modify splices) then you must have base and template-haskell versions aligned with host GHC used versions. The same restriction is GHC plugins, isn't it, except `template-haskell` is replaced with `ghc`? - Oleg On 17.10.2023 18.54, Adam Gundry wrote: > Hi Simon, > > Thanks for starting this discussion, it would be good to see progress > in this direction. As it happens I was discussing this question with > Ben and Matt over dinner last night, and unfortunately they explained > to me that it is more difficult than I naively hoped, even once > wired-in and known-key things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will > be compiled against a particular version of base. Then when Template > Haskell is used (with the internal interpreter), code will be > dynamically loaded into a process that already has symbols for ghc's > version of base, which means it is not safe for the code to depend on > a different version of base. This is rather like the situation with TH > and cross-compilers. > > Adam > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: >> Dear GHC devs >> >> Given the now-agreed split between ghc-internal and base >> , what >> stands in the way of a "reinstallable base"? >> >> Specifically, suppose that >> >>   * GHC 9.8 comes out with base-4.9 >>   * The CLC decides to make some change to `base`, so we get base-4.10 >>   * Then GHC 9.10 comes out with base-4.10 >> >> I think we'd all like it if someone could use GHC 9.10 to compile a >> library L that depends on base-4.9 and either L doesn't work at all >> with base-4.10, or L's dependency bounds have not yet been adjusted >> to allow base-4.10. >> >> We'd like to have a version of `base`, say `base-4.9.1` that has the >> exact same API as `base-4.9` but works with GHC 9.10. >> >> Today, GHC 9.10 comes with a specific version of base, /and you can't >> change it/. The original reason for that was, I recall, that GHC >> knows the precise place where (say) the type Int is declared, and >> it'll get very confused if that data type definition moves around. >> >> But now we have `ghc-internal`, all these "things that GHC magically >> knows" are in `ghc-internal`, not `base`. >> >> *Hence my question: what (now) stops us making `base` behave like any >> other library*?  That would be a big step forward, because it would >> mean that a newer GHC could compile old libraries against their old >> dependencies. >> >> (Some changes would still be difficult.  If, for example, we removed >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to >> simulate the old `base` with a shim.  But getting 99% of the way >> there would still be fantastic.) >> >> Simon > From arnaud.spiwack at tweag.io Fri Oct 20 07:56:54 2023 From: arnaud.spiwack at tweag.io (Arnaud Spiwack) Date: Fri, 20 Oct 2023 09:56:54 +0200 Subject: Reinstallable - base In-Reply-To: <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> Message-ID: A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything. On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote: > For what it worth, `template-haskell` itself depends on a `base`. So if > `base` if different base is used, different `template-haskell` is to be > used. > > In my opinion is not *too unfair* to require that if you actually splice > in (i.e. the code not only provides template-haskell combinators to > create/modify splices) then you must have base and template-haskell > versions aligned with host GHC used versions. > > The same restriction is GHC plugins, isn't it, except `template-haskell` > is replaced with `ghc`? > > - Oleg > > On 17.10.2023 18.54, Adam Gundry wrote: > > Hi Simon, > > > > Thanks for starting this discussion, it would be good to see progress > > in this direction. As it happens I was discussing this question with > > Ben and Matt over dinner last night, and unfortunately they explained > > to me that it is more difficult than I naively hoped, even once > > wired-in and known-key things are moved to ghc-internal. > > > > The difficulty is that, as a normal Haskell library, ghc itself will > > be compiled against a particular version of base. Then when Template > > Haskell is used (with the internal interpreter), code will be > > dynamically loaded into a process that already has symbols for ghc's > > version of base, which means it is not safe for the code to depend on > > a different version of base. This is rather like the situation with TH > > and cross-compilers. > > > > Adam > > > > > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: > >> Dear GHC devs > >> > >> Given the now-agreed split between ghc-internal and base > >> , what > >> stands in the way of a "reinstallable base"? > >> > >> Specifically, suppose that > >> > >> * GHC 9.8 comes out with base-4.9 > >> * The CLC decides to make some change to `base`, so we get base-4.10 > >> * Then GHC 9.10 comes out with base-4.10 > >> > >> I think we'd all like it if someone could use GHC 9.10 to compile a > >> library L that depends on base-4.9 and either L doesn't work at all > >> with base-4.10, or L's dependency bounds have not yet been adjusted > >> to allow base-4.10. > >> > >> We'd like to have a version of `base`, say `base-4.9.1` that has the > >> exact same API as `base-4.9` but works with GHC 9.10. > >> > >> Today, GHC 9.10 comes with a specific version of base, /and you can't > >> change it/. The original reason for that was, I recall, that GHC > >> knows the precise place where (say) the type Int is declared, and > >> it'll get very confused if that data type definition moves around. > >> > >> But now we have `ghc-internal`, all these "things that GHC magically > >> knows" are in `ghc-internal`, not `base`. > >> > >> *Hence my question: what (now) stops us making `base` behave like any > >> other library*? That would be a big step forward, because it would > >> mean that a newer GHC could compile old libraries against their old > >> dependencies. > >> > >> (Some changes would still be difficult. If, for example, we removed > >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to > >> simulate the old `base` with a shim. But getting 99% of the way > >> there would still be fantastic.) > >> > >> Simon > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -- Arnaud Spiwack Director, Research at https://moduscreate.com and https://tweag.io. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.peytonjones at gmail.com Fri Oct 20 08:00:22 2023 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Fri, 20 Oct 2023 09:00:22 +0100 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> Message-ID: > > A very large proportion of libraries, and virtually all end-user > applications, transitively depend on Template Haskell. Whether they use > Template Haskell directly or not. So if we're saying “base is > reinstallable, except when you have Template Haskell somewhere”, we're > effectively saying “base is not reinstallable”. Now, it could be a good > stepping-stone, from an engineering standpoint, but I don't think we could > deliver this and be satisfied that we've accomplished anything. > No one has yet answered my naive question (from 3 days ago) asking why Template Haskell stops base being reinstallable. I'll quote it here for completeness. Let's say that > - An old library mylib (which uses TH) depends on base-4.7. - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. - At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API. At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note that the ghc-9.10 binary includes a compiled form of `base-4.9`.) - That produces compiled object files, such as, mylib:M.o. - To run TH we need to link them with the running binary - So we need to link the compiled `base-4.7.1` as well. No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10 So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`. I think that's already supported: each has a distinct "installed package id". (End of quote) What am I missing? Simon On Fri, 20 Oct 2023 at 08:57, Arnaud Spiwack wrote: > A very large proportion of libraries, and virtually all end-user > applications, transitively depend on Template Haskell. Whether they use > Template Haskell directly or not. So if we're saying “base is > reinstallable, except when you have Template Haskell somewhere”, we're > effectively saying “base is not reinstallable”. Now, it could be a good > stepping-stone, from an engineering standpoint, but I don't think we could > deliver this and be satisfied that we've accomplished anything. > > On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote: > >> For what it worth, `template-haskell` itself depends on a `base`. So if >> `base` if different base is used, different `template-haskell` is to be >> used. >> >> In my opinion is not *too unfair* to require that if you actually splice >> in (i.e. the code not only provides template-haskell combinators to >> create/modify splices) then you must have base and template-haskell >> versions aligned with host GHC used versions. >> >> The same restriction is GHC plugins, isn't it, except `template-haskell` >> is replaced with `ghc`? >> >> - Oleg >> >> On 17.10.2023 18.54, Adam Gundry wrote: >> > Hi Simon, >> > >> > Thanks for starting this discussion, it would be good to see progress >> > in this direction. As it happens I was discussing this question with >> > Ben and Matt over dinner last night, and unfortunately they explained >> > to me that it is more difficult than I naively hoped, even once >> > wired-in and known-key things are moved to ghc-internal. >> > >> > The difficulty is that, as a normal Haskell library, ghc itself will >> > be compiled against a particular version of base. Then when Template >> > Haskell is used (with the internal interpreter), code will be >> > dynamically loaded into a process that already has symbols for ghc's >> > version of base, which means it is not safe for the code to depend on >> > a different version of base. This is rather like the situation with TH >> > and cross-compilers. >> > >> > Adam >> > >> > >> > >> > On 17/10/2023 11:08, Simon Peyton Jones wrote: >> >> Dear GHC devs >> >> >> >> Given the now-agreed split between ghc-internal and base >> >> , what >> >> stands in the way of a "reinstallable base"? >> >> >> >> Specifically, suppose that >> >> >> >> * GHC 9.8 comes out with base-4.9 >> >> * The CLC decides to make some change to `base`, so we get base-4.10 >> >> * Then GHC 9.10 comes out with base-4.10 >> >> >> >> I think we'd all like it if someone could use GHC 9.10 to compile a >> >> library L that depends on base-4.9 and either L doesn't work at all >> >> with base-4.10, or L's dependency bounds have not yet been adjusted >> >> to allow base-4.10. >> >> >> >> We'd like to have a version of `base`, say `base-4.9.1` that has the >> >> exact same API as `base-4.9` but works with GHC 9.10. >> >> >> >> Today, GHC 9.10 comes with a specific version of base, /and you can't >> >> change it/. The original reason for that was, I recall, that GHC >> >> knows the precise place where (say) the type Int is declared, and >> >> it'll get very confused if that data type definition moves around. >> >> >> >> But now we have `ghc-internal`, all these "things that GHC magically >> >> knows" are in `ghc-internal`, not `base`. >> >> >> >> *Hence my question: what (now) stops us making `base` behave like any >> >> other library*? That would be a big step forward, because it would >> >> mean that a newer GHC could compile old libraries against their old >> >> dependencies. >> >> >> >> (Some changes would still be difficult. If, for example, we removed >> >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to >> >> simulate the old `base` with a shim. But getting 99% of the way >> >> there would still be fantastic.) >> >> >> >> Simon >> > >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > > > -- > Arnaud Spiwack > Director, Research at https://moduscreate.com and https://tweag.io. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Fri Oct 20 10:17:19 2023 From: ben at well-typed.com (Ben Gamari) Date: Fri, 20 Oct 2023 06:17:19 -0400 Subject: Reinstallable - base In-Reply-To: References: Message-ID: <87pm191tuh.fsf@smart-cactus.org> Simon Peyton Jones writes: > Dear GHC devs > > Given the now-agreed split between ghc-internal and base > , what stands > in the way of a "reinstallable base"? > > Specifically, suppose that > > - GHC 9.8 comes out with base-4.9 > - The CLC decides to make some change to `base`, so we get base-4.10 > - Then GHC 9.10 comes out with base-4.10 > We thought about this quite a bit at the Well-Typed meeting this week. In short, I suspect that after the `ghc-internal` split is merged and some further work we will be able to use multiple `base` versions with a single compiler version. I imagine this would be done in roughly three phases. The first two phases can happen in the short term; the last is a bit further off. I'll describe these in turn below. # Phase 1: Reinstallable `base` Specifically, (in the absence of TH) we need to... * Give `base` a proper unit ID. This is necessary as GHC may allow multiple versions of a single package in a single build; under the status quo this would result in overlapping symbol names. * Ensure that all known-key and wired-in declarations are placed in `ghc-internal`, not `base`. This is necessary as wired-in and known-key declarations are detected by their `Name`, which we cannot know a priori if `base` has a proper unit ID. * Teach `Cabal` that `base` is a reinstallable package (`ghc-internal` will remain non-reinstallable) * With each GHC version, provide minor releases of `base` which can be built against the new GHC's `ghc-internal` With this done, we can offer (limited) compatibility across GHC/`base` version pairs. Naturally, changes in semantics of things defined in `ghc-internal` will make general compatibility difficult or impossible. However, Simon suggests, limited compatibility is much better than the status quo. # Phase 2: Template Haskell The above story is made considerably messier by `template-haskell`. Specifically, `template-haskell` depends upon `base`. Typically (i.e. when not using the external interpreter) values (e.g., splice results) are communicated between the splice and the compiler as heap objects; since these heap objects may be of types defined in `base` we must guarantee that `template-haskell` and `ghc` are built against the same `base` ABI. Since GHC doesn't have a notion of ABI stability, this would render reinstallable `base` unusable for a majority of programs. The first step to fixing this would be to eliminate the ABI dependency between `template-haskell` and `ghc`. We can do this by either: * using the external interpreter (`-fexternal-interpreter`), in which case splice results are communicated by serialisation * teaching the internal interpreter to communicate using the same serialisation mechanism used in the external interpreter However, `template-haskell` suffers from the same problem as `base` in that it contains definitions of known-key declarations. For this reason, we must apply the same approach that we used in `base` above, moving all known-key definitions into a new `template-haskell-internal` package against which `template-haskell` can link. At this point we can teach `Cabal` that `template-haskell` can be reinstalled and users will be able to build packages using `TemplateHaskell` while linking against arbitrary `base` versions. # Phase 3: Compatibility across `template-haskell` versions One of the major compatibility headaches of moving between GHC versions is accommodating changes in the `template-haskell` package. While there are various interface-level changes which we can use to alleviate some of this pressure (#20828, #24021), being able to provide compatibility with multiple `template-haskell` versions would be very convenient. Once we have split `template-haskell-internal` and `template-haskell`, we can potentially use a similar approach to that proposed above: support for multiple `template-haskell` major versions can be provided for each compiler. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Fri Oct 20 10:23:29 2023 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 20 Oct 2023 06:23:29 -0400 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: <87mswd1tjz.fsf@smart-cactus.org> Viktor Dukhovni writes: > On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote: > >> Thanks for starting this discussion, it would be good to see progress in >> this direction. As it happens I was discussing this question with Ben and >> Matt over dinner last night, and unfortunately they explained to me that it >> is more difficult than I naively hoped, even once wired-in and known-key >> things are moved to ghc-internal. >> >> The difficulty is that, as a normal Haskell library, ghc itself will be >> compiled against a particular version of base. Then when Template Haskell is >> used (with the internal interpreter), code will be dynamically loaded into a >> process that already has symbols for ghc's version of base, which means it >> is not safe for the code to depend on a different version of base. This is >> rather like the situation with TH and cross-compilers. > > To avoid that problem, GHC's own dependency on "base" could be indirect > via a shared object with versioned symbol names and a version-specific > SONAME (possibly even a private to GHC SONAME and private symbol version > names). Say "libbase.so.4.19.1". > The problem here is deeper than simply the symbol names. For instance, the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. Since lists are used in the `template-haskell` AST, we would be unable to share lists between `template-haskell` and `ghc`. As noted in my recent reply elsewhere in the thread, this can be avoided by communicating via serialisation instead of heap objects. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri Oct 20 10:25:12 2023 From: ben at well-typed.com (Ben Gamari) Date: Fri, 20 Oct 2023 06:25:12 -0400 Subject: Reinstallable - base In-Reply-To: References: Message-ID: <87jzrh1th3.fsf@smart-cactus.org> "Howard B. Golden" writes: > My _very_ naive question is whether Template Haskell should have all > the features of the latest language? Perhaps it could instead be > limited to a simpler subset so that it could become a bolt-on to the > rest of the compiler and it could be compiled and implemented > independently using an older compiler and libraries. > While I think users should be able to access all of the features of the language from TemplateHaskell, I agree that most users only need a small subset. We have explored some ideas exploiting this property in #24021. While it's not trivial per se, I think there is a real opportunity here. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri Oct 20 10:31:25 2023 From: ben at well-typed.com (Ben Gamari) Date: Fri, 20 Oct 2023 06:31:25 -0400 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> Message-ID: <87h6ml1t6q.fsf@smart-cactus.org> Simon Peyton Jones writes: > (Meta-question: on reflection, would this discussion perhaps be better on a > ticket? But where? GHC's repo? Or HF's?) > Given the current state of the discussion and the fact that the necessary steps are mostly internal to GHC, I think it is better to keep the discussion here or the issue tracker. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From andreyborzenkov2002 at gmail.com Fri Oct 20 10:35:16 2023 From: andreyborzenkov2002 at gmail.com (Andrei Borzenkov) Date: Fri, 20 Oct 2023 14:35:16 +0400 Subject: Reinstallable - base In-Reply-To: <87mswd1tjz.fsf@smart-cactus.org> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <87mswd1tjz.fsf@smart-cactus.org> Message-ID: <6e1994d4-e921-4a34-a425-8c56241b64bd@gmail.com> > the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. This is not actually true, `List` will obviously live in `ghc-internal` and `ghc-internal` would be the same for `base-4.9` and for `base-4.7.1`. This raises the question about do we really need to depend on `base` in `template-haskell`? Perhaps we may be satisfied by small set of things that will contain `ghc-internal`? 20.10.2023 14:23, Ben Gamari writes: > Viktor Dukhovni writes: > >> On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote: >> >>> Thanks for starting this discussion, it would be good to see progress in >>> this direction. As it happens I was discussing this question with Ben and >>> Matt over dinner last night, and unfortunately they explained to me that it >>> is more difficult than I naively hoped, even once wired-in and known-key >>> things are moved to ghc-internal. >>> >>> The difficulty is that, as a normal Haskell library, ghc itself will be >>> compiled against a particular version of base. Then when Template Haskell is >>> used (with the internal interpreter), code will be dynamically loaded into a >>> process that already has symbols for ghc's version of base, which means it >>> is not safe for the code to depend on a different version of base. This is >>> rather like the situation with TH and cross-compilers. >> To avoid that problem, GHC's own dependency on "base" could be indirect >> via a shared object with versioned symbol names and a version-specific >> SONAME (possibly even a private to GHC SONAME and private symbol version >> names). Say "libbase.so.4.19.1". >> > The problem here is deeper than simply the symbol names. For instance, > the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. > Since lists are used in the `template-haskell` AST, we would be unable > to share lists between `template-haskell` and `ghc`. > > As noted in my recent reply elsewhere in the thread, this can be avoided > by communicating via serialisation instead of heap objects. > > Cheers, > > - Ben > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Fri Oct 20 12:12:57 2023 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Fri, 20 Oct 2023 14:12:57 +0200 Subject: Reinstallable - base In-Reply-To: <6e1994d4-e921-4a34-a425-8c56241b64bd@gmail.com> References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <87mswd1tjz.fsf@smart-cactus.org> <6e1994d4-e921-4a34-a425-8c56241b64bd@gmail.com> Message-ID: Hi, Thanks, Ben, that sounds very interesting. Allow me to provide the following perspective. It seems that those `*-internal` packages take the role of a static library's symbol table in the absence of a fixed ABI: It allows clients (such as `base` and `template-haskell`) to link against fixed, internal implementations provided by GHC. "Reinstallable" then just means that a library can be linked against multiple different GHC versions providing the same set of internal APIs. But perhaps this analogy is not all to useful, given the clash with the traditional use of the term "linking" and "static library". Cheers, Sebastian Am Fr., 20. Okt. 2023 um 12:35 Uhr schrieb Andrei Borzenkov < andreyborzenkov2002 at gmail.com>: > > the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. > This is not actually true, `List` will obviously live in `ghc-internal` and > `ghc-internal` would be the same for `base-4.9` and for `base-4.7.1`. This > raises the question about do we really need to depend on `base` in > `template-haskell`? Perhaps we may be satisfied by small set of things that > will contain `ghc-internal`? > 20.10.2023 14:23, Ben Gamari writes: > > Viktor Dukhovni writes: > > > On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote: > > > Thanks for starting this discussion, it would be good to see progress in > this direction. As it happens I was discussing this question with Ben and > Matt over dinner last night, and unfortunately they explained to me that it > is more difficult than I naively hoped, even once wired-in and known-key > things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will be > compiled against a particular version of base. Then when Template Haskell is > used (with the internal interpreter), code will be dynamically loaded into a > process that already has symbols for ghc's version of base, which means it > is not safe for the code to depend on a different version of base. This is > rather like the situation with TH and cross-compilers. > > To avoid that problem, GHC's own dependency on "base" could be indirect > via a shared object with versioned symbol names and a version-specific > SONAME (possibly even a private to GHC SONAME and private symbol version > names). Say "libbase.so.4.19.1". > > > The problem here is deeper than simply the symbol names. For instance, > the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. > Since lists are used in the `template-haskell` AST, we would be unable > to share lists between `template-haskell` and `ghc`. > > As noted in my recent reply elsewhere in the thread, this can be avoided > by communicating via serialisation instead of heap objects. > > Cheers, > > - Ben > > > > _______________________________________________ > ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From list at JohnEricson.me Fri Oct 20 14:10:32 2023 From: list at JohnEricson.me (John Ericson) Date: Fri, 20 Oct 2023 10:10:32 -0400 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> Message-ID: <2fa05bfd-6dfd-42fb-8325-441ad39fe939@JohnEricson.me> I think it is making progress. It's not just an engineering milestone but also messaging. Right now a huge issue that every Haskeller faces is new GHCs having new major versions of base. If we make clear that "it's not base's fault, it's Template Haskell's fault", then we will have an easier time coordinating people and eventually fundraising to get issued fixed with Template Haskell. John On 10/20/23 03:56, Arnaud Spiwack wrote: > A very large proportion of libraries, and virtually all end-user > applications, transitively depend on Template Haskell. Whether they > use Template Haskell directly or not. So if we're saying “base is > reinstallable, except when you have Template Haskell somewhere”, we're > effectively saying “base is not reinstallable”. Now, it could be a > good stepping-stone, from an engineering standpoint, but I don't think > we could deliver this and be satisfied that we've accomplished anything. > > On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote: > > For what it worth, `template-haskell` itself depends on a `base`. > So if > `base` if different base is used, different `template-haskell` is > to be > used. > > In my opinion is not *too unfair* to require that if you actually > splice > in (i.e. the code not only provides template-haskell combinators to > create/modify splices) then you must have base and template-haskell > versions aligned with host GHC used versions. > > The same restriction is GHC plugins, isn't it, except > `template-haskell` > is replaced with `ghc`? > > - Oleg > > On 17.10.2023 18.54, Adam Gundry wrote: > > Hi Simon, > > > > Thanks for starting this discussion, it would be good to see > progress > > in this direction. As it happens I was discussing this question > with > > Ben and Matt over dinner last night, and unfortunately they > explained > > to me that it is more difficult than I naively hoped, even once > > wired-in and known-key things are moved to ghc-internal. > > > > The difficulty is that, as a normal Haskell library, ghc itself > will > > be compiled against a particular version of base. Then when > Template > > Haskell is used (with the internal interpreter), code will be > > dynamically loaded into a process that already has symbols for > ghc's > > version of base, which means it is not safe for the code to > depend on > > a different version of base. This is rather like the situation > with TH > > and cross-compilers. > > > > Adam > > > > > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: > >> Dear GHC devs > >> > >> Given the now-agreed split between ghc-internal and base > >> , > what > >> stands in the way of a "reinstallable base"? > >> > >> Specifically, suppose that > >> > >>   * GHC 9.8 comes out with base-4.9 > >>   * The CLC decides to make some change to `base`, so we get > base-4.10 > >>   * Then GHC 9.10 comes out with base-4.10 > >> > >> I think we'd all like it if someone could use GHC 9.10 to > compile a > >> library L that depends on base-4.9 and either L doesn't work at > all > >> with base-4.10, or L's dependency bounds have not yet been > adjusted > >> to allow base-4.10. > >> > >> We'd like to have a version of `base`, say `base-4.9.1` that > has the > >> exact same API as `base-4.9` but works with GHC 9.10. > >> > >> Today, GHC 9.10 comes with a specific version of base, /and you > can't > >> change it/. The original reason for that was, I recall, that GHC > >> knows the precise place where (say) the type Int is declared, and > >> it'll get very confused if that data type definition moves around. > >> > >> But now we have `ghc-internal`, all these "things that GHC > magically > >> knows" are in `ghc-internal`, not `base`. > >> > >> *Hence my question: what (now) stops us making `base` behave > like any > >> other library*?  That would be a big step forward, because it > would > >> mean that a newer GHC could compile old libraries against their > old > >> dependencies. > >> > >> (Some changes would still be difficult.  If, for example, we > removed > >> Monad and replaced it with classes Mo1 and Mo2, it might be > hard to > >> simulate the old `base` with a shim.  But getting 99% of the way > >> there would still be fantastic.) > >> > >> Simon > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > -- > Arnaud Spiwack > Director, Research at https://moduscreate.com and https://tweag.io. > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From list at JohnEricson.me Fri Oct 20 15:01:03 2023 From: list at JohnEricson.me (John Ericson) Date: Fri, 20 Oct 2023 11:01:03 -0400 Subject: Reinstallable - base In-Reply-To: References: <790db448-edc1-42c4-8e65-f736feed484c@well-typed.com> <33bcf746-dc06-8cdb-3b0e-fb84d9ab1718@iki.fi> Message-ID: On 10/20/23 04:00, Simon Peyton Jones wrote: > A very large proportion of libraries, and virtually all end-user > applications, transitively depend on Template Haskell. Whether > they use Template Haskell directly or not. So if we're saying > “base is reinstallable, except when you have Template Haskell > somewhere”, we're effectively saying “base is not reinstallable”. > Now, it could be a good stepping-stone, from an engineering > standpoint, but I don't think we could deliver this and be > satisfied that we've accomplished anything. > > > No one has yet answered my naive question (from 3 days ago) asking why > Template Haskell stops base being reinstallable. I'll quote it here > for completeness. > > Let's say that > > * An old library mylib (which uses TH) depends on base-4.7. > * A new GHC, say GHC 9.10, depends on a newer version of > base-4.9, which in turn depends on ghc-internal-9.10. > * At the same time, though, we release base-4.7.1, which depends > on ghc-internal-9.10, and exposes the base-4.7 API. > > At this point we use ghc-9.10 to compile L, against base-4.7.1.   > (Note that the ghc-9.10 binary includes a compiled form of > `base-4.9`.) > > * That produces compiled object files, such as, mylib:M.o. > * To run TH we need to link them with the running binary > * So we need to link the compiled `base-4.7.1` as well.  No > problem: it contains very little code; it is mostly a shim > for ghc-internal-9.10 > > So the only thing we need is the ability to have a single linked > binary that includes (the compiled form for) two different > versions/instantiations of `base`.   I think that's already > supported: each has a distinct "installed package id". > > (End of quote) > > What am I missing? > > Simon Simon I think you are right on the level of GHC itself: GHC can indeed cope with multiple versions of libraries just fine. However if we do this we run the risk of the user getting "base.x.y FooType is not the same as base.x.(y+1) FooType" errors, and I don't think that is good. cabal-install and stack current enforce build plans such that those sorts of errors are not possible, and I think that is a good thing we should not revisit at this time. I don't think this should stop anything about reinstallable base, however. We just need to make a nice error when someone tries to splice in code using the wrong version of template haskell (or otherwise interact with GHC) using ABI hashes. Then we have the following situation: 1. Users that use stack/cabal-install plans only get one version of base, and if they have template-haskell using the the "wrong" version of base (identified via API hashes) they will get a nice error. 2. Users that "go rogue" and manually set up their project to mix base versions and avoid other errors will have their TH work as you describe./ So the thing Simon talks about is not /encouraged/, but it is a "free feature" we shouldn't go out of our way to prevent either. John -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Fri Oct 20 17:26:19 2023 From: ben at well-typed.com (Ben Gamari) Date: Fri, 20 Oct 2023 13:26:19 -0400 Subject: Reinstallable - base In-Reply-To: <87pm191tuh.fsf@smart-cactus.org> References: <87pm191tuh.fsf@smart-cactus.org> Message-ID: <87a5sd19zb.fsf@smart-cactus.org> Ben Gamari writes: > Simon Peyton Jones writes: > >> Dear GHC devs >> >> Given the now-agreed split between ghc-internal and base >> , what stands >> in the way of a "reinstallable base"? >> >> Specifically, suppose that >> >> - GHC 9.8 comes out with base-4.9 >> - The CLC decides to make some change to `base`, so we get base-4.10 >> - Then GHC 9.10 comes out with base-4.10 >> > We thought about this quite a bit at the Well-Typed meeting this week. > In short, I suspect that after the `ghc-internal` split is merged and > some further work we will be able to use multiple `base` versions with > a single compiler version. > > I imagine this would be done in roughly three phases. The first two > phases can happen in the short term; the last is a bit further off. > I'll describe these in turn below. > One final point that I forgot to mention: None of this addresses the problem of compiler plugins. This is problematic as `Cabal`'s one-version-per-install-plan restriction means means that any package using a plugin will be forced to use the precise `base` that `ghc` itself is linked against. I can think of three ways to address this: * Teach `Cabal` to distinguish between plugin dependencies (which are only needed at build time and needn't be linked into final build results) and normal runtime dependencies. This is not a complete solution as many plugins have both compile-time and runtime components, but it would help in some cases. * Make `ghc` reinstallable. This allows `Cabal` to rebuild the compiler when we need to link against a different `base`. We have started this process but in general it is quite tricky to get right (see #20742) and will require cooperation from `Cabal` . * Serialise structures used by plugins, just as we do for the TH AST. Unfortunately, the surface area of the plugin interface is significantly larger than that of TH. None of these options are easy. For this reason, I think it would be wise to leave plugins as future work (if we decide to address it at all). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From howard.b.golden at gmail.com Fri Oct 20 19:06:43 2023 From: howard.b.golden at gmail.com (Howard B. Golden) Date: Fri, 20 Oct 2023 12:06:43 -0700 Subject: Reinstallable - base In-Reply-To: <87a5sd19zb.fsf@smart-cactus.org> References: <87a5sd19zb.fsf@smart-cactus.org> Message-ID: <9B7F1F77-B70F-4D85-B861-AFD8DEB3E76F@gmail.com> Cut the Gordian Knot by serializing all the interfaces. This may be a good first step. Howard > On Oct 20, 2023, at 10:26 AM, Ben Gamari wrote: > > Ben Gamari writes: > >> Simon Peyton Jones writes: >> >>> Dear GHC devs >>> >>> Given the now-agreed split between ghc-internal and base >>> , what stands >>> in the way of a "reinstallable base"? >>> >>> Specifically, suppose that >>> >>> - GHC 9.8 comes out with base-4.9 >>> - The CLC decides to make some change to `base`, so we get base-4.10 >>> - Then GHC 9.10 comes out with base-4.10 >>> >> We thought about this quite a bit at the Well-Typed meeting this week. >> In short, I suspect that after the `ghc-internal` split is merged and >> some further work we will be able to use multiple `base` versions with >> a single compiler version. >> >> I imagine this would be done in roughly three phases. The first two >> phases can happen in the short term; the last is a bit further off. >> I'll describe these in turn below. >> > One final point that I forgot to mention: None of this addresses the > problem of compiler plugins. This is problematic as `Cabal`'s > one-version-per-install-plan restriction means means that any package > using a plugin will be forced to use the precise `base` that `ghc` > itself is linked against. > > I can think of three ways to address this: > > * Teach `Cabal` to distinguish between plugin dependencies (which are > only needed at build time and needn't be linked into final build > results) and normal runtime dependencies. This is not a complete > solution as many plugins have both compile-time and runtime > components, but it would help in some cases. > > * Make `ghc` reinstallable. This allows `Cabal` to rebuild the compiler > when we need to link against a different `base`. We have started this > process but in general it is quite tricky to get right (see #20742) > and will require cooperation from `Cabal` . > > * Serialise structures used by plugins, just as we do for the TH AST. > Unfortunately, the surface area of the plugin interface is > significantly larger than that of TH. > > None of these options are easy. For this reason, I think it would be > wise to leave plugins as future work (if we decide to address it at > all). > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Fri Oct 20 20:23:36 2023 From: ben at well-typed.com (Ben Gamari) Date: Fri, 20 Oct 2023 16:23:36 -0400 Subject: Reinstallable - base In-Reply-To: <87pm191tuh.fsf@smart-cactus.org> References: <87pm191tuh.fsf@smart-cactus.org> Message-ID: <874jil11ru.fsf@smart-cactus.org> Ben Gamari writes: > ... > > # Phase 2: Template Haskell > > The above story is made considerably messier by `template-haskell`. > ... One further clarification as I fear this may be unclear: Phase 2 does *not* allow multiple versions of `template-haskell` to be used with a single GHC. That is, each GHC version will have precisely one corresponding `template-haskell` version; consequently, programs using TemplateHaskell's combinator's will still need to adapt to new `template-haskell` releases when moving to a new GHC. Phase 3 describes how we could eliminate this need by allowing multiple `template-haskell` versions to be used with a single GHC release. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From howard.b.golden at gmail.com Fri Oct 20 20:54:58 2023 From: howard.b.golden at gmail.com (Howard B. Golden) Date: Fri, 20 Oct 2023 13:54:58 -0700 Subject: Reinstallable - base In-Reply-To: <874jil11ru.fsf@smart-cactus.org> References: <874jil11ru.fsf@smart-cactus.org> Message-ID: <001E6BE8-468E-4DC6-99DD-661035E3CAFE@gmail.com> Perhaps there may be useful ideas in https://draugus.github.io/pdf/Software%20Design%20for%20Flexibility%20-%20Chris%20Hanson.pdf . Howard > On Oct 20, 2023, at 1:23 PM, Ben Gamari wrote: > > Ben Gamari writes: > >> ... >> >> # Phase 2: Template Haskell >> >> The above story is made considerably messier by `template-haskell`. >> ... > > One further clarification as I fear this may be unclear: > > Phase 2 does *not* allow multiple versions of `template-haskell` to > be used with a single GHC. That is, each GHC version will have precisely > one corresponding `template-haskell` version; consequently, programs > using TemplateHaskell's combinator's will still need to adapt to new > `template-haskell` releases when moving to a new GHC. > > Phase 3 describes how we could eliminate this need by allowing multiple > `template-haskell` versions to be used with a single GHC release. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From zubin at well-typed.com Tue Oct 24 12:40:39 2023 From: zubin at well-typed.com (Zubin Duggal) Date: Tue, 24 Oct 2023 18:10:39 +0530 Subject: Upcoming GHC 9.4.8 release Message-ID: Hi all, We hope to prepare the 9.4.8 release in the coming weeks. This will likely be the final release in the 9.4 series. The major patches currently slated for inclusion into the 9.4.8 are: 1. A process submodule bump to 1.6.17.0 avoiding segfaults on Darwin (https://github.com/haskell/process/issues/295) 2. A fix for a recompilation checking bug where GHC may miss changes in transitive dependencies when deciding to relink a program (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11178) 3. A fix for a packaging bug so that we can ensure all released binaries and bundled libraries are built with `-split-sections` enabled (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9492) If you would like any patches to be considered for inclusion in this release please ensure that the corresponding Merge Requests are marked with the ~"backport needed:9.4" by **the end of this week**. Please note that we will be quite conservative with the changes we accept for this branch at this point in its lifecycle. The current set of all MRs being considered for inclusion can be viewed at https://gitlab.haskell.org/ghc/ghc/-/merge_requests?scope=all&state=all&label_name[]=backport%20needed%3A9.4 Thanks, Zubin From kazu at iij.ad.jp Wed Oct 25 01:57:57 2023 From: kazu at iij.ad.jp (Kazu Yamamoto (=?iso-2022-jp?B?GyRCOzNLXE9CSScbKEI=?=)) Date: Wed, 25 Oct 2023 10:57:57 +0900 (JST) Subject: Upcoming GHC 9.4.8 release In-Reply-To: References: Message-ID: <20231025.105757.548341736261247555.kazu@iij.ad.jp> Hello, Thank you for your effort. How can I confirm that fix for #23746 is included in GHC 9.4.8? Without this, I cannot take profile on M1 mac. I'm now taking profile on a remote Linux machine. It's very inconvenient. --Kazu > Hi all, > > We hope to prepare the 9.4.8 release in the coming weeks. This will > likely be the final release in the 9.4 series. > > The major patches currently slated for inclusion into the 9.4.8 are: > > 1. A process submodule bump to 1.6.17.0 avoiding segfaults on Darwin > (https://github.com/haskell/process/issues/295) > 2. A fix for a recompilation checking bug where GHC may miss changes > in > transitive dependencies when deciding to relink a program > (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11178) 3. A > fix > for a packaging bug so that we can ensure all released binaries > and bundled libraries are built with `-split-sections` enabled > (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9492) > > If you would like any patches to be considered for inclusion in this > release please ensure that the corresponding Merge Requests are > marked > with the ~"backport needed:9.4" by **the end of this week**. > > Please note that we will be quite conservative with the changes we > accept for this branch at this point in its lifecycle. > > The current set of all MRs being considered for inclusion can be > viewed at > https://gitlab.haskell.org/ghc/ghc/-/merge_requests?scope=all&state=all&label_name[]=backport%20needed%3A9.4 > > Thanks, > Zubin > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From zubin at well-typed.com Wed Oct 25 08:22:52 2023 From: zubin at well-typed.com (Zubin Duggal) Date: Wed, 25 Oct 2023 13:52:52 +0530 Subject: Upcoming GHC 9.4.8 release In-Reply-To: <20231025.105757.548341736261247555.kazu@iij.ad.jp> References: <20231025.105757.548341736261247555.kazu@iij.ad.jp> Message-ID: The fix for the issue (!11254) has been marked for backport and will be included in the release. On 23/10/25 10:57, Kazu Yamamoto wrote: >Hello, > >Thank you for your effort. > >How can I confirm that fix for #23746 is included in GHC 9.4.8? >Without this, I cannot take profile on M1 mac. >I'm now taking profile on a remote Linux machine. >It's very inconvenient. > >--Kazu > >>Hi all, >> >>We hope to prepare the 9.4.8 release in the coming weeks. This will >>likely be the final release in the 9.4 series. >> >>The major patches currently slated for inclusion into the 9.4.8 are: >> >>1. A process submodule bump to 1.6.17.0 avoiding segfaults on Darwin >> (https://github.com/haskell/process/issues/295) >>2. A fix for a recompilation checking bug where GHC may miss changes >>in >> transitive dependencies when deciding to relink a program >> (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11178) 3. A >>fix >> for a packaging bug so that we can ensure all released binaries >> and bundled libraries are built with `-split-sections` enabled >> (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9492) >> >>If you would like any patches to be considered for inclusion in this >>release please ensure that the corresponding Merge Requests are >>marked >>with the ~"backport needed:9.4" by **the end of this week**. >> >>Please note that we will be quite conservative with the changes we >>accept for this branch at this point in its lifecycle. >> >>The current set of all MRs being considered for inclusion can be >>viewed at >>https://gitlab.haskell.org/ghc/ghc/-/merge_requests?scope=all&state=all&label_name[]=backport%20needed%3A9.4 >> >>Thanks, >>Zubin >>_______________________________________________ >>ghc-devs mailing list >>ghc-devs at haskell.org >>http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > From kazu at iij.ad.jp Thu Oct 26 01:37:12 2023 From: kazu at iij.ad.jp (Kazu Yamamoto (=?iso-2022-jp?B?GyRCOzNLXE9CSScbKEI=?=)) Date: Thu, 26 Oct 2023 10:37:12 +0900 (JST) Subject: Upcoming GHC 9.4.8 release In-Reply-To: References: <20231025.105757.548341736261247555.kazu@iij.ad.jp> Message-ID: <20231026.103712.109163977995540406.kazu@iij.ad.jp> Hello Zubin, > The fix for the issue (!11254) has been marked for backport and will > be included in the release. Thanks in advance! --Kazu From alan.zimm at gmail.com Thu Oct 26 17:10:21 2023 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Thu, 26 Oct 2023 17:10:21 +0000 Subject: "Merge-buddy" request Message-ID: Hi all I have been landing a series of MRs to simplify the exact print annotations. They are split into pieces so each change is not too big. I still have quite a number of them to land, but am finding the process slow, as I - make the MR - wait for CI to go green - wait for a review, or ping people on chat to do the review - once approved, land it. I don't want to loudly shout for reviewers each time, and become a general irritant. Is there anyone (ideally with an interest in the exact print annotations) that would like to be my "merge buddy" that I can easily ask to do the reviews? I am happy for each one to take a couple of days, so it is not a high pressure thing, I just want to get into a cadence on them. My current one is https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11496. Admittedly only one reviewer requested, and he is always busy, and I do not want to make him even more so. Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.peytonjones at gmail.com Fri Oct 27 08:25:34 2023 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Fri, 27 Oct 2023 09:25:34 +0100 Subject: "Merge-buddy" request In-Reply-To: References: Message-ID: I hope someone responds to Alan's call. Simplifying exact-print annotations is a noble goal, and Alan has been working hard on it. I hope someone feels able to support him. I don't think you need to be an exact-print expert. I'm sure Alan would be happy to teach you. And that learning journey might be very useful in itself, because it'll tell you want extra documentation (especially overview notes) is needed. Big thanks to Alan Simon On Thu, 26 Oct 2023 at 18:11, Alan & Kim Zimmerman wrote: > Hi all > > I have been landing a series of MRs to simplify the exact print > annotations. > They are split into pieces so each change is not too big. > I still have quite a number of them to land, but am finding the process > slow, as I > > - make the MR > - wait for CI to go green > - wait for a review, or ping people on chat to do the review > - once approved, land it. > > I don't want to loudly shout for reviewers each time, and become a general > irritant. > > Is there anyone (ideally with an interest in the exact print annotations) > that would like to be my "merge buddy" that I can easily ask to do the > reviews? > > I am happy for each one to take a couple of days, so it is not a high > pressure thing, I just want to get into a cadence on them. > > My current one is > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11496. Admittedly > only one reviewer requested, and he is always busy, and I do not want to > make him even more so. > > Alan > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zubin at well-typed.com Fri Oct 27 11:24:11 2023 From: zubin at well-typed.com (Zubin Duggal) Date: Fri, 27 Oct 2023 16:54:11 +0530 Subject: "Merge-buddy" request In-Reply-To: References: Message-ID: Feel free to assign me for reviews. On 23/10/26 17:10, Alan & Kim Zimmerman wrote: >Hi all > >I have been landing a series of MRs to simplify the exact print annotations. >They are split into pieces so each change is not too big. >I still have quite a number of them to land, but am finding the process >slow, as I > >- make the MR >- wait for CI to go green >- wait for a review, or ping people on chat to do the review >- once approved, land it. > >I don't want to loudly shout for reviewers each time, and become a general >irritant. > >Is there anyone (ideally with an interest in the exact print annotations) >that would like to be my "merge buddy" that I can easily ask to do the >reviews? > >I am happy for each one to take a couple of days, so it is not a high >pressure thing, I just want to get into a cadence on them. > >My current one is https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11496. >Admittedly only one reviewer requested, and he is always busy, and I do not >want to make him even more so. > >Alan >_______________________________________________ >ghc-devs mailing list >ghc-devs at haskell.org >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From alan.zimm at gmail.com Fri Oct 27 15:03:18 2023 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Fri, 27 Oct 2023 16:03:18 +0100 Subject: "Merge-buddy" request In-Reply-To: References: Message-ID: Thanks Zubin. And to re-iterate on what Simon said, I will cheerfully have a discussion with anyone who needs context for these. Alan On Fri, 27 Oct 2023, 12:24 Zubin Duggal, wrote: > Feel free to assign me for reviews. > > On 23/10/26 17:10, Alan & Kim Zimmerman wrote: > >Hi all > > > >I have been landing a series of MRs to simplify the exact print > annotations. > >They are split into pieces so each change is not too big. > >I still have quite a number of them to land, but am finding the process > >slow, as I > > > >- make the MR > >- wait for CI to go green > >- wait for a review, or ping people on chat to do the review > >- once approved, land it. > > > >I don't want to loudly shout for reviewers each time, and become a general > >irritant. > > > >Is there anyone (ideally with an interest in the exact print annotations) > >that would like to be my "merge buddy" that I can easily ask to do the > >reviews? > > > >I am happy for each one to take a couple of days, so it is not a high > >pressure thing, I just want to get into a cadence on them. > > > >My current one is > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11496. > >Admittedly only one reviewer requested, and he is always busy, and I do > not > >want to make him even more so. > > > >Alan > > >_______________________________________________ > >ghc-devs mailing list > >ghc-devs at haskell.org > >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rodrigo.m.mesquita at gmail.com Fri Oct 27 15:34:29 2023 From: rodrigo.m.mesquita at gmail.com (Rodrigo Mesquita) Date: Fri, 27 Oct 2023 16:34:29 +0100 Subject: PMC: addConCt and newtypes bottom info Message-ID: Dear Sebastian and GHC devs, Regarding this bit from the function addConCt in the GHC.HsToCore.Pmc.Solver module, Nothing -> do let pos' = PACA alt tvs args : pos let nabla_with bot' = nabla{ nabla_tm_st = ts{ts_facts = addToUSDFM env x (vi{vi_pos = pos', vi_bot = bot'})} } -- Do (2) in Note [Coverage checking Newtype matches] case (alt, args) of (PmAltConLike (RealDataCon dc), [y]) | isNewDataCon dc -> case bot of MaybeBot -> pure (nabla_with MaybeBot) IsBot -> addBotCt (nabla_with MaybeBot) y IsNotBot -> addNotBotCt (nabla_with MaybeBot) y _ -> assert (isPmAltConMatchStrict alt ) pure (nabla_with IsNotBot) -- strict match ==> not ⊥ My understanding is that given some x which we know e.g. cannot be bottom, if we learn that x ~ N y, where N is a newtype (NT), we move our knowledge of x not being bottom to the underlying NT Id y, since forcing the newtype in a pattern is equivalent to forcing the underlying NT Id. Additionally, we set x’s BottomInfo to MaybeBot — However, I don’t understand why we must reset x’s BotInfo to MaybeBot — couldn’t we keep it as it is while setting y’s BotInfo to the same info? An example where resetting this info on the newtype-match is important/necessary would be excellent. FWIW, I built and tested the PMC of ghc devel2 with MaybeBot -> pure (nabla_with MaybeBot) IsBot -> addBotCt (nabla_with IsBot) y IsNotBot -> addNotBotCt (nabla_with IsNotBot) y And it worked without warnings or errors… Thanks in advance! Rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Fri Oct 27 18:53:39 2023 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Fri, 27 Oct 2023 18:53:39 +0000 Subject: PMC: addConCt and newtypes bottom info In-Reply-To: References: Message-ID: Hi Rodrigo, Happy to see that you resumed your work on the pattern-match checker. I think you are right; we could reasonably just go with your code, not least because it is less confusing to retain as much info about `x` as possible. I don't think it makes a difference, because whenever we add a constraint `x ≁ ⊥` afterwards, we call `addNotBotCt` which will interpret this constraint as `y ≁ ⊥` via `lookupVarInfoNT`, and we have accurate BotInfo for `y`. Basically whenever we have seen `x ~ T y`, `T` Newtype, we will never look at `BotInfo` of `x` again. I thought it might become relevant in `generateInhabitingPatterns` for warning messages, but there we eagerly instantiate through NTs anyway. So by all means, open an MR for your change. Good work! Sebastian ------ Originalnachricht ------ Von: "Rodrigo Mesquita" An: "Sebastian Graf" Cc: "GHC developers" Gesendet: 27.10.2023 17:34:29 Betreff: PMC: addConCt and newtypes bottom info >Dear Sebastian and GHC devs, > >Regarding this bit from the function addConCt in the >GHC.HsToCore.Pmc.Solver module, > > Nothing -> do > let pos' = PACA alt tvs args : pos > let nabla_with bot' = > nabla{ nabla_tm_st = ts{ts_facts = addToUSDFM env x >(vi{vi_pos = pos', vi_bot = bot'})} } > -- Do (2) in Note [Coverage checking Newtype matches] > case (alt, args) of > (PmAltConLike (RealDataCon dc), [y]) | isNewDataCon dc -> > case bot of > MaybeBot -> pure (nabla_with MaybeBot) > IsBot -> addBotCt (nabla_with MaybeBot) y > IsNotBot -> addNotBotCt (nabla_with MaybeBot) y > _ -> assert (isPmAltConMatchStrict alt ) > pure (nabla_with IsNotBot) -- strict match ==> not ⊥ > >My understanding is that given some x which we know e.g. cannot be >bottom, if we learn that x ~ N y, where N is a newtype (NT), we move >our knowledge of x not being bottom to the underlying NT Id y, since >forcing the newtype in a pattern is equivalent to forcing the >underlying NT Id. > >Additionally, we set x’s BottomInfo to MaybeBot — >However, I don’t understand why we must reset x’s BotInfo to MaybeBot — >couldn’t we keep it as it is while setting y’s BotInfo to the same >info? >An example where resetting this info on the newtype-match is >important/necessary would be excellent. > >FWIW, I built and tested the PMC of ghc devel2 with > > MaybeBot -> pure (nabla_with MaybeBot) > IsBot -> addBotCt (nabla_with IsBot) y > IsNotBot -> addNotBotCt (nabla_with IsNotBot) y > >And it worked without warnings or errors… > >Thanks in advance! >Rodrigo -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandrer_b at outlook.com Sat Oct 28 01:34:44 2023 From: alexandrer_b at outlook.com (=?iso-8859-1?Q?Alexandre_Rodrigues_Bald=E9?=) Date: Sat, 28 Oct 2023 01:34:44 +0000 Subject: "Merge-buddy" request In-Reply-To: References: Message-ID: Zubin has already kindly volunteered; if a second review is needed, I also don't mind taking a look - though I might need some context. I'm @rockbmb ________________________________ From: ghc-devs on behalf of Alan & Kim Zimmerman Sent: 27 October 2023 16:03 To: Zubin Duggal Cc: ghc-devs Subject: Re: "Merge-buddy" request Thanks Zubin. And to re-iterate on what Simon said, I will cheerfully have a discussion with anyone who needs context for these. Alan On Fri, 27 Oct 2023, 12:24 Zubin Duggal, > wrote: Feel free to assign me for reviews. On 23/10/26 17:10, Alan & Kim Zimmerman wrote: >Hi all > >I have been landing a series of MRs to simplify the exact print annotations. >They are split into pieces so each change is not too big. >I still have quite a number of them to land, but am finding the process >slow, as I > >- make the MR >- wait for CI to go green >- wait for a review, or ping people on chat to do the review >- once approved, land it. > >I don't want to loudly shout for reviewers each time, and become a general >irritant. > >Is there anyone (ideally with an interest in the exact print annotations) >that would like to be my "merge buddy" that I can easily ask to do the >reviews? > >I am happy for each one to take a couple of days, so it is not a high >pressure thing, I just want to get into a cadence on them. > >My current one is https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11496. >Admittedly only one reviewer requested, and he is always busy, and I do not >want to make him even more so. > >Alan >_______________________________________________ >ghc-devs mailing list >ghc-devs at haskell.org >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ietf-dane at dukhovni.org Mon Oct 30 05:06:12 2023 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Mon, 30 Oct 2023 01:06:12 -0400 Subject: Type-level sized Word literals??? Message-ID: I am working on some code where it is useful to have types indexed by a 16-bit unsigned value. Presently, I am using type-level naturals and having to now and then provide witnesses that a 'Nat' value I am working with is at most 65535. Also, perhaps there's some inefficiency here, as Naturals have two constructors, and so a more complex representation with (AFAIK) no unboxed forms. I was wondering what it would take to have type-level fixed-size Words (Word8, Word16, Word32, Word64) to go along with the Nats? It looks like some of the machinery (KnownWord16, SomeWord16, wordVal16, etc.) can be copied straight out of GHC.TypeNats with minor changes, and that much works, but the three major things that are't easily done seem to be: - There are it seems no TypeReps for types of Kind Word16, so one can't have (Typeable (Foo w)) with (w :: Word16). - There are no literals of a promoted Word16 Kind. type Foo :: Word16 -> Type data Foo w = MkFoo Int -- 1 has Kind 'Natural' (a.k.a. Nat) x = MkFoo 13 :: Foo 1 -- Rejected, -- The new ExtendedLiterals syntax does not help -- x = MkFoo 13 :: Foo (W16# 1#Word16) -- Syntax error! - There are unsurprisingly also no built-in 'KnownWord16' instances for any hypothetical type-level literals of Kind Word16. Likely the use case for type-level fixed-size words is too specialised to rush to shoehorn into GHC, but is there something on the not too distant horizon that would make it easier and reasonable to have fixed-size unsigned integral type literals available? [ I don't see a use-case for unsigned versions, they can trivially be represented by the unsigned value of the same width. ] With some inconvenience, in many cases I can perhaps synthesise Proxies for types of Kind Word16, and just never use literals directly. -- Viktor. From vladislav at serokell.io Mon Oct 30 08:20:16 2023 From: vladislav at serokell.io (Vladislav Zavialov) Date: Mon, 30 Oct 2023 09:20:16 +0100 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: > I am working on some code where it is useful to have types indexed by a 16-bit unsigned value. This is great to hear. I've always wanted to make it possible to promote all numeric types: Natural, Word8, Word16, Word32, Word64, Integer, Int8, Int16, Int32, Int64. (Then, as the next step, even the floating-point types Float and Double). I see it as a step towards universal promotion, i.e. being able to use any data type as a kind. The problem is that such a change would require a GHC proposal, and I don't have a strong motivating use case to write one. But you seem to have one! If you'd like to co-author a GHC proposal and if the proposal gets accepted, I can implement the feature. Here's how I imagine it could work. 1. Currently, a type-level literal like `15` is inferred to have kind `Nat` (and `Nat` is a synonym for `Natural` nowadays). At the term-level, however, the type of `15` is `forall {a}. Num a => a`. I'd like to follow the term-level precedent as closely as possible, except we don't have type class constraints in kinds, so it's going to be simply `15 :: forall {k}. k`. 2. The desugaring should also follow the term-level precedent. `15` actually stands for `fromInteger (15 :: Integer)`, and I expect no less at the type level, where we could introduce a type family `FromInteger :: Integer -> k`, with the following instances type instance FromInteger @Natural n = NaturalFromInteger n type instance FromInteger @Word8 n = Word8FromInteger n type instance FromInteger @Word16 n = Word16FromInteger n ... The helper type families `NaturalFromInteger`, `Word8FromInteger`, `Word16FromInteger` etc. are partial, e.g. `NaturalFromInteger (10 :: Integer)` yields `10 :: Natural` whereas `NaturalFromInteger (-10)` is stuck. I have a fairly good idea of what it'd take to implement this (i.e. the changes to the GHC parser, type checker, and libraries), and the change has been on my mind for a while. The use case that you have might be the last piece of the puzzle to get this thing rolling. Can you tell more about the code you're writing? Would it be possible to use it as the basis for the "Motivation" section of a GHC proposal? Vlad On Mon, Oct 30, 2023 at 6:06 AM Viktor Dukhovni wrote: > > I am working on some code where it is useful to have types indexed by a > 16-bit unsigned value. > > Presently, I am using type-level naturals and having to now and then > provide witnesses that a 'Nat' value I am working with is at most 65535. > > Also, perhaps there's some inefficiency here, as Naturals have two > constructors, and so a more complex representation with (AFAIK) no > unboxed forms. > > I was wondering what it would take to have type-level fixed-size > Words (Word8, Word16, Word32, Word64) to go along with the Nats? > > It looks like some of the machinery (KnownWord16, SomeWord16, wordVal16, > etc.) can be copied straight out of GHC.TypeNats with minor changes, and > that much works, but the three major things that are't easily done seem > to be: > > - There are it seems no TypeReps for types of Kind Word16, so one can't > have (Typeable (Foo w)) with (w :: Word16). > > - There are no literals of a promoted Word16 Kind. > > type Foo :: Word16 -> Type > data Foo w = MkFoo Int > > -- 1 has Kind 'Natural' (a.k.a. Nat) > x = MkFoo 13 :: Foo 1 -- Rejected, > > -- The new ExtendedLiterals syntax does not help > -- > x = MkFoo 13 :: Foo (W16# 1#Word16) -- Syntax error! > > - There are unsurprisingly also no built-in 'KnownWord16' instances > for any hypothetical type-level literals of Kind Word16. > > Likely the use case for type-level fixed-size words is too specialised > to rush to shoehorn into GHC, but is there something on the not too > distant horizon that would make it easier and reasonable to have > fixed-size unsigned integral type literals available? > > [ I don't see a use-case for unsigned versions, they can trivially be > represented by the unsigned value of the same width. ] > > With some inconvenience, in many cases I can perhaps synthesise Proxies > for types of Kind Word16, and just never use literals directly. > > -- > Viktor. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From simon.peytonjones at gmail.com Mon Oct 30 09:04:48 2023 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Mon, 30 Oct 2023 09:04:48 +0000 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: I'm pretty cautious about attempting to replicate type classes (or a weaker version thereof) at the kind level. An alternative would be to us *non-overloaded* literals. Simon On Mon, 30 Oct 2023 at 08:20, Vladislav Zavialov via ghc-devs < ghc-devs at haskell.org> wrote: > > I am working on some code where it is useful to have types indexed by a > 16-bit unsigned value. > > This is great to hear. I've always wanted to make it possible to > promote all numeric types: Natural, Word8, Word16, Word32, Word64, > Integer, Int8, Int16, Int32, Int64. (Then, as the next step, even the > floating-point types Float and Double). I see it as a step towards > universal promotion, i.e. being able to use any data type as a kind. > > The problem is that such a change would require a GHC proposal, and I > don't have a strong motivating use case to write one. But you seem to > have one! If you'd like to co-author a GHC proposal and if the > proposal gets accepted, I can implement the feature. > > Here's how I imagine it could work. > > 1. Currently, a type-level literal like `15` is inferred to have kind > `Nat` (and `Nat` is a synonym for `Natural` nowadays). At the > term-level, however, the type of `15` is `forall {a}. Num a => a`. I'd > like to follow the term-level precedent as closely as possible, except > we don't have type class constraints in kinds, so it's going to be > simply `15 :: forall {k}. k`. > > 2. The desugaring should also follow the term-level precedent. `15` > actually stands for `fromInteger (15 :: Integer)`, and I expect no > less at the type level, where we could introduce a type family > `FromInteger :: Integer -> k`, with the following instances > > type instance FromInteger @Natural n = NaturalFromInteger n > type instance FromInteger @Word8 n = Word8FromInteger n > type instance FromInteger @Word16 n = Word16FromInteger n > ... > > The helper type families `NaturalFromInteger`, `Word8FromInteger`, > `Word16FromInteger` etc. are partial, e.g. `NaturalFromInteger (10 :: > Integer)` yields `10 :: Natural` whereas `NaturalFromInteger (-10)` is > stuck. > > I have a fairly good idea of what it'd take to implement this (i.e. > the changes to the GHC parser, type checker, and libraries), and the > change has been on my mind for a while. The use case that you have > might be the last piece of the puzzle to get this thing rolling. > > Can you tell more about the code you're writing? Would it be possible > to use it as the basis for the "Motivation" section of a GHC proposal? > > Vlad > > On Mon, Oct 30, 2023 at 6:06 AM Viktor Dukhovni > wrote: > > > > I am working on some code where it is useful to have types indexed by a > > 16-bit unsigned value. > > > > Presently, I am using type-level naturals and having to now and then > > provide witnesses that a 'Nat' value I am working with is at most 65535. > > > > Also, perhaps there's some inefficiency here, as Naturals have two > > constructors, and so a more complex representation with (AFAIK) no > > unboxed forms. > > > > I was wondering what it would take to have type-level fixed-size > > Words (Word8, Word16, Word32, Word64) to go along with the Nats? > > > > It looks like some of the machinery (KnownWord16, SomeWord16, wordVal16, > > etc.) can be copied straight out of GHC.TypeNats with minor changes, and > > that much works, but the three major things that are't easily done seem > > to be: > > > > - There are it seems no TypeReps for types of Kind Word16, so one > can't > > have (Typeable (Foo w)) with (w :: Word16). > > > > - There are no literals of a promoted Word16 Kind. > > > > type Foo :: Word16 -> Type > > data Foo w = MkFoo Int > > > > -- 1 has Kind 'Natural' (a.k.a. Nat) > > x = MkFoo 13 :: Foo 1 -- Rejected, > > > > -- The new ExtendedLiterals syntax does not help > > -- > > x = MkFoo 13 :: Foo (W16# 1#Word16) -- Syntax error! > > > > - There are unsurprisingly also no built-in 'KnownWord16' instances > > for any hypothetical type-level literals of Kind Word16. > > > > Likely the use case for type-level fixed-size words is too specialised > > to rush to shoehorn into GHC, but is there something on the not too > > distant horizon that would make it easier and reasonable to have > > fixed-size unsigned integral type literals available? > > > > [ I don't see a use-case for unsigned versions, they can trivially be > > represented by the unsigned value of the same width. ] > > > > With some inconvenience, in many cases I can perhaps synthesise Proxies > > for types of Kind Word16, and just never use literals directly. > > > > -- > > Viktor. > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladislav at serokell.io Mon Oct 30 09:32:16 2023 From: vladislav at serokell.io (Vladislav Zavialov) Date: Mon, 30 Oct 2023 10:32:16 +0100 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: I agree caution is warranted, but I still want the type level to behave as closely as possible to the term level, where literals are currently overloaded. I don't care if it's monomorphic literals everywhere or overloaded literals everywhere, but I oppose a discrepancy. Vlad On Mon, Oct 30, 2023, 10:05 Simon Peyton Jones wrote: > I'm pretty cautious about attempting to replicate type classes (or a > weaker version thereof) at the kind level. An alternative would be to us > *non-overloaded* literals. > > Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at richarde.dev Mon Oct 30 11:15:27 2023 From: lists at richarde.dev (Richard Eisenberg) Date: Mon, 30 Oct 2023 11:15:27 +0000 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: <010f018b804bb758-6ed99d94-bbd7-4e5d-be22-0406f316af73-000000@us-east-2.amazonses.com> Modulo the backward-compatibility piece around today's type-level numbers, I'm in support of this direction. No new type machinery is needed, other than a new interpretation for literals, because type families can already infer a kind argument from the return kind. This is almost entirely a change to libraries, not to GHC itself. Richard > On Oct 30, 2023, at 5:32 AM, Vladislav Zavialov via ghc-devs wrote: > > I agree caution is warranted, but I still want the type level to behave as closely as possible to the term level, where literals are currently overloaded. > > I don't care if it's monomorphic literals everywhere or overloaded literals everywhere, but I oppose a discrepancy. > > Vlad > > On Mon, Oct 30, 2023, 10:05 Simon Peyton Jones > wrote: > I'm pretty cautious about attempting to replicate type classes (or a weaker version thereof) at the kind level. An alternative would be to us *non-overloaded* literals. > > Simon > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ietf-dane at dukhovni.org Mon Oct 30 17:26:08 2023 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Mon, 30 Oct 2023 13:26:08 -0400 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: On Mon, Oct 30, 2023 at 09:20:16AM +0100, Vladislav Zavialov via ghc-devs wrote: > Can you tell more about the code you're writing? Would it be possible > to use it as the basis for the "Motivation" section of a GHC proposal? > Working with DNS resource records (A, NS, CNAME, SOA, MX, SRV, ...) requires a runtime extensible data model: - Some RR types will be known at DNS-library compile-time. - Some RR types can defined and added at runtime (application compile-time, and registered with the DNS library). - Some RR types may appear "on the wire" in serialised form, with the RRTYPE not known to either the library or application code. The library data model has separate types for the "rrData" parts of each resource record type, which are existentially quantified in the RR: RR { rrOwner, rrTTL, rrClass, rrData :: RData } RData = forall a. KnownRData a => RData a with the usual: fromRData :: KnownRData a => RData -> Maybe a fromRData (RData a) = cast a To filter a list of resource records to those of a particular RR type, I have: monoList :: forall a t. (KnownRData a, Foldable t) => t RData -> [a] monoList = foldr (maybe id (:) . fromRData) go [] I'd like to be able to use this also to distinguish between "OpaqueRData" resource record types, so there's actually a separate opaque type for each 16-bit RR number: {-# LANGUAGE AllowAmbiguousTypes #-} type OpaqueRData :: Word16 -> Type data OpaqueRData w = OpaqueRData { getOpaqueRData :: Bytes16 } -- Nat16 constraint enforces 65535 ceiling -- natVal16 returns Nat as Word16. -- instance Nat16 n => KnownRData (OpaqueRData n) where rdType = RRTYPE $ natVal16 @n This works, because the phantom indices have to match for "cast" to return a Just value, so that, for example: λ> x1 = RData $ (OpaqueRData (coerce ("abc" :: ShortByteString)) :: OpaqueRData 1) λ> fromRData x1 :: (Maybe (OpaqueRData 1)) Just (OpaqueRData @1 "616263") λ> fromRData x1 :: (Maybe (OpaqueRData 2)) Nothing λ> l1 = monoList [x1] :: [OpaqueRData 1] λ> l2 = monoList [x1] :: [OpaqueRData 2] λ> hPutBuilder stdout $ foldr (presentLn) ("That's all folks!\n") l1 \# 3 616263 That's all folks! λ> hPutBuilder stdout $ foldr (presentLn) ("That's all folks!\n") l2 That's all folks! In addition to labeling unknown RData with Word16 values, I also type-index unknown EDNS options (they're elements of the OPT pseudo RR that carries DNS protocol extensions) and unknown SVCB/HTTPS key/value service parameter pairs. Applications can register novel RData types, EDNS options, and SVCB key/value types at runtime, and the extended code points behave just like the "built-in" ones, because the only "built-in" code points are the opaque ones, the others are registered at runtime by the library as part of default resolver settings. So this is how I end up with Word16-indexed types. One might argue that "OpaqueRData" could be a single type, and that filtering by RRTYPE should have the "rrType" method taking a value to optionally inspect, but I like the type-level separation even between Opaque data of different RRTYPEs, and ditto for EDNS options and SVCB/HTTPS fields. This supports view patterns: f (fromRData -> Just (T_a ipv4)) = ... f (fromRData -> Just (T_aaaa ipv6)) = ... which should "morally" also work for: getBlob42 :: OpaqueRData 42 -> ShortByteString getBlob42 = fromBytes16 . getOpaqueRData f (fmap getBlob42 . fromRData -> Just blob) = ... yielding just the serialised blobs of RRTYPE 42, with little chance of accidentally pulling in blobs of the wrong RRTYPE. I may before long add an associated type to the KnownRData class: type RdType :: Nat -- Ideally some day Word16 making it possible to write: -- Identity functions on the actual Opaque types. toOpaque :: a -> Opaque (RdType a) fromOpaque :: Opaque (RdType a) -> a at which point a simple tweak to the above "blob" pattern match could also work when the RRtype 42 was decoded as known: f (fmap getBlob42 . fromRData . toOpaque -> Just blob)) = ... -- Viktor. From lists at qseep.net Tue Oct 31 20:40:06 2023 From: lists at qseep.net (Lyle Kopnicky) Date: Tue, 31 Oct 2023 13:40:06 -0700 Subject: Type-level sized Word literals??? In-Reply-To: References: Message-ID: We do have monomorphic unboxed literals via ExtendedLiterals. Can we make use of those or perhaps a “boxed” variant of those to get monomorphic literals at the value and type levels? E.g., 0xFF#Word8 is monomorphically a Word8#. So could we have: 0xFF at Word8 (or similar notation) is a boxed Word8? And that would automatically be promoted to the type level. Really, shouldn’t both the boxed and unboxed variants be promotable to the type level? — Lyle On Oct 30, 2023 at 2:04:48 AM, Simon Peyton Jones < simon.peytonjones at gmail.com> wrote: > I'm pretty cautious about attempting to replicate type classes (or a > weaker version thereof) at the kind level. An alternative would be to us > *non-overloaded* literals. > > Simon > > On Mon, 30 Oct 2023 at 08:20, Vladislav Zavialov via ghc-devs < > ghc-devs at haskell.org> wrote: > >> > I am working on some code where it is useful to have types indexed by a >> 16-bit unsigned value. >> >> This is great to hear. I've always wanted to make it possible to >> promote all numeric types: Natural, Word8, Word16, Word32, Word64, >> Integer, Int8, Int16, Int32, Int64. (Then, as the next step, even the >> floating-point types Float and Double). I see it as a step towards >> universal promotion, i.e. being able to use any data type as a kind. >> >> The problem is that such a change would require a GHC proposal, and I >> don't have a strong motivating use case to write one. But you seem to >> have one! If you'd like to co-author a GHC proposal and if the >> proposal gets accepted, I can implement the feature. >> >> Here's how I imagine it could work. >> >> 1. Currently, a type-level literal like `15` is inferred to have kind >> `Nat` (and `Nat` is a synonym for `Natural` nowadays). At the >> term-level, however, the type of `15` is `forall {a}. Num a => a`. I'd >> like to follow the term-level precedent as closely as possible, except >> we don't have type class constraints in kinds, so it's going to be >> simply `15 :: forall {k}. k`. >> >> 2. The desugaring should also follow the term-level precedent. `15` >> actually stands for `fromInteger (15 :: Integer)`, and I expect no >> less at the type level, where we could introduce a type family >> `FromInteger :: Integer -> k`, with the following instances >> >> type instance FromInteger @Natural n = NaturalFromInteger n >> type instance FromInteger @Word8 n = Word8FromInteger n >> type instance FromInteger @Word16 n = Word16FromInteger n >> ... >> >> The helper type families `NaturalFromInteger`, `Word8FromInteger`, >> `Word16FromInteger` etc. are partial, e.g. `NaturalFromInteger (10 :: >> Integer)` yields `10 :: Natural` whereas `NaturalFromInteger (-10)` is >> stuck. >> >> I have a fairly good idea of what it'd take to implement this (i.e. >> the changes to the GHC parser, type checker, and libraries), and the >> change has been on my mind for a while. The use case that you have >> might be the last piece of the puzzle to get this thing rolling. >> >> Can you tell more about the code you're writing? Would it be possible >> to use it as the basis for the "Motivation" section of a GHC proposal? >> >> Vlad >> >> On Mon, Oct 30, 2023 at 6:06 AM Viktor Dukhovni >> wrote: >> > >> > I am working on some code where it is useful to have types indexed by a >> > 16-bit unsigned value. >> > >> > Presently, I am using type-level naturals and having to now and then >> > provide witnesses that a 'Nat' value I am working with is at most 65535. >> > >> > Also, perhaps there's some inefficiency here, as Naturals have two >> > constructors, and so a more complex representation with (AFAIK) no >> > unboxed forms. >> > >> > I was wondering what it would take to have type-level fixed-size >> > Words (Word8, Word16, Word32, Word64) to go along with the Nats? >> > >> > It looks like some of the machinery (KnownWord16, SomeWord16, wordVal16, >> > etc.) can be copied straight out of GHC.TypeNats with minor changes, and >> > that much works, but the three major things that are't easily done seem >> > to be: >> > >> > - There are it seems no TypeReps for types of Kind Word16, so one >> can't >> > have (Typeable (Foo w)) with (w :: Word16). >> > >> > - There are no literals of a promoted Word16 Kind. >> > >> > type Foo :: Word16 -> Type >> > data Foo w = MkFoo Int >> > >> > -- 1 has Kind 'Natural' (a.k.a. Nat) >> > x = MkFoo 13 :: Foo 1 -- Rejected, >> > >> > -- The new ExtendedLiterals syntax does not help >> > -- >> > x = MkFoo 13 :: Foo (W16# 1#Word16) -- Syntax error! >> > >> > - There are unsurprisingly also no built-in 'KnownWord16' instances >> > for any hypothetical type-level literals of Kind Word16. >> > >> > Likely the use case for type-level fixed-size words is too specialised >> > to rush to shoehorn into GHC, but is there something on the not too >> > distant horizon that would make it easier and reasonable to have >> > fixed-size unsigned integral type literals available? >> > >> > [ I don't see a use-case for unsigned versions, they can trivially be >> > represented by the unsigned value of the same width. ] >> > >> > With some inconvenience, in many cases I can perhaps synthesise Proxies >> > for types of Kind Word16, and just never use literals directly. >> > >> > -- >> > Viktor. >> > _______________________________________________ >> > ghc-devs mailing list >> > ghc-devs at haskell.org >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: