From lexi.lambda at gmail.com Wed Jun 1 01:37:47 2022 From: lexi.lambda at gmail.com (Alexis King) Date: Tue, 31 May 2022 20:37:47 -0500 Subject: The GHC(i)/RTS linker and Template Haskell Message-ID: Hi all, I’ve recently been trying to better understand how and where time is spent at compile-time when running Template Haskell splices, and one of the areas I’ve been struggling to figure out is the operation of the linker. From reading the source code, here’s a summary of what I think I’ve figured out so far: - TH splices are executed using the GHCi interpreter, though it may be internal or external (if -fexternal-interpreter is used). - Regardless of which mode is used, TH splices need their dependencies loaded into the interpreter context before they can be run. This is handled by the call to loadDecls in hscCompileCoreExpr', which in turn calls loadDependencies in GHC.Linker.Loader. - loadDependencies loads packages and modules in different ways. Package dependencies are just loaded via the appropriate built shared libraries, but modules from the current package have to be loaded a different way, via loadObjects (also in GHC.Linker.Loader). Here, however, is where I get a bit lost. GHC has two strategies for loading individual objects, which it chooses between depending on whether the current value of interpreterDynamic is True. But I don’t actually understand what interpreterDynamic means! The Haddock comment just says that it determines whether or not the “interpreter uses the Dynamic way”, but I don’t see why that matters. My understanding was that GHCi *always* requires dynamic linking, since it is, after all, loading code dynamically. Under what circumstances would interpreterDynamic ever be False? Furthermore, I don’t actually understand precisely how and why this influences the choice of loading strategy. In the case that interpreterDynamic is True, GHC appears to convert the desired dyn_o object into a shared library by calling the system linker, then loads that, which can be very slow but otherwise works. However, when interpreterDynamic is False, it loads the object directly. Both paths eventually call into “the RTS linker”, implemented in rts/Linker.c, to actually load the resulting object. I have found precious little information on what the RTS linker does, in which contexts it’s used, or how precisely it works. Note [runtime-linker-phases] at the top of Linker.c has some information, but it’s mostly a high-level description of what the code actually does rather than an explanation of its role in the bigger picture. Does anyone know of any resources they could possibly point me to that help to explain how all the pieces fit together here? I’ve spent quite a bit of time reading the code, but I’m afraid I still haven’t managed to see the forest for the trees. Thanks, Alexis -------------- next part -------------- An HTML attachment was scrubbed... URL: From lonetiger at gmail.com Wed Jun 1 05:09:39 2022 From: lonetiger at gmail.com (Phyx) Date: Wed, 1 Jun 2022 06:09:39 +0100 Subject: The GHC(i)/RTS linker and Template Haskell In-Reply-To: References: Message-ID: Hi Alexis, Most information on this can be found on the Wiki, where a lot of these design decisions are made. e.g. https://gitlab.haskell.org/ghc/ghc/-/wikis/dynamic-ghc-programs The points you've figured out are correct so far, to answer some of your questions: > But I don’t actually understand what interpreterDynamic means! The Haddock comment just says that it determines whether or not the “interpreter uses the Dynamic way”, but I don’t see why that matters. My understanding was that GHCi *always* requires dynamic linking, since it is, after all, loading code dynamically. DynamicWay essentially means whether or not the runtime linker uses the platform linker under the hood. When dynamic way your object files will be linked into a shared library by the RTS linker and that shared library loaded. This means that the linker itself doesn't have to do a bunch of work such as relocation processing etc. For most Unix platforms this is the default. The downside of this approach is that on every change, i.e. if you load a new object file into scope, you have to relink the shared library, unload the old one, and link the new one in. This brings with it its own set of problems, such as what happens to references you already hold to symbols on the old shared library etc. > Under what circumstances would interpreterDynamic ever be False? For instance, on Windows. Linking on Windows using the system linker is generally slower, so creating multiple shared libraries on the fly is time consuming. There are also some practical issues, for instance base is so big that it doesn't fit into a single DLL. or how Windows handles data and code accesses to symbols Shared libraries, etc. This means that on Windows we load object files and internally do all relocation processing, run initializers etc. Everything you would need to do to be able to run the code inside the object file. There are several other platforms as well, such as Android, where there's no system linker to call etc. > In the case that interpreterDynamic is True, GHC appears to convert the desired dyn_o object into a shared library by calling the system linker, then loads that, which can be very slow but otherwise works. However, when interpreterDynamic is False, it loads the object directly. Both paths eventually call into “the RTS linker”, implemented in rts/Linker.c, to actually load the resulting object. Yes, the end goal is to be able to resolve a function name to an address. So whichever strategy is chosen, we must in the end register the functions with the RTS. Though loading a shared lib is much less error prone than loading the object files directly. It also uses less memory and can benefit from linker level optimizations that we don't implement in the RTS linker. Also loading a shared library has additional benefits such as that the system loader deals with running initializers, registering exception tables, etc. Hope this clarified it somewhat, but if you have any more questions feel free to ask. Regards, Tamar On Wed, Jun 1, 2022 at 2:38 AM Alexis King wrote: > Hi all, > > I’ve recently been trying to better understand how and where time is spent > at compile-time when running Template Haskell splices, and one of the areas > I’ve been struggling to figure out is the operation of the linker. From > reading the source code, here’s a summary of what I think I’ve figured out > so far: > > - TH splices are executed using the GHCi interpreter, though it may be > internal or external (if -fexternal-interpreter is used). > > - Regardless of which mode is used, TH splices need their dependencies > loaded into the interpreter context before they can be run. This is handled > by the call to loadDecls in hscCompileCoreExpr', which in turn calls > loadDependencies in GHC.Linker.Loader. > > - loadDependencies loads packages and modules in different ways. > Package dependencies are just loaded via the appropriate built shared > libraries, but modules from the current package have to be loaded a > different way, via loadObjects (also in GHC.Linker.Loader). > > Here, however, is where I get a bit lost. GHC has two strategies for > loading individual objects, which it chooses between depending on whether > the current value of interpreterDynamic is True. But I don’t actually > understand what interpreterDynamic means! The Haddock comment just says > that it determines whether or not the “interpreter uses the Dynamic way”, > but I don’t see why that matters. My understanding was that GHCi *always* > requires dynamic linking, since it is, after all, loading code dynamically. > Under what circumstances would interpreterDynamic ever be False? > > Furthermore, I don’t actually understand precisely how and why this > influences the choice of loading strategy. In the case that > interpreterDynamic is True, GHC appears to convert the desired dyn_o object > into a shared library by calling the system linker, then loads that, which > can be very slow but otherwise works. However, when interpreterDynamic is > False, it loads the object directly. Both paths eventually call into “the > RTS linker”, implemented in rts/Linker.c, to actually load the resulting > object. > > I have found precious little information on what the RTS linker does, in > which contexts it’s used, or how precisely it works. Note > [runtime-linker-phases] at the top of Linker.c has some information, but > it’s mostly a high-level description of what the code actually does rather > than an explanation of its role in the bigger picture. Does anyone know of > any resources they could possibly point me to that help to explain how all > the pieces fit together here? I’ve spent quite a bit of time reading the > code, but I’m afraid I still haven’t managed to see the forest for the > trees. > > Thanks, > Alexis > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Wed Jun 1 05:16:51 2022 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 1 Jun 2022 07:16:51 +0200 Subject: The GHC(i)/RTS linker and Template Haskell In-Reply-To: References: Message-ID: Hi Alexis, let me try to provide the high up view. I'm sorry if I'm going a bit overboard on details you already know. But let's start with clearing up a misconception first. No, GHCi does not always require dynamic linking. At the very abstract level we have a compiler that knows how to turn various inputs into object code. This includes, C, Cmm, Haskell, assembly (.c, .S, .cmm, .hs -> .o). Thus a complete haskell package ends up as a bunch of object code files. We know that for dynamic linkers, we may need slightly different arguments (e.g. PIC). We next roll these up into archives (.a) and occasionally a pre-linked object file (e.g. link the whole set of object files into one object file, and resolve internal references); as well as a dynamic (shared object, dylib) file. For GHCi's purposes (and TH), we ultimately want to call a haskell function to produce some AST to splice in. This haskell function might not be defined in a different package, but the same, so we'll have to deal with some in-flight packages anyway. We may habe some Byte Code Object (BCO) glue code to invoke the haskell function, which GHCi will interpret during evaluation. However that function can depend on a large dependency tree, and we don't have BCO for everything. I still think it would be nice to have an abstract machine and an Intermediate Representation/ByteCode, that's a much larger project though. Also until recently BCO's couldn't encode unboxed types/sums even. So given the BCO glue code, we really want to call into object code (also for performance). You can instruct GHCi to prefer object code as well via -fobject-code. This now leads us to the need of getting the object code somehow into memory and running it. The dynamic system linker approach would be to turn the object code with the function we want to call into a shared library, and just hand that over to the linker (e.g. dlopen). However, GHC has for a long time grown it's own in-memory static linker. As such it has the capability to load object file (.o) and resolve them on the fly. There is no need for system shared libraries, a system linker, and to deal with potential bugs in that linker. It also means we can link on platforms that don't have a system linker or a severely restricted one (e.g. iOS). So from a high level you can look at GHC's RTS linker as a special feature of GHC that allows us to not need a system provided dynamic linker, if there is none available, or using it is undesirable. Whether or not stuff is loaded through the internal or external interpreter has near no difference. You _can_ load different abi's through the external iserv (as that iserv can be built against a different abi). Hope this helps a bit? Feel free to ask more questions. Cheers, Moritz On Wed, 1 Jun 2022 at 03:38, Alexis King wrote: > Hi all, > > I’ve recently been trying to better understand how and where time is spent > at compile-time when running Template Haskell splices, and one of the areas > I’ve been struggling to figure out is the operation of the linker. From > reading the source code, here’s a summary of what I think I’ve figured out > so far: > > - TH splices are executed using the GHCi interpreter, though it may be > internal or external (if -fexternal-interpreter is used). > > - Regardless of which mode is used, TH splices need their dependencies > loaded into the interpreter context before they can be run. This is handled > by the call to loadDecls in hscCompileCoreExpr', which in turn calls > loadDependencies in GHC.Linker.Loader. > > - loadDependencies loads packages and modules in different ways. > Package dependencies are just loaded via the appropriate built shared > libraries, but modules from the current package have to be loaded a > different way, via loadObjects (also in GHC.Linker.Loader). > > Here, however, is where I get a bit lost. GHC has two strategies for > loading individual objects, which it chooses between depending on whether > the current value of interpreterDynamic is True. But I don’t actually > understand what interpreterDynamic means! The Haddock comment just says > that it determines whether or not the “interpreter uses the Dynamic way”, > but I don’t see why that matters. My understanding was that GHCi *always* > requires dynamic linking, since it is, after all, loading code dynamically. > Under what circumstances would interpreterDynamic ever be False? > > Furthermore, I don’t actually understand precisely how and why this > influences the choice of loading strategy. In the case that > interpreterDynamic is True, GHC appears to convert the desired dyn_o object > into a shared library by calling the system linker, then loads that, which > can be very slow but otherwise works. However, when interpreterDynamic is > False, it loads the object directly. Both paths eventually call into “the > RTS linker”, implemented in rts/Linker.c, to actually load the resulting > object. > > I have found precious little information on what the RTS linker does, in > which contexts it’s used, or how precisely it works. Note > [runtime-linker-phases] at the top of Linker.c has some information, but > it’s mostly a high-level description of what the code actually does rather > than an explanation of its role in the bigger picture. Does anyone know of > any resources they could possibly point me to that help to explain how all > the pieces fit together here? I’ve spent quite a bit of time reading the > code, but I’m afraid I still haven’t managed to see the forest for the > trees. > > Thanks, > Alexis > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at richarde.dev Thu Jun 2 17:37:12 2022 From: lists at richarde.dev (Richard Eisenberg) Date: Thu, 2 Jun 2022 17:37:12 +0000 Subject: error codes in GHC Message-ID: <010f0181257e22ff-341aac97-de26-40f2-9cc2-d13c7cf2c0d4-000000@us-east-2.amazonses.com> Hi devs, I just wanted to call your attention to https://gitlab.haskell.org/ghc/ghc/-/issues/21684, which will add a little more bureaucracy to adding error messages in the future, but all for a good cause. Please take a look and offer any feedback! Thanks, Richard From hecate at glitchbra.in Mon Jun 6 11:47:54 2022 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Mon, 6 Jun 2022 13:47:54 +0200 Subject: Haddock needs help In-Reply-To: References: Message-ID: <5cbf449c-325d-2060-a534-0cda89822092@glitchbra.in> Hi Mikolaj, you highlight something very interesting. Indeed there is no shame in disclosing technical debt. That being said the problem isn't entirely technical. The main challenges here are the consolidation of expertise, onboarding of newcomers, a sensible product roadmap, and a realistic path forward when it comes to the big features I'd like to ship in the future. At the moment I would be more comfortable with starting by onboarding a restricted group of people so that my attention can be properly focused on ramping up their skills, which is why I posted on the ghc-devs mailing-list, as GHC is our main (if not only) consumer, and GHC developers have already made contributions to Haddock. Cheers, Hécate Le 26/05/2022 à 17:12, Mikolaj Konarski a écrit : > Talking as a Haskell user, but also a cabal maintainer: > the Haddock work Hécate describes is crucial for the health > and efficiency of the whole ecosystem and the sooner > we can start it, the less drag it's going to have on the rest. > > Given that GHC expertise is not necessary for many > of the tasks involved, could we forward this message > to other media? I don't think there is shame in disclosing > technical debt (as opposed to hiding it) and I think the way > it's worded, it may be viewed as a challenge, not a turn-off. > Still, are there any suggestions on how to tweak the text > before an announcement on discourse, reddit, etc.? > Would, e.g., HF, like to chime in early in each of the ensuing > discussions? If so, how would we know where it gets posted to? > > Cheers, > Mikolaj > > On Thu, May 26, 2022 at 4:39 PM Hécate wrote: >> Hi everyone, >> >> Haddock needs help. >> >> Since I joined the Haddock team to help triage the tickets and interact >> with the community, we lost all of our experts, and I didn't have time >> to level up quickly enough to handle the mass of incoming tickets, let >> alone actually reduce the number of tickets to number below two hundred. >> >> As things stand now, the Haddock code base is in a disastrous state, >> largely not understood and its CI is in shambles. >> There are things that we can improve on the short and longer term – see >> https://github.com/haskell/haddock/issues/1465 – but the greater lack of >> expertise means that any project involving some core business logic is >> bound to be utterly and unnecessarily painful. The Hi Haddock GSOC >> proposal, whilst fully implemented in GHC, cannot be brought in Haddock >> at this moment in a reasonable timeline without any help. >> >> At present time, I need: >> >> * People who can refactor the code base, following modern software >> engineering practices, like domain-driven design and test-driven >> development. >> * UI developers, proficient in CSS and web accessibility. >> >> If you feel like you fit some of these criteria, please do contact me at >> this address. If your company can spare some engineering hours for you >> to give a hand, you're most welcome to do so. >> >> Just so we are clear, I am immensely grateful to the people who have >> submitted fixes and patches these past months, but this situation is >> untenable. >> >> Hécate ✨ >> 🐦: @TechnoEmpress >> IRC: Hecate >> WWW: https://glitchbra.in >> RUN: BSD >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD From ben at well-typed.com Tue Jun 7 19:27:42 2022 From: ben at well-typed.com (Ben Gamari) Date: Tue, 07 Jun 2022 15:27:42 -0400 Subject: GitLab upgrade tonight Message-ID: <87wnds46li.fsf@smart-cactus.org> Hello everyone, Tonight, around 20:00 EST, I will be performing a GitLab upgrade. As always, the downtime should be minimal (less than 30 minutes) but I will provide status updates as necessary. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Tue Jun 7 23:55:56 2022 From: ben at well-typed.com (Ben Gamari) Date: Tue, 07 Jun 2022 19:55:56 -0400 Subject: GitLab upgrade starting shortly Message-ID: <87r1403u6t.fsf@smart-cactus.org> Hello all, I will be starting the GitLab upgrade shortly. Expect approximately 30 minutes of downtime. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Wed Jun 8 00:40:32 2022 From: ben at well-typed.com (Ben Gamari) Date: Tue, 07 Jun 2022 20:40:32 -0400 Subject: GitLab upgrade starting shortly In-Reply-To: <87r1403u6t.fsf@smart-cactus.org> References: <87r1403u6t.fsf@smart-cactus.org> Message-ID: <87mteo3s46.fsf@smart-cactus.org> Ben Gamari writes: > Hello all, > > I will be starting the GitLab upgrade shortly. Expect approximately 30 > minutes of downtime. > The service is back up temporarily but currently performing background migrations, which are taking longer than expected. There will be two more periods of brief downtime as we migrate to 4.10 and then 5.0. I will write to the list when the next down period begins. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Wed Jun 8 02:54:31 2022 From: ben at well-typed.com (Ben Gamari) Date: Tue, 07 Jun 2022 22:54:31 -0400 Subject: GitLab upgrade starting shortly In-Reply-To: <87mteo3s46.fsf@smart-cactus.org> References: <87r1403u6t.fsf@smart-cactus.org> <87mteo3s46.fsf@smart-cactus.org> Message-ID: <87k09r50he.fsf@smart-cactus.org> Ben Gamari writes: > Ben Gamari writes: > >> Hello all, >> >> I will be starting the GitLab upgrade shortly. Expect approximately 30 >> minutes of downtime. >> > The service is back up temporarily but currently performing background > migrations, which are taking longer than expected. There will be two > more periods of brief downtime as we migrate to 4.10 and then 5.0. I > will write to the list when the next down period begins. > The second phase of the migration will begin shortly. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Wed Jun 8 04:45:25 2022 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 08 Jun 2022 00:45:25 -0400 Subject: GitLab upgrade starting shortly In-Reply-To: <87k09r50he.fsf@smart-cactus.org> References: <87r1403u6t.fsf@smart-cactus.org> <87mteo3s46.fsf@smart-cactus.org> <87k09r50he.fsf@smart-cactus.org> Message-ID: <72DCC717-F43C-4B76-98F5-93980EEB818E@smart-cactus.org> GitLab is again up and churning through the next round of migrations which is yet again taking quite a bit longer than expected. I will need to arrange another r short outage tomorrow to finish the final phase of the upgrade but it will remain up and usable until then. Cheers, - Ben On June 7, 2022 10:54:31 PM EDT, Ben Gamari wrote: >Ben Gamari writes: > >> Ben Gamari writes: >> >>> Hello all, >>> >>> I will be starting the GitLab upgrade shortly. Expect approximately 30 >>> minutes of downtime. >>> >> The service is back up temporarily but currently performing background >> migrations, which are taking longer than expected. There will be two >> more periods of brief downtime as we migrate to 4.10 and then 5.0. I >> will write to the list when the next down period begins. >> >The second phase of the migration will begin shortly. > > >Cheers, > > - Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Wed Jun 8 15:24:54 2022 From: ben at well-typed.com (Ben Gamari) Date: Wed, 08 Jun 2022 11:24:54 -0400 Subject: Two-factor authentication enforced on gitlab.haskell.org Message-ID: <87fskf41qo.fsf@smart-cactus.org> Hi all, Due to a recent up-tick in spam activity, we have started enforcing two-factor authentication on gitlab.haskell.org. We hope that this isn't too much of a burden, but do let us know if so and we can evaluate other options. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Wed Jun 8 23:37:27 2022 From: ben at well-typed.com (Ben Gamari) Date: Wed, 08 Jun 2022 19:37:27 -0400 Subject: Two-factor authentication enforced on gitlab.haskell.org In-Reply-To: <87fskf41qo.fsf@smart-cactus.org> References: <87fskf41qo.fsf@smart-cactus.org> Message-ID: <87zgim3ext.fsf@smart-cactus.org> Ben Gamari writes: > Hi all, > > Due to a recent up-tick in spam activity, we have started enforcing > two-factor authentication on gitlab.haskell.org. We hope that this isn't > too much of a burden, but do let us know if so and we can evaluate other > options. > Hi all, Due to user response we have reverted the two-factor authentication requirement and will instead manually review new account creations for now. Do feel free to ping either Matt, Andreas, or I via IRC or email if you attempt to create an account and your request sits unapproved for more than an hour or two. Hopefully in the near term we will be able to able to find less onerous way of dealing with the growing spam problem. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Wed Jun 8 23:39:11 2022 From: ben at well-typed.com (Ben Gamari) Date: Wed, 08 Jun 2022 19:39:11 -0400 Subject: GitLab upgrade starting shortly In-Reply-To: <87r1403u6t.fsf@smart-cactus.org> References: <87r1403u6t.fsf@smart-cactus.org> Message-ID: <87wndq3eut.fsf@smart-cactus.org> Hello all, As noted yesterday, there is one final (hopefully short!) outage needed to complete the upgrade process. This will begin in around 20 minutes, at 20:00 EST. I'll notify the list when it has finished. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Thu Jun 9 00:16:22 2022 From: ben at well-typed.com (Ben Gamari) Date: Wed, 08 Jun 2022 20:16:22 -0400 Subject: GitLab upgrade starting shortly In-Reply-To: <87wndq3eut.fsf@smart-cactus.org> References: <87r1403u6t.fsf@smart-cactus.org> <87wndq3eut.fsf@smart-cactus.org> Message-ID: <87tu8u3d4u.fsf@smart-cactus.org> Hello all, GitLab is back up and is fully upgraded. You may now continue with your usual business. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From carter.schonwald at gmail.com Thu Jun 9 01:23:05 2022 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Wed, 8 Jun 2022 21:23:05 -0400 Subject: Haddock needs help In-Reply-To: <5cbf449c-325d-2060-a534-0cda89822092@glitchbra.in> References: <5cbf449c-325d-2060-a534-0cda89822092@glitchbra.in> Message-ID: i might have some bandwidth to help out in small spurts if needed in the second half of this summer, lets catsup sometime soon on IRC or whatever and maybe i can help kick the can a smidge On Mon, Jun 6, 2022 at 7:49 AM Hécate wrote: > Hi Mikolaj, you highlight something very interesting. Indeed there is no > shame in disclosing technical debt. > > That being said the problem isn't entirely technical. The main > challenges here are the consolidation of expertise, onboarding of > newcomers, a sensible product roadmap, and a realistic path forward when > it comes to the big features I'd like to ship in the future. > > At the moment I would be more comfortable with starting by onboarding a > restricted group of people so that my attention can be properly focused > on ramping up their skills, which is why I posted on the ghc-devs > mailing-list, as GHC is our main (if not only) consumer, and GHC > developers have already made contributions to Haddock. > > Cheers, > Hécate > > Le 26/05/2022 à 17:12, Mikolaj Konarski a écrit : > > Talking as a Haskell user, but also a cabal maintainer: > > the Haddock work Hécate describes is crucial for the health > > and efficiency of the whole ecosystem and the sooner > > we can start it, the less drag it's going to have on the rest. > > > > Given that GHC expertise is not necessary for many > > of the tasks involved, could we forward this message > > to other media? I don't think there is shame in disclosing > > technical debt (as opposed to hiding it) and I think the way > > it's worded, it may be viewed as a challenge, not a turn-off. > > Still, are there any suggestions on how to tweak the text > > before an announcement on discourse, reddit, etc.? > > Would, e.g., HF, like to chime in early in each of the ensuing > > discussions? If so, how would we know where it gets posted to? > > > > Cheers, > > Mikolaj > > > > On Thu, May 26, 2022 at 4:39 PM Hécate wrote: > >> Hi everyone, > >> > >> Haddock needs help. > >> > >> Since I joined the Haddock team to help triage the tickets and interact > >> with the community, we lost all of our experts, and I didn't have time > >> to level up quickly enough to handle the mass of incoming tickets, let > >> alone actually reduce the number of tickets to number below two hundred. > >> > >> As things stand now, the Haddock code base is in a disastrous state, > >> largely not understood and its CI is in shambles. > >> There are things that we can improve on the short and longer term – see > >> https://github.com/haskell/haddock/issues/1465 – but the greater lack > of > >> expertise means that any project involving some core business logic is > >> bound to be utterly and unnecessarily painful. The Hi Haddock GSOC > >> proposal, whilst fully implemented in GHC, cannot be brought in Haddock > >> at this moment in a reasonable timeline without any help. > >> > >> At present time, I need: > >> > >> * People who can refactor the code base, following modern software > >> engineering practices, like domain-driven design and test-driven > >> development. > >> * UI developers, proficient in CSS and web accessibility. > >> > >> If you feel like you fit some of these criteria, please do contact me at > >> this address. If your company can spare some engineering hours for you > >> to give a hand, you're most welcome to do so. > >> > >> Just so we are clear, I am immensely grateful to the people who have > >> submitted fixes and patches these past months, but this situation is > >> untenable. > >> > >> Hécate ✨ > >> 🐦: @TechnoEmpress > >> IRC: Hecate > >> WWW: https://glitchbra.in > >> RUN: BSD > >> > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > -- > Hécate ✨ > 🐦: @TechnoEmpress > IRC: Hecate > WWW: https://glitchbra.in > RUN: BSD > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.peytonjones at gmail.com Thu Jun 9 14:37:27 2022 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Thu, 9 Jun 2022 15:37:27 +0100 Subject: GitLab bullets Message-ID: Ben GitLab's rendering has become broken. See this ticket https://gitlab.haskell.org/ghc/ghc/-/issues/20686 - look under "Common ground...". The bullets all have a newline after them - Ditto the numbered points under "Generalisation" - But some bullets under "Generalisation" are OK. - Bullets have a handle that seems to let you drag the bullet around. But if you do so, it messa up the format completely! Ben, can you revert this Gitlab change? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From giorgio at marinel.li Thu Jun 9 17:02:48 2022 From: giorgio at marinel.li (Giorgio Marinelli) Date: Thu, 9 Jun 2022 19:02:48 +0200 Subject: GitLab bullets In-Reply-To: References: Message-ID: It might be fixed by [1], that is part [2] of the latest release v15.0.2. Giorgio [1] https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87904 [2] https://about.gitlab.com/releases/2022/06/06/gitlab-15-0-2-released/ On Thu, 9 Jun 2022 at 16:37, Simon Peyton Jones wrote: > > Ben > > GitLab's rendering has become broken. See this ticket > https://gitlab.haskell.org/ghc/ghc/-/issues/20686 > > look under "Common ground...". The bullets all have a newline after them > Ditto the numbered points under "Generalisation" > But some bullets under "Generalisation" are OK. > Bullets have a handle that seems to let you drag the bullet around. But if you do so, it messa up the format completely! > > Ben, can you revert this Gitlab change? > > Simon > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Fri Jun 10 15:01:23 2022 From: ben at well-typed.com (Ben Gamari) Date: Fri, 10 Jun 2022 11:01:23 -0400 Subject: GitLab bullets In-Reply-To: References: Message-ID: <87leu436mq.fsf@smart-cactus.org> Simon Peyton Jones writes: > Ben > > GitLab's rendering has become broken. See this ticket > https://gitlab.haskell.org/ghc/ghc/-/issues/20686 > > - look under "Common ground...". The bullets all have a newline after > them > - Ditto the numbered points under "Generalisation" > - But some bullets under "Generalisation" are OK. > - Bullets have a handle that seems to let you drag the bullet around. > But if you do so, it messa up the format completely! > > Ben, can you revert this Gitlab change? > Indeed I also noticed this. I sneakily updated to 15.0.2 this morning, which indeed seems to have fixed the regression. Do let me know if anything else looks amiss. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simon.peytonjones at gmail.com Fri Jun 10 15:05:54 2022 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Fri, 10 Jun 2022 16:05:54 +0100 Subject: GitLab bullets In-Reply-To: <87leu436mq.fsf@smart-cactus.org> References: <87leu436mq.fsf@smart-cactus.org> Message-ID: Thank you. Looks nice again! S On Fri, 10 Jun 2022 at 16:01, Ben Gamari wrote: > Simon Peyton Jones writes: > > > Ben > > > > GitLab's rendering has become broken. See this ticket > > https://gitlab.haskell.org/ghc/ghc/-/issues/20686 > > > > - look under "Common ground...". The bullets all have a newline > after > > them > > - Ditto the numbered points under "Generalisation" > > - But some bullets under "Generalisation" are OK. > > - Bullets have a handle that seems to let you drag the bullet around. > > But if you do so, it messa up the format completely! > > > > Ben, can you revert this Gitlab change? > > > Indeed I also noticed this. I sneakily updated to 15.0.2 this morning, > which indeed seems to have fixed the regression. > > Do let me know if anything else looks amiss. > > Cheers, > > - Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.peytonjones at gmail.com Fri Jun 10 16:20:29 2022 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Fri, 10 Jun 2022 17:20:29 +0100 Subject: GHC code size Message-ID: Dear GHC devs Is it possible to get a "lines-of-code" summary of GHC these days? Like the one below, from 2011. It needs more than `wc` because it's helpful to split lines of code from lines of comments and notes. We used to have `count_lines` but I'm not sure whether it is still extant. I'm giving a talk at Zurihac on Sunday morning, about the internals of GHC. Any data before then, preferably in a form comparable to that below, would be terrific. But you have a lot else to do. This isn't do-or-die, just nice to have. Thanks Simon [image: image.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From chessai1996 at gmail.com Fri Jun 10 16:29:59 2022 From: chessai1996 at gmail.com (chessai) Date: Fri, 10 Jun 2022 11:29:59 -0500 Subject: GHC code size In-Reply-To: References: Message-ID: You might be able to do something with cloc and a shell script for a rough estimate. ``` $ cd ghc $ nix-shell -p clock --run "cloc ." ``` will output a detailed report of the loc and language breakdown of the top level ghc directory (it is comment-aware and aware of many languages). there might be a way to get cloc or a similar tool to output something more inspect able (eg json), and then use a shell script to gather everything from the appropriate directories/files. I suspect something could be hacked up in less than a day, but it would require a bit of research. Hopefully this is helpful and gets you going - I'd be happy to hear of better solutions. Thanks On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones wrote: > Dear GHC devs > > Is it possible to get a "lines-of-code" summary of GHC these days? Like > the one below, from 2011. > > It needs more than `wc` because it's helpful to split lines of code from > lines of comments and notes. > > We used to have `count_lines` but I'm not sure whether it is still extant. > > I'm giving a talk at Zurihac on Sunday morning, about the internals of > GHC. Any data before then, preferably in a form comparable to that below, > would be terrific. > > But you have a lot else to do. This isn't do-or-die, just nice to have. > > Thanks > > Simon > > [image: image.png] > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From hecate at glitchbra.in Fri Jun 10 16:43:42 2022 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Fri, 10 Jun 2022 18:43:42 +0200 Subject: GHC code size In-Reply-To: References: Message-ID: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> If you don't have a nix shell handy, here is what I'm getting: ❯ cloc compiler rts driver     1148 text files.     1137 unique files.      108 files ignored. github.com/AlDanial/cloc v 1.88  T=1.31 s (794.3 files/s, 431269.4 lines/s) --------------------------------------------------------------------------------------- Language                             files          blank comment           code --------------------------------------------------------------------------------------- Haskell                                635 68541         140216         231567 C                                      158 10529          16953          51162 C/C++ Header                           209 4329           8984          14536 yacc                                     2 971             10           5024 Logos                                    3 530              0           3642 Pascal                                   1 661            936           2312 make                                    14 252            409            850 Windows Module Definition                7 27              0            489 Assembly                                 5 76            269            478 Puppet                                   1 106              0            445 Python                                   1 32             19            162 D                                        1 16             42             60 YAML                                     1 6             10             18 Lisp                                     1 2              4              7 Windows Resource File                    1 0              0              1 --------------------------------------------------------------------------------------- SUM:                                  1040 86078         167852         310753 --------------------------------------------------------------------------------------- Le 10/06/2022 à 18:29, chessai a écrit : > You might be able to do something with cloc and a shell script for a > rough estimate. > > ``` > $ cd ghc > $ nix-shell -p clock --run "cloc ." > ``` > > will output a detailed report of the loc and language breakdown of the > top level ghc directory (it is comment-aware and aware of many > languages). there might be a way to get cloc or a similar tool to > output something more inspect able (eg json), and then use a shell > script to gather everything from the appropriate directories/files. > > I suspect something could be hacked up in less than a day, but it > would require a bit of research. Hopefully this is helpful and gets > you going - I'd be happy to hear of better solutions. > > Thanks > > On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones > wrote: > > Dear GHC devs > > Is it possible to get a "lines-of-code" summary of GHC these > days?   Like the one below, from 2011. > > It needs more than `wc` because it's helpful to split lines of > code from lines of comments and notes. > > We used to have `count_lines` but I'm not sure whether it is still > extant. > > I'm giving a talk at Zurihac on Sunday morning, about the > internals of GHC. Any data before then, preferably in a form > comparable to that below, would be terrific. > > But you have a lot else to do.  This isn't do-or-die, just nice to > have. > > Thanks > > Simon > > image.png > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW:https://glitchbra.in RUN: BSD -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From simon.peytonjones at gmail.com Mon Jun 13 10:29:03 2022 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Mon, 13 Jun 2022 11:29:03 +0100 Subject: Type vs Constraint Message-ID: Dear GHC devs and Core Libraries folk The tension between Type and Constraint (are they equal or not?) has plaugued GHC for years. This GHC proposal #518 suggests how to fix it. Please do offer your thoughts. Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Tue Jun 14 15:04:21 2022 From: bryan at haskell.foundation (Bryan Richter) Date: Tue, 14 Jun 2022 17:04:21 +0200 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> Message-ID: A quick googling discovered https://githubnext.com/projects/repo-visualization, which has some of the desired features. :) (CC'ing the author and team mentioned in the article, too.) Has: * very visual * subdirectory breakdown * filetype breakdown Doesn't have: * Separating code from comments * History is listed under "future work" * Drop-in support for gitlab (it's presented as a GitHub Action) The article has an interactive widget you can point at a repo. I pointed it at ghc/ghc, and although my browser is still churning ten minutes later, here's a preview. It's pretty cool! On 14/06/2022 16:20, Hécate wrote: > > I'm taking the liberty of forwarding this to Bryan, as he's in a > unique position to help on this front. :) > > Le 14/06/2022 à 16:18, Simon Peyton Jones a écrit : >> Thanks Hecate. I used your figures in my talk.  Really helpful. >> >>  A note to all ghc-devs: it's be lovely to have a regularly-updated >> summary visualisation of GHC's source code: >> >> * Separating code from comments >> * Broken up by sub-directory >> * As visual as possible >> * Ideally with some kind of historical time-line ability >> >> This can't be new.  Zillions of GitHub repositories could be >> visualised like this.  There must be prior art; probably a lot of >> it.  Can we just press a button and get it? >> >> Simon >> >> On Fri, 10 Jun 2022 at 17:45, Hécate wrote: >> >> If you don't have a nix shell handy, here is what I'm getting: >> >> ❯ cloc compiler rts driver >>     1148 text files. >>     1137 unique files. >>      108 files ignored. >> >> github.com/AlDanial/cloc v >> 1.88  T=1.31 s (794.3 files/s, 431269.4 lines/s) >> --------------------------------------------------------------------------------------- >> Language                             files blank        >> comment           code >> --------------------------------------------------------------------------------------- >> Haskell                                635 68541         >> 140216         231567 >> C                                      158 10529          >> 16953          51162 >> C/C++ Header                           209 4329           >> 8984          14536 >> yacc                                     2 971             >> 10           5024 >> Logos                                    3 530              >> 0           3642 >> Pascal                                   1 661            >> 936           2312 >> make                                    14 252            >> 409            850 >> Windows Module Definition                7 27              >> 0            489 >> Assembly                                 5 76            >> 269            478 >> Puppet                                   1 106              >> 0            445 >> Python                                   1 32             >> 19            162 >> D                                        1 16             >> 42             60 >> YAML 1              6             10             18 >> Lisp 1              2              4              7 >> Windows Resource File 1              0              0              1 >> --------------------------------------------------------------------------------------- >> SUM:                                  1040 86078         >> 167852         310753 >> --------------------------------------------------------------------------------------- >> >> Le 10/06/2022 à 18:29, chessai a écrit : >>> You might be able to do something with cloc and a shell script >>> for a rough estimate. >>> >>> ``` >>> $ cd ghc >>> $ nix-shell -p clock --run "cloc ." >>> ``` >>> >>> will output a detailed report of the loc and language breakdown >>> of the top level ghc directory (it is comment-aware and aware of >>> many languages). there might be a way to get cloc or a similar >>> tool to output something more inspect able (eg json), and then >>> use a shell script to gather everything from the appropriate >>> directories/files. >>> >>> I suspect something could be hacked up in less than a day, but >>> it would require a bit of research. Hopefully this is helpful >>> and gets you going - I'd be happy to hear of better solutions. >>> >>> Thanks >>> >>> On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones >>> wrote: >>> >>> Dear GHC devs >>> >>> Is it possible to get a "lines-of-code" summary of GHC these >>> days?   Like the one below, from 2011. >>> >>> It needs more than `wc` because it's helpful to split lines >>> of code from lines of comments and notes. >>> >>> We used to have `count_lines` but I'm not sure whether it is >>> still extant. >>> >>> I'm giving a talk at Zurihac on Sunday morning, about the >>> internals of GHC.  Any data before then, preferably in a >>> form comparable to that below, would be terrific. >>> >>> But you have a lot else to do.  This isn't do-or-die, just >>> nice to have. >>> >>> Thanks >>> >>> Simon >>> >>> image.png >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> -- >> Hécate ✨ >> 🐦: @TechnoEmpress >> IRC: Hecate >> WWW:https://glitchbra.in >> RUN: BSD >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -- > Hécate ✨ > 🐦: @TechnoEmpress > IRC: Hecate > WWW:https://glitchbra.in > RUN: BSD -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: s0NTyQh7STmE0t2T.png Type: image/png Size: 439175 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From sam.derbyshire at gmail.com Tue Jun 14 15:33:11 2022 From: sam.derbyshire at gmail.com (Sam Derbyshire) Date: Tue, 14 Jun 2022 17:33:11 +0200 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> Message-ID: That's really cool, thanks Bryan. On Tue, 14 Jun 2022 at 17:04, Bryan Richter via ghc-devs < ghc-devs at haskell.org> wrote: > A quick googling discovered > https://githubnext.com/projects/repo-visualization, which has some of the > desired features. :) > > (CC'ing the author and team mentioned in the article, too.) > > Has: > > * very visual > * subdirectory breakdown > * filetype breakdown > > Doesn't have: > > * Separating code from comments > * History is listed under "future work" > * Drop-in support for gitlab (it's presented as a GitHub Action) > > The article has an interactive widget you can point at a repo. I pointed > it at ghc/ghc, and although my browser is still churning ten minutes later, > here's a preview. It's pretty cool! > > > On 14/06/2022 16:20, Hécate wrote: > > I'm taking the liberty of forwarding this to Bryan, as he's in a unique > position to help on this front. :) > Le 14/06/2022 à 16:18, Simon Peyton Jones a écrit : > > Thanks Hecate. I used your figures in my talk. Really helpful. > > A note to all ghc-devs: it's be lovely to have a regularly-updated > summary visualisation of GHC's source code: > > - Separating code from comments > - Broken up by sub-directory > - As visual as possible > - Ideally with some kind of historical time-line ability > > This can't be new. Zillions of GitHub repositories could be visualised > like this. There must be prior art; probably a lot of it. Can we just > press a button and get it? > > Simon > > On Fri, 10 Jun 2022 at 17:45, Hécate wrote: > >> If you don't have a nix shell handy, here is what I'm getting: >> >> ❯ cloc compiler rts driver >> 1148 text files. >> 1137 unique files. >> 108 files ignored. >> >> github.com/AlDanial/cloc v 1.88 T=1.31 s (794.3 files/s, 431269.4 >> lines/s) >> >> --------------------------------------------------------------------------------------- >> Language files blank >> comment code >> >> --------------------------------------------------------------------------------------- >> Haskell 635 68541 >> 140216 231567 >> C 158 10529 >> 16953 51162 >> C/C++ Header 209 4329 >> 8984 14536 >> yacc 2 971 >> 10 5024 >> Logos 3 530 >> 0 3642 >> Pascal 1 661 >> 936 2312 >> make 14 252 >> 409 850 >> Windows Module Definition 7 27 >> 0 489 >> Assembly 5 76 >> 269 478 >> Puppet 1 106 >> 0 445 >> Python 1 32 >> 19 162 >> D 1 16 >> 42 60 >> YAML 1 6 >> 10 18 >> Lisp 1 2 >> 4 7 >> Windows Resource File 1 0 >> 0 1 >> >> --------------------------------------------------------------------------------------- >> SUM: 1040 86078 >> 167852 310753 >> >> --------------------------------------------------------------------------------------- >> Le 10/06/2022 à 18:29, chessai a écrit : >> >> You might be able to do something with cloc and a shell script for a >> rough estimate. >> >> ``` >> $ cd ghc >> $ nix-shell -p clock --run "cloc ." >> ``` >> >> will output a detailed report of the loc and language breakdown of the >> top level ghc directory (it is comment-aware and aware of many languages). >> there might be a way to get cloc or a similar tool to output something more >> inspect able (eg json), and then use a shell script to gather everything >> from the appropriate directories/files. >> >> I suspect something could be hacked up in less than a day, but it would >> require a bit of research. Hopefully this is helpful and gets you going - >> I'd be happy to hear of better solutions. >> >> Thanks >> >> On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones < >> simon.peytonjones at gmail.com> wrote: >> >>> Dear GHC devs >>> >>> Is it possible to get a "lines-of-code" summary of GHC these days? >>> Like the one below, from 2011. >>> >>> It needs more than `wc` because it's helpful to split lines of code from >>> lines of comments and notes. >>> >>> We used to have `count_lines` but I'm not sure whether it is still >>> extant. >>> >>> I'm giving a talk at Zurihac on Sunday morning, about the internals of >>> GHC. Any data before then, preferably in a form comparable to that below, >>> would be terrific. >>> >>> But you have a lot else to do. This isn't do-or-die, just nice to have. >>> >>> Thanks >>> >>> Simon >>> >>> [image: image.png] >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> >> _______________________________________________ >> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> -- >> Hécate ✨ >> 🐦: @TechnoEmpress >> IRC: Hecate >> WWW: https://glitchbra.in >> RUN: BSD >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -- > Hécate ✨ > 🐦: @TechnoEmpress > IRC: Hecate > WWW: https://glitchbra.in > RUN: BSD > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: s0NTyQh7STmE0t2T.png Type: image/png Size: 439175 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From simon.peytonjones at gmail.com Tue Jun 14 16:14:55 2022 From: simon.peytonjones at gmail.com (Simon Peyton Jones) Date: Tue, 14 Jun 2022 17:14:55 +0100 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> Message-ID: Indeed cool. - Can it do numeric breakdowns too? Like I needed for my talk? - Can it distinguish code from comments? Simon On Tue, 14 Jun 2022 at 16:04, Bryan Richter wrote: > A quick googling discovered > https://githubnext.com/projects/repo-visualization, which has some of the > desired features. :) > > (CC'ing the author and team mentioned in the article, too.) > > Has: > > * very visual > * subdirectory breakdown > * filetype breakdown > > Doesn't have: > > * Separating code from comments > * History is listed under "future work" > * Drop-in support for gitlab (it's presented as a GitHub Action) > > The article has an interactive widget you can point at a repo. I pointed > it at ghc/ghc, and although my browser is still churning ten minutes later, > here's a preview. It's pretty cool! > > > On 14/06/2022 16:20, Hécate wrote: > > I'm taking the liberty of forwarding this to Bryan, as he's in a unique > position to help on this front. :) > Le 14/06/2022 à 16:18, Simon Peyton Jones a écrit : > > Thanks Hecate. I used your figures in my talk. Really helpful. > > A note to all ghc-devs: it's be lovely to have a regularly-updated > summary visualisation of GHC's source code: > > - Separating code from comments > - Broken up by sub-directory > - As visual as possible > - Ideally with some kind of historical time-line ability > > This can't be new. Zillions of GitHub repositories could be visualised > like this. There must be prior art; probably a lot of it. Can we just > press a button and get it? > > Simon > > On Fri, 10 Jun 2022 at 17:45, Hécate wrote: > >> If you don't have a nix shell handy, here is what I'm getting: >> >> ❯ cloc compiler rts driver >> 1148 text files. >> 1137 unique files. >> 108 files ignored. >> >> github.com/AlDanial/cloc v 1.88 T=1.31 s (794.3 files/s, 431269.4 >> lines/s) >> >> --------------------------------------------------------------------------------------- >> Language files blank >> comment code >> >> --------------------------------------------------------------------------------------- >> Haskell 635 68541 >> 140216 231567 >> C 158 10529 >> 16953 51162 >> C/C++ Header 209 4329 >> 8984 14536 >> yacc 2 971 >> 10 5024 >> Logos 3 530 >> 0 3642 >> Pascal 1 661 >> 936 2312 >> make 14 252 >> 409 850 >> Windows Module Definition 7 27 >> 0 489 >> Assembly 5 76 >> 269 478 >> Puppet 1 106 >> 0 445 >> Python 1 32 >> 19 162 >> D 1 16 >> 42 60 >> YAML 1 6 >> 10 18 >> Lisp 1 2 >> 4 7 >> Windows Resource File 1 0 >> 0 1 >> >> --------------------------------------------------------------------------------------- >> SUM: 1040 86078 >> 167852 310753 >> >> --------------------------------------------------------------------------------------- >> Le 10/06/2022 à 18:29, chessai a écrit : >> >> You might be able to do something with cloc and a shell script for a >> rough estimate. >> >> ``` >> $ cd ghc >> $ nix-shell -p clock --run "cloc ." >> ``` >> >> will output a detailed report of the loc and language breakdown of the >> top level ghc directory (it is comment-aware and aware of many languages). >> there might be a way to get cloc or a similar tool to output something more >> inspect able (eg json), and then use a shell script to gather everything >> from the appropriate directories/files. >> >> I suspect something could be hacked up in less than a day, but it would >> require a bit of research. Hopefully this is helpful and gets you going - >> I'd be happy to hear of better solutions. >> >> Thanks >> >> On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones < >> simon.peytonjones at gmail.com> wrote: >> >>> Dear GHC devs >>> >>> Is it possible to get a "lines-of-code" summary of GHC these days? >>> Like the one below, from 2011. >>> >>> It needs more than `wc` because it's helpful to split lines of code from >>> lines of comments and notes. >>> >>> We used to have `count_lines` but I'm not sure whether it is still >>> extant. >>> >>> I'm giving a talk at Zurihac on Sunday morning, about the internals of >>> GHC. Any data before then, preferably in a form comparable to that below, >>> would be terrific. >>> >>> But you have a lot else to do. This isn't do-or-die, just nice to have. >>> >>> Thanks >>> >>> Simon >>> >>> [image: image.png] >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> >> _______________________________________________ >> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> -- >> Hécate ✨ >> 🐦: @TechnoEmpress >> IRC: Hecate >> WWW: https://glitchbra.in >> RUN: BSD >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -- > Hécate ✨ > 🐦: @TechnoEmpress > IRC: Hecate > WWW: https://glitchbra.in > RUN: BSD > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: s0NTyQh7STmE0t2T.png Type: image/png Size: 439175 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From freizl at gmail.com Tue Jun 14 16:43:31 2022 From: freizl at gmail.com (Haisheng Wu) Date: Tue, 14 Jun 2022 09:43:31 -0700 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> Message-ID: I tried to run `cloc` in the `compiler` folder and got this. Not sure how to further break down the Compiler into smaller modules. ``` ➜ ghc git:(master) ✗ cloc compiler 723 text files. 655 unique files. 68 files ignored. github.com/AlDanial/cloc v 1.92 T=0.64 s (1025.4 files/s, 716024.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Haskell 635 68541 147036 224747 yacc 2 971 10 5024 Logos 3 530 0 3642 Pascal 1 661 936 2312 C/C++ Header 7 187 304 1512 Puppet 1 106 0 445 make 2 47 84 187 C 3 11 16 37 YAML 1 6 10 18 ------------------------------------------------------------------------------- SUM: 655 71060 148396 237924 ------------------------------------------------------------------------------- ``` -Haisheng On Tue, Jun 14, 2022 at 9:14 AM Simon Peyton Jones < simon.peytonjones at gmail.com> wrote: > Indeed cool. > > - Can it do numeric breakdowns too? Like I needed for my talk? > - Can it distinguish code from comments? > > Simon > > On Tue, 14 Jun 2022 at 16:04, Bryan Richter > wrote: > >> A quick googling discovered >> https://githubnext.com/projects/repo-visualization, which has some of >> the desired features. :) >> >> (CC'ing the author and team mentioned in the article, too.) >> >> Has: >> >> * very visual >> * subdirectory breakdown >> * filetype breakdown >> >> Doesn't have: >> >> * Separating code from comments >> * History is listed under "future work" >> * Drop-in support for gitlab (it's presented as a GitHub Action) >> >> The article has an interactive widget you can point at a repo. I pointed >> it at ghc/ghc, and although my browser is still churning ten minutes later, >> here's a preview. It's pretty cool! >> >> >> On 14/06/2022 16:20, Hécate wrote: >> >> I'm taking the liberty of forwarding this to Bryan, as he's in a unique >> position to help on this front. :) >> Le 14/06/2022 à 16:18, Simon Peyton Jones a écrit : >> >> Thanks Hecate. I used your figures in my talk. Really helpful. >> >> A note to all ghc-devs: it's be lovely to have a regularly-updated >> summary visualisation of GHC's source code: >> >> - Separating code from comments >> - Broken up by sub-directory >> - As visual as possible >> - Ideally with some kind of historical time-line ability >> >> This can't be new. Zillions of GitHub repositories could be visualised >> like this. There must be prior art; probably a lot of it. Can we just >> press a button and get it? >> >> Simon >> >> On Fri, 10 Jun 2022 at 17:45, Hécate wrote: >> >>> If you don't have a nix shell handy, here is what I'm getting: >>> >>> ❯ cloc compiler rts driver >>> 1148 text files. >>> 1137 unique files. >>> 108 files ignored. >>> >>> github.com/AlDanial/cloc v 1.88 T=1.31 s (794.3 files/s, 431269.4 >>> lines/s) >>> >>> --------------------------------------------------------------------------------------- >>> Language files blank >>> comment code >>> >>> --------------------------------------------------------------------------------------- >>> Haskell 635 68541 >>> 140216 231567 >>> C 158 10529 >>> 16953 51162 >>> C/C++ Header 209 4329 >>> 8984 14536 >>> yacc 2 971 >>> 10 5024 >>> Logos 3 530 >>> 0 3642 >>> Pascal 1 661 >>> 936 2312 >>> make 14 252 >>> 409 850 >>> Windows Module Definition 7 27 >>> 0 489 >>> Assembly 5 76 >>> 269 478 >>> Puppet 1 106 >>> 0 445 >>> Python 1 32 >>> 19 162 >>> D 1 16 >>> 42 60 >>> YAML 1 6 >>> 10 18 >>> Lisp 1 2 >>> 4 7 >>> Windows Resource File 1 0 >>> 0 1 >>> >>> --------------------------------------------------------------------------------------- >>> SUM: 1040 86078 >>> 167852 310753 >>> >>> --------------------------------------------------------------------------------------- >>> Le 10/06/2022 à 18:29, chessai a écrit : >>> >>> You might be able to do something with cloc and a shell script for a >>> rough estimate. >>> >>> ``` >>> $ cd ghc >>> $ nix-shell -p clock --run "cloc ." >>> ``` >>> >>> will output a detailed report of the loc and language breakdown of the >>> top level ghc directory (it is comment-aware and aware of many languages). >>> there might be a way to get cloc or a similar tool to output something more >>> inspect able (eg json), and then use a shell script to gather everything >>> from the appropriate directories/files. >>> >>> I suspect something could be hacked up in less than a day, but it would >>> require a bit of research. Hopefully this is helpful and gets you going - >>> I'd be happy to hear of better solutions. >>> >>> Thanks >>> >>> On Fri, Jun 10, 2022, 11:20 Simon Peyton Jones < >>> simon.peytonjones at gmail.com> wrote: >>> >>>> Dear GHC devs >>>> >>>> Is it possible to get a "lines-of-code" summary of GHC these days? >>>> Like the one below, from 2011. >>>> >>>> It needs more than `wc` because it's helpful to split lines of code >>>> from lines of comments and notes. >>>> >>>> We used to have `count_lines` but I'm not sure whether it is still >>>> extant. >>>> >>>> I'm giving a talk at Zurihac on Sunday morning, about the internals of >>>> GHC. Any data before then, preferably in a form comparable to that below, >>>> would be terrific. >>>> >>>> But you have a lot else to do. This isn't do-or-die, just nice to have. >>>> >>>> Thanks >>>> >>>> Simon >>>> >>>> [image: image.png] >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >>> >>> _______________________________________________ >>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >>> -- >>> Hécate ✨ >>> 🐦: @TechnoEmpress >>> IRC: Hecate >>> WWW: https://glitchbra.in >>> RUN: BSD >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -- >> Hécate ✨ >> 🐦: @TechnoEmpress >> IRC: Hecate >> WWW: https://glitchbra.in >> RUN: BSD >> >> _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: s0NTyQh7STmE0t2T.png Type: image/png Size: 439175 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 336143 bytes Desc: not available URL: From ben at smart-cactus.org Tue Jun 14 18:14:43 2022 From: ben at smart-cactus.org (Ben Gamari) Date: Tue, 14 Jun 2022 14:14:43 -0400 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> Message-ID: <87wndjun7p.fsf@smart-cactus.org> Bryan Richter via ghc-devs writes: > A quick googling discovered > https://githubnext.com/projects/repo-visualization, which has some of > the desired features. :) > Another somewhat-related visualiation tool that can produce some pretty pictures is gource [1]. I wouldn't call the output "useful" per se, but it is mildly amusing to see the avatars frenetically flying about the source tree. It gives you a sense of just how many people are responsible for building the GHC that we know and love. Here [2] is a rendering of ghc's history. Best to skip the first minute or so, which is largely just Will Partain setting things up. Things really start to pick up around 2012 (around the 6 minute mark); it's truly dizzying. Happily, this momentum has persisted to this day. Cheers, - Ben [1] https://gource.io/ [2] http://home.smart-cactus.org/~ben/ghc/gource-2022-06-14.mkv -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From steven at steshaw.org Wed Jun 15 00:05:29 2022 From: steven at steshaw.org (Steven Shaw) Date: Wed, 15 Jun 2022 10:05:29 +1000 Subject: GHC code size In-Reply-To: <87wndjun7p.fsf@smart-cactus.org> References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> <87wndjun7p.fsf@smart-cactus.org> Message-ID: Hi Simon, If you prefer to use the old `count_lines`, you can retrieve it like this: git show 0cd989577a8b8d2666741fcac4fd3032ae212b80^:utils/count_lines/ count_lines.pl >/tmp/count_lines.pl It appears to need a list of files. The output is quite busy (not quite like your nice summary above). For example, the output of perl /tmp/count_lines.pl $(find compiler -name '*.hs') on `master` is https://gist.github.com/steshaw/b636fb76c805bfa0fff7484f5da11ed6 Perhaps if you recall how you used `count_lines` in the past, you can make an accurate comparison! Cheers, Steve On Wed, 15 Jun 2022 at 04:15, Ben Gamari wrote: > Bryan Richter via ghc-devs writes: > > > A quick googling discovered > > https://githubnext.com/projects/repo-visualization, which has some of > > the desired features. :) > > > Another somewhat-related visualiation tool that can produce some pretty > pictures is gource [1]. I wouldn't call the output "useful" per se, but > it is mildly amusing to see the avatars frenetically flying about the > source tree. It gives you a sense of just how many people are > responsible for building the GHC that we know and love. > > Here [2] is a rendering of ghc's history. Best to skip the first minute > or so, which is largely just Will Partain setting things up. Things > really start to pick up around 2012 (around the 6 minute mark); it's > truly dizzying. Happily, this momentum has persisted to this day. > > Cheers, > > - Ben > > [1] https://gource.io/ > [2] http://home.smart-cactus.org/~ben/ghc/gource-2022-06-14.mkv > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Wed Jun 15 01:01:58 2022 From: ben at smart-cactus.org (Ben Gamari) Date: Tue, 14 Jun 2022 21:01:58 -0400 Subject: GHC code size In-Reply-To: References: <6a802d30-64fa-2161-2f23-f285863c2c32@glitchbra.in> <87wndjun7p.fsf@smart-cactus.org> Message-ID: <87tu8mvixa.fsf@smart-cactus.org> Rodrigo Mesquita writes: > The visualisation is very cool. I’m left wondering: > > - What triggered this “explosion” in 2012? That is a good question. I'll admit, I'm not entirely sure as I myself only really started contributing to GHC in 2012. > - Do we have statistics on the number of contributors over time? > Yes, we usually report these during the "State of the compiler" talk that is given during the Haskell Implementor's Workshop. I use this script [1] to generate these. It's current output is: "1 year ago" 768 71 "2 year ago" 695 73 "3 year ago" 611 84 "4 year ago" 532 79 "5 year ago" 467 65 "6 year ago" 403 64 "7 year ago" 340 63 "8 year ago" 272 68 "9 year ago" 223 49 "10 year ago" 199 24 Where: * the first column is the beginning of the measured period * the second is the total number of unique contributors in GHC at the end of that period * the third is the number of *new* contributors during that period Cheers, - Ben [1] https://gitlab.haskell.org/bgamari/ghc-utils/-/blob/master/analytics/new-contribs.hs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From Gergo.Erdi at sc.com Thu Jun 16 03:59:09 2022 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Thu, 16 Jun 2022 03:59:09 +0000 Subject: Migration guide for multiple home units Message-ID: PUBLIC Hi, Is there a migration guide for GHC API clients for the new "multiple home units" feature? In particular, I have the following two functions in my code that used to interact with related features of GHC: addUnit :: UnitInfo -> HscEnv -> HscEnv addUnit unitInfo at GenericUnitInfo{..} = modifyUnitState $ \us -> us { packageNameMap = addToUFM (packageNameMap us) unitPackageName unitId , unitInfoMap = M.insert unitId unitInfo $ unitInfoMap us } registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () registerModule details iface = modifySession $ extendHpt . addModule where hmi = HomeModInfo iface details Nothing mod = mi_module iface modOrig = ModOrigin (Just True) [] [] True addModule = modifyUnitState $ \us -> us { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us } extendHpt env = env { hsc_unit_env = let ue = hsc_unit_env env in ue { ue_hpt = addHomeModInfoToHpt hmi (hsc_HPT env) } } I implemented these using the following utility function: modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env { hsc_unit_env = let ue = hsc_unit_env env in ue { ue_units = f (ue_units ue) } } With the recent changes to GHC, `modifyUnitState` doesn't work anymore because there is no single `UnitEnv` in the `HscEnv`. I tried updating the `HomeUnitEnv` of the current unit: modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env { hsc_unit_env = let ue = hsc_unit_env env in ue_updateHomeUnitEnv f' (ue_currentUnit ue) ue } where f' hue = let units = homeUnitEnv_units hue in hue { homeUnitEnv_units = f units } but this leads to a panic: GHC version 9.5.20220613: Unit unknown to the internal unit environment unit (cortex-prim) pprInternalUnitMap main (flags: main, Just main) -> Call stack: CallStack (from HasCallStack): callStackDoc, called at compiler/GHC/Utils/Panic.hs:182:37 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Utils.Panic pprPanic, called at compiler/GHC/Unit/Env.hs:450:14 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Unit.Env ue_findHomeUnitEnv, called at compiler/GHC/Unit/Env.hs:394:48 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Unit.Env ue_unitFlags, called at compiler/GHC/Driver/Env.hs:421:18 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Env hscSetActiveUnitId, called at compiler/GHC/Driver/Env.hs:416:34 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Env hscSetActiveHomeUnit, called at compiler/GHC/Driver/Make.hs:1853:15 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Make What is the new way of registering a new unit or new module with GHC? Thanks, Gergo This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Thu Jun 16 14:54:45 2022 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Thu, 16 Jun 2022 15:54:45 +0100 Subject: Migration guide for multiple home units In-Reply-To: References: Message-ID: Hi Gergo The error tells you what you only have setup the HscEnv so that there's one unit called main, but you are attempting to compile something with unit id cortex-prim. Perhaps you could be inspired by `setSessionDynFlags` which deals with renaming the main unit id if you are compiling a unit which have a different unit id. (See setUnitDynFlagsNoCheck). Matt On Thu, Jun 16, 2022 at 5:01 AM Erdi, Gergo via ghc-devs wrote: > > PUBLIC > > > Hi, > > > > Is there a migration guide for GHC API clients for the new “multiple home units” feature? > > > > In particular, I have the following two functions in my code that used to interact with related features of GHC: > > > > addUnit :: UnitInfo -> HscEnv -> HscEnv > > addUnit unitInfo at GenericUnitInfo{..} = modifyUnitState $ \us -> us > > { packageNameMap = addToUFM (packageNameMap us) unitPackageName unitId > > , unitInfoMap = M.insert unitId unitInfo $ unitInfoMap us > > } > > > > registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () > > registerModule details iface = modifySession $ extendHpt . addModule > > where > > hmi = HomeModInfo iface details Nothing > > > > mod = mi_module iface > > modOrig = ModOrigin (Just True) [] [] True > > > > addModule = modifyUnitState $ \us -> us > > { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us > > } > > > > extendHpt env = env > > { hsc_unit_env = let ue = hsc_unit_env env in ue > > { ue_hpt = addHomeModInfoToHpt hmi (hsc_HPT env) > > } > > } > > > > I implemented these using the following utility function: > > > > modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv > > modifyUnitState f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in ue > > { ue_units = f (ue_units ue) > > } > > } > > > > With the recent changes to GHC, `modifyUnitState` doesn’t work anymore because there is no single `UnitEnv` in the `HscEnv`. I tried updating the `HomeUnitEnv` of the current unit: > > > > modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv > > modifyUnitState f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in > > ue_updateHomeUnitEnv f' (ue_currentUnit ue) ue > > } > > where > > f' hue = let units = homeUnitEnv_units hue in hue > > { homeUnitEnv_units = f units > > } > > > > but this leads to a panic: > > > > GHC version 9.5.20220613: > > Unit unknown to the internal unit environment > > > > unit (cortex-prim) > > pprInternalUnitMap > > main (flags: main, Just main) -> > > Call stack: > > CallStack (from HasCallStack): > > callStackDoc, called at compiler/GHC/Utils/Panic.hs:182:37 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Utils.Panic > > pprPanic, called at compiler/GHC/Unit/Env.hs:450:14 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Unit.Env > > ue_findHomeUnitEnv, called at compiler/GHC/Unit/Env.hs:394:48 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Unit.Env > > ue_unitFlags, called at compiler/GHC/Driver/Env.hs:421:18 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Env > > hscSetActiveUnitId, called at compiler/GHC/Driver/Env.hs:416:34 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Env > > hscSetActiveHomeUnit, called at compiler/GHC/Driver/Make.hs:1853:15 in ghc-lib-0.20220613-HQTH0nHhxeOCnJy5jYkZiX:GHC.Driver.Make > > > > > > What is the new way of registering a new unit or new module with GHC? > > > > Thanks, > > Gergo > > > This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations > > Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm > > Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. > > Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. > > Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. > > Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From gergo at erdi.hu Sun Jun 19 04:31:48 2022 From: gergo at erdi.hu (=?ISO-8859-2?Q?=C9RDI_Gerg=F5?=) Date: Sun, 19 Jun 2022 12:31:48 +0800 (+08) Subject: Migration guide for multiple home units In-Reply-To: References: Message-ID: On Thu, 16 Jun 2022, Erdi, Gergo via ghc-devs wrote: > Is there a migration guide for GHC API clients for the new “multiple home units” > feature? OK so in concrete terms, please see my attached program which is a heavily cut-down, standalone version of my real program. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400^ (i.e. 3219610e3ba6cb6a5cd1f4e32e2b4befea5bd384) it compiles and works as expected. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400 onwards, two problems pop up: 1. `extendMG` has changed and now requires manually specifying outgoing dependency edges. I thought the whole point of `summariseFile` was to collect this information? The reason I need to `extendMG` at that point is to get intra-unit orphan instances working. 2. `modifyUnitState` and its two uses (`addUnit` and `registerModule`) need to be updated to the new API. I think it makes sense that these need changing, since they touch exactly on the issue of which units are being compiled right now. However, I don't know how to update these. Also, I guess `setHomeUnit` should change the `CurrentUnit` instead of the `HomeUnit` now? Thanks, Gergo -------------- next part -------------- A non-text attachment was scrubbed... Name: GHCAPI.hs Type: text/x-haskell Size: 9209 bytes Desc: URL: From Gergo.Erdi at sc.com Mon Jun 20 05:15:45 2022 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Mon, 20 Jun 2022 05:15:45 +0000 Subject: Migration guide for multiple home units Message-ID: PUBLIC I managed to get this working, but I would like some feedback from Matt or others on whether this is the intended way of doing this. 1. The `extendMG` change can be brute-forced just to keep things going, by looking up dependency `ModSummary`s by `ModName` (we don't have full `Module`s at this point yet!): registerModSummary :: (GhcMonad m) => ModSummary -> m () registerModSummary ms = modifySession \env -> let mg = hsc_mod_graph env -- TODO: of course with a bit more housekeeping we can do better than this... edges = [ NodeKey_Module $ msKey ms' | dep <- deps, ms' <- mgModSummaries mg, ms_mod_name ms' == dep ] mg' = extendMG mg edges ms in env{ hsc_mod_graph = mg' } where deps = map (unLoc . snd) $ ms_textual_imps ms 2. `registerModule` can be kept mostly as-is, since it only uses `modifyUnitState` to change the active unit: registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () registerModule details iface = modifySession $ extendHpt . addModule where hmi = HomeModInfo iface details Nothing mod = mi_module iface modOrig = ModOrigin (Just True) [] [] True addModule = modifyUnitState $ \us -> us { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us } extendHpt = hscUpdateHUG $ addHomeModInfoToHug hmi modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env { hsc_unit_env = let ue = hsc_unit_env env in let units = ue_units ue units' = f units in ue_setUnits units' ue } 3. The tricky part is getting `addUnit` right. For this, based on the implementation of `initUnits`, I came up with the following: modifyUnitEnv :: (UnitEnv -> UnitEnv) -> HscEnv -> HscEnv modifyUnitEnv f env = env { hsc_unit_env = let ue = hsc_unit_env env in f ue } addUnit :: DynFlags -> UnitId -> HscEnv -> HscEnv addUnit dflags unitId = modifyUnitEnv $ \ue -> let dbs = ue_unit_dbs ue unit_state = ue_units ue home_unit = ue_homeUnit ue in flip updateHug ue $ unitEnv_insert unitId $ HomeUnitEnv { homeUnitEnv_units = unit_state , homeUnitEnv_unit_dbs = dbs , homeUnitEnv_dflags = dflags , homeUnitEnv_hpt = emptyHomePackageTable , homeUnitEnv_home_unit = home_unit } setCurrentUnit :: UnitId -> HscEnv -> HscEnv setCurrentUnit unitId = modifyUnitEnv $ ue_setActiveUnit unitId So my questions about this: 1. How does setting the home unit make sense? By doing this, I am effectively setting the home unit to `main` for all units, since that's the initial `ue_homeUnit` of the initial unit environment. Or does it not matter because after `addUnit`, I call `setCurrentUnit` anyway? I've found that I can't use the unit I am just adding as its own home unit, because that then leads to module name resolution problems in `main`: every imported module from `main` is searched for in `main` instead of its correct unit. 2. Speaking of `main`, why is it that when adding units, I have to skip `mainUnitId`, otherwise module resolution breaks again? 3. Unlike the previous version, I am no longer creating and putting `UnitInfo`s anywhere. Where is this going to bite me? Where (if anywhere) should I put `UnitInfo`s with the new setup? Thanks, Gergo -----Original Message----- From: ÉRDI Gergő Sent: Sunday, June 19, 2022 12:32 PM To: Erdi, Gergo Cc: GHC Devs ; Montelatici, Raphael Laurent Subject: [External] Re: Migration guide for multiple home units On Thu, 16 Jun 2022, Erdi, Gergo via ghc-devs wrote: > Is there a migration guide for GHC API clients for the new “multiple home units” > feature? OK so in concrete terms, please see my attached program which is a heavily cut-down, standalone version of my real program. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400^ (i.e. 3219610e3ba6cb6a5cd1f4e32e2b4befea5bd384) it compiles and works as expected. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400 onwards, two problems pop up: 1. `extendMG` has changed and now requires manually specifying outgoing dependency edges. I thought the whole point of `summariseFile` was to collect this information? The reason I need to `extendMG` at that point is to get intra-unit orphan instances working. 2. `modifyUnitState` and its two uses (`addUnit` and `registerModule`) need to be updated to the new API. I think it makes sense that these need changing, since they touch exactly on the issue of which units are being compiled right now. However, I don't know how to update these. Also, I guess `setHomeUnit` should change the `CurrentUnit` instead of the `HomeUnit` now? Thanks, Gergo This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. From matthewtpickering at gmail.com Mon Jun 20 16:47:21 2022 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Mon, 20 Jun 2022 17:47:21 +0100 Subject: Migration guide for multiple home units In-Reply-To: References: Message-ID: Gergo, I rewrote your example program for the new API but didn't quite manage to finish today. I will finish it off and post it tomorrow. Matt On Mon, Jun 20, 2022 at 6:16 AM Erdi, Gergo wrote: > > PUBLIC > > I managed to get this working, but I would like some feedback from Matt or others on whether this is the intended way of doing this. > > 1. The `extendMG` change can be brute-forced just to keep things going, by looking up dependency `ModSummary`s by `ModName` (we don't have full `Module`s at this point yet!): > > registerModSummary :: (GhcMonad m) => ModSummary -> m () registerModSummary ms = modifySession \env -> > let mg = hsc_mod_graph env > -- TODO: of course with a bit more housekeeping we can do better than this... > edges = [ NodeKey_Module $ msKey ms' | dep <- deps, ms' <- mgModSummaries mg, ms_mod_name ms' == dep ] > mg' = extendMG mg edges ms > in env{ hsc_mod_graph = mg' } > where > deps = map (unLoc . snd) $ ms_textual_imps ms > > 2. `registerModule` can be kept mostly as-is, since it only uses `modifyUnitState` to change the active unit: > > registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () registerModule details iface = modifySession $ extendHpt . addModule > where > hmi = HomeModInfo iface details Nothing > > mod = mi_module iface > modOrig = ModOrigin (Just True) [] [] True > > addModule = modifyUnitState $ \us -> us > { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us > } > > extendHpt = hscUpdateHUG $ addHomeModInfoToHug hmi > > modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env > { hsc_unit_env = let ue = hsc_unit_env env in > let units = ue_units ue > units' = f units > in ue_setUnits units' ue > } > > 3. The tricky part is getting `addUnit` right. For this, based on the implementation of `initUnits`, I came up with the following: > > modifyUnitEnv :: (UnitEnv -> UnitEnv) -> HscEnv -> HscEnv modifyUnitEnv f env = env > { hsc_unit_env = let ue = hsc_unit_env env in f ue > } > > addUnit :: DynFlags -> UnitId -> HscEnv -> HscEnv addUnit dflags unitId = modifyUnitEnv $ \ue -> > let dbs = ue_unit_dbs ue > unit_state = ue_units ue > home_unit = ue_homeUnit ue > in flip updateHug ue $ unitEnv_insert unitId $ HomeUnitEnv > { homeUnitEnv_units = unit_state > , homeUnitEnv_unit_dbs = dbs > , homeUnitEnv_dflags = dflags > , homeUnitEnv_hpt = emptyHomePackageTable > , homeUnitEnv_home_unit = home_unit > } > > setCurrentUnit :: UnitId -> HscEnv -> HscEnv setCurrentUnit unitId = modifyUnitEnv $ ue_setActiveUnit unitId > > > So my questions about this: > > 1. How does setting the home unit make sense? By doing this, I am effectively setting the home unit to `main` for all units, since that's the initial `ue_homeUnit` of the initial unit environment. Or does it not matter because after `addUnit`, I call `setCurrentUnit` anyway? I've found that I can't use the unit I am just adding as its own home unit, because that then leads to module name resolution problems in `main`: every imported module from `main` is searched for in `main` instead of its correct unit. > > 2. Speaking of `main`, why is it that when adding units, I have to skip `mainUnitId`, otherwise module resolution breaks again? > > 3. Unlike the previous version, I am no longer creating and putting `UnitInfo`s anywhere. Where is this going to bite me? Where (if anywhere) should I put `UnitInfo`s with the new setup? > > Thanks, > Gergo > > > -----Original Message----- > From: ÉRDI Gergő > Sent: Sunday, June 19, 2022 12:32 PM > To: Erdi, Gergo > Cc: GHC Devs ; Montelatici, Raphael Laurent > Subject: [External] Re: Migration guide for multiple home units > > > On Thu, 16 Jun 2022, Erdi, Gergo via ghc-devs wrote: > > > Is there a migration guide for GHC API clients for the new “multiple home units” > > feature? > > OK so in concrete terms, please see my attached program which is a heavily cut-down, standalone version of my real program. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400^ (i.e. > 3219610e3ba6cb6a5cd1f4e32e2b4befea5bd384) it compiles and works as expected. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400 onwards, two problems pop up: > > 1. `extendMG` has changed and now requires manually specifying outgoing dependency edges. I thought the whole point of `summariseFile` was to collect this information? The reason I need to `extendMG` at that point is to get intra-unit orphan instances working. > > 2. `modifyUnitState` and its two uses (`addUnit` and `registerModule`) need to be updated to the new API. I think it makes sense that these need changing, since they touch exactly on the issue of which units are being compiled right now. However, I don't know how to update these. Also, I guess `setHomeUnit` should change the `CurrentUnit` instead of the `HomeUnit` now? > > Thanks, > Gergo > > This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations > > Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm > > Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. > > Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. > > Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. > > Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. From matthewtpickering at gmail.com Mon Jun 20 16:53:37 2022 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Mon, 20 Jun 2022 17:53:37 +0100 Subject: Migration guide for multiple home units In-Reply-To: References: Message-ID: My issue was caused by running the program in a dirty tree :) Find the modified program here - https://gist.github.com/df64f8cda24dd0c66137f10ad9488912 I compiled it with HEAD. Hope you can make sense of it, I think it's simpler than before. Cheers, Matt On Mon, Jun 20, 2022 at 5:47 PM Matthew Pickering wrote: > > Gergo, I rewrote your example program for the new API but didn't quite > manage to finish today. I will finish it off and post it tomorrow. > > Matt > > On Mon, Jun 20, 2022 at 6:16 AM Erdi, Gergo wrote: > > > > PUBLIC > > > > I managed to get this working, but I would like some feedback from Matt or others on whether this is the intended way of doing this. > > > > 1. The `extendMG` change can be brute-forced just to keep things going, by looking up dependency `ModSummary`s by `ModName` (we don't have full `Module`s at this point yet!): > > > > registerModSummary :: (GhcMonad m) => ModSummary -> m () registerModSummary ms = modifySession \env -> > > let mg = hsc_mod_graph env > > -- TODO: of course with a bit more housekeeping we can do better than this... > > edges = [ NodeKey_Module $ msKey ms' | dep <- deps, ms' <- mgModSummaries mg, ms_mod_name ms' == dep ] > > mg' = extendMG mg edges ms > > in env{ hsc_mod_graph = mg' } > > where > > deps = map (unLoc . snd) $ ms_textual_imps ms > > > > 2. `registerModule` can be kept mostly as-is, since it only uses `modifyUnitState` to change the active unit: > > > > registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () registerModule details iface = modifySession $ extendHpt . addModule > > where > > hmi = HomeModInfo iface details Nothing > > > > mod = mi_module iface > > modOrig = ModOrigin (Just True) [] [] True > > > > addModule = modifyUnitState $ \us -> us > > { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us > > } > > > > extendHpt = hscUpdateHUG $ addHomeModInfoToHug hmi > > > > modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in > > let units = ue_units ue > > units' = f units > > in ue_setUnits units' ue > > } > > > > 3. The tricky part is getting `addUnit` right. For this, based on the implementation of `initUnits`, I came up with the following: > > > > modifyUnitEnv :: (UnitEnv -> UnitEnv) -> HscEnv -> HscEnv modifyUnitEnv f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in f ue > > } > > > > addUnit :: DynFlags -> UnitId -> HscEnv -> HscEnv addUnit dflags unitId = modifyUnitEnv $ \ue -> > > let dbs = ue_unit_dbs ue > > unit_state = ue_units ue > > home_unit = ue_homeUnit ue > > in flip updateHug ue $ unitEnv_insert unitId $ HomeUnitEnv > > { homeUnitEnv_units = unit_state > > , homeUnitEnv_unit_dbs = dbs > > , homeUnitEnv_dflags = dflags > > , homeUnitEnv_hpt = emptyHomePackageTable > > , homeUnitEnv_home_unit = home_unit > > } > > > > setCurrentUnit :: UnitId -> HscEnv -> HscEnv setCurrentUnit unitId = modifyUnitEnv $ ue_setActiveUnit unitId > > > > > > So my questions about this: > > > > 1. How does setting the home unit make sense? By doing this, I am effectively setting the home unit to `main` for all units, since that's the initial `ue_homeUnit` of the initial unit environment. Or does it not matter because after `addUnit`, I call `setCurrentUnit` anyway? I've found that I can't use the unit I am just adding as its own home unit, because that then leads to module name resolution problems in `main`: every imported module from `main` is searched for in `main` instead of its correct unit. > > > > 2. Speaking of `main`, why is it that when adding units, I have to skip `mainUnitId`, otherwise module resolution breaks again? > > > > 3. Unlike the previous version, I am no longer creating and putting `UnitInfo`s anywhere. Where is this going to bite me? Where (if anywhere) should I put `UnitInfo`s with the new setup? > > > > Thanks, > > Gergo > > > > > > -----Original Message----- > > From: ÉRDI Gergő > > Sent: Sunday, June 19, 2022 12:32 PM > > To: Erdi, Gergo > > Cc: GHC Devs ; Montelatici, Raphael Laurent > > Subject: [External] Re: Migration guide for multiple home units > > > > > > On Thu, 16 Jun 2022, Erdi, Gergo via ghc-devs wrote: > > > > > Is there a migration guide for GHC API clients for the new “multiple home units” > > > feature? > > > > OK so in concrete terms, please see my attached program which is a heavily cut-down, standalone version of my real program. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400^ (i.e. > > 3219610e3ba6cb6a5cd1f4e32e2b4befea5bd384) it compiles and works as expected. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400 onwards, two problems pop up: > > > > 1. `extendMG` has changed and now requires manually specifying outgoing dependency edges. I thought the whole point of `summariseFile` was to collect this information? The reason I need to `extendMG` at that point is to get intra-unit orphan instances working. > > > > 2. `modifyUnitState` and its two uses (`addUnit` and `registerModule`) need to be updated to the new API. I think it makes sense that these need changing, since they touch exactly on the issue of which units are being compiled right now. However, I don't know how to update these. Also, I guess `setHomeUnit` should change the `CurrentUnit` instead of the `HomeUnit` now? > > > > Thanks, > > Gergo > > > > This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations > > > > Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm > > > > Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. > > > > Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. > > > > Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. > > > > Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. From Gergo.Erdi at sc.com Tue Jun 21 09:41:36 2022 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Tue, 21 Jun 2022 09:41:36 +0000 Subject: Migration guide for multiple home units Message-ID: PUBLIC Thanks, based on this I got things going. I might have to come back to you if it turns out something's still broken in my original full application. -----Original Message----- From: Matthew Pickering Sent: Tuesday, June 21, 2022 12:54 AM To: Erdi, Gergo Cc: ÉRDI Gergo ; GHC Devs ; Montelatici, Raphael Laurent Subject: [External] Re: Migration guide for multiple home units My issue was caused by running the program in a dirty tree :) Find the modified program here - https://clicktime.symantec.com/39Wzk4MM32vW4JJm9h9nYC96xn?u=https%3A%2F%2Fgist.github.com%2Fdf64f8cda24dd0c66137f10ad9488912 I compiled it with HEAD. Hope you can make sense of it, I think it's simpler than before. Cheers, Matt On Mon, Jun 20, 2022 at 5:47 PM Matthew Pickering wrote: > > Gergo, I rewrote your example program for the new API but didn't quite > manage to finish today. I will finish it off and post it tomorrow. > > Matt > > On Mon, Jun 20, 2022 at 6:16 AM Erdi, Gergo wrote: > > > > PUBLIC > > > > I managed to get this working, but I would like some feedback from Matt or others on whether this is the intended way of doing this. > > > > 1. The `extendMG` change can be brute-forced just to keep things going, by looking up dependency `ModSummary`s by `ModName` (we don't have full `Module`s at this point yet!): > > > > registerModSummary :: (GhcMonad m) => ModSummary -> m () registerModSummary ms = modifySession \env -> > > let mg = hsc_mod_graph env > > -- TODO: of course with a bit more housekeeping we can do better than this... > > edges = [ NodeKey_Module $ msKey ms' | dep <- deps, ms' <- mgModSummaries mg, ms_mod_name ms' == dep ] > > mg' = extendMG mg edges ms > > in env{ hsc_mod_graph = mg' } > > where > > deps = map (unLoc . snd) $ ms_textual_imps ms > > > > 2. `registerModule` can be kept mostly as-is, since it only uses `modifyUnitState` to change the active unit: > > > > registerModule :: (GhcMonad m) => ModDetails -> ModIface -> m () registerModule details iface = modifySession $ extendHpt . addModule > > where > > hmi = HomeModInfo iface details Nothing > > > > mod = mi_module iface > > modOrig = ModOrigin (Just True) [] [] True > > > > addModule = modifyUnitState $ \us -> us > > { moduleNameProvidersMap = M.insert (moduleName mod) (M.singleton mod modOrig) $ moduleNameProvidersMap us > > } > > > > extendHpt = hscUpdateHUG $ addHomeModInfoToHug hmi > > > > modifyUnitState :: (UnitState -> UnitState) -> HscEnv -> HscEnv modifyUnitState f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in > > let units = ue_units ue > > units' = f units > > in ue_setUnits units' ue > > } > > > > 3. The tricky part is getting `addUnit` right. For this, based on the implementation of `initUnits`, I came up with the following: > > > > modifyUnitEnv :: (UnitEnv -> UnitEnv) -> HscEnv -> HscEnv modifyUnitEnv f env = env > > { hsc_unit_env = let ue = hsc_unit_env env in f ue > > } > > > > addUnit :: DynFlags -> UnitId -> HscEnv -> HscEnv addUnit dflags unitId = modifyUnitEnv $ \ue -> > > let dbs = ue_unit_dbs ue > > unit_state = ue_units ue > > home_unit = ue_homeUnit ue > > in flip updateHug ue $ unitEnv_insert unitId $ HomeUnitEnv > > { homeUnitEnv_units = unit_state > > , homeUnitEnv_unit_dbs = dbs > > , homeUnitEnv_dflags = dflags > > , homeUnitEnv_hpt = emptyHomePackageTable > > , homeUnitEnv_home_unit = home_unit > > } > > > > setCurrentUnit :: UnitId -> HscEnv -> HscEnv setCurrentUnit unitId = > > modifyUnitEnv $ ue_setActiveUnit unitId > > > > > > So my questions about this: > > > > 1. How does setting the home unit make sense? By doing this, I am effectively setting the home unit to `main` for all units, since that's the initial `ue_homeUnit` of the initial unit environment. Or does it not matter because after `addUnit`, I call `setCurrentUnit` anyway? I've found that I can't use the unit I am just adding as its own home unit, because that then leads to module name resolution problems in `main`: every imported module from `main` is searched for in `main` instead of its correct unit. > > > > 2. Speaking of `main`, why is it that when adding units, I have to skip `mainUnitId`, otherwise module resolution breaks again? > > > > 3. Unlike the previous version, I am no longer creating and putting `UnitInfo`s anywhere. Where is this going to bite me? Where (if anywhere) should I put `UnitInfo`s with the new setup? > > > > Thanks, > > Gergo > > > > > > -----Original Message----- > > From: ÉRDI Gergő > > Sent: Sunday, June 19, 2022 12:32 PM > > To: Erdi, Gergo > > Cc: GHC Devs ; Montelatici, Raphael Laurent > > > > Subject: [External] Re: Migration guide for multiple home units > > > > > > On Thu, 16 Jun 2022, Erdi, Gergo via ghc-devs wrote: > > > > > Is there a migration guide for GHC API clients for the new “multiple home units” > > > feature? > > > > OK so in concrete terms, please see my attached program which is a heavily cut-down, standalone version of my real program. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400^ (i.e. > > 3219610e3ba6cb6a5cd1f4e32e2b4befea5bd384) it compiles and works as expected. On commit fd42ab5fa1df847a6b595dfe4b63d9c7eecbf400 onwards, two problems pop up: > > > > 1. `extendMG` has changed and now requires manually specifying outgoing dependency edges. I thought the whole point of `summariseFile` was to collect this information? The reason I need to `extendMG` at that point is to get intra-unit orphan instances working. > > > > 2. `modifyUnitState` and its two uses (`addUnit` and `registerModule`) need to be updated to the new API. I think it makes sense that these need changing, since they touch exactly on the issue of which units are being compiled right now. However, I don't know how to update these. Also, I guess `setHomeUnit` should change the `CurrentUnit` instead of the `HomeUnit` now? > > > > Thanks, > > Gergo > > > > This email and any attachments are confidential and may also be > > privileged. If you are not the intended recipient, please delete all > > copies and notify the sender immediately. You may wish to refer to > > the incorporation details of Standard Chartered PLC, Standard > > Chartered Bank and their subsidiaries at https: > > //www.sc.com/en/our-locations > > > > Where you have a Financial Markets relationship with Standard > > Chartered PLC, Standard Chartered Bank and their subsidiaries (the > > "Group"), information on the regulatory standards we adhere to and > > how it may affect you can be found in our Regulatory Compliance > > Statement at https: //www.sc.com/rcs/ and Regulatory Compliance > > Disclosures at http: //www.sc.com/rcs/fm > > > > Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. > > > > Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. > > > > Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. > > > > Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. From ben at well-typed.com Fri Jun 24 20:56:33 2022 From: ben at well-typed.com (Ben Gamari) Date: Fri, 24 Jun 2022 16:56:33 -0400 Subject: GHC 9.4.1-alpha3 now available Message-ID: <878rpl3hn7.fsf@smart-cactus.org> The GHC developers are happy to announce the availability of the third alpha release of the GHC 9.4 series. Binary distributions, source distributions, and documentation are available at [downloads.haskell.org](https://downloads.haskell.org/ghc/9.4.1-alpha3). This major release will include: - A new profiling mode, `-fprof-late`, which adds automatic cost-center annotations to all top-level functions *after* Core optimisation has run. This incurs significantly less performance cost while still providing informative profiles. - A variety of plugin improvements including the introduction of a new plugin type, *defaulting plugins*, and the ability for typechecking plugins to rewrite type-families. - An improved constructed product result analysis, allowing unboxing of nested structures, and a new boxity analysis, leading to less reboxing. - Introduction of a tag-check elision optimisation, bringing significant performance improvements in strict programs. - Generalisation of a variety of primitive types to be levity polymorphic. Consequently, the `ArrayArray#` type can at long last be retired, replaced by standard `Array#`. - Introduction of the `\cases` syntax from [GHC proposal 0302] - A complete overhaul of GHC's Windows support. This includes a migration to a fully Clang-based C toolchain, a deep refactoring of the linker, and many fixes in WinIO. - Support for multiple home packages, significantly improving support in IDEs and other tools for multi-package projects. - A refactoring of GHC's error message infrastructure, allowing GHC to provide diagnostic information to downstream consumers as structured data, greatly easing IDE support. - Significant compile-time improvements to runtime and memory consumption. - On overhaul of our packaging infrastructure, allowing full traceability of release artifacts and more reliable binary distributions. - ... and much more. See the [release notes] for a full accounting. Note that, as 9.4.1 is the first release for which the released artifacts will all be generated by our Hadrian build system, it's possible that there will be packaging issues. If you enounter trouble while using a binary distribution, please open a [ticket]. Likewise, if you are a downstream packager, do consider migrating to [Hadrian] to run your build; the Hadrian build system can be built using `cabal-install`, `stack`, or the in-tree [bootstrap script]. We would like to thank Microsoft Azure, GitHub, IOG, the Zw3rk stake pool, Tweag I/O, Serokell, Equinix, SimSpace, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release. As always, do give this release a try and open a [ticket] if you see anything amiss. Happy testing, - Ben [GHC proposal 0302]: https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0302-cases.rst [ticket]: https://gitlab.haskell.org/ghc/ghc/-/issues/new [bootstrap script]: https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian/bootstrap/README.md [Hadrian]: https://gitlab.haskell.org/ghc/ghc/-/blob/e2520df3fffa0cf22fb19c5fb872832d11c07d35/hadrian [release notes]: https://downloads.haskell.org/~ghc/9.4.1-alpha3/docs/users_guide/9.4.1-notes.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From matthewtpickering at gmail.com Thu Jun 30 07:36:03 2022 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Thu, 30 Jun 2022 08:36:03 +0100 Subject: Release Plans - 9.4.1 and 9.2.4 Message-ID: Hello all, Welcome to another update on our release plans. 9.4.1 (Ben Gamari) ~~~~ We proceed smoothly onwards towards the release at the end of July. The alpha3 was released last week - https://discourse.haskell.org/t/ghc-9-4-1-alpha3-now-available/4701 The next release is scheduled to be the first release candidate. 9.2.4 (Zubin Duggal) ~~~~ We intend to also publish a new version in the 9.2 series before the end of July. The priority of this release is motivated by: * Adding support for the DeepSubsumption language extension * Fixing some critical RTS bugs Cheers, Matt From matthewtpickering at gmail.com Thu Jun 30 07:46:43 2022 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Thu, 30 Jun 2022 08:46:43 +0100 Subject: Imminent Removal of Make Build System Message-ID: Hi all, We are imminently going to remove the make build system. See: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7094 There is quite extensive documentation about using hadrian, the best place to start is the in-tree documentation: https://gitlab.haskell.org/ghc/ghc/-/blob/master/hadrian/README.md For packagers: * The 9.4.* series will be the last release series which can be built with the make build system. * 9.6.* will only be able to be built using hadrian. * See ghcs-nix for an example of how to modify your own build system to use hadrian: https://gitlab.haskell.org/bgamari/ghcs-nix/-/blob/master/hadrian.nix * Sam Derbyshire is creating a guide aimed at packagers describing how to migrate their build plans. Cheers, Matt From Gergo.Erdi at sc.com Thu Jun 30 09:55:40 2022 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Thu, 30 Jun 2022 09:55:40 +0000 Subject: Migration guide for multiple home units In-Reply-To: References: Message-ID: PUBLIC In case anyone finds this interesting, I ended up splitting the unit initialization into two parts: one where all units are registered, without any package dependencies, and one where the package dependencies of a single unit are registered. This allowed us to start with just knowing which units are in play, use summariseFile to discover module dependencies, and then use that to compute package dependencies. It's basically just a refactoring of Matt's code from his email up the chain. setUnits :: [(UnitId, DynFlags -> DynFlags)] -> HscEnv -> IO HscEnv setUnits units hsc_env = do home_unit_graph <- createHomeUnitGraph hsc_env units' setHomeUnitGraph hsc_env dummy_home_unit home_unit_graph where dflags0 = hsc_dflags hsc_env units'@((dummy_home_unit, _):_) = [ (unit_id, unitFlags dflags0{ homeUnitId_ = unit_id, packageFlags = [] }) | (unit_id, unitFlags) <- units ] setUnitDeps :: UnitId -> [UnitId] -> HscEnv -> IO HscEnv setUnitDeps unit_id deps hsc_env = do home_unit_env <- fillHomeUnitEnv hsc_env home_units new_dflags let home_unit_graph = unitEnv_insert unit_id home_unit_env home_unit_graph0 setHomeUnitGraph hsc_env unit_id home_unit_graph where old_dflags = ue_unitFlags unit_id $ hsc_unit_env hsc_env new_dflags = old_dflags { packageFlags = [ ExposePackage "" (UnitIdArg (RealUnit $ Definite uid)) (ModRenaming True []) | uid <- deps] } home_unit_graph0 = ue_home_unit_graph . hsc_unit_env $ hsc_env home_units = unitEnv_keys home_unit_graph0 setHomeUnitGraph :: HscEnv -> UnitId -> HomeUnitGraph -> IO HscEnv setHomeUnitGraph hsc_env unit_id home_unit_graph = do unit_env <- assertUnitEnvInvariant <$> initUnitEnv unit_id home_unit_graph (ghcNameVersion dflags0) (targetPlatform dflags0) return $ hsc_env { hsc_unit_env = unit_env } where dflags0 = hsc_dflags hsc_env createHomeUnitGraph :: HscEnv -> [(UnitId, DynFlags)] -> IO HomeUnitGraph createHomeUnitGraph hsc_env units = unitEnv_new <$> traverse (fillHomeUnitEnv hsc_env home_units) (Map.fromList units) where home_units = Set.fromList $ map fst units fillHomeUnitEnv :: HscEnv -> Set.Set UnitId -> DynFlags -> IO HomeUnitEnv fillHomeUnitEnv hsc_env home_units dflags = do (dbs, unit_state, home_unit, _) <- initUnits logger dflags Nothing home_units pure $ HomeUnitEnv { homeUnitEnv_units = unit_state , homeUnitEnv_unit_dbs = Just dbs , homeUnitEnv_dflags = dflags , homeUnitEnv_hpt = emptyHomePackageTable , homeUnitEnv_home_unit = Just home_unit } where logger = hsc_logger hsc_env -----Original Message----- From: Erdi, Gergo Sent: Tuesday, June 21, 2022 5:42 PM To: Matthew Pickering Cc: ÉRDI Gergo ; GHC Devs ; Montelatici, Raphael Laurent Subject: Re: Migration guide for multiple home units Thanks, based on this I got things going. I might have to come back to you if it turns out something's still broken in my original full application. This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. From amindfv at mailbox.org Thu Jun 30 19:48:16 2022 From: amindfv at mailbox.org (amindfv at mailbox.org) Date: Thu, 30 Jun 2022 12:48:16 -0700 Subject: Imminent Removal of Make Build System In-Reply-To: References: Message-ID: As a site note, it's probably worth removing or adding a deprecation note to this repo: https://github.com/ghc/hadrian It's the first search result for me for "ghc hadrian" but it's 4 years behind the code in the GHC GitLab. Tom On Thu, Jun 30, 2022 at 08:46:43AM +0100, Matthew Pickering wrote: > Hi all, > > We are imminently going to remove the make build system. > > See: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7094 > > There is quite extensive documentation about using hadrian, the best > place to start is the in-tree documentation: > > https://gitlab.haskell.org/ghc/ghc/-/blob/master/hadrian/README.md > > For packagers: > > * The 9.4.* series will be the last release series which can be built > with the make build system. > * 9.6.* will only be able to be built using hadrian. > * See ghcs-nix for an example of how to modify your own build system > to use hadrian: > https://gitlab.haskell.org/bgamari/ghcs-nix/-/blob/master/hadrian.nix > * Sam Derbyshire is creating a guide aimed at packagers describing how > to migrate their build plans. > > Cheers, > > Matt > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs