From andrew.lelechenko at gmail.com Sat Jan 4 16:42:45 2025 From: andrew.lelechenko at gmail.com (Andrew Lelechenko) Date: Sat, 4 Jan 2025 16:42:45 +0000 Subject: CLC Elections January 2025 Message-ID: <194DEEE9-369C-4D61-894D-72FCA12C0FFE@gmail.com> Core Libraries Committee seeks nominations for three expiring seats (including my own). I’m specifically sending this email to ghc-devs, because im my opinion CLC can benefit from representatives of GHC team. # What is CLC? CLC is a volunteer body, primarily tasked with management of API of `base` package, but also owning so-called Core Libraries and responsible for PVP. You can find out more at https://github.com/haskell/core-libraries-committee/ # Who should apply? Anyone who meets the following criteria should apply: * Candidates should have enough bandwidth to review merge requests to `base` on a semi-frequent basis (~3-4 per month), and sustain this for their 3 years term in a healthy manner. * Candidates should be able to contribute opinions and analysis to issues raised by the community as a part of the CLC proposal process on a semi-frequent basis (~7 per month). * Candidates should be good communicators, and at least be able to articulate to the CLC team when they will be available vs. unavailable. * Candidates should be productive, and be able to follow through on merge requests and conversations to their completion in a diligent and timely manner. We encourage any and all who satisfy these requirements to apply. Please note that we are not looking for the biggest galaxy brain in the room -- quite the opposite. We are looking for productive, motivated individuals who want to help support the ecosystem that we love. As such, we hope to build a broad sample of the community. # How can I apply? To apply for one of these positions, send an email to clc.nominations.2025 at gmail.com that consists of the following data: * The subject “CLC Election September 2025 - {your name}”. * Why you think you’re a good fit given the above criteria. * If applicable, please point us to some code you’ve written. Please apply before Feb 2. Best regards, Andrew From liisikerik at hotmail.com Sat Jan 4 19:48:01 2025 From: liisikerik at hotmail.com (Liisi Kerik) Date: Sat, 4 Jan 2025 21:48:01 +0200 Subject: Approving gitlab account for creating an issue Message-ID: Hi, Could you please approve my gitlab account (@LiisiKerik)? I recently ran into a performance problem (ghc plugins make builds very slow on Windows) and would like to create an issue about this. Best regards, Liisi Kerik From allbery.b at gmail.com Sat Jan 4 19:54:42 2025 From: allbery.b at gmail.com (Brandon Allbery) Date: Sat, 4 Jan 2025 14:54:42 -0500 Subject: Approving gitlab account for creating an issue In-Reply-To: References: Message-ID: I'm afraid I'm not seeing an account application for that username. On Sat, Jan 4, 2025 at 2:48 PM Liisi Kerik wrote: > Hi, > > Could you please approve my gitlab account (@LiisiKerik)? I recently ran > into a performance problem (ghc plugins make builds very slow on > Windows) and would like to create an issue about this. > > Best regards, > > Liisi Kerik > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -- brandon s allbery kf8nh allbery.b at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Thu Jan 9 07:54:24 2025 From: bryan at haskell.foundation (Bryan Richter) Date: Thu, 9 Jan 2025 09:54:24 +0200 Subject: gitlab.haskell.org maintenance 2025-01-10 Message-ID: Hi folks, In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Fri Jan 10 07:48:12 2025 From: bryan at haskell.foundation (Bryan Richter) Date: Fri, 10 Jan 2025 09:48:12 +0200 Subject: gitlab.haskell.org maintenance starting now In-Reply-To: References: Message-ID: On Thu, 9 Jan 2025 at 09:54, Bryan Richter wrote: > Hi folks, > > In about 24 hours, I will perform some maintenance on gitlab.haskell.org. > I will be addressing the frequent disk outages by improving how partitions > are used. This will only require a system restart, so you can expect some > short downtime. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Fri Jan 10 08:44:14 2025 From: bryan at haskell.foundation (Bryan Richter) Date: Fri, 10 Jan 2025 10:44:14 +0200 Subject: gitlab.haskell.org maintenance starting now In-Reply-To: References: Message-ID: Well, everything went according to plan, but the plan had a flaw. ;) GitLab is back up, but the maintenance did not succeed. I will try again in an hour. On Fri, 10 Jan 2025 at 09:48, Bryan Richter wrote: > > On Thu, 9 Jan 2025 at 09:54, Bryan Richter > wrote: > >> Hi folks, >> >> In about 24 hours, I will perform some maintenance on gitlab.haskell.org. >> I will be addressing the frequent disk outages by improving how partitions >> are used. This will only require a system restart, so you can expect some >> short downtime. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Fri Jan 10 12:04:18 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 10 Jan 2025 12:04:18 +0000 Subject: gitlab.haskell.org maintenance starting now In-Reply-To: References:

Message-ID: Hi, Thanks Bryan for taking on the upgrade. Is it possible to perform a rollback to the last known good state for this afternoon so that we can make some progress with work? Cheers, Matt On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < ghc-devs at haskell.org> wrote: > Well, everything went according to plan, but the plan had a flaw. ;) > > GitLab is back up, but the maintenance did not succeed. I will try again > in an hour. > > On Fri, 10 Jan 2025 at 09:48, Bryan Richter > wrote: > >> >> On Thu, 9 Jan 2025 at 09:54, Bryan Richter >> wrote: >> >>> Hi folks, >>> >>> In about 24 hours, I will perform some maintenance on gitlab.haskell.org. >>> I will be addressing the frequent disk outages by improving how partitions >>> are used. This will only require a system restart, so you can expect some >>> short downtime. >>> >> _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Fri Jan 10 13:13:51 2025 From: bryan at haskell.foundation (Bryan Richter) Date: Fri, 10 Jan 2025 15:13:51 +0200 Subject: gitlab.haskell.org maintenance starting now In-Reply-To: References:

Message-ID: Sadly, no. The issue is at a level below backups. And for a reason I haven't looked into yet, there are no previous NixOS generations available to roll back to. The principal issue is that while booting, NixOS Stage 1 prints the message "cannot import 'rpool': more than one matching pool". The root filesystem is in that pool, so booting fails. Unfortunately I don't know why it says that. From within a rescue system, I can see just one pool named "rpool", and I can import it and mount filesystems from it just fine. So the current task is figuring out why NixOS Stage 1 sees otherwise. On Fri, 10 Jan 2025 at 14:04, Matthew Pickering wrote: > Hi, > > Thanks Bryan for taking on the upgrade. > > Is it possible to perform a rollback to the last known good state for this > afternoon so that we can make some progress with work? > > Cheers, > > Matt > > On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < > ghc-devs at haskell.org> wrote: > >> Well, everything went according to plan, but the plan had a flaw. ;) >> >> GitLab is back up, but the maintenance did not succeed. I will try again >> in an hour. >> >> On Fri, 10 Jan 2025 at 09:48, Bryan Richter >> wrote: >> >>> >>> On Thu, 9 Jan 2025 at 09:54, Bryan Richter >>> wrote: >>> >>>> Hi folks, >>>> >>>> In about 24 hours, I will perform some maintenance on >>>> gitlab.haskell.org. I will be addressing the frequent disk outages by >>>> improving how partitions are used. This will only require a system restart, >>>> so you can expect some short downtime. >>>> >>> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryan at haskell.foundation Fri Jan 10 16:00:56 2025 From: bryan at haskell.foundation (Bryan Richter) Date: Fri, 10 Jan 2025 18:00:56 +0200 Subject: gitlab.haskell.org maintenance starting now In-Reply-To: References:

Message-ID: It's back! Ancient, bad ZFS metadata was causing an issue during boot. I eventually found where the metadata was lurking and was able to remove it. I can speculate why this wasn't a problem in earlier boots, but I haven't actually looked into it. (The last boot was 2 years ago!) The problem had nothing to do with recent changes made by me or anyone else. It may be related to operating system upgrades that have happened over the last two years. Changes to the OS may have exposed the lurking filesystem issue. As a bonus, my *completely unrelated* filesystem changes have taken effect, and I believe we'll see a lot fewer disk outages going forward. -Bryan On Fri, 10 Jan 2025 at 15:13, Bryan Richter wrote: > Sadly, no. The issue is at a level below backups. And for a reason I > haven't looked into yet, there are no previous NixOS generations available > to roll back to. > > The principal issue is that while booting, NixOS Stage 1 prints the > message "cannot import 'rpool': more than one matching pool". The root > filesystem is in that pool, so booting fails. > > Unfortunately I don't know why it says that. From within a rescue system, > I can see just one pool named "rpool", and I can import it and mount > filesystems from it just fine. So the current task is figuring out why > NixOS Stage 1 sees otherwise. > > > > On Fri, 10 Jan 2025 at 14:04, Matthew Pickering < > matthewtpickering at gmail.com> wrote: > >> Hi, >> >> Thanks Bryan for taking on the upgrade. >> >> Is it possible to perform a rollback to the last known good state for >> this afternoon so that we can make some progress with work? >> >> Cheers, >> >> Matt >> >> On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < >> ghc-devs at haskell.org> wrote: >> >>> Well, everything went according to plan, but the plan had a flaw. ;) >>> >>> GitLab is back up, but the maintenance did not succeed. I will try again >>> in an hour. >>> >>> On Fri, 10 Jan 2025 at 09:48, Bryan Richter >>> wrote: >>> >>>> >>>> On Thu, 9 Jan 2025 at 09:54, Bryan Richter >>>> wrote: >>>> >>>>> Hi folks, >>>>> >>>>> In about 24 hours, I will perform some maintenance on >>>>> gitlab.haskell.org. I will be addressing the frequent disk outages by >>>>> improving how partitions are used. This will only require a system restart, >>>>> so you can expect some short downtime. >>>>> >>>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergo at erdi.hu Fri Jan 17 08:07:38 2025 From: gergo at erdi.hu (=?UTF-8?B?R2VyZ8WRIMOJcmRp?=) Date: Fri, 17 Jan 2025 16:07:38 +0800 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces Message-ID: Hi, I’m using the GHC API to typecheck 35,000 modules that form a complicated dependency graph (with multiple top-level modules, i.e. there’s no single “god module” that would transitively depend on everything else), and I noticed that peak memory usage is wildly different when everything is done from scratch vs. when everything is loaded from files containing ModIfaces: 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs 33M for 407 modules. I’m aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586 and so when I finish typechecking a module, I take the resulting ModIface and create the ModDetails that ends up in the HomeUnitGraph from that. My understanding of Matt’s original GHC fix in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478 is that it does the same, i.e. it only makes a fresh ModDetails only once per module, after the ModIface is ready. But of course that still means that ModDetails can only keep growing as more and more parts of it are used for typechecking more and more dependants. Could that be the cause? I tried a crude experiment of “putting the toothpaste back in the tube” by replacing all ModDetails with a fresh one in the HUG after each finished typechecking , but that’s a complete disaster for memory usage: even for the small 407 module example, the memory usage shoots up to 1.5G. I can imagine it’s because imported Ids are probably not shared anymore between different importer modules. Any ideas on how I could improve memory usage in the from-scratch case, so that it's more similar to the from-ModIface case? Thanks, Gergo -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gergo.Erdi at sc.com Fri Jan 17 08:54:52 2025 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Fri, 17 Jan 2025 08:54:52 +0000 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: Message-ID: PUBLIC Looking at this with ghc-debug, at least I can see why we have the huge memory usage when recreating ModDetails: there are lots of HscEnvs stored all over the heap (I assume in thunks inside already compiled modules), when I recreate the ModDetails and replace them in the HUG, I am accumulating more and more copies of the same MOdDetails, since other HscEnvs are of course still pointing to the HUG that had the previous ModDetails, and so on. Given this, I don't really know yet if remaking the ModDetails would help me or not, since trying it is now blocking on figuring out how I could avoid having multiple HscEnvs in memory at the same time... From: ghc-devs On Behalf Of Gergo Érdi Sent: Friday, January 17, 2025 4:08 PM To: GHC Devs Cc: Montelatici, Raphael Laurent Subject: [External] GHC memory usage when typechecking from source vs. loading ModIfaces Hi, I'm using the GHC API to typecheck 35,000 modules that form a complicated dependency graph (with multiple top-level modules, i.e. there's no single "god module" that would transitively depend on everything else), and I noticed that peak memory usage is wildly different when everything is done from scratch vs. when everything is loaded from files containing ModIfaces: 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs 33M for 407 modules. I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586 and so when I finish typechecking a module, I take the resulting ModIface and create the ModDetails that ends up in the HomeUnitGraph from that. My understanding of Matt's original GHC fix in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478 is that it does the same, i.e. it only makes a fresh ModDetails only once per module, after the ModIface is ready. But of course that still means that ModDetails can only keep growing as more and more parts of it are used for typechecking more and more dependants. Could that be the cause? I tried a crude experiment of "putting the toothpaste back in the tube" by replacing all ModDetails with a fresh one in the HUG after each finished typechecking , but that's a complete disaster for memory usage: even for the small 407 module example, the memory usage shoots up to 1.5G. I can imagine it's because imported Ids are probably not shared anymore between different importer modules. Any ideas on how I could improve memory usage in the from-scratch case, so that it's more similar to the from-ModIface case? Thanks, Gergo ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com -------------- next part -------------- An HTML attachment was scrubbed... URL: From zubin at well-typed.com Fri Jan 17 10:30:49 2025 From: zubin at well-typed.com (Zubin Duggal) Date: Fri, 17 Jan 2025 16:00:49 +0530 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: Message-ID: See https://gitlab.haskell.org/ghc/ghc/-/issues/25511 On 25/01/17 08:54, Erdi, Gergo via ghc-devs wrote: >PUBLIC > >Looking at this with ghc-debug, at least I can see why we have the huge memory usage when recreating ModDetails: there are lots of HscEnvs stored all over the heap (I assume in thunks inside already compiled modules), when I recreate the ModDetails and replace them in the HUG, I am accumulating more and more copies of the same MOdDetails, since other HscEnvs are of course still pointing to the HUG that had the previous ModDetails, and so on. > >Given this, I don't really know yet if remaking the ModDetails would help me or not, since trying it is now blocking on figuring out how I could avoid having multiple HscEnvs in memory at the same time... > >From: ghc-devs On Behalf Of Gergo Érdi >Sent: Friday, January 17, 2025 4:08 PM >To: GHC Devs >Cc: Montelatici, Raphael Laurent >Subject: [External] GHC memory usage when typechecking from source vs. loading ModIfaces > >Hi, > >I'm using the GHC API to typecheck 35,000 modules that form a complicated dependency graph (with multiple top-level modules, i.e. there's no single "god module" that would transitively depend on everything else), and I noticed that peak memory usage is wildly different when everything is done from scratch vs. when everything is loaded from files containing ModIfaces: 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs 33M for 407 modules. > >I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586 and so when I finish typechecking a module, I take the resulting ModIface and create the ModDetails that ends up in the HomeUnitGraph from that. My understanding of Matt's original GHC fix in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478 is that it does the same, i.e. it only makes a fresh ModDetails only once per module, after the ModIface is ready. > >But of course that still means that ModDetails can only keep growing as more and more parts of it are used for typechecking more and more dependants. Could that be the cause? I tried a crude experiment of "putting the toothpaste back in the tube" by replacing all ModDetails with a fresh one in the HUG after each finished typechecking , but that's a complete disaster for memory usage: even for the small 407 module example, the memory usage shoots up to 1.5G. I can imagine it's because imported Ids are probably not shared anymore between different importer modules. > >Any ideas on how I could improve memory usage in the from-scratch case, so that it's more similar to the from-ModIface case? > >Thanks, > Gergo > > >---------------------------------------------------------------------- >This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations > >Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm > >Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer. > >Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. > >Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same. > >Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products. > >---------------------------------------------------------------------- >This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. > >---------------------------------------------------------------------- >This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com >_______________________________________________ >ghc-devs mailing list >ghc-devs at haskell.org >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From klebinger.andreas at gmx.at Fri Jan 17 11:02:55 2025 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Fri, 17 Jan 2025 12:02:55 +0100 Subject: [PSA]: Correctness issue in GHC-9.12 Message-ID: <80fdf81c-84eb-4265-8214-ad5f6ea50174@gmx.at> A bug in 9.12.1 was reported and confirmed (ghc bug https://gitlab.haskell.org/ghc/ghc/-/issues/25653) which can lead to incorrect runtime results in compiled programs. In particular this seems to happen for sub-word divisions with a known divisor when optimizations are enabled and affects at least the x86 backend, although it’s likely that other backends are also affected. We plan to release a bug fix release 9.12.2 in the near future to address this issue and recommend people do not upgrade until then. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Fri Jan 17 11:19:20 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 17 Jan 2025 11:19:20 +0000 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: Message-ID: Hi Gergo, I think there a few things to say here. 1. As Zubin points out we have recently been concerned with improving the memory usage of large module sessions (#25511, !13675, !13593) I imagine all these patches will greatly help the memory usage in your use case. 2. You are absolutely right that ModDetails can get forced and is never reset. If you try !13675, it should be much more easily possible to reset the ModDetails by writing into the IORef which stores each home package. 3. If you share your example or perhaps even a trace from ghc-debug then I will be happy to investigate further as it seems like a great test case for the work we have recently been doing. Cheers, Matt On Fri, Jan 17, 2025 at 10:31 AM Zubin Duggal wrote: > See https://gitlab.haskell.org/ghc/ghc/-/issues/25511 > > On 25/01/17 08:54, Erdi, Gergo via ghc-devs wrote: > >PUBLIC > > > >Looking at this with ghc-debug, at least I can see why we have the huge > memory usage when recreating ModDetails: there are lots of HscEnvs stored > all over the heap (I assume in thunks inside already compiled modules), > when I recreate the ModDetails and replace them in the HUG, I am > accumulating more and more copies of the same MOdDetails, since other > HscEnvs are of course still pointing to the HUG that had the previous > ModDetails, and so on. > > > >Given this, I don't really know yet if remaking the ModDetails would help > me or not, since trying it is now blocking on figuring out how I could > avoid having multiple HscEnvs in memory at the same time... > > > >From: ghc-devs On Behalf Of Gergo Érdi > >Sent: Friday, January 17, 2025 4:08 PM > >To: GHC Devs > >Cc: Montelatici, Raphael Laurent > >Subject: [External] GHC memory usage when typechecking from source vs. > loading ModIfaces > > > >Hi, > > > >I'm using the GHC API to typecheck 35,000 modules that form a complicated > dependency graph (with multiple top-level modules, i.e. there's no single > "god module" that would transitively depend on everything else), and I > noticed that peak memory usage is wildly different when everything is done > from scratch vs. when everything is loaded from files containing ModIfaces: > 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs > 33M for 407 modules. > > > >I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586< > https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/issues/13586__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuRzNhyzs$> > and so when I finish typechecking a module, I take the resulting ModIface > and create the ModDetails that ends up in the HomeUnitGraph from that. My > understanding of Matt's original GHC fix in > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478< > https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuEf-sjms$> > is that it does the same, i.e. it only makes a fresh ModDetails only once > per module, after the ModIface is ready. > > > >But of course that still means that ModDetails can only keep growing as > more and more parts of it are used for typechecking more and more > dependants. Could that be the cause? I tried a crude experiment of "putting > the toothpaste back in the tube" by replacing all ModDetails with a fresh > one in the HUG after each finished typechecking , but that's a complete > disaster for memory usage: even for the small 407 module example, the > memory usage shoots up to 1.5G. I can imagine it's because imported Ids are > probably not shared anymore between different importer modules. > > > >Any ideas on how I could improve memory usage in the from-scratch case, > so that it's more similar to the from-ModIface case? > > > >Thanks, > > Gergo > > > > > >---------------------------------------------------------------------- > >This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries at https: //www.sc.com/en/our-locations > > > >Where you have a Financial Markets relationship with Standard Chartered > PLC, Standard Chartered Bank and their subsidiaries (the "Group"), > information on the regulatory standards we adhere to and how it may affect > you can be found in our Regulatory Compliance Statement at https: // > www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: // > www.sc.com/rcs/fm > > > >Insofar as this communication is not sent by the Global Research team and > contains any market commentary, the market commentary has been prepared by > the sales and/or trading desk of Standard Chartered Bank or its affiliate. > It is not and does not constitute research material, independent research, > recommendation or financial advice. Any market commentary is for > information purpose only and shall not be relied on for any other purpose > and is subject to the relevant disclaimers available at https: // > www.sc.com/en/regulatory-disclosures/#market-disclaimer. > > > >Insofar as this communication is sent by the Global Research team and > contains any research materials prepared by members of the team, the > research material is for information purpose only and shall not be relied > on for any other purpose, and is subject to the relevant disclaimers > available at https: // > research.sc.com/research/api/application/static/terms-and-conditions. > > > >Insofar as this e-mail contains the term sheet for a proposed > transaction, by responding affirmatively to this e-mail, you agree that you > have understood the terms and conditions in the attached term sheet and > evaluated the merits and risks of the transaction. We may at times also > request you to sign the term sheet to acknowledge the same. > > > >Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ > for important information with respect to derivative products. > > > >---------------------------------------------------------------------- > >This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries together with Standard Chartered Bank’s Privacy > Policy via our public website. > > > >---------------------------------------------------------------------- > >This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries together with Standard Chartered Bank’s Privacy > Policy via our main Standard Chartered PLC (UK) website at sc. com > > >_______________________________________________ > >ghc-devs mailing list > >ghc-devs at haskell.org > >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergo at erdi.hu Fri Jan 17 11:26:51 2025 From: gergo at erdi.hu (=?ISO-8859-2?Q?=C9RDI_Gerg=F5?=) Date: Fri, 17 Jan 2025 19:26:51 +0800 (+08) Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: Message-ID: <981b668c-5657-6cd7-7fcb-aec0ae2bbcf8@erdi.hu> On Fri, 17 Jan 2025, Zubin Duggal wrote: > See https://gitlab.haskell.org/ghc/ghc/-/issues/25511 Thank you, reading this ticket and the linked PR, this looks EXACTLY related to what I'm seeing! Unfortunately, I can't just try it out quickly, because my code is currently based on GHC 9.8, but once I've rebased on the linked PR I'll be sure to let you know if it has solved my problem. From gergo at erdi.hu Fri Jan 17 11:30:54 2025 From: gergo at erdi.hu (=?ISO-8859-2?Q?=C9RDI_Gerg=F5?=) Date: Fri, 17 Jan 2025 19:30:54 +0800 (+08) Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References:

Message-ID: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> On Fri, 17 Jan 2025, Matthew Pickering wrote: > 1. As Zubin points out we have recently been concerned with improving the memory usage > of large module sessions (#25511, !13675, !13593) > > I imagine all these patches will greatly help the memory usage in your use case. I'll try these out and report back. > 2. You are absolutely right that ModDetails can get forced and is never reset. > > If you try !13675, it should be much more easily possible to reset the ModDetails by > writing into the IORef which stores each home package. Yes, that makes sense. > 3. If you share your example or perhaps even a trace from ghc-debug then I will be > happy to investigate further as it seems like a great test case for the work we have > recently been doing. Untangling just the parts that exercise the GHC API from all the other in-house bits will be quite a lot of work. But if just a ghc-debug snapshot of e.g. a small example from scratch vs. from existing ModIfaces would be helpful (with e.g. the top HscEnv at the time of finishing all typechecking as a saved closure), I can provide that no prob. Thanks, Gergo From aaronallen8455 at gmail.com Mon Jan 20 00:27:45 2025 From: aaronallen8455 at gmail.com (Aaron Allen) Date: Sun, 19 Jan 2025 18:27:45 -0600 Subject: Google Summer of Code Project Ideas Message-ID: Haskell will be applying for the Google Summer of Code 2025 program and it would be great to have some project ideas involving GHC. If anything comes to mind, please submit an idea proposal using the instructions found here: https://summer.haskell.org/ideas.html. Thanks, Aaron -------------- next part -------------- An HTML attachment was scrubbed... URL: From vladislav at serokell.io Mon Jan 20 15:46:06 2025 From: vladislav at serokell.io (Vladislav Zavialov) Date: Mon, 20 Jan 2025 18:46:06 +0300 Subject: History of DH contributions Message-ID: Dear GHC devs, I’ll be soon publishing a task breakdown for DH (as I see it). In addition to the planned changes, I decided to include the recent history of related work (2018–2024) in the form of links to specific proposals, issues, and commits. Here’s an example of what a single section is going to look like: https://discourse.haskell.org/uploads/default/optimized/2X/4/4954f5fac0a03eba2918901471f047d92496f5e8_2_804x1000.jpeg and there will be about 50 of such sections. However, as the volume of GHC commits and issues is too high for me to go through all of them, I had to limit the scope and focus on my own work. If you have any DH-related contributions that you want represented, please let me know and I’ll try to find a place to include a reference. Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Tue Jan 21 12:24:19 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Tue, 21 Jan 2025 12:24:19 +0000 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> References:

<4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> Message-ID: Thanks Gergo, I think that unless we have access to your code base or a realistic example then the before vs after snapshot will not be so helpful. It's known that `ModDetails` will leak space like this. Let us know how it goes for you. Cheers, Matt On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő wrote: > On Fri, 17 Jan 2025, Matthew Pickering wrote: > > > 1. As Zubin points out we have recently been concerned with improving > the memory usage > > of large module sessions (#25511, !13675, !13593) > > > > I imagine all these patches will greatly help the memory usage in your > use case. > > I'll try these out and report back. > > > 2. You are absolutely right that ModDetails can get forced and is never > reset. > > > > If you try !13675, it should be much more easily possible to reset the > ModDetails by > > writing into the IORef which stores each home package. > > Yes, that makes sense. > > > 3. If you share your example or perhaps even a trace from ghc-debug then > I will be > > happy to investigate further as it seems like a great test case for the > work we have > > recently been doing. > > Untangling just the parts that exercise the GHC API from all the other > in-house bits will be quite a lot of work. But if just a ghc-debug > snapshot of e.g. a small example from scratch vs. from existing ModIfaces > would be helpful (with e.g. the top HscEnv at the time of finishing all > typechecking as a saved closure), I can provide that no prob. > > Thanks, > Gergo > -------------- next part -------------- An HTML attachment was scrubbed... URL: