From Gergo.Erdi at sc.com Wed Feb 5 07:02:18 2025 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Wed, 5 Feb 2025 07:02:18 +0000 Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> Message-ID: PUBLIC Hi Matt, Thanks for your help so far! One vacation later, I am back looking at this. Unfortunately, the latest results I am seeing only confuse me more. I have this small test load of a 4313 module forest that I am typechecking. The baseline resource usage, i.e. before any tricks about rehydrating the ModDetails in the HPT, is 1 GB maximum residency, 113s MUT time and 87s GC time. My aim is to reduce the maximum residency with as little disruption as possible to the total runtime. My first test was the completely brute-force approach of rehydrating every single ModDetails in the HPT after typechecking every single module. Of course, this has catastrophic runtime performance, since I end up re-re-re-re-rehydrating every ModDetails for a total of 8,443,380 times (not counting the initial rehydration just after typechecking to put it in the HPT). So I get 290s MUT time, 252s GC time. But, the max residency goes down to 490 MB, showing that the idea, at least in principle, has legs. So far so good. But then my problem starts -- how do I get this max residency improvement with acceptable runtime? My idea was that when typechecking a module, it should only unfold parts of ModDetails that are its transitive dependencies, so it should be enough to rehydrate only those ModDetails. Since this still results in 3,603,206 rehydrations, I shouldn't be too optimistic about its performance, but it should still cut the overhead in half. When I try this out, I get MUT time of 257s, GC time of 186s. However, the max residency is 883 MB! But how is it possible that max residency is not the same 490 MB?!?! Does that mean typechecking a module can unfold parts of ModDetails that are not transitive dependencies of it? How would I track this down? For reference, here is how I do the rehydration of the HPT, let me know if it seems fishy: ``` recreateModDetailsInHpt :: HscEnv -> [ModuleName] -> IO () recreateModDetailsInHpt hsc_env mods = do hpt <- readIORef hptr fixIO \hpt' -> do writeIORef hptr hpt' traverse recreate_hmi hpt pure () where hpt at HPT{ table = hptr } = hsc_HPT hsc_env recreate_hmi hmi@(HomeModInfo iface _details linkable) | moduleName mod `elem` mods = do !fresh_details <- genModDetails hsc_env iface pure $ HomeModInfo iface fresh_details linkable | otherwise = pure hmi where mod = mi_module iface ``` In summary, my questions going forward are: * How come rehydrating transitive dependencies doesn't help as much for max residency as rehydrating all already-loaded modules? * What exactly does GHC itself do to use this new mutable HPT feature to good effect? I'm sure it doesn't suffer from the above-described quadratic slowdown. Thanks for the tip on the other two memory usage improvement MRs -- I haven't had time yet to backport them. !12582 in particular seems like it will need quite a bit of work to be applied on 9.8. Unfortunately, I couldn't get eventlog2html to work -- if I pass an .hp file with the `-p` parameter, I get an HTML file that claims "This eventlog was generated without heap profiling.". Thanks, Gergo From: Matthew Pickering Sent: Thursday, January 23, 2025 5:51 PM To: Erdi, Gergo Cc: ÉRDI Gergő ; Zubin Duggal ; Montelatici, Raphael Laurent ; GHC Devs Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces That's good news. I don't think the first idea will do very much as there are other references to the final "HomeModInfo" not stored in the HPT. Have you constructed a time profile to determine why the runtime is higher? With the second approach you are certainly trading space usage for repeating work. If you actually do have a forest, then ideally you would replace the ModDetails after it will never be used again. You are likely also missing other patches important for memory usage. * https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12582__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2nUOva2t$ * https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12347__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2kDCIO5S$ I can't comment about the 17 HPT, what do the retainer stacks look like in ghc-debug? PS. Please use eventlog2html so the profiles are readable! You can use it on .hp profiles. Cheers, Matt On Thu, Jan 23, 2025 at 3:19 AM Erdi, Gergo wrote: PUBLIC Hi Matt & Zubin, Thanks for the help on this so far! I managed to hack the linked MR onto 9.8.4 (see https://urldefense.com/v3/__https://gitlab.haskell.org/cactus/ghc/-/tree/cactus/backport-13675__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2mon4aUz$) and basically it seems to do what it says on the tin on a small example (see attached heap profile examples for typechecking 4313 modules), but I am unsure how to actually use it. So my understanding of the improvement here is that since now there is only one single HPT [*], I should be able to avoid unnecessary ballooning by doing two things: • Evicting `HomeModInfo`s wholesale from the HPT that are not going to be needed anymore, because I am done with all modules that would transitively depend on them. This of course only makes sense when typechecking a forest. • Replacing remaining `HomeModInfo`s with new ones that contain the same ModInterface but the ModDetails is replaced with a fresh one from initModDetails. The attached `-after` profile shows typechecking with both of these ideas implemented. The first one doesn’t seem to help much on its own, but it’s tricky to evaluate that because it is very dependent on the shape of the workload (how tree-y it is). But the second one shows some serious promise in curtailing memory usage. However, it is also very slow – even on this small example, you can see its effect. On my full 35k+ module example, it more than doubles the runtime. What would be a good policy on when to replace ModDetails with thunks to avoid both the space leak and excessive rehydration churn? Also, perhaps unrelated, perhaps not – what’s with all those lists?! Thanks, Gergo [*] BTW is it normal that I am still seeing several (17 in a small test case involving a couple hundred modules) HPT constructors in the heap? (I hacked it locally to be a datatype instead of a newtype just so I can see it in the heap). I expected to see only one. From: Matthew Pickering Sent: Tuesday, January 21, 2025 8:24 PM To: ÉRDI Gergő Cc: Zubin Duggal ; Erdi, Gergo ; Montelatici, Raphael Laurent ; GHC Devs Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces Thanks Gergo, I think that unless we have access to your code base or a realistic example then the before vs after snapshot will not be so helpful. It's known that `ModDetails` will leak space like this. Let us know how it goes for you. Cheers, Matt On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő wrote: On Fri, 17 Jan 2025, Matthew Pickering wrote: > 1. As Zubin points out we have recently been concerned with improving the memory usage > of large module sessions (#25511, !13675, !13593) > > I imagine all these patches will greatly help the memory usage in your use case. I'll try these out and report back. > 2. You are absolutely right that ModDetails can get forced and is never reset. > > If you try !13675, it should be much more easily possible to reset the ModDetails by > writing into the IORef which stores each home package. Yes, that makes sense. > 3. If you share your example or perhaps even a trace from ghc-debug then I will be > happy to investigate further as it seems like a great test case for the work we have > recently been doing. Untangling just the parts that exercise the GHC API from all the other in-house bits will be quite a lot of work. But if just a ghc-debug snapshot of e.g. a small example from scratch vs. from existing ModIfaces would be helpful (with e.g. the top HscEnv at the time of finishing all typechecking as a saved closure), I can provide that no prob. Thanks, Gergo ________________________________________ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ________________________________________ This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com From Gergo.Erdi at sc.com Thu Feb 6 02:16:55 2025 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Thu, 6 Feb 2025 02:16:55 +0000 Subject: GHC 9.10 release tags missing from Git Message-ID: PUBLIC Hi, The latest release tag for the ghc-9.10 branch on the Git repo is ghc-9.10.1-release, even though we've had 9.10.2 and 9.10.3 in the meantime. Could someone who knows the exact commits tag them please? Thanks, Gergo ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com From juhpetersen at gmail.com Thu Feb 6 04:27:32 2025 From: juhpetersen at gmail.com (Jens Petersen) Date: Thu, 6 Feb 2025 12:27:32 +0800 Subject: GHC 9.10 release tags missing from Git In-Reply-To: References: Message-ID: On Thu, 6 Feb 2025 at 10:17, Erdi, Gergo via ghc-devs wrote: > The latest release tag for the ghc-9.10 branch on the Git repo is > ghc-9.10.1-release, even though we've had 9.10.2 and 9.10.3 in the > meantime. Could someone who knows the exact commits tag them please? > I think you are confused perhaps? Unfortunately the latest 9.10 release is still 9.10.1. Though, awaiting 9.10.2 with anticipation :-) Jens -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gergo.Erdi at sc.com Thu Feb 6 04:37:09 2025 From: Gergo.Erdi at sc.com (Erdi, Gergo) Date: Thu, 6 Feb 2025 04:37:09 +0000 Subject: GHC 9.10 release tags missing from Git In-Reply-To: References: Message-ID: PUBLIC Indeed. For some reason I thought I saw 9.10.2 and 9.10.3 in `ghcup tui`. I must have hallucinated it :D From: Jens Petersen Sent: Thursday, February 6, 2025 12:28 PM To: Erdi, Gergo Cc: ghc-devs at haskell.org Subject: [External] Re: GHC 9.10 release tags missing from Git On Thu, 6 Feb 2025 at 10:17, Erdi, Gergo via ghc-devs > wrote: The latest release tag for the ghc-9.10 branch on the Git repo is ghc-9.10.1-release, even though we've had 9.10.2 and 9.10.3 in the meantime. Could someone who knows the exact commits tag them please? I think you are confused perhaps? Unfortunately the latest 9.10 release is still 9.10.1. Though, awaiting 9.10.2 with anticipation :-) Jens ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website. ---------------------------------------------------------------------- This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Feb 7 17:08:46 2025 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 07 Feb 2025 12:08:46 -0500 Subject: Docker registry outage Message-ID: <878qqhlm2n.fsf@smart-cactus.org> Hi all, This evening, starting at 1800 EST, I plan on bringing down the Docker registry for an upgrade; I expect that the maintenance will take one or two hours. Docker images will be unable to be pushed or pulled for this duration. While the CI runners generally cache images locally, there is a non-negligible chance that some jobs would fail during this period. To mitigate this, I will keep an eye on the runner pool and pause adversely affected runners during the outage. As always, let me know if you have any questions or concerns about the timing of this maintenance. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1312 bytes Desc: not available URL: From ben at smart-cactus.org Sun Feb 9 12:47:48 2025 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 09 Feb 2025 07:47:48 -0500 Subject: CI instability Message-ID: <875xljl1yg.fsf@smart-cactus.org> Hi all, It appears that there is currently some CI instability arising from Docker registry changes made recently. I am investigating. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 861 bytes Desc: not available URL: From matthewtpickering at gmail.com Mon Feb 10 11:49:40 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Mon, 10 Feb 2025 11:49:40 +0000 Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> Message-ID: Gergo, I wonder if you have got your condition the wrong way around. The only "safe" time to perform rehydration is AFTER the point it can never be used again. If you rehydrate it just before it is used then you will repeat work which has already been done. If you do this, you will always have a trade-off between space used and runtime. PS: The eventlog2html bug with -p rendering is fixed by https://github.com/mpickering/eventlog2html/pull/192 but you should just generate an eventlog anyway with `-l`. Cheers, Matt On Wed, Feb 5, 2025 at 7:03 AM Erdi, Gergo wrote: > PUBLIC > > Hi Matt, > > Thanks for your help so far! > > One vacation later, I am back looking at this. Unfortunately, the latest > results I am seeing only confuse me more. > > I have this small test load of a 4313 module forest that I am > typechecking. The baseline resource usage, i.e. before any tricks about > rehydrating the ModDetails in the HPT, is 1 GB maximum residency, 113s MUT > time and 87s GC time. My aim is to reduce the maximum residency with as > little disruption as possible to the total runtime. > > My first test was the completely brute-force approach of rehydrating every > single ModDetails in the HPT after typechecking every single module. Of > course, this has catastrophic runtime performance, since I end up > re-re-re-re-rehydrating every ModDetails for a total of 8,443,380 times > (not counting the initial rehydration just after typechecking to put it in > the HPT). So I get 290s MUT time, 252s GC time. But, the max residency goes > down to 490 MB, showing that the idea, at least in principle, has legs. > > So far so good. But then my problem starts -- how do I get this max > residency improvement with acceptable runtime? My idea was that when > typechecking a module, it should only unfold parts of ModDetails that are > its transitive dependencies, so it should be enough to rehydrate only those > ModDetails. Since this still results in 3,603,206 rehydrations, I shouldn't > be too optimistic about its performance, but it should still cut the > overhead in half. When I try this out, I get MUT time of 257s, GC time of > 186s. However, the max residency is 883 MB! But how is it possible that max > residency is not the same 490 MB?!?! Does that mean typechecking a module > can unfold parts of ModDetails that are not transitive dependencies of it? > How would I track this down? > > For reference, here is how I do the rehydration of the HPT, let me know if > it seems fishy: > > ``` > recreateModDetailsInHpt :: HscEnv -> [ModuleName] -> IO () > recreateModDetailsInHpt hsc_env mods = do > hpt <- readIORef hptr > fixIO \hpt' -> do > writeIORef hptr hpt' > traverse recreate_hmi hpt > pure () > where > hpt at HPT{ table = hptr } = hsc_HPT hsc_env > > recreate_hmi hmi@(HomeModInfo iface _details linkable) > | moduleName mod `elem` mods > = do > !fresh_details <- genModDetails hsc_env iface > pure $ HomeModInfo iface fresh_details linkable > > | otherwise > = pure hmi > where > mod = mi_module iface > ``` > > In summary, my questions going forward are: > > * How come rehydrating transitive dependencies doesn't help as much for > max residency as rehydrating all already-loaded modules? > > * What exactly does GHC itself do to use this new mutable HPT feature to > good effect? I'm sure it doesn't suffer from the above-described quadratic > slowdown. > > Thanks for the tip on the other two memory usage improvement MRs -- I > haven't had time yet to backport them. !12582 in particular seems like it > will need quite a bit of work to be applied on 9.8. > > Unfortunately, I couldn't get eventlog2html to work -- if I pass an .hp > file with the `-p` parameter, I get an HTML file that claims "This eventlog > was generated without heap profiling.". > > Thanks, > Gergo > > From: Matthew Pickering > Sent: Thursday, January 23, 2025 5:51 PM > To: Erdi, Gergo > Cc: ÉRDI Gergő ; Zubin Duggal ; > Montelatici, Raphael Laurent ; GHC Devs < > ghc-devs at haskell.org> > Subject: [External] Re: GHC memory usage when typechecking from source vs. > loading ModIfaces > > That's good news. > > I don't think the first idea will do very much as there are other > references to the final "HomeModInfo" not stored in the HPT. > > Have you constructed a time profile to determine why the runtime is > higher? With the second approach you are certainly trading space usage for > repeating work. > > If you actually do have a forest, then ideally you would replace the > ModDetails after it will never be used again. > > You are likely also missing other patches important for memory usage. > > * > https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12582__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2nUOva2t$ > * > https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12347__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2kDCIO5S$ > > I can't comment about the 17 HPT, what do the retainer stacks look like in > ghc-debug? > > PS. Please use eventlog2html so the profiles are readable! You can use it > on .hp profiles. > > Cheers, > > Matt > > > > > > > > On Thu, Jan 23, 2025 at 3:19 AM Erdi, Gergo > wrote: > PUBLIC > > Hi Matt & Zubin, > > Thanks for the help on this so far! > > I managed to hack the linked MR onto 9.8.4 (see > https://urldefense.com/v3/__https://gitlab.haskell.org/cactus/ghc/-/tree/cactus/backport-13675__;!!ASp95G87aa5DoyK5mB3l!8j2-zkmKQghR93XL-RPF1V9V1kplxBgAdAb456h8PjDVH7dx9jPdv0xP7GyikMyzP3qbiZPYaJL0ytEl2mon4aUz$) > and basically it seems to do what it says on the tin on a small example > (see attached heap profile examples for typechecking 4313 modules), but I > am unsure how to actually use it. > > So my understanding of the improvement here is that since now there is > only one single HPT [*], I should be able to avoid unnecessary ballooning > by doing two things: > > • Evicting `HomeModInfo`s wholesale from the HPT that are not going to be > needed anymore, because I am done with all modules that would transitively > depend on them. This of course only makes sense when typechecking a forest. > • Replacing remaining `HomeModInfo`s with new ones that contain the same > ModInterface but the ModDetails is replaced with a fresh one from > initModDetails. > > The attached `-after` profile shows typechecking with both of these ideas > implemented. The first one doesn’t seem to help much on its own, but it’s > tricky to evaluate that because it is very dependent on the shape of the > workload (how tree-y it is). But the second one shows some serious promise > in curtailing memory usage. However, it is also very slow – even on this > small example, you can see its effect. On my full 35k+ module example, it > more than doubles the runtime. > > What would be a good policy on when to replace ModDetails with thunks to > avoid both the space leak and excessive rehydration churn? > > Also, perhaps unrelated, perhaps not – what’s with all those lists?! > > Thanks, > Gergo > > [*] BTW is it normal that I am still seeing several (17 in a small test > case involving a couple hundred modules) HPT constructors in the heap? (I > hacked it locally to be a datatype instead of a newtype just so I can see > it in the heap). I expected to see only one. > > From: Matthew Pickering > Sent: Tuesday, January 21, 2025 8:24 PM > To: ÉRDI Gergő > Cc: Zubin Duggal ; Erdi, Gergo Gergo.Erdi at sc.com>; Montelatici, Raphael Laurent Raphael.Montelatici at sc.com>; GHC Devs > Subject: [External] Re: GHC memory usage when typechecking from source vs. > loading ModIfaces > > Thanks Gergo, I think that unless we have access to your code base or a > realistic example then the before vs after snapshot will not be so helpful. > It's known that `ModDetails` will leak space like this. > > Let us know how it goes for you. > > Cheers, > > Matt > > > > On Fri, Jan 17, 2025 at 11:30 AM ÉRDI Gergő wrote: > On Fri, 17 Jan 2025, Matthew Pickering wrote: > > > 1. As Zubin points out we have recently been concerned with improving > the memory usage > > of large module sessions (#25511, !13675, !13593) > > > > I imagine all these patches will greatly help the memory usage in your > use case. > > I'll try these out and report back. > > > 2. You are absolutely right that ModDetails can get forced and is never > reset. > > > > If you try !13675, it should be much more easily possible to reset the > ModDetails by > > writing into the IORef which stores each home package. > > Yes, that makes sense. > > > 3. If you share your example or perhaps even a trace from ghc-debug then > I will be > > happy to investigate further as it seems like a great test case for the > work we have > > recently been doing. > > Untangling just the parts that exercise the GHC API from all the other > in-house bits will be quite a lot of work. But if just a ghc-debug > snapshot of e.g. a small example from scratch vs. from existing ModIfaces > would be helpful (with e.g. the top HscEnv at the time of finishing all > typechecking as a saved closure), I can provide that no prob. > > Thanks, > Gergo > ________________________________________ > This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries together with Standard Chartered Bank’s Privacy > Policy via our public website. > ________________________________________ > This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries together with Standard Chartered Bank’s Privacy > Policy via our main Standard Chartered PLC (UK) website at sc. com > > ---------------------------------------------------------------------- > This email and any attachments are confidential and may also be > privileged. If you are not the intended recipient, please delete all copies > and notify the sender immediately. You may wish to refer to the > incorporation details of Standard Chartered PLC, Standard Chartered Bank > and their subsidiaries together with Standard Chartered Bank’s Privacy > Policy via our main Standard Chartered PLC (UK) website at sc. com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gergo at erdi.hu Mon Feb 10 11:56:34 2025 From: gergo at erdi.hu (=?ISO-8859-2?Q?=C9RDI_Gerg=F5?=) Date: Mon, 10 Feb 2025 19:56:34 +0800 (+08) Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> Message-ID: <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu> On Mon, 10 Feb 2025, Matthew Pickering wrote: > I wonder if you have got your condition the wrong way around.  > > The only "safe" time to perform rehydration is AFTER the point it can never be used > again. > > If you rehydrate it just before it is used then you will repeat work which has already > been done. If you do this, you will always have a trade-off between space used and > runtime. Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails. But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT? From matthewtpickering at gmail.com Mon Feb 10 12:13:24 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Mon, 10 Feb 2025 12:13:24 +0000 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu> References: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu> Message-ID: Sure, you can remove them once you are sure they are not used anymore. For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine. On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő wrote: > On Mon, 10 Feb 2025, Matthew Pickering wrote: > > > I wonder if you have got your condition the wrong way around. > > > > The only "safe" time to perform rehydration is AFTER the point it can > never be used > > again. > > > > If you rehydrate it just before it is used then you will repeat work > which has already > > been done. If you do this, you will always have a trade-off between > space used and > > runtime. > > Oops. Yes, I have misunderstood the idea. I thought the idea was that > after loading a given module into the HPT, its ModDetails would start > out small (because of laziness) and then keep > growing in size as more and more of it are traversed, and thus > forced, during the typechecking of its dependees, so at some point we > would want to reset that into the small initial representation as created > by initModDetails. > > But if the idea is that I should rehydrate modules when they can't be used > anymore, then that brings up the question why even do that, instead of > straight removing the HomeModInfos from the HPT? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Wed Feb 12 11:08:17 2025 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Wed, 12 Feb 2025 11:08:17 +0000 Subject: GHC memory usage when typechecking from source vs. loading ModIfaces In-Reply-To: References: <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu> <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu> Message-ID: You do also raise a good point about rehydration costs. In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well. On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering < matthewtpickering at gmail.com> wrote: > Sure, you can remove them once you are sure they are not used anymore. > > For clients like `GHCi` that doesn't work obviously as they can be used at > any point in the future but for a batch compiler it would be fine. > > On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő wrote: > >> On Mon, 10 Feb 2025, Matthew Pickering wrote: >> >> > I wonder if you have got your condition the wrong way around. >> > >> > The only "safe" time to perform rehydration is AFTER the point it can >> never be used >> > again. >> > >> > If you rehydrate it just before it is used then you will repeat work >> which has already >> > been done. If you do this, you will always have a trade-off between >> space used and >> > runtime. >> >> Oops. Yes, I have misunderstood the idea. I thought the idea was that >> after loading a given module into the HPT, its ModDetails would start >> out small (because of laziness) and then keep >> growing in size as more and more of it are traversed, and thus >> forced, during the typechecking of its dependees, so at some point we >> would want to reset that into the small initial representation as created >> by initModDetails. >> >> But if the idea is that I should rehydrate modules when they can't be >> used >> anymore, then that brings up the question why even do that, instead of >> straight removing the HomeModInfos from the HPT? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stegeman at gmail.com Sun Feb 16 05:45:11 2025 From: stegeman at gmail.com (Luite Stegeman) Date: Sun, 16 Feb 2025 14:45:11 +0900 Subject: GHC 9.6.7 rc2 available Message-ID: The GHC developers are very pleased to announce the availability of the second release candidate of GHC 9.6.7. Binary distributions, source distributions, and documentation are available at https://downloads.haskell.org/ghc/9.6.7-rc2 We hope to have this release available via ghcup shortly. Haskell Language Server does not yet build with GHC 9.6.7. See the note below. If all goes according to plan, we expect that the final release of GHC 9.6.7 will be available on February 28. Release candidate 2 contains the following changes relative to (the unannounced) release candidate 1: - the symbol atomic_inc64 is no longer exported from the RTS GHC 9.6.7 will bring a number of fixes, including: - GHC’s internal Unique type has been widened to 64-bits on 32-bit architectures, avoiding potential miscompilations on large projects. - Fix a runtime crash when using the compacting GC, caused by black holes in large objects. - Added new flags -fspec-eval and -fspec-eval-dictfun to allow switching off speculative evaluation. The following libraries have been updated since GHC 9.6.6: - base-4.18.3.0 (the version number in the rc2 release notes is incorrect) - filepath-1.4.301.0 - unix-2.8.6.0 - bytestring-0.11.5.4 - array-0.5.8.0 A more detailed description can be found in the [release notes] and a full list of changes in [GHC!13841] and [GHC!13891] Note about Haskell Language Server and building GHC 9.8+: The change of Unique to 64 bit [GHC#22010] adds the exported symbol ghc_unique_counter64 to the RTS. Unfortunately it's impossible to avoid this without breaking other things. If you encounter a linker error related to ghc_unique_counter64 when building GHC (or building a GHC-derived package like ghc-lib-parser) with GHC 9.6.7, you probably have to add this [fix] to the program you're building. At the time of writing all released GHC 9.8 versions are affected, as well as the latest version of ghc-lib-parser-9.8 on hackage (which is a dependency of Haskell Language Server). We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release. Please give this release candidate a try and open a [ticket] if you see anything amiss. ~ Luite [release notes]: https://downloads.haskell.org/ghc/9.6.7-rc2/docs/users_guide/9.6.7-notes.html [GHC!13841]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13841 [GHC!13891]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13891 [GHC#22010]: https://gitlab.haskell.org/ghc/ghc/-/issues/22010 [fix] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13743 [ticket] https://gitlab.haskell.org/ghc/ghc/-/issues/new From stegeman at gmail.com Thu Feb 13 22:55:31 2025 From: stegeman at gmail.com (Luite Stegeman) Date: Fri, 14 Feb 2025 07:55:31 +0900 Subject: GHC 9.6.7 rc2 available Message-ID: The GHC developers are very pleased to announce the availability of the second release candidate of GHC 9.6.7. Binary distributions, source distributions, and documentation are available at https://downloads.haskell.org/ghc/9.6.7-rc2 We hope to have this release available via ghcup shortly. Haskell Language Server does not yet build with GHC 9.6.7. See the note below. If all goes according to plan, we expect that the final release of GHC 9.6.7 will be available on February 28. Release candidate 2 contains the following changes relative to (the unannounced) release candidate 1: - the symbol atomic_inc64 is no longer exported from the RTS GHC 9.6.7 will bring a number of fixes, including: - GHC’s internal Unique type has been widened to 64-bits on 32-bit architectures, avoiding potential miscompilations on large projects. - Fix a runtime crash when using the compacting GC, caused by black holes in large objects. - Added new flags -fspec-eval and -fspec-eval-dictfun to allow switching off speculative evaluation. The following libraries have been updated since GHC 9.6.6: - base-4.18.3.0 (the version number in the rc2 release notes is incorrect) - filepath-1.4.301.0 - unix-2.8.6.0 - bytestring-0.11.5.4 - array-0.5.8.0 A more detailed description can be found in the [release notes] and a full list of changes in [GHC!13841] and [GHC!13891] Note about Haskell Language Server and building GHC 9.8+: The change of Unique to 64 bit [GHC#22010] adds the exported symbol ghc_unique_counter64 to the RTS. Unfortunately it's impossible to avoid this without breaking other things. If you encounter a linker error related to ghc_unique_counter64 when building GHC (or building a GHC-derived package like ghc-lib-parser) with GHC 9.6.7, you probably have to add this [fix] to the program you're building. At the time of writing all released GHC 9.8 versions are affected, as well as the latest version of ghc-lib-parser-9.8 on hackage (which is a dependency of Haskell Language Server). We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release. Please give this release candidate a try and open a [ticket] if you see anything amiss. ~ Luite [release notes]: https://downloads.haskell.org/ghc/9.6.7-rc2/docs/users_guide/9.6.7-notes.html [GHC!13841]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13841 [GHC!13891]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13891 [GHC#22010]: https://gitlab.haskell.org/ghc/ghc/-/issues/22010 [fix] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13743 [ticket] https://gitlab.haskell.org/ghc/ghc/-/issues/new From stegeman at gmail.com Sat Feb 15 14:27:14 2025 From: stegeman at gmail.com (Luite Stegeman) Date: Sat, 15 Feb 2025 23:27:14 +0900 Subject: GHC 9.6.7 rc2 available Message-ID: The GHC developers are very pleased to announce the availability of the second release candidate of GHC 9.6.7. Binary distributions, source distributions, and documentation are available at https://downloads.haskell.org/ghc/9.6.7-rc2 We hope to have this release available via ghcup shortly. Haskell Language Server does not yet build with GHC 9.6.7. See the note below. If all goes according to plan, we expect that the final release of GHC 9.6.7 will be available on February 28. Release candidate 2 contains the following changes relative to (the unannounced) release candidate 1: - the symbol atomic_inc64 is no longer exported from the RTS GHC 9.6.7 will bring a number of fixes, including: - GHC’s internal Unique type has been widened to 64-bits on 32-bit architectures, avoiding potential miscompilations on large projects. - Fix a runtime crash when using the compacting GC, caused by black holes in large objects. - Added new flags -fspec-eval and -fspec-eval-dictfun to allow switching off speculative evaluation. The following libraries have been updated since GHC 9.6.6: - base-4.18.3.0 (the version number in the rc2 release notes is incorrect) - filepath-1.4.301.0 - unix-2.8.6.0 - bytestring-0.11.5.4 - array-0.5.8.0 A more detailed description can be found in the [release notes] and a full list of changes in [GHC!13841] and [GHC!13891] Note about Haskell Language Server and building GHC 9.8+: The change of Unique to 64 bit [GHC#22010] adds the exported symbol ghc_unique_counter64 to the RTS. Unfortunately it's impossible to avoid this without breaking other things. If you encounter a linker error related to ghc_unique_counter64 when building GHC (or building a GHC-derived package like ghc-lib-parser) with GHC 9.6.7, you probably have to add this [fix] to the program you're building. At the time of writing all released GHC 9.8 versions are affected, as well as the latest version of ghc-lib-parser-9.8 on hackage (which is a dependency of Haskell Language Server). We would like to thank GitHub, IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O, Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous contributors whose on-going financial and in-kind support has facilitated GHC maintenance and release management over the years. Finally, this release would not have been possible without the hundreds of open-source contributors whose work comprise this release. Please give this release candidate a try and open a [ticket] if you see anything amiss. ~ Luite [release notes]: https://downloads.haskell.org/ghc/9.6.7-rc2/docs/users_guide/9.6.7-notes.html [GHC!13841]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13841 [GHC!13891]: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13891 [GHC#22010]: https://gitlab.haskell.org/ghc/ghc/-/issues/22010 [fix] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13743 [ticket] https://gitlab.haskell.org/ghc/ghc/-/issues/new P.S. Apologies if you already received this mail. From facundo.dominguez at tweag.io Fri Feb 21 19:41:29 2025 From: facundo.dominguez at tweag.io (=?UTF-8?Q?Facundo_Dom=C3=ADnguez?=) Date: Fri, 21 Feb 2025 16:41:29 -0300 Subject: GHC API stability: draft report of actions Message-ID: Dear GHC devs, As part of the initiative to stabilize the GHC API [1], I've written a draft report [2] proposing actions to address some of the collected feedback. Some of these actions can potentially affect the processes used to maintain GHC, so your input is very much welcome to assess these or to propose other actions. [1] https://discourse.haskell.org/t/ghc-api-stability-update-3/11407 [2] https://docs.google.com/document/d/1w88bz03bZ9J7BP1WDxdpUTLqGrV4C9MAM1C06dDJZAU/edit Best, Facundo -- All views and opinions expressed in this email message are the personal opinions of the author and do not represent those of the organization or its customers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreash87 at gmx.ch Fri Feb 21 15:54:58 2025 From: andreash87 at gmx.ch (Andreas Herrmann) Date: Fri, 21 Feb 2025 16:54:58 +0100 Subject: CFP - Haskell Implementors' Workshop 2025 Message-ID: Call for proposals for the Haskell Implementors' Workshop https://haskell.foundation/events/2025-haskell-implementors-workshop.html June 6, 2025 Organized by the Haskell Community Co-located with ZuriHac 2025 and Haskell Ecosystem Workshop 2025 Hosted by the Haskell Foundation at Eastern Switzerland University of Applied Sciences (OST) https://www.ost.ch/en/university-of-applied-sciences/campus/rapperswil-jona-campus ## Overview * Deadline: April 4, 2025 * Notification: May 5, 2025 * Workshop: June 6, 2025 The 17th Haskell Implementors' Workshop is to be held alongside ZuriHac 2025 this year near Zurich. It is a forum for people involved in the design and development of Haskell implementations, tools, libraries, and supporting infrastructure to share their work and to discuss future directions and collaborations with others. Talks and/or demos are proposed by submitting an abstract, and selected by a small program committee. There will be no published proceedings. The workshop will be informal and interactive, with open spaces in the timetable and room for ad-hoc discussion, demos, and lightning talks. In the past the Haskell Implementors’ Workshop was co-located with ICFP (International Conference on Functional Programming). However, in recent years it has become more and more challenging to attract a large enough audience and sufficiently many speakers for an appealing program. ZuriHac and the Haskell Ecosystem Workshop have become an important annual gathering of a large part of the Haskell community. This year the Haskell Implementors’ Workshop will be co-located with these events to be accessible to a broader audience. ## Scope and Target Audience The Haskell Implementors' Workshop is an ideal place to describe a Haskell extension, describe works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. Members of the wider Haskell community are encouraged to attend the workshop - we need your feedback to keep the Haskell ecosystem thriving. Students working with Haskell are especially encouraged to share their work. The scope covers any of the following topics. There may be some topics that people feel we've missed, so by all means submit a proposal even if it doesn't fit exactly into one of these buckets: * Compilation techniques * Language features and extensions * Type system implementation * Concurrency and parallelism: language design and implementation * Performance, optimization and benchmarking * Virtual machines and run-time systems * Libraries and tools for development or deployment ## Talks We invite proposals from potential speakers for talks and demonstrations. We are aiming for 20-minute talks with 5 minutes for questions and changeovers. We want to hear from people writing compilers, tools, or libraries, people with cool ideas for directions in which we should take the platform, proposals for new features to be implemented, and half-baked crazy ideas. Submissions can be made via the form linked below until April 4, 2025 (anywhere on earth). https://docs.google.com/forms/d/e/1FAIpQLSdczGbxJYGc4eusvPrxwBbZl561PnKeYnoZ2hYsdw_ZpSfupQ/viewform?usp=header We will also have a lightning talks session. Lightning talks should be ~7mins and are scheduled on the day of the workshop. Suggested topics for lightning talks are to present a single idea, a work-in-progress project, a problem to intrigue and perplex Haskell implementors, or simply to ask for feedback and collaborators. ## Program Committee * Luite Stegeman * Jaro Reinders * Emily Pillmore * Rodrigo Mesquita * Ian-Woo Kim * Andreas Herrmann (chair) ## Contact * Andreas Herrmann -------------- next part -------------- An HTML attachment was scrubbed... URL: