From Gergo.Erdi at sc.com  Tue Apr  1 04:51:17 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Tue, 1 Apr 2025 04:51:17 +0000
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
Message-ID: <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.

Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric?

From: Matthew Pickering <matthewtpickering at gmail.com>
Sent: Saturday, March 29, 2025 1:53 AM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces


Hi Gergo,

I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs.

It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?

Cheers,

Matt

On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------


----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/96d935c7/attachment.html>

From Gergo.Erdi at sc.com  Tue Apr  1 06:04:46 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Tue, 1 Apr 2025 06:04:46 +0000
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
 <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <SEYPR03MB7463B4D13EF9C71F4A36DDD4E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

OK scratch that, I was looking at wrong ghc-debug output. Indeed there are 2301 ModuleGraphs in the heap at the end of typechecking :O

From: Erdi, Gergo
Sent: Tuesday, April 1, 2025 12:51 PM
To: Matthew Pickering <matthewtpickering at gmail.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: Re: GHC memory usage when typechecking from source vs. loading ModIfaces

This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.

Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric?

From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Saturday, March 29, 2025 1:53 AM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces


Hi Gergo,

I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs.

It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?

Cheers,

Matt

On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/be3a1ef4/attachment-0001.html>

From matthewtpickering at gmail.com  Tue Apr  1 09:44:16 2025
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Tue, 1 Apr 2025 10:44:16 +0100
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <SEYPR03MB7463B4D13EF9C71F4A36DDD4E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
 <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463B4D13EF9C71F4A36DDD4E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <CALuQ0m9ZJ8z5T2PoANvw2xcy2V1X2e-OO_LRoB8kjYFSmSWA3g@mail.gmail.com>

Hi Gergo,

I looked in the detailed pane, searched for ModuleGraph, hovered my mouse
over the "ModuleGraph` constructor, recorded the number of live bytes,
divided that by 32.

Matt

On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:

> PUBLIC
>
> OK scratch that, I was looking at wrong ghc-debug output. Indeed there are
> 2301 ModuleGraphs in the heap at the end of typechecking :O
>
>
>
> *From:* Erdi, Gergo
> *Sent:* Tuesday, April 1, 2025 12:51 PM
> *To:* Matthew Pickering <matthewtpickering at gmail.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* Re: GHC memory usage when typechecking from source vs. loading
> ModIfaces
>
>
>
> This sounds extremely interesting, but I don’t understand where you are
> getting this number from! How do you see in the eventlog HTMLs that I’ve
> included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug
> to find all ModuleGraph constructors at two points in the run: just before
> typechecking the first module (after all the extendMG calls) and just after
> typechecking the last module, and even in the cold case I only see 1
> ModuleGraph before and 13 ModuleGraphs after.
>
>
>
> Also, what do you mean by “precisely one loaded per interface loaded into
> the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294
> ModuleGraphs by that metric?
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Saturday, March 29, 2025 1:53 AM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
>
>
> Hi Gergo,
>
>
>
> I quickly tried building `Cabal` with the master branch. There is
> precisely 1 ModuleGraph allocated for the home session, and precisely one
> loaded per interface loaded into the EPS. No leaky behaviour like you can
> see in your eventlogs.
>
>
>
> It seems there are about 2000 live module graphs in your program, are you
> doing something with the API to create this many?
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <
> matthewtpickering at gmail.com> wrote:
>
> HI Gergo,
>
>
>
> Do you have a (synthetic?) reproducer? You have probably identified some
> memory leak. However, without any means to reproduce it becomes very
> difficult to investigate. I feel like we are getting into very precise
> details now, where speculating is not going to be so useful.
>
>
>
> It seems like this is an important thing for you and your company. Is
> there any budget to pay for some investigation? If that was the case then
> some effort could be made to create a synthetic producer and make the
> situation more robust going into the future if your requirements were
> precisely understood.
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
> Just to add that I get the same "equalizing" behaviour (but in a more
> "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just
> call `hugInstancesBelow` before processing each module. So that's
> definitely one source of extra memory usage. I wonder if it would be
> possible to rebuild the ModuleGraph periodically (similar to the ModDetails
> dehydration), or if there are references to it stored all over the place
> from `HscEnv`s scattered around in closures etc. (basically the same
> problem the HPT had before it was made into a mutable reference).
>
> -----Original Message-----
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Erdi, Gergo
> via ghc-devs
> Sent: Friday, March 28, 2025 4:49 PM
> To: Matthew Pickering <matthewtpickering at gmail.com>; GHC Devs <
> ghc-devs at haskell.org>
> Cc: ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <
> Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> Hi,
>
> Unfortunately, I am forced to return to this problem. Everything below is
> now in the context of GHC 9.12 plus the mutable HPT patch backported.
>
> My test case is typechecking a tree of 2294 modules that form the
> transitive closure of a single module's dependencies, all in a single
> process. I have done this typechecking three times, here's what `+RTS -s
> -RTS` reports for max residency:
>
> * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB
>
> * "cold-top": With all `ModIface`s already on disk, except for the
>   single top-level module: 302 MB
>
> * "warm": With all `ModIface`s already on disk: 211 MB
>
> So my stupidly naive question is, why is the "cold" case also not 302 MB?
>
> In earlier discussion, `ModDetails` unfolding has come up. Dehydrating
> `ModDetails` in the HPT all the time is disastrous for runtime, but based
> on this model I would expect to see improvements from dehydrating "every
> now and then". So I tried a stupid simple example where after every 100th
> typechecked module, I run this function on the topologically sorted list of
> modules processed so far:
>
>
> ```
> dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods
> = do
>     let HPT{ table = hptr } = hsc_HPT hsc_env
>     hpt <- readIORef hptr
>     for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface
> _details _linkable) -> do
>         !details <- initModDetails hsc_env iface
>         pure ()
> ```
>
> Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact,
> the profile looks exactly the same.
>
> Speaking of the profile, in the "cold" case I see a lot of steadily
> increasing heap usage from the `ModuleGraph`. I could see this happening if
> typechecking from scratch involves more `modulesUnder` calls which in turn
> force more and more of the `ModuleGraph`. If so, then maybe this could be
> worked around by repeatedly remaking the `ModuleGraph` just like I remake
> the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing
> the `ModuleGraph` at the start, with the idea being that this should
> "equalize" the three scenarios if this really is a substantial source of
> extra memory usage. This pushes up the warm case's memory usage to 381 MB,
> which is promising, but I still see a `Word64Map` that is steadily
> increasing in the "cold-force-modulegraph" case and contributes a lot to
> the memory usage. Unfortunately, I don't know where that `Word64Map` is (it
> could be any `Unique`-keyed environment...).
>
> So I am now stuck at this point. To spell out my goal explicitly, I would
> like to typecheck one module after another and not keep anything more in
> memory around than if I loaded them from `ModIface` files.
>
> Thanks,
>         Gergo
>
> p.s.: I couldn't find a way in the EventLog output HTML to turn event
> markers on/off or filter them, so to avoid covering the whole graph with
> gray lines, I mark only every 100th module.
>
>
>
>
> From: Matthew Pickering <matthewtpickering at gmail.com>
> Sent: Wednesday, February 12, 2025 7:08 PM
> To: ÉRDI Gergő <gergo at erdi.hu>
> Cc: Erdi, Gergo <Gergo.Erdi at sc.com>; Zubin Duggal <zubin at well-typed.com>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; GHC Devs <
> ghc-devs at haskell.org>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> You do also raise a good point about rehydration costs.
>
> In oneshot mode, you are basically rehydrating the entire transitive
> closure of each module when you compile it, which obviously results in a
> large amount of repeated work. This is why people are investigating ideas
> of a persistent worker to at least avoid rehydrating all external
> dependencies as well.
>
> On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:
> matthewtpickering at gmail.com> wrote:
> Sure, you can remove them once you are sure they are not used anymore.
>
> For clients like `GHCi` that doesn't work obviously as they can be used at
> any point in the future but for a batch compiler it would be fine.
>
> On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu> wrote:
> On Mon, 10 Feb 2025, Matthew Pickering wrote:
>
> > I wonder if you have got your condition the wrong way around.
> >
> > The only "safe" time to perform rehydration is AFTER the point it can
> > never be used again.
> >
> > If you rehydrate it just before it is used then you will repeat work
> > which has already been done. If you do this, you will always have a
> > trade-off between space used and runtime.
>
> Oops. Yes, I have misunderstood the idea. I thought the idea was that
> after loading a given module into the HPT, its ModDetails would start out
> small (because of laziness) and then keep growing in size as more and more
> of it are traversed, and thus forced, during the typechecking of its
> dependees, so at some point we would want to reset that into the small
> initial representation as created by initModDetails.
>
> But if the idea is that I should rehydrate modules when they can't be used
> anymore, then that brings up the question why even do that, instead of
> straight removing the HomeModInfos from the HPT?
>
> ----------------------------------------------------------------------
>
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/f6a1c04f/attachment.html>

From Gergo.Erdi at sc.com  Tue Apr  1 12:24:14 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Tue, 1 Apr 2025 12:24:14 +0000
Subject: [External] Re: GHC memory usage when typechecking from source vs.
 loading ModIfaces
In-Reply-To: <CALuQ0m9ZJ8z5T2PoANvw2xcy2V1X2e-OO_LRoB8kjYFSmSWA3g@mail.gmail.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
 <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463B4D13EF9C71F4A36DDD4E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9ZJ8z5T2PoANvw2xcy2V1X2e-OO_LRoB8kjYFSmSWA3g@mail.gmail.com>
Message-ID: <SEYPR03MB7463CDCAD948DE28ED6B9C51E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

OK so this `ModuleGraph` leak has the potential to be a huge explainer because of the large number of ModuleNodes and NodeKey_Modules in the heap profile. It turns out the reason we had one ModuleGraph per module was because I was populating the ModuleGraph by repeated calls to `extendMG`, where the new edges were created via `mgModSummaries` on the old `ModuleGraph`, and so the new edge list was a thunk pointing to the old `ModuleGraph`.

I’ve now changed it to a single `mkModuleGraph` call, which gets rid of the extra ModuleGraphs (see attached profile), but unfortunately, the ModuleNodes/ edge NodeKey_Modules are still not shared. Here’s how I build the graph, trying hard with a Data.Map.Map to ensure only use one NodeKey_Module per each target module:

```
registerModulesToGraph :: GhcMonad m => [(UnitId, ModuleName, [ModuleName])] -> m ()
registerModulesToGraph entries = modifySessionM \hsc_env -> do
    mod_deps <- for entries \(unit, mod_name, deps) -> do
        mod <- liftIO $ registerToModuleFinder hsc_env unit mod_name
        let ms = dummyModSummary (hsc_dflags hsc_env) mod deps fingerprint0
        pure (ms, deps)
    let nodes = [ModuleNode (edges deps) ms | (ms, deps) <- mod_deps]
    pure hsc_env{ hsc_mod_graph = mkModuleGraph nodes }
  where
    edge_map :: Map.Map ModuleName NodeKey
    edge_map = Map.fromList [(mod_name, NodeKey_Module $ ModNodeKeyWithUid (notBoot mod_name) uid) | (uid, mod_name, _) <- entries]

    edges deps = map (edge_map Map.!) deps

registerToModuleFinder :: HscEnv -> UnitId -> ModuleName -> IO Module
registerToModuleFinder hsc_env unitId modName = do
    addHomeModuleToFinder (hsc_FC hsc_env) homeUnit (notBoot modName) location
  where
    homeUnit = DefiniteHomeUnit unitId Nothing
    location = dummyModLocation modName
```

But, as you can see in the attached profile, I still get way more than 2k ModuleNodes and NodeKey_Modules….

From: Matthew Pickering <matthewtpickering at gmail.com>
Sent: Tuesday, April 1, 2025 5:44 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi Gergo,

I looked in the detailed pane, searched for ModuleGraph, hovered my mouse over the "ModuleGraph` constructor, recorded the number of live bytes, divided that by 32.

Matt

On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:

PUBLIC

OK scratch that, I was looking at wrong ghc-debug output. Indeed there are 2301 ModuleGraphs in the heap at the end of typechecking :O

From: Erdi, Gergo
Sent: Tuesday, April 1, 2025 12:51 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: Re: GHC memory usage when typechecking from source vs. loading ModIfaces

This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.

Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294 ModuleGraphs by that metric?

From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Saturday, March 29, 2025 1:53 AM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces


Hi Gergo,

I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the EPS. No leaky behaviour like you can see in your eventlogs.

It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?

Cheers,

Matt

On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/c3387938/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cold-no-modulegraph-leak.zip
Type: application/x-zip-compressed
Size: 1337463 bytes
Desc: cold-no-modulegraph-leak.zip
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/c3387938/attachment-0001.bin>

From george.colpitts at gmail.com  Tue Apr  1 14:48:34 2025
From: george.colpitts at gmail.com (George Colpitts)
Date: Tue, 1 Apr 2025 11:48:34 -0300
Subject: can't move to latest version of llvm on ghc 9.12.2
Message-ID: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>

llvm 20 is out but unlike in earlier versions of ghc moving to it means you
can no longer use llvm:

compiling:

ghc -fllvm hello.hs
Loaded package environment from
/Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
[1 of 2] Compiling Main             ( hello.hs, hello.o )
<no location info>: error: [GHC-66599]
    GHC was not configured with a supported LLVM toolchain
    Make sure you have installed LLVM between [13 and 20) and reinstall GHC
to make -fllvm work


from configure:

configure: We only support llvm 13 upto 20 (non-inclusive) (found 20.1.1).


Can we move to llvm 20 on HEAD and can we revert to the old behavior on ghc
9.12.3?

Should I file an ER?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/53092005/attachment.html>

From allbery.b at gmail.com  Tue Apr  1 16:27:41 2025
From: allbery.b at gmail.com (Brandon Allbery)
Date: Tue, 1 Apr 2025 12:27:41 -0400
Subject: can't move to latest version of llvm on ghc 9.12.2
In-Reply-To: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
References: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
Message-ID: <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>

Have you demonstrated that it works? IIRC the current behavior is because
some LLVM version (16, IIRC) didn't work with GHC (threw errors from opt, I
think).

On Tue, Apr 1, 2025 at 10:49 AM George Colpitts <george.colpitts at gmail.com>
wrote:

> llvm 20 is out but unlike in earlier versions of ghc moving to it means
> you can no longer use llvm:
>
> compiling:
>
> ghc -fllvm hello.hs
> Loaded package environment from
> /Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
> [1 of 2] Compiling Main             ( hello.hs, hello.o )
> <no location info>: error: [GHC-66599]
>     GHC was not configured with a supported LLVM toolchain
>     Make sure you have installed LLVM between [13 and 20) and reinstall
> GHC to make -fllvm work
>
>
> from configure:
>
> configure: We only support llvm 13 upto 20 (non-inclusive) (found 20.1.1).
>
>
> Can we move to llvm 20 on HEAD and can we revert to the old behavior on
> ghc 9.12.3?
>
> Should I file an ER?
>
> Thanks
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>


-- 
brandon s allbery kf8nh
allbery.b at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/8d947be9/attachment.html>

From george.colpitts at gmail.com  Tue Apr  1 17:05:59 2025
From: george.colpitts at gmail.com (George Colpitts)
Date: Tue, 1 Apr 2025 14:05:59 -0300
Subject: can't move to latest version of llvm on ghc 9.12.2
In-Reply-To: <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>
References: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
 <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>
Message-ID: <CAB-d4A4w-ghx4hCmizaw0Ebdye2i0ohe+KbP06XuQJUx0vpn3Q@mail.gmail.com>

I was trying to find out if it works but I ran into the problem I
described. Yes, there were a few years where the latest version didn't work
but in many years it did. ghc used to give a warning, this is an
unsupported version but we'll try it anyways. I would like to return to
that.

In any case I believe ghc dev should test to see if it works and if so use
it in HEAD. If it doesn't work than we should file a bug and fix it. In the
past we got a few versions behind llvm. I think we want to be on the latest
version available for each new release if possible. The earlier we look
into this the more chance we have to succeed at that.

On Tue, Apr 1, 2025 at 1:27 PM Brandon Allbery <allbery.b at gmail.com> wrote:

> Have you demonstrated that it works? IIRC the current behavior is because
> some LLVM version (16, IIRC) didn't work with GHC (threw errors from opt, I
> think).
>
> On Tue, Apr 1, 2025 at 10:49 AM George Colpitts <george.colpitts at gmail.com>
> wrote:
>
>> llvm 20 is out but unlike in earlier versions of ghc moving to it means
>> you can no longer use llvm:
>>
>> compiling:
>>
>> ghc -fllvm hello.hs
>> Loaded package environment from
>> /Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
>> [1 of 2] Compiling Main             ( hello.hs, hello.o )
>> <no location info>: error: [GHC-66599]
>>     GHC was not configured with a supported LLVM toolchain
>>     Make sure you have installed LLVM between [13 and 20) and reinstall
>> GHC to make -fllvm work
>>
>>
>> from configure:
>>
>> configure: We only support llvm 13 upto 20 (non-inclusive) (found 20.1.1).
>>
>>
>> Can we move to llvm 20 on HEAD and can we revert to the old behavior on
>> ghc 9.12.3?
>>
>> Should I file an ER?
>>
>> Thanks
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
>
> --
> brandon s allbery kf8nh
> allbery.b at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/125e4d5a/attachment.html>

From klebinger.andreas at gmx.at  Tue Apr  1 17:11:24 2025
From: klebinger.andreas at gmx.at (Andreas Klebinger)
Date: Tue, 1 Apr 2025 19:11:24 +0200
Subject: can't move to latest version of llvm on ghc 9.12.2
In-Reply-To: <CAB-d4A4w-ghx4hCmizaw0Ebdye2i0ohe+KbP06XuQJUx0vpn3Q@mail.gmail.com>
References: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
 <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>
 <CAB-d4A4w-ghx4hCmizaw0Ebdye2i0ohe+KbP06XuQJUx0vpn3Q@mail.gmail.com>
Message-ID: <4db7eafd-69b0-476c-bffd-e2a8827d7346@gmx.at>

Looking at https://gitlab.haskell.org/ghc/ghc/-/issues/25011 it seems
the desire was to fail with a proper error when LLVM was not found at all.

To me failing when the llvm version is too new seems like a unintended
side effect of fixing the former.

I agree that simply warning and trying anway is the better default. A
ticket would be appreciated!

Am 01/04/2025 um 19:05 schrieb George Colpitts:
> I was trying to find out if it works but I ran into the problem I
> described. Yes, there were a few years where the latest version didn't
> work but in many years it did. ghc used to give a warning, this is an
> unsupported version but we'll try it anyways. I would like to
> return to that.
>
> In any case I believe ghc dev should test to see if it works and if so
> use it in HEAD. If it doesn't work than we should file a bug and fix
> it. In the past we got a few versions behind llvm. I think we want to
> be on the latest version available for each new release if possible.
> The earlier we look into this the more chance we have to succeed at that.
>
> On Tue, Apr 1, 2025 at 1:27 PM Brandon Allbery <allbery.b at gmail.com>
> wrote:
>
>     Have you demonstrated that it works? IIRC the current behavior is
>     because some LLVM version (16, IIRC) didn't work with GHC (threw
>     errors from opt, I think).
>
>     On Tue, Apr 1, 2025 at 10:49 AM George Colpitts
>     <george.colpitts at gmail.com> wrote:
>
>         llvm 20 is out but unlike in earlier versions of ghc moving to
>         it means you can no longer use llvm:
>
>         compiling:
>
>             ghc -fllvm hello.hs
>             Loaded package environment from
>             /Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
>             [1 of 2] Compiling Main             ( hello.hs, hello.o )
>             <no location info>: error: [GHC-66599]
>               GHC was not configured with a supported LLVM toolchain
>               Make sure you have installed LLVM between [13 and 20)
>             and reinstall GHC to make -fllvm work
>
>
>         from configure:
>
>             configure: We only support llvm 13 upto 20 (non-inclusive)
>             (found 20.1.1).
>
>
>         Can we move to llvm 20 on HEAD and can we revert to the old
>         behavior on ghc 9.12.3?
>
>         Should I file an ER?
>
>         Thanks
>
>         _______________________________________________
>         ghc-devs mailing list
>         ghc-devs at haskell.org
>         http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
>
>     --
>     brandon s allbery kf8nh
>     allbery.b at gmail.com
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/e8b4524b/attachment.html>

From george.colpitts at gmail.com  Tue Apr  1 20:15:46 2025
From: george.colpitts at gmail.com (George Colpitts)
Date: Tue, 1 Apr 2025 17:15:46 -0300
Subject: can't move to latest version of llvm on ghc 9.12.2
In-Reply-To: <4db7eafd-69b0-476c-bffd-e2a8827d7346@gmx.at>
References: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
 <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>
 <CAB-d4A4w-ghx4hCmizaw0Ebdye2i0ohe+KbP06XuQJUx0vpn3Q@mail.gmail.com>
 <4db7eafd-69b0-476c-bffd-e2a8827d7346@gmx.at>
Message-ID: <CAB-d4A4dLMUocv2ZWp_7yPns0WbRZpHicmfkcn+za9ADC4YK4g@mail.gmail.com>

Thanks, I'll file a ticket today or tomorrow.

On Tue, Apr 1, 2025 at 2:11 PM Andreas Klebinger via ghc-devs <
ghc-devs at haskell.org> wrote:

> Looking at https://gitlab.haskell.org/ghc/ghc/-/issues/25011 it seems the
> desire was to fail with a proper error when LLVM was not found at all.
>
> To me failing when the llvm version is too new seems like a unintended
> side effect of fixing the former.
>
> I agree that simply warning and trying anway is the better default. A
> ticket would be appreciated!
> Am 01/04/2025 um 19:05 schrieb George Colpitts:
>
> I was trying to find out if it works but I ran into the problem I
> described. Yes, there were a few years where the latest version didn't work
> but in many years it did. ghc used to give a warning, this is an
> unsupported version but we'll try it anyways. I would like to return to
> that.
>
> In any case I believe ghc dev should test to see if it works and if so use
> it in HEAD. If it doesn't work than we should file a bug and fix it. In the
> past we got a few versions behind llvm. I think we want to be on the latest
> version available for each new release if possible. The earlier we look
> into this the more chance we have to succeed at that.
>
> On Tue, Apr 1, 2025 at 1:27 PM Brandon Allbery <allbery.b at gmail.com>
> wrote:
>
>> Have you demonstrated that it works? IIRC the current behavior is because
>> some LLVM version (16, IIRC) didn't work with GHC (threw errors from opt, I
>> think).
>>
>> On Tue, Apr 1, 2025 at 10:49 AM George Colpitts <
>> george.colpitts at gmail.com> wrote:
>>
>>> llvm 20 is out but unlike in earlier versions of ghc moving to it means
>>> you can no longer use llvm:
>>>
>>> compiling:
>>>
>>> ghc -fllvm hello.hs
>>> Loaded package environment from
>>> /Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
>>> [1 of 2] Compiling Main             ( hello.hs, hello.o )
>>> <no location info>: error: [GHC-66599]
>>>     GHC was not configured with a supported LLVM toolchain
>>>     Make sure you have installed LLVM between [13 and 20) and reinstall
>>> GHC to make -fllvm work
>>>
>>>
>>> from configure:
>>>
>>> configure: We only support llvm 13 upto 20 (non-inclusive) (found
>>> 20.1.1).
>>>
>>>
>>> Can we move to llvm 20 on HEAD and can we revert to the old behavior on
>>> ghc 9.12.3?
>>>
>>> Should I file an ER?
>>>
>>> Thanks
>>>
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>
>>
>> --
>> brandon s allbery kf8nh
>> allbery.b at gmail.com
>>
>
> _______________________________________________
> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250401/5753cb1a/attachment-0001.html>

From matthewtpickering at gmail.com  Wed Apr  2 09:14:44 2025
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Wed, 2 Apr 2025 10:14:44 +0100
Subject: [External] Re: GHC memory usage when typechecking from source vs.
 loading ModIfaces
In-Reply-To: <SEYPR03MB7463CDCAD948DE28ED6B9C51E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <CALuQ0m9Cj3dqaOdxUxn_0Vz3rOChzBGu5xpZMVV=N8gd2cZMsQ@mail.gmail.com>
 <SEYPR03MB74632ECD48E921CA918584E1E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463B4D13EF9C71F4A36DDD4E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9ZJ8z5T2PoANvw2xcy2V1X2e-OO_LRoB8kjYFSmSWA3g@mail.gmail.com>
 <SEYPR03MB7463CDCAD948DE28ED6B9C51E9AC2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <CALuQ0m9xD=rNk6v7ZpmkmQ+c3Qh+dpazHq6p3Q-WDKS9tfYJkQ@mail.gmail.com>

Hi Gergo,

Since this is a non-reproducible issue with your custom code, I think that
we've reached the limit of free debugging via email. There are several
consultancies which can be employed to investigate complicated issues like
this and perform the necessary changes to GHC to make your use-case easier.

If you have a reproducer on a distributed GHC release then it could be
investigated.

Cheers,

Matt

On Tue, Apr 1, 2025 at 1:24 PM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:

> PUBLIC
>
> OK so this `ModuleGraph` leak has the potential to be a huge explainer
> because of the large number of ModuleNodes and NodeKey_Modules in the heap
> profile. It turns out the reason we had one ModuleGraph per module was
> because I was populating the ModuleGraph by repeated calls to `extendMG`,
> where the new edges were created via `mgModSummaries` on the old
> `ModuleGraph`, and so the new edge list was a thunk pointing to the old
> `ModuleGraph`.
>
>
>
> I’ve now changed it to a single `mkModuleGraph` call, which gets rid of
> the extra ModuleGraphs (see attached profile), but unfortunately, the
> ModuleNodes/ edge NodeKey_Modules are still not shared. Here’s how I build
> the graph, trying hard with a Data.Map.Map to ensure only use one
> NodeKey_Module per each target module:
>
>
>
> ```
>
> registerModulesToGraph :: GhcMonad m => [(UnitId, ModuleName,
> [ModuleName])] -> m ()
>
> registerModulesToGraph entries = modifySessionM \hsc_env -> do
>
>     mod_deps <- for entries \(unit, mod_name, deps) -> do
>
>         mod <- liftIO $ registerToModuleFinder hsc_env unit mod_name
>
>         let ms = dummyModSummary (hsc_dflags hsc_env) mod deps fingerprint0
>
>         pure (ms, deps)
>
>     let nodes = [ModuleNode (edges deps) ms | (ms, deps) <- mod_deps]
>
>     pure hsc_env{ hsc_mod_graph = mkModuleGraph nodes }
>
>   where
>
>     edge_map :: Map.Map ModuleName NodeKey
>
>     edge_map = Map.fromList [(mod_name, NodeKey_Module $ ModNodeKeyWithUid
> (notBoot mod_name) uid) | (uid, mod_name, _) <- entries]
>
>
>
>     edges deps = map (edge_map Map.!) deps
>
>
>
> registerToModuleFinder :: HscEnv -> UnitId -> ModuleName -> IO Module
>
> registerToModuleFinder hsc_env unitId modName = do
>
>     addHomeModuleToFinder (hsc_FC hsc_env) homeUnit (notBoot modName)
> location
>
>   where
>
>     homeUnit = DefiniteHomeUnit unitId Nothing
>
>     location = dummyModLocation modName
>
> ```
>
>
>
> But, as you can see in the attached profile, I still get way more than 2k
> ModuleNodes and NodeKey_Modules….
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Tuesday, April 1, 2025 5:44 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
> Hi Gergo,
>
>
>
> I looked in the detailed pane, searched for ModuleGraph, hovered my mouse
> over the "ModuleGraph` constructor, recorded the number of live bytes,
> divided that by 32.
>
>
>
> Matt
>
>
>
> On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
>
>
> OK scratch that, I was looking at wrong ghc-debug output. Indeed there are
> 2301 ModuleGraphs in the heap at the end of typechecking :O
>
>
>
> *From:* Erdi, Gergo
> *Sent:* Tuesday, April 1, 2025 12:51 PM
> *To:* Matthew Pickering <matthewtpickering at gmail.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* Re: GHC memory usage when typechecking from source vs. loading
> ModIfaces
>
>
>
> This sounds extremely interesting, but I don’t understand where you are
> getting this number from! How do you see in the eventlog HTMLs that I’ve
> included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug
> to find all ModuleGraph constructors at two points in the run: just before
> typechecking the first module (after all the extendMG calls) and just after
> typechecking the last module, and even in the cold case I only see 1
> ModuleGraph before and 13 ModuleGraphs after.
>
>
>
> Also, what do you mean by “precisely one loaded per interface loaded into
> the EPS”? Since my repro has 2294 modules, wouldn’t that mean 2294
> ModuleGraphs by that metric?
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Saturday, March 29, 2025 1:53 AM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
>
>
> Hi Gergo,
>
>
>
> I quickly tried building `Cabal` with the master branch. There is
> precisely 1 ModuleGraph allocated for the home session, and precisely one
> loaded per interface loaded into the EPS. No leaky behaviour like you can
> see in your eventlogs.
>
>
>
> It seems there are about 2000 live module graphs in your program, are you
> doing something with the API to create this many?
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <
> matthewtpickering at gmail.com> wrote:
>
> HI Gergo,
>
>
>
> Do you have a (synthetic?) reproducer? You have probably identified some
> memory leak. However, without any means to reproduce it becomes very
> difficult to investigate. I feel like we are getting into very precise
> details now, where speculating is not going to be so useful.
>
>
>
> It seems like this is an important thing for you and your company. Is
> there any budget to pay for some investigation? If that was the case then
> some effort could be made to create a synthetic producer and make the
> situation more robust going into the future if your requirements were
> precisely understood.
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
> Just to add that I get the same "equalizing" behaviour (but in a more
> "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just
> call `hugInstancesBelow` before processing each module. So that's
> definitely one source of extra memory usage. I wonder if it would be
> possible to rebuild the ModuleGraph periodically (similar to the ModDetails
> dehydration), or if there are references to it stored all over the place
> from `HscEnv`s scattered around in closures etc. (basically the same
> problem the HPT had before it was made into a mutable reference).
>
> -----Original Message-----
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Erdi, Gergo
> via ghc-devs
> Sent: Friday, March 28, 2025 4:49 PM
> To: Matthew Pickering <matthewtpickering at gmail.com>; GHC Devs <
> ghc-devs at haskell.org>
> Cc: ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <
> Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> Hi,
>
> Unfortunately, I am forced to return to this problem. Everything below is
> now in the context of GHC 9.12 plus the mutable HPT patch backported.
>
> My test case is typechecking a tree of 2294 modules that form the
> transitive closure of a single module's dependencies, all in a single
> process. I have done this typechecking three times, here's what `+RTS -s
> -RTS` reports for max residency:
>
> * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB
>
> * "cold-top": With all `ModIface`s already on disk, except for the
>   single top-level module: 302 MB
>
> * "warm": With all `ModIface`s already on disk: 211 MB
>
> So my stupidly naive question is, why is the "cold" case also not 302 MB?
>
> In earlier discussion, `ModDetails` unfolding has come up. Dehydrating
> `ModDetails` in the HPT all the time is disastrous for runtime, but based
> on this model I would expect to see improvements from dehydrating "every
> now and then". So I tried a stupid simple example where after every 100th
> typechecked module, I run this function on the topologically sorted list of
> modules processed so far:
>
>
> ```
> dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods
> = do
>     let HPT{ table = hptr } = hsc_HPT hsc_env
>     hpt <- readIORef hptr
>     for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface
> _details _linkable) -> do
>         !details <- initModDetails hsc_env iface
>         pure ()
> ```
>
> Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact,
> the profile looks exactly the same.
>
> Speaking of the profile, in the "cold" case I see a lot of steadily
> increasing heap usage from the `ModuleGraph`. I could see this happening if
> typechecking from scratch involves more `modulesUnder` calls which in turn
> force more and more of the `ModuleGraph`. If so, then maybe this could be
> worked around by repeatedly remaking the `ModuleGraph` just like I remake
> the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing
> the `ModuleGraph` at the start, with the idea being that this should
> "equalize" the three scenarios if this really is a substantial source of
> extra memory usage. This pushes up the warm case's memory usage to 381 MB,
> which is promising, but I still see a `Word64Map` that is steadily
> increasing in the "cold-force-modulegraph" case and contributes a lot to
> the memory usage. Unfortunately, I don't know where that `Word64Map` is (it
> could be any `Unique`-keyed environment...).
>
> So I am now stuck at this point. To spell out my goal explicitly, I would
> like to typecheck one module after another and not keep anything more in
> memory around than if I loaded them from `ModIface` files.
>
> Thanks,
>         Gergo
>
> p.s.: I couldn't find a way in the EventLog output HTML to turn event
> markers on/off or filter them, so to avoid covering the whole graph with
> gray lines, I mark only every 100th module.
>
>
>
>
> From: Matthew Pickering <matthewtpickering at gmail.com>
> Sent: Wednesday, February 12, 2025 7:08 PM
> To: ÉRDI Gergő <gergo at erdi.hu>
> Cc: Erdi, Gergo <Gergo.Erdi at sc.com>; Zubin Duggal <zubin at well-typed.com>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; GHC Devs <
> ghc-devs at haskell.org>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> You do also raise a good point about rehydration costs.
>
> In oneshot mode, you are basically rehydrating the entire transitive
> closure of each module when you compile it, which obviously results in a
> large amount of repeated work. This is why people are investigating ideas
> of a persistent worker to at least avoid rehydrating all external
> dependencies as well.
>
> On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:
> matthewtpickering at gmail.com> wrote:
> Sure, you can remove them once you are sure they are not used anymore.
>
> For clients like `GHCi` that doesn't work obviously as they can be used at
> any point in the future but for a batch compiler it would be fine.
>
> On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu> wrote:
> On Mon, 10 Feb 2025, Matthew Pickering wrote:
>
> > I wonder if you have got your condition the wrong way around.
> >
> > The only "safe" time to perform rehydration is AFTER the point it can
> > never be used again.
> >
> > If you rehydrate it just before it is used then you will repeat work
> > which has already been done. If you do this, you will always have a
> > trade-off between space used and runtime.
>
> Oops. Yes, I have misunderstood the idea. I thought the idea was that
> after loading a given module into the HPT, its ModDetails would start out
> small (because of laziness) and then keep growing in size as more and more
> of it are traversed, and thus forced, during the typechecking of its
> dependees, so at some point we would want to reset that into the small
> initial representation as created by initModDetails.
>
> But if the idea is that I should rehydrate modules when they can't be used
> anymore, then that brings up the question why even do that, instead of
> straight removing the HomeModInfos from the HPT?
>
> ----------------------------------------------------------------------
>
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/1268c59a/attachment.html>

From Gergo.Erdi at sc.com  Wed Apr  2 09:25:56 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Wed, 2 Apr 2025 09:25:56 +0000
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
Message-ID: <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,
            Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

From: Matthew Pickering <matthewtpickering at gmail.com>
Sent: Friday, March 28, 2025 8:40 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/2b24b7ce/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ghc-prof.zip
Type: application/x-zip-compressed
Size: 1722494 bytes
Desc: ghc-prof.zip
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/2b24b7ce/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: repro-hs.patch.gz
Type: application/x-gzip
Size: 80437 bytes
Desc: repro-hs.patch.gz
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/2b24b7ce/attachment-0003.bin>

From matthewtpickering at gmail.com  Wed Apr  2 09:38:34 2025
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Wed, 2 Apr 2025 10:38:34 +0100
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>

What command do I run to generate the files from this patch file? Perhaps a
link to a git repo would be a suitable way to share the reproducer?

On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:

> PUBLIC
>
> Hi Matt,
>
>
>
> I think I have something that might demonstrate that GHC (at least GHC
> 9.12.1) might have a similar problem!
>
>
>
> With the attached vacuous module hierarchy, I tried compiling M2294 from
> scratch, and then with `.hi` files for everything except the toplevel
> module. I did the same with our GHC-API-using compiler as well. As you can
> see from the attached event logs, while the details differ, the overall
> shape of the memory used by ModuleGraph edges (750k of GWIB and
> NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct
> dependency edges) is pretty much the same between our compiler and GHC
> 9.12, suggesting to me that GHC is duplicating ModuleGraph node information
> in the dependency edges when building the transitive closure.
>
>
>
> Based on these measurements, do you agree that this is a GHC-side problem
> of memory usage scaling quadratically with the number of dependency edges?
>
>
>
> Thanks,
>
>             Gergo
>
>
>
> p.s.: Sorry for including the reproducer module tree in this weird format
> as a patch file, but I am behind a mail server that won’t let me send mails
> with too many individual files in attached archives…
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Friday, March 28, 2025 8:40 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
> HI Gergo,
>
>
>
> Do you have a (synthetic?) reproducer? You have probably identified some
> memory leak. However, without any means to reproduce it becomes very
> difficult to investigate. I feel like we are getting into very precise
> details now, where speculating is not going to be so useful.
>
>
>
> It seems like this is an important thing for you and your company. Is
> there any budget to pay for some investigation? If that was the case then
> some effort could be made to create a synthetic producer and make the
> situation more robust going into the future if your requirements were
> precisely understood.
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
> Just to add that I get the same "equalizing" behaviour (but in a more
> "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just
> call `hugInstancesBelow` before processing each module. So that's
> definitely one source of extra memory usage. I wonder if it would be
> possible to rebuild the ModuleGraph periodically (similar to the ModDetails
> dehydration), or if there are references to it stored all over the place
> from `HscEnv`s scattered around in closures etc. (basically the same
> problem the HPT had before it was made into a mutable reference).
>
> -----Original Message-----
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Erdi, Gergo
> via ghc-devs
> Sent: Friday, March 28, 2025 4:49 PM
> To: Matthew Pickering <matthewtpickering at gmail.com>; GHC Devs <
> ghc-devs at haskell.org>
> Cc: ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <
> Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> Hi,
>
> Unfortunately, I am forced to return to this problem. Everything below is
> now in the context of GHC 9.12 plus the mutable HPT patch backported.
>
> My test case is typechecking a tree of 2294 modules that form the
> transitive closure of a single module's dependencies, all in a single
> process. I have done this typechecking three times, here's what `+RTS -s
> -RTS` reports for max residency:
>
> * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB
>
> * "cold-top": With all `ModIface`s already on disk, except for the
>   single top-level module: 302 MB
>
> * "warm": With all `ModIface`s already on disk: 211 MB
>
> So my stupidly naive question is, why is the "cold" case also not 302 MB?
>
> In earlier discussion, `ModDetails` unfolding has come up. Dehydrating
> `ModDetails` in the HPT all the time is disastrous for runtime, but based
> on this model I would expect to see improvements from dehydrating "every
> now and then". So I tried a stupid simple example where after every 100th
> typechecked module, I run this function on the topologically sorted list of
> modules processed so far:
>
>
> ```
> dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods
> = do
>     let HPT{ table = hptr } = hsc_HPT hsc_env
>     hpt <- readIORef hptr
>     for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface
> _details _linkable) -> do
>         !details <- initModDetails hsc_env iface
>         pure ()
> ```
>
> Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact,
> the profile looks exactly the same.
>
> Speaking of the profile, in the "cold" case I see a lot of steadily
> increasing heap usage from the `ModuleGraph`. I could see this happening if
> typechecking from scratch involves more `modulesUnder` calls which in turn
> force more and more of the `ModuleGraph`. If so, then maybe this could be
> worked around by repeatedly remaking the `ModuleGraph` just like I remake
> the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing
> the `ModuleGraph` at the start, with the idea being that this should
> "equalize" the three scenarios if this really is a substantial source of
> extra memory usage. This pushes up the warm case's memory usage to 381 MB,
> which is promising, but I still see a `Word64Map` that is steadily
> increasing in the "cold-force-modulegraph" case and contributes a lot to
> the memory usage. Unfortunately, I don't know where that `Word64Map` is (it
> could be any `Unique`-keyed environment...).
>
> So I am now stuck at this point. To spell out my goal explicitly, I would
> like to typecheck one module after another and not keep anything more in
> memory around than if I loaded them from `ModIface` files.
>
> Thanks,
>         Gergo
>
> p.s.: I couldn't find a way in the EventLog output HTML to turn event
> markers on/off or filter them, so to avoid covering the whole graph with
> gray lines, I mark only every 100th module.
>
>
>
>
> From: Matthew Pickering <matthewtpickering at gmail.com>
> Sent: Wednesday, February 12, 2025 7:08 PM
> To: ÉRDI Gergő <gergo at erdi.hu>
> Cc: Erdi, Gergo <Gergo.Erdi at sc.com>; Zubin Duggal <zubin at well-typed.com>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; GHC Devs <
> ghc-devs at haskell.org>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> You do also raise a good point about rehydration costs.
>
> In oneshot mode, you are basically rehydrating the entire transitive
> closure of each module when you compile it, which obviously results in a
> large amount of repeated work. This is why people are investigating ideas
> of a persistent worker to at least avoid rehydrating all external
> dependencies as well.
>
> On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:
> matthewtpickering at gmail.com> wrote:
> Sure, you can remove them once you are sure they are not used anymore.
>
> For clients like `GHCi` that doesn't work obviously as they can be used at
> any point in the future but for a batch compiler it would be fine.
>
> On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu> wrote:
> On Mon, 10 Feb 2025, Matthew Pickering wrote:
>
> > I wonder if you have got your condition the wrong way around.
> >
> > The only "safe" time to perform rehydration is AFTER the point it can
> > never be used again.
> >
> > If you rehydrate it just before it is used then you will repeat work
> > which has already been done. If you do this, you will always have a
> > trade-off between space used and runtime.
>
> Oops. Yes, I have misunderstood the idea. I thought the idea was that
> after loading a given module into the HPT, its ModDetails would start out
> small (because of laziness) and then keep growing in size as more and more
> of it are traversed, and thus forced, during the typechecking of its
> dependees, so at some point we would want to reset that into the small
> initial representation as created by initModDetails.
>
> But if the idea is that I should rehydrate modules when they can't be used
> anymore, then that brings up the question why even do that, instead of
> straight removing the HomeModInfos from the HPT?
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/1ed2a0c0/attachment.html>

From Gergo.Erdi at sc.com  Wed Apr  2 09:47:30 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Wed, 2 Apr 2025 09:47:30 +0000
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>
Message-ID: <SEYPR03MB7463AA22D073E313B431B865E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

zcat ../repro-hs.patch.gz |patch -p0

From: Matthew Pickering <matthewtpickering at gmail.com>
Sent: Wednesday, April 2, 2025 5:39 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces


What command do I run to generate the files from this patch file? Perhaps a link to a git repo would be a suitable way to share the reproducer?

On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,
            Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Friday, March 28, 2025 8:40 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
________________________________
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/cebeb4f6/attachment.html>

From matthewtpickering at gmail.com  Wed Apr  2 10:02:42 2025
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Wed, 2 Apr 2025 11:02:42 +0100
Subject: GHC memory usage when typechecking from source vs. loading
 ModIfaces
In-Reply-To: <SEYPR03MB7463AA22D073E313B431B865E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>
 <SEYPR03MB7463AA22D073E313B431B865E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <CALuQ0m88xhEMWG=OSeKSOo4f2Dm7nEgGkFo-7cKX9txAK=cNKA@mail.gmail.com>

I think you are missing
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593

On HEAD I get maximum residency of about 200M, NodeKey usage is constant.

On 9.10.1, I get maximum residency of 400M, NodeKey usage looks quadratic.



On Wed, Apr 2, 2025 at 10:47 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:

> PUBLIC
>
> zcat ../repro-hs.patch.gz |patch -p0
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Wednesday, April 2, 2025 5:39 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
>
>
> What command do I run to generate the files from this patch file? Perhaps
> a link to a git repo would be a suitable way to share the reproducer?
>
>
>
> On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
>
>
> Hi Matt,
>
>
>
> I think I have something that might demonstrate that GHC (at least GHC
> 9.12.1) might have a similar problem!
>
>
>
> With the attached vacuous module hierarchy, I tried compiling M2294 from
> scratch, and then with `.hi` files for everything except the toplevel
> module. I did the same with our GHC-API-using compiler as well. As you can
> see from the attached event logs, while the details differ, the overall
> shape of the memory used by ModuleGraph edges (750k of GWIB and
> NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct
> dependency edges) is pretty much the same between our compiler and GHC
> 9.12, suggesting to me that GHC is duplicating ModuleGraph node information
> in the dependency edges when building the transitive closure.
>
>
>
> Based on these measurements, do you agree that this is a GHC-side problem
> of memory usage scaling quadratically with the number of dependency edges?
>
>
>
> Thanks,
>
>             Gergo
>
>
>
> p.s.: Sorry for including the reproducer module tree in this weird format
> as a patch file, but I am behind a mail server that won’t let me send mails
> with too many individual files in attached archives…
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Friday, March 28, 2025 8:40 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
> HI Gergo,
>
>
>
> Do you have a (synthetic?) reproducer? You have probably identified some
> memory leak. However, without any means to reproduce it becomes very
> difficult to investigate. I feel like we are getting into very precise
> details now, where speculating is not going to be so useful.
>
>
>
> It seems like this is an important thing for you and your company. Is
> there any budget to pay for some investigation? If that was the case then
> some effort could be made to create a synthetic producer and make the
> situation more robust going into the future if your requirements were
> precisely understood.
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
> Just to add that I get the same "equalizing" behaviour (but in a more
> "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just
> call `hugInstancesBelow` before processing each module. So that's
> definitely one source of extra memory usage. I wonder if it would be
> possible to rebuild the ModuleGraph periodically (similar to the ModDetails
> dehydration), or if there are references to it stored all over the place
> from `HscEnv`s scattered around in closures etc. (basically the same
> problem the HPT had before it was made into a mutable reference).
>
> -----Original Message-----
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Erdi, Gergo
> via ghc-devs
> Sent: Friday, March 28, 2025 4:49 PM
> To: Matthew Pickering <matthewtpickering at gmail.com>; GHC Devs <
> ghc-devs at haskell.org>
> Cc: ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <
> Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> Hi,
>
> Unfortunately, I am forced to return to this problem. Everything below is
> now in the context of GHC 9.12 plus the mutable HPT patch backported.
>
> My test case is typechecking a tree of 2294 modules that form the
> transitive closure of a single module's dependencies, all in a single
> process. I have done this typechecking three times, here's what `+RTS -s
> -RTS` reports for max residency:
>
> * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB
>
> * "cold-top": With all `ModIface`s already on disk, except for the
>   single top-level module: 302 MB
>
> * "warm": With all `ModIface`s already on disk: 211 MB
>
> So my stupidly naive question is, why is the "cold" case also not 302 MB?
>
> In earlier discussion, `ModDetails` unfolding has come up. Dehydrating
> `ModDetails` in the HPT all the time is disastrous for runtime, but based
> on this model I would expect to see improvements from dehydrating "every
> now and then". So I tried a stupid simple example where after every 100th
> typechecked module, I run this function on the topologically sorted list of
> modules processed so far:
>
>
> ```
> dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods
> = do
>     let HPT{ table = hptr } = hsc_HPT hsc_env
>     hpt <- readIORef hptr
>     for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface
> _details _linkable) -> do
>         !details <- initModDetails hsc_env iface
>         pure ()
> ```
>
> Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact,
> the profile looks exactly the same.
>
> Speaking of the profile, in the "cold" case I see a lot of steadily
> increasing heap usage from the `ModuleGraph`. I could see this happening if
> typechecking from scratch involves more `modulesUnder` calls which in turn
> force more and more of the `ModuleGraph`. If so, then maybe this could be
> worked around by repeatedly remaking the `ModuleGraph` just like I remake
> the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing
> the `ModuleGraph` at the start, with the idea being that this should
> "equalize" the three scenarios if this really is a substantial source of
> extra memory usage. This pushes up the warm case's memory usage to 381 MB,
> which is promising, but I still see a `Word64Map` that is steadily
> increasing in the "cold-force-modulegraph" case and contributes a lot to
> the memory usage. Unfortunately, I don't know where that `Word64Map` is (it
> could be any `Unique`-keyed environment...).
>
> So I am now stuck at this point. To spell out my goal explicitly, I would
> like to typecheck one module after another and not keep anything more in
> memory around than if I loaded them from `ModIface` files.
>
> Thanks,
>         Gergo
>
> p.s.: I couldn't find a way in the EventLog output HTML to turn event
> markers on/off or filter them, so to avoid covering the whole graph with
> gray lines, I mark only every 100th module.
>
>
>
>
> From: Matthew Pickering <matthewtpickering at gmail.com>
> Sent: Wednesday, February 12, 2025 7:08 PM
> To: ÉRDI Gergő <gergo at erdi.hu>
> Cc: Erdi, Gergo <Gergo.Erdi at sc.com>; Zubin Duggal <zubin at well-typed.com>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; GHC Devs <
> ghc-devs at haskell.org>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> You do also raise a good point about rehydration costs.
>
> In oneshot mode, you are basically rehydrating the entire transitive
> closure of each module when you compile it, which obviously results in a
> large amount of repeated work. This is why people are investigating ideas
> of a persistent worker to at least avoid rehydrating all external
> dependencies as well.
>
> On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:
> matthewtpickering at gmail.com> wrote:
> Sure, you can remove them once you are sure they are not used anymore.
>
> For clients like `GHCi` that doesn't work obviously as they can be used at
> any point in the future but for a batch compiler it would be fine.
>
> On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu> wrote:
> On Mon, 10 Feb 2025, Matthew Pickering wrote:
>
> > I wonder if you have got your condition the wrong way around.
> >
> > The only "safe" time to perform rehydration is AFTER the point it can
> > never be used again.
> >
> > If you rehydrate it just before it is used then you will repeat work
> > which has already been done. If you do this, you will always have a
> > trade-off between space used and runtime.
>
> Oops. Yes, I have misunderstood the idea. I thought the idea was that
> after loading a given module into the HPT, its ModDetails would start out
> small (because of laziness) and then keep growing in size as more and more
> of it are traversed, and thus forced, during the typechecking of its
> dependees, so at some point we would want to reset that into the small
> initial representation as created by initModDetails.
>
> But if the idea is that I should rehydrate modules when they can't be used
> anymore, then that brings up the question why even do that, instead of
> straight removing the HomeModInfos from the HPT?
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/e55205f6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2025-04-02-110039_1254x783.png
Type: image/png
Size: 138132 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/e55205f6/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2025-04-02-110049_1676x558.png
Type: image/png
Size: 95386 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/e55205f6/attachment-0003.png>

From george.colpitts at gmail.com  Wed Apr  2 12:06:05 2025
From: george.colpitts at gmail.com (George Colpitts)
Date: Wed, 2 Apr 2025 09:06:05 -0300
Subject: can't move to latest version of llvm on ghc 9.12.2
In-Reply-To: <4db7eafd-69b0-476c-bffd-e2a8827d7346@gmx.at>
References: <CAB-d4A5WCbBX-G+3RvV8AQ_+5_-fR2P8ED7qO-14PhwMVdKY-g@mail.gmail.com>
 <CAKFCL4WDj31se4=nEo4ts8-OwQRUwaXioncPgB1PUgvA2q1HKQ@mail.gmail.com>
 <CAB-d4A4w-ghx4hCmizaw0Ebdye2i0ohe+KbP06XuQJUx0vpn3Q@mail.gmail.com>
 <4db7eafd-69b0-476c-bffd-e2a8827d7346@gmx.at>
Message-ID: <CAB-d4A4teaamw8dOkR1wLjYhPW0uknorvDJyYVnp0gag+No3QQ@mail.gmail.com>

Filed https://gitlab.haskell.org/ghc/ghc/-/issues/25915

On Tue, Apr 1, 2025 at 2:11 PM Andreas Klebinger via ghc-devs <
ghc-devs at haskell.org> wrote:

> Looking at https://gitlab.haskell.org/ghc/ghc/-/issues/25011 it seems the
> desire was to fail with a proper error when LLVM was not found at all.
>
> To me failing when the llvm version is too new seems like a unintended
> side effect of fixing the former.
>
> I agree that simply warning and trying anway is the better default. A
> ticket would be appreciated!
> Am 01/04/2025 um 19:05 schrieb George Colpitts:
>
> I was trying to find out if it works but I ran into the problem I
> described. Yes, there were a few years where the latest version didn't work
> but in many years it did. ghc used to give a warning, this is an
> unsupported version but we'll try it anyways. I would like to return to
> that.
>
> In any case I believe ghc dev should test to see if it works and if so use
> it in HEAD. If it doesn't work than we should file a bug and fix it. In the
> past we got a few versions behind llvm. I think we want to be on the latest
> version available for each new release if possible. The earlier we look
> into this the more chance we have to succeed at that.
>
> On Tue, Apr 1, 2025 at 1:27 PM Brandon Allbery <allbery.b at gmail.com>
> wrote:
>
>> Have you demonstrated that it works? IIRC the current behavior is because
>> some LLVM version (16, IIRC) didn't work with GHC (threw errors from opt, I
>> think).
>>
>> On Tue, Apr 1, 2025 at 10:49 AM George Colpitts <
>> george.colpitts at gmail.com> wrote:
>>
>>> llvm 20 is out but unlike in earlier versions of ghc moving to it means
>>> you can no longer use llvm:
>>>
>>> compiling:
>>>
>>> ghc -fllvm hello.hs
>>> Loaded package environment from
>>> /Users/avie/.ghc/aarch64-darwin-9.12.2/environments/default
>>> [1 of 2] Compiling Main             ( hello.hs, hello.o )
>>> <no location info>: error: [GHC-66599]
>>>     GHC was not configured with a supported LLVM toolchain
>>>     Make sure you have installed LLVM between [13 and 20) and reinstall
>>> GHC to make -fllvm work
>>>
>>>
>>> from configure:
>>>
>>> configure: We only support llvm 13 upto 20 (non-inclusive) (found
>>> 20.1.1).
>>>
>>>
>>> Can we move to llvm 20 on HEAD and can we revert to the old behavior on
>>> ghc 9.12.3?
>>>
>>> Should I file an ER?
>>>
>>> Thanks
>>>
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>>
>>
>> --
>> brandon s allbery kf8nh
>> allbery.b at gmail.com
>>
>
> _______________________________________________
> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/925cd23b/attachment.html>

From teofilcamarasu at gmail.com  Wed Apr  2 17:26:02 2025
From: teofilcamarasu at gmail.com (Teo Camarasu)
Date: Wed, 2 Apr 2025 18:26:02 +0100
Subject: Next steps forward for a stable Template Haskell
Message-ID: <CAPBO51EE6JB-tFsuKEKQZwBmnqruq6s=cdTREJBCcpNigRiiZw@mail.gmail.com>

Hi folks,

At Tuesday's GHC devs meeting, we talked about having a discussion about
template-haskell stability. I've jotted down some notes to help focus our
future discussion. It's mostly just a list of what I take to be the next
most important issues to tackle, and some details about each.

https://docs.google.com/document/d/1gui1YXOvrNihR3NpXuMVCKV3lTkKPL8y_HDRjbZyRxk/edit?usp=sharing

Cheers,
Teo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250402/07f0d46c/attachment.html>

From Gergo.Erdi at sc.com  Thu Apr  3 10:54:11 2025
From: Gergo.Erdi at sc.com (Erdi, Gergo)
Date: Thu, 3 Apr 2025 10:54:11 +0000
Subject: [External] Re: GHC memory usage when typechecking from source vs.
 loading ModIfaces
In-Reply-To: <CALuQ0m88xhEMWG=OSeKSOo4f2Dm7nEgGkFo-7cKX9txAK=cNKA@mail.gmail.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>
 <SEYPR03MB7463AA22D073E313B431B865E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m88xhEMWG=OSeKSOo4f2Dm7nEgGkFo-7cKX9txAK=cNKA@mail.gmail.com>
Message-ID: <SEYPR03MB74634F6071582CC556AF980AE9AE2@SEYPR03MB7463.apcprd03.prod.outlook.com>

PUBLIC

Yes I was, since I am using 9.12.1.

This is great stuff -- with !13593 backported, I don’t see any ModuleGraph-related stuff showing up in top spots of the heap profile! Thank you for pointing me towards this!

I’ll have to do some more digging to see if there is anything else concretely blameable now. But in theory, morally, abstractly, from the 10,000 feet view, would it be correct to say that there should be no memory usage difference between typechecking a large number of modules from scratch vs. loading the `.hi` files for those same modules? (Modulo, of course, the memory usage during any single module’s typechecking).

From: Matthew Pickering <matthewtpickering at gmail.com>
Sent: Wednesday, April 2, 2025 6:03 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

I think you are missing https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593<https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593__;!!ASp95G87aa5DoyK5mB3l!736R3M2loLHwk60BLtBarIhVc0mTMNZ41vyzyjTmqQJ81DibLFqtrJNUeZLxM8YOIGmg8SbzoAYXhGK97EyBWDm9$>

On HEAD I get maximum residency of about 200M, NodeKey usage is constant.

On 9.10.1, I get maximum residency of 400M, NodeKey usage looks quadratic.



On Wed, Apr 2, 2025 at 10:47 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:

PUBLIC

zcat ../repro-hs.patch.gz |patch -p0

From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, April 2, 2025 5:39 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces


What command do I run to generate the files from this patch file? Perhaps a link to a git repo would be a suitable way to share the reproducer?

On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:

PUBLIC

Hi Matt,

I think I have something that might demonstrate that GHC (at least GHC 9.12.1) might have a similar problem!

With the attached vacuous module hierarchy, I tried compiling M2294 from scratch, and then with `.hi` files for everything except the toplevel module. I did the same with our GHC-API-using compiler as well. As you can see from the attached event logs, while the details differ, the overall shape of the memory used by ModuleGraph edges (750k of GWIB and NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct dependency edges) is pretty much the same between our compiler and GHC 9.12, suggesting to me that GHC is duplicating ModuleGraph node information in the dependency edges when building the transitive closure.

Based on these measurements, do you agree that this is a GHC-side problem of memory usage scaling quadratically with the number of dependency edges?

Thanks,
            Gergo

p.s.: Sorry for including the reproducer module tree in this weird format as a patch file, but I am behind a mail server that won’t let me send mails with too many individual files in attached archives…

From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Friday, March 28, 2025 8:40 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>; ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

HI Gergo,

Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like we are getting into very precise details now, where speculating is not going to be so useful.

It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.

Cheers,

Matt

On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>> wrote:
PUBLIC

Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before it was made into a mutable reference).

-----Original Message-----
From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: Friday, March 28, 2025 4:49 PM
To: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; Dijkstra, Atze <Atze.Dijkstra at sc.com<mailto:Atze.Dijkstra at sc.com>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

Hi,

Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.

My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:

* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB

* "cold-top": With all `ModIface`s already on disk, except for the
  single top-level module: 302 MB

* "warm": With all `ModIface`s already on disk: 211 MB

So my stupidly naive question is, why is the "cold" case also not 302 MB?

In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:


```
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do
    let HPT{ table = hptr } = hsc_HPT hsc_env
    hpt <- readIORef hptr
    for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do
        !details <- initModDetails hsc_env iface
        pure ()
```

Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.

Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`. If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize" the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).

So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.

Thanks,
        Gergo

p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.




From: Matthew Pickering <matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>>
Sent: Wednesday, February 12, 2025 7:08 PM
To: ÉRDI Gergő <gergo at erdi.hu<mailto:gergo at erdi.hu>>
Cc: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>; Zubin Duggal <zubin at well-typed.com<mailto:zubin at well-typed.com>>; Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; GHC Devs <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces

You do also raise a good point about rehydration costs.

In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid rehydrating all external dependencies as well.

On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:matthewtpickering at gmail.com<mailto:matthewtpickering at gmail.com>> wrote:
Sure, you can remove them once you are sure they are not used anymore.

For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.

On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu<mailto:gergo at erdi.hu>> wrote:
On Mon, 10 Feb 2025, Matthew Pickering wrote:

> I wonder if you have got your condition the wrong way around.
>
> The only "safe" time to perform rehydration is AFTER the point it can
> never be used again.
>
> If you rehydrate it just before it is used then you will repeat work
> which has already been done. If you do this, you will always have a
> trade-off between space used and runtime.

Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced, during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.

But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
________________________________
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
________________________________
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com

----------------------------------------------------------------------
This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250403/7c5f56ad/attachment-0001.html>

From matthewtpickering at gmail.com  Thu Apr  3 11:01:33 2025
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Thu, 3 Apr 2025 12:01:33 +0100
Subject: [External] Re: GHC memory usage when typechecking from source vs.
 loading ModIfaces
In-Reply-To: <SEYPR03MB74634F6071582CC556AF980AE9AE2@SEYPR03MB7463.apcprd03.prod.outlook.com>
References: <CAO7EwMoZPheo=ubP6S5VkTNEuWxSP-BiG0NkfJnMPwA_sg9AQQ@mail.gmail.com>
 <SEYPR03MB74634DE2A8BDE8A481B919AAE91B2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <pom2vzn7ff2ntiwn7gub7tj2d7dghdfrcqqwjqylrstsv4kvuh@fnspm4m4ji3e>
 <CALuQ0m-fn3rTRs1PgK1WdoV3F8FJnF28HST5aOeEdyrK1L8JyQ@mail.gmail.com>
 <4f980efd-58bc-2814-b4cf-b5742c040b97@erdi.hu>
 <CALuQ0m-6RYuJAMT0kw272E2-R_9LsbJ4ZJPpr7m64ymYnaREVg@mail.gmail.com>
 <SEYPR03MB7463A4ED0988986744263839E9E02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m9FXh4fzhmVRBiFW82rS3W4dPuDnH+7u6yqs3Gwff=B_Q@mail.gmail.com>
 <SEYPR03MB74638EDF52238E57708B6C0EE9F72@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m8VJKXz3y3y3Mv+dK1pFdY2DDVgdWM_oTGPCC51JL5jTg@mail.gmail.com>
 <3a5dd981-7fdc-a814-88c9-6768395cae39@erdi.hu>
 <CALuQ0m__FtS-cfz4TcWXmv2JV1r4-sz1e73RK+JPridQTB0EVQ@mail.gmail.com>
 <CALuQ0m-3iB+2oY1x45FhT+G_OypYo2T0gyzjXUoo3yvm8H6JmA@mail.gmail.com>
 <SEYPR03MB7463AABC8385A5FCCCAC330CE9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <SEYPR03MB7463EC99669F8723A3156872E9A02@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_FHoDPF8ZV+-441AS22bQbLDrS9GsX-+vaUYD9okrKXA@mail.gmail.com>
 <SEYPR03MB7463223D4085BA61D2B64FE7E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m_R_oerbKZ43t+1-RA_9R7ec9D3=O7ZhUECohsC3EAXmQ@mail.gmail.com>
 <SEYPR03MB7463AA22D073E313B431B865E9AF2@SEYPR03MB7463.apcprd03.prod.outlook.com>
 <CALuQ0m88xhEMWG=OSeKSOo4f2Dm7nEgGkFo-7cKX9txAK=cNKA@mail.gmail.com>
 <SEYPR03MB74634F6071582CC556AF980AE9AE2@SEYPR03MB7463.apcprd03.prod.outlook.com>
Message-ID: <CALuQ0m_LFQdNPMWv3SWAqLHx6MDcZS3b9NokUq1CmVuF7oHbPQ@mail.gmail.com>

Yes, but

* With all the caveats around forcing ModDetails and using extra memory.
* When you typecheck from scratch you may load additional interfaces for
external dependencies into the EPS.

After a module is typechecked, everything apart from the interface is
discarded and everything is regenerated from that interface.

It's possible there are some leaks in forcing the interface, since the deep
forcing was incomplete (see
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14078)

Anyway, best thing is to profile with these invariants in mind, and fix
situations where the invariants are violated.

Cheers,

Matt

On Thu, Apr 3, 2025 at 11:54 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:

> PUBLIC
>
> Yes I was, since I am using 9.12.1.
>
>
>
> This is great stuff -- with !13593 backported, I don’t see any
> ModuleGraph-related stuff showing up in top spots of the heap profile!
> Thank you for pointing me towards this!
>
> I’ll have to do some more digging to see if there is anything else
> concretely blameable now. But in theory, morally, abstractly, from the
> 10,000 feet view, would it be correct to say that there should be no memory
> usage difference between typechecking a large number of modules from
> scratch vs. loading the `.hi` files for those same modules? (Modulo, of
> course, the memory usage during any single module’s typechecking).
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Wednesday, April 2, 2025 6:03 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
> I think you are missing
> https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593
> <https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13593__;!!ASp95G87aa5DoyK5mB3l!736R3M2loLHwk60BLtBarIhVc0mTMNZ41vyzyjTmqQJ81DibLFqtrJNUeZLxM8YOIGmg8SbzoAYXhGK97EyBWDm9$>
>
>
>
> On HEAD I get maximum residency of about 200M, NodeKey usage is constant.
>
>
>
> On 9.10.1, I get maximum residency of 400M, NodeKey usage looks quadratic.
>
>
>
>
>
>
>
> On Wed, Apr 2, 2025 at 10:47 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
>
>
> zcat ../repro-hs.patch.gz |patch -p0
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Wednesday, April 2, 2025 5:39 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
>
>
> What command do I run to generate the files from this patch file? Perhaps
> a link to a git repo would be a suitable way to share the reproducer?
>
>
>
> On Wed, Apr 2, 2025 at 10:26 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
>
>
> Hi Matt,
>
>
>
> I think I have something that might demonstrate that GHC (at least GHC
> 9.12.1) might have a similar problem!
>
>
>
> With the attached vacuous module hierarchy, I tried compiling M2294 from
> scratch, and then with `.hi` files for everything except the toplevel
> module. I did the same with our GHC-API-using compiler as well. As you can
> see from the attached event logs, while the details differ, the overall
> shape of the memory used by ModuleGraph edges (750k of GWIB and
> NodeKey_Module constructors for the 2321 ModuleNodes and ~60k direct
> dependency edges) is pretty much the same between our compiler and GHC
> 9.12, suggesting to me that GHC is duplicating ModuleGraph node information
> in the dependency edges when building the transitive closure.
>
>
>
> Based on these measurements, do you agree that this is a GHC-side problem
> of memory usage scaling quadratically with the number of dependency edges?
>
>
>
> Thanks,
>
>             Gergo
>
>
>
> p.s.: Sorry for including the reproducer module tree in this weird format
> as a patch file, but I am behind a mail server that won’t let me send mails
> with too many individual files in attached archives…
>
>
>
> *From:* Matthew Pickering <matthewtpickering at gmail.com>
> *Sent:* Friday, March 28, 2025 8:40 PM
> *To:* Erdi, Gergo <Gergo.Erdi at sc.com>
> *Cc:* GHC Devs <ghc-devs at haskell.org>; ÉRDI Gergő <gergo at erdi.hu>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; Dijkstra, Atze
> <Atze.Dijkstra at sc.com>
> *Subject:* [External] Re: GHC memory usage when typechecking from source
> vs. loading ModIfaces
>
>
>
> HI Gergo,
>
>
>
> Do you have a (synthetic?) reproducer? You have probably identified some
> memory leak. However, without any means to reproduce it becomes very
> difficult to investigate. I feel like we are getting into very precise
> details now, where speculating is not going to be so useful.
>
>
>
> It seems like this is an important thing for you and your company. Is
> there any budget to pay for some investigation? If that was the case then
> some effort could be made to create a synthetic producer and make the
> situation more robust going into the future if your requirements were
> precisely understood.
>
>
>
> Cheers,
>
>
>
> Matt
>
>
>
> On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <Gergo.Erdi at sc.com> wrote:
>
> PUBLIC
>
> Just to add that I get the same "equalizing" behaviour (but in a more
> "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just
> call `hugInstancesBelow` before processing each module. So that's
> definitely one source of extra memory usage. I wonder if it would be
> possible to rebuild the ModuleGraph periodically (similar to the ModDetails
> dehydration), or if there are references to it stored all over the place
> from `HscEnv`s scattered around in closures etc. (basically the same
> problem the HPT had before it was made into a mutable reference).
>
> -----Original Message-----
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Erdi, Gergo
> via ghc-devs
> Sent: Friday, March 28, 2025 4:49 PM
> To: Matthew Pickering <matthewtpickering at gmail.com>; GHC Devs <
> ghc-devs at haskell.org>
> Cc: ÉRDI Gergő <gergo at erdi.hu>; Montelatici, Raphael Laurent <
> Raphael.Montelatici at sc.com>; Dijkstra, Atze <Atze.Dijkstra at sc.com>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> Hi,
>
> Unfortunately, I am forced to return to this problem. Everything below is
> now in the context of GHC 9.12 plus the mutable HPT patch backported.
>
> My test case is typechecking a tree of 2294 modules that form the
> transitive closure of a single module's dependencies, all in a single
> process. I have done this typechecking three times, here's what `+RTS -s
> -RTS` reports for max residency:
>
> * "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB
>
> * "cold-top": With all `ModIface`s already on disk, except for the
>   single top-level module: 302 MB
>
> * "warm": With all `ModIface`s already on disk: 211 MB
>
> So my stupidly naive question is, why is the "cold" case also not 302 MB?
>
> In earlier discussion, `ModDetails` unfolding has come up. Dehydrating
> `ModDetails` in the HPT all the time is disastrous for runtime, but based
> on this model I would expect to see improvements from dehydrating "every
> now and then". So I tried a stupid simple example where after every 100th
> typechecked module, I run this function on the topologically sorted list of
> modules processed so far:
>
>
> ```
> dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods
> = do
>     let HPT{ table = hptr } = hsc_HPT hsc_env
>     hpt <- readIORef hptr
>     for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface
> _details _linkable) -> do
>         !details <- initModDetails hsc_env iface
>         pure ()
> ```
>
> Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact,
> the profile looks exactly the same.
>
> Speaking of the profile, in the "cold" case I see a lot of steadily
> increasing heap usage from the `ModuleGraph`. I could see this happening if
> typechecking from scratch involves more `modulesUnder` calls which in turn
> force more and more of the `ModuleGraph`. If so, then maybe this could be
> worked around by repeatedly remaking the `ModuleGraph` just like I remake
> the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing
> the `ModuleGraph` at the start, with the idea being that this should
> "equalize" the three scenarios if this really is a substantial source of
> extra memory usage. This pushes up the warm case's memory usage to 381 MB,
> which is promising, but I still see a `Word64Map` that is steadily
> increasing in the "cold-force-modulegraph" case and contributes a lot to
> the memory usage. Unfortunately, I don't know where that `Word64Map` is (it
> could be any `Unique`-keyed environment...).
>
> So I am now stuck at this point. To spell out my goal explicitly, I would
> like to typecheck one module after another and not keep anything more in
> memory around than if I loaded them from `ModIface` files.
>
> Thanks,
>         Gergo
>
> p.s.: I couldn't find a way in the EventLog output HTML to turn event
> markers on/off or filter them, so to avoid covering the whole graph with
> gray lines, I mark only every 100th module.
>
>
>
>
> From: Matthew Pickering <matthewtpickering at gmail.com>
> Sent: Wednesday, February 12, 2025 7:08 PM
> To: ÉRDI Gergő <gergo at erdi.hu>
> Cc: Erdi, Gergo <Gergo.Erdi at sc.com>; Zubin Duggal <zubin at well-typed.com>;
> Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; GHC Devs <
> ghc-devs at haskell.org>
> Subject: [External] Re: GHC memory usage when typechecking from source vs.
> loading ModIfaces
>
> You do also raise a good point about rehydration costs.
>
> In oneshot mode, you are basically rehydrating the entire transitive
> closure of each module when you compile it, which obviously results in a
> large amount of repeated work. This is why people are investigating ideas
> of a persistent worker to at least avoid rehydrating all external
> dependencies as well.
>
> On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:
> matthewtpickering at gmail.com> wrote:
> Sure, you can remove them once you are sure they are not used anymore.
>
> For clients like `GHCi` that doesn't work obviously as they can be used at
> any point in the future but for a batch compiler it would be fine.
>
> On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:gergo at erdi.hu> wrote:
> On Mon, 10 Feb 2025, Matthew Pickering wrote:
>
> > I wonder if you have got your condition the wrong way around.
> >
> > The only "safe" time to perform rehydration is AFTER the point it can
> > never be used again.
> >
> > If you rehydrate it just before it is used then you will repeat work
> > which has already been done. If you do this, you will always have a
> > trade-off between space used and runtime.
>
> Oops. Yes, I have misunderstood the idea. I thought the idea was that
> after loading a given module into the HPT, its ModDetails would start out
> small (because of laziness) and then keep growing in size as more and more
> of it are traversed, and thus forced, during the typechecking of its
> dependees, so at some point we would want to reset that into the small
> initial representation as created by initModDetails.
>
> But if the idea is that I should rehydrate modules when they can't be used
> anymore, then that brings up the question why even do that, instead of
> straight removing the HomeModInfos from the HPT?
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ----------------------------------------------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
>
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> ------------------------------
> This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250403/31fa6fe4/attachment.html>