GHC memory usage when typechecking from source vs. loading ModIfaces

Fri Jan 17 11:19:20 UTC 2025

Hi Gergo,

I think there a few things to say here.

1. As Zubin points out we have recently been concerned with improving the
memory usage of large module sessions (#25511, !13675, !13593)

I imagine all these patches will greatly help the memory usage in your use
case.

2. You are absolutely right that ModDetails can get forced and is never
reset.

If you try !13675, it should be much more easily possible to reset the
ModDetails by writing into the IORef which stores each home package.

3. If you share your example or perhaps even a trace from ghc-debug then I
will be happy to investigate further as it seems like a great test case for
the work we have recently been doing.

Cheers,

Matt

On Fri, Jan 17, 2025 at 10:31 AM Zubin Duggal <zubin at well-typed.com> wrote:

> See https://gitlab.haskell.org/ghc/ghc/-/issues/25511
>
> On 25/01/17 08:54, Erdi, Gergo via ghc-devs wrote:
> >PUBLIC
> >
> >Looking at this with ghc-debug, at least I can see why we have the huge
> memory usage when recreating ModDetails: there are lots of HscEnvs stored
> all over the heap (I assume in thunks inside already compiled modules),
> when I recreate the ModDetails and replace them in the HUG, I am
> accumulating more and more copies of the same MOdDetails, since other
> HscEnvs are of course still pointing to the HUG that had the previous
> ModDetails, and so on.
> >
> >Given this, I don't really know yet if remaking the ModDetails would help
> me or not, since trying it is now blocking on figuring out how I could
> avoid having multiple HscEnvs in memory at the same time...
> >
> >From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Gergo Érdi
> >Sent: Friday, January 17, 2025 4:08 PM
> >To: GHC Devs <ghc-devs at haskell.org>
> >Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>
> >Subject: [External] GHC memory usage when typechecking from source vs.
> loading ModIfaces
> >
> >Hi,
> >
> >I'm using the GHC API to typecheck 35,000 modules that form a complicated
> dependency graph (with multiple top-level modules, i.e. there's no single
> "god module" that would transitively depend on everything else), and I
> noticed that peak memory usage is wildly different when everything is done
> from scratch vs. when everything is loaded from files containing ModIfaces:
> 17G vs. 8G. This ratio replicates for smaller samples as well, e.g. 80M vs
> 33M for 407 modules.
> >
> >I'm aware of https://gitlab.haskell.org/ghc/ghc/-/issues/13586<
> https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/issues/13586__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuRzNhyzs$>
> and so when I finish typechecking a module, I take the resulting ModIface
> and create the ModDetails that ends up in the HomeUnitGraph from that. My
> understanding of Matt's original GHC fix in
> https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478<
> https://urldefense.com/v3/__https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5478__;!!ASp95G87aa5DoyK5mB3l!5iF7gCTy5JYm4P-C5-UCaCPVmrNbVCvKbNCIy59XuTRWVJOQGbsLyRBEOGuFElS5o9KuEf-sjms$>
> is that it does the same, i.e. it only makes a fresh ModDetails only once
> per module, after the ModIface is ready.
> >
> >But of course that still means that ModDetails can only keep growing as
> more and more parts of it are used for typechecking more and more
> dependants. Could that be the cause? I tried a crude experiment of "putting
> the toothpaste back in the tube" by replacing all ModDetails with a fresh
> one in the HUG after each finished typechecking , but that's a complete
> disaster for memory usage: even for the small 407 module example, the
> memory usage shoots up to 1.5G. I can imagine it's because imported Ids are
> probably not shared anymore between different importer modules.
> >
> >Any ideas on how I could improve memory usage in the from-scratch case,
> so that it's more similar to the from-ModIface case?
> >
> >Thanks,
> >            Gergo
> >
> >
> >----------------------------------------------------------------------
> >This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries at https: //www.sc.com/en/our-locations
> >
> >Where you have a Financial Markets relationship with Standard Chartered
> PLC, Standard Chartered Bank and their subsidiaries (the "Group"),
> information on the regulatory standards we adhere to and how it may affect
> you can be found in our Regulatory Compliance Statement at https: //
> www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //
> www.sc.com/rcs/fm
> >
> >Insofar as this communication is not sent by the Global Research team and
> contains any market commentary, the market commentary has been prepared by
> the sales and/or trading desk of Standard Chartered Bank or its affiliate.
> It is not and does not constitute research material, independent research,
> recommendation or financial advice. Any market commentary is for
> information purpose only and shall not be relied on for any other purpose
> and is subject to the relevant disclaimers available at https: //
> www.sc.com/en/regulatory-disclosures/#market-disclaimer.
> >
> >Insofar as this communication is sent by the Global Research team and
> contains any research materials prepared by members of the team, the
> research material is for information purpose only and shall not be relied
> on for any other purpose, and is subject to the relevant disclaimers
> available at https: //
> research.sc.com/research/api/application/static/terms-and-conditions.
> >
> >Insofar as this e-mail contains the term sheet for a proposed
> transaction, by responding affirmatively to this e-mail, you agree that you
> have understood the terms and conditions in the attached term sheet and
> evaluated the merits and risks of the transaction. We may at times also
> request you to sign the term sheet to acknowledge the same.
> >
> >Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/
> for important information with respect to derivative products.
> >
> >----------------------------------------------------------------------
> >This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our public website.
> >
> >----------------------------------------------------------------------
> >This email and any attachments are confidential and may also be
> privileged. If you are not the intended recipient, please delete all copies
> and notify the sender immediately. You may wish to refer to the
> incorporation details of Standard Chartered PLC, Standard Chartered Bank
> and their subsidiaries together with Standard Chartered Bank’s Privacy
> Policy via our main Standard Chartered PLC (UK) website at sc. com
>
> >_______________________________________________
> >ghc-devs mailing list
> >ghc-devs at haskell.org
> >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20250117/f9fe751a/attachment.html>