GHC memory usage when typechecking from source vs. loading ModIfaces

[ÉRDI Gergő]

I'll get back to you with details as I can.

But one thing I noticed is that contrary to my expectations, when 
replacing the ModDetails with fresh ones, the real cost comes not from the 
fact that we have to re-evaluate the various parts of the ModDetails, but 
rather the call to `fillModDetails` itself!

In other words, I can replicate most of the slowdown by calling 
`fillModDetails` on the interfaces in the HPT but throwing its result away 
and keeping the HPT as is. This sounds like bad news to me.

Your remark that there are HomeModInfos referenced outside the HPT 
confuses me. I thought the whole reason this could work, and the whole 
reason I can see memory usage improvement, is because the HomeModInfos are 
not stored outside the HPT? If they are, home come my replacing them in 
the HPT with fresh ModDetails helps with memory usage instead of making it 
worse?

On Thu, 23 Jan 2025, Matthew Pickering wrote:

> That's good news.
> 
> I don't think the first idea will do very much as there are other references to the
> final "HomeModInfo" not stored in the HPT.
> 
> Have you constructed a time profile to determine why the runtime is higher? With the
> second approach you are certainly trading space usage for repeating work.
> 
> If you actually do have a forest, then ideally you would replace the ModDetails after
> it will never be used again.
> 
> You are likely also missing other patches important for memory usage.
> 
> * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12582
> * https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12347
> 
> I can't comment about the 17 HPT, what do the retainer stacks look like in ghc-debug?
> 
> PS.  Please use eventlog2html so the profiles are readable! You can use it on .hp
> profiles.
> 
> Cheers,
> 
> Matt