<div dir="ltr"><div>Hi Gergo,</div><div><br></div><div>Since this is a non-reproducible issue with your custom code, I think that we've reached the limit of free debugging via email. There are several consultancies which can be employed to investigate complicated issues like this and perform the necessary changes to GHC to make your use-case easier. <br></div><div><br></div><div>If you have a reproducer on a distributed GHC release then it could be investigated. <br></div><div><br></div><div>Cheers,</div><div><br></div><div>Matt</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Apr 1, 2025 at 1:24 PM Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com">Gergo.Erdi@sc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg4071090311710062270">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<p style="font-family:Arial;font-size:9pt;color:rgb(49,113,0);margin:5pt;font-style:normal;font-weight:normal;text-decoration:none" align="Left">
PUBLIC<br>
</p>
<br>
<div>
<div class="m_4071090311710062270WordSection1">
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">OK so this `ModuleGraph` leak has the potential to be a huge explainer because of the large number of ModuleNodes and NodeKey_Modules in the heap profile.
It turns out the reason we had one ModuleGraph per module was because I was populating the ModuleGraph by repeated calls to `extendMG`, where the new edges were created via `mgModSummaries` on the old `ModuleGraph`, and so the new edge list was a thunk pointing
to the old `ModuleGraph`.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">I’ve now changed it to a single `mkModuleGraph` call, which gets rid of the extra ModuleGraphs (see attached profile), but unfortunately, the ModuleNodes/
edge NodeKey_Modules are still not shared. Here’s how I build the graph, trying hard with a Data.Map.Map to ensure only use one NodeKey_Module per each target module:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">```<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">registerModulesToGraph :: GhcMonad m => [(UnitId, ModuleName, [ModuleName])] -> m ()<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">registerModulesToGraph entries = modifySessionM \hsc_env -> do<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> mod_deps <- for entries \(unit, mod_name, deps) -> do<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> mod <- liftIO $ registerToModuleFinder hsc_env unit mod_name<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> let ms = dummyModSummary (hsc_dflags hsc_env) mod deps fingerprint0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> pure (ms, deps)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> let nodes = [ModuleNode (edges deps) ms | (ms, deps) <- mod_deps]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> pure hsc_env{ hsc_mod_graph = mkModuleGraph nodes }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> where<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> edge_map :: Map.Map ModuleName NodeKey<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> edge_map = Map.fromList [(mod_name, NodeKey_Module $ ModNodeKeyWithUid (notBoot mod_name) uid) | (uid, mod_name, _) <- entries]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> edges deps = map (edge_map Map.!) deps<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">registerToModuleFinder :: HscEnv -> UnitId -> ModuleName -> IO Module<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">registerToModuleFinder hsc_env unitId modName = do<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> addHomeModuleToFinder (hsc_FC hsc_env) homeUnit (notBoot modName) location<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> where<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> homeUnit = DefiniteHomeUnit unitId Nothing<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> location = dummyModLocation modName<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">```<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">But, as you can see in the attached profile, I still get way more than 2k ModuleNodes and NodeKey_Modules….<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(225,225,225) currentcolor currentcolor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>>
<br>
<b>Sent:</b> Tuesday, April 1, 2025 5:44 PM<br>
<b>To:</b> Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com" target="_blank">Gergo.Erdi@sc.com</a>><br>
<b>Cc:</b> GHC Devs <<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>>; ÉRDI Gergő <<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>>; Montelatici, Raphael Laurent <<a href="mailto:Raphael.Montelatici@sc.com" target="_blank">Raphael.Montelatici@sc.com</a>>; Dijkstra, Atze <<a href="mailto:Atze.Dijkstra@sc.com" target="_blank">Atze.Dijkstra@sc.com</a>><br>
<b>Subject:</b> [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Hi Gergo,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I looked in the detailed pane, searched for ModuleGraph, hovered my mouse over the "ModuleGraph` constructor, recorded the number of live bytes, divided that by 32.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Matt<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Apr 1, 2025 at 7:04 AM Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com" target="_blank">Gergo.Erdi@sc.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentcolor currentcolor currentcolor rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p style="margin:5pt"><span style="font-size:9pt;font-family:"Arial",sans-serif;color:rgb(49,113,0)">PUBLIC<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">OK scratch that, I was looking at wrong ghc-debug output. Indeed there are 2301 ModuleGraphs in the heap at the end
of typechecking :O</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> </span><u></u><u></u></p>
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;border-color:currentcolor">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Erdi, Gergo
<br>
<b>Sent:</b> Tuesday, April 1, 2025 12:51 PM<br>
<b>To:</b> Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>><br>
<b>Cc:</b> GHC Devs <<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>>; ÉRDI Gergő <<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>>; Montelatici, Raphael Laurent <<a href="mailto:Raphael.Montelatici@sc.com" target="_blank">Raphael.Montelatici@sc.com</a>>;
Dijkstra, Atze <<a href="mailto:Atze.Dijkstra@sc.com" target="_blank">Atze.Dijkstra@sc.com</a>><br>
<b>Subject:</b> Re: GHC memory usage when typechecking from source vs. loading ModIfaces</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">This sounds extremely interesting, but I don’t understand where you are getting this number from! How do you see in
the eventlog HTMLs that I’ve included that there are ~2000 ModuleGraphs? I’ve now tried using ghc-debug to find all ModuleGraph constructors at two points in the run: just before typechecking the first module (after all the extendMG calls) and just after typechecking
the last module, and even in the cold case I only see 1 ModuleGraph before and 13 ModuleGraphs after.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif">Also, what do you mean by “precisely one loaded per interface loaded into the EPS”? Since my repro has 2294 modules,
wouldn’t that mean 2294 ModuleGraphs by that metric? </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif"> </span><u></u><u></u></p>
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;border-color:currentcolor">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>>
<br>
<b>Sent:</b> Saturday, March 29, 2025 1:53 AM<br>
<b>To:</b> Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com" target="_blank">Gergo.Erdi@sc.com</a>><br>
<b>Cc:</b> GHC Devs <<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>>; ÉRDI Gergő <<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>>; Montelatici, Raphael Laurent <<a href="mailto:Raphael.Montelatici@sc.com" target="_blank">Raphael.Montelatici@sc.com</a>>;
Dijkstra, Atze <<a href="mailto:Atze.Dijkstra@sc.com" target="_blank">Atze.Dijkstra@sc.com</a>><br>
<b>Subject:</b> [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Hi Gergo,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I quickly tried building `Cabal` with the master branch. There is precisely 1 ModuleGraph allocated for the home session, and precisely one loaded per interface loaded into the
EPS. No leaky behaviour like you can see in your eventlogs. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It seems there are about 2000 live module graphs in your program, are you doing something with the API to create this many?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Matt<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 28, 2025 at 12:40 PM Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal">HI Gergo,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Do you have a (synthetic?) reproducer? You have probably identified some memory leak. However, without any means to reproduce it becomes very difficult to investigate. I feel like
we are getting into very precise details now, where speculating is not going to be so useful.
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It seems like this is an important thing for you and your company. Is there any budget to pay for some investigation? If that was the case then some effort could be made to create
a synthetic producer and make the situation more robust going into the future if your requirements were precisely understood.
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Matt<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 28, 2025 at 10:12 AM Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com" target="_blank">Gergo.Erdi@sc.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<p class="MsoNormal" style="margin-bottom:12pt">PUBLIC<br>
<br>
Just to add that I get the same "equalizing" behaviour (but in a more "natural" way) if instead of deepseq-ing the ModuleGraph upfront, I just call `hugInstancesBelow` before processing each module. So that's definitely one source of extra memory usage. I wonder
if it would be possible to rebuild the ModuleGraph periodically (similar to the ModDetails dehydration), or if there are references to it stored all over the place from `HscEnv`s scattered around in closures etc. (basically the same problem the HPT had before
it was made into a mutable reference).<br>
<br>
-----Original Message-----<br>
From: ghc-devs <<a href="mailto:ghc-devs-bounces@haskell.org" target="_blank">ghc-devs-bounces@haskell.org</a>> On Behalf Of Erdi, Gergo via ghc-devs<br>
Sent: Friday, March 28, 2025 4:49 PM<br>
To: Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>>; GHC Devs <<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>><br>
Cc: ÉRDI Gergő <<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>>; Montelatici, Raphael Laurent <<a href="mailto:Raphael.Montelatici@sc.com" target="_blank">Raphael.Montelatici@sc.com</a>>; Dijkstra, Atze <<a href="mailto:Atze.Dijkstra@sc.com" target="_blank">Atze.Dijkstra@sc.com</a>><br>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces<br>
<br>
Hi,<br>
<br>
Unfortunately, I am forced to return to this problem. Everything below is now in the context of GHC 9.12 plus the mutable HPT patch backported.<br>
<br>
My test case is typechecking a tree of 2294 modules that form the transitive closure of a single module's dependencies, all in a single process. I have done this typechecking three times, here's what `+RTS -s -RTS` reports for max residency:<br>
<br>
* "cold": With no on-disk `ModIface` files, i.e. from scratch: 537 MB<br>
<br>
* "cold-top": With all `ModIface`s already on disk, except for the<br>
single top-level module: 302 MB<br>
<br>
* "warm": With all `ModIface`s already on disk: 211 MB<br>
<br>
So my stupidly naive question is, why is the "cold" case also not 302 MB?<br>
<br>
In earlier discussion, `ModDetails` unfolding has come up. Dehydrating `ModDetails` in the HPT all the time is disastrous for runtime, but based on this model I would expect to see improvements from dehydrating "every now and then". So I tried a stupid simple
example where after every 100th typechecked module, I run this function on the topologically sorted list of modules processed so far:<br>
<br>
<br>
```<br>
dehydrateHpt :: HscEnv -> [ModuleName] -> IO () dehydrateHpt hsc_env mods = do<br>
let HPT{ table = hptr } = hsc_HPT hsc_env<br>
hpt <- readIORef hptr<br>
for_ mods \mod -> for_ (lookupUDFM hpt mod) \(HomeModInfo iface _details _linkable) -> do<br>
!details <- initModDetails hsc_env iface<br>
pure ()<br>
```<br>
<br>
Buuut the max residency is still 534 MB (see "cold-dehydrate"); in fact, the profile looks exactly the same.<br>
<br>
Speaking of the profile, in the "cold" case I see a lot of steadily increasing heap usage from the `ModuleGraph`. I could see this happening if typechecking from scratch involves more `modulesUnder` calls which in turn force more and more of the `ModuleGraph`.
If so, then maybe this could be worked around by repeatedly remaking the `ModuleGraph` just like I remake the `ModDetails` above. I tried getting rid of this effect by `deepseq`'ing the `ModuleGraph` at the start, with the idea being that this should "equalize"
the three scenarios if this really is a substantial source of extra memory usage. This pushes up the warm case's memory usage to 381 MB, which is promising, but I still see a `Word64Map` that is steadily increasing in the "cold-force-modulegraph" case and
contributes a lot to the memory usage. Unfortunately, I don't know where that `Word64Map` is (it could be any `Unique`-keyed environment...).<br>
<br>
So I am now stuck at this point. To spell out my goal explicitly, I would like to typecheck one module after another and not keep anything more in memory around than if I loaded them from `ModIface` files.<br>
<br>
Thanks,<br>
Gergo<br>
<br>
p.s.: I couldn't find a way in the EventLog output HTML to turn event markers on/off or filter them, so to avoid covering the whole graph with gray lines, I mark only every 100th module.<br>
<br>
<br>
<br>
<br>
From: Matthew Pickering <<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>><br>
Sent: Wednesday, February 12, 2025 7:08 PM<br>
To: ÉRDI Gergő <<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>><br>
Cc: Erdi, Gergo <<a href="mailto:Gergo.Erdi@sc.com" target="_blank">Gergo.Erdi@sc.com</a>>; Zubin Duggal <<a href="mailto:zubin@well-typed.com" target="_blank">zubin@well-typed.com</a>>; Montelatici, Raphael Laurent <<a href="mailto:Raphael.Montelatici@sc.com" target="_blank">Raphael.Montelatici@sc.com</a>>;
GHC Devs <<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>><br>
Subject: [External] Re: GHC memory usage when typechecking from source vs. loading ModIfaces<br>
<br>
You do also raise a good point about rehydration costs.<br>
<br>
In oneshot mode, you are basically rehydrating the entire transitive closure of each module when you compile it, which obviously results in a large amount of repeated work. This is why people are investigating ideas of a persistent worker to at least avoid
rehydrating all external dependencies as well.<br>
<br>
On Mon, Feb 10, 2025 at 12:13 PM Matthew Pickering <mailto:<a href="mailto:matthewtpickering@gmail.com" target="_blank">matthewtpickering@gmail.com</a>> wrote:<br>
Sure, you can remove them once you are sure they are not used anymore.<br>
<br>
For clients like `GHCi` that doesn't work obviously as they can be used at any point in the future but for a batch compiler it would be fine.<br>
<br>
On Mon, Feb 10, 2025 at 11:56 AM ÉRDI Gergő <mailto:<a href="mailto:gergo@erdi.hu" target="_blank">gergo@erdi.hu</a>> wrote:<br>
On Mon, 10 Feb 2025, Matthew Pickering wrote:<br>
<br>
> I wonder if you have got your condition the wrong way around.<br>
><br>
> The only "safe" time to perform rehydration is AFTER the point it can<br>
> never be used again.<br>
><br>
> If you rehydrate it just before it is used then you will repeat work<br>
> which has already been done. If you do this, you will always have a<br>
> trade-off between space used and runtime.<br>
<br>
Oops. Yes, I have misunderstood the idea. I thought the idea was that after loading a given module into the HPT, its ModDetails would start out small (because of laziness) and then keep growing in size as more and more of it are traversed, and thus forced,
during the typechecking of its dependees, so at some point we would want to reset that into the small initial representation as created by initModDetails.<br>
<br>
But if the idea is that I should rehydrate modules when they can't be used anymore, then that brings up the question why even do that, instead of straight removing the HomeModInfos from the HPT?<br>
<br>
----------------------------------------------------------------------<u></u><u></u></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<hr>This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our public website.<br>
<hr>This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries together with Standard Chartered Bank’s Privacy Policy via our main Standard Chartered PLC (UK) website at sc. com<br>
</div>
</div></blockquote></div>