-O* does more than what's in optLevelFlags?

[Erdi, Gergo]

PUBLIC


PUBLIC

https://gitlab.haskell.org/ghc/ghc/-/issues/20500

I tried assigning it to myself but failed; I guess I don’t have enough entitlements even to pick up tickets :/

From: Simon Peyton Jones <simonpj at microsoft.com>
Sent: Monday, October 11, 2021 4:33 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com>
Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com>; 'GHC' <ghc-devs at haskell.org>
Subject: [External] RE: -O* does more than what's in optLevelFlags?


PUBLIC

ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails. Always report suspicious emails using the Report As Phishing button in Outlook to protect the Bank and our clients.

Oh, now I get it.  I misunderstood you.

But the point is, I do *not* want to set optLevel. I want to turn on individual optimizations. But as it stands today, if I turn on any optimizations, that doesn’t do anything unless I *also* set optLevel to >= 1.

Yes I agree.   Nothing should consult `optLevel`; indeed maybe we shouldn’t even record it permanently.

The place in Pipeline that Sylvain identified is most important.

So when we compute the CoreTodos, instead of looking at the optLevel, we’d consult the set of optimizations turned on, and go “yup, at least one of these needs core-to-core simpl passes, so we better get on it”, going to the same branch that current is gated by optLevel >= 1.

Yes I agree with this.   Another alternative might be to kill off the -O0 pipeline entirely (in GHC.Core.Opt.Pipeline.getCoreToDo) and just have one pipeline that does not-very-much when no optimisations are enabled.  This would be more robust (no need to have a list of flags to consult to decide which path to follow), but could have unforeseen consequences (what happens in the -0 pipeline if all optimisations are off?  Same as the -O0 pipeline?).

I’d be happy with either of these changes, if someone wants to offer a patch.    Certainly open a ticket with this discussion so it doesn’t get lost.

Simon

PS: I am leaving Microsoft at the end of November 2021, at which point simonpj at microsoft.com<mailto:simonpj at microsoft.com> will cease to work.  Use simon.peytonjones at gmail.com<mailto:simon.peytonjones at gmail.com> instead.  (For now, it just forwards to simonpj at microsoft.com<mailto:simonpj at microsoft.com>.)

From: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Sent: 11 October 2021 09:21
To: Simon Peyton Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>>
Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; 'GHC' <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: RE: -O* does more than what's in optLevelFlags?


PUBLIC


PUBLIC

But the point is, I do *not* want to set optLevel. I want to turn on individual optimizations. But as it stands today, if I turn on any optimizations, that doesn’t do anything unless I *also* set optLevel to >= 1.

A nicer design in my mind would be if the semantics of optLevel is fully defined in terms of optimization flags. I want “-O1 -fno-strictness” to be exactly equivalent to “-fpolynomial-complexity -fremove-space-leaks -fstrictness -fno-strictness” which of course is equivalent to “-fpolynomial-complexity -fremove-space-leaks -fno-strictness” (assuming that the O1 flag is defined to turn on three flags that ensure polynomial complexity, remove space leaks, and turn on strictness analysis, just as a way of example). So when we compute the CoreTodos, instead of looking at the optLevel, we’d consult the set of optimizations turned on, and go “yup, at least one of these needs core-to-core simpl passes, so we better get on it”, going to the same branch that current is gated by optLevel >= 1.

Would that make sense?

Also, this is NOT just an API documentation issue! The GHC user’s guide itself wrongly claims:

The -O* options specify convenient “packages” of optimisation flags; the -f* options described later on specify individual optimisations to be turned on/off; the -m* options specify machine-specific optimisations to be turned on/off.
[…]
There are many options that affect the quality of code produced by GHC. Most people only have a general goal, something like “Compile quickly” or “Make my program run like greased lightning.” The following “packages” of optimisations (or lack thereof) should suffice.

https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/using-optimisation.html#o-convenient-packages-of-optimisation-flags<https://clicktime.symantec.com/3VfBWn4WsD9Jis7gDmxmv4a7Vc?u=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fdownloads.haskell.org%252F~ghc%252Flatest%252Fdocs%252Fhtml%252Fusers_guide%252Fusing-optimisation.html%2523o-convenient-packages-of-optimisation-flags%26data%3D04%257C01%257Csimonpj%2540microsoft.com%257C5afa7f7726d04c4e8e7008d98c903bd6%257C72f988bf86f141af91ab2d7cd011db47%257C1%257C0%257C637695374886882049%257CUnknown%257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%253D%257C1000%26sdata%3DEE6ARdxQR07lCZCjIPLJKxjIBcvCufBj6yWk8VpFPvE%253D%26reserved%3D0>


I’m happy to submit a documentation patch if there is no way to actually support this model, but I would prefer much more if we can implement what is claimed in the documentation here 😊


From: Simon Peyton Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>>
Sent: Monday, October 11, 2021 4:06 PM
To: Erdi, Gergo <Gergo.Erdi at sc.com<mailto:Gergo.Erdi at sc.com>>
Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>; 'GHC' <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Subject: [External] RE: -O* does more than what's in optLevelFlags?


Is this a documentation bug, an implementation bug (as in, if any of the relevant opts are set, then the CoreToDos should always include the optimization steps selected), or a design bug (there is no way to support this meaningfully)?

Maybe a documentation bug?  You should update the `optLevel` field of `DynFlags` only via calling `setOptLevel`, not by setting it directly.

What other design would make sense?  We want to support ghc -O -fno-strictness, where the -O switches on a bunch of flags, and -fno-strictness turns off strictness.  The order matters.

One difficulty is that I’m not even sure where one would look for that documentation.  We don’t really have a comprehensive GHC User Manual description of the GHC API: Section 7.2 “Using GHC as a library” is vestigial.  I would be Absolutely Fantastic, if someone (Gergo, even) felt able to flesh it out.

Simon

PS: I am leaving Microsoft at the end of November 2021, at which point simonpj at microsoft.com<mailto:simonpj at microsoft.com> will cease to work.  Use simon.peytonjones at gmail.com<mailto:simon.peytonjones at gmail.com> instead.  (For now, it just forwards to simonpj at microsoft.com<mailto:simonpj at microsoft.com>.)

From: ghc-devs <ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>> On Behalf Of Erdi, Gergo via ghc-devs
Sent: 11 October 2021 08:54
To: 'GHC' <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>
Subject: RE: -O* does more than what's in optLevelFlags?


PUBLIC

I’ve done some digging into this, and it turns out the DynFlag’s `optLevel` itself is used at some places, most notably when creating the main [CoreToDo]. So turning on all these flags on their own doesn’t equal setting -On for the right “n”; in fact, currently setting most of these flags does NOTHING on its own unless -On with n>=1 is *also* passed on the command line, and there is no command line flag to *only* turn on Core optimizations in the abstract, without actually turning any specific ones on.

Is this a documentation bug, an implementation bug (as in, if any of the relevant opts are set, then the CoreToDos should always include the optimization steps selected), or a design bug (there is no way to support this meaningfully)?

From: Erdi, Gergo
Sent: Monday, October 11, 2021 12:09 PM
To: 'GHC' <ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>>
Cc: Montelatici, Raphael Laurent <Raphael.Montelatici at sc.com<mailto:Raphael.Montelatici at sc.com>>
Subject: -O* does more than what's in optLevelFlags?


PUBLIC

What is set by -O* that is not included in optLevelFlags?

I would have thought that setting all the flags implied by, e.g., -O1, would be the same as setting -O1 itself. But this is not the case! Here are all the flags for O1 from optLevelFlags:

Opt_DoLambdaEtaExpansion
Opt_DoEtaReduction
Opt_LlvmTBAA
Opt_CallArity
Opt_Exitification
Opt_CaseMerge
Opt_CaseFolding
Opt_CmmElimCommonBlocks
Opt_CmmSink
Opt_CmmStaticPred
Opt_CSE
Opt_StgCSE
Opt_EnableRewriteRules
Opt_FloatIn
Opt_FullLaziness
Opt_IgnoreAsserts
Opt_Loopification
Opt_CfgBlocklayout
Opt_Specialise
Opt_CrossModuleSpecialise
Opt_InlineGenerics
Opt_Strictness
Opt_UnboxSmallStrictFields
Opt_CprAnal
Opt_WorkerWrapper
Opt_SolveConstantDicts
Opt_NumConstantFolding

And here are the ones that are set by O0 (the default) but not by O1:

Opt_IgnoreInterfacePragmas
Opt_OmitInterfacePragmas

So I expected that the following two invocations of GHC would be equivalent:


  1.  ghc -O1
  2.  ghc -fdo-lambda-eta-expansion -fdo-eta-reduction -fllvm-tbaa -fcall-arity -fexitification -fcase-merge -fcase-folding -fcmm-elim-common-blocks -fcmm-sink -fcmm-static-pred -fcse -fstg-cse -fenable-rewrite-rules -ffloat-in -ffull-laziness -fignore-asserts -floopification -fblock-layout-cfg -fspecialise -fcross-module-specialise -finline-generics -fstrictness -funbox-small-strict-fields -fcpr-anal -fworker-wrapper -fsolve-constant-dicts -fnum-constant-folding -fno-ignore-interface-pragmas -fno-omit-interface-pragmas

However, just by observing the output of -dshow-passes, I can see that while -O1 applies all these optimizations, the second version does NOT, even though I have turned on each and every one of them one by one.

Looking at compiler/GHC/Driver/Session.hs, it is not at all clear that -O* should do more than just setting the flags from optLevelFlags. What other flags are implied by -O*?

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations

Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm

Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer.

Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions.

Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same.

Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations

Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm

Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer.

Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions.

Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same.

Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.

This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please delete all copies and notify the sender immediately. You may wish to refer to the incorporation details of Standard Chartered PLC, Standard Chartered Bank and their subsidiaries at https: //www.sc.com/en/our-locations

Where you have a Financial Markets relationship with Standard Chartered PLC, Standard Chartered Bank and their subsidiaries (the "Group"), information on the regulatory standards we adhere to and how it may affect you can be found in our Regulatory Compliance Statement at https: //www.sc.com/rcs/ and Regulatory Compliance Disclosures at http: //www.sc.com/rcs/fm

Insofar as this communication is not sent by the Global Research team and contains any market commentary, the market commentary has been prepared by the sales and/or trading desk of Standard Chartered Bank or its affiliate. It is not and does not constitute research material, independent research, recommendation or financial advice. Any market commentary is for information purpose only and shall not be relied on for any other purpose and is subject to the relevant disclaimers available at https: //www.sc.com/en/regulatory-disclosures/#market-disclaimer.

Insofar as this communication is sent by the Global Research team and contains any research materials prepared by members of the team, the research material is for information purpose only and shall not be relied on for any other purpose, and is subject to the relevant disclaimers available at https: //research.sc.com/research/api/application/static/terms-and-conditions. 

Insofar as this e-mail contains the term sheet for a proposed transaction, by responding affirmatively to this e-mail, you agree that you have understood the terms and conditions in the attached term sheet and evaluated the merits and risks of the transaction. We may at times also request you to sign the term sheet to acknowledge the same.

Please visit https: //www.sc.com/en/regulatory-disclosures/dodd-frank/ for important information with respect to derivative products.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20211011/aa9908c3/attachment-0001.html>