<html><head></head><body style="zoom: 0%;"><div dir="auto">Hi Tristan,<br><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->I wouldn't do this with Core (cf inlining issue and issue associating what you find with source syntax).<!-- tmjah_g_1299e --><br><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->I think you should use the output of the renamer instead. Either with a GHC plugin using `renamedResultAction` or just by dumping the renamed AST (fully qualified) with -ddump-rn-ast -ddump-to-file and grepping for the names you want.<!-- tmjah_g_1299e --><br><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->Cheers,<!-- tmjah_g_1299e --><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->Sylvain <!-- tmjah_g_1299e --><br></div>
<div class="gmail_quote" >Le 9 août 2023, à 21:07, Tristan Cacqueray <<a href="mailto:tdecacqu@redhat.com" target="_blank">tdecacqu@redhat.com</a>> a écrit:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="blue"><br>On Mon, Jul 31, 2023 at 16:26 Tristan Cacqueray wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> On Mon, Jul 31, 2023 at 11:05 David Christiansen via ghc-devs wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"> Dear GHC devs,<br><br> I think that having automated security advisory warnings from build tools<br> is important for Haskell adoption in certain industries. This can be done<br> based on build plans, but a package is really the wrong granularity - a<br> large, widely-used package might export a little-used definition that is<br> the subject of an advisory, and it would be good to warn only the users of<br> said definition (cf base and readFloat).<br><br> Tristan is exploring using HIE files to do this check, but I don't know if<br> you read Discourse, where he posted the question:<br> <a href="https://discourse.haskell.org/t/rfc-using-hie-files-to-list-external-declarations-for-cabal-audit/7147">https://discourse.haskell.org/t/rfc-using-hie-files-to-list-external-declarations-for-cabal-audit/7147</a></blockquote><br><br> Thank you David for bringing this up here. One thing to note is that we<br> would need hie files for ghc libraries, as proposed in:<br> <a href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/1337">https://gitlab.haskell.org/ghc/ghc/-/merge_requests/1337</a><br><br> Cheers,<br> -Tristan<br></blockquote><br>Dear GHC devs,<br><br>To recap, the goal of this project is to check if a given declaration is<br>used by a package. For example, I would like to check if such<br>definition: "package:<a href="http://Module.name">Module.name</a>" is reachable from another module.<br><br>In this post I list the considered options, and raise some questions<br>about using the simplified core from .hi files. <br><br>I would appreciate if you could have a look and help me figure out the<br>remaining blockers. Note that I'm not very familiar with the GHC<br>internals and how to properly read Core expressions, so any feedback<br>would be appreciated.<br><br><br># Context and Problem Statement<br><br>We would like to check if a package is affected by a known<br>vulnerability. Instead of looking at the build dependencies names and<br>versions, we would like to search for individual functions. This is<br>particularly important to avoid false alarm when a given vulnerability<br>only appears in a rarely used declaration of a popular package. <br><br>Therefor, we need a way to search the whole call graph to assert with<br>confidence that a given declaration is not used (e.g. reachable).<br><br><br># Considered Options<br><br>To obtain the call graph data, the following options are considered:<br><br>* .hie files produced when using the `-fwrite-ide-info` flag.<br>* .modpack files produced by the [wpc-plugin][grin].<br>* custom GHC plugin.<br>* .hi files containing the simplified core when using the<br> `-fwrite-if-simplified-core` flag. <br><br><br># Pros and Cons of the Options<br><br>### Hie files<br><br>This option is similar to what [weeder][weeder] already implements.<br>However this file format is designed for IDE, and it may not be suitable<br>for our problem. For example, RULES, deriving, RebindableSyntax and<br>template haskell are not well captured.<br><br>[weeder]: <a href="https://github.com/ocharles/weeder">https://github.com/ocharles/weeder</a>/<br><br>### Modpack<br><br>This option appears to work, but it seems overkill. I don't think we<br>need to reach for STG representation.<br><br>[grin]: <a href="https://github.com/grin-compiler/ghc-whole-program-compiler-project">https://github.com/grin-compiler/ghc-whole-program-compiler-project</a><br><br>### Custom GHC plugin<br><br>This option enables extra metadata to be collected, but if using the<br>simplified core is enough, then it is just an extra step compared to<br>using .hi files.<br><br>### Hi files<br><br>Using .hi files is the only option that doesn't require an extra<br>compilation artifacts, the necessary files are already part of the<br>packages.<br><br>To collect hie files or files generated by a GHC plugin, ghc/cabal/stack<br>all need some extra work:<br><br>- ghc libraries doesn't ship hie files ([issue!16901](<a href="https://gitlab.haskell.org/ghc/ghc/-/issues/16901">https://gitlab.haskell.org/ghc/ghc/-/issues/16901</a>)).<br>- cabal needs recent changes for hie files ([PR#9019](<a href="https://github.com/haskell/cabal/pull/9019">https://github.com/haskell/cabal/pull/9019</a>)) and plugin artifacts ([PR#8662](<a href="https://github.com/haskell/cabal/pull/8662">https://github.com/haskell/cabal/pull/8662</a>)).<br>- stack doesn't seem to install hie files for global library.<br><br>Moreover, creating artifacts with a plugin for ghc libraries may<br>requires manual steps because these libraries are not built by the<br>end user.<br><br>Therefor, using .hi files is the most straightforward solution.<br><br><br># Questions<br><br>In this section I present the current implementation of<br>[cabal-audit](<a href="https://github.com/TristanCacqueray/cabal-audit">https://github.com/TristanCacqueray/cabal-audit</a>/).<br><br><br>## Collecting dependencies from core<br><br>In the [cabal-audit-core:CabalAudit.Core](<a href="https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-core/src/CabalAudit/Core.hs">https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-core/src/CabalAudit/Core.hs</a>)<br>module I implemented the logic to extract the call graph from core<br>expression into a list of declarations composed of<br> `UnitId:ModuleName.OccName` and their dependencies.<br><br>Here is an example output for the [cabal-audit-test:CabalAudit.Test.User](<a href="https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-test/src/CabalAudit/Test/User.hs">https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-test/src/CabalAudit/Test/User.hs</a>) module:<br><br>```ShellSession<br>$ cabal run -O0 --write-ghc-environment=always cabal-audit-hi -- CabalAudit.Test.User<br>cabal-audit-test:CabalAudit.Test.Inline.fonctionInlined: base:GHC.Num.$fNumInt, base:GHC.Num.-, ghc-prim:GHC.Types.I#<br>cabal-audit-test:CabalAudit.Test.Instance.$fTestClassTea: cabal-audit-test:CabalAudit.Test.Instance.$ctasty1<br>cabal-audit-test:CabalAudit.Test.Instance.$fTestClassCofee: cabal-audit-test:CabalAudit.Test.Instance.$ctasty<br>cabal-audit-test:CabalAudit.Test.Instance.$ctasty: ghc-prim:GHC.Classes.&&, ghc-prim:GHC.Types.True<br>cabal-audit-test:CabalAudit.Test.Instance.$ctasty1: base:GHC.Base.., cabal-audit-test:CabalAudit.Test.Instance.alwaysTrue, ghc-prim:GHC.Classes.not<br>cabal-audit-test:CabalAudit.Test.Instance.alwaysTrue: base:GHC.Base.const, ghc-prim:GHC.Types.True<br>cabal-audit-test:CabalAudit.Test.User.monDoubleDecr: base:GHC.Num.$fNumInt, base:GHC.Num.-, cabal-audit-test:CabalAudit.Test.Inline.fonctionInlined, ghc-prim:GHC.Types.I#<br>cabal-audit-test:CabalAudit.Test.User.useAlwaysTrue: cabal-audit-test:CabalAudit.Test.Instance.Tea, cabal-audit-test:CabalAudit.Test.Instance.$fTestClassTea<br>cabal-audit-test:CabalAudit.Test.User.useCofeeInstance: cabal-audit-test:CabalAudit.Test.Instance.Cofee, cabal-audit-test:CabalAudit.Test.Instance.$fTestClassCofee<br>```<br><br>This appears correct, in particular:<br><br>- Type class instances are uniquely identified (that was not working well when using a custom plugin).<br>- Inlined declaration are not inlined in the simplified core when built with `-O0`.<br><br>However this is collecting extra definitions that are not part of the<br>source file. I understand that '$fTestClassTea' means the 'TestClass'<br>instance of 'Tea'. But it seems like the actual implementation is behind<br>the extra '$ctasty' declaration. Moreover, when analyzing the other test<br>modules, I see many declarations named 'lvlXX', which I guess are local<br>names that have been floated out.<br><br>This is not ideal because the resulting graph contains extra edges that<br>are not relevant for the end user. I tried to tidy this using<br>'isExportedId' and 'idDetails' from 'GHC.Types.Var' but I worry that<br>this not a good strategy. So my question is: how to recover the original<br>declarations context of core expressions, so that the resulting<br>dependency graph only contains edges that are part of the source<br>declaration? I assume this can be done by dissolving the declarations<br>starting with '$' or 'lvl', but it would be good to know how to do that<br>reliably. <br><br><br>## Handling inlined declaration<br><br>When compiling with `-O1`, declarations seem to be inlined in the<br>simplified core. In that case, is it possible to recover the original<br>inlined OccName?<br><br>If not, I guess we have to use a GHC plugin.<br>I investigated this strategy in [cabal-audit-plugin:CabalAudit.Plugin](<a href="https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-plugin/src/CabalAudit/Plugin.hs">https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-plugin/src/CabalAudit/Plugin.hs</a>). <br>However I am not sure this is done correctly and I could use some <br>guidances on how to proceed.<br><br><br>## Loading hidden module<br><br>If I understand correctly, accessing the ModIface mi_extra_decls to get<br>the simplified core requires an HscEnv. <br>In the [cabal-audit-hi:GhcExtras](<a href="https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-hi/src/GhcExtras.hs">https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-hi/src/GhcExtras.hs</a>)<br>module, I put together the following helpers using GHC as a library:<br><br>```haskell<br>-- | Setup a Ghc session using the packages found in the local environment file<br>runGhcWithEnv :: Ghc a -> IO a<br><br>-- | Lookup a module and extract the simplified core.<br>getCoreBind :: ModuleName -> Maybe FastString -> Ghc (Maybe (Module, [CoreBind]))<br>```<br><br>However this doesn't work for hidden modules, trying to load them with<br>'GHC.lookupModule' fails with this error:<br><br>```ShellSession<br> Could not load module `GHC.Event.Thread'<br> it is a hidden module in the package `base-<a href="http://4.18.0.0">4.18.0.0</a>'<br>```<br><br>I tried to reset the hsc_env.hsc_dflags.hiddenModules but without luck.<br>Is there a trick to access the ModIface of hidden modules?<br><br><br>## Including simplified core in .hi files by default<br><br>In the cabal-audit flake, I am using a nix override to set the<br>`-fwrite-if-simplified-core` ghc-options by default and to patch the ghc<br>build phase to use the `+hi_core` hadrian transformers.<br><br>To avoid rebuilding the dependencies, it would be great to have the<br>simplified core in the hi file by default.<br>Is there an issue or a downside when enabling the flag by default?<br>Could the libraries shipped with GHC contains the simplified core in the<br>future?<br><br><br>## Declaration identifications<br><br>In the [cabal-audit-command:CabalAudit.Command](<a href="https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-command/src/CabalAudit/Command.hs">https://github.com/TristanCacqueray/cabal-audit/blob/main/cabal-audit-command/src/CabalAudit/Command.hs</a>)<br>module, I implemented a proof of concept reverse lookup to find<br>reachable declarations. For example using this command:<br><br>```ShellSession<br>$ cabal-audit-hi --target GHC.Exception.throw CabalAudit.Test.Simple<br>base:GHC.Exception.throw<br>|<br>`- base:GHC.IO.Handle.Internals.ioe_finalizedHandle<br> |<br> `- base:GHC.IO.Handle.FD.$wstdHandleFinalizer<br> |<br> `- base:GHC.IO.Handle.FD.stdout<br> |<br> +- base:System.IO.putStrLn1<br> | |<br> | `- base:System.IO.putStrLn<br> | |<br> | `- cabal-audit-test:CabalAudit.Test.Simple.afficheNombre<br> |<br> `- base:System.IO.putStr1<br> |<br> `- base:System.IO.putStr<br> |<br> `- cabal-audit-test:CabalAudit.Test.Simple.maFonction<br>```<br><br>In the event a vulnerability happens in a type class instance, how to<br>identify the affected instance?<br>Instead of using 'package:Module.$fClassNameDataName', is there an<br>established format we could use (for example "Typeclass X instance of T").<br><br>What about data types or type families, would it makes sense to include<br>them in the graph? If so, how to identify them in the advisory database?<br><br><br>Please let me know if I miss something.<br>Thanks for your time!<br>-Tristan<br></pre><pre class="blue"><hr><br>ghc-devs mailing list<br>ghc-devs@haskell.org<br><a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a><br></pre></blockquote></div></body></html>