DriverPipeline/HscMain DynFlags mystery

[Ömer Sinan Ağacan]

> The question would be easier to answer if it included a bit more context:
>
> - this is in reference to !1304, correct?

This is done in two parts: !1633 and !1969. It's mainly for !1304 and !17004,
but I'm talking about the code already merged here. No need to look at any MRs.

(There will be one more MR on this topic, but it's not relevant for this
problem)

> - specifically which tests fail and in which ways?

I don't remember... There are two places in DriverPipeline that writes interface
files. Currently because of this problem I pass a DynFlags to those sites so
that I can generate interfaces without breaking stuff, e.g.

    HscRecomp { hscs_guts = cgguts,
                hscs_summary = mod_summary,
                hscs_partial_iface = partial_iface,
                hscs_old_iface_hash = mb_old_iface_hash,
                hscs_iface_dflags = iface_dflags } -> do

        ...
        liftIO $ hscMaybeWriteIface if_dflags final_iface mb_old_iface_hash
                                        (ms_location mod_summary)
        ...


Instead of using if_dflags if I use the DynFlags in the current environment (in
`HscEnv`) some tests fail. I'll try to give more details of which tests are
failing.

> - what is the "more stuff" that you are adding?

Currently nothing, but we'll be adding CafInfos after !1304.

> In general when it comes to bugs like this I find it help to reduce the size
> of the patch as much as possible. In your case, the CAF refactor is probably
> quite irrelevant to the issue you are seeing. I would try to extract the
> pipeline refactor that is triggering your bug into a separate MR which can be
> assessed independently from the CAF business.
>
> This may take a few minutes but in my experience this sort of exercise is
> almost always worth the effort. Even if you don't find the bug while splitting
> up the patch it will be significantly easier for others to help with the
> result.

This is the patch:
https://gitlab.haskell.org/ghc/ghc/commit/bbdd54aab2f727bd90efe237eeb72e5e014b0cb2

It's not the smallest patch that demonstrates, but hopefully it's small enough.

Ömer

Ben Gamari <ben at smart-cactus.org>, 1 Kas 2019 Cum, 03:40 tarihinde şunu yazdı:
>
> On October 31, 2019 2:45:09 PM EDT, "Ömer Sinan Ağacan" <omeragacan at gmail.com> wrote:
> >Hi,
> >
> >We recently did some refactoring in HscMain and DriverPipeline to
> >generate
> >interfaces after final Cmms are generated (previously interfaces would
> >be
> >generated after the tidying pass). It's mostly done but there's one
> >thing that I
> >couldn't figure out after two full days of debugging (I also asked a
> >few people
> >about it on IRC), so I wanted to ask here in case someone here knows.
> >
> >Previously the interface value (ModIface) would be generated and
> >written to disk
> >in `HscMain.finish`. The DynFlags we use to generate the ModIface and
> >to write
> >it to disk would be the one passed to `HscMain.hscIncrementalCompile`.
> >
> >In the new implementation part of the interface is still generated in
> >`HscMain.hscIncrementalCompile` (see mkPartialIface), using the same
> >DynFlags as
> >before. But more stuff is added after the final Cmms are generated (see
> >mkFullIface calls in DriverPipeline) using DynFlags in `compileOne'` or
> >`runPhase` (called by `runPipeline`). It turns out these DynFlags are
> >different
> >enough from the one passed to `HscMain.hscIncrementalCompile` that some
> >tests
> >fail (I remember a backpack test, but there may be more).
> >
> >("Full" interfaces are written to disk right after generation)
> >
> >See [1] for the hack I added as a workaround. Basically I keep the
> >DynFlags
> >passed to hscIncrementalCompile so that I can generate the final
> >interfaces
> >correctly.
> >
> >The question is what's changing in DynFlags that makes things go wrong.
> >I tried
> >looking at the fields used by mkFullIface and hscMaybeWriteIface, but
> >as far as
> >I can see none of the fields used by these functions are different from
> >the
> >DynFlags passed to hscIncrementalCompile.
> >
> >If anyone knows what's going on any help would be appreciated.
> >
> >Thanks,
> >
> >Ömer
> >
> >[1]:
> >https://gitlab.haskell.org/ghc/ghc/blob/master/compiler/main/HscTypes.hs#L255-259
> >_______________________________________________
> >ghc-devs mailing list
> >ghc-devs at haskell.org
> >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> The question would be easier to answer if it included a bit more context:
>
> - this is in reference to !1304, correct?
> - specifically which tests fail and in which ways?
> - what is the "more stuff" that you are adding?
>
> In general when it comes to bugs like this I find it help to reduce the size of the patch as much as possible. In your case, the CAF refactor is probably quite irrelevant to the issue you are seeing. I would try to extract the pipeline refactor that is triggering your bug into a separate MR which can be assessed independently from the CAF business.
>
> This may take a few minutes but in my experience this sort of exercise is almost always worth the effort. Even if you don't find the bug while splitting up the patch it will be significantly easier for others to help with the result.
>
> Cheers,
>
> - Ben