[GHC] #10871: Implement "fat" interface files which can be directly compiled without source

[GHC]

#10871: Implement "fat" interface files which can be directly compiled without
source
-------------------------------------+-------------------------------------
        Reporter:  ezyang            |                Owner:  ezyang
            Type:  feature request   |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  7.11
      Resolution:                    |             Keywords:  backpack
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by ezyang):

 SM, I think you have convinced me that eventually, we will need a version
 of cabal-install that is Backpack-aware, and has a work item in its
 install plan for each instantiated Backpack package.

 I am also going to claim this implementation of `cabal-install` is not
 going to be "simple".   Let me first state an principle which we have both
 agreed is desirable:

 **It should be possible to build a Backpack unit without Cabal**
 (...assuming that all external dependencies have been built already.)

 This principle forces a few choices:

 **We need a separate file format for Backpack.**  Suppose that we don't
 have a separate file format for Backpack, e.g. that all the Backpack
 information is recorded in a Cabal file (as it was in our older design
 from a year ago.)  GHC can't parse a Cabal file, but it needs to know
 specifically what unit provides any module it wants to import.  How can
 you get this info to GHC?

 1. In our old design, you could pass a pile of `-package` flags
    with renamings to make sure every module name points to the correct
    module from the correct instantiated unit.  Obviously this is
    completely unusable by hand; you'd need a tool which knows about
    Backpack to generate these flags.  So much for using Backpack without
    Cabal!  My advisor complained a lot about this, and that was the
    impetus that lead us down the route to fat interface files.

 2. You could write any Backpack-specific information to its own file and
    pass that to GHC.  This is a lot better! Now the only extra information
 you might need to pass is how the holes are to be instantiated (in case
 the unit is indefinite but you want to compile it).

 **It is GHC, not Cabal, who knows how to instantiate units.**  Simple
 corollary of the above.  And I also assume that we don't want to duplicate
 code,
 so this code for instantiation should only live in one place.

 Notice that in the old design, it was Cabal that had the Backpack smarts!
 But I think this was the wrong choice: GHC should have the Backpack
 smarts.

 **Cabal needs to ask GHC for information about how it instantiated
 units.**
 Since the code for unit instantiation lives in GHC, but each instantiated
 unit needs to get installed to the database (a Cabal concern), it follows
 that Cabal needs to call GHC in order to get information about how units
 are instantiated.

 This is directly relevant to cabal-install.  Traditionally,
 cabal-install takes the package to be built, resolves the dependency
 graph, and then builds the dependency graph (now an install plan) in
 topological order.  But if GHC, not Cabal, does instantiation, we don't
 know what the full set of instantiated units a Backpack package needs
 until after we have configured it.  Essentially, as we process packages
 in the install plan, after we configure, we have to ask GHC to tell us
 what other dependencies are needed, and then splice those dependencies
 into the install plan as extra targets that must be built. This is further
 complicated
 by the fact that the Cabal library may be embedded in a custom Setup
 script; in which case we have to define an interchange format between
 Cabal and cabal-install to communicate this information. I don't think
 this is the wrong way to do it, but it will take work to implement.

 Let me argue for fat interface files one more time. One of my top
 priorities at this stage is to get Backpack working "just enough" so that
 we can roll it out and let users (including me) play with it and use it
 for real applications.  So if there is a way to get thing working today
 (maybe not entirely the Right(TM) way) in a way that doesn't impair our
 ability to make it work right later, I would quite like to take it.  Fat
 interfaces are a way to get Backpack integrated with the Cabal ecosystem
 without having to rewrite cabal-install's install plan runner, make it
 easier to experiment with Backpack, have the property, "If you don't care
 about it, you don't have to", and most importantly, is mostly implemented
 at this point.

 This discussion has also reminded me that there is still an elephant in
 the
 room with regards to recompilation avoidance in Backpack.  In a package
 model, GHC will not recompile if the package key of the depended upon
 package has not changed.  Since we don't generally expect package keys
 to change on an edit/recompile cycle, it's unsound to keep around build
 files from an old compilation.  This means that you end up doing a lot
 of recompiling with Cabal today when you change a package that is
 depended upon by other packages you are building.  In fact, we want
 something like this: we don't *care* about recompilation avoidance
 when we are installing, but if we are edit/rebuilding, we want accurate
 recompilation avoidance for the entire set of packages which we may
 be editing and building.  This sounds exactly like the "local package
 database" that we end up creating when compiling Backpack units using fat
 interfaces.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10871#comment:22>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler