Implementing a compilation server

[Domínguez, Facundo]

Thanks Matthew for your pointers.

Since originally posting, I managed to simplify the problem by terminating
the compilation server at the end of a build, which allows to introduce the
assumption that the code doesn't change during the lifetime of the server.

Now, I'm observing that sometimes different compilation requests place the
same package databases at different paths using the -package-db flags. From
the point of view of GHC, it is as if the package databases had been moved
from one location to another. In newer requests, GHC still looks for the
interface files at the old locations, and fails when it doesn't find them.

Another difference between requests is that, even for a same package
database, different interface files are present, depending on what the
module under compilation imports transitively. This is causing failures
sometimes but not always, I still need to pin exactly the circumstances.
The error manifests as an attempt to load a missing interface file that is
apparently not transitively needed.

If I understand correctly, all the packages pointed with -package-id and
-package-db end up in the EPS. And this means that we can't expect to
update the locations of the interface files without discarding and
repopulating the EPS, correct? I'm thinking of this as approximately as
costly as restarting the compilation server.

I can reasonably ensure that package databases aren't moved around between
compilation requests. But from the standpoint of the build system, it would
require some compromises to demand that all of the interface files of a
package be available even when not all of them are transitively imported.
Can we hope to have GHC cope with this dynamic membership of modules to
Haskell packages during the build? Is this an ability that 8.10.7 already
has?

Thanks,
Facundo

On Thu, May 5, 2022 at 5:13 AM Matthew Pickering <
matthewtpickering at gmail.com> wrote:

> Hi Facundo
>
> Some pointers...
>
> 1. Only put things in the EPS if they are not going to change
> throughout the whole compilation
> 2. Treat everything which can change as a home package
> 2a. I suppose you have performed your own dependency analysis, so
> build your own `ModGraph` and start looking from `load'`, you might
> just want to call `upsweep_mod/compileOne'` directly yourself.
> 2b. I suppose you are NOT targeting 9.4.1 yet, but that will make
> things easier as you can use support for multiple home packages,
> otherwise you will get into severe difficulties if you load a package
> you later want to compile into the EPS. The only thing you can do here
>       is restart the compilation session I think.
> 3. To my knowledge, there is no issue using different -this-unit-id in
> the same session. Not sure what errors you have seen.
> 4. You need to use --make mode rather than -c (oneshot) because
> oneshot mode loads all interfaces into the EPS (see point 1)
>
> ghcide is the closest program to this kind of compilation server you
> imagine so you can look at how that uses the GHC API.
>
> Cheers,
>
> Matt
>
>
> On Thu, May 5, 2022 at 1:06 AM Domínguez, Facundo
> <facundo.dominguez at tweag.io> wrote:
> >
> > Dear ghc devs,
> >
> > I'm using the ghc API to write a compilation server (a.k.a. persistent
> worker). The idea is to serve requests to compile individual modules. In
> this fashion, we can compile modules with different compilation flags and
> yet pay only once for the startup overheads of the compiler.
> >
> > One challenge of this approach is to reuse as much as possible from the
> ghc API session/environment from one compilation request to the next, so we
> save the trouble of reconstructing it each time. This message is to ask for
> advise on how to better accomplish this reuse.
> >
> > I tried reusing the whole environment for multiple requests, but I'm
> conjecturing that this might cause troubles when the requests require
> building modules with different values of -this-unit-id. Another problem
> that streams from this is that recompiling a module which defines a type
> class instance fails because it encounters in the environment the type
> class instance from the
> > previous compilation.
> >
> > My work-in-progress implementation is here [1]. There appears to be
> multiple ways to compile a module in the API, so far I have been trying
> DriverPipeline.compileFile.
> >
> > My best lead right now is to look for inspiration in how GHCi implements
> the load command, but this does a sort of --make compilation while I'm
> going here for the one-shot style.
> >
> > Thanks in advance,
> > Facundo
> >
> > [1]
> https://github.com/tweag/rules_haskell/blob/16ba422457ea4daa5dbf40d46327ebcb20588e97/tools/haskell_module_worker/src/Compile.hs#L188
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20220529/4c148dc9/attachment.html>