Parallel building multiple targets

John Lato jwlato at gmail.com
Thu Jan 23 18:22:47 UTC 2014 

On Jan 23, 2014 1:28 AM, "Simon Marlow" <marlowsd at gmail.com> wrote:
>
> On 23/01/14 03:52, John Lato wrote:
>
>> However, these are all rather obviously fixable as part of the build
>> system.  For me, the worst problems have to do with cleaning.  If you're
>> using a Makefile, typically you want to leave intermediate object files
>> around and only rebuild them when the sources have changed.  However,
>> there are various issues with ghc batch-mode that make this difficult
>> (e.g. https://ghc.haskell.org/trac/ghc/ticket/8029 ).  The workarounds
>> to deal with this are not as straightforward.  The alternative is to
>> live with the occasional build error that can only be fixed by blowing
>> away the entire build dir (a remedy that I often need with ghc's source
>> tree, as even make maintainer-clean doesn't always cut it.  Hopefully my
>> experience here is unique, but I do not believe it is).
>
>
> You said "various issues", but you've only mentioned *one* specific issue
so far: #8029, and we concluded that was not a bug, although I do see how
it could require manually deleting a .hi file if you have a module that
shadows a package module and then remove it.  This seems a rare occurrence
to me, but perhaps it is something you do often.  If it really hurts, then
you could have a way to tell your build system about a file when it is
removed from the project, so that it can delete the build artifacts that go
with it.

It seems uncommon until you're developing in a branch that does so, and try
to go back and forth between that branch and another.

>
> Anyway, are there other problems you'd like to bring to our attention?

If the one bug I linked to earlier is to be closed as "not a bug" (seems
correct to me), there doesn't seem much point to raising other issues
relating to out-of-date intermediate files.  The general solution is
exactly as you suggested, leading to an increasingly baroque build system.

I don't think any of this detracts from my original thrust, which is that
something that looks like an afternoon's work is much more complicated.
Plus, you'll end up fighting with/hacking on a build system instead of what
you meant to work on.

>
> Cheers,
> Simon
>
>
>
>
>
>> Also, the most common use case seems to be for parallel building of
>> modules.  As ghc-7.8 provides this with --make, I'd expect the demand
>> for ghc -M will be greatly reduced.  That's why I'm not certain it's
>> worth the time it would take to resolve these issues.
>>
>> Cheers,
>> John
>>
>>
>>     Cheers,
>>     Simon
>>
>>
>>
>>        For an example of some of the extra steps necessary to make
>>
>>         something like this work, see e.g.
>>         https://github.com/nh2/__multishake
>>
>>         <https://github.com/nh2/multishake> (which is admittedly for a
more
>>         complicated setup, and also has some issues).  The especially
>>         frustrating part is, just when you think you have everything
>>         working,
>>         someone wants to add some other tool to a workflow (hsc2hs, .cmm
>>         files,
>>         etc), and your build system doesn't support it.
>>
>>         ghc --make doesn't allow building several binaries in one run,
>>         however
>>         if you use cabal all the separate runs will use a shared build
>>         directory, so subsequent builds will be able to take advantage
>>         of the
>>         intermediate output of the first build.  Of course you could do
>>         the same
>>         without cabal, but it's a convenient way to create a common build
>>         directory and manage multiple targets.  This is the approach I
would
>>         take to building multiple executables from the same source files.
>>
>>         ghc doesn't do any locking of build files AFAIK.  Running
>>         parallel ghc
>>         commands for two main modules that have the same import, using
>>         the same
>>         working directory, is not safe.  In pathological cases the two
>>         different
>>         main modules may even generate different code *for the imported
>>         module*.
>>            This sort of situation can arise with the IncoherentInstances
>>         extension, for example.
>>
>>         The obvious approach is of course to make a library out of your
>>         common
>>         files.  This has the downsides of requiring a bit more work on
the
>>         developer's part, but if the common files are relatively stable
>>         it'll
>>         probably lead to the fastest builds of your executables.  Also
>>         in this
>>         case you could run multiple `ghc --make`s in parallel, using
>>         different
>>         build directories, since they won't be rebuilding any common
code.
>>
>>         John L.
>>
>>         On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes at iki.fi
>>         <mailto:sami.liedes at iki.fi>
>>         <mailto:sami.liedes at iki.fi <mailto:sami.liedes at iki.fi>>> wrote:
>>
>>              Hi,
>>
>>              I have a Haskell project where a number of executables are
>>         produced
>>              from mostly the same modules. I'm using a Makefile to
>>         enable parallel
>>              builds. I received advice[1] that ghc -M is broken, but
>>         that there
>>              is parallel ghc --make in HEAD.
>>
>>              As far as I can tell, ghc --make does not allow building
>>         several
>>              binaries in one run, so I think it may not still be a full
>>         replacement
>>              for Makefiles.
>>
>>              However I have a question about ghc --make that is also
>>         relevant
>>              without parallel ghc --make:
>>
>>              If I have two main modules, prog1.hs and prog2.hs, which
>>         have mutual
>>              dependencies (for example, both import A from A.hs), is it
>>         safe to run
>>              "ghc --make prog1" in parallel with "ghc --make prog2"?
>>         IOW, is there
>>              some kind of locking to prevent both from building module A
>>         at the
>>              same time and interfering with each other?
>>
>>              Is there a good way (either in current releases or HEAD) to
>>         build
>>              multiple binaries partially from the same sources in
parallel?
>>
>>                       Sami
>>
>>
>>              [1]
>>
http://stackoverflow.com/__questions/20938894/generating-__correct-link-dependencies-for-__ghc-and-makefile-style-builds
>>         <
http://stackoverflow.com/questions/20938894/generating-correct-link-dependencies-for-ghc-and-makefile-style-builds
>
>>
>>              _________________________________________________
>>              Glasgow-haskell-users mailing list
>>         Glasgow-haskell-users at haskell.__org
>>         <mailto:Glasgow-haskell-users at haskell.org>
>>              <mailto:Glasgow-haskell-users at __haskell.org
>>         <mailto:Glasgow-haskell-users at haskell.org>>
>>         http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users<
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
>>
>>
>>
>>
>>
>>         _________________________________________________
>>         Glasgow-haskell-users mailing list
>>         Glasgow-haskell-users at haskell.__org
>>         <mailto:Glasgow-haskell-users at haskell.org>
>>         http://www.haskell.org/__mailman/listinfo/glasgow-__haskell-users<
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/glasgow-haskell-users/attachments/20140123/cae6db93/attachment.html>

More information about the Glasgow-haskell-users mailing list