Parallel building multiple targets

John Lato jwlato at gmail.com
Thu Jan 23 03:52:14 UTC 2014 

On Wed, Jan 22, 2014 at 12:25 AM, Simon Marlow <marlowsd at gmail.com> wrote:

> On 05/01/2014 23:48, John Lato wrote:
>
>> (FYI, I expect I'm the source of the suggestion that ghc -M is broken)
>>
>> First, just to clarify, I don't think ghc -M is obviously broken.
>>   Rather, I think it's broken in subtle, unobvious ways, such that
>> trying to develop a make-based project with ghc -M will fail at various
>> times in a non-obvious fashion, at least without substantial additional
>> rules.
>>
>
> If I understand you correctly, you're not saying that ghc -M is broken,
> but that it would be easier to use if it did more.  Right?  Maybe you could
> make specific suggestions?  Saying it is "broken" is a bit FUD-ish.  We use
> it in GHC's build system, so by an existence proof it is certainly not
> broken.
>

This is more-or-less true.  To be a bit more precise, I'm saying that the
raw output from ghc -M is insufficient in several interesting/useful cases.
 I'm also not convinced that it's worth anyone's time to add the necessary
features to ghc -M to cover these cases, or indeed that it's even
necessarily the correct place to fix it.

Some specific shortcomings of ghc -M are:

1.  no support for .cmm file dependencies (though in fairness, --make
doesn't track these either)
2.  no support for Language.Haskell.TH.Syntax.addDependentFile
3.  no support for preprocessing, such as .hsc or #include'd files.  (I
think it would work to write a .hs file that uses -pgmF to call hsc2hs, but
that seems rather non-obvious and I'm not sure it's a good idea).

However, these are all rather obviously fixable as part of the build
system.  For me, the worst problems have to do with cleaning.  If you're
using a Makefile, typically you want to leave intermediate object files
around and only rebuild them when the sources have changed.  However, there
are various issues with ghc batch-mode that make this difficult (e.g.
https://ghc.haskell.org/trac/ghc/ticket/8029 ).  The workarounds to deal
with this are not as straightforward.  The alternative is to live with the
occasional build error that can only be fixed by blowing away the entire
build dir (a remedy that I often need with ghc's source tree, as even make
maintainer-clean doesn't always cut it.  Hopefully my experience here is
unique, but I do not believe it is).

Also, the most common use case seems to be for parallel building of
modules.  As ghc-7.8 provides this with --make, I'd expect the demand for
ghc -M will be greatly reduced.  That's why I'm not certain it's worth the
time it would take to resolve these issues.

Cheers,
John


> Cheers,
> Simon
>
>
>
>   For an example of some of the extra steps necessary to make
>
>> something like this work, see e.g.
>> https://github.com/nh2/multishake (which is admittedly for a more
>> complicated setup, and also has some issues).  The especially
>> frustrating part is, just when you think you have everything working,
>> someone wants to add some other tool to a workflow (hsc2hs, .cmm files,
>> etc), and your build system doesn't support it.
>>
>> ghc --make doesn't allow building several binaries in one run, however
>> if you use cabal all the separate runs will use a shared build
>> directory, so subsequent builds will be able to take advantage of the
>> intermediate output of the first build.  Of course you could do the same
>> without cabal, but it's a convenient way to create a common build
>> directory and manage multiple targets.  This is the approach I would
>> take to building multiple executables from the same source files.
>>
>> ghc doesn't do any locking of build files AFAIK.  Running parallel ghc
>> commands for two main modules that have the same import, using the same
>> working directory, is not safe.  In pathological cases the two different
>> main modules may even generate different code *for the imported module*.
>>   This sort of situation can arise with the IncoherentInstances
>> extension, for example.
>>
>> The obvious approach is of course to make a library out of your common
>> files.  This has the downsides of requiring a bit more work on the
>> developer's part, but if the common files are relatively stable it'll
>> probably lead to the fastest builds of your executables.  Also in this
>> case you could run multiple `ghc --make`s in parallel, using different
>> build directories, since they won't be rebuilding any common code.
>>
>> John L.
>>
>> On Sun, Jan 5, 2014 at 1:47 PM, Sami Liedes <sami.liedes at iki.fi
>> <mailto:sami.liedes at iki.fi>> wrote:
>>
>>     Hi,
>>
>>     I have a Haskell project where a number of executables are produced
>>     from mostly the same modules. I'm using a Makefile to enable parallel
>>     builds. I received advice[1] that ghc -M is broken, but that there
>>     is parallel ghc --make in HEAD.
>>
>>     As far as I can tell, ghc --make does not allow building several
>>     binaries in one run, so I think it may not still be a full replacement
>>     for Makefiles.
>>
>>     However I have a question about ghc --make that is also relevant
>>     without parallel ghc --make:
>>
>>     If I have two main modules, prog1.hs and prog2.hs, which have mutual
>>     dependencies (for example, both import A from A.hs), is it safe to run
>>     "ghc --make prog1" in parallel with "ghc --make prog2"? IOW, is there
>>     some kind of locking to prevent both from building module A at the
>>     same time and interfering with each other?
>>
>>     Is there a good way (either in current releases or HEAD) to build
>>     multiple binaries partially from the same sources in parallel?
>>
>>              Sami
>>
>>
>>     [1]
>>     http://stackoverflow.com/questions/20938894/generating-
>> correct-link-dependencies-for-ghc-and-makefile-style-builds
>>
>>     _______________________________________________
>>     Glasgow-haskell-users mailing list
>>     Glasgow-haskell-users at haskell.org
>>     <mailto:Glasgow-haskell-users at haskell.org>
>>     http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>
>>
>>
>>
>>
>> _______________________________________________
>> Glasgow-haskell-users mailing list
>> Glasgow-haskell-users at haskell.org
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/glasgow-haskell-users/attachments/20140122/13f6c098/attachment-0001.html>

More information about the Glasgow-haskell-users mailing list