Hadrian

[Andrey Mokhov]

Hi Mathieu,

Yes, in principle we can merge right now, as long as it's clear that Hadrian still requires more work before taking over.

My only concern is that merging will make it more difficult for us to quickly iterate on Hadrian: the GitHub workflow is more convenient (at least for me) than the Phabricator one. Perhaps, we can keep Hadrian on GitHub as a submodule? This also has the advantage that we can keep all existing references to GitHub issues/PRs without migrating everything to GHC Trac. It would be very unfortunate to lose all history during the merge.

Cheers,
Andrey

-----Original Message-----
From: Boespflug, Mathieu [mailto:m at tweag.io] 
Sent: 19 October 2017 19:21
To: Andrey Mokhov <andrey.mokhov at newcastle.ac.uk>
Cc: Ben Gamari <ben at well-typed.com>; Manuel M T Chakravarty <manuel.chakravarty at tweag.io>; Moritz Angermann <moritz.angermann at gmail.com>; Jonas Pfenniger Chevalier <jonas.chevalier at tweag.io>; ghc-devs <ghc-devs at haskell.org>
Subject: Re: Hadrian

Hi all,

As a user who tried to be an early adopter of Hadrian, I can attest to
Andrey's remarks about GHC progress sometimes (frequently?) breaking
Hadrian.

Ben, Andrey - how about this strawman proposal:

we merge Hadrian into GHC *now*, not because ./validate with Hadrian
works (it doesn't yet), but because the build of GHC succeeds with
Hadrian. The resulting compiler might well be garbage. But that's fine
- we can iterate in the official ghc repo, all the while knowing that
CI has our back if ever some change makes Hadrian no longer even build
a compiler. Once ./validate passes with Hadrian, the guarantee that CI
gives will become stronger still: we'll know if any change would make
the Hadrian-produced compiler garbage again.

Maybe that does sound totally bonkers to you? :) Maybe it's radical,
but sounds to me like the best way to get the benefit *now* of
ensuring Make and Hadrian move forward in lock step thanks to CI.

The above all assumes that we're committed to Hadrian being the future
of GHC's build system. But I take it that's a given by now.

Best,

Mathieu


On 19 October 2017 at 19:05, Andrey Mokhov
<andrey.mokhov at newcastle.ac.uk> wrote:
> Hi Ben, Manuel and all,
>
> Ben has already covered most questions in his answer, but let me add a couple of comments.
>
>> So, Mathieu had the clever idea that having the two build system in
>> GHC side-by-side and then build in CI both ways might be a good way of
>> making sure that keep constant tabs on Hadian and get a clear (and
>> continuous picture) of where it is and what is missing.
>
> It would be great to build every GHC commit both by Make and Hadrian. Not only this would allow to monitor the readiness of Hadrian, it would also make it easier for us to catch up with changes in GHC and the Make build system. At the moment, Hadrian is frequently broken by changes in GHC/Make and we need to do detective work of figuring out which commit broke what and fix Hadrian accordingly. These fixes are trivial in many cases (e.g. adding a new flag to one of the builders) so GHC developers would probably be able to easily mirror these changes in Hadrian if only their commits were marked as Hadrian-breaking by the common CI.
>
>> In other words, why can’t we have Hadrian *today*?
>
> I'd say the biggest issue is #299 -- i.e. making sure that the GHC built by Hadrian passes validation. At the moment we still have 100-200 unexpected failures, most of which probably have to do with outdated command line flags (GHC's build system continuously evolves and it wasn't possible to keep track of all flag changes over time, so there are certainly some differences in Hadrian that lead to validation failures). This is where help is currently needed.
>
> Cheers,
> Andrey
>
> -----Original Message-----
> From: Ben Gamari [mailto:ben at well-typed.com]
> Sent: 19 October 2017 15:08
> To: Manuel M T Chakravarty <manuel.chakravarty at tweag.io>
> Cc: Mathieu Boespflug <m at tweag.io>; Moritz Angermann <moritz.angermann at gmail.com>; Jonas Pfenniger Chevalier <jonas.chevalier at tweag.io>; Andrey Mokhov <andrey.mokhov at newcastle.ac.uk>
> Subject: Re: Hadrian
>
> CCing Andrey and Zhen, who are the principle authors of Hadrian.
>
> Manuel M T Chakravarty <manuel.chakravarty at tweag.io> writes:
>
>> Hi Ben,
>>
>> I our exploration of cross-compilation for GHC and the CI integration,
>> we talked with Moritz and got to the topic of Hadrian. It seems that
>> there are few interdependent issues here, and I am really interested
>> in getting your take on them.
>>
>> * Hadrian is slated for inclusion for 8.4, but I couldn’t find any
>> tickets except #12599. Is this because Hadrian has its own issue
>> tracker on GitHub? How can I get an overview over what’s missing in
>> getting Hadrian into GHC?
>>
> Right, Hadrian tickets are currently tracked in its own issue tracker.
>
>> * For Moritz’ work on ARM and also for CI cross-compiling is
>> important, but Moritz pointed out rightly that it seems wasteful put
>> much time into improving this in the make-based build system if that
>> is going to go soon anyway. However, because Hadrian is not in yet,
>> doing the work on Hadrian now also doesn’t really make sense.
>>
> Indeed. Hadrian is one of the things I am currently working on for
> precisely this reason.
>
>> * Obviously, one issue with Hadrian is stability and feature
>> completeness. This has ramifications on our desire to really get 8.4
>> out of the door in February and not delay the release and also get the
>> CI done, because good CI is the best way of making sure the Hadrian
>> integration goes smoothly. (There is obviously a cyclic dependence
>> between this point and the previous.)
>>
>> So, Mathieu had the clever idea that having the two build system in
>> GHC side-by-side and then build in CI both ways might be a good way of
>> making sure that keep constant tabs on Hadian and get a clear (and
>> continuous picture) of where it is and what is missing. It would also
>> allow Moritz to do cross-compilation work right there on Hadrian.
>>
>> In other words, why can’t we have Hadrian *today*?
>>
> At this point the plan is for Hadrian and the Make-based build system to
> coexist in 8.4, with the make system serving as a backup.
>
> Well, we could (and potentially should) merge Hadrian
>
>> If the limiting factor is resources, maybe we can do something about
>> that, but it would require a good solid plan that we can execute
>> perfectly when given the opportunity.
>>
> The blocking issues in Hadrian are tagged with the "GHC Merge" project
> in the issue tracker. The overall themes currently are,
>
>  * Fixing validation support (finishing #187)
>
>  * Support the remaining test targets (#360)
>
>  * Documentation, so we don't overwhelm Andrey with questions when
>    things break. (#308)
>
>  * Finish and test install rules (#318, #373, #379)
>
>  * bindist install (the last part of #219). I started working on this,
>    although for the moment bugs and 8.2.2 have taken priority.
>
> Andrey, do correct me if I've missed something.
>
> Help is of course always welcome!
>
> Cheers,
>
> - Ben
>