Host-Oriented Template Haskell

[Ericson, John]

Thanks for filing the issue Edward!

Moritz I hope you are right. I suspect most of the autoconf stuff
relating to this exists because GCC and binutils are single-target by
default, whereas LLVM is (always?) multi-target. Even the other
targets may also benefit if LLD can be used instead of binutils--LLD
was WIP last I checked, but I assume like the rest of LLVM it plans to
be multi-target.

Insofar as configuration and the build system might be the biggest
headache in bringing this about, I hope the shaking-up-ghc work will
be very useful.

John

On Wed, Jan 20, 2016 at 11:20 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
> I've filed a Trac ticket for this: https://ghc.haskell.org/trac/ghc/ticket/11470#ticket
>
> Edward
>
> Excerpts from Moritz Angermann's message of 2016-01-20 22:55:36 -0800:
>> Colour me an interested party!  For iOS one currently builds two
>> different ghc compiler.  One for the simulator (i386) and one for the
>> actual device (arm).  My initial work on bringing ghcjs’s TH support,
>> (which basically ships the TH out to a process on the target to compile
>> and brings back the compiled TH into the cross compiler) to ghc, was
>> based on the idea of having a TH handler plugin, which would handle the
>> TH parts, and could then be adapted for different TH situations.  This
>> of course could only work if the plugin was compiled with the same version
>> of ghc, that the cross compiler was.
>>
>> I wonder if the multi-target pipeline might be simpler to achieve with
>> the llvm backend.
>>
>> If someone is taking a shot at this, I’d like to help!
>>
>> Cheers,
>>  Moritz
>>
>> > On Jan 21, 2016, at 2:44 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
>> >
>> > Yes, in principle, a compiler that supports cross-compiling to many
>> > targets without needing to be configured (at build time) which
>> > target it should support is ideal.  Unfortunately, currently GHC
>> > relies on autoconf to determine all of the necessary parameters
>> > for a cross-compile.  I am not sure how feasible it would be to
>> > allow these parameters to be configured at runtime: it might be
>> > easy, it might be hard!  (And of course you have to arrange
>> > to have cross-compiled all the boot libraries too...)
>> >
>> > In any case, I think that a GHC that permits changing cross-compilation
>> > targets at runtime is a fine goal, and if it is supported, then we
>> > would only have to make adjustments to GHC internally to distinguish
>> > between the interfaces/objects for various targets (including itself.)
>> >
>> > Edward
>> >
>> > Excerpts from Ericson, John's message of 2016-01-20 21:23:28 -0800:
>> >> Oops, Sent just to Edward by mistake.
>> >>
>> >> ----
>> >>
>> >> That all sounds good to me---I think what I was thinking all along in
>> >> fact, unless I am still missing some nuance :).
>> >>
>> >> My idea of using a multi-target compiler was to avoid needing stage0
>> >> to be the same version as stage1 (or adding a new stage, stage1 as in
>> >> your alternative). The cross compiler is multi-target so it doesn't
>> >> need to link another compiler to load plugins / run "host-oriented
>> >> Template Haskell". I mentioned the usecase of trying to build a stage2
>> >> which needs plugins / Host-Oriented Template Haskell, but it could
>> >> well apply to building anything else with the need for those to things
>> >> (and not normal TH, which cannot be built with a cross compiler)
>> >>
>> >> In diagrams, rather than:
>> >>
>> >>> stage0 (version foo, platform-x -> platform-x)
>> >>> ==> stage1 (version foo, platform-x -> platform-y)
>> >>> ==> some program (with plugins, with host-oriented TH, without normal TH)
>> >>
>> >> or
>> >>
>> >>> stage0 (version bar, platform-x -> platform-x)
>> >>> ==> stage1 (version foo, platform-x -> platform-x) ->
>> >>> ==> stage2 (version foo, platform-x -> platform-y) ->
>> >>> ==> some program (with plugins, with host-oriented TH, without normal TH)
>> >>
>> >> do just
>> >>
>> >>> stage0 (version bar, platform-x -> platform-x)
>> >>> ==> stage1 (version foo, platform-x -> {platform-x, platform-y})
>> >>> ==> some program (with plugins, with host-oriented TH, without normal TH)
>> >>
>> >> Now it may be that building such a multi-target compiler today is just
>> >> as arduous as making sure one has a native compiler of the same
>> >> version before building the cross compiler, so my idea is of no
>> >> practical value. But if so, I hope in the long term that could be
>> >> fixed.
>> >>
>> >> John
>> >>
>> >> On Wed, Jan 20, 2016 at 8:37 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
>> >>> Hello John,
>> >>>
>> >>> Almost!
>> >>>
>> >>> As you point out, if you just want a cross-compiler, you don't
>> >>> need to build stage2.  The real problem is that we want stage0
>> >>> to be *the same version* as stage1, so that stage0 has the correct
>> >>> API in order to load stage0 libraries which are going to be used
>> >>> for TH, plugins, etc.  (Alternately, build a non-cross-compiler
>> >>> stage1, and then a cross-compiler stage2--NOT a cross compiled
>> >>> stage 2.)
>> >>>
>> >>> Edward
>> >>>
>> >>> Excerpts from Ericson, John's message of 2016-01-19 16:13:24 -0800:
>> >>>> From #11378:
>> >>>>
>> >>>>> Can we actually link against the old version of ghc? Yes we can! All we need is
>> >>>>> (1) for it to have been built with a different IPID, and
>> >>>>> (2) to use module renaming to rename the relevant modules to a different
>> >>>>> module name, so that we can import them without a conflict.
>> >>>>
>> >>>>
>> >>>> If I understand this, you are saying stage1 links stage0 (the normal,
>> >>>> non-cross-compiler used to build stage1), to build stage2? I agree
>> >>>> that should work, but if stage1 is multi-target isn't that level of
>> >>>> complexity not even needed? I'm not too familiar with GHC or its build
>> >>>> system, but a multi-target compiler that doesn't support normal TH
>> >>>> doesn't seem that hard to build in principle.
>> >>>>
>> >>>> John
>> >>>>
>> >>>> On Tue, Jan 19, 2016 at 3:59 PM, Ericson, John <john_ericson at brown.edu> wrote:
>> >>>>> Ah thanks for the link. Template Haskell with only {-# TH-ONLY #-}
>> >>>>> imports is precisely equivalent to what I propose. I find a clean
>> >>>>> separation with that and normal TH useful, however, precisely so the
>> >>>>> stage2 compiler's source can include such a thing.
>> >>>>>
>> >>>>> On Tue, Jan 19, 2016 at 3:20 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
>> >>>>>> Hello John
>> >>>>>>
>> >>>>>> You may find this implementation ticket of interest:
>> >>>>>> https://ghc.haskell.org/trac/ghc/ticket/11378
>> >>>>>>
>> >>>>>> Edward
>> >>>>>>
>> >>>>>> Excerpts from Ericson, John's message of 2016-01-19 15:15:15 -0800:
>> >>>>>>> Cross-posting this as was suggested on the Haskell-Cafe list. While I
>> >>>>>>> envision this as normal feature that anyone can use, in practice its
>> >>>>>>> biggest consumers would be GHC devs.
>> >>>>>>>
>> >>>>>>> John
>> >>>>>>>
>> >>>>>>> ---------- Forwarded message ----------
>> >>>>>>> From: Ericson, John <john_ericson at brown.edu>
>> >>>>>>> Date: Tue, Jan 19, 2016 at 1:17 PM
>> >>>>>>> Subject: Host-Oriented Template Haskell
>> >>>>>>> To: Haskell-Cafe <haskell-cafe at haskell.org>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> As is well known, TH and cross-compiling do not get along. There are
>> >>>>>>> various proposals on how to make this interaction less annoying, and I
>> >>>>>>> am not against them. But as I see it, the problem is largely inherent
>> >>>>>>> to the design of TH itself: since values can (usually) be lifted from
>> >>>>>>> compile-time to run-time, and normal definitions from upstream modules
>> >>>>>>> to downstream modules' TH, TH and normal code must "live in the same
>> >>>>>>> world".
>> >>>>>>>
>> >>>>>>> Now this restriction in turn bequeaths TH with much expressive power,
>> >>>>>>> and I wouldn't advocate getting rid of it. But many tasks do not need
>> >>>>>>> it, and in some cases, say in bootstrapping compilers[1] themselves,
>> >>>>>>> it is impossible to use TH because of it, even were all the current
>> >>>>>>> proposals implemented.
>> >>>>>>>
>> >>>>>>> For these reason, I propose a new TH variant which has much harsher
>> >>>>>>> phase separation. Normal definitions from upstream modules can not be
>> >>>>>>> used, lifting values is either not permitted or is allowed to fail
>> >>>>>>> (because of missing/incompatible definitions), and IO is defined to
>> >>>>>>> match the behavior of the host, not target, platform (in the cross
>> >>>>>>> compiling case). The only interaction between the two phases is that
>> >>>>>>> quoted syntax is resolved against the the run-time phase's definitions
>> >>>>>>> (just like today).
>> >>>>>>>
>> >>>>>>> Some of you may find this a shoddy substitute for defining a subset of
>> >>>>>>> Haskell which behaves identically on all platforms, and optionally
>> >>>>>>> constraining TH to it. But the big feature that my proposal offers and
>> >>>>>>> that one doesn't is to be able to independently specify compile-time
>> >>>>>>> dependencies for the host-oriented TH---this is analogous to the
>> >>>>>>> newish `Setup.hs` dependencies. That in turns leads to what I think is
>> >>>>>>> the "killer app" for Host-Oriented TH: exposing the various
>> >>>>>>> prepossessors we use (alex, happy, hsc2hs, even CPP) into libraries,
>> >>>>>>> and side-stepping any need for "executable dependencies" in Cabal.
>> >>>>>>> Note also that at least hsc2hs additionally requires host-IO---there
>> >>>>>>> may not even exist a C compiler on the target platform at all.
>> >>>>>>>
>> >>>>>>> Finally, forgive me if this has been brought up before. I've been
>> >>>>>>> thinking about this a while, and did a final pass over the GHC wiki to
>> >>>>>>> make sure it wasn't already proposed, but I could have missed
>> >>>>>>> something (this is also my first post to the list).
>> >>>>>>>
>> >>>>>>> John
>> >>>>>>>
>> >>>>>>> [1]: https://github.com/ghcjs/ghcjs/blob/master/lib/ghcjs-prim/GHCJS/Prim/Internal/Build.hs
>> >>>>>> _______________________________________________
>> >>>>>> ghc-devs mailing list
>> >>>>>> ghc-devs at haskell.org
>> >>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> > _______________________________________________
>> > ghc-devs mailing list
>> > ghc-devs at haskell.org
>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>> —————————————————
>> Moritz Angermann
>> +49 170 54 33 0 74
>> moritz at lichtzwerge.de
>>
>> lichtzwerge GmbH
>> Raiffeisenstr. 8
>> 93185 Michelsneukirchen
>>
>> Amtsgericht Regensburg HRB 14723
>> Geschäftsführung: Moritz Angermann, Ralf Sangl
>> USt-Id: DE291948767
>>
>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
>> E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den
>> Absender und vernichten Sie diese Mail.
>> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail
>> ist nicht gestattet.
>> This e-mail may contain confidential and/or privileged information.
>> If you are not the intended recipient (or have received this e-mail in
>> error) please notify the sender immediately and destroy this e-mail.
>> Any unauthorized copying, disclosure or distribution of the material in
>> this e-mail is strictly forbidden.