Static values language extension proposal

[Tim Watson]

Hi Mathieu,

On 28 Jan 2014, at 12:53, Mathieu Boespflug wrote:
> We would prefer to do it that way, to be honest. As explained in my
> previous email, we identified two problems with this approach:
> 
> 1) User friendliness. It's important for us that Cloud Haskell be
> pretty much as user friendly and easy to use as Erlang is.
> 

Exactly!

>    a) I don't know that it's possible from Template Haskell to detect
> and warn the user when dependent modules have not been compiled into
> dynamic object code or into static code with the right flags.
> 

I don't think that it is, from what I've seen, though I'm by no means an expert.

>    b)  It's very convenient in practice to be able to send not just
> `f` if `f` is a global identifier, but in general `e` where `e` is any
> closed expression mentioning only global names. That can easily be
> done by having the compiler float the expression `e` to the top-level
> and give it a global name. I don't see how to do that in TH in a user
> friendly way.

Agreed.

> 
> 2) A technical issue: you ought to be able to send unexported
> functions across the wire, just as you can pass unexported functions
> as arguments to higher-order functions. Yet GHC does not create linker
> symbols for unexported identifiers, so our approach would break down.
> Worse, I don't think that it's even possible to detect in TH whether
> an identifier is exported or not, in order to warn the user. One could
> imagine a compiler flag to force the creation of linker symbols for
> all toplevel bindings, exported or unexported. But that seems
> wasteful, and potentially not very user friendly.

Interesting.

> 
> If the above can be solved, all the better!
> 
> If not: we don't always want to touch the compiler, but when we do,
> ideally it should be in an unintrusive way. I contend our proposal
> fits that criterion. And our cursory implementation efforts seem to
> confirm that so far.

Good!

> 
>> But I really think insisting that the linker symbol names denote the "datum
>> agreement" in a distributed system is punting on what should be handled at
>> the application level. Simon Marlow put some improvements into GHC to help
>> improve doing dynamic code (un)loading, stress test that!
> 
> We could use either the system linker or rts linker. Not sure that it
> makes any difference at the application level.

No indeed.

> 
>> 2) I've a work in progress on specing out a proper (and sound :) ) static
>> values type extension for ghc, that will be usable perhaps in your your case
>> (though by dint of being sound, will preclude some of the things you think
>> you want).
> 
> I look forward to hearing more about that.

+1

> How is the existing proposal not (type?) sound?
> 

I'd like to hear more about the concerns too.

>> As for *how* to send an AST fragment, edward kmett and other have some
>> pretty nice typed AST models that are easy to adapt and extend for an
>> application specific use case. Bound
>> http://hackage.haskell.org/package/bound is one nice one.
>> 
>> heres a really really good school of haskell exposition
>> https://www.fpcomplete.com/user/edwardk/bound
> 
> These are nice encodings for AST's. But they don't address how to
> minimize the amount of code to ship around the cluster. If you have no
> agreement about what functions are commonly available, then the AST
> needs to include the code for the function you are sending, + any
> functions it depends, + any of their dependencies, and so on
> transitively.

That was precisely my concern with the idea of shipping *something* AST-like around. It's a lot of overhead for every application you want to develop, or a *massive* overhead to cover all bases.

> 
> Tim, perhaps the following also answers some of your questions. This
> is where the current proposal comes in: if you choose to ship around
> AST's, you can minimize their size by having them mention shared
> linker symbol names.

Indeed, that does seem to simplify things.

> Mind, that's already possible today, by means of
> the global RemoteTable, but it's building that remote table safely,
> conveniently, in a modular way, and with static checking that no
> symbols from any of the modules that were linked at build time were
> missed, that is difficult.
> 

Yep. It's awkward and when you get it wrong, you're either fighting with TH-obscured compiler errors or worse, the damn thing just doesn't work (because you can't decode properly on the remote node and things just crash, or worse still, just hang on waiting for the *correct* input types, which never arrive because they're not "known" to the RTS).

> By avoiding a RemoteTable entirely, we avoid having to solve that
> difficult problem. :)

Not having a RemoteTable sounds like a plus to me.

Cheers,
Tim