Static values language extension proposal

[Mathieu Boespflug]

Hi Carter,

On Tue, Jan 28, 2014 at 6:03 PM, Carter Schonwald
<carter.schonwald at gmail.com> wrote:
> Theres actually a missing piece of information in this thread: what are the
> example computations that are being sent?

Quite simply, the same as those considered in the original Cloud
Haskell paper, that already advocates the extension that Facundo's
first email merely fleshed out a tiny bit. Here's the link once again:

"Towards Haskell in the Cloud", Jeff Epstein, Andrew P. Black, and
Simon Peyton-Jones (2011).
http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/remote.pdf

We are emphatically not considering "arbitrary closures" as you say
below, anymore than the original paper does. As such...

> My understanding is that erlang has not way to send file handles, shared
> variables, Tvars, Mvars, memory mapped binary files, GPU code / memory
> pointers , and other fun unportable things between nodes, and I don't really
> expect / see how we can hope to sanely do that in haskell!

... the above is completely impossible. The original paper explains
why this is so (see Sections 2.3 and 5.1). Here's the gist:

1. you can only send remotely serializable values, i.e. that have an
instance of class Serializable.
2. none of the above have a Serializable instance, and are hence not
"send"-able.

When it comes to sending closures capturing any of the above types of
values, the reasoning goes like this:

3. a closure in the sense of CH is a pair of a static value and an environment,
4. a closure can only be sent if it is serializable,
5. a closure is serializable only if its its environment can be serialized,
5. its environment can be serialized only if all free variables of the
closure can,
6. none of the above have a Serializable instance,
7. hence any closure capturing file handles, MVars, memory pointers,
etc cannot be sent.

> point in fact, even when restricted to "exactly the same binary, running on
> a cluster of homogeneous machines with the exact same hardware, with a
> modern linux distro " you hit some gnarly problems doing this for arbitrary
> closures!  Its for a very simple (and fun) reason: address randomization!

Which is why neither we nor the original paper considered using
addresses as labels for static values. We use linker labels, which are
stable.

> on the GHC API front,
> http://www.haskell.org/ghc/docs/latest/html/libraries/ghc-7.6.3/DynamicLoading.html
> along with (and more appropriately
> http://www.haskell.org/ghc/docs/latest/html/libraries/ghc-7.6.3/ObjLink.html
> ) should actually give enough basic tooling to make this possible as a
> userland library, mind you unload was recently fixed up in HEAD by Simon
> Marlow to support the dynamic code loading / unloading use case he has in
> facebook.  Point being the GHC 7.8 version of the ObjLink api should
> actually give enough support tooling to prototype this idea in user land,
> and that plus better support for writing "direct haskell code" and getting
> out both a local computation and an AST we can serialize would probably be a
> good set of primitives for making this feasible in user land.  I

For the third time: we can of course use any linker API that the
system or the compiler happens to provide, so long as it allows
resolving linker symbols to Haskell values. The (small) extension
under consideration does not replace or add to any existing linker
API. It just transparently floats closed expressions to the top-level,
makes sure linker symbols will exist at runtime (they currently don't
always do),  and does some basic sanity checks so the user doesn't
lose his.

I listed problems labeled 1a), 1b) and 2) in my previous email. You
still haven't showed us how to address those in pure TH userland.

> In Fact, because in general not every computation
> will be properly serializable, you need not even bother with tracking an
> explicit symbol table on each side, just try to load it at a given type and
> if it fails it wasn't there!
>
> The point being, linkers are a thing, ghc exposes an API for linking, have
> you tried that api?
> http://www.haskell.org/ghc/docs/latest/html/libraries/ghc-7.6.3/ObjLink.html

Yes we have. But I don't see how using it or not using it makes any
difference to the user interface of the proposed compiler extension.
It's an implementation detail with tradeoffs that Facundo could
explain in detail in GHC ticket #8711 if you hadn't rudely closed it
as a "duplicate" of some future and unspecified work of yours.

Best,

Mathieu