Remote GHCi

Alan & Kim Zimmerman alan.zimm at gmail.com
Tue Nov 24 11:04:40 UTC 2015 

What kind of timescale can we expect on this, and will it be possible to
backport it (via a ghci-ng or similar)?

We are currently wrestling with ghci stdio issues in haskell-ide-engine. If
this will be out soon we can wait for it.

Alan

On Tue, Nov 24, 2015 at 2:25 AM, Manuel M T Chakravarty <
chak at justtesting.org> wrote:

> Hi Simon,
>
> Sounds great!
>
> This may very well what you have got in mind anyway, but I could imagine
> to run the interpreter on a different thread in the
> -fno-external-interpreter case and arrange communication through the same
> messaging API that you outlined for the seperate-process interpreter. Then,
> the essential difference between the two modes would be whether memory is
> shared or not (i.e., multithreading vs multi-process).
>
> Cheers,
> Manuel
>
> > Simon Marlow <marlowsd at gmail.com>:
> >
> > Hi Manuel,
> >
> > Thanks for the detailed reply, I have a much better understanding of
> your requirements now.
> >
> > I'm going to support both models of running interpreted code.  The
> current plan is to have a flag, -fexternal-interpreter, which GHC will use
> by default when running Template Haskell during compilation, and perhaps
> for GHCi, but for compatibility with applications like yours I'll probably
> leave it off for GHC API users.
> >
> > There's really no downside to doing this, it's not much more complicated
> than implementing the separate-process model.
> >
> > Cheers,
> > Simon
> >
> > On 21/11/2015 03:38, Manuel M T Chakravarty wrote:
> >>> Simon Marlow <marlowsd at gmail.com>:
> >>> On 18/11/2015 01:41, Manuel M T Chakravarty wrote:
> >>>> Hi Simon,
> >>>>
> >>>> While this is an interesting proposal, Haskell for Mac strongly
> >>>> relies on running interpreted code in the same process. I’m using
> >>>> ’dynCompileExpr’ as well as ’hscStmtWithLocation’ and some other
> >>>> stuff.
> >>>
> >>> Let me say first of all that I'm not going to remove anything, so
> there's no need to worry.  But I'd like to explore exactly what you need,
> so that we can see whether there's a way to accommodate it with a
> separate-process implementation.
> >>>
> >>> hscStmtWithLocation is part of the core GHCi functionality, it is
> definitely supported.  It has a slightly different signature:
> >>>
> >>> hscStmtWithLocation :: HscEnv
> >>>                    -> String -- ^ The statement
> >>>                    -> String -- ^ The source
> >>>                    -> Int    -- ^ Starting line
> >>>                    -> IO ( Maybe ([Id]
> >>>                          , RemoteHValue {- IO [HValue] -}
> >>>                          , FixityEnv))
> >>>
> >>> RemoteHValue is a reference to a value in the interpreter's context.
> These have to be evaluated via an explicit API, rather than just
> unsafeCoercing Value as we do now.  (this is not strictly speaking part of
> the GHC API, so a separate but interesting question is: why did you need to
> use this directly, and what should we add to the GHC API?)
> >>
> >> The GHC API basically assumes that the ”result” of statement execution
> is the *side-effect* of printing the result to stdout. This is not
> sufficient for an interactive graphical environment as
> >>
> >> (1) I want to have the result (even if it is a string) separate from
> anything else interpreted code execution writes to stdout. (In Haskell for
> Mac, these things are displayed in different places.)
> >>
> >> (2) I want results that are not just strings. For example, a result (of
> running Haskell code) may be a ForeignPtr to a C-land data structure
> representing an image (e.g., an in-memory representation of a PNG image
> rendered by Diagrams).
> >>
> >> For the latter, I’m actually using `compileExpr`, then `unsafeCoerce`
> the `hValue` into `IO (ForeignPtr ())` and `try` that (to also catch any
> exceptions). When this code runs, in some cases, it calls back and forth
> between interpreted Haskell code and the host application using the FFI.
> >>
> >>> I believe that many uses of dynCompileExpr can be changed so that the
> code using the resulting value is moved into the interpreter’s context, and
> then there’s no problem.
> >>
> >> This is difficult in my case, because the resulting value is used in
> the GUI code written in Swift. Code running in a different process cannot
> call the Cocoa framework methods for the GUI of the main process.
> >>
> >>>> This is quite crucial for some of the interactive
> >>>> functionality. Imagine a game where the game engine is in Swift
> >>>> linked into the main application and the game logic is in
> >>>> *interpreted* Haskell code. The engine calls into the Haskell code
> >>>> multiple times per frame of the animation and for all
> >>>> keyboard/mouse/etc input (using StablePtr and ForeignPtr to construct
> >>>> the scene graph across the Swift and Haskell heap).
> >>>
> >>> So my question is, why wouldn't you run the whole game engine in the
> interpreter's context?  That’s what would happen if you were to load the
> program into GHCi and run it.
> >>
> >> On a fundamental level: The game engine runs on OpenGL. If it is in a
> different process, it cannot access the OpenGL context of the main process
> (which it needs to do to render into a specific view of a specific window
> of the main process).
> >>
> >> In practice, it is not just an OpenGL problem as I’m using a framework
> called SpriteKit with its own event and rendering loop that in turn uses
> OpenGL for the actual rendering. It does a lot of things behind the scenes
> (which makes it convenient to use), which requires you to be careful which
> threads you use to execute some operations. Running in an entire different
> process is surely going to break things.
> >>
> >>>   Directly calling back and forth between the client of the GHC API
> and the program being interpreted is arguably a strange thing to do, and
> it’s kind of accidental that we allow it.
> >>
> >> I understand that, but I also think that it is an artefact of Haskell
> mostly being used in a command line program set up. I don’t think, it is
> just by chance that the IHaskell people do some quite similar things to at
> least some of what I’m doing. Once you want a more interactive experience,
> call patterns get more complicated.
> >>
> >>>> I actually also might have a use for the architecture that you are
> >>>> proposing. However, I really would like to keep the ability to, at
> >>>> least, optionally run interpreted code in the same process (without
> >>>> profiling etc). Do you think we could have both?
> >>>
> >>> We can certainly have both, it's straightforward to implement, but I
> don't get to throw away some of the hacks we have to support same-process
> execution, which would be a shame.  We just add more code rather than
> >>
> >> Yes, I understand that and, as I wrote, I do like the idea of running
> in a separate process. However, it would also be a shame to prevent richer
> and more interactive experiences than CLI applications.
> >>
> >> I have thought a bit more about what the fundamental obstacle is. I
> think, it is two things:
> >>
> >> (1) I have interpreted Haskell code that (via a compiled Haskell
> library) uses FFI calls to call Cocoa system framework methods to create
> Cocoa objects. In Haskell, these Cocoa objects are referenced via a
> ForeignPtr and I need the interpreter to be able to return these foreign
> pointers. The ForeignPtr’s need to refer to memory of the main host
> process; hence, the FFI calls need to run the Cocoa framework code in the
> host process.
> >>
> >> (2) The Cocoa objects from (1) include both StablePtrs as well as C
> function pointers created via foreign dynamic wrapper. At least some of the
> StablePtrs refer to Haskell heap structures that need to be accessed by
> interpreted Haskell code. And calling the dynamic wrapper code from Swift
> in the main process needs to execute Haskell code that may refer to
> closures created by interpreted code.
> >>
> >> So, the issue really is that I would need FFI calls in the interpreter
> process that call Cocoa code in the main process and dynamic wrapper entry
> code in the main process that needs to call Haskell code in the interpreter
> process. (Crossing the FFI language chasm corresponds to cross-process
> calls.)
> >>
> >> I cannot move the Cocoa code from the main process to the interpreter
> process, as Cocoa requires that it runs on the *main* thread of the main
> process (to interact with the GUI and also to render via OpenGL).
> >>
> >> Does that make sense?
> >>
> >> Cheers,
> >> Manuel
> >>
> >>>>> Simon Marlow <marlowsd at gmail.com>:
> >>>>>
> >>>>> Hi folks - I've been thinking about changing the way we run
> interpreted code so that it would be run in a separate process.  It turns
> out this has quite a few benefits, and would let us kill some of the really
> awkward hacks we have in GHC to work around problems that arise because
> we're running interpreted code and the compiler on the same runtime.
> >>>>>
> >>>>> I summarised the idea here:
> https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> >>>>>
> >>>>> I'd be interested to hear if anyone has any thoughts around this,
> particularly if doing this would make your life difficult in some way. Are
> people relying on dynCompileExpr for anything?
> >>>>>
> >>>>> Cheers,
> >>>>> Simon
> >>>>> _______________________________________________
> >>>>> ghc-devs mailing list
> >>>>> ghc-devs at haskell.org
> >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> >>>>
> >>
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20151124/da639a94/attachment-0001.html>

More information about the ghc-devs mailing list