Remote GHCi
Simon Marlow
marlowsd at gmail.com
Tue Nov 24 21:32:58 UTC 2015
I want to get it done within the next few weeks. Currently GHCi is
mostly working, and the main missing pieces are TH and the debugger. I
plan to backport it to the 7.10 branch so that I can have it in our
local GHC builds at Facebook.
My WIP branch is here: https://github.com/simonmar/ghc/commits/prof-ghci
Cheers
Simon
On 24/11/2015 11:04, Alan & Kim Zimmerman wrote:
> What kind of timescale can we expect on this, and will it be possible to
> backport it (via a ghci-ng or similar)?
>
> We are currently wrestling with ghci stdio issues in haskell-ide-engine.
> If this will be out soon we can wait for it.
>
> Alan
>
> On Tue, Nov 24, 2015 at 2:25 AM, Manuel M T Chakravarty
> <chak at justtesting.org <mailto:chak at justtesting.org>> wrote:
>
> Hi Simon,
>
> Sounds great!
>
> This may very well what you have got in mind anyway, but I could
> imagine to run the interpreter on a different thread in the
> -fno-external-interpreter case and arrange communication through the
> same messaging API that you outlined for the seperate-process
> interpreter. Then, the essential difference between the two modes
> would be whether memory is shared or not (i.e., multithreading vs
> multi-process).
>
> Cheers,
> Manuel
>
> > Simon Marlow <marlowsd at gmail.com <mailto:marlowsd at gmail.com>>:
> >
> > Hi Manuel,
> >
> > Thanks for the detailed reply, I have a much better understanding
> of your requirements now.
> >
> > I'm going to support both models of running interpreted code.
> The current plan is to have a flag, -fexternal-interpreter, which
> GHC will use by default when running Template Haskell during
> compilation, and perhaps for GHCi, but for compatibility with
> applications like yours I'll probably leave it off for GHC API users.
> >
> > There's really no downside to doing this, it's not much more
> complicated than implementing the separate-process model.
> >
> > Cheers,
> > Simon
> >
> > On 21/11/2015 03:38, Manuel M T Chakravarty wrote:
> >>> Simon Marlow <marlowsd at gmail.com <mailto:marlowsd at gmail.com>>:
> >>> On 18/11/2015 01:41, Manuel M T Chakravarty wrote:
> >>>> Hi Simon,
> >>>>
> >>>> While this is an interesting proposal, Haskell for Mac strongly
> >>>> relies on running interpreted code in the same process. I’m using
> >>>> ’dynCompileExpr’ as well as ’hscStmtWithLocation’ and some other
> >>>> stuff.
> >>>
> >>> Let me say first of all that I'm not going to remove anything,
> so there's no need to worry. But I'd like to explore exactly what
> you need, so that we can see whether there's a way to accommodate it
> with a separate-process implementation.
> >>>
> >>> hscStmtWithLocation is part of the core GHCi functionality, it
> is definitely supported. It has a slightly different signature:
> >>>
> >>> hscStmtWithLocation :: HscEnv
> >>> -> String -- ^ The statement
> >>> -> String -- ^ The source
> >>> -> Int -- ^ Starting line
> >>> -> IO ( Maybe ([Id]
> >>> , RemoteHValue {- IO [HValue] -}
> >>> , FixityEnv))
> >>>
> >>> RemoteHValue is a reference to a value in the interpreter's
> context. These have to be evaluated via an explicit API, rather than
> just unsafeCoercing Value as we do now. (this is not strictly
> speaking part of the GHC API, so a separate but interesting question
> is: why did you need to use this directly, and what should we add to
> the GHC API?)
> >>
> >> The GHC API basically assumes that the ”result” of statement
> execution is the *side-effect* of printing the result to stdout.
> This is not sufficient for an interactive graphical environment as
> >>
> >> (1) I want to have the result (even if it is a string) separate
> from anything else interpreted code execution writes to stdout. (In
> Haskell for Mac, these things are displayed in different places.)
> >>
> >> (2) I want results that are not just strings. For example, a
> result (of running Haskell code) may be a ForeignPtr to a C-land
> data structure representing an image (e.g., an in-memory
> representation of a PNG image rendered by Diagrams).
> >>
> >> For the latter, I’m actually using `compileExpr`, then
> `unsafeCoerce` the `hValue` into `IO (ForeignPtr ())` and `try` that
> (to also catch any exceptions). When this code runs, in some cases,
> it calls back and forth between interpreted Haskell code and the
> host application using the FFI.
> >>
> >>> I believe that many uses of dynCompileExpr can be changed so
> that the code using the resulting value is moved into the
> interpreter’s context, and then there’s no problem.
> >>
> >> This is difficult in my case, because the resulting value is
> used in the GUI code written in Swift. Code running in a different
> process cannot call the Cocoa framework methods for the GUI of the
> main process.
> >>
> >>>> This is quite crucial for some of the interactive
> >>>> functionality. Imagine a game where the game engine is in Swift
> >>>> linked into the main application and the game logic is in
> >>>> *interpreted* Haskell code. The engine calls into the Haskell code
> >>>> multiple times per frame of the animation and for all
> >>>> keyboard/mouse/etc input (using StablePtr and ForeignPtr to
> construct
> >>>> the scene graph across the Swift and Haskell heap).
> >>>
> >>> So my question is, why wouldn't you run the whole game engine
> in the interpreter's context? That’s what would happen if you were
> to load the program into GHCi and run it.
> >>
> >> On a fundamental level: The game engine runs on OpenGL. If it is
> in a different process, it cannot access the OpenGL context of the
> main process (which it needs to do to render into a specific view of
> a specific window of the main process).
> >>
> >> In practice, it is not just an OpenGL problem as I’m using a
> framework called SpriteKit with its own event and rendering loop
> that in turn uses OpenGL for the actual rendering. It does a lot of
> things behind the scenes (which makes it convenient to use), which
> requires you to be careful which threads you use to execute some
> operations. Running in an entire different process is surely going
> to break things.
> >>
> >>> Directly calling back and forth between the client of the GHC
> API and the program being interpreted is arguably a strange thing to
> do, and it’s kind of accidental that we allow it.
> >>
> >> I understand that, but I also think that it is an artefact of
> Haskell mostly being used in a command line program set up. I don’t
> think, it is just by chance that the IHaskell people do some quite
> similar things to at least some of what I’m doing. Once you want a
> more interactive experience, call patterns get more complicated.
> >>
> >>>> I actually also might have a use for the architecture that you are
> >>>> proposing. However, I really would like to keep the ability to, at
> >>>> least, optionally run interpreted code in the same process
> (without
> >>>> profiling etc). Do you think we could have both?
> >>>
> >>> We can certainly have both, it's straightforward to implement,
> but I don't get to throw away some of the hacks we have to support
> same-process execution, which would be a shame. We just add more
> code rather than
> >>
> >> Yes, I understand that and, as I wrote, I do like the idea of
> running in a separate process. However, it would also be a shame to
> prevent richer and more interactive experiences than CLI applications.
> >>
> >> I have thought a bit more about what the fundamental obstacle
> is. I think, it is two things:
> >>
> >> (1) I have interpreted Haskell code that (via a compiled Haskell
> library) uses FFI calls to call Cocoa system framework methods to
> create Cocoa objects. In Haskell, these Cocoa objects are referenced
> via a ForeignPtr and I need the interpreter to be able to return
> these foreign pointers. The ForeignPtr’s need to refer to memory of
> the main host process; hence, the FFI calls need to run the Cocoa
> framework code in the host process.
> >>
> >> (2) The Cocoa objects from (1) include both StablePtrs as well
> as C function pointers created via foreign dynamic wrapper. At least
> some of the StablePtrs refer to Haskell heap structures that need to
> be accessed by interpreted Haskell code. And calling the dynamic
> wrapper code from Swift in the main process needs to execute Haskell
> code that may refer to closures created by interpreted code.
> >>
> >> So, the issue really is that I would need FFI calls in the
> interpreter process that call Cocoa code in the main process and
> dynamic wrapper entry code in the main process that needs to call
> Haskell code in the interpreter process. (Crossing the FFI language
> chasm corresponds to cross-process calls.)
> >>
> >> I cannot move the Cocoa code from the main process to the
> interpreter process, as Cocoa requires that it runs on the *main*
> thread of the main process (to interact with the GUI and also to
> render via OpenGL).
> >>
> >> Does that make sense?
> >>
> >> Cheers,
> >> Manuel
> >>
> >>>>> Simon Marlow <marlowsd at gmail.com <mailto:marlowsd at gmail.com>>:
> >>>>>
> >>>>> Hi folks - I've been thinking about changing the way we run
> interpreted code so that it would be run in a separate process. It
> turns out this has quite a few benefits, and would let us kill some
> of the really awkward hacks we have in GHC to work around problems
> that arise because we're running interpreted code and the compiler
> on the same runtime.
> >>>>>
> >>>>> I summarised the idea here:
> https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> >>>>>
> >>>>> I'd be interested to hear if anyone has any thoughts around
> this, particularly if doing this would make your life difficult in
> some way. Are people relying on dynCompileExpr for anything?
> >>>>>
> >>>>> Cheers,
> >>>>> Simon
> >>>>> _______________________________________________
> >>>>> ghc-devs mailing list
> >>>>> ghc-devs at haskell.org <mailto:ghc-devs at haskell.org>
> >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> >>>>
> >>
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org <mailto:ghc-devs at haskell.org>
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org <mailto:ghc-devs at haskell.org>
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
More information about the ghc-devs
mailing list