Remote GHCi

Simon Marlow marlowsd at
Tue Nov 24 21:32:58 UTC 2015

I want to get it done within the next few weeks.  Currently GHCi is 
mostly working, and the main missing pieces are TH and the debugger.  I 
plan to backport it to the 7.10 branch so that I can have it in our 
local GHC builds at Facebook.

My WIP branch is here:


On 24/11/2015 11:04, Alan & Kim Zimmerman wrote:
> What kind of timescale can we expect on this, and will it be possible to
> backport it (via a ghci-ng or similar)?
> We are currently wrestling with ghci stdio issues in haskell-ide-engine.
> If this will be out soon we can wait for it.
> Alan
> On Tue, Nov 24, 2015 at 2:25 AM, Manuel M T Chakravarty
> <chak at <mailto:chak at>> wrote:
>     Hi Simon,
>     Sounds great!
>     This may very well what you have got in mind anyway, but I could
>     imagine to run the interpreter on a different thread in the
>     -fno-external-interpreter case and arrange communication through the
>     same messaging API that you outlined for the seperate-process
>     interpreter. Then, the essential difference between the two modes
>     would be whether memory is shared or not (i.e., multithreading vs
>     multi-process).
>     Cheers,
>     Manuel
>      > Simon Marlow <marlowsd at <mailto:marlowsd at>>:
>      >
>      > Hi Manuel,
>      >
>      > Thanks for the detailed reply, I have a much better understanding
>     of your requirements now.
>      >
>      > I'm going to support both models of running interpreted code.
>     The current plan is to have a flag, -fexternal-interpreter, which
>     GHC will use by default when running Template Haskell during
>     compilation, and perhaps for GHCi, but for compatibility with
>     applications like yours I'll probably leave it off for GHC API users.
>      >
>      > There's really no downside to doing this, it's not much more
>     complicated than implementing the separate-process model.
>      >
>      > Cheers,
>      > Simon
>      >
>      > On 21/11/2015 03:38, Manuel M T Chakravarty wrote:
>      >>> Simon Marlow <marlowsd at <mailto:marlowsd at>>:
>      >>> On 18/11/2015 01:41, Manuel M T Chakravarty wrote:
>      >>>> Hi Simon,
>      >>>>
>      >>>> While this is an interesting proposal, Haskell for Mac strongly
>      >>>> relies on running interpreted code in the same process. I’m using
>      >>>> ’dynCompileExpr’ as well as ’hscStmtWithLocation’ and some other
>      >>>> stuff.
>      >>>
>      >>> Let me say first of all that I'm not going to remove anything,
>     so there's no need to worry.  But I'd like to explore exactly what
>     you need, so that we can see whether there's a way to accommodate it
>     with a separate-process implementation.
>      >>>
>      >>> hscStmtWithLocation is part of the core GHCi functionality, it
>     is definitely supported.  It has a slightly different signature:
>      >>>
>      >>> hscStmtWithLocation :: HscEnv
>      >>>                    -> String -- ^ The statement
>      >>>                    -> String -- ^ The source
>      >>>                    -> Int    -- ^ Starting line
>      >>>                    -> IO ( Maybe ([Id]
>      >>>                          , RemoteHValue {- IO [HValue] -}
>      >>>                          , FixityEnv))
>      >>>
>      >>> RemoteHValue is a reference to a value in the interpreter's
>     context. These have to be evaluated via an explicit API, rather than
>     just unsafeCoercing Value as we do now.  (this is not strictly
>     speaking part of the GHC API, so a separate but interesting question
>     is: why did you need to use this directly, and what should we add to
>     the GHC API?)
>      >>
>      >> The GHC API basically assumes that the ”result” of statement
>     execution is the *side-effect* of printing the result to stdout.
>     This is not sufficient for an interactive graphical environment as
>      >>
>      >> (1) I want to have the result (even if it is a string) separate
>     from anything else interpreted code execution writes to stdout. (In
>     Haskell for Mac, these things are displayed in different places.)
>      >>
>      >> (2) I want results that are not just strings. For example, a
>     result (of running Haskell code) may be a ForeignPtr to a C-land
>     data structure representing an image (e.g., an in-memory
>     representation of a PNG image rendered by Diagrams).
>      >>
>      >> For the latter, I’m actually using `compileExpr`, then
>     `unsafeCoerce` the `hValue` into `IO (ForeignPtr ())` and `try` that
>     (to also catch any exceptions). When this code runs, in some cases,
>     it calls back and forth between interpreted Haskell code and the
>     host application using the FFI.
>      >>
>      >>> I believe that many uses of dynCompileExpr can be changed so
>     that the code using the resulting value is moved into the
>     interpreter’s context, and then there’s no problem.
>      >>
>      >> This is difficult in my case, because the resulting value is
>     used in the GUI code written in Swift. Code running in a different
>     process cannot call the Cocoa framework methods for the GUI of the
>     main process.
>      >>
>      >>>> This is quite crucial for some of the interactive
>      >>>> functionality. Imagine a game where the game engine is in Swift
>      >>>> linked into the main application and the game logic is in
>      >>>> *interpreted* Haskell code. The engine calls into the Haskell code
>      >>>> multiple times per frame of the animation and for all
>      >>>> keyboard/mouse/etc input (using StablePtr and ForeignPtr to
>     construct
>      >>>> the scene graph across the Swift and Haskell heap).
>      >>>
>      >>> So my question is, why wouldn't you run the whole game engine
>     in the interpreter's context?  That’s what would happen if you were
>     to load the program into GHCi and run it.
>      >>
>      >> On a fundamental level: The game engine runs on OpenGL. If it is
>     in a different process, it cannot access the OpenGL context of the
>     main process (which it needs to do to render into a specific view of
>     a specific window of the main process).
>      >>
>      >> In practice, it is not just an OpenGL problem as I’m using a
>     framework called SpriteKit with its own event and rendering loop
>     that in turn uses OpenGL for the actual rendering. It does a lot of
>     things behind the scenes (which makes it convenient to use), which
>     requires you to be careful which threads you use to execute some
>     operations. Running in an entire different process is surely going
>     to break things.
>      >>
>      >>>   Directly calling back and forth between the client of the GHC
>     API and the program being interpreted is arguably a strange thing to
>     do, and it’s kind of accidental that we allow it.
>      >>
>      >> I understand that, but I also think that it is an artefact of
>     Haskell mostly being used in a command line program set up. I don’t
>     think, it is just by chance that the IHaskell people do some quite
>     similar things to at least some of what I’m doing. Once you want a
>     more interactive experience, call patterns get more complicated.
>      >>
>      >>>> I actually also might have a use for the architecture that you are
>      >>>> proposing. However, I really would like to keep the ability to, at
>      >>>> least, optionally run interpreted code in the same process
>     (without
>      >>>> profiling etc). Do you think we could have both?
>      >>>
>      >>> We can certainly have both, it's straightforward to implement,
>     but I don't get to throw away some of the hacks we have to support
>     same-process execution, which would be a shame.  We just add more
>     code rather than
>      >>
>      >> Yes, I understand that and, as I wrote, I do like the idea of
>     running in a separate process. However, it would also be a shame to
>     prevent richer and more interactive experiences than CLI applications.
>      >>
>      >> I have thought a bit more about what the fundamental obstacle
>     is. I think, it is two things:
>      >>
>      >> (1) I have interpreted Haskell code that (via a compiled Haskell
>     library) uses FFI calls to call Cocoa system framework methods to
>     create Cocoa objects. In Haskell, these Cocoa objects are referenced
>     via a ForeignPtr and I need the interpreter to be able to return
>     these foreign pointers. The ForeignPtr’s need to refer to memory of
>     the main host process; hence, the FFI calls need to run the Cocoa
>     framework code in the host process.
>      >>
>      >> (2) The Cocoa objects from (1) include both StablePtrs as well
>     as C function pointers created via foreign dynamic wrapper. At least
>     some of the StablePtrs refer to Haskell heap structures that need to
>     be accessed by interpreted Haskell code. And calling the dynamic
>     wrapper code from Swift in the main process needs to execute Haskell
>     code that may refer to closures created by interpreted code.
>      >>
>      >> So, the issue really is that I would need FFI calls in the
>     interpreter process that call Cocoa code in the main process and
>     dynamic wrapper entry code in the main process that needs to call
>     Haskell code in the interpreter process. (Crossing the FFI language
>     chasm corresponds to cross-process calls.)
>      >>
>      >> I cannot move the Cocoa code from the main process to the
>     interpreter process, as Cocoa requires that it runs on the *main*
>     thread of the main process (to interact with the GUI and also to
>     render via OpenGL).
>      >>
>      >> Does that make sense?
>      >>
>      >> Cheers,
>      >> Manuel
>      >>
>      >>>>> Simon Marlow <marlowsd at <mailto:marlowsd at>>:
>      >>>>>
>      >>>>> Hi folks - I've been thinking about changing the way we run
>     interpreted code so that it would be run in a separate process.  It
>     turns out this has quite a few benefits, and would let us kill some
>     of the really awkward hacks we have in GHC to work around problems
>     that arise because we're running interpreted code and the compiler
>     on the same runtime.
>      >>>>>
>      >>>>> I summarised the idea here:
>      >>>>>
>      >>>>> I'd be interested to hear if anyone has any thoughts around
>     this, particularly if doing this would make your life difficult in
>     some way. Are people relying on dynCompileExpr for anything?
>      >>>>>
>      >>>>> Cheers,
>      >>>>> Simon
>      >>>>> _______________________________________________
>      >>>>> ghc-devs mailing list
>      >>>>> ghc-devs at <mailto:ghc-devs at>
>      >>>>>
>      >>>>
>      >>
>      > _______________________________________________
>      > ghc-devs mailing list
>      > ghc-devs at <mailto:ghc-devs at>
>      >
>     _______________________________________________
>     ghc-devs mailing list
>     ghc-devs at <mailto:ghc-devs at>

More information about the ghc-devs mailing list