[GHC] #7746: Support loading/unloading profiled objects from a profiled executable

[GHC]

#7746: Support loading/unloading profiled objects from a profiled executable
-------------------------------------+------------------------------------
        Reporter:  ezyang            |            Owner:  ezyang
            Type:  feature request   |           Status:  new
        Priority:  normal            |        Milestone:  7.8.1
       Component:  Runtime System    |          Version:  7.7
      Resolution:                    |         Keywords:
Operating System:  Unknown/Multiple  |     Architecture:  Unknown/Multiple
 Type of failure:  None/Unknown      |       Difficulty:  Unknown
       Test Case:                    |       Blocked By:  5435, 8039
        Blocking:                    |  Related Tickets:
-------------------------------------+------------------------------------
Changes (by ezyang):

 * blockedby:   => 5435, 8039


Old description:

> This is closely related to #3360, but it is a bit less ambitious and
> should be possible to implement without too many extra changes to the
> byte code compiler and interpreter (e.g. we just have to teach the linker
> how to handle things). Here is a simple test program off of 'plugins' to
> get working:
>
> {{{
> {-# LANGUAGE ScopedTypeVariables #-}
>
> import System.Plugins.Make
> import System.Plugins.Load
> import Data.List
>
> boot :: FilePath -> IO ()
> boot path = do
>     r <- make path ["-prof"]
>     case r of
>       MakeSuccess _ p -> do
>         r' <- load p [] [] "result"
>         case r' of
>           LoadSuccess _ (v :: Int) -> print v
>           LoadFailure msg -> print msg
>       MakeFailure es -> putStrLn ("Failed: " ++ intercalate " " es)
>
> main = do
>     boot "Foo.hs"
> }}}
>
> where Foo.hs is
>
> {{{
> module Foo where
> result = 2 :: Int
> }}}
>
> Here are the things that, as far as I can tell, need to be handled:
>
>  * We should ensure consistency between the host and the object file
> being uploaded. For example, if you load an un-profiled object file into
> a profiled binary, GHC will eat all your puppies. A simple way to do this
> is look for a symbol (e.g. CC_LIST) which is only ever exported when
> something is profiled and barf it is encountered.
>
>  * This current code fails with {{{test: Foo.o: unknown symbol
> `CC_LIST'}}}, much the same way GHCi used to fail.  This particular
> problem is (I think) that we don’t store CC_LIST and other externsymbols
> in our global hash table, so the linker thinks that they don’t exist,
> when they do. CC_LIST and friends should be special-cased or added to the
> table.
>
>  * We don’t run ctors which setup CC_LIST with all of the cost-centres
> from the loaded module; we need to teach the linker to do that (that's
> the {{{/* ignore constructor section for now */}}})
>
>  * We need to come up with some sensible way of unloading cost-centres
> from CC_LIST and friends; we could make CC_LIST doubly-linked and then
> just excise the cost-centre in a destructor, but freeing the actual
> allocated CostCentre is more difficult. For now, we might just live with
> the memory leak, but see wiki:"Commentary/ResourceLimits#Memoryleaks" for
> a possible better implementation strategy.
>
> But that’s it; everything else should work normally. Something similar
> should apply to ticky builds. Something we will have to think about is
> how to handle these special-cases as we move from static objects to
> dynamic objects and push more of the runtime linking burden to the
> standard libraries.

New description:

 This is closely related to #3360, but it is a bit less ambitious and
 should be possible to implement without too many extra changes to the byte
 code compiler and interpreter (e.g. we just have to teach the linker how
 to handle things). Here is a simple test program off of 'plugins' to get
 working:

 {{{
 {-# LANGUAGE ScopedTypeVariables #-}

 import System.Plugins.Make
 import System.Plugins.Load
 import Data.List

 boot :: FilePath -> IO ()
 boot path = do
     r <- make path ["-prof"]
     case r of
       MakeSuccess _ p -> do
         r' <- load p [] [] "result"
         case r' of
           LoadSuccess _ (v :: Int) -> print v
           LoadFailure msg -> print msg
       MakeFailure es -> putStrLn ("Failed: " ++ intercalate " " es)

 main = do
     boot "Foo.hs"
 }}}

 where Foo.hs is

 {{{
 module Foo where
 result = 2 :: Int
 }}}

 Here are the things that, as far as I can tell, need to be handled:

  * We should ensure consistency between the host and the object file being
 uploaded. For example, if you load an un-profiled object file into a
 profiled binary, GHC will eat all your puppies. A simple way to do this is
 look for a symbol (e.g. CC_LIST) which is only ever exported when
 something is profiled and barf it is encountered.

  * This current code fails with {{{test: Foo.o: unknown symbol
 `CC_LIST'}}}, much the same way GHCi used to fail.  This particular
 problem is (I think) that we don’t store CC_LIST and other externsymbols
 in our global hash table, so the linker thinks that they don’t exist, when
 they do. CC_LIST and friends should be special-cased or added to the
 table. I have a patch for that; it's pretty easy.

  * We don’t run ctors which setup CC_LIST with all of the cost-centres
 from the loaded module; we need to teach the linker to do that. The
 relevant bug is #5435.

  * We need to come up with some sensible way of unloading cost-centres
 from CC_LIST and friends; we could make CC_LIST doubly-linked and then
 just excise the cost-centre in a destructor, but freeing the actual
 allocated CostCentre is more difficult. For now, we might just live with
 the memory leak, but see wiki:"Commentary/ResourceLimits#Memoryleaks" for
 a possible better implementation strategy. Whatever cleanup is done here
 should be registered as a destructor for the library. Maybe #8039 solves
 this problem.

  * Tests!

 But that’s it; everything else should work normally. Something similar
 should apply to ticky builds. Sans destructors, there is a good chance
 this shindig may already work for dynamically linked apps.

--

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/7746#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler