[Haskell-cafe] Re: hs-plugins and memory leaks

[Andy Stewart]

Evan Laforge <qdunkan at gmail.com> writes:

>> Last, i remove pdynload code from my project temporary with below reasons:
>>
>> 1) Hold running state is difficult, like network state in browser or
>> running state in terminal emulator.
>
> This doesn't seem too hard to me.  Provided you are not swapping the
> module that defines the state in the first place, simply reload the
> module, and replace the old symbol in the state with the reloaded one.
>
>> 2) Linking time is too long, I have haskell OS project
>> (http://www.flickr.com/photos/48809572@N02/) have many sub-module, every
>> sub-module is very big, and linking time is too long.
>
> This is discouraging, since one of the main reasons to use dynamically
> loaded code instead of recompiling the whole app is to avoid long link
> times.  Presumably you would compile the majority of the app (the API
> that the plugins use, and the internal code also uses) as a dynamic
> library:
>
> main.o -> tiny stub that just calls app.so
> app.so -> large library containing all app logic
> plugin.so -> links against app.so when loaded
>
> So the plugin needs to read a lot of hi files when recompiling, but
> the dynamic link time should be proportional to the number of
> unresolved symbols in plugin.so that point into app.so, not
> proportional to the overall size of the app, right?
Yes, not proportional the size of application, 
but link time depend on the dependent packages that haven't linked.

Example like the GHC API in  'pdynload' package, it will search symbol define in GHC
database to get which packageId that need re-link, then use below code
link:

   Linker.linkPackages flags [packageId]
   
Function 'linkPackages' will link specified package and it's "dependent
packages", if dependents packages is bigger, link time is longer.

So the long link time is unavoidable for *big* package.

>
>>> So would freeing oc->image fix the leak?  In my case, it's not too
>>> hard to force all data structures that might reference it.
>> It's not safe for GHC runtime system since you don't know when time
>> unload old code is safe.
>
> But that's just my question, I *do* (think I) know when it's safe,
> which is after the data that has passed through plugged-in code has
> been fully forced.  Can't I just call unloadObj then?
Yes, unloadObj can work if you careful design, well it's also easy to crash
your program if something miss.

>
> E.g., loading and unloading plugins for audio processing is totally
> standard.  Since the data is strict arrays of primitive types, there's
> no risk of stray pointers to unloaded code.
>
>> Anyway, i was re-thinking hot-swap haskell some time, my idea
>> is :
>>
>>     multi-processes framework
>>   + hot-swapping core entry
>>   + mix old/new sub-module in runtime
>>
>> Core and sub-module all in separate processes.
>
> How would you pass state between processes?
Infact, i won't pass any state between processes.

My framework like this: http://www.flickr.com/photos/48809572@N02/5031811365/lightbox/

Every sub-module running in render process, and render process for
daemon process just a *Tab*. 

When you need update current sub-module, just recompile new code in
Cabal/GHC database, then startup *new* process to load new code, and we
can use dyre technology to restore state in new process.

Though it's not powerful as hs-plugins do, but perfect safety and no
*memory leak*.

  -- Andy