[Haskell-cafe] Re: hs-plugins and memory leaks

[Andy Stewart]

Hi Evan,
Evan Laforge <qdunkan at gmail.com> writes:

>>> So my questions are:
>>>
>>> Why did lambdabot and yi abandon plugins?
>>
>> Because it was unmaintained for around 5 years, and was fundamentally
>> less portable than simpler state serialization solutions that offered
>> some of the same benefits as full code hot swapping.
>
> Fair enough.  The idea of being able to make changes and see them
> quickly enough for it to have an interactive feel is very appealing,
> but maybe there are other ways to get there, such as improving link
> time with dynamic linking (my current link time is around 24 seconds).
>  State serialization + restart is definitely simpler and more robust.
> But if it's impossible to get it fast enough otherwise, and there
> aren't any other show stopping problems (I think even a known memory
> leak may be dwarfed by the amount of data the app keeps in memory
> anyway), then it might be worth it to me to maintain hs-plugins.
>
I have project design for use dynamic linking, i even build 'pdynload'
(http://hackage.haskell.org/package/pdynload-0.0.3) with Don's PhD
thesis.

Last, i remove pdynload code from my project temporary with below reasons:

1) Hold running state is difficult, like network state in browser or
running state in terminal emulator. 

2) Linking time is too long, I have haskell OS project
(http://www.flickr.com/photos/48809572@N02/) have many sub-module, every
sub-module is very big, and linking time is too long.

3) Memory leak like you said.

>>> Is unloadObj a guaranteed memory leak?  As far as I can tell, it's
>>> never called within ghc itself.  If the choices are between a memory
>>> leak no matter how you use it and dangerous but correct if you use it
>>> right, shouldn't we at least have the latter available as an option?
>>> E.g. a reallyUnloadObj function that also frees the image.
>>
>> GHC never unloads object code, so yes, it will "leak" old code.
>
> So would freeing oc->image fix the leak?  In my case, it's not too
> hard to force all data structures that might reference it.
It's not safe for GHC runtime system since you don't know when time
unload old code is safe.

Don's idea is hold old state in memory even you load new state for
hot-swapping safely.
>
>>> Long shot, but are there any more principled ways to guarantee no
>>> pointers to a chunk of code exist?  The only thing I can think of is
>>> to have the state be totally strict and consist only of types from the
>>> static core.  Would it be possible to hand responsibility for the
>>> memory off to the garbage collector?
>>
>> It's really hard.
>
> It happens in python for python bytecode, since it exists as a plain
> data structure in the language.  E.g. 'code = compile('xyz')'.
> Couldn't a haskell solution be along the same lines?  'code <- load
> "X.o"; makeFunction code', and then makeFunction holds a ForeignPtr to
> the actual code and there's some kind of primitive to call a chunk of
> code as a function.
Anyway, i was re-thinking hot-swap haskell some time, my idea
is : 

     multi-processes framework 
   + hot-swapping core entry 
   + mix old/new sub-module in runtime

Core and sub-module all in separate processes.

With my project (http://www.flickr.com/photos/48809572@N02/), editor and
browser (many other sub-module ...) are sub-module.

Core don't do anything, just control how to load sub-module.

Core have 'entry code', like 'pageBufferNewFun' in
https://patch-tag.com/r/AndyStewart/manatee/snapshot/current/content/pretty/Manatee.hs
'sourceBufferNew', 'browserBufferNew' are 'entry function' to load sub-module in *new*
process.

Core process always running, so we just need hot-swapping 'entry code' after we update sub-module
library by cabal, then we can use new 'entry code' load sub-module in new process, at the same
time, old sub-module code still running in old process.

Welcome to discuss. :)

Cheers,

  -- Andy