memory fragmentation with ghc-7.6.1

[John Lato]

Yes, that's my current understanding.  I see this with ByteString and
Data.Vector.Storable, but not
Data.Vector/Data.Vector.Unboxed/Data.Text.  As ByteStrings are pretty
widely used for IO, I expected that somebody else would have
experienced this too.

I would expect some memory fragmentation with pinned memory, but the
change from ghc-7.4 to ghc-7.6 is rather extreme (no fragmentation to
several GB).

John L.

On Fri, Sep 21, 2012 at 10:53 AM, Carter Schonwald
<carter.schonwald at gmail.com> wrote:
> So the problem is only with the data structures on the heap that are pinned
> in place to play nice with C?
>
> I'd be curious to understand the change too, though per se pinned memory (a
> la storable or or bytestring) will by definition cause memory fragmentation
> in a gc'd lang as a rule,  (or at least one like Haskell).
> -Carter
>
> On Thu, Sep 20, 2012 at 8:59 PM, John Lato <jwlato at gmail.com> wrote:
>>
>> Hello,
>>
>> We've noticed that some applications exhibit significantly worse
>> memory usage when compiled with ghc-7.6.1 compared to ghc-7.4, leading
>> to out of memory errors in some cases.  Running one app with +RTS -s,
>> I see this:
>>
>> ghc-7.4
>>  525,451,699,736 bytes allocated in the heap
>>   53,404,833,048 bytes copied during GC
>>       39,097,600 bytes maximum residency (2439 sample(s))
>>        1,547,040 bytes maximum slop
>>              628 MB total memory in use (0 MB lost due to fragmentation)
>>
>> ghc-7.6
>> 512,535,907,752 bytes allocated in the heap
>>   53,327,184,712 bytes copied during GC
>>       40,038,584 bytes maximum residency (2391 sample(s))
>>        1,456,472 bytes maximum slop
>>             3414 MB total memory in use (2744 MB lost due to
>> fragmentation)
>>
>> The total memory in use (consistent with 'top's output) is much higher
>> when built with ghc-7.6, due entirely to fragmentation.
>>
>> I've filed a bug report
>> (http://hackage.haskell.org/trac/ghc/ticket/7257,
>> http://hpaste.org/74987), but I was wondering if anyone else has
>> noticed this?  I'm not entirely sure what's triggering this behavior
>> (some applications work fine), although I suspect it has to do with
>> allocation of pinned memory.
>>
>> John L.
>>
>> _______________________________________________
>> Glasgow-haskell-users mailing list
>> Glasgow-haskell-users at haskell.org
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
>