[Haskell-cafe] Aeson memory use

[Claude Heiland-Allen]

Hi,

The test.data is very repetitive:

{"1":["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"],"10":["a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"],...}

Perhaps (after parsing (which might fuse enough to avoid a memory spike), otherwise during parsing might require modifications to aeson?) you could compress it by interning the symbols using a `Map Text Text` to generate one canonical `Text` object for each unique string.

`pack "a" == pack "a"` under `Eq` but they might be different `Text` objects.

You might also need to `copy` the `Text` objects, which might be slices referencing the input.


On 08/08/18 21:24, Vanessa McHale wrote:
> I'm not 100% sure it's applicable in your case, but perhaps trying
> threadscope would be a good idea?
>
> Based on the heap profile, I'm guessing that most of the allocations are
> due to slicing ByteStrings up. What happens when you try to read a map
> of Integers?
>
> I'd also note that your example uses evaluate and performGC which isn't
> entirely realistic and thus may not actually correspond to the
> performance of the application in practice. I gather it *was* a problem
> in practice but nonetheless it's always good to benchmark the right thing.
>
> On 08/08/2018 02:54 PM, Lana Black wrote:
>> Hello Cafe,
>>
>> Recently while trying to optimize some code, I noticed that aeson
>> requires quite a substantial amount of memory when parsing map-like
>> structures. The quick example I crafted [1] is quite surprising. In
>> order to parse an 11 megabyte file aeson requires around 300 megabytes
>> of memory (according to the heap profiler), and the total memory
>> footprint (according to htop) jumps to around 700 megabytes.
>>
>> Is there a way to reduce this number to something less insane? I
>> suspect that I might be doing something wrong here, but I can't
>> exactly see what.
>>
>> [1] https://github.com/greydot/aeson-example

Claude
-- 
https://mathr.co.uk