[Haskell-cafe] Is it possible to have constant-space JSON decoding?

Iustin Pop iustin at google.com
Tue Dec 4 16:23:08 CET 2012


On Tue, Dec 04, 2012 at 03:58:24PM +0100, Herbert Valerio Riedel wrote:
> Iustin Pop <iustin at google.com> writes:
> 
> [...]
> 
> > Let's say we have a simple JSON message: an array of 5 million numbers.
> > I would like to parse this in constant space, such that if I only need
> > the last element, overall memory usage is low (yes, unrealistic use, but
> > please bear with me for a moment).
> >
> > Using aeson, I thought the following program will be nicely-behaved:
> 
> part of the problem is that aeson builds an intermediate JSON parse-tree
> which has quite an overhead for representing a list of numbers on the
> heap as each numbers requires multiple heap objects (see also [1]). This
> is an area where e.g. Python has a significantly smaller footprint
> (mostly due to a more efficient heap representation).

Ah, I see. Thanks for the link, so that's from where the 'S' constructor
was coming from in the -hd output.

And indeed, I was surprised as well that Python has a more efficient
representation for this.

> [...]
> 
> > It seems that the Array constructor holds a vector, and this results in
> > too much strictness?
> 
> btw, using a list on the other hand would add an overhead of 2 words
> (due to the ':' constructor) for representing each JSON array element in
> the parse-tree, that's probably why aeson uses vectors instead of lists.

Ack.

thanks,
iustin



More information about the Haskell-Cafe mailing list