[jhc] Re: Ids in Jhc.

[John Meacham]

On Thu, Feb 14, 2008 at 01:23:08AM +0100, Lemmih wrote:
> I'd like to fix it in another way, if you have no objects.
> 
> By my count there are 120 unique atoms in base-1.0.hl. Those 120 atoms
> are saved ~2344 times each (there's a total of 281338 atoms).
> Compressing the files with gzip reduces the negative effect on file
> size but the overhead of parsing 281k atoms is still there.
> On my system, testing with HelloWorld.hs, parsing those atoms took 16%
> of the runtime and 27% of the memory usage.
> 
> I propose scrapping the current system of random atom ids in favor for
> a per-module map of symbols.
> Using an ADT for the ids would make the system more transparent. I'll
> have to investigate the performance impact of this.

What do you mean? just for saving binary files? it already converts them
to an ADT on the way out and back in since atom numbers can be different
between different runs of the program. 

If you mean using an ADT internally for ids,
stopping doing that is what brought memory usage from gigabytes to
megabytes. The text of the atoms only needs to be stored in a single
place, also the fact they are int's means they can be unboxed in
datatypes, meanding the garbage collector does not even have to care
about them, another huge gain. 

Also, IntSet/IntMap are orders of magnitude faster than Set and Map and
sets/maps of ids are the bed and butter of optimization passes.

my only issue with them is that it is

type Id = Int
instead of
newtype Id = Id Int

I have been working on the binary format with my recent changes, it is
now much less of an issue than it used to be in terms of speed and
memory usage. I modified it to use bytestrings and split the file into
various 'chunks' that can be lazily loaded independently. so when it
only needs the dependency information that is all it needs to pull out.
and so forth. I also put a hard limit on the length of atoms to 256
bytes, which means I can chop 7 bytes off the storage of each one.

By far the biggest win was in Info/Info, which used to store the type of
each field alongside of it as a string, now it stores a hash of the type
as a number, this reduced the file size (after compression) by about 30%.

        John

-- 
John Meacham - ⑆repetae.net⑆john⑈