Pickling a finite map (Binary + zlib) [was: [Haskell-cafe]Data.Binary poor read performance]

[Claus Reinke]

>> btw, i always thought that it should be a way to overcome any export
>> lists and go directly to module internals. limiting export is the way
>> to protect programmer from errors, not security feature, and it should
>> be left to programmer to decide when he don't need it. compilers
>> should just be able to check whether some module/library imported
>> abstractly or with internals too. this will render useless all those
>> .Internals modules that now we need to add everywhere

You're not alone!-) This has been called "Open Implementation" 
(OI, a pre-cursor of aspect-oriented programming):

    http://www2.parc.com/csl/groups/sda/projects/oi/

They argue for an explicit auxiliary interface instead of full access 
to module internals. Since these same folks worked on meta-object 
protocols as well (at the meta-level, the boundaries can be bypassed 
entirely), that suggestion probably comes from experience. They do 
allow for the auxiliary interface to use meta-programming style 
features, though that depends on the language in question (in 
Haskell, type classes or type functions might be used instead,
but rewrite rules and Template Haskell are also available).

    Open Implementation Design Guidelines 
    http://www2.parc.com/csl/groups/sda/publications/papers/Kiczales-ICSE97/

is a short paper discussing a Set API/Open Implementation example.

> I agree in principle, but GHC also uses that knowledge to optimize the
> code better - if a function is exported it has to produce the most
> polymorphic possible code for its type, if it isn't it can specialize
> better... that sort of thing.

That refers to optimization in the provider module. As the OI
people argued, optimization in the client modules also needs to
be taken into account. If the default one-size-fits-all-uses
implementation behind the default API doesn't work well 
enough, there'll be a proliferation of library variants. If there 
is a way to fine-tune the implementation via an auxiliary API,
much of that can be avoided. 

In other words, instead of half a dozen Maps and a dozen 
Array variants, there'd just be one of each, but with auxiliary 
interfaces that would allow client code to choose and tune 
the most suitable implementations behind the main interfaces.

It isn't magic, though: if, say, even the auxiliary API doesn't
allow you to say "thanks, but I know the length already", you're
still stuck. But it does help to think about designing 2 APIs
(the default functionality one and the auxiliary fine-tuning one)
instead of offering only the choice of 1 API or full access to 
internals.

Claus