Restricted commandline lenghts

[Duncan Coutts]

On Fri, 2006-08-18 at 01:50 +0300, Esa Ilari Vuokko wrote:
> Hi,
> 
> Ticket #19 [1], and maybe some big projects in general, need to be able
> to invoke ar/ld so that we don't run into issues in commandline lenght.
> (And in theory, compiler's as well, but we that'd require the dreaded
> deps analysis.)  I think this is major feature we should resolve before
> GHC 6.6 goes out.  It affects significantly some libraries, at least
> gtk2hs, and I can see real size benefits on some of my own libraries
> as well.
> 
> I posted patch to fix this in some cases for Windows [2], but upon
> talking about it with Duncan, in irc, there's few exta points:
>  * Some unixy shells apparently have restrictions as well, so this
>    shouldn't be just Windows.
>  * On my old implementation, I just split the commandlines by filecount,
>    not according to actual space the paths take.  Is this a major
>    point?  It can affect the number of tool invokations quite radically.
>  * The algorithm should really be parametriseable because it is, I believe,
>    in all cases heuristic.
> 
> As how to proceed on implementing this, I'm a bit unsure.
>  * Because of unicode conversion issues, I am not entirely sure if we
>    can accurately know the length of commandline, atleast in Windows.
>    Or going for the bad case, 4 bytes per character gives us a lot of
>    setback in common case.
>  * I think the common way to workaround this is by using xargs with
>    some constant number per params (much like my current algos).
>  * I think there is a diffrence between complexities, because with
>    ar we really want to do append, but with ld we can do a
>    tree-style build.  I haven't benchmarked this.
>  * This might be useful to generalise into library function(s).

I've sent a patch to the list in another email.

What it does is take an xargs approach. It adds an xargs function and
uses for both the ar and ld cases. It calculates the length of the
command line string and uses as large a number of arguments as will fit,
given a maximum size.

Doing a minimum number of invocations of ar is quite important as at
least GNU ar is shockingly bad when linking thousands of little .o files
into an archive .a file. At the moment, linking GHC's libHSbase.a takes
several invocations of ar via xargs and takes >500Mb of memory. I did
actually submit a patch to binutils to bring this down to a mere 100Mb
but that's only in the very latest binutils versions and it's still
quite a lot. So I've put an arbitrary 32k in for unix systems, but I
think many are actually larger than this, eg 128k so it might be good to
either statically or dynamically find this.

For ld since it doesn't do append I've made output to a temp file and it
link in the previous accumulated .o file if it exists. Then it renames
the temp file to the final target.

I've tested it on linux, just with a very small maximum commandline size
and checked that it does invoke ar/ld multiple times and that the
resulting binaries work.

> I could use some advice or experiences from other buildsystems.  If nobody
> else steps up, I'll probably implement some choice (but I'm only prepared
> to test it in Windows.)

It'd be great if you could test this patch on Windows.

Duncan