hackage-server: index format

[Antoine Latter]

On Fri, Nov 19, 2010 at 7:01 AM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> On Fri, 2010-11-19 at 12:27 +0000, Duncan Coutts wrote:
>
>> Matt and I also discussed making the 00-index.tar.gz into a RESTful
>> format by adding proper URLs for package tarballs.
>
> Indeed we could go further and use a single general format for
> describing or distributing bundles of packages.
>
> Use case: local build trees
> ---------------------------
>
> A bunch of related packages (e.g. gtk2hs, happstack-* etc) unpacked
> locally.
>
> /home/me/prgs/myproj/foo/            --top of source tree for foo
> /home/me/prgs/myproj/foo/foo.cabal
> /home/me/prgs/myproj/bar/
> /home/me/prgs/myproj/bar/bar.cabal
>
> Now we can have an index.tar containing symlinks to .cabal files!
>
> /home/me/prgs/myproj/index.tar: containing
>        foo.cabal -> foo/foo.cabal
>        bar.cabal -> bar/bar.cabal
>
> So these are not copies of the .cabal files, these really are symlinks
> to the local .cabal files (but inside the tarball). I guess we need some
> extra index entry to point to the location of the source tree, though
> it's not a .tar.gz kind.
>
> Now just as we can have symlinks (or really URLs) inside the tarball, we
> could also have full file contents there too. Next use case...
>
> Use case: distribution bundles
> ------------------------------
>
> Shipping a bunch of source packages as a single file
>
> some-name.tar: containing
>        foo.cabal
>        foo-1.0.tar.gz
>        bar.cabal
>        bar-1.0.tar.gz
>
> So now instead of symlinks/URLs to separate tarballs, the whole file
> contents is right there. We have a hackage-like index plus the file
> tarballs.
>
>
> We might have to have a different naming convention than simply blah.tar
> for these indexes, otherwise cabal install might not know how to
> interpret  "cabal install foo.tar" should it interpret foo.tar as an
> index or as a single package?
>
> Opinions?
>

It feels like an abuse of tar-files to me - if we want to have a set
of meta-data about the location of resources in a package repository,
I think it would be better to come up with a file format that has the
information we want directly and then serve it up.

This hypothetical cabal-repository.description file would be pointed
at by a user's .cabal/conf, and the config file would describe either
what resources the repo makes available or how to discover what
resources it makes available.

So for a small repo, this file could contain a listing of package ids
and where the tar-ball/package descriptions are.

We could even have a special case for local or file-share hosted
repositories - the presence of an empty repo description file would
imply that the contents of the repo is every tar, tar.gz or directory
containing a .cabal file in the top level.

A larger repository would point to another file which contains a
collection of packages and their meta-data. One of the resources could
be "here's where to find a tarball containing the package descriptions
of every package I know how to serve" to support the current model of
solving dependencies based. In this scenario the 'repo description'
files would exactly be a REST description of the contents of Hackage
Server.

It's the same information as what you'd wanted to put in the index
tarball, and we might even want to make it so that the repo config
file can live in the tarball and address resources in the tarball it
is hosted in (so I can deply a local cabal repo by dropping a tarball
into a fileshare).

But slipstreaming metadata into soft-links in a tarball feels weird,
and since we need client changes to make it work we may as well do it
right.

Does this sort of approach sound sensible? I don't mind fleshing it
out more as a start.

Antoine