Hackage updating of distro data

[Matthew Gruen]

For Hacakge 2, the distro ecosystem hasn't had quite as much tooling put
into it as, say, documentation, and it has changed a little bit from
Hackage 1. Previously, there was a cron job that, among tasks like updating
documentation, also downloaded distro info and ran <
http://code.haskell.org/~ross/hackage-scripts/DistroInfo.hs> on it; the
resulting files were read when generating the package page in <
http://code.haskell.org/~ross/hackage-scripts/PackagePage.hs>.

No such job exists now. However, there is something to replace it in
Hackage 2. It is not a cron job; it's more push than pull, actually. Distro
maintainers can upload distro information directly to Hackage (either
manually, using a form, or programmatically using something like curl). It
is currently not exposed or documented that well, unfortunately, and it
needs a last 10% of work to be fully functional. If you'd like, I think I'd
be able to do it this week.

So what does this new API look like? Well, this is the list of distros we
know about:
http://hackage.haskell.org/distros/
And the set of packages we know about for Fedora:
http://hackage.haskell.org/distro/Fedora/packages
http://hackage.haskell.org/distro/Fedora/packages.csv
The status for a given package:
http://hackage.haskell.org/distro/Fedora/package/text
Finally, shown on the package page itself:
http://hackage.haskell.org/package/text

There are per-distro ACLs so that the above information can kept up to date
by authorized accounts. There are a few issues with this system currently,
such as not being able to edit these ACLs over HTTP (!). Once this is
fixed, it will be possible to do:
1. Bulk update: PUT a csv formatted like the one above, which is similar to
the current format (the current format is a tuple, like
http://people.debian.org/~nomeata/cabalDebianMap.txt). This will wipe the
previous set of distro statuses for a given distro.
2. Add/update the distro status of an individual package: this is also a
PUT, using two form fields (e.g. version=0.11.2.3&uri=
https://apps.fedoraproject.org/packages/ghc-text)
3. Delete the distro status of an individual package: this is a DELETE,
perhaps meaning that the package is no longer offered

Unlike the previous system, using this *does* require Hackage credentials,
and if this data comes from an automatic task, a bot account could possibly
be used. It depends on what will work best with the current flow.

Best,
Matt



On Mon, Oct 14, 2013 at 12:08 AM, Jens Petersen
<juhp at community.haskell.org>wrote:

> Hi, it is great that Hackage 2 finally was deployed in production. :-)
>
> How often does Hackage 2 update/pull distro data?
>
> Hackage 1 had a cronjob I believe which ran every few hours I think.
> But the new Hackage does not seem to pull often/regularly.
>
> For Fedora I updated the data on 18 Sept and then 26 Sept.
> The update from 18th Sept is reflected in Hackage but not 26th Sept.
> Checking more I guess the 18 Sept was still during Hackage1
> so it looks like Hackage2 has not updated it distro data since
> going into production?
>
> Is there a cronjob setup on the new hackage server currently
> for pulling distro package data?
>
> Thanks, Jens
>
> _______________________________________________
> cabal-devel mailing list
> cabal-devel at haskell.org
> http://www.haskell.org/mailman/listinfo/cabal-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/cabal-devel/attachments/20131014/47f87525/attachment.html>