hackage reverse dependencies

[Duncan Coutts]

On Mon, 2013-10-14 at 12:18 -0400, Carter Schonwald wrote:
> So you're saying we need to add a db proper to hackage2 server, like SQLite
> or Postgres so as to make it more performant for interesting features?
>  What's needed to do that?

I don't think that's needed at all.

What we need is to look carefully at what we want from a revdeps feature
and then look carefully at what info should be stored (ie cached) and
what should be calculated on demand.

It just needs someone to spend some time on the feature, there's no need
to panic about the data storage (especially before looking at what is
actually needed).

Duncan

> On Monday, October 14, 2013, Matthew Gruen wrote:
> 
> > Hey, thanks for taking an interest in this. There is kind of a
> > space-time-featureset tradeoff with revdeps, unfortunately. It seems to me
> > like the challenge doesn't come so much from indirect deps, but rather from
> > the sheer number of versions there are! There are 5600+ packages and 33800+
> > versions, and the set of dependencies of foo-1.2 vs foo-1.3 can be
> > completely different. Not to mention, when you look at the package page for
> > a particular version, you may want to see the revdeps *for that version*,
> > and at this point there are a billion possible combinations. npm's UI, in
> > contrast, has an "always in HEAD" kind of feel.
> >
> > If you are interested in how the disabled implementation worked, by the
> > way, I have a post here (as gracenotes):
> > https://github.com/haskell/hackage-server/issues/40
> >
> > So revdeps is great, and there are many different killer uses for it, such
> > as popularity metrics and finding upper version bounds which need updating.
> > In sum, these require keeping around a lot of data. Because there's so
> > much, acid-state is probably not a good place to store it. Not to mention,
> > it's time-consuming to generate it from scratch and space-consuming to keep
> > around data structures to incrementally update it. So there are some tool
> > limitations, but it seems fundamentally tricky to do frugally.
> >
> > Matt
> >
> >
> >
> > On Mon, Oct 14, 2013 at 12:20 AM, Jens Petersen <
> > juhp at community.haskell.org> wrote:
> >
> >> Not really complaining :) but one of the things I had been looking
> >> forward to with Hackage2 was the display of reverse-dependencies, but I
> >> gather it was disabled for now (because it loads the server too much when
> >> updating the data iirc?).
> >>
> >> I was thinking perhaps a compromise would be to show only direct
> >> reverse-dependencies rather than including all indirect dependencies too.
> >>  Perhaps that would make it scale better?  I think direct dependency
> >> information is the most useful anyway (and also what packdeps provides).
> >>
> >> Before doing anything I thought I would ask here first if this makes
> >> sense and should be work?
> >> I for one would still really like to see basic reverse dependencies info
> >> listed on hackage (I noticed yesterday that npm has them too).
> >>
> >> Jens
> >>
> >> _______________________________________________
> >> cabal-devel mailing list
> >> cabal-devel at haskell.org
> >> http://www.haskell.org/mailman/listinfo/cabal-devel
> >>
> >>
> >
> _______________________________________________
> cabal-devel mailing list
> cabal-devel at haskell.org
> http://www.haskell.org/mailman/listinfo/cabal-devel


-- 
Duncan Coutts, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/