hackage reverse dependencies

[Gershom B]

It seems to me that all we really need is a list of all first
generation revdeps on a given package (any version), each specified
maybe as (in pseudodata):

type VersionRange = (Version,Version)
type RevDep = (PackageName,[VersionRange])

RevDeps = [RevDep] (alt M.Map PackageName [VersionRange])

RevDepStructure = M.Map PackageName RevDeps

I can see how things get uglier if we instead do:

RevDepStructure = M.Map PackageName (M.Map Version RevDeps)

but I think the latter structure is potentially too granular for most uses?

--Gershom


On Mon, Oct 14, 2013 at 2:44 PM, Duncan Coutts <duncan at well-typed.com> wrote:
> On Mon, 2013-10-14 at 12:18 -0400, Carter Schonwald wrote:
>> So you're saying we need to add a db proper to hackage2 server, like SQLite
>> or Postgres so as to make it more performant for interesting features?
>>  What's needed to do that?
>
> I don't think that's needed at all.
>
> What we need is to look carefully at what we want from a revdeps feature
> and then look carefully at what info should be stored (ie cached) and
> what should be calculated on demand.
>
> It just needs someone to spend some time on the feature, there's no need
> to panic about the data storage (especially before looking at what is
> actually needed).
>
> Duncan
>
>> On Monday, October 14, 2013, Matthew Gruen wrote:
>>
>> > Hey, thanks for taking an interest in this. There is kind of a
>> > space-time-featureset tradeoff with revdeps, unfortunately. It seems to me
>> > like the challenge doesn't come so much from indirect deps, but rather from
>> > the sheer number of versions there are! There are 5600+ packages and 33800+
>> > versions, and the set of dependencies of foo-1.2 vs foo-1.3 can be
>> > completely different. Not to mention, when you look at the package page for
>> > a particular version, you may want to see the revdeps *for that version*,
>> > and at this point there are a billion possible combinations. npm's UI, in
>> > contrast, has an "always in HEAD" kind of feel.
>> >
>> > If you are interested in how the disabled implementation worked, by the
>> > way, I have a post here (as gracenotes):
>> > https://github.com/haskell/hackage-server/issues/40
>> >
>> > So revdeps is great, and there are many different killer uses for it, such
>> > as popularity metrics and finding upper version bounds which need updating.
>> > In sum, these require keeping around a lot of data. Because there's so
>> > much, acid-state is probably not a good place to store it. Not to mention,
>> > it's time-consuming to generate it from scratch and space-consuming to keep
>> > around data structures to incrementally update it. So there are some tool
>> > limitations, but it seems fundamentally tricky to do frugally.
>> >
>> > Matt
>> >
>> >
>> >
>> > On Mon, Oct 14, 2013 at 12:20 AM, Jens Petersen <
>> > juhp at community.haskell.org> wrote:
>> >
>> >> Not really complaining :) but one of the things I had been looking
>> >> forward to with Hackage2 was the display of reverse-dependencies, but I
>> >> gather it was disabled for now (because it loads the server too much when
>> >> updating the data iirc?).
>> >>
>> >> I was thinking perhaps a compromise would be to show only direct
>> >> reverse-dependencies rather than including all indirect dependencies too.
>> >>  Perhaps that would make it scale better?  I think direct dependency
>> >> information is the most useful anyway (and also what packdeps provides).
>> >>
>> >> Before doing anything I thought I would ask here first if this makes
>> >> sense and should be work?
>> >> I for one would still really like to see basic reverse dependencies info
>> >> listed on hackage (I noticed yesterday that npm has them too).
>> >>
>> >> Jens
>> >>
>> >> _______________________________________________
>> >> cabal-devel mailing list
>> >> cabal-devel at haskell.org
>> >> http://www.haskell.org/mailman/listinfo/cabal-devel
>> >>
>> >>
>> >
>> _______________________________________________
>> cabal-devel mailing list
>> cabal-devel at haskell.org
>> http://www.haskell.org/mailman/listinfo/cabal-devel
>
>
> --
> Duncan Coutts, Haskell Consultant
> Well-Typed LLP, http://www.well-typed.com/
>
> _______________________________________________
> cabal-devel mailing list
> cabal-devel at haskell.org
> http://www.haskell.org/mailman/listinfo/cabal-devel