[Haskell-cafe] GSoC: Hackage 2.0

[Matthew Gruen]

Hi Haskellers,

I'm Matt Gruen (Gracenotes in #haskell), and the Hackage 2.0 SoC
project at <http://hackage.haskell.org/trac/summer-of-code/ticket/1587>
really piqued my interest. It seems doable, in a summer, to make the
new hackage-server more-than-deployment-ready as well as clearing out
some items in the hackage bug tracker[0]; so, I've been working on a
proposal. In this email I'd like to consolidate my mental notes for
haskell-cafe and formulate a roadmap towards a more social Hackage.

The most vital part is getting hackage-server
<http://code.haskell.org/hackage-server/> to a state where it can be
switched in place of hackage-scripts
<http://darcs.haskell.org/hackage-scripts/>, and doing it properly,
organizing the code so it can be extended painlessly in the future.
Duncan Coutts, Antoine Latter, and others have done some great work on
it in the past few years. I've been using Haskell for 1.33 of those
years. I think I could become fluent in the codebases after a week of
dedicated study, although even the hackage-server and hackage-scripts
repositories don't include some components such as the current build
report script.

For putting the 2.0 in Hackage 2.0, any interface changes should help
the library users and the library writers/uploaders without hurting
either of them. Hackage should contain more of the right kind of
information. Statistics help everyone, and they're a pretty good gauge
on the direction of Hackage as a whole. Package popularity contents
are one form of this. Reverse dependencies and even dependency
graphs[1] are great, if I can integrate and expand Roel van Dijk's
work[2].

There should also be some space on package pages, or on pages a link
away from them, for users to contribute information and suggestions.
Coders can explain why or why not the package met their needs, as a
sort of informal bug/enhancement tracking service. Another helpful
flavor of information is package relationships beyond dependencies:
'Deprecated in favor of Foo', 'a fork of Foo', or 'does blah instead
of Foo's plugh functionality'. This will help people find the 'right'
package among look-alikes. There could also be links to illustrative
usage examples for getting started. Happstack's state management will
be a good match for updating and serving this information in real
time.

There's also a need for a more interactive form of package
documentation, but this should strengthen relationships with existing
tools like Haddock and Cabal, not bypass the tools. For example,
adding a changelog[3] or making Haddock's declaration-by-declaration
commentary more wiki-like[4]. Changelogs seem to be within the scope
of Hackage 2.0, integrating with Cabal; Haddock wikification might not
be, perhaps deserving a separate student-summer session of its own.
These can improve the package page and documentation subtrees.

More generally, how can library users find the package they want?
Categories themselves are great, but a tag system could identify and
group specific package functionality. There could be sorting by
ratings and reviews (4/5 lambdas!). Metadata searches, like those
Sascha Böhme implemented in SoC 2007[5], could be integrated. It's not
always obvious which ideas will help and which won't see good returns,
which makes it all the more important to bring hackage-server to a
state where future extensions can be easily written, submitted and
deployed. That's the goal here.

On the technical side, I realize I'd need to spend a not-insignificant
amount of time on a user account system, dealing with authentication
and related issues. One additional bit of functionality to manage is
the hackage build system, which is used to ensure that packages build
and to generate documentation. When building depends on FFI or
OS-specific bindings, specific versions of other packages, compiler
choice or compiler version choice, including language extensions, this
is not trivial. One of two good routes is running cabal server-side to
generate build reports and alleviate the miscompiling a smidge. Given
that cabal install dependency calculation seems to be up in the air
still, the current Setup-running script might end up staying for the
moment, with some basic integrity checking (no cycles or other
impossible dependency scenarios). The other not-mutually-exclusive
approach is accepting failed build reports from users as a web
service[6] to generate a matrix of the platforms that seem to
encounter the most trouble.

Ultimately, it's important that I, or whoever ends up doing the
project, plans for it to benefit you guys. What do you think?

—Matt Gruen

PS: how do you feel about augmenting the .cabal format to provide more
information for Hackage?

[0] Active Tickets by Component, scroll down to HackageDB.
http://hackage.haskell.org/trac/hackage/report/9?USER=anonymous&page=3
[1] Re: ANN: Reverse Dependencies in Hackage (demo).
http://www.haskell.org/pipermail/haskell-cafe/2009-October/067847.html
[2] Show reverse dependencies. Includes a patch for hackage-scripts.
http://hackage.haskell.org/trac/hackage/ticket/576
[3] Add changelog feature to hackagedb.
http://hackage.haskell.org/trac/hackage/ticket/299
[4] Reddit comment. The other comments are good too.
http://www.reddit.com/r/haskell/comments/8bylw/ask_haskell_reddit_how_can_we_improve_the/c08tc0q
[5] Hackage Web Interface, Doc-Browser.
http://code.google.com/soc/2007/haskell/appinfo.html?csaid=D505A03B8B64C3BF
[6] cabal-install should report build results to hackage server.
http://hackage.haskell.org/trac/hackage/ticket/184