[Haskell-cafe] Improvements to package hosting and security

[Mathieu Boespflug]

Thank you for that Gershom. I think everything that you're saying in
that last email is very much on the mark. Multiple proposals is
certainly a good thing for diversity, a good thing to help take our
infrastructure in a good direction, and a good thing to help it evolve
over time. It's true that most of us are volunteer contributors,
working on improving infrastructure only so long as it's fun. So it's
not always easy to ask for more upfront clarity and pitch perfect
coordination. Then again as a community we make more progress faster
when a little bit of process is followed.

While a millions different tools or libraries to do the same thing can
coexist just fine, with infrastructure that's much more difficult. A
single global view of all code that people choose to contribute as
open source is much healthier than a fragmented set of sub communities
each working with their own infrastructure. So the degree of
coordination required to make infrastructure evolve is much higher. To
this end, I'd like to strongly encourage all interested parties to
publish into the open proposals covering one or both of the topics
that are currently hot infrastructure topics in the community:

1. reliable and efficient distribution of package metadata, package
content and of incremental updates thereof.
2. robust and convenient checking of the provenance of a package
version and policies for rejecting such package versions as
potentially unsafe.

These two topics overlap of course, so as has been the case so far
often folks will be addressing both simultaneously. I submit that it
would be most helpful if these proposals were structured as follows:

* Requirements addressed by the proposal (including *thread model*
where relevant)
* Technical details
* Ideally, some indication of the resources needed and a timeline.

I know that the last point is of particular interest to commercial
users, who like predictability in order to decide whether or not they
need to be chipping in their own meagre resources to make the proposal
happen and happen soon. But to some extent so does everyone else: no
one likes to see the same discussions drag on for 2+ years. Openness
really helps here - if things end up dragging out others can pick up
the baton where it was left lying.

So far we have at least 2 proposals that cover at least the first two
sections above:

* Chris Done's package signing proposal:
https://github.com/commercialhaskell/commercialhaskell/wiki/Package-signing-proposal
* Duncan Coutts and Austin Seipp's proposal for improving Hackage
security: http://www.well-typed.com/blog/2015/04/improving-hackage-security/

There are other draft (or "strawman") proposals (including one of
mine) floating around out there, mentioned earlier in this thread. And
then some (including prototype implementations) that I can say I and
others have engaged with via private communication, but it would
really help this discussion move forward if they were public.

> One idea I have been thinking about, is a Birds of a Feather meeting at the upcoming ICFP in Vancouver focused just on Haskell Open-Source Infrastructure.

I think that's a great idea.

Best,

Mathieu

On 16 April 2015 at 17:06, Gershom B <gershomb at gmail.com> wrote:
> On April 16, 2015 at 8:39:40 AM, Mathieu Boespflug (mboes at tweag.net) wrote:
>
>> It ultimately hurts the community when people repeatedly say things to
>> the effect of, "yep, I hear you, interesting topic, I have a really
>> cool solution to all of what you're saying - will be done Real Soon
>> Now(tm)", or are happy to share details but only within a limited
>> circle of cognoscenti. Because the net result is that other interested
>> parties either unknowingly duplicate effort, or stall thinking that
>> others are tackling the issue, sometimes for years.
>
> I think this is a valid concern. Let me make a suggestion as to why this does not happen as much as we might like as well (other than not-enough-time which is always a common reason). Knowing a little about different people’s style of working on open source projects, I have observed that some people are keen to throw out lots of ideas and blog while their projects are in the very early stages of formation. Sometimes this leads to useful discussions, sometimes it leads to lots of premature bikeshedding. But, often, other people don’t feel comfortable throwing out what they know are rough and unfinished thoughts to the world. They would rather either polish the proposal more fully, or would like to have a sufficient proof-of-concept that they feel confident the idea is actually tractable. I do not mean to suggest one or the other style is “better” — just that these are different ways that people are comfortable working, and they are hardwired rather deeply into their habits.
>
> In a single commercial development environment, these things are relatively more straightforward to mediate, because project plans are often set top down, and there are in fact people whose job it is to amalgamate information between different developers and teams. In an open source community things are necessarily looser. There are going to be a range of such styles and approaches, and while it is sort of a pain to negotiate between all of them, I don’t really see an alternative.
>
> So let me pose the opposite thing too: if there is a set of concerns/ideas involving core infrastructure and possible future plans, it would be good to reach out to the people most involved with that work and check if they have any projects underway but perhaps not widely announced that you might want to be aware of. I know that it feels it would be better to have more frequent updates on what projects are kicking around and what timetables. But contrariwise, it also feels it would be better to have more people investigate more as they start to pursue such projects.
>
> Also, it is good to have different proposals on the table, so that we can compare them and stack up what they do and don’t solve more clearly. So, to an extent, I welcome duplication of proposals as long as the discussion doesn’t fragment too far. And it is also good to have a few proofs-of-concept floating about to help pin down the issues better. All this is also very much in the open source spirit.
>
> One idea I have been thinking about, is a Birds of a Feather meeting at the upcoming ICFP in Vancouver focused just on Haskell Open-Source Infrastructure. That way a variety of people with a range of different ideas/projects/etc. could all get together in one room and share what they’re worried about and what they’re working on and what they’re maybe vaguely contemplating on working on. It’s great to see so much interest from so many quarters in various systems and improvements. Now to try and facilitate a bit more (loose) coordination between these endeavors!
>
> Cheers,
> Gershom
>
> P.S. as a general point to bystanders in this conversation — it seems to me one of the best ways to help the pace of “big ticket” cabal/hackage-server work would be to take a look at their outstanding lists of tracker issues and see if you feel comfortable jumping in on the smaller stuff. The more we can keep the little stuff under control, the better for the developers as a whole to start to implement more sweeping changes.
>
>