[Haskell-cafe] Slow documentation generation on Hackage

[wren ng thornton]

Thomas ten Cate wrote:
> Niemeijer, R.A. wrote:
> > which, face it, is going to be all of them; I doubt Haskell
> > is popular enough yet to be the target of DoS attacks
> 
> Second that. I think this is a good case in which some security should
> be traded in for usability.

Those who would trade security for usability deserve neither usability 
nor security ;)

Seriously, all the Haskell hackers I've encountered have been good 
people, but Haskell is the language of the hair shirt afterall. Security 
is hard enough to come by in the first place, sacrificing what little we 
have is not the right path. The Haskell interwebs are already too 
susceptible to downtimes from non-malicious sources, and it floods 
#haskell whenever it happens.

I agree that the turnaround time is a bit long, but I think server 
stability is more important than instant feedback. It's easy enough to 
get an account on community.haskell.org and just upload your own docs 
there[1]. The thing I'd be more interested in getting quick feedback on 
is whether compilation succeeds in the Hackage environment, which is 
very different from my own build environment.

Given the various constraints mentioned, and depending on the load 
averages for the servers, perhaps the simplest thing to do would be to 
just reduce the latency somewhat. Flushing the queue every 4~6 hrs seems 
both long enough to circumvent the major DoS problems, and short enough 
to be helpful to developers. Especially if the queue could be set up to 
be fair among users (e.g. giving each user some fixed number of slots or 
cycles per flush, delaying the rest until the next cycle). A different 
approach would be to do exponential slowdown per user. So if a user has 
submitted N jobs in the last Window (e.g. 24 hours), then the job is run 
around Epsilon^N after submission (where Epsilon is, say, 3 minutes).


[1] I do: http://community.haskell.org/~wren/ The scripts to automate it 
are trivial, but I can share them if people like.

-- 
Live well,
~wren