[Haskell-community] 2018 state of Haskell survey results

[Richard Eisenberg]

OK. Thanks for sharing some statistics. I'm now convinced as to the characterization of the attack. I'm still glad for how the public post diplomatically handled this.

> I will also say, though there's speculation about "false flags" and

Oof. That thought never crossed my mind. I can only imagine this is on some social media where I don't participate. Every day, I am more and more pleased with my non-presence on most social media. :) Besides, just keeping up with email is enough of a challenge.

Thanks for the clarification.

Richard

> other silliness floating around that I personally have a very good
> guess as to who did this. There's one well-known troll who has these
> preoccupations and is known for creating serial sockpuppet accounts,
> and is just the right amount of obsessed to do something like this. A
> few of the bogus responses actually had comments, and the comments
> were all written in a voice that was unmistakeable as this troll as
> well. Occam's razor seems to apply.
> 
> Finally, let me add why I don't think this was a "false flag" -- while
> there were enough telltale markers that the fake answers could seem to
> be detected, I don't think this was on purpose. There was _too much_
> effort put into distributions of other choices, etc. If they had
> wanted the fakes to be detected they would have left much stronger
> evidence. Rather, from a forensic standpoint, this seems pretty clear
> to me that the pattern of data is of someone _trying_ to cover their
> tracks, but just making four or five errors which I could assemble
> into a pattern. If they hadn't made those errors -- likely based on
> bad priors about what the organic data would be that theirs would need
> to "mesh" into -- then I think the deception would have been much
> harder to detect.
> 
> --Gershom
> 
>> Given the contention around cabal vs stack, I agree that sociological concerns suggest that the troll meant to tilt those scales. But I wouldn't want a public accusation without at least some statistical analysis that independently supports that conclusion.
>> 
>> In any case, thanks to all for putting this together!
>> 
>> Richard
>> 
>> On Nov 18, 2018, at 4:31 PM, Taylor Fausak <taylor at fausak.me> wrote:
>> 
>> Oops, the ordering of the answer choices is manual because some questions have a natural order while others should just be most to least popular. I've made another run through to make sure everything is sorted properly. I'll probably hit publish in the next half hour or so unless there are any objections.
>> 
>> https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown
>> 
>> 
>> On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote:
>> 
>> The language extensions section doesn’t appear to be sorted properly. Outside of that, I think that these results are looking much better and any effort to find any additional outliers is probably not worth it for the moment. Thanks for your work on this, and I appreciate you being responsive and attentive when problems with the data were pointed out. There’s certainly some interesting and helpful information to be gleaned from this data.
>> 
>> Cheers,
>> Gershom
>> 
>> 
>> 
>> 
>> On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me) wrote:
>> 
>> 
>> 
>> 
>> Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this.
>> 
>> https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown
>> 
>> 
>> On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote:
>> 
>> Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report.
>> 
>> On 18 Nov 2018, at 21:17, Taylor Fausak <taylor at fausak.me> wrote:
>> 
>> Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java.
>> 
>> That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses.
>> 
>> 
>> On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote:
>> 
>> What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year?
>> 
>> The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary.
>> 
>> 
>> Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels?
>> 
>> Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats.
>> 
>> Is either library likely to be influenced by this survey?
>> _______________________________________________
>> Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> 
>> 
>> _______________________________________________
>> Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> 
>> 
>> _______________________________________________
>> Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> 
>> 
>> _______________________________________________
>> Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> 
>>