[Haskell-community] 2018 state of Haskell survey results

Sun Nov 18 13:11:07 UTC 2018

Thanks for finding those anomalies, Gershom! I'm disappointed that
someone submitted bogus responses, apparently to tip the scales of Cabal
versus Stack. I intend to identify those responses and exclude them from
the results. The work you've done so far will help a great deal in
finding them.
You said that there are about 1,200 responses with demographic
information. That makes sense considering the number of submissions I
got last year. Also, there are 1,185 responses that included an answer
to at least one of the free-response questions. So perhaps whoever wrote
the script didn't bother to put an answer for those types of questions.
Unfortunately I do not have precise submission times or IP address
information about submissions. Beyond what's in the CSV, the only other
thing I have is (some) email addresses.
Fortunately I wrote a script to output all the charts and tables from
the survey responses. Once I've identified the problematic responses, I
should be able to update the script to ignore them and regenerate all
the output.

On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote:
> Sadly, it looks like a Cabal/Stack thing.  Of the responses with a
> country provided, 618 of 1226 claim to use Cabal, and 948 of 1226
> claim to use Stack. Of the responses with no country, only 35 of 3868
> claim to use Cabal, while 3781 of the 3868 claim to use Stack.
> Assuming independence, you'd expect that last number to be about 50,
> meaning there are probably around 3700 fake responses generated just
> to answer "Stack".> 
> To partially answer Simon's question, the flood of no-demographics
> responses started on November 2, around the 750-response point, and
> continued unabated through the close of the survey.  And, indeed,
> looking at just the first 750 responses gives similar distributions
> to what we get by ignoring the no-demographic responses.  For
> example, of the first 750 responses, 359 claim to use Cabal, and 568
> claim to use Stack.> 
> On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow
> <marlowsd at gmail.com> wrote:>> Good spot Gershom. Maybe it would be revealing to look at the times
>> that responses were received for the no-demographics group?>> 
>> On Sun, 18 Nov 2018, 07:17 Gershom B <gershomb at gmail.com wrote:
>>> I also noticed a number of other bizarre statistical anomolies when
>>> looking at the full results. I know this is a bit much to ask — but
>>> if you could rerun the statistics filtering out people that did not
>>> give demographic information (i.e. country of origin or education,
>>> etc) I think the results will change drastically. By all statistical
>>> logic, this should _not_ be the case, and points to a serious
>>> problem.>>> 
>>> In particular, this drops the results by a huge amount — only 1,200
>>> or so remain. However, the remaining results tend to make a lot more
>>> sense. For example — of the “no demographics” group, there are 713
>>> users who claim to develop with notepad++ but all of these say they
>>> develop on mac and linux, and none on windows — which is impossible,
>>> as notepad++ is a windows program. Further if you drop the “no
>>> demographics” group, then you find that almost everyone uses at
>>> least ghc 8.0.2, while in the “no demographics” group,  a stunning
>>> number of people claim to be on 7.8.3. Even more bizarrely, people
>>> claim to be using the 7.8 series while only having used Haskell for
>>> less than one year. And people claim to have used haskell for “one
>>> week to one month” and also to be advanced and expert users!>>> 
>>> The differences continue and defy all probability. Of the “no
>>> demographics” group, almost everyone dislikes the new release
>>> schedule. Of the “demographics” group there are answers that like
>>> it, were not aware of it, or are indifferent, but almost nobody
>>> dislikes it. There is naturally a difference in proportions of
>>> cabal/stack and hackage/stackage responses as well.>>> 
>>> There are a lot of other things I could point to as well. But,
>>> bluntly put, I think that some disaffected party or parties wrote a
>>> crude script and submitted over 3,000 fake responses. Luckily for
>>> us, they were not very smart, and made some obvious errors, so in
>>> this case we can weed out the bad responses (although, sadly, losing
>>> at least a few real ones as well).>>> 
>>> However, assuming  this party isn’t entirely stupid, it doesn’t bode
>>> well for future surveys as they may get at least slightly less dumb
>>> in the future if they decide to keep it up :-/>>> 
>>> —Gershom
>>> 
>>> 
>>> 
>>> On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com)
>>> wrote:>>>> 
>>>> 
>>>> This is interesting, but I’m thoroughly confused. Over 2500 people
>>>> said they took last year’s survey, but it only had roughly 1,300
>>>> respondants?>>>> 
>>>> 
>>>> On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak <taylor at fausak.me>
>>>> wrote:
>>>>> Hello! It took a little longer than I expected, but I am nearly
>>>>> ready to announce the 2018 state of Haskell survey results. Some
>>>>> community members have expressed interest in seeing the
>>>>> announcement post before it's published. If you are one of those
>>>>> people, you can see the results here:
>>>>> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown
>>>>>
>>>>>  If you would like to suggest changes to the announcement post,
>>>>>  please respond to this email, send me an email directly, or reply
>>>>>  to this pull request on GitHub:
>>>>>  https://github.com/tfausak/tfausak.github.io/pull/148
>>>>>
>>>>>  I plan on publishing the results tomorrow. Once the results are
>>>>>  published, the post is by no means set in stone. I will happily
>>>>>  accept suggestions from anyone at any time.
>>>>>
>>>>>  Thank you!
>>>>>  _______________________________________________
>>>>>  Haskell-community mailing list Haskell-community at haskell.org
>>>>>  http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>>> 
>>> _______________________________________________
>>>  Haskell-community mailing list
>>> Haskell-community at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> _______________________________________________
>>  Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
> _________________________________________________
> Haskell-community mailing list
> Haskell-community at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-community/attachments/20181118/76df66b8/attachment.html>