[Haskell-community] 2018 state of Haskell survey results

Sun Nov 18 15:58:26 UTC 2018

I have filtered out the bogus responses and re-generated all the charts
and tables. You can see the updated results here:
https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown
Note that until I post the results on my blog, they are not published.
Please don't share the preliminary results on social media!

On Sun, Nov 18, 2018, at 8:11 AM, Taylor Fausak wrote:
> Thanks for finding those anomalies, Gershom! I'm disappointed that
> someone submitted bogus responses, apparently to tip the scales of
> Cabal versus Stack. I intend to identify those responses and exclude
> them from the results. The work you've done so far will help a great
> deal in finding them.> 
> You said that there are about 1,200 responses with demographic
> information. That makes sense considering the number of submissions I
> got last year. Also, there are 1,185 responses that included an answer
> to at least one of the free-response questions. So perhaps whoever
> wrote the script didn't bother to put an answer for those types of
> questions.> 
> Unfortunately I do not have precise submission times or IP address
> information about submissions. Beyond what's in the CSV, the only
> other thing I have is (some) email addresses.> 
> Fortunately I wrote a script to output all the charts and tables from
> the survey responses. Once I've identified the problematic responses,
> I should be able to update the script to ignore them and regenerate
> all the output.> 
> 
> On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote:
>> Sadly, it looks like a Cabal/Stack thing.  Of the responses with a
>> country provided, 618 of 1226 claim to use Cabal, and 948 of 1226
>> claim to use Stack. Of the responses with no country, only 35 of 3868
>> claim to use Cabal, while 3781 of the 3868 claim to use Stack.
>> Assuming independence, you'd expect that last number to be about 50,
>> meaning there are probably around 3700 fake responses generated just
>> to answer "Stack".>> 
>> To partially answer Simon's question, the flood of no-demographics
>> responses started on November 2, around the 750-response point, and
>> continued unabated through the close of the survey.  And, indeed,
>> looking at just the first 750 responses gives similar distributions
>> to what we get by ignoring the no-demographic responses.  For
>> example, of the first 750 responses, 359 claim to use Cabal, and 568
>> claim to use Stack.>> 
>> On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow
>> <marlowsd at gmail.com> wrote:>>> Good spot Gershom. Maybe it would be revealing to look at the times
>>> that responses were received for the no-demographics group?>>> 
>>> On Sun, 18 Nov 2018, 07:17 Gershom B <gershomb at gmail.com wrote:
>>>> I also noticed a number of other bizarre statistical anomolies when
>>>> looking at the full results. I know this is a bit much to ask — but
>>>> if you could rerun the statistics filtering out people that did not
>>>> give demographic information (i.e. country of origin or education,
>>>> etc) I think the results will change drastically. By all
>>>> statistical logic, this should _not_ be the case, and points to a
>>>> serious problem.>>>> 
>>>> In particular, this drops the results by a huge amount — only 1,200
>>>> or so remain. However, the remaining results tend to make a lot
>>>> more sense. For example — of the “no demographics” group, there are
>>>> 713 users who claim to develop with notepad++ but all of these say
>>>> they develop on mac and linux, and none on windows — which is
>>>> impossible, as notepad++ is a windows program. Further if you drop
>>>> the “no demographics” group, then you find that almost everyone
>>>> uses at least ghc 8.0.2, while in the “no demographics” group,  a
>>>> stunning number of people claim to be on 7.8.3. Even more
>>>> bizarrely, people claim to be using the 7.8 series while only
>>>> having used Haskell for less than one year. And people claim to
>>>> have used haskell for “one week to one month” and also to be
>>>> advanced and expert users!>>>> 
>>>> The differences continue and defy all probability. Of the “no
>>>> demographics” group, almost everyone dislikes the new release
>>>> schedule. Of the “demographics” group there are answers that like
>>>> it, were not aware of it, or are indifferent, but almost nobody
>>>> dislikes it. There is naturally a difference in proportions of
>>>> cabal/stack and hackage/stackage responses as well.>>>> 
>>>> There are a lot of other things I could point to as well. But,
>>>> bluntly put, I think that some disaffected party or parties wrote a
>>>> crude script and submitted over 3,000 fake responses. Luckily for
>>>> us, they were not very smart, and made some obvious errors, so in
>>>> this case we can weed out the bad responses (although, sadly,
>>>> losing at least a few real ones as well).>>>> 
>>>> However, assuming  this party isn’t entirely stupid, it doesn’t
>>>> bode well for future surveys as they may get at least slightly less
>>>> dumb in the future if they decide to keep it up :-/>>>> 
>>>> —Gershom
>>>> 
>>>> 
>>>> 
>>>> On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com)
>>>> wrote:>>>>> 
>>>>> 
>>>>> This is interesting, but I’m thoroughly confused. Over 2500 people
>>>>> said they took last year’s survey, but it only had roughly 1,300
>>>>> respondants?>>>>> 
>>>>> 
>>>>> On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak <taylor at fausak.me>
>>>>> wrote:
>>>>>> Hello! It took a little longer than I expected, but I am nearly
>>>>>> ready to announce the 2018 state of Haskell survey results. Some
>>>>>> community members have expressed interest in seeing the
>>>>>> announcement post before it's published. If you are one of those
>>>>>> people, you can see the results here:
>>>>>> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown
>>>>>>
>>>>>>  If you would like to suggest changes to the announcement post,
>>>>>>  please respond to this email, send me an email directly, or
>>>>>>  reply to this pull request on GitHub:
>>>>>>  https://github.com/tfausak/tfausak.github.io/pull/148
>>>>>>
>>>>>>  I plan on publishing the results tomorrow. Once the results are
>>>>>>  published, the post is by no means set in stone. I will happily
>>>>>>  accept suggestions from anyone at any time.
>>>>>>
>>>>>>  Thank you!
>>>>>>  _______________________________________________
>>>>>>  Haskell-community mailing list Haskell-community at haskell.org
>>>>>>  http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>>>> 
>>>> _______________________________________________
>>>> Haskell-community mailing list
>>>> Haskell-community at haskell.org
>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>> _______________________________________________
>>> Haskell-community mailing list
>>> Haskell-community at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
>> _________________________________________________
>> Haskell-community mailing list
>> Haskell-community at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-community/attachments/20181118/c18b2775/attachment-0001.html>