<div dir="ltr">If I could make a suggestion... although this is at the forefront of our minds right now, I don't think that you want the attempted hack of survey responses to be THE big news about the survey. I have no doubt it will garner lots of attention anyway, and you are certainly right to explain what happened and what your methodology was; but I think it would be better to state the legitimate results first... i.e., by saying "This year we received 1,679 [*] responses, which is quite an improvement.", and waiting until later to explain about the bogus submissions. Hopefully, then, more of the reaction will be around the data this provides, and less around ugly drama with what seems like only ONE bad actor.</div><br><div class="gmail_quote"><div dir="ltr">On Sun, Nov 18, 2018 at 10:58 AM Taylor Fausak <<a href="mailto:taylor@fausak.me">taylor@fausak.me</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><div>I have filtered out the bogus responses and re-generated all the charts and tables. You can see the updated results here: <a href="https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown" target="_blank">https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown</a><br></div>
<div><br></div>
<div>Note that until I post the results on my blog, they are not published. Please don't share the preliminary results on social media! <br></div>
<div><br></div>
<div><br></div>
<div>On Sun, Nov 18, 2018, at 8:11 AM, Taylor Fausak wrote:<br></div>
<blockquote type="cite"><div>Thanks for finding those anomalies, Gershom! I'm disappointed that someone submitted bogus responses, apparently to tip the scales of Cabal versus Stack. I intend to identify those responses and exclude them from the results. The work you've done so far will help a great deal in finding them. <br></div>
<div><br></div>
<div>You said that there are about 1,200 responses with demographic information. That makes sense considering the number of submissions I got last year. Also, there are 1,185 responses that included an answer to at least one of the free-response questions. So perhaps whoever wrote the script didn't bother to put an answer for those types of questions. <br></div>
<div><br></div>
<div>Unfortunately I do not have precise submission times or IP address information about submissions. Beyond what's in the CSV, the only other thing I have is (some) email addresses.<br></div>
<div><br></div>
<div>Fortunately I wrote a script to output all the charts and tables from the survey responses. Once I've identified the problematic responses, I should be able to update the script to ignore them and regenerate all the output. <br></div>
<div><br></div>
<div><br></div>
<div>On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div>Sadly, it looks like a Cabal/Stack thing. Of the responses with a country provided, 618 of 1226 claim to use Cabal, and 948 of 1226 claim to use Stack. Of the responses with no country, only 35 of 3868 claim to use Cabal, while 3781 of the 3868 claim to use Stack. Assuming independence, you'd expect that last number to be about 50, meaning there are probably around 3700 fake responses generated just to answer "Stack".<br></div>
<div><br></div>
<div>To partially answer Simon's question, the flood of no-demographics responses started on November 2, around the 750-response point, and continued unabated through the close of the survey. And, indeed, looking at just the first 750 responses gives similar distributions to what we get by ignoring the no-demographic responses. For example, of the first 750 responses, 359 claim to use Cabal, and 568 claim to use Stack.<br></div>
</div>
<div><br></div>
<div><div dir="ltr">On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow <<a href="mailto:marlowsd@gmail.com" target="_blank">marlowsd@gmail.com</a>> wrote:<br></div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204,204,204);border-left-style:solid;border-left-width:1px;padding-left:1ex"><div>Good spot Gershom. Maybe it would be revealing to look at the times that responses were received for the no-demographics group?<br></div>
<div><br></div>
<div><div dir="ltr">On Sun, 18 Nov 2018, 07:17 Gershom B <<a href="mailto:gershomb@gmail.com" target="_blank">gershomb@gmail.com</a> wrote:<br></div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204,204,204);border-left-style:solid;border-left-width:1px;padding-left:1ex"><div><div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">I also noticed a number of other bizarre statistical anomolies when looking at the full results. I know this is a bit much to ask — but if you could rerun the statistics filtering out people that did not give demographic information (i.e. country of origin or education, etc) I think the results will change drastically. By all statistical logic, this should _not_ be the case, and points to a serious problem.<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">In particular, this drops the results by a huge amount — only 1,200 or so remain. However, the remaining results tend to make a lot more sense. For example — of the “no demographics” group, there are 713 users who claim to develop with notepad++ but all of these say they develop on mac and linux, and none on windows — which is impossible, as notepad++ is a windows program. Further if you drop the “no demographics” group, then you find that almost everyone uses at least ghc 8.0.2, while in the “no demographics” group, a stunning number of people claim to be on 7.8.3. Even more bizarrely, people claim to be using the 7.8 series while only having used Haskell for less than one year. And people claim to have used haskell for “one week to one month” and also to be advanced and expert users!<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">The differences continue and defy all probability. Of the “no demographics” group, almost everyone dislikes the new release schedule. Of the “demographics” group there are answers that like it, were not aware of it, or are indifferent, but almost nobody dislikes it. There is naturally a difference in proportions of cabal/stack and hackage/stackage responses as well.<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">There are a lot of other things I could point to as well. But, bluntly put, I think that some disaffected party or parties wrote a crude script and submitted over 3,000 fake responses. Luckily for us, they were not very smart, and made some obvious errors, so in this case we can weed out the bad responses (although, sadly, losing at least a few real ones as well).<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">However, assuming this party isn’t entirely stupid, it doesn’t bode well for future surveys as they may get at least slightly less dumb in the future if they decide to keep it up :-/<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">—Gershom<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></div>
<div><br></div>
<div><br></div>
<p>On November 18, 2018 at 1:10:31 AM, Gershom B (<a href="mailto:gershomb@gmail.com" target="_blank">gershomb@gmail.com</a>) wrote:<br></p><blockquote type="cite"><div><span></span><br></div>
<div><div><br></div>
<div><div><div><span>This is interesting, but I’m thoroughly confused.
Over 2500 people said they took last year’s survey, but it only had
roughly 1,300 respondants?</span><br></div>
</div>
<div><span></span><br></div>
<div><div><span></span><br></div>
<div><div dir="ltr"><span>On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak
<<a href="mailto:taylor@fausak.me" target="_blank">taylor@fausak.me</a>>
wrote:</span><br></div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204,204,204);border-left-style:solid;border-left-width:1px;padding-left:1ex"><span>Hello! It took a little longer than I expected, but I am nearly
ready to announce the 2018 state of Haskell survey results. Some
community members have expressed interest in seeing the
announcement post before it's published. If you are one of those
people, you can see the results here: <a href="https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown" target="_blank">https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown</a><br> <br> If you would like to suggest changes to the announcement post,
please respond to this email, send me an email directly, or reply
to this pull request on GitHub: <a href="https://github.com/tfausak/tfausak.github.io/pull/148" target="_blank">https://github.com/tfausak/tfausak.github.io/pull/148</a><br> <br> I plan on publishing the results tomorrow. Once the results are
published, the post is by no means set in stone. I will happily
accept suggestions from anyone at any time.<br> <br> Thank you!<br> _______________________________________________<br> Haskell-community mailing list<br> <a href="mailto:Haskell-community@haskell.org" target="_blank">Haskell-community@haskell.org</a><br> <a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community</a></span></blockquote></div>
</div>
</div>
</div>
<div><span></span><br></div>
</blockquote></div>
<div>_______________________________________________<br></div>
<div>Haskell-community mailing list<br></div>
<div><a href="mailto:Haskell-community@haskell.org" target="_blank">Haskell-community@haskell.org</a><br></div>
<div><a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community</a><br></div>
</blockquote></div>
<div>_______________________________________________<br></div>
<div>Haskell-community mailing list<br></div>
<div><a href="mailto:Haskell-community@haskell.org" target="_blank">Haskell-community@haskell.org</a><br></div>
<div><a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community</a><br></div>
</blockquote></div>
<div><u>_______________________________________________</u><br></div>
<div>Haskell-community mailing list<br></div>
<div><a href="mailto:Haskell-community@haskell.org" target="_blank">Haskell-community@haskell.org</a><br></div>
<div><a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community</a><br></div>
</blockquote><div><br></div>
</blockquote><div><br></div>
</div>
_______________________________________________<br>
Haskell-community mailing list<br>
<a href="mailto:Haskell-community@haskell.org" target="_blank">Haskell-community@haskell.org</a><br>
<a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community" rel="noreferrer" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community</a><br>
</blockquote></div>