From taylor at fausak.me Thu Nov 1 02:49:47 2018 From: taylor at fausak.me (Taylor Fausak) Date: Wed, 31 Oct 2018 22:49:47 -0400 Subject: [Haskell-community] 2018 state of Haskell survey In-Reply-To: <0EDA54EC-7F0C-4826-8E70-A01899CAEC0F@fausak.me> References: <1539646963.657263.1543110624.3024DAB9@webmail.messagingengine.com> <98189428-DFDC-4D3B-B605-A128C83E0658@fausak.me> <20181026093149.GB1508@colony6.localdomain> <123F6C33-097F-4B64-B764-6FF635AFC42C@fausak.me> <1540752136.1144019.1557451144.55BDBE50@webmail.messagingengine.com> <20181028200616.3nzfcu5xdtotcwtt@x60s.casa> <0EDA54EC-7F0C-4826-8E70-A01899CAEC0F@fausak.me> Message-ID: <13C0922C-BDA7-4C9A-BDCA-11BE0C4143AA@fausak.me> I received confirmation from Airtable that they do not support arbitrary markup in forms. So I put in separator questions between each of section. At this point the survey is ready to publish. I recognize that there are many more questions that could be asked, but they’ll have to wait until next year. Thank you all for your feedback! I look forward to sharing the results with you in a couple weeks. In the meantime, if there’s anything I can do for you, please let me know. > On Oct 29, 2018, at 7:38 PM, Taylor Fausak wrote: > > Thanks for the feedback! > > - I would like to separate the survey into sections, but Airtable does not provide that functionality. I have sent a message to their support asking if I’m just missing it. Worst case scenario I can put some bogus questions in to act as dividers. I’ve put an example of such a divider question at the top of the survey. > > - The “Add an option” questions allow you to select multiple answers rather than choosing a single one. I’ve updated the questions to make that clearer by adding this help text: “Select all that apply." > > - I’ve added a followup question to the one about GHC’s new release schedule: "Why do you feel the way that you do about the new GHC release schedule?” I’m open to better wording there. > > - I have added follow up questions of the form “What would you change about X?” where X is the language, compiler, build tool, or package repository. Hopefully that will provide meaningful guidance about how to improve those things without overwhelming the user with questions. > > - For information about using Haskell at work, I think that is covered by existing questions. Last year’s survey asked if people used Haskell at work, and this year’s added some followup questions to that. Company size is covered by the demographic questions at the end. The only missing piece is asking about the size of the team of Haskell programmers. Is that worth asking about separately? > > - I have removed “Official” from the title of the survey. > > - I changed the Haskell Prime question to ask about importance rather than interest: “How important do you feel it would be to have a new version of the Haskell standard?” It uses the answer scale from here: https://help.surveymonkey.com/articles/en_US/kb/Likert-Scales > > - I split academic and commercial conferences in the question about interacting with the Haskell community. > > - For the question about which type of Haskell software is developed at the respondents company, would it suffice to ask if the software is used internally by other employees and/or externally by customers? Another question already covers the type of software (web, CLI, GUI, …). > > - I like the idea of drilling down into performance bottlenecks. How do you feel about phrasing it like this: “Which performance bottlenecks does your Haskell software typically hit?” With answer choices: CPU, RAM, disk, network, other, none.(I’m not sure what you mean by “bound by serialization.” Can you expand on that?) > > - I think the way that the software runs is covered by another question about the type of software (web, CLI, GUI, …). Is it worth it to have a separate question? > > I hope that addresses all the feedback so far. If not, please let me know! Thanks again! > >> On Oct 29, 2018, at 1:14 PM, Gershom B > wrote: >> >> HI Taylor. >> >> A few thoughts. First, even with joint sponsorship, I don't think >> saying "Official" in the name of the survey is a good idea. Everything >> is "official" from whatever group supports it, but that seems besides >> the point. I think that the intended meaning here is a bit slippery >> since it can be interpreted as "approved by some body" but is often >> used to mean "authoritative" and as we've discussed, you can't really >> be authoritative with things like this, just "better". Ok, that said, >> on to some other points: >> >> "Are you interested in a new version of the Haskell standard?" >> >> Interested is a very vague thing to ask. I'd want something more >> specific like "how important do you feel it would be to have a new >> version..." >> >> On "Where do you interact with the Haskell community?" I think that we >> should distinguish between "conferences (academic)" and "conferences >> (commercial)" because ICFP and HaskellX, for example, are very >> different sorts of things. >> >> I'd also like a question, as I mentioned earlier, like "What sort of >> Haskell software is developed at your company" with options for >> "in-house" "binaries deployed to customers" and "webapps used by >> customers" among maybe other options. Also perhaps "is the software >> you work on A) bound by memory B) bound by processor utilization C) >> bound by wire/disk speed D) bound by serialization E) not running >> against any performance limits at this time" and additionally is the >> software intended A) for continuous (server) operation or B) batched >> operation or C) interactive user-driven operation. >> >> Cheers, >> Gershom >> On Sun, Oct 28, 2018 at 4:06 PM Francesco Ariis > wrote: >>> >>> Hello Taylor, >>> >>> On Sun, Oct 28, 2018 at 02:42:16PM -0400, Taylor Fausak wrote: >>>> Please > take a look at the survey to make sure that you're happy >>>> with it! Let me know if there are any questions that you would like >>>> to be added, removed, or changed. You can view the survey here: >>>> https://airtable.com/shr8G4RBPD9T6tnDf >>>> You can deliver feedback to me either in this thread or on GitHub: >>>> https://github.com/haskellweekly/haskellweekly.github.io/issues/206 >>> >>> Suggestions: >>> - state under which specific one of the "permissive license"s the >>> results will be available; >>> - if it not mission critical, axe the last question. >>> >>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Thu Nov 1 12:00:03 2018 From: taylor at fausak.me (Taylor Fausak) Date: Thu, 1 Nov 2018 08:00:03 -0400 Subject: [Haskell-community] 2018 state of Haskell survey In-Reply-To: <13C0922C-BDA7-4C9A-BDCA-11BE0C4143AA@fausak.me> References: <1539646963.657263.1543110624.3024DAB9@webmail.messagingengine.com> <98189428-DFDC-4D3B-B605-A128C83E0658@fausak.me> <20181026093149.GB1508@colony6.localdomain> <123F6C33-097F-4B64-B764-6FF635AFC42C@fausak.me> <1540752136.1144019.1557451144.55BDBE50@webmail.messagingengine.com> <20181028200616.3nzfcu5xdtotcwtt@x60s.casa> <0EDA54EC-7F0C-4826-8E70-A01899CAEC0F@fausak.me> <13C0922C-BDA7-4C9A-BDCA-11BE0C4143AA@fausak.me> Message-ID: The survey is now open! You can read the announcement post here: http://taylor.fausak.me/2018/11/01/2018-state-of-haskell-survey/ You can go directly to the survey here: https://bit.ly/haskell2018 Or, if you don’t like link shorteners, you can go here: https://airtable.com/shr8G4RBPD9T6tnDf > On Oct 31, 2018, at 10:49 PM, Taylor Fausak wrote: > > I received confirmation from Airtable that they do not support arbitrary markup in forms. So I put in separator questions between each of section. > > At this point the survey is ready to publish. I recognize that there are many more questions that could be asked, but they’ll have to wait until next year. Thank you all for your feedback! I look forward to sharing the results with you in a couple weeks. In the meantime, if there’s anything I can do for you, please let me know. > >> On Oct 29, 2018, at 7:38 PM, Taylor Fausak > wrote: >> >> Thanks for the feedback! >> >> - I would like to separate the survey into sections, but Airtable does not provide that functionality. I have sent a message to their support asking if I’m just missing it. Worst case scenario I can put some bogus questions in to act as dividers. I’ve put an example of such a divider question at the top of the survey. >> >> - The “Add an option” questions allow you to select multiple answers rather than choosing a single one. I’ve updated the questions to make that clearer by adding this help text: “Select all that apply." >> >> - I’ve added a followup question to the one about GHC’s new release schedule: "Why do you feel the way that you do about the new GHC release schedule?” I’m open to better wording there. >> >> - I have added follow up questions of the form “What would you change about X?” where X is the language, compiler, build tool, or package repository. Hopefully that will provide meaningful guidance about how to improve those things without overwhelming the user with questions. >> >> - For information about using Haskell at work, I think that is covered by existing questions. Last year’s survey asked if people used Haskell at work, and this year’s added some followup questions to that. Company size is covered by the demographic questions at the end. The only missing piece is asking about the size of the team of Haskell programmers. Is that worth asking about separately? >> >> - I have removed “Official” from the title of the survey. >> >> - I changed the Haskell Prime question to ask about importance rather than interest: “How important do you feel it would be to have a new version of the Haskell standard?” It uses the answer scale from here: https://help.surveymonkey.com/articles/en_US/kb/Likert-Scales >> >> - I split academic and commercial conferences in the question about interacting with the Haskell community. >> >> - For the question about which type of Haskell software is developed at the respondents company, would it suffice to ask if the software is used internally by other employees and/or externally by customers? Another question already covers the type of software (web, CLI, GUI, …). >> >> - I like the idea of drilling down into performance bottlenecks. How do you feel about phrasing it like this: “Which performance bottlenecks does your Haskell software typically hit?” With answer choices: CPU, RAM, disk, network, other, none.(I’m not sure what you mean by “bound by serialization.” Can you expand on that?) >> >> - I think the way that the software runs is covered by another question about the type of software (web, CLI, GUI, …). Is it worth it to have a separate question? >> >> I hope that addresses all the feedback so far. If not, please let me know! Thanks again! >> >>> On Oct 29, 2018, at 1:14 PM, Gershom B > wrote: >>> >>> HI Taylor. >>> >>> A few thoughts. First, even with joint sponsorship, I don't think >>> saying "Official" in the name of the survey is a good idea. Everything >>> is "official" from whatever group supports it, but that seems besides >>> the point. I think that the intended meaning here is a bit slippery >>> since it can be interpreted as "approved by some body" but is often >>> used to mean "authoritative" and as we've discussed, you can't really >>> be authoritative with things like this, just "better". Ok, that said, >>> on to some other points: >>> >>> "Are you interested in a new version of the Haskell standard?" >>> >>> Interested is a very vague thing to ask. I'd want something more >>> specific like "how important do you feel it would be to have a new >>> version..." >>> >>> On "Where do you interact with the Haskell community?" I think that we >>> should distinguish between "conferences (academic)" and "conferences >>> (commercial)" because ICFP and HaskellX, for example, are very >>> different sorts of things. >>> >>> I'd also like a question, as I mentioned earlier, like "What sort of >>> Haskell software is developed at your company" with options for >>> "in-house" "binaries deployed to customers" and "webapps used by >>> customers" among maybe other options. Also perhaps "is the software >>> you work on A) bound by memory B) bound by processor utilization C) >>> bound by wire/disk speed D) bound by serialization E) not running >>> against any performance limits at this time" and additionally is the >>> software intended A) for continuous (server) operation or B) batched >>> operation or C) interactive user-driven operation. >>> >>> Cheers, >>> Gershom >>> On Sun, Oct 28, 2018 at 4:06 PM Francesco Ariis > wrote: >>>> >>>> Hello Taylor, >>>> >>>> On Sun, Oct 28, 2018 at 02:42:16PM -0400, Taylor Fausak wrote: >>>>> Please > take a look at the survey to make sure that you're happy >>>>> with it! Let me know if there are any questions that you would like >>>>> to be added, removed, or changed. You can view the survey here: >>>>> https://airtable.com/shr8G4RBPD9T6tnDf >>>>> You can deliver feedback to me either in this thread or on GitHub: >>>>> https://github.com/haskellweekly/haskellweekly.github.io/issues/206 >>>> >>>> Suggestions: >>>> - state under which specific one of the "permissive license"s the >>>> results will be available; >>>> - if it not mission critical, axe the last question. >>>> >>>> _______________________________________________ >>>> Haskell-community mailing list >>>> Haskell-community at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Sun Nov 18 01:00:35 2018 From: cdsmith at gmail.com (Chris Smith) Date: Sat, 17 Nov 2018 20:00:35 -0500 Subject: [Haskell-community] Creating a new @haskell.org mailing list? In-Reply-To: References: Message-ID: Any news on this? Would love to help any way I can, but I am not sure what to do next. Thanks, Chris On Wed, Oct 24, 2018 at 3:12 PM Gershom B wrote: > Sounds good. Ccing Sandy, who has volunteered to start helping with > mail stuff. Sandy -- do you need any further details in setting this > up, or do you think it should be straightforward? > > -g > On Wed, Oct 24, 2018 at 10:18 AM Chris Smith wrote: > > > > Good point, Simon. education@ sounds like a good choice, with the > understanding that we mean education for the general population, not > classes in type theory or category theory! > > > > Is this a possibility? Anything else I can do to move this forward? > > > > On Mon, Oct 22, 2018 at 11:32 AM Simon Peyton Jones < > simonpj at microsoft.com> wrote: > >> > >> Good idea. “k12” is rather USA specific. What about > education at haskell.org? > >> > >> > >> > >> Simon > >> > >> > >> > >> From: Haskell-community On > Behalf Of Chris Smith > >> Sent: 22 October 2018 15:32 > >> To: Haskell-community > >> Subject: [Haskell-community] Creating a new @haskell.org mailing list? > >> > >> > >> > >> Hey, > >> > >> > >> > >> Is there a process to request a new mailing list on the haskell.org > domain? > >> > >> > >> > >> Here's my use case. About 25 Haskell programmers met at ICFP to > discuss uses of Haskell in K-12 education (for non-US readers, that means > before university). I'm also in touch with another half-dozen people who > either have done, or are doing, something pre-university with Haskell, but > could not be at ICFP. The main result of our conversation was that we > wanted a common place to discuss, report on our experiences, look for > productive collaborations and common threads, etc. There are already a few > project-specific places, e.g. the codeworld-discuss mailing list for my own > project, but we were explicitly looking for something general-purpose and > universal. It would be great if this could be, say, "k12 at haskell.org" or > something like that. > >> > >> > >> > >> I'm pretty open in terms of how we'd administer the list. I'm willing > to do the work of handling obvious spam bots and things like that. If > there's a feeling we'd need something more than that, then let's have that > discussion. We explicitly don't want a strict topicality enforcement, > though. For example, several people who attended the dinner at ICFP were > also interested in functional programming for non-majors at the university > level, or were using Elm and other Haskell-like languages - even a few > people from the Racket community. I'd hope to rely on the name of the > mailing list to keep things a bit focused, but not really police it at all. > >> > >> > >> > >> Thoughts? > >> > >> > >> > >> Thanks, > >> > >> Chris Smith > > > > _______________________________________________ > > Haskell-community mailing list > > Haskell-community at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 02:56:37 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sat, 17 Nov 2018 21:56:37 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results Message-ID: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Hello! It took a little longer than I expected, but I am nearly ready to announce the 2018 state of Haskell survey results. Some community members have expressed interest in seeing the announcement post before it's published. If you are one of those people, you can see the results here: https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown If you would like to suggest changes to the announcement post, please respond to this email, send me an email directly, or reply to this pull request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 I plan on publishing the results tomorrow. Once the results are published, the post is by no means set in stone. I will happily accept suggestions from anyone at any time. Thank you! From gershomb at gmail.com Sun Nov 18 06:10:31 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 01:10:31 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: This is interesting, but I’m thoroughly confused. Over 2500 people said they took last year’s survey, but it only had roughly 1,300 respondants? On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > Hello! It took a little longer than I expected, but I am nearly ready to > announce the 2018 state of Haskell survey results. Some community members > have expressed interest in seeing the announcement post before it's > published. If you are one of those people, you can see the results here: > https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > If you would like to suggest changes to the announcement post, please > respond to this email, send me an email directly, or reply to this pull > request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 > > I plan on publishing the results tomorrow. Once the results are published, > the post is by no means set in stone. I will happily accept suggestions > from anyone at any time. > > Thank you! > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 06:11:05 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 01:11:05 -0500 Subject: [Haskell-community] Creating a new @haskell.org mailing list? In-Reply-To: References: Message-ID: Sorry for the delay - sysadmim turnover. I’ll handle it tomorrow. -g On Sat, Nov 17, 2018 at 8:00 PM Chris Smith wrote: > Any news on this? Would love to help any way I can, but I am not sure > what to do next. > > Thanks, > Chris > > On Wed, Oct 24, 2018 at 3:12 PM Gershom B wrote: > >> Sounds good. Ccing Sandy, who has volunteered to start helping with >> mail stuff. Sandy -- do you need any further details in setting this >> up, or do you think it should be straightforward? >> >> -g >> On Wed, Oct 24, 2018 at 10:18 AM Chris Smith wrote: >> > >> > Good point, Simon. education@ sounds like a good choice, with the >> understanding that we mean education for the general population, not >> classes in type theory or category theory! >> > >> > Is this a possibility? Anything else I can do to move this forward? >> > >> > On Mon, Oct 22, 2018 at 11:32 AM Simon Peyton Jones < >> simonpj at microsoft.com> wrote: >> >> >> >> Good idea. “k12” is rather USA specific. What about >> education at haskell.org? >> >> >> >> >> >> >> >> Simon >> >> >> >> >> >> >> >> From: Haskell-community On >> Behalf Of Chris Smith >> >> Sent: 22 October 2018 15:32 >> >> To: Haskell-community >> >> Subject: [Haskell-community] Creating a new @haskell.org mailing list? >> >> >> >> >> >> >> >> Hey, >> >> >> >> >> >> >> >> Is there a process to request a new mailing list on the haskell.org >> domain? >> >> >> >> >> >> >> >> Here's my use case. About 25 Haskell programmers met at ICFP to >> discuss uses of Haskell in K-12 education (for non-US readers, that means >> before university). I'm also in touch with another half-dozen people who >> either have done, or are doing, something pre-university with Haskell, but >> could not be at ICFP. The main result of our conversation was that we >> wanted a common place to discuss, report on our experiences, look for >> productive collaborations and common threads, etc. There are already a few >> project-specific places, e.g. the codeworld-discuss mailing list for my own >> project, but we were explicitly looking for something general-purpose and >> universal. It would be great if this could be, say, "k12 at haskell.org" >> or something like that. >> >> >> >> >> >> >> >> I'm pretty open in terms of how we'd administer the list. I'm willing >> to do the work of handling obvious spam bots and things like that. If >> there's a feeling we'd need something more than that, then let's have that >> discussion. We explicitly don't want a strict topicality enforcement, >> though. For example, several people who attended the dinner at ICFP were >> also interested in functional programming for non-majors at the university >> level, or were using Elm and other Haskell-like languages - even a few >> people from the Racket community. I'd hope to rely on the name of the >> mailing list to keep things a bit focused, but not really police it at all. >> >> >> >> >> >> >> >> Thoughts? >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Chris Smith >> > >> > _______________________________________________ >> > Haskell-community mailing list >> > Haskell-community at haskell.org >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Sun Nov 18 06:17:27 2018 From: cdsmith at gmail.com (Chris Smith) Date: Sun, 18 Nov 2018 01:17:27 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: This isn't TOO surprising to me. There's been a lot of confusion lately about different surveys by several different parties. It seems likely that several hundred people took some other survey, and thought it was last year's version of this one. Also, people have extremely bad memories of what they did last year. Some hundreds more probably remember seeing last year's survey and being interested, but they've forgotten that they never took it. On Sun, Nov 18, 2018 at 1:10 AM Gershom B wrote: > This is interesting, but I’m thoroughly confused. Over 2500 people said > they took last year’s survey, but it only had roughly 1,300 respondants? > > > On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > >> Hello! It took a little longer than I expected, but I am nearly ready to >> announce the 2018 state of Haskell survey results. Some community members >> have expressed interest in seeing the announcement post before it's >> published. If you are one of those people, you can see the results here: >> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown >> >> If you would like to suggest changes to the announcement post, please >> respond to this email, send me an email directly, or reply to this pull >> request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 >> >> I plan on publishing the results tomorrow. Once the results are >> published, the post is by no means set in stone. I will happily accept >> suggestions from anyone at any time. >> >> Thank you! >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 07:17:34 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 02:17:34 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: I also noticed a number of other bizarre statistical anomolies when looking at the full results. I know this is a bit much to ask — but if you could rerun the statistics filtering out people that did not give demographic information (i.e. country of origin or education, etc) I think the results will change drastically. By all statistical logic, this should _not_ be the case, and points to a serious problem. In particular, this drops the results by a huge amount — only 1,200 or so remain. However, the remaining results tend to make a lot more sense. For example — of the “no demographics” group, there are 713 users who claim to develop with notepad++ but all of these say they develop on mac and linux, and none on windows — which is impossible, as notepad++ is a windows program. Further if you drop the “no demographics” group, then you find that almost everyone uses at least ghc 8.0.2, while in the “no demographics” group, a stunning number of people claim to be on 7.8.3. Even more bizarrely, people claim to be using the 7.8 series while only having used Haskell for less than one year. And people claim to have used haskell for “one week to one month” and also to be advanced and expert users! The differences continue and defy all probability. Of the “no demographics” group, almost everyone dislikes the new release schedule. Of the “demographics” group there are answers that like it, were not aware of it, or are indifferent, but almost nobody dislikes it. There is naturally a difference in proportions of cabal/stack and hackage/stackage responses as well. There are a lot of other things I could point to as well. But, bluntly put, I think that some disaffected party or parties wrote a crude script and submitted over 3,000 fake responses. Luckily for us, they were not very smart, and made some obvious errors, so in this case we can weed out the bad responses (although, sadly, losing at least a few real ones as well). However, assuming this party isn’t entirely stupid, it doesn’t bode well for future surveys as they may get at least slightly less dumb in the future if they decide to keep it up :-/ —Gershom On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: This is interesting, but I’m thoroughly confused. Over 2500 people said they took last year’s survey, but it only had roughly 1,300 respondants? On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > Hello! It took a little longer than I expected, but I am nearly ready to > announce the 2018 state of Haskell survey results. Some community members > have expressed interest in seeing the announcement post before it's > published. If you are one of those people, you can see the results here: > https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > If you would like to suggest changes to the announcement post, please > respond to this email, send me an email directly, or reply to this pull > request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 > > I plan on publishing the results tomorrow. Once the results are published, > the post is by no means set in stone. I will happily accept suggestions > from anyone at any time. > > Thank you! > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 07:27:33 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 02:27:33 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: Finally, if anyone doubts some sort of scripted attack on the survey, is it really possible that over 200 people each want NPlusKPatterns, DatatypeContexts, or JavascriptFFI enabled by default!? Even a bot should make _some_ sense! —gershom On November 18, 2018 at 2:17:34 AM, Gershom B (gershomb at gmail.com) wrote: I also noticed a number of other bizarre statistical anomolies when looking at the full results. I know this is a bit much to ask — but if you could rerun the statistics filtering out people that did not give demographic information (i.e. country of origin or education, etc) I think the results will change drastically. By all statistical logic, this should _not_ be the case, and points to a serious problem. In particular, this drops the results by a huge amount — only 1,200 or so remain. However, the remaining results tend to make a lot more sense. For example — of the “no demographics” group, there are 713 users who claim to develop with notepad++ but all of these say they develop on mac and linux, and none on windows — which is impossible, as notepad++ is a windows program. Further if you drop the “no demographics” group, then you find that almost everyone uses at least ghc 8.0.2, while in the “no demographics” group, a stunning number of people claim to be on 7.8.3. Even more bizarrely, people claim to be using the 7.8 series while only having used Haskell for less than one year. And people claim to have used haskell for “one week to one month” and also to be advanced and expert users! The differences continue and defy all probability. Of the “no demographics” group, almost everyone dislikes the new release schedule. Of the “demographics” group there are answers that like it, were not aware of it, or are indifferent, but almost nobody dislikes it. There is naturally a difference in proportions of cabal/stack and hackage/stackage responses as well. There are a lot of other things I could point to as well. But, bluntly put, I think that some disaffected party or parties wrote a crude script and submitted over 3,000 fake responses. Luckily for us, they were not very smart, and made some obvious errors, so in this case we can weed out the bad responses (although, sadly, losing at least a few real ones as well). However, assuming this party isn’t entirely stupid, it doesn’t bode well for future surveys as they may get at least slightly less dumb in the future if they decide to keep it up :-/ —Gershom On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: This is interesting, but I’m thoroughly confused. Over 2500 people said they took last year’s survey, but it only had roughly 1,300 respondants? On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > Hello! It took a little longer than I expected, but I am nearly ready to > announce the 2018 state of Haskell survey results. Some community members > have expressed interest in seeing the announcement post before it's > published. If you are one of those people, you can see the results here: > https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > If you would like to suggest changes to the announcement post, please > respond to this email, send me an email directly, or reply to this pull > request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 > > I plan on publishing the results tomorrow. Once the results are published, > the post is by no means set in stone. I will happily accept suggestions > from anyone at any time. > > Thank you! > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marlowsd at gmail.com Sun Nov 18 07:30:54 2018 From: marlowsd at gmail.com (Simon Marlow) Date: Sun, 18 Nov 2018 07:30:54 +0000 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: Good spot Gershom. Maybe it would be revealing to look at the times that responses were received for the no-demographics group? On Sun, 18 Nov 2018, 07:17 Gershom B I also noticed a number of other bizarre statistical anomolies when > looking at the full results. I know this is a bit much to ask — but if you > could rerun the statistics filtering out people that did not give > demographic information (i.e. country of origin or education, etc) I think > the results will change drastically. By all statistical logic, this should > _not_ be the case, and points to a serious problem. > > In particular, this drops the results by a huge amount — only 1,200 or so > remain. However, the remaining results tend to make a lot more sense. For > example — of the “no demographics” group, there are 713 users who claim to > develop with notepad++ but all of these say they develop on mac and linux, > and none on windows — which is impossible, as notepad++ is a windows > program. Further if you drop the “no demographics” group, then you find > that almost everyone uses at least ghc 8.0.2, while in the “no > demographics” group, a stunning number of people claim to be on 7.8.3. > Even more bizarrely, people claim to be using the 7.8 series while only > having used Haskell for less than one year. And people claim to have used > haskell for “one week to one month” and also to be advanced and expert > users! > > The differences continue and defy all probability. Of the “no > demographics” group, almost everyone dislikes the new release schedule. Of > the “demographics” group there are answers that like it, were not aware of > it, or are indifferent, but almost nobody dislikes it. There is naturally a > difference in proportions of cabal/stack and hackage/stackage responses as > well. > > There are a lot of other things I could point to as well. But, bluntly > put, I think that some disaffected party or parties wrote a crude script > and submitted over 3,000 fake responses. Luckily for us, they were not very > smart, and made some obvious errors, so in this case we can weed out the > bad responses (although, sadly, losing at least a few real ones as well). > > However, assuming this party isn’t entirely stupid, it doesn’t bode well > for future surveys as they may get at least slightly less dumb in the > future if they decide to keep it up :-/ > > —Gershom > > > On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: > > This is interesting, but I’m thoroughly confused. Over 2500 people said > they took last year’s survey, but it only had roughly 1,300 respondants? > > > On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > >> Hello! It took a little longer than I expected, but I am nearly ready to >> announce the 2018 state of Haskell survey results. Some community members >> have expressed interest in seeing the announcement post before it's >> published. If you are one of those people, you can see the results here: >> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown >> >> If you would like to suggest changes to the announcement post, please >> respond to this email, send me an email directly, or reply to this pull >> request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 >> >> I plan on publishing the results tomorrow. Once the results are >> published, the post is by no means set in stone. I will happily accept >> suggestions from anyone at any time. >> >> Thank you! >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Sun Nov 18 08:40:52 2018 From: cdsmith at gmail.com (Chris Smith) Date: Sun, 18 Nov 2018 03:40:52 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: Sadly, it looks like a Cabal/Stack thing. Of the responses with a country provided, 618 of 1226 claim to use Cabal, and 948 of 1226 claim to use Stack. Of the responses with no country, only 35 of 3868 claim to use Cabal, while 3781 of the 3868 claim to use Stack. Assuming independence, you'd expect that last number to be about 50, meaning there are probably around 3700 fake responses generated just to answer "Stack". To partially answer Simon's question, the flood of no-demographics responses started on November 2, around the 750-response point, and continued unabated through the close of the survey. And, indeed, looking at just the first 750 responses gives similar distributions to what we get by ignoring the no-demographic responses. For example, of the first 750 responses, 359 claim to use Cabal, and 568 claim to use Stack. On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow wrote: > Good spot Gershom. Maybe it would be revealing to look at the times that > responses were received for the no-demographics group? > > On Sun, 18 Nov 2018, 07:17 Gershom B >> I also noticed a number of other bizarre statistical anomolies when >> looking at the full results. I know this is a bit much to ask — but if you >> could rerun the statistics filtering out people that did not give >> demographic information (i.e. country of origin or education, etc) I think >> the results will change drastically. By all statistical logic, this should >> _not_ be the case, and points to a serious problem. >> >> In particular, this drops the results by a huge amount — only 1,200 or so >> remain. However, the remaining results tend to make a lot more sense. For >> example — of the “no demographics” group, there are 713 users who claim to >> develop with notepad++ but all of these say they develop on mac and linux, >> and none on windows — which is impossible, as notepad++ is a windows >> program. Further if you drop the “no demographics” group, then you find >> that almost everyone uses at least ghc 8.0.2, while in the “no >> demographics” group, a stunning number of people claim to be on 7.8.3. >> Even more bizarrely, people claim to be using the 7.8 series while only >> having used Haskell for less than one year. And people claim to have used >> haskell for “one week to one month” and also to be advanced and expert >> users! >> >> The differences continue and defy all probability. Of the “no >> demographics” group, almost everyone dislikes the new release schedule. Of >> the “demographics” group there are answers that like it, were not aware of >> it, or are indifferent, but almost nobody dislikes it. There is naturally a >> difference in proportions of cabal/stack and hackage/stackage responses as >> well. >> >> There are a lot of other things I could point to as well. But, bluntly >> put, I think that some disaffected party or parties wrote a crude script >> and submitted over 3,000 fake responses. Luckily for us, they were not very >> smart, and made some obvious errors, so in this case we can weed out the >> bad responses (although, sadly, losing at least a few real ones as well). >> >> However, assuming this party isn’t entirely stupid, it doesn’t bode well >> for future surveys as they may get at least slightly less dumb in the >> future if they decide to keep it up :-/ >> >> —Gershom >> >> >> On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: >> >> This is interesting, but I’m thoroughly confused. Over 2500 people said >> they took last year’s survey, but it only had roughly 1,300 respondants? >> >> >> On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: >> >>> Hello! It took a little longer than I expected, but I am nearly ready to >>> announce the 2018 state of Haskell survey results. Some community members >>> have expressed interest in seeing the announcement post before it's >>> published. If you are one of those people, you can see the results here: >>> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown >>> >>> If you would like to suggest changes to the announcement post, please >>> respond to this email, send me an email directly, or reply to this pull >>> request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 >>> >>> I plan on publishing the results tomorrow. Once the results are >>> published, the post is by no means set in stone. I will happily accept >>> suggestions from anyone at any time. >>> >>> Thank you! >>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >>> >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 13:11:07 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sun, 18 Nov 2018 08:11:07 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> Message-ID: <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> Thanks for finding those anomalies, Gershom! I'm disappointed that someone submitted bogus responses, apparently to tip the scales of Cabal versus Stack. I intend to identify those responses and exclude them from the results. The work you've done so far will help a great deal in finding them. You said that there are about 1,200 responses with demographic information. That makes sense considering the number of submissions I got last year. Also, there are 1,185 responses that included an answer to at least one of the free-response questions. So perhaps whoever wrote the script didn't bother to put an answer for those types of questions. Unfortunately I do not have precise submission times or IP address information about submissions. Beyond what's in the CSV, the only other thing I have is (some) email addresses. Fortunately I wrote a script to output all the charts and tables from the survey responses. Once I've identified the problematic responses, I should be able to update the script to ignore them and regenerate all the output. On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote: > Sadly, it looks like a Cabal/Stack thing. Of the responses with a > country provided, 618 of 1226 claim to use Cabal, and 948 of 1226 > claim to use Stack. Of the responses with no country, only 35 of 3868 > claim to use Cabal, while 3781 of the 3868 claim to use Stack. > Assuming independence, you'd expect that last number to be about 50, > meaning there are probably around 3700 fake responses generated just > to answer "Stack".> > To partially answer Simon's question, the flood of no-demographics > responses started on November 2, around the 750-response point, and > continued unabated through the close of the survey. And, indeed, > looking at just the first 750 responses gives similar distributions > to what we get by ignoring the no-demographic responses. For > example, of the first 750 responses, 359 claim to use Cabal, and 568 > claim to use Stack.> > On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow > wrote:>> Good spot Gershom. Maybe it would be revealing to look at the times >> that responses were received for the no-demographics group?>> >> On Sun, 18 Nov 2018, 07:17 Gershom B >> I also noticed a number of other bizarre statistical anomolies when >>> looking at the full results. I know this is a bit much to ask — but >>> if you could rerun the statistics filtering out people that did not >>> give demographic information (i.e. country of origin or education, >>> etc) I think the results will change drastically. By all statistical >>> logic, this should _not_ be the case, and points to a serious >>> problem.>>> >>> In particular, this drops the results by a huge amount — only 1,200 >>> or so remain. However, the remaining results tend to make a lot more >>> sense. For example — of the “no demographics” group, there are 713 >>> users who claim to develop with notepad++ but all of these say they >>> develop on mac and linux, and none on windows — which is impossible, >>> as notepad++ is a windows program. Further if you drop the “no >>> demographics” group, then you find that almost everyone uses at >>> least ghc 8.0.2, while in the “no demographics” group, a stunning >>> number of people claim to be on 7.8.3. Even more bizarrely, people >>> claim to be using the 7.8 series while only having used Haskell for >>> less than one year. And people claim to have used haskell for “one >>> week to one month” and also to be advanced and expert users!>>> >>> The differences continue and defy all probability. Of the “no >>> demographics” group, almost everyone dislikes the new release >>> schedule. Of the “demographics” group there are answers that like >>> it, were not aware of it, or are indifferent, but almost nobody >>> dislikes it. There is naturally a difference in proportions of >>> cabal/stack and hackage/stackage responses as well.>>> >>> There are a lot of other things I could point to as well. But, >>> bluntly put, I think that some disaffected party or parties wrote a >>> crude script and submitted over 3,000 fake responses. Luckily for >>> us, they were not very smart, and made some obvious errors, so in >>> this case we can weed out the bad responses (although, sadly, losing >>> at least a few real ones as well).>>> >>> However, assuming this party isn’t entirely stupid, it doesn’t bode >>> well for future surveys as they may get at least slightly less dumb >>> in the future if they decide to keep it up :-/>>> >>> —Gershom >>> >>> >>> >>> On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) >>> wrote:>>>> >>>> >>>> This is interesting, but I’m thoroughly confused. Over 2500 people >>>> said they took last year’s survey, but it only had roughly 1,300 >>>> respondants?>>>> >>>> >>>> On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak >>>> wrote: >>>>> Hello! It took a little longer than I expected, but I am nearly >>>>> ready to announce the 2018 state of Haskell survey results. Some >>>>> community members have expressed interest in seeing the >>>>> announcement post before it's published. If you are one of those >>>>> people, you can see the results here: >>>>> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown >>>>> >>>>> If you would like to suggest changes to the announcement post, >>>>> please respond to this email, send me an email directly, or reply >>>>> to this pull request on GitHub: >>>>> https://github.com/tfausak/tfausak.github.io/pull/148 >>>>> >>>>> I plan on publishing the results tomorrow. Once the results are >>>>> published, the post is by no means set in stone. I will happily >>>>> accept suggestions from anyone at any time. >>>>> >>>>> Thank you! >>>>> _______________________________________________ >>>>> Haskell-community mailing list Haskell-community at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>>> >>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > _________________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 15:58:26 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sun, 18 Nov 2018 10:58:26 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> Message-ID: <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> I have filtered out the bogus responses and re-generated all the charts and tables. You can see the updated results here: https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown Note that until I post the results on my blog, they are not published. Please don't share the preliminary results on social media! On Sun, Nov 18, 2018, at 8:11 AM, Taylor Fausak wrote: > Thanks for finding those anomalies, Gershom! I'm disappointed that > someone submitted bogus responses, apparently to tip the scales of > Cabal versus Stack. I intend to identify those responses and exclude > them from the results. The work you've done so far will help a great > deal in finding them.> > You said that there are about 1,200 responses with demographic > information. That makes sense considering the number of submissions I > got last year. Also, there are 1,185 responses that included an answer > to at least one of the free-response questions. So perhaps whoever > wrote the script didn't bother to put an answer for those types of > questions.> > Unfortunately I do not have precise submission times or IP address > information about submissions. Beyond what's in the CSV, the only > other thing I have is (some) email addresses.> > Fortunately I wrote a script to output all the charts and tables from > the survey responses. Once I've identified the problematic responses, > I should be able to update the script to ignore them and regenerate > all the output.> > > On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote: >> Sadly, it looks like a Cabal/Stack thing. Of the responses with a >> country provided, 618 of 1226 claim to use Cabal, and 948 of 1226 >> claim to use Stack. Of the responses with no country, only 35 of 3868 >> claim to use Cabal, while 3781 of the 3868 claim to use Stack. >> Assuming independence, you'd expect that last number to be about 50, >> meaning there are probably around 3700 fake responses generated just >> to answer "Stack".>> >> To partially answer Simon's question, the flood of no-demographics >> responses started on November 2, around the 750-response point, and >> continued unabated through the close of the survey. And, indeed, >> looking at just the first 750 responses gives similar distributions >> to what we get by ignoring the no-demographic responses. For >> example, of the first 750 responses, 359 claim to use Cabal, and 568 >> claim to use Stack.>> >> On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow >> wrote:>>> Good spot Gershom. Maybe it would be revealing to look at the times >>> that responses were received for the no-demographics group?>>> >>> On Sun, 18 Nov 2018, 07:17 Gershom B >>> I also noticed a number of other bizarre statistical anomolies when >>>> looking at the full results. I know this is a bit much to ask — but >>>> if you could rerun the statistics filtering out people that did not >>>> give demographic information (i.e. country of origin or education, >>>> etc) I think the results will change drastically. By all >>>> statistical logic, this should _not_ be the case, and points to a >>>> serious problem.>>>> >>>> In particular, this drops the results by a huge amount — only 1,200 >>>> or so remain. However, the remaining results tend to make a lot >>>> more sense. For example — of the “no demographics” group, there are >>>> 713 users who claim to develop with notepad++ but all of these say >>>> they develop on mac and linux, and none on windows — which is >>>> impossible, as notepad++ is a windows program. Further if you drop >>>> the “no demographics” group, then you find that almost everyone >>>> uses at least ghc 8.0.2, while in the “no demographics” group, a >>>> stunning number of people claim to be on 7.8.3. Even more >>>> bizarrely, people claim to be using the 7.8 series while only >>>> having used Haskell for less than one year. And people claim to >>>> have used haskell for “one week to one month” and also to be >>>> advanced and expert users!>>>> >>>> The differences continue and defy all probability. Of the “no >>>> demographics” group, almost everyone dislikes the new release >>>> schedule. Of the “demographics” group there are answers that like >>>> it, were not aware of it, or are indifferent, but almost nobody >>>> dislikes it. There is naturally a difference in proportions of >>>> cabal/stack and hackage/stackage responses as well.>>>> >>>> There are a lot of other things I could point to as well. But, >>>> bluntly put, I think that some disaffected party or parties wrote a >>>> crude script and submitted over 3,000 fake responses. Luckily for >>>> us, they were not very smart, and made some obvious errors, so in >>>> this case we can weed out the bad responses (although, sadly, >>>> losing at least a few real ones as well).>>>> >>>> However, assuming this party isn’t entirely stupid, it doesn’t >>>> bode well for future surveys as they may get at least slightly less >>>> dumb in the future if they decide to keep it up :-/>>>> >>>> —Gershom >>>> >>>> >>>> >>>> On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) >>>> wrote:>>>>> >>>>> >>>>> This is interesting, but I’m thoroughly confused. Over 2500 people >>>>> said they took last year’s survey, but it only had roughly 1,300 >>>>> respondants?>>>>> >>>>> >>>>> On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak >>>>> wrote: >>>>>> Hello! It took a little longer than I expected, but I am nearly >>>>>> ready to announce the 2018 state of Haskell survey results. Some >>>>>> community members have expressed interest in seeing the >>>>>> announcement post before it's published. If you are one of those >>>>>> people, you can see the results here: >>>>>> https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown >>>>>> >>>>>> If you would like to suggest changes to the announcement post, >>>>>> please respond to this email, send me an email directly, or >>>>>> reply to this pull request on GitHub: >>>>>> https://github.com/tfausak/tfausak.github.io/pull/148 >>>>>> >>>>>> I plan on publishing the results tomorrow. Once the results are >>>>>> published, the post is by no means set in stone. I will happily >>>>>> accept suggestions from anyone at any time. >>>>>> >>>>>> Thank you! >>>>>> _______________________________________________ >>>>>> Haskell-community mailing list Haskell-community at haskell.org >>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>>>> >>>> _______________________________________________ >>>> Haskell-community mailing list >>>> Haskell-community at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> _________________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 17:51:23 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 12:51:23 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> Message-ID: Hi Taylor. I think we're closer to the real results here, but I'm still pretty sure that there are a fair number of phony responses. In particular, looking at your filter function, I don't think that _all_ bogus responses said "I dislike it" with regards to the ghc release schedule. A fair number that hit all the other criteria also seem to have left it blank. I suspect this will be enough to do the trick, but can't be sure... This attempted sabotage of the survey is really frustrating and disappointing. -g On Sun, Nov 18, 2018 at 10:58 AM Taylor Fausak wrote: > > I have filtered out the bogus responses and re-generated all the charts and tables. You can see the updated results here: https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > Note that until I post the results on my blog, they are not published. Please don't share the preliminary results on social media! > > > On Sun, Nov 18, 2018, at 8:11 AM, Taylor Fausak wrote: > > Thanks for finding those anomalies, Gershom! I'm disappointed that someone submitted bogus responses, apparently to tip the scales of Cabal versus Stack. I intend to identify those responses and exclude them from the results. The work you've done so far will help a great deal in finding them. > > You said that there are about 1,200 responses with demographic information. That makes sense considering the number of submissions I got last year. Also, there are 1,185 responses that included an answer to at least one of the free-response questions. So perhaps whoever wrote the script didn't bother to put an answer for those types of questions. > > Unfortunately I do not have precise submission times or IP address information about submissions. Beyond what's in the CSV, the only other thing I have is (some) email addresses. > > Fortunately I wrote a script to output all the charts and tables from the survey responses. Once I've identified the problematic responses, I should be able to update the script to ignore them and regenerate all the output. > > > On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote: > > Sadly, it looks like a Cabal/Stack thing. Of the responses with a country provided, 618 of 1226 claim to use Cabal, and 948 of 1226 claim to use Stack. Of the responses with no country, only 35 of 3868 claim to use Cabal, while 3781 of the 3868 claim to use Stack. Assuming independence, you'd expect that last number to be about 50, meaning there are probably around 3700 fake responses generated just to answer "Stack". > > To partially answer Simon's question, the flood of no-demographics responses started on November 2, around the 750-response point, and continued unabated through the close of the survey. And, indeed, looking at just the first 750 responses gives similar distributions to what we get by ignoring the no-demographic responses. For example, of the first 750 responses, 359 claim to use Cabal, and 568 claim to use Stack. > > On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow wrote: > > Good spot Gershom. Maybe it would be revealing to look at the times that responses were received for the no-demographics group? > > On Sun, 18 Nov 2018, 07:17 Gershom B > I also noticed a number of other bizarre statistical anomolies when looking at the full results. I know this is a bit much to ask — but if you could rerun the statistics filtering out people that did not give demographic information (i.e. country of origin or education, etc) I think the results will change drastically. By all statistical logic, this should _not_ be the case, and points to a serious problem. > > In particular, this drops the results by a huge amount — only 1,200 or so remain. However, the remaining results tend to make a lot more sense. For example — of the “no demographics” group, there are 713 users who claim to develop with notepad++ but all of these say they develop on mac and linux, and none on windows — which is impossible, as notepad++ is a windows program. Further if you drop the “no demographics” group, then you find that almost everyone uses at least ghc 8.0.2, while in the “no demographics” group, a stunning number of people claim to be on 7.8.3. Even more bizarrely, people claim to be using the 7.8 series while only having used Haskell for less than one year. And people claim to have used haskell for “one week to one month” and also to be advanced and expert users! > > The differences continue and defy all probability. Of the “no demographics” group, almost everyone dislikes the new release schedule. Of the “demographics” group there are answers that like it, were not aware of it, or are indifferent, but almost nobody dislikes it. There is naturally a difference in proportions of cabal/stack and hackage/stackage responses as well. > > There are a lot of other things I could point to as well. But, bluntly put, I think that some disaffected party or parties wrote a crude script and submitted over 3,000 fake responses. Luckily for us, they were not very smart, and made some obvious errors, so in this case we can weed out the bad responses (although, sadly, losing at least a few real ones as well). > > However, assuming this party isn’t entirely stupid, it doesn’t bode well for future surveys as they may get at least slightly less dumb in the future if they decide to keep it up :-/ > > —Gershom > > > > On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: > > > > This is interesting, but I’m thoroughly confused. Over 2500 people said they took last year’s survey, but it only had roughly 1,300 respondants? > > > On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > > Hello! It took a little longer than I expected, but I am nearly ready to announce the 2018 state of Haskell survey results. Some community members have expressed interest in seeing the announcement post before it's published. If you are one of those people, you can see the results here: https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > If you would like to suggest changes to the announcement post, please respond to this email, send me an email directly, or reply to this pull request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 > > I plan on publishing the results tomorrow. Once the results are published, the post is by no means set in stone. I will happily accept suggestions from anyone at any time. > > Thank you! > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community From cdsmith at gmail.com Sun Nov 18 17:58:27 2018 From: cdsmith at gmail.com (Chris Smith) Date: Sun, 18 Nov 2018 12:58:27 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> Message-ID: If I could make a suggestion... although this is at the forefront of our minds right now, I don't think that you want the attempted hack of survey responses to be THE big news about the survey. I have no doubt it will garner lots of attention anyway, and you are certainly right to explain what happened and what your methodology was; but I think it would be better to state the legitimate results first... i.e., by saying "This year we received 1,679 [*] responses, which is quite an improvement.", and waiting until later to explain about the bogus submissions. Hopefully, then, more of the reaction will be around the data this provides, and less around ugly drama with what seems like only ONE bad actor. On Sun, Nov 18, 2018 at 10:58 AM Taylor Fausak wrote: > I have filtered out the bogus responses and re-generated all the charts > and tables. You can see the updated results here: > https://github.com/tfausak/tfausak.github.io/blob/ee29da5bd8389c19763ac2b4dbe27ff5204161f5/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > Note that until I post the results on my blog, they are not published. > Please don't share the preliminary results on social media! > > > On Sun, Nov 18, 2018, at 8:11 AM, Taylor Fausak wrote: > > Thanks for finding those anomalies, Gershom! I'm disappointed that someone > submitted bogus responses, apparently to tip the scales of Cabal versus > Stack. I intend to identify those responses and exclude them from the > results. The work you've done so far will help a great deal in finding > them. > > You said that there are about 1,200 responses with demographic > information. That makes sense considering the number of submissions I got > last year. Also, there are 1,185 responses that included an answer to at > least one of the free-response questions. So perhaps whoever wrote the > script didn't bother to put an answer for those types of questions. > > Unfortunately I do not have precise submission times or IP address > information about submissions. Beyond what's in the CSV, the only other > thing I have is (some) email addresses. > > Fortunately I wrote a script to output all the charts and tables from the > survey responses. Once I've identified the problematic responses, I should > be able to update the script to ignore them and regenerate all the output. > > > On Sun, Nov 18, 2018, at 3:40 AM, Chris Smith wrote: > > Sadly, it looks like a Cabal/Stack thing. Of the responses with a country > provided, 618 of 1226 claim to use Cabal, and 948 of 1226 claim to use > Stack. Of the responses with no country, only 35 of 3868 claim to use > Cabal, while 3781 of the 3868 claim to use Stack. Assuming independence, > you'd expect that last number to be about 50, meaning there are probably > around 3700 fake responses generated just to answer "Stack". > > To partially answer Simon's question, the flood of no-demographics > responses started on November 2, around the 750-response point, and > continued unabated through the close of the survey. And, indeed, looking > at just the first 750 responses gives similar distributions to what we get > by ignoring the no-demographic responses. For example, of the first 750 > responses, 359 claim to use Cabal, and 568 claim to use Stack. > > On Sun, Nov 18, 2018 at 2:31 AM Simon Marlow wrote: > > Good spot Gershom. Maybe it would be revealing to look at the times that > responses were received for the no-demographics group? > > On Sun, 18 Nov 2018, 07:17 Gershom B > I also noticed a number of other bizarre statistical anomolies when > looking at the full results. I know this is a bit much to ask — but if you > could rerun the statistics filtering out people that did not give > demographic information (i.e. country of origin or education, etc) I think > the results will change drastically. By all statistical logic, this should > _not_ be the case, and points to a serious problem. > > In particular, this drops the results by a huge amount — only 1,200 or so > remain. However, the remaining results tend to make a lot more sense. For > example — of the “no demographics” group, there are 713 users who claim to > develop with notepad++ but all of these say they develop on mac and linux, > and none on windows — which is impossible, as notepad++ is a windows > program. Further if you drop the “no demographics” group, then you find > that almost everyone uses at least ghc 8.0.2, while in the “no > demographics” group, a stunning number of people claim to be on 7.8.3. > Even more bizarrely, people claim to be using the 7.8 series while only > having used Haskell for less than one year. And people claim to have used > haskell for “one week to one month” and also to be advanced and expert > users! > > The differences continue and defy all probability. Of the “no > demographics” group, almost everyone dislikes the new release schedule. Of > the “demographics” group there are answers that like it, were not aware of > it, or are indifferent, but almost nobody dislikes it. There is naturally a > difference in proportions of cabal/stack and hackage/stackage responses as > well. > > There are a lot of other things I could point to as well. But, bluntly > put, I think that some disaffected party or parties wrote a crude script > and submitted over 3,000 fake responses. Luckily for us, they were not very > smart, and made some obvious errors, so in this case we can weed out the > bad responses (although, sadly, losing at least a few real ones as well). > > However, assuming this party isn’t entirely stupid, it doesn’t bode well > for future surveys as they may get at least slightly less dumb in the > future if they decide to keep it up :-/ > > —Gershom > > > > On November 18, 2018 at 1:10:31 AM, Gershom B (gershomb at gmail.com) wrote: > > > > This is interesting, but I’m thoroughly confused. Over 2500 people said > they took last year’s survey, but it only had roughly 1,300 respondants? > > > On Sat, Nov 17, 2018 at 9:56 PM Taylor Fausak wrote: > > Hello! It took a little longer than I expected, but I am nearly ready to > announce the 2018 state of Haskell survey results. Some community members > have expressed interest in seeing the announcement post before it's > published. If you are one of those people, you can see the results here: > https://github.com/tfausak/tfausak.github.io/blob/7e4937e284a3068add9e9af6b585c8d0215ff360/_posts/2018-11-16-2018-state-of-haskell-survey-results.markdown > > If you would like to suggest changes to the announcement post, please > respond to this email, send me an email directly, or reply to this pull > request on GitHub: https://github.com/tfausak/tfausak.github.io/pull/148 > > I plan on publishing the results tomorrow. Once the results are published, > the post is by no means set in stone. I will happily accept suggestions > from anyone at any time. > > Thank you! > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > *_______________________________________________* > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imantc at gmail.com Sun Nov 18 18:26:30 2018 From: imantc at gmail.com (Imants Cekusins) Date: Sun, 18 Nov 2018 19:26:30 +0100 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> Message-ID: What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. Is either library likely to be influenced by this survey? -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 19:17:43 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sun, 18 Nov 2018 14:17:43 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> Message-ID: <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: > What if the announcement mentioned a large number of potentially bogus > responses, explained the grounds for this conclusion, with a new > survey conducted early next year?> > The next survey would then need to be done differently from this one > somehow. To improve the reliability, some authentication may be > necessary.> > > Maybe Stack, Cabal questions could be grouped as separate distinct > surveys, conducted by their maintainers through own channels?> > Not sure how much value is in exact numbers of users of Stack or > Cabal. Both groups are large enough. The maintainers of both groups > are aware about usage stats.> > Is either library likely to be influenced by this survey? > _________________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at snoyman.com Sun Nov 18 19:32:29 2018 From: michael at snoyman.com (Michael Snoyman) Date: Sun, 18 Nov 2018 21:32:29 +0200 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> Message-ID: Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. > On 18 Nov 2018, at 21:17, Taylor Fausak wrote: > > Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. > > That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. > > > On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: >> What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? >> >> The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. >> >> >> Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? >> >> Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. >> >> Is either library likely to be influenced by this survey? >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 19:55:00 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sun, 18 Nov 2018 14:55:00 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> Message-ID: <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this. https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: > Just wanted to add in: good catch Gershom on identifying the problem, > and thank you Taylor for working to remove them from the report.> >> On 18 Nov 2018, at 21:17, Taylor Fausak wrote: >> >> Great catch, Gershom! There are indeed about 300 responses that tick >> all the boxes except for disliking the new GHC release schedule. The >> main thing the attacker seemed to be interested in was over- >> representing Stack and Stackage. Also, bizarrely, Java.>> >> That brings the number of bogus responses up to 3,735, which puts the >> number of legitimate responses at 1,361. For context, last year's >> survey asked far fewer questions and had 1,335 responses.>> >> >> On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: >>> What if the announcement mentioned a large number of potentially >>> bogus responses, explained the grounds for this conclusion, with a >>> new survey conducted early next year?>>> >>> The next survey would then need to be done differently from this one >>> somehow. To improve the reliability, some authentication may be >>> necessary.>>> >>> >>> Maybe Stack, Cabal questions could be grouped as separate distinct >>> surveys, conducted by their maintainers through own channels?>>> >>> Not sure how much value is in exact numbers of users of Stack or >>> Cabal. Both groups are large enough. The maintainers of both groups >>> are aware about usage stats.>>> >>> Is either library likely to be influenced by this survey? >>> _________________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 20:07:45 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 12:07:45 -0800 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> Message-ID: The language extensions section doesn’t appear to be sorted properly. Outside of that, I think that these results are looking much better and any effort to find any additional outliers is probably not worth it for the moment. Thanks for your work on this, and I appreciate you being responsive and attentive when problems with the data were pointed out. There’s certainly some interesting and helpful information to be gleaned from this data. Cheers, Gershom On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me) wrote: Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this. https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. On 18 Nov 2018, at 21:17, Taylor Fausak wrote: Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. Is either library likely to be influenced by this survey? *_______________________________________________* Haskell-community mailing list Haskell-community at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community _______________________________________________ Haskell-community mailing list Haskell-community at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community _______________________________________________ Haskell-community mailing list Haskell-community at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at fausak.me Sun Nov 18 21:31:46 2018 From: taylor at fausak.me (Taylor Fausak) Date: Sun, 18 Nov 2018 16:31:46 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> Message-ID: <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> Oops, the ordering of the answer choices is manual because some questions have a natural order while others should just be most to least popular. I've made another run through to make sure everything is sorted properly. I'll probably hit publish in the next half hour or so unless there are any objections. https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote: > The language extensions section doesn’t appear to be sorted properly. > Outside of that, I think that these results are looking much better > and any effort to find any additional outliers is probably not worth > it for the moment. Thanks for your work on this, and I appreciate you > being responsive and attentive when problems with the data were > pointed out. There’s certainly some interesting and helpful > information to be gleaned from this data.> > Cheers, > Gershom > > > > > On November 18, 2018 at 2:55:10 PM, Taylor Fausak > (taylor at fausak.me) wrote:>> >> >> >> Ok, I updated the function that checks for bad responses, re-ran the >> script, and updated the announcement along with all the assets >> (charts, tables, and CSV). Hopefully it's the last time, as I can't >> justify spending much more time on this.>> >> https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown>> >> >> On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: >>> Just wanted to add in: good catch Gershom on identifying the >>> problem, and thank you Taylor for working to remove them from the >>> report.>>> >>>> On 18 Nov 2018, at 21:17, Taylor Fausak wrote: >>>> >>>> Great catch, Gershom! There are indeed about 300 responses that >>>> tick all the boxes except for disliking the new GHC release >>>> schedule. The main thing the attacker seemed to be interested in >>>> was over-representing Stack and Stackage. Also, bizarrely, Java.>>>> >>>> That brings the number of bogus responses up to 3,735, which puts >>>> the number of legitimate responses at 1,361. For context, last >>>> year's survey asked far fewer questions and had 1,335 responses.>>>> >>>> >>>> On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: >>>>> What if the announcement mentioned a large number of potentially >>>>> bogus responses, explained the grounds for this conclusion, with a >>>>> new survey conducted early next year?>>>>> >>>>> The next survey would then need to be done differently from this >>>>> one somehow. To improve the reliability, some authentication may >>>>> be necessary.>>>>> >>>>> >>>>> Maybe Stack, Cabal questions could be grouped as separate distinct >>>>> surveys, conducted by their maintainers through own channels?>>>>> >>>>> Not sure how much value is in exact numbers of users of Stack or >>>>> Cabal. Both groups are large enough. The maintainers of both >>>>> groups are aware about usage stats.>>>>> >>>>> Is either library likely to be influenced by this survey? >>>>> _________________________________________________ >>>>> Haskell-community mailing list >>>>> Haskell-community at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>>>> >>>> _______________________________________________ >>>> Haskell-community mailing list >>>> Haskell-community at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>> >> _______________________________________________ >> Haskell-community mailing list Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gershomb at gmail.com Sun Nov 18 23:50:14 2018 From: gershomb at gmail.com (Gershom B) Date: Sun, 18 Nov 2018 18:50:14 -0500 Subject: [Haskell-community] Creating a new @haskell.org mailing list? In-Reply-To: References: Message-ID: Ok, education@ should now be created, and chris should be list admin. Feel free to reach out to me if there are any issues. Cheers, Gershom On Sat, Nov 17, 2018 at 8:00 PM Chris Smith wrote: > > Any news on this? Would love to help any way I can, but I am not sure what to do next. > > Thanks, > Chris > > On Wed, Oct 24, 2018 at 3:12 PM Gershom B wrote: >> >> Sounds good. Ccing Sandy, who has volunteered to start helping with >> mail stuff. Sandy -- do you need any further details in setting this >> up, or do you think it should be straightforward? >> >> -g >> On Wed, Oct 24, 2018 at 10:18 AM Chris Smith wrote: >> > >> > Good point, Simon. education@ sounds like a good choice, with the understanding that we mean education for the general population, not classes in type theory or category theory! >> > >> > Is this a possibility? Anything else I can do to move this forward? >> > >> > On Mon, Oct 22, 2018 at 11:32 AM Simon Peyton Jones wrote: >> >> >> >> Good idea. “k12” is rather USA specific. What about education at haskell.org? >> >> >> >> >> >> >> >> Simon >> >> >> >> >> >> >> >> From: Haskell-community On Behalf Of Chris Smith >> >> Sent: 22 October 2018 15:32 >> >> To: Haskell-community >> >> Subject: [Haskell-community] Creating a new @haskell.org mailing list? >> >> >> >> >> >> >> >> Hey, >> >> >> >> >> >> >> >> Is there a process to request a new mailing list on the haskell.org domain? >> >> >> >> >> >> >> >> Here's my use case. About 25 Haskell programmers met at ICFP to discuss uses of Haskell in K-12 education (for non-US readers, that means before university). I'm also in touch with another half-dozen people who either have done, or are doing, something pre-university with Haskell, but could not be at ICFP. The main result of our conversation was that we wanted a common place to discuss, report on our experiences, look for productive collaborations and common threads, etc. There are already a few project-specific places, e.g. the codeworld-discuss mailing list for my own project, but we were explicitly looking for something general-purpose and universal. It would be great if this could be, say, "k12 at haskell.org" or something like that. >> >> >> >> >> >> >> >> I'm pretty open in terms of how we'd administer the list. I'm willing to do the work of handling obvious spam bots and things like that. If there's a feeling we'd need something more than that, then let's have that discussion. We explicitly don't want a strict topicality enforcement, though. For example, several people who attended the dinner at ICFP were also interested in functional programming for non-majors at the university level, or were using Elm and other Haskell-like languages - even a few people from the Racket community. I'd hope to rely on the name of the mailing list to keep things a bit focused, but not really police it at all. >> >> >> >> >> >> >> >> Thoughts? >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Chris Smith >> > >> > _______________________________________________ >> > Haskell-community mailing list >> > Haskell-community at haskell.org >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community From rae at cs.brynmawr.edu Mon Nov 19 04:20:52 2018 From: rae at cs.brynmawr.edu (Richard Eisenberg) Date: Sun, 18 Nov 2018 23:20:52 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> Message-ID: <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> I have not analyzed the data myself, but I wonder how we jumped to the conclusion that the troll was trying to promote Stack. Is there statistical data that supports that conclusion? For example, just reading this thread, it sounds like the bogus responses also really don't like the new release schedule. Maybe the troll wants the old release schedule back and was just lazy about programming the tool to vary the stack/cabal question answers adequately. Given the contention around cabal vs stack, I agree that sociological concerns suggest that the troll meant to tilt those scales. But I wouldn't want a public accusation without at least some statistical analysis that independently supports that conclusion. In any case, thanks to all for putting this together! Richard > On Nov 18, 2018, at 4:31 PM, Taylor Fausak wrote: > > Oops, the ordering of the answer choices is manual because some questions have a natural order while others should just be most to least popular. I've made another run through to make sure everything is sorted properly. I'll probably hit publish in the next half hour or so unless there are any objections. > > https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown > > > On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote: >> The language extensions section doesn’t appear to be sorted properly. Outside of that, I think that these results are looking much better and any effort to find any additional outliers is probably not worth it for the moment. Thanks for your work on this, and I appreciate you being responsive and attentive when problems with the data were pointed out. There’s certainly some interesting and helpful information to be gleaned from this data. >> >> Cheers, >> Gershom >> >> >> >> >> On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me ) wrote: >> >>> >>> >>> >>> Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this. >>> >>> https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown >>> >>> >>> On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: >>>> Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. >>>> >>>>> On 18 Nov 2018, at 21:17, Taylor Fausak > wrote: >>>>> >>>>> Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. >>>>> >>>>> That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. >>>>> >>>>> >>>>> On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: >>>>>> What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? >>>>>> >>>>>> The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. >>>>>> >>>>>> >>>>>> Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? >>>>>> >>>>>> Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. >>>>>> >>>>>> Is either library likely to be influenced by this survey? >>>>>> _______________________________________________ >>>>>> Haskell-community mailing list >>>>>> Haskell-community at haskell.org >>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >>>>> >>>>> _______________________________________________ >>>>> Haskell-community mailing list >>>>> Haskell-community at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >>> >>> _______________________________________________ >>> Haskell-community mailing list >>> Haskell-community at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >>> > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdsmith at gmail.com Mon Nov 19 04:56:49 2018 From: cdsmith at gmail.com (Chris Smith) Date: Sun, 18 Nov 2018 23:56:49 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> Message-ID: > For example, just reading this thread, it sounds like the bogus responses also really don't like the new release schedule. Maybe the troll wants the old release schedule back and was just lazy about programming the tool to vary the stack/cabal question answers adequately. There is another scenario, though, which should caution against making official statements about motivation. There was a set of people who worked very hard while the survey was open to preemptively cast doubt on its motivation and goals. It may be that someone was mainly attempting to sabotage the survey results themselves, rather than taking a side in any specific dispute. Of course, had the results been published claiming that a mere 12% of Haskellers use Cabal, it would have been immediately dismissed by many people as obviously biased, which would have achieved that goal, too. I think Taylor's post handled this well, saying what we know to be true, that the attack targeted divisive issues, but without drawing unnecessary conclusions. On Sun, Nov 18, 2018 at 11:21 PM Richard Eisenberg wrote: > I have not analyzed the data myself, but I wonder how we jumped to the > conclusion that the troll was trying to promote Stack. Is there statistical > data that supports that conclusion? For example, just reading this thread, > it sounds like the bogus responses also really don't like the new release > schedule. Maybe the troll wants the old release schedule back and was just > lazy about programming the tool to vary the stack/cabal question answers > adequately. > > Given the contention around cabal vs stack, I agree that sociological > concerns suggest that the troll meant to tilt those scales. But I wouldn't > want a public accusation without at least some statistical analysis that > independently supports that conclusion. > > In any case, thanks to all for putting this together! > > Richard > > On Nov 18, 2018, at 4:31 PM, Taylor Fausak wrote: > > Oops, the ordering of the answer choices is manual because some questions > have a natural order while others should just be most to least popular. > I've made another run through to make sure everything is sorted properly. > I'll probably hit publish in the next half hour or so unless there are any > objections. > > > https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown > > > On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote: > > The language extensions section doesn’t appear to be sorted properly. > Outside of that, I think that these results are looking much better and any > effort to find any additional outliers is probably not worth it for the > moment. Thanks for your work on this, and I appreciate you being responsive > and attentive when problems with the data were pointed out. There’s > certainly some interesting and helpful information to be gleaned from this > data. > > Cheers, > Gershom > > > > > On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me) > wrote: > > > > > Ok, I updated the function that checks for bad responses, re-ran the > script, and updated the announcement along with all the assets (charts, > tables, and CSV). Hopefully it's the last time, as I can't justify spending > much more time on this. > > > https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown > > > On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: > > Just wanted to add in: good catch Gershom on identifying the problem, and > thank you Taylor for working to remove them from the report. > > On 18 Nov 2018, at 21:17, Taylor Fausak wrote: > > Great catch, Gershom! There are indeed about 300 responses that tick all > the boxes except for disliking the new GHC release schedule. The main thing > the attacker seemed to be interested in was over-representing Stack and > Stackage. Also, bizarrely, Java. > > That brings the number of bogus responses up to 3,735, which puts the > number of legitimate responses at 1,361. For context, last year's survey > asked far fewer questions and had 1,335 responses. > > > On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: > > What if the announcement mentioned a large number of potentially bogus > responses, explained the grounds for this conclusion, with a new survey > conducted early next year? > > The next survey would then need to be done differently from this one > somehow. To improve the reliability, some authentication may be necessary. > > > Maybe Stack, Cabal questions could be grouped as separate distinct > surveys, conducted by their maintainers through own channels? > > Not sure how much value is in exact numbers of users of Stack or Cabal. > Both groups are large enough. The maintainers of both groups are aware > about usage stats. > > Is either library likely to be influenced by this survey? > *_______________________________________________* > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at cs.brynmawr.edu Mon Nov 19 04:58:36 2018 From: rae at cs.brynmawr.edu (Richard Eisenberg) Date: Sun, 18 Nov 2018 23:58:36 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> Message-ID: > On Nov 18, 2018, at 11:56 PM, Chris Smith wrote: > > I think Taylor's post handled this well, saying what we know to be true, that the attack targeted divisive issues, but without drawing unnecessary conclusions. I agree 100%. I hadn't read the post before writing my email on this thread. Thanks, Taylor. Richard From fa-ml at ariis.it Mon Nov 19 05:03:23 2018 From: fa-ml at ariis.it (Francesco Ariis) Date: Mon, 19 Nov 2018 06:03:23 +0100 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> References: <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> Message-ID: <20181119050323.vp2u2wpr26hb67es@x60s.casa> Hello Richard, On Sun, Nov 18, 2018 at 11:20:52PM -0500, Richard Eisenberg wrote: > I have not analyzed the data myself, but I wonder how we jumped to the > conclusion that the troll was trying to promote Stack. Is there > statistical data that supports that conclusion? For example, just reading > this thread, it sounds like the bogus responses also really don't like > the new release schedule. Maybe the troll wants the old release schedule > back and was just lazy about programming the tool to vary the > stack/cabal question answers adequately. If you filter the results for the (impossible) "linux/mac AND notepad++" combination, you can check the pattern-of-action of the troll. Every demographic question is skipped; every "write in" answer is skipped; all the other questions are filled in with a random value, bar the "build tools" one and the "release schedule" one, both having a constant value. From gershomb at gmail.com Mon Nov 19 05:06:07 2018 From: gershomb at gmail.com (Gershom B) Date: Mon, 19 Nov 2018 00:06:07 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> Message-ID: On Sun, Nov 18, 2018 at 11:20 PM Richard Eisenberg wrote: > > I have not analyzed the data myself, but I wonder how we jumped to the conclusion that the troll was trying to promote Stack. Is there statistical data that supports that conclusion? For example, just reading this thread, it sounds like the bogus responses also really don't like the new release schedule. Maybe the troll wants the old release schedule back and was just lazy about programming the tool to vary the stack/cabal question answers adequately. Roughly 90% of the bogus responses disliked the new ghc schedule and 10% left the answer blank. As far as I know, 100% of the bogus responses said they used stack exclusively. The answers to almost every other question (except, I think, for targeted platform?) varied significantly (although according to either uniform, linear, or normal distributions for the most part). So as guesses go, this seems pretty strong. I will also say, though there's speculation about "false flags" and other silliness floating around that I personally have a very good guess as to who did this. There's one well-known troll who has these preoccupations and is known for creating serial sockpuppet accounts, and is just the right amount of obsessed to do something like this. A few of the bogus responses actually had comments, and the comments were all written in a voice that was unmistakeable as this troll as well. Occam's razor seems to apply. Finally, let me add why I don't think this was a "false flag" -- while there were enough telltale markers that the fake answers could seem to be detected, I don't think this was on purpose. There was _too much_ effort put into distributions of other choices, etc. If they had wanted the fakes to be detected they would have left much stronger evidence. Rather, from a forensic standpoint, this seems pretty clear to me that the pattern of data is of someone _trying_ to cover their tracks, but just making four or five errors which I could assemble into a pattern. If they hadn't made those errors -- likely based on bad priors about what the organic data would be that theirs would need to "mesh" into -- then I think the deception would have been much harder to detect. --Gershom > Given the contention around cabal vs stack, I agree that sociological concerns suggest that the troll meant to tilt those scales. But I wouldn't want a public accusation without at least some statistical analysis that independently supports that conclusion. > > In any case, thanks to all for putting this together! > > Richard > > On Nov 18, 2018, at 4:31 PM, Taylor Fausak wrote: > > Oops, the ordering of the answer choices is manual because some questions have a natural order while others should just be most to least popular. I've made another run through to make sure everything is sorted properly. I'll probably hit publish in the next half hour or so unless there are any objections. > > https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown > > > On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote: > > The language extensions section doesn’t appear to be sorted properly. Outside of that, I think that these results are looking much better and any effort to find any additional outliers is probably not worth it for the moment. Thanks for your work on this, and I appreciate you being responsive and attentive when problems with the data were pointed out. There’s certainly some interesting and helpful information to be gleaned from this data. > > Cheers, > Gershom > > > > > On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me) wrote: > > > > > Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this. > > https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown > > > On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: > > Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. > > On 18 Nov 2018, at 21:17, Taylor Fausak wrote: > > Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. > > That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. > > > On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: > > What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? > > The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. > > > Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? > > Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. > > Is either library likely to be influenced by this survey? > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > > _______________________________________________ > Haskell-community mailing list > Haskell-community at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community > > From rae at cs.brynmawr.edu Mon Nov 19 05:16:23 2018 From: rae at cs.brynmawr.edu (Richard Eisenberg) Date: Mon, 19 Nov 2018 00:16:23 -0500 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> <1542570900.2315274.1581087368.019A69D2@webmail.messagingengine.com> <1542576706.2385291.1581151704.622356C8@webmail.messagingengine.com> <7B085E96-149B-4886-B5EF-4C9C34FF410E@cs.brynmawr.edu> Message-ID: <926E3B20-529D-4CF4-BBFF-C25D14E32AD7@cs.brynmawr.edu> OK. Thanks for sharing some statistics. I'm now convinced as to the characterization of the attack. I'm still glad for how the public post diplomatically handled this. > I will also say, though there's speculation about "false flags" and Oof. That thought never crossed my mind. I can only imagine this is on some social media where I don't participate. Every day, I am more and more pleased with my non-presence on most social media. :) Besides, just keeping up with email is enough of a challenge. Thanks for the clarification. Richard > other silliness floating around that I personally have a very good > guess as to who did this. There's one well-known troll who has these > preoccupations and is known for creating serial sockpuppet accounts, > and is just the right amount of obsessed to do something like this. A > few of the bogus responses actually had comments, and the comments > were all written in a voice that was unmistakeable as this troll as > well. Occam's razor seems to apply. > > Finally, let me add why I don't think this was a "false flag" -- while > there were enough telltale markers that the fake answers could seem to > be detected, I don't think this was on purpose. There was _too much_ > effort put into distributions of other choices, etc. If they had > wanted the fakes to be detected they would have left much stronger > evidence. Rather, from a forensic standpoint, this seems pretty clear > to me that the pattern of data is of someone _trying_ to cover their > tracks, but just making four or five errors which I could assemble > into a pattern. If they hadn't made those errors -- likely based on > bad priors about what the organic data would be that theirs would need > to "mesh" into -- then I think the deception would have been much > harder to detect. > > --Gershom > >> Given the contention around cabal vs stack, I agree that sociological concerns suggest that the troll meant to tilt those scales. But I wouldn't want a public accusation without at least some statistical analysis that independently supports that conclusion. >> >> In any case, thanks to all for putting this together! >> >> Richard >> >> On Nov 18, 2018, at 4:31 PM, Taylor Fausak wrote: >> >> Oops, the ordering of the answer choices is manual because some questions have a natural order while others should just be most to least popular. I've made another run through to make sure everything is sorted properly. I'll probably hit publish in the next half hour or so unless there are any objections. >> >> https://github.com/tfausak/tfausak.github.io/blob/fce97d07c369856d4c05b756c492eb6229a1b5c7/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown >> >> >> On Sun, Nov 18, 2018, at 3:07 PM, Gershom B wrote: >> >> The language extensions section doesn’t appear to be sorted properly. Outside of that, I think that these results are looking much better and any effort to find any additional outliers is probably not worth it for the moment. Thanks for your work on this, and I appreciate you being responsive and attentive when problems with the data were pointed out. There’s certainly some interesting and helpful information to be gleaned from this data. >> >> Cheers, >> Gershom >> >> >> >> >> On November 18, 2018 at 2:55:10 PM, Taylor Fausak (taylor at fausak.me) wrote: >> >> >> >> >> Ok, I updated the function that checks for bad responses, re-ran the script, and updated the announcement along with all the assets (charts, tables, and CSV). Hopefully it's the last time, as I can't justify spending much more time on this. >> >> https://github.com/tfausak/tfausak.github.io/blob/6f9991758ffeed085c45dd97e4ce6a82a8b1a73f/_posts/2018-11-18-2018-state-of-haskell-survey-results.markdown >> >> >> On Sun, Nov 18, 2018, at 2:32 PM, Michael Snoyman wrote: >> >> Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. >> >> On 18 Nov 2018, at 21:17, Taylor Fausak wrote: >> >> Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. >> >> That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. >> >> >> On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: >> >> What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? >> >> The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. >> >> >> Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? >> >> Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. >> >> Is either library likely to be influenced by this survey? >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> >> >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> >> >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> >> >> _______________________________________________ >> Haskell-community mailing list >> Haskell-community at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community >> >> From simonpj at microsoft.com Mon Nov 19 10:51:34 2018 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 19 Nov 2018 10:51:34 +0000 Subject: [Haskell-community] 2018 state of Haskell survey results In-Reply-To: References: <1542509797.1396708.1580603208.6D1AEB39@webmail.messagingengine.com> <1542546667.1525337.1580851352.337BF18B@webmail.messagingengine.com> <1542556706.1565147.1580944272.24C33287@webmail.messagingengine.com> <1542568663.1608876.1581059240.65290B1B@webmail.messagingengine.com> Message-ID: Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. I'd like to add +1 to that. It's a source of astonishment, and some dismay, to me that anyone would go to so much trouble to affect a survey about Haskell. (Brexit, perhaps, but Haskell??) But many thanks to Gershom and Taylor for dealing with it so professionally. Simon From: Haskell-community On Behalf Of Michael Snoyman Sent: 18 November 2018 19:32 To: Taylor Fausak Cc: haskell-community at haskell.org Subject: Re: [Haskell-community] 2018 state of Haskell survey results Just wanted to add in: good catch Gershom on identifying the problem, and thank you Taylor for working to remove them from the report. On 18 Nov 2018, at 21:17, Taylor Fausak > wrote: Great catch, Gershom! There are indeed about 300 responses that tick all the boxes except for disliking the new GHC release schedule. The main thing the attacker seemed to be interested in was over-representing Stack and Stackage. Also, bizarrely, Java. That brings the number of bogus responses up to 3,735, which puts the number of legitimate responses at 1,361. For context, last year's survey asked far fewer questions and had 1,335 responses. On Sun, Nov 18, 2018, at 1:26 PM, Imants Cekusins wrote: What if the announcement mentioned a large number of potentially bogus responses, explained the grounds for this conclusion, with a new survey conducted early next year? The next survey would then need to be done differently from this one somehow. To improve the reliability, some authentication may be necessary. Maybe Stack, Cabal questions could be grouped as separate distinct surveys, conducted by their maintainers through own channels? Not sure how much value is in exact numbers of users of Stack or Cabal. Both groups are large enough. The maintainers of both groups are aware about usage stats. Is either library likely to be influenced by this survey? _______________________________________________ Haskell-community mailing list Haskell-community at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community _______________________________________________ Haskell-community mailing list Haskell-community at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-community -------------- next part -------------- An HTML attachment was scrubbed... URL: From naidunitin1 at gmail.com Thu Nov 8 07:20:07 2018 From: naidunitin1 at gmail.com (Nitin Naidu) Date: Thu, 08 Nov 2018 07:20:07 -0000 Subject: [Haskell-community] New curious user contribution Message-ID: Hi there! I'm Nitin. I wish to contribute in Haskell project. I am a little lost here. Any suggestions where I can start? Regards, Nitin Naidu -------------- next part -------------- An HTML attachment was scrubbed... URL: