[Haskell-community] 2018 state of Haskell survey

Tue Oct 16 01:33:32 UTC 2018

Hi Taylor. I like the way you pose things here: "I don't expect that
to remove selection bias, but it will let me (us, really) say: We're
doing this together for the benefit of all sides". I think that's a
better place to start from. (I personally would just say "all" and
remove the word "sides" -- there certainly are sides and alignments,
but in my ideal world, the "sides" would be episodic with regards to
particular issues and decisions, and vary with regards to them rather
than having a cross-cutting us/them over time, which can be harmful. I
think we can get there!) I see it in positive contrast to something
earlier I've been thinking about a bit, where you wrote: "My goal is
for this survey to be *the* authoritative Haskell survey and for the
community to broadly accept it's results." I do think it is good to
start with what a goal for a survey would be, and I was somewhat
surprised that the Clojure, etc. surveys I looked at didn't say
explicitly what their goal was. I don't think there's necessarily
*the* authoritative survey on many things though -- in election
polling, for example, Nate Silver made a name for himself by looking
at a _lot_ of surveys, understanding that none was authoritative, and
trying to use statistics to interpolate information between them.
Every survey will have strengths and weaknesses in what it represents
and what it asks and what we can learn from it. So I don't think that
people didn't "accept the results" of the fpco survey, for example. I
think they accepted them _as results of a particular survey_ with all
the attendant caveats that may entail. So maybe a better way to put it
is that the desire is to have as representative a survey as possible,
and the working idea, at least is that haskell.org backing would help
extend the reach and response to it.

Some of the anxiety in that reddit thread reflects, I think, the
concern that poll results would be used as _prescriptive_ rather than
_descriptive_ -- as has been the case in the past. I.e. "since the
survey says that some percent of people do X, then Y". So the anxiety
is that people don't want to participate in something that they are
worried is going to be more a popularity contest than a way of just
sounding out current moods and practices.

(by the way, Simon -- I don't think people were complaining that HWN
gave links to the FPCo survey -- rather they were opining more
generally that the relative percentages of any given survey are
subject to sample bias, some more than others, and as such one
shouldn't take the percentages yielded by such surveys as the "true"
measure of any particular distribution -- rather they're just the
distribution among those who answered the survey, almost by
definition. In particular, the "stop posting these surveys" comment
was made in the heat of one or another flamewar, about people invoking
the surveys in the _context_ of the flamewar, not about people posting
surveys in a more general sense. That said I too am glad Taylor has
been carrying on the weekly haskell torch.).

Another related thought that hit me -- and here, perhaps working with
others in _analyzing_ the results could be useful -- is that
percentages of X vs. Y are in many ways less informative -- they're
more subject to sample bias, and really all they tell you, assuming
that they're not insignificant, is that there is a measurable and
significant amount of people who answer X or Y. Far more interesting
are the cross-sectional correlations. These let us ask questions about
what sorts of things go together, and let us get a sense of different
clusters of users with different concerns, niches, and needs.
Furthermore, almost definitionally, correlations are more stable
across variations in sampling than are absolute relationships.
(There's probably something really interesting mathematical here that
I'm curious about -- statistical correlation as a form of
symmetry-invariance of parametricity [and stability as a measure of
how "correlations fail to correlate" as perhaps reflecting higher
topological structure] -- references welcome!).

Anyway, this is all a long-winded way of suggesting that it might be
good if the purpose of the survey was explicitly set out as trying to
inform developers of haskell libraries and tools (and educational
materials) regarding the systems their potential users work on and
develop, and their habits and practices in doing so, and where they
encounter difficulty. That is explicitly as a way of learning rather
than as any sort of horse-race or popularity contest. If things are
framed in this way, then while having as broad a range of users
surveyed as possible is still good, the problem of sample bias becomes
much less significant, because the answers ascertained to these sorts
of questions from a survey are good answers, even in the presence of
such problems. And this also helps frame what a good analysis and
writeup of survey results is, because it helps pick out the
interesting questions one would use survey data to answer.

On a smaller and more concrete note -- I don't think I'd be interested
in what particular JS frameworks haskell users picked. I want to know
if they have an FP solution to the "javascript problem", not what
non-fp JS flavor of the month they're using currently. (I may be in a
minority here :-))

And I also agree that there is a concern in adding too many questions
-- I noticed a recent rust survey attributed its higher response rate
in part to having cut down on how many questions it asked.

Cheers,
Gershom

On Mon, Oct 15, 2018 at 7:42 PM Taylor Fausak <taylor at fausak.me> wrote:
>
> Thanks for the kind words, Simon! They mean a lot :)
>
> I would be happy to include questions that would benefit the GHC team, including:
>
> - Simon's question
> - Reaction to the new pace of GHC releases
> - Target of the GHC team's focus: performance, features, ergonomics, etc.
> - Average wait time before upgrading GHC
>
> Are there any other questions the GHC team would be interested in asking? Perhaps I should ask on a different mailing list.
>
> In response to Gershom's comments:
>
> 1. Asking how people heard about the survey is a great idea. Not only would it let me identify the best ways to reach people, it could also be useful in dealing with selection bias.
>
> 2. Addressed above.
>
> 3. In general, distinguishing between work and home is something I would love to do for basically every question. Unfortunately I think that would balloon the size of the survey. Maybe identifying a few key questions for the work/home split would be the best way to go? Build systems, as you identified, are certainly one of those key questions. Maybe GHC versions used is another?
>
> 4. I also like Go's survey and have been trying to crib as much as I can from it. Questions worth asking:
> - Area of development (web, embedded, etc.)
> - Type of development (server, CLI, desktop, library, etc.)
> - Deploy environments / infrastructure
> - Internal versus external
>
> 5. Asking about JS solutions for web developers is a great idea! I like the choices you've given, and there are a whole slew of JS libraries to include as well, such as React or Vue.
>
> 6. Giving multiple choice answers to the "why did you stop" question (and, in fact, as many questions as possible) is awesome and would make the results much easier to digest. It also makes things easier to compare across time, which could be used to gauge the effectiveness of various endeavors.
>
> Thank you all for your feedback so far! I am very excited about this year's survey. I want to include as many useful questions as I can without overwhelming respondents. As I continue to develop the survey, I constantly ask myself this question: "How would I act on responses to this question?" For example, last year's survey asked if people had contributed to an open source Haskell project. I suspect I will exclude that question because it's not really actionable.
>
> With regards to Haskell.org sponsorship, I still think that throwing around the words "official" and "Haskell.org" would do a lot in terms of credibility. I don't expect that to remove selection bias, but it will let me (us, really) say: We're doing this together for the benefit of all sides. And if people have problems with the survey, I want them to feel comfortable trying to fix those problems, even if they're not on my "side".
>
> On Mon, Oct 15, 2018, at 3:55 AM, Simon Peyton Jones wrote:
> > The GHC team (coincidentally) is cooking up a very short survey
> > that is intended to help guide our strategic priorities.
> >
> > There is only one substantial question:
> >
> >       Imagine that you had one developer working on GHC for
> >       six months full-time, that you were paying for yourself.
> >       What would you ask that person to do?   Keep in mind that
> >       the project should be within GHC itself and tractable
> >       in a six-month time-frame.
> >
> > It has a rather different purpose to Taylor's proposed survey
> > and FP Complete's, because it is not comprehensive, but instead
> > focuses on a single question about a single artefact.
> >
> > Returning to Taylor's question list, yes, it would be interesting to
> > know whether the increased release tempo is perceived as helpful or
> > unhelpful.  (It's helpful for the development /process/ because it makes
> > reduces the pressure to squeeze "just one more thing" into a release:
> > the next bus will be along in only 6 months.)
> >
> > Incidentally, I for one think that it's a Good Thing that you included
> > FP Complete's survey in the HWN, Taylor.  First, I think it's a
> > substantial and interesting piece of work -- and /any/ survey is
> > vulnerable to response bias.  Second, I don’t think anyone should expect
> > you as HWN editor to play a role as community censor. Third,
> > deliberately excluding it would in itself be a divisive act in a
> > community that needs less division and more love.
> >
> > You do a fantastic job with HWN.  Please keep doing it!
> >
> > Thanks
> >
> > Simon
> >
> > | -----Original Message-----
> > | From: Haskell-community <haskell-community-bounces at haskell.org> On
> > | Behalf Of Gershom B
> > | Sent: 15 October 2018 01:56
> > | To: taylor at fausak.me; Mathieu Boespflug <mboes at tweag.net>; Ben Gamari
> > | <bgamari at gmail.com>
> > | Cc: haskell-community at haskell.org
> > | Subject: Re: [Haskell-community] 2018 state of Haskell survey
> > |
> > | (cc mathieu boespflug, ben gamari)
> > |
> > | One more thought:
> > |
> > | mathieu, ben -- do you think you would be interested in any questions
> > | on the frequency of ghc releases -- if people appreciate more
> > | frequent, smaller releases, or not, etc?
> > |
> > | I wonder if there are any other questions as well about core libraries
> > | vs. performance vs. "big features" (like type system things) vs. small
> > | ergonomic features etc. that the core ghc team might be interested in
> > | sounding out people on, bearing in mind the necessary limitations of
> > | survey derived data.
> > |
> > | --g
> > | On Sun, Oct 14, 2018 at 8:36 PM Gershom B <gershomb at gmail.com> wrote:
> > | >
> > | > Hi Taylor.
> > | >
> > | > We're discussing this in the committee. I agree that to the extent
> > | > they can accurately reflect something, language surveys are useful,
> > | > and appreciate that you want to run a useful survey, and certainly
> > | > want to encourage and help you in making it as broad and useful as
> > | > possible. That said, I don't know if slapping a "haskell.org" label on
> > | > the survey will help manage the biggest drivers of selection bias --
> > | > which is not only about who chooses to respond, but about who is
> > | > reached through what mechanisms. (I don't know the relative importance
> > | > of response-bias vs. outreach-bias in general, and would be curious if
> > | > somebody has some good research on that to point to). I honestly don't
> > | > know if we have enough channels _in general_ to do a good survey, no
> > | > matter who runs it or how at all! Regardless of the decision we come
> > | > to, here are a few of my personal thoughts on the questions you have
> > | > thus far, and what could be added:
> > | >
> > | > 1) A question "how did you hear about this survey" -- this could at
> > | > least help to disentangle outreach-bias, or notice it, depending on if
> > | > it induces any correlations.
> > | >
> > | > 2) A question on how long after a new GHC release users upgrade --
> > | > both personally, and at work.
> > | >
> > | > 3) Distinguishing between personal and work build-systems in the
> > | > relevant question.
> > | >
> > | > 4) I think the sorts of questions that make sense to ask in this early
> > | > part can resemble those in the first part of the Go survey:
> > | >
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.go
> > | lang.org%2Fsurvey2017-
> > | results&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420bbf2
> > | 008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6367516175958
> > | 02229&sdata=2wB5Ph5%2Be6wZm0CWP7Yzx%2Fxe4dOlKHqNPKZReiJmE50%3D&r
> > | eserved=0 (I especially like the
> > | > questions about the area of development and server vs cli apps vs
> > | > libraries). I also like their questions about what environments teams
> > | > deploy programs to. It would also be worth asking if the apps
> > | > developed are customer-facing or internal.
> > | >
> > | > 5) I think an interesting question would be what preferred js
> > | > solution, if any, teams adopt -- i.e. ghcjs, typescript, purescript,
> > | > raw js, etc.
> > | >
> > | > 6)  for the "why did you stop" question, there are a good range of
> > | > potential multi-choice answers that can be drawn from with the rust
> > | > user survey:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.ru
> > | st-lang.org%2F2017%2F09%2F05%2FRust-2017-Survey-
> > | Results.html&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e42
> > | 0bbf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63675161
> > | 7595802229&sdata=MpnYeOui%2FfTPfpYRLlzzpStJHqYWJ4ENZd77qdrQ1BY%3D&am
> > | p;reserved=0
> > | >
> > | > Cheers,
> > | > Gershom
> > | >
> > | > On Sun, Oct 14, 2018 at 3:32 PM Taylor Fausak <taylor at fausak.me>
> > | wrote:
> > | > >
> > | > > I'm not entirely sure what official support would look like. A few
> > | things come to mind:
> > | > >
> > | > > 1. Simply putting "official" somewhere in the title, such as:
> > | Official 2018 state of Haskell survey.
> > | > >
> > | > > 2. Putting something about Haskell.org in the description of the
> > | survey, such as: Sponsored by Haskell.org.
> > | > >
> > | > > 3. Announcing the survey through channels that I may not be aware
> > | of. Or helping me announce the survey through various channels (mailing
> > | lists, Reddit, and so on) by mentioning Haskell.org.
> > | > >
> > | > > I'm sure there are more ways that I'm not thinking of.
> > | > >
> > | > > Perhaps it would be better to state my goal and see what, if
> > | anything, can be to achieve that goal? My goal is for this survey to be
> > | *the* authoritative Haskell survey and for the community to broadly
> > | accept it results. In particular I would like to avoid reactions like
> > | these to the recent FP Complete survey:
> > | > >
> > | > > > I browse r/haskell all the time and follow FPco on Twitter, and I
> > | wasn't aware of this survey. [1]
> > | > >
> > | > > > It should not be surprising to think that fp complete has much
> > | better outreach to Stackage users than to non Stackage users. [2]
> > | > >
> > | > > > Any survey hosted by FPComplete is biased towards users of stack,
> > | for reasons that should be self evident. [3]
> > | > >
> > | > > Similar sentiments have been expressed about last year's Haskell
> > | Weekly survey:
> > | > >
> > | > > > Haskell Weekly's reputation is tainted as it appears to be seen as
> > | partisan (and I tend to agree). [4]
> > | > >
> > | > > > one of those surveys is from FP Complete and one of them is from
> > | someone who I would consider very partisan in these kind of discussions.
> > | [5]
> > | > >
> > | > > > For the love of god stop posting those surveys. They're not
> > | convincing and obviously flawed. [6]
> > | > >
> > | > > I don't expect to be able to make everyone happy, but I think that
> > | presenting this year's survey as sponsored by both Haskell Weekly and
> > | Haskell.org would go a long way toward making it acceptable to a broad
> > | range of the Haskell community.
> > | > >
> > | > > I hope that helps!
> > | > >
> > | > > [1]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F9mm05d%2F2018_haskell_survey_results%2
> > | Fe7gplya%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420
> > | bbf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636751617
> > | 595802229&sdata=X518Wt%2BsB5yepxQO7iKxYzbcOBqNTwH4QvesKNVccBg%3D&amp
> > | ;reserved=0
> > | > > [2]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F8tc8pr%2Ffp_complete_launches_new_bloc
> > | kchain_auditing%2Fe188ftv%2F%3Fcontext%3D3&data=02%7C01%7Csimonpj%40
> > | microsoft.com%7C9fbbfac1fd9e420bbf2008d63238f7c4%7C72f988bf86f141af91ab2
> > | d7cd011db47%7C1%7C0%7C636751617595802229&sdata=4oPTVet8hrgE2lTzjt%2B
> > | 02f13o%2Fq%2BOrnGoITlsBsdKgw%3D&reserved=0
> > | > > [3]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F9mm05d%2F2018_haskell_survey_results%2
> > | Fe7gpfwe%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420
> > | bbf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636751617
> > | 595802229&sdata=9L8ztYN3kOArpdAhOiydFI6gfuz8Evu5%2FmtHZ5j3vyY%3D&amp
> > | ;reserved=0
> > | > > [4]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F9mm05d%2F2018_haskell_survey_results%2
> > | Fe7ka8xn%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420
> > | bbf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636751617
> > | 595802229&sdata=tpyLQ%2BA3Lj4GLfxjMUgglfRyl46jH6go1ZE3OhVLwLo%3D&amp
> > | ;reserved=0
> > | > > [5]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F8tc8pr%2Ffp_complete_launches_new_bloc
> > | kchain_auditing%2Fe187pzq%2F%3Fcontext%3D1&data=02%7C01%7Csimonpj%40
> > | microsoft.com%7C9fbbfac1fd9e420bbf2008d63238f7c4%7C72f988bf86f141af91ab2
> > | d7cd011db47%7C1%7C0%7C636751617595812242&sdata=ZbjRYFCXl4rraHE%2Bno7
> > | kefm5N3%2FvtNpaI%2FXTz3N6i7w%3D&reserved=0
> > | > > [6]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnp.redd
> > | it.com%2Fr%2Fhaskell%2Fcomments%2F8uw9hw%2Fpsa_cabal_breaks_with_yaml083
> > | 1_on_ghc_710_and%2Fe1lfzgr%2F%3Fcontext%3D1&data=02%7C01%7Csimonpj%4
> > | 0microsoft.com%7C9fbbfac1fd9e420bbf2008d63238f7c4%7C72f988bf86f141af91ab
> > | 2d7cd011db47%7C1%7C0%7C636751617595812242&sdata=oyy8raidob%2Bjf8Evw0
> > | Kb6aCul0knqYlOlxN8zLjzl3I%3D&reserved=0
> > | > >
> > | > >
> > | > > On Sun, Oct 14, 2018, at 2:21 PM, Neil Mitchell wrote:
> > | > >
> > | > > Hi Taylor,
> > | > >
> > | > > What does official support look like? I don't think there's anything
> > | above and beyond what you're already doing.
> > | > >
> > | > > Thanks, Neil
> > | > >
> > | > > On Sun, 14 Oct 2018 at 3:49 pm, Taylor Fausak <taylor at fausak.me>
> > | wrote:
> > | > >
> > | > > (I have CCed <committee at haskell.org> because the haskell-community
> > | mailing list seems relatively quiet and I want to make sure this is
> > | seen.)
> > | > >
> > | > > Hello! My name is Taylor Fausak. I run the Haskell Weekly
> > | newsletter. Last year I published a survey [1] for the Haskell
> > | community. I collected and reported [2] on about 1,335 responses. I plan
> > | on publishing another survey this year on the same date, November 1st. I
> > | am developing it in the open again [3] and would love to hear from any
> > | interested parties. Please let me know if you have any ideas about the
> > | survey!
> > | > >
> > | > > This year I am interested in making the survey official by seeking
> > | support from Haskell.org. Is such a thing possible and desirable?
> > | > >
> > | > > Thanks for your consideration! I hope to hear from you soon.
> > | > >
> > | > > [1]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhaskell
> > | weekly.news%2Fsurveys%2F2017.html&data=02%7C01%7Csimonpj%40microsoft
> > | .com%7C9fbbfac1fd9e420bbf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db
> > | 47%7C1%7C0%7C636751617595812242&sdata=0eGdCpPFIyxgNp5YNXvP6TZQMAvTJ%
> > | 2BIck4UvSV7wWSU%3D&reserved=0
> > | > > [2]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftaylor.
> > | fausak.me%2F2017%2F11%2F15%2F2017-state-of-haskell-survey-
> > | results%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420b
> > | bf2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6367516175
> > | 95812242&sdata=QZjow3PKFG7BitCQCdTe7xW%2BXl4YA4ICVmA%2B87GXOCs%3D&am
> > | p;reserved=0
> > | > > [3]:
> > | https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> > | com%2Fhaskellweekly%2Fhaskellweekly.github.io%2Fissues%2F206&data=02
> > | %7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420bbf2008d63238f7c4%7C72f
> > | 988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636751617595812242&sdata=v2b
> > | W4fjDtWrJnKACnQaoLRoaYw0kADGucwasn2X0wtI%3D&reserved=0
> > | > > _______________________________________________
> > | > > Haskell-community mailing list
> > | > > Haskell-community at haskell.org
> > | > >
> > | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has
> > | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fhaskell-
> > | community&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420bb
> > | f2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63675161759
> > | 5812242&sdata=2SB8gyaEQI2zxvUE%2FEdw%2B%2B2OLcJW10hpH8A1McoP0MY%3D&a
> > | mp;reserved=0
> > | > >
> > | > >
> > | > > _______________________________________________
> > | > > Haskell-community mailing list
> > | > > Haskell-community at haskell.org
> > | > >
> > | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has
> > | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fhaskell-
> > | community&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420bb
> > | f2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63675161759
> > | 5812242&sdata=2SB8gyaEQI2zxvUE%2FEdw%2B%2B2OLcJW10hpH8A1McoP0MY%3D&a
> > | mp;reserved=0
> > | _______________________________________________
> > | Haskell-community mailing list
> > | Haskell-community at haskell.org
> > | https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has
> > | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fhaskell-
> > | community&data=02%7C01%7Csimonpj%40microsoft.com%7C9fbbfac1fd9e420bb
> > | f2008d63238f7c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63675161759
> > | 5812242&sdata=2SB8gyaEQI2zxvUE%2FEdw%2B%2B2OLcJW10hpH8A1McoP0MY%3D&a
> > | mp;reserved=0