From lexi.lambda at gmail.com Wed Jul 1 19:12:58 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Wed, 1 Jul 2020 14:12:58 -0500 Subject: Async exceptions and delimited continuations Message-ID: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> Hi all, As some of you are likely aware, I have an open GHC proposal[1] to add native support for delimited continuations to the RTS. I also have an incomplete implementation,[2] and the only major remaining obstacle concerns async exceptions. The issue is subtle, so I’ve tried to provide all the necessary context in this email. If you’re already familiar with the ideas at play, you can skip the context about how delimited continuations work. For those unfamiliar, delimited continuations allow capturing slices of the call stack and restoring them later. For example, the program do y <- prompt $ do x <- control0 $ \k -> k (pure 10) pure (x + 5) print y will print 15. To understand what’s happening operationally, we can imagine an abstract call stack made up of continuation frames: ┌──────────┐ │ ● + 5 │ redex: control0 $ \k -> k (pure 10) ├──────────┤ │ prompt ● │ ├──────────┤ │ print ● │ ├──────────┤ │ ... │ ├──────────┤ Here, each ● represents the “hole” where the evaluated result of the redex will be returned. `control0` moves all the frames between the top of the stack and the first `prompt` into the heap and returns a reference to them, so after a single reduction step, we have ┌──────────┐ │ print ● │ redex: k1 (pure 10) ├──────────┤ heap: k1 = ┌──────────┐ │ ... │ │ ● + 5 │ ├──────────┤ └──────────┘ When a continuation is applied, its stored stack frames are copied back onto the top of the current stack, and the argument becomes the new redex: ┌──────────┐ │ ● + 5 │ redex: pure 10 ├──────────┤ │ print ● │ ├──────────┤ │ ... │ ├──────────┤ Now it should hopefully be clear how we end up printing 15. With that context, consider the following expression: prompt $ mask_ $ do x <- control0 $ \k -> k (pure 10) f x The call stack at the point of control0 looks very similar in this program, but now we have a use of `mask_` in there as well: ┌──────────┐ │ f ● │ redex: control0 $ \k -> k (pure 10) ├──────────┤ exns: masked │ mask_ ● │ ├──────────┤ │ prompt ● │ ├──────────┤ │ ... │ ├──────────┤ When capturing the continuation, we’ll unwind the stack the same way we did before. Because we’re popping mask_ off the stack, we’ll unmask async exceptions: ┌──────────┐ redex: k1 (pure 10) │ ... │ exns: not masked ├──────────┤ heap: k1 = ┌──────────┐ │ f ● │ ├──────────┤ │ mask_ ● │ └──────────┘ Now when we apply `k1`, we’ll copy the `mask_` frame back onto the stack, and we must re-mask async exceptions. Otherwise, exceptions will not be masked during the call to `f`, which would be wrong. Why is this a problem? The RTS applies an optimization: if you call mask_ (actually maskAsyncExceptions#) while exceptions are already masked, it doesn’t push a new stack frame at all. So, for example, if you write mask_ $ mask_ $ foo bar you’ll end up with only one mask_ frame on the call stack, not two. This tiny optimization actually allows not one but two savings: 1. The immediate benefit is that we save a stack frame. 2. The hidden benefit is that we never need to push the old exception masking state onto the stack. If we had multiple mask_ frames on the stack simultaneously, we wouldn’t know what to do when returning: should we unmask them, or should they stay masked? We’d need to push that information onto the stack in the same way we must push a return address when calling a function. By skipping these redundant stack frames, we can always be certain the right thing to do on return is to unmask async exceptions. No need to store anything else. (This explanation is a slight simplification, especially once maskUninterruptible comes into play, but it’s close enough.) Now you may see the looming problem: this strategy completely breaks down in the presence of delimited continuations. With delimited continuations, we might have a program like this: mask_ $ prompt $ mask_ $ ... If we capture a continuation up to this prompt, we expect the inner mask_ frame to be captured along with it. But that won’t happen if we never pushed a mask_ frame at all due to the aforementioned optimization. So now I can finally state my question: what is the right solution for this? I see three obvious possible ways forward: 1. Keep track of whether or not we’re inside a prompt and skip the optimization in that case. 2. Keep some bookkeeping that tracks the modifications to the async exception masking state since the most recently pushed prompt. 3. Just don’t bother with the optimization at all. Option 3 certainly seems the most appealing from a simplicity point of view, and I suspect the optimization doesn’t matter much in practice. Why? Because the real `mask` implementation from Control.Exception already avoids re-masking exceptions if they’re masked! (And that’s okay, because prompt# and control0# are not intended to be used directly in IO, so code that uses them can provide its own version of `mask`.) However, it is admittedly possible for the restore action passed to the argument of `mask` to create redundant calls, as the check is only performed in `mask` itself. Is eliminating this optimization an acceptable compromise? Or is there reason to believe this is important for performance of real programs? Thanks, Alexis [1]: https://github.com/ghc-proposals/ghc-proposals/pull/313 [2]: https://gitlab.haskell.org/lexi.lambda/ghc/-/commits/first-class-continuations From harendra.kumar at gmail.com Thu Jul 2 10:48:02 2020 From: harendra.kumar at gmail.com (Harendra Kumar) Date: Thu, 2 Jul 2020 16:18:02 +0530 Subject: perf regression with TypeFamilies In-Reply-To: References: Message-ID: I opened a GHC ticket (https://gitlab.haskell.org/ghc/ghc/-/issues/18414) with full details. -harendra On Tue, 30 Jun 2020 at 18:25, Harendra Kumar wrote: > Hi Richard, > > I am glad that you are interested in it. It is the runtime performance > that degrades, and yes it is fully reproducible and publicly shareable, the > library is open source so anyone can run these benchmarks. I had postponed > the problem raising an issue to investigate it further here > https://github.com/composewell/streamly/issues/567 . Since you are > interested I will investigate it sooner. I have added some perf numbers in > that issue. > > I was also surprised by it, thinking what can type families possibly do to > degrade run time performance like that. I can try compiling one worst > affected benchmark code and look at the core-2-core passes to see where it > makes the difference. > > -harendra > > On Mon, 29 Jun 2020 at 19:45, Richard Eisenberg wrote: > >> Hi Harendra, >> >> I saw your comment on a ghc proposal ( >> https://github.com/ghc-proposals/ghc-proposals/pull/343#issuecomment-650797297) >> that said you experienced perf regressions with -XTypeFamilies enabled (but >> no other changes). Are these reproducible in a way that can be shared >> publicly? And are the regressions in compile times or run times? >> >> -XTypeFamilies enables -XMonoLocalBinds, which disables >> let-generalization on some nested lets. It is thus just barely conceivable >> that different Core is produced depending on this extension, and that there >> may be a possibility of performance changes. But this would be unexpected, >> and something worth investigating. >> >> In other words: if you can, do please post a bug -- simply enabling >> -XTypeFamilies should not slow anything down! >> >> Thanks, >> Richard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davide at well-typed.com Thu Jul 2 15:24:48 2020 From: davide at well-typed.com (David Eichmann) Date: Thu, 2 Jul 2020 16:24:48 +0100 Subject: Perf notes In-Reply-To: References: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> <96926365-d098-c17c-f94b-1047c5c962fa@well-typed.com> Message-ID: <13efec05-f82c-fb09-03e7-05be207fc55a@well-typed.com> Hello, So CI metrics are being pushed again, good. The immediate issue was perf test T9803. I've looked at metrics across all CI test environments and the last 300 commit on master. Metric looks stable. I've attached the output of: `python3 ./testsuite/driver/perf_notes.py --chart T9203.html --ci --test-name T9203  origin/master~300..origin/master`. Simon, you are on windows, correct? If so the result you posted is using the "x86_64-linux-deb9" environment as a baseline when it should be using "x86_64" which is much closer to the result you're getting. > So it’s just an implementation detail whether the numbers you save are gotten from one run, or another identical one. For the most part yes, but since we usually rebase and batch commit with marge bot, we are really creating a new commit. So it makes some sense to rerun CI. > The CI log tells you the comparison between the preceding commit and this one I'd love for this to be the behavior too. The problem is, most of the time we don't have the metrics for the parent commit, and generating them is expensive. We could automatically checkout / build the previous commit and run perf tests, but this doesn't seem like good design. That's why we've resorted to searching for "most recent" local or CI metrics to establish an approximate baseline. Requiring the developers to always run perf tests locally on the previous commit will be extremely annoying. Another option is to just disable perf tests by default. It's not satisfying but it's hard to think of a better solution. I think the current implementation is still more convenient than before, where baselines were just hard coded into test files. If you wanted to investigate a particular commit, you had to manually establish a baseline. Perhaps the problem with this new system is that it's a bit too "magical" and it's unclear how to interpret results. Perhaps this can be remedied with better output form the test runner. > Would it be possible to output a table (in the log) like we get from nofib-analyse Absolutely. This should be fairly easy to implement. I've created #18417. - David E On 6/29/20 3:08 PM, Simon Peyton Jones wrote: > > Re the doubling of bytes-allocated on T9803, that's a good point. Due > to the recent change in RSA keys, CI is recently failing to upload > metrics (e.g. [1])! I'll fix that then see if I can track down where / > if the metric has really regressed in master. > > Thanks > > Yes we run CI on MRs, but once merged into master CI is run again. > It's only those metrics from CI on master (post merge) that are > ultimately uploaded / used as a baseline. > > OK. But they are guaranteed to be 100.0% identical to the ones > discovered by CI, aren’t they?   So it’s just an implementation detail > whether the numbers you save are gotten from one run, or another > identical one. > > I’m still lost about when I can rely on the perf output of CI and when > I can’t.  I’m really hoping for a simple answer like: > > * The CI log tells you the comparison between the preceding commit > and this one > > No ifs, no buts.  Simple! > > Incidentally, would it be possible to output a table (in the log) like > we get from nofib-analyse.  It looks like this > > Program           Size    Allocs   Runtime   Elapsed TotalMem > > -------------------------------------------------------------------------------- > > boyer          -0.3%     +5.4%     +0.7%     +1.0%      0.0% > > cichelli          -0.3%     +5.9%     -9.9%     -9.5% 0.0% > > compress2          -0.4%     +9.6%     +7.2%     +6.4% 0.0% > > constraints          -0.3%     +0.2%     -3.0% -3.4%      0.0% > > cryptarithm2          -0.3%     -3.9%     -2.2% -2.4%      0.0% > > gamteb          -0.4%     +2.5%     +2.8%     +2.8% 0.0% > > life          -0.3%     -2.2%     -4.7%     -4.9%      0.0% > > lift          -0.3%     -0.3%     -0.8%     -0.5%      0.0% > > linear          -0.3%     -0.1%     -4.1%     -4.5% 0.0% > > mate          -0.2%     +1.4%     -2.2%     -1.9%    -14.3% > > parser          -0.3%     -2.1%     -5.4%     -4.6% 0.0% > > puzzle          -0.3%     +2.1%     -6.6%     -6.3% 0.0% > > simple          -0.4%     +2.8%     -3.4%     -3.3% -2.2% > > veritas          -0.1%     +0.7%     -0.6%     -1.1% 0.0% > > wheel-sieve2          -0.3%    -19.2%    -24.9%    -24.5% -42.9% > > -------------------------------------------------------------------------------- > > Min          -0.4%    -19.2%    -24.9%    -24.5%    -42.9% > > Max          +0.1%     +9.6%     +7.2%     +6.4%    +33.3% > > Geometric Mean          -0.3%     -0.0%     -3.0% -2.9%     -0.3% > > Instantly comprehensible, one line per benchmark.  I find I spent > quite a lot of time search manually in the log and manually building a > table (or excerpts thereof) looking like this. > > I don’t have an opinion about the columns, just wanting a table with > one line per benchmark, and a number of columns. > > Thanks > > Simon > > *From:*David Eichmann > *Sent:* 27 June 2020 20:39 > *To:* Simon Peyton Jones ; ghc-devs at haskell.org > *Subject:* Re: Perf notes > > > I thought that wasn’t possible. Isn’t that what CI is **for**? > > Yes we run CI on MRs, but once merged into master CI is run again. > It's only those metrics from CI on master (post merge) that are > ultimately uploaded / used as a baseline. > > Re the doubling of bytes-allocated on T9803, that's a good point. Due > to the recent change in RSA keys, CI is recently failing to upload > metrics (e.g. [1])! I'll fix that then see if I can track down where / > if the metric has really regressed in master. > > [1] "fatal: Could not read from remote repository." > https://gitlab.haskell.org/ghc/ghc/-/jobs/378487 > > > -- > > David Eichmann, Haskell Consultant > Well-Typed LLP,http://www.well-typed.com > Registered in England & Wales, OC335890 > 118 Wymering Mansions, Wymering Road, London W9 2NF, England -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Jul 2 15:41:50 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 2 Jul 2020 15:41:50 +0000 Subject: Perf notes In-Reply-To: <13efec05-f82c-fb09-03e7-05be207fc55a@well-typed.com> References: <844cb770-201f-238f-235c-78f13114bf24@well-typed.com> <96926365-d098-c17c-f94b-1047c5c962fa@well-typed.com> <13efec05-f82c-fb09-03e7-05be207fc55a@well-typed.com> Message-ID: Simon, you are on windows, correct? If so the result you posted is using the "x86_64-linux-deb9" environment as a baseline when it should be using "x86_64" which is much closer to the result you're getting. WSL actually. (Windows Subsystem for Linux.) Simon From: David Eichmann Sent: 02 July 2020 16:25 To: Simon Peyton Jones ; ghc-devs at haskell.org Subject: Re: Perf notes Hello, So CI metrics are being pushed again, good. The immediate issue was perf test T9803. I've looked at metrics across all CI test environments and the last 300 commit on master. Metric looks stable. I've attached the output of: `python3 ./testsuite/driver/perf_notes.py --chart T9203.html --ci --test-name T9203 origin/master~300..origin/master`. Simon, you are on windows, correct? If so the result you posted is using the "x86_64-linux-deb9" environment as a baseline when it should be using "x86_64" which is much closer to the result you're getting. > So it's just an implementation detail whether the numbers you save are gotten from one run, or another identical one. For the most part yes, but since we usually rebase and batch commit with marge bot, we are really creating a new commit. So it makes some sense to rerun CI. > The CI log tells you the comparison between the preceding commit and this one I'd love for this to be the behavior too. The problem is, most of the time we don't have the metrics for the parent commit, and generating them is expensive. We could automatically checkout / build the previous commit and run perf tests, but this doesn't seem like good design. That's why we've resorted to searching for "most recent" local or CI metrics to establish an approximate baseline. Requiring the developers to always run perf tests locally on the previous commit will be extremely annoying. Another option is to just disable perf tests by default. It's not satisfying but it's hard to think of a better solution. I think the current implementation is still more convenient than before, where baselines were just hard coded into test files. If you wanted to investigate a particular commit, you had to manually establish a baseline. Perhaps the problem with this new system is that it's a bit too "magical" and it's unclear how to interpret results. Perhaps this can be remedied with better output form the test runner. > Would it be possible to output a table (in the log) like we get from nofib-analyse Absolutely. This should be fairly easy to implement. I've created #18417. - David E On 6/29/20 3:08 PM, Simon Peyton Jones wrote: Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master. Thanks Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline. OK. But they are guaranteed to be 100.0% identical to the ones discovered by CI, aren't they? So it's just an implementation detail whether the numbers you save are gotten from one run, or another identical one. I'm still lost about when I can rely on the perf output of CI and when I can't. I'm really hoping for a simple answer like: * The CI log tells you the comparison between the preceding commit and this one No ifs, no buts. Simple! Incidentally, would it be possible to output a table (in the log) like we get from nofib-analyse. It looks like this Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- boyer -0.3% +5.4% +0.7% +1.0% 0.0% cichelli -0.3% +5.9% -9.9% -9.5% 0.0% compress2 -0.4% +9.6% +7.2% +6.4% 0.0% constraints -0.3% +0.2% -3.0% -3.4% 0.0% cryptarithm2 -0.3% -3.9% -2.2% -2.4% 0.0% gamteb -0.4% +2.5% +2.8% +2.8% 0.0% life -0.3% -2.2% -4.7% -4.9% 0.0% lift -0.3% -0.3% -0.8% -0.5% 0.0% linear -0.3% -0.1% -4.1% -4.5% 0.0% mate -0.2% +1.4% -2.2% -1.9% -14.3% parser -0.3% -2.1% -5.4% -4.6% 0.0% puzzle -0.3% +2.1% -6.6% -6.3% 0.0% simple -0.4% +2.8% -3.4% -3.3% -2.2% veritas -0.1% +0.7% -0.6% -1.1% 0.0% wheel-sieve2 -0.3% -19.2% -24.9% -24.5% -42.9% -------------------------------------------------------------------------------- Min -0.4% -19.2% -24.9% -24.5% -42.9% Max +0.1% +9.6% +7.2% +6.4% +33.3% Geometric Mean -0.3% -0.0% -3.0% -2.9% -0.3% Instantly comprehensible, one line per benchmark. I find I spent quite a lot of time search manually in the log and manually building a table (or excerpts thereof) looking like this. I don't have an opinion about the columns, just wanting a table with one line per benchmark, and a number of columns. Thanks Simon From: David Eichmann Sent: 27 June 2020 20:39 To: Simon Peyton Jones ; ghc-devs at haskell.org Subject: Re: Perf notes > I thought that wasn't possible. Isn't that what CI is *for*? Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline. Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master. [1] "fatal: Could not read from remote repository." https://gitlab.haskell.org/ghc/ghc/-/jobs/378487 -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England -------------- next part -------------- An HTML attachment was scrubbed... URL: From iavor.diatchki at gmail.com Thu Jul 2 16:24:05 2020 From: iavor.diatchki at gmail.com (Iavor Diatchki) Date: Thu, 2 Jul 2020 09:24:05 -0700 Subject: Async exceptions and delimited continuations In-Reply-To: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> References: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> Message-ID: Hi, I am by no means an expert on the GHC RTS but all 3 suggestions seem quite reasonable to me. A good way to make a decision might be to collect some data, at least for the things that might be easy to measure. In particular, it would be interesting to temporarily disable the optimization and run some benchmarks on some IO/exceptions heavy code (some sort of server? or maybe a synthetic benchmark to really stress the masking/unmaksing) and see what's the change in performance. -Iavor On Wed, Jul 1, 2020 at 12:13 PM Alexis King wrote: > Hi all, > > As some of you are likely aware, I have an open GHC proposal[1] to add > native support for delimited continuations to the RTS. I also have an > incomplete implementation,[2] and the only major remaining obstacle > concerns async exceptions. The issue is subtle, so I’ve tried to > provide all the necessary context in this email. If you’re already > familiar with the ideas at play, you can skip the context about how > delimited continuations work. > > For those unfamiliar, delimited continuations allow capturing slices > of the call stack and restoring them later. For example, the program > > do y <- prompt $ do x <- control0 $ \k -> k (pure 10) > pure (x + 5) > print y > > will print 15. To understand what’s happening operationally, we can > imagine an abstract call stack made up of continuation frames: > > ┌──────────┐ > │ ● + 5 │ redex: control0 $ \k -> k (pure 10) > ├──────────┤ > │ prompt ● │ > ├──────────┤ > │ print ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > Here, each ● represents the “hole” where the evaluated result of the > redex will be returned. `control0` moves all the frames between the > top of the stack and the first `prompt` into the heap and returns a > reference to them, so after a single reduction step, we have > > ┌──────────┐ > │ print ● │ redex: k1 (pure 10) > ├──────────┤ heap: k1 = ┌──────────┐ > │ ... │ │ ● + 5 │ > ├──────────┤ └──────────┘ > > When a continuation is applied, its stored stack frames are copied > back onto the top of the current stack, and the argument becomes the > new redex: > > ┌──────────┐ > │ ● + 5 │ redex: pure 10 > ├──────────┤ > │ print ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > Now it should hopefully be clear how we end up printing 15. > > With that context, consider the following expression: > > prompt $ mask_ $ do x <- control0 $ \k -> k (pure 10) > f x > > The call stack at the point of control0 looks very similar in this > program, but now we have a use of `mask_` in there as well: > > ┌──────────┐ > │ f ● │ redex: control0 $ \k -> k (pure 10) > ├──────────┤ exns: masked > │ mask_ ● │ > ├──────────┤ > │ prompt ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > When capturing the continuation, we’ll unwind the stack the same way > we did before. Because we’re popping mask_ off the stack, we’ll unmask > async exceptions: > > ┌──────────┐ redex: k1 (pure 10) > │ ... │ exns: not masked > ├──────────┤ heap: k1 = ┌──────────┐ > │ f ● │ > ├──────────┤ > │ mask_ ● │ > └──────────┘ > > Now when we apply `k1`, we’ll copy the `mask_` frame back onto the > stack, and we must re-mask async exceptions. Otherwise, exceptions > will not be masked during the call to `f`, which would be wrong. > > Why is this a problem? The RTS applies an optimization: if you call > mask_ (actually maskAsyncExceptions#) while exceptions are already > masked, it doesn’t push a new stack frame at all. So, for example, if > you write > > mask_ $ mask_ $ foo bar > > you’ll end up with only one mask_ frame on the call stack, not two. > This tiny optimization actually allows not one but two savings: > > 1. The immediate benefit is that we save a stack frame. > > 2. The hidden benefit is that we never need to push the old > exception masking state onto the stack. > > If we had multiple mask_ frames on the stack simultaneously, we > wouldn’t know what to do when returning: should we unmask them, > or should they stay masked? We’d need to push that information > onto the stack in the same way we must push a return address > when calling a function. > > By skipping these redundant stack frames, we can always be > certain the right thing to do on return is to unmask async > exceptions. No need to store anything else. > > (This explanation is a slight simplification, especially once > maskUninterruptible comes into play, but it’s close enough.) > > Now you may see the looming problem: this strategy completely breaks > down in the presence of delimited continuations. With delimited > continuations, we might have a program like this: > > mask_ $ prompt $ mask_ $ ... > > If we capture a continuation up to this prompt, we expect the inner > mask_ frame to be captured along with it. But that won’t happen if we > never pushed a mask_ frame at all due to the aforementioned > optimization. > > So now I can finally state my question: what is the right solution for > this? I see three obvious possible ways forward: > > 1. Keep track of whether or not we’re inside a prompt and skip the > optimization in that case. > > 2. Keep some bookkeeping that tracks the modifications to the > async exception masking state since the most recently pushed > prompt. > > 3. Just don’t bother with the optimization at all. > > Option 3 certainly seems the most appealing from a simplicity point of > view, and I suspect the optimization doesn’t matter much in practice. > Why? Because the real `mask` implementation from Control.Exception > already avoids re-masking exceptions if they’re masked! (And that’s > okay, because prompt# and control0# are not intended to be used > directly in IO, so code that uses them can provide its own version of > `mask`.) However, it is admittedly possible for the restore action > passed to the argument of `mask` to create redundant calls, as the > check is only performed in `mask` itself. > > Is eliminating this optimization an acceptable compromise? Or is there > reason to believe this is important for performance of real programs? > > Thanks, > Alexis > > [1]: https://github.com/ghc-proposals/ghc-proposals/pull/313 > [2]: > https://gitlab.haskell.org/lexi.lambda/ghc/-/commits/first-class-continuations > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Thu Jul 2 17:46:20 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 02 Jul 2020 13:46:20 -0400 Subject: Async exceptions and delimited continuations In-Reply-To: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> References: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> Message-ID: <87k0zl6c9r.fsf@smart-cactus.org> Alexis King writes: > Hi all, > > As some of you are likely aware, I have an open GHC proposal[1] to add > native support for delimited continuations to the RTS. I also have an > incomplete implementation,[2] and the only major remaining obstacle > concerns async exceptions. The issue is subtle, so I’ve tried to > provide all the necessary context in this email. If you’re already > familiar with the ideas at play, you can skip the context about how > delimited continuations work. > ... > This tiny optimization actually allows not one but two savings: > > 1. The immediate benefit is that we save a stack frame. > > 2. The hidden benefit is that we never need to push the old > exception masking state onto the stack. > > If we had multiple mask_ frames on the stack simultaneously, we > wouldn’t know what to do when returning: should we unmask them, > or should they stay masked? We’d need to push that information > onto the stack in the same way we must push a return address > when calling a function. > > By skipping these redundant stack frames, we can always be > certain the right thing to do on return is to unmask async > exceptions. No need to store anything else. > Indeed. However, I don't think it would be that expensive to store the masking state with a suitable implementation strategy. Specifically, don't add a "masking state" field to the return frame. Rather, I think you want to have `mask` push one of two possible return frame types: * in the case that we are already masked push an "already masked" frame. The entry code for this would be a no-op. * in the case that we aren't currently masked push the usual stg_maskAsyncExceptionszh_ret_info return frame. This incurs minimal overhead while preserving the information that you need. We avoid adding a word to the stack frame (likely saving an instruction) and in cases where today the optimsation fires the user would only pay for a return to a trivial entry frame. It seems to me that this is as close to free as we will get. Under this scheme `control` can simply look for "already masked" frames when copying the stack. > (This explanation is a slight simplification, especially once > maskUninterruptible comes into play, but it’s close enough.) > > Now you may see the looming problem: this strategy completely breaks > down in the presence of delimited continuations. With delimited > continuations, we might have a program like this: > > mask_ $ prompt $ mask_ $ ... > > If we capture a continuation up to this prompt, we expect the inner > mask_ frame to be captured along with it. But that won’t happen if we > never pushed a mask_ frame at all due to the aforementioned > optimization. > > So now I can finally state my question: what is the right solution for > this? I see three obvious possible ways forward: > > 1. Keep track of whether or not we’re inside a prompt and skip the > optimization in that case. > > 2. Keep some bookkeeping that tracks the modifications to the > async exception masking state since the most recently pushed > prompt. > > 3. Just don’t bother with the optimization at all. > > Option 3 certainly seems the most appealing from a simplicity point of > view, and I suspect the optimization doesn’t matter much in practice. > Why? Because the real `mask` implementation from Control.Exception > already avoids re-masking exceptions if they’re masked! (And that’s > okay, because prompt# and control0# are not intended to be used > directly in IO, so code that uses them can provide its own version of > `mask`.) However, it is admittedly possible for the restore action > passed to the argument of `mask` to create redundant calls, as the > check is only performed in `mask` itself. > > Is eliminating this optimization an acceptable compromise? Or is there > reason to believe this is important for performance of real programs? > It never hurts to measure. Perhaps establish the worst-case performance impact by looking at a simple program which just masks repeatedly. For what it's worth, I rather doubt that this optimisation is going to make or break any program. Exception operations in general are somewhat rare in my experience and the cost of pushing a stack frame (or, perhaps more importantly on modern hardware, returning through it) should be reasonably cheap. By the way, when you go to implement this do add a few comments to the masking primitives. It took a bit of staring to work out what was going on in there. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From alexander.kjeldaas at gmail.com Fri Jul 3 10:20:19 2020 From: alexander.kjeldaas at gmail.com (Alexander Kjeldaas) Date: Fri, 3 Jul 2020 12:20:19 +0200 Subject: A small documentation PR on github Message-ID: Hi devs! I created a small documentation PR for the GHC FFI on github and noticed that there's another one-liner PR from May 2019 that was not merged. https://github.com/ghc/ghc/pull/260 https://github.com/ghc/ghc/pull/255 Just checking that simple PRs are still accepted on github. Alexander -------------- next part -------------- An HTML attachment was scrubbed... URL: From takenobu.hs at gmail.com Fri Jul 3 10:29:58 2020 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Fri, 3 Jul 2020 19:29:58 +0900 Subject: Async exceptions and delimited continuations In-Reply-To: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> References: <2F29943E-55E1-4172-9E6C-C19E47B5EED1@gmail.com> Message-ID: Hi Alexis, I prepared a framework page on ghc-wiki about this proposal: * https://gitlab.haskell.org/ghc/ghc/-/wikis/delimited-continuations If it helps, please use the page to share ideas with developers and users. It is a draft page. Please feel free to rewrite all contents as you like. You could also create sub pages. Some similar pages for proposals are here: * https://gitlab.haskell.org/ghc/ghc/-/wikis/linear-types * https://gitlab.haskell.org/ghc/ghc/-/wikis/dependent-haskell Regards, Takenobu On Thu, Jul 2, 2020 at 4:13 AM Alexis King wrote: > > Hi all, > > As some of you are likely aware, I have an open GHC proposal[1] to add > native support for delimited continuations to the RTS. I also have an > incomplete implementation,[2] and the only major remaining obstacle > concerns async exceptions. The issue is subtle, so I’ve tried to > provide all the necessary context in this email. If you’re already > familiar with the ideas at play, you can skip the context about how > delimited continuations work. > > For those unfamiliar, delimited continuations allow capturing slices > of the call stack and restoring them later. For example, the program > > do y <- prompt $ do x <- control0 $ \k -> k (pure 10) > pure (x + 5) > print y > > will print 15. To understand what’s happening operationally, we can > imagine an abstract call stack made up of continuation frames: > > ┌──────────┐ > │ ● + 5 │ redex: control0 $ \k -> k (pure 10) > ├──────────┤ > │ prompt ● │ > ├──────────┤ > │ print ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > Here, each ● represents the “hole” where the evaluated result of the > redex will be returned. `control0` moves all the frames between the > top of the stack and the first `prompt` into the heap and returns a > reference to them, so after a single reduction step, we have > > ┌──────────┐ > │ print ● │ redex: k1 (pure 10) > ├──────────┤ heap: k1 = ┌──────────┐ > │ ... │ │ ● + 5 │ > ├──────────┤ └──────────┘ > > When a continuation is applied, its stored stack frames are copied > back onto the top of the current stack, and the argument becomes the > new redex: > > ┌──────────┐ > │ ● + 5 │ redex: pure 10 > ├──────────┤ > │ print ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > Now it should hopefully be clear how we end up printing 15. > > With that context, consider the following expression: > > prompt $ mask_ $ do x <- control0 $ \k -> k (pure 10) > f x > > The call stack at the point of control0 looks very similar in this > program, but now we have a use of `mask_` in there as well: > > ┌──────────┐ > │ f ● │ redex: control0 $ \k -> k (pure 10) > ├──────────┤ exns: masked > │ mask_ ● │ > ├──────────┤ > │ prompt ● │ > ├──────────┤ > │ ... │ > ├──────────┤ > > When capturing the continuation, we’ll unwind the stack the same way > we did before. Because we’re popping mask_ off the stack, we’ll unmask > async exceptions: > > ┌──────────┐ redex: k1 (pure 10) > │ ... │ exns: not masked > ├──────────┤ heap: k1 = ┌──────────┐ > │ f ● │ > ├──────────┤ > │ mask_ ● │ > └──────────┘ > > Now when we apply `k1`, we’ll copy the `mask_` frame back onto the > stack, and we must re-mask async exceptions. Otherwise, exceptions > will not be masked during the call to `f`, which would be wrong. > > Why is this a problem? The RTS applies an optimization: if you call > mask_ (actually maskAsyncExceptions#) while exceptions are already > masked, it doesn’t push a new stack frame at all. So, for example, if > you write > > mask_ $ mask_ $ foo bar > > you’ll end up with only one mask_ frame on the call stack, not two. > This tiny optimization actually allows not one but two savings: > > 1. The immediate benefit is that we save a stack frame. > > 2. The hidden benefit is that we never need to push the old > exception masking state onto the stack. > > If we had multiple mask_ frames on the stack simultaneously, we > wouldn’t know what to do when returning: should we unmask them, > or should they stay masked? We’d need to push that information > onto the stack in the same way we must push a return address > when calling a function. > > By skipping these redundant stack frames, we can always be > certain the right thing to do on return is to unmask async > exceptions. No need to store anything else. > > (This explanation is a slight simplification, especially once > maskUninterruptible comes into play, but it’s close enough.) > > Now you may see the looming problem: this strategy completely breaks > down in the presence of delimited continuations. With delimited > continuations, we might have a program like this: > > mask_ $ prompt $ mask_ $ ... > > If we capture a continuation up to this prompt, we expect the inner > mask_ frame to be captured along with it. But that won’t happen if we > never pushed a mask_ frame at all due to the aforementioned > optimization. > > So now I can finally state my question: what is the right solution for > this? I see three obvious possible ways forward: > > 1. Keep track of whether or not we’re inside a prompt and skip the > optimization in that case. > > 2. Keep some bookkeeping that tracks the modifications to the > async exception masking state since the most recently pushed > prompt. > > 3. Just don’t bother with the optimization at all. > > Option 3 certainly seems the most appealing from a simplicity point of > view, and I suspect the optimization doesn’t matter much in practice. > Why? Because the real `mask` implementation from Control.Exception > already avoids re-masking exceptions if they’re masked! (And that’s > okay, because prompt# and control0# are not intended to be used > directly in IO, so code that uses them can provide its own version of > `mask`.) However, it is admittedly possible for the restore action > passed to the argument of `mask` to create redundant calls, as the > check is only performed in `mask` itself. > > Is eliminating this optimization an acceptable compromise? Or is there > reason to believe this is important for performance of real programs? > > Thanks, > Alexis > > [1]: https://github.com/ghc-proposals/ghc-proposals/pull/313 > [2]: https://gitlab.haskell.org/lexi.lambda/ghc/-/commits/first-class-continuations > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From hecate at glitchbra.in Fri Jul 3 18:10:55 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Fri, 3 Jul 2020 20:10:55 +0200 Subject: A small documentation PR on github In-Reply-To: References: Message-ID: <432c6470-c2a8-9491-c97a-b5c2b57b7fec@glitchbra.in> Hi Alexander, and thank you for the heads-up! The development process has largely migrated to the GHC GitLab instance, so the pull-requests may receive less attentions on GitHub. I encourage you to read our process https://gitlab.haskell.org/ghc/ghc/-/issues/17929#merge-requests-etiquette so that the PRs receive all the love they deserve! Cheers! Hécate Le 03/07/2020 à 12:20, Alexander Kjeldaas a écrit : > > Hi devs! > > I created a small documentation PR for the GHC FFI on github and > noticed that there's another one-liner PR from May 2019 that was not > merged. > > https://github.com/ghc/ghc/pull/260 > https://github.com/ghc/ghc/pull/255 > > Just checking that simple PRs are still accepted on github. > > Alexander > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Fri Jul 3 18:18:23 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 03 Jul 2020 14:18:23 -0400 Subject: A small documentation PR on github In-Reply-To: References: Message-ID: <87lfk04g40.fsf@smart-cactus.org> Alexander Kjeldaas writes: > Hi devs! > > I created a small documentation PR for the GHC FFI on github and noticed > that there's another one-liner PR from May 2019 that was not merged. > > https://github.com/ghc/ghc/pull/260 > https://github.com/ghc/ghc/pull/255 > > Just checking that simple PRs are still accepted on github. > An excellent point. In my mind the move to GitLab has addressed the principle reason why we started accepted small PRs on GitHub. My sense is that we should move these PRs to GitLab and formally stop accepting PRs via GitHub. If there is no objection I will do this in three days. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From moritz.angermann at gmail.com Sat Jul 4 00:28:37 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Sat, 4 Jul 2020 08:28:37 +0800 Subject: A small documentation PR on github In-Reply-To: <87lfk04g40.fsf@smart-cactus.org> References: <87lfk04g40.fsf@smart-cactus.org> Message-ID: The performance of GH is still better than GL. Reading the code on GH is faster and easier to navigate than GL. This might be an artifact of my location? The GL UI feels a lot more sluggish. Though GH is doing their part with service downtimes recently as well. Making a small change on GH to a file is almost comically trivial. Press Edit, make the change, commit and open the PR. All from within the browser in a few seconds. Wasn’t this this primary motivation for allowing documentation PRs on GH? On Sat, 4 Jul 2020 at 2:18 AM, Ben Gamari wrote: > Alexander Kjeldaas writes: > > > Hi devs! > > > > I created a small documentation PR for the GHC FFI on github and noticed > > that there's another one-liner PR from May 2019 that was not merged. > > > > https://github.com/ghc/ghc/pull/260 > > https://github.com/ghc/ghc/pull/255 > > > > Just checking that simple PRs are still accepted on github. > > > An excellent point. In my mind the move to GitLab has addressed the > principle reason why we started accepted small PRs on GitHub. My sense > is that we should move these PRs to GitLab and formally stop accepting > PRs via GitHub. > > If there is no objection I will do this in three days. > > Cheers, > > - Ben > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.kjeldaas at gmail.com Sat Jul 4 06:51:36 2020 From: alexander.kjeldaas at gmail.com (Alexander Kjeldaas) Date: Sat, 4 Jul 2020 08:51:36 +0200 Subject: A small documentation PR on github In-Reply-To: References: <87lfk04g40.fsf@smart-cactus.org> Message-ID: Wrt my change, I've made a merge request on Gitlab. Absolutely no problems using that tool. Alexander On Sat, Jul 4, 2020 at 2:28 AM Moritz Angermann wrote: > The performance of GH is still better than GL. Reading the code on GH is > faster and easier to navigate than GL. This might be an artifact of my > location? The GL UI feels a lot more sluggish. Though GH is doing their > part with service downtimes recently as well. > > Making a small change on GH to a file is almost comically trivial. Press > Edit, make the change, commit and open the PR. All from within the browser > in a few seconds. Wasn’t this this primary motivation for allowing > documentation PRs on GH? > > On Sat, 4 Jul 2020 at 2:18 AM, Ben Gamari wrote: > >> Alexander Kjeldaas writes: >> >> > Hi devs! >> > >> > I created a small documentation PR for the GHC FFI on github and noticed >> > that there's another one-liner PR from May 2019 that was not merged. >> > >> > https://github.com/ghc/ghc/pull/260 >> > https://github.com/ghc/ghc/pull/255 >> > >> > Just checking that simple PRs are still accepted on github. >> > >> An excellent point. In my mind the move to GitLab has addressed the >> principle reason why we started accepted small PRs on GitHub. My sense >> is that we should move these PRs to GitLab and formally stop accepting >> PRs via GitHub. >> >> If there is no objection I will do this in three days. >> >> Cheers, >> >> - Ben >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Sun Jul 5 15:26:01 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 05 Jul 2020 11:26:01 -0400 Subject: A small documentation PR on github In-Reply-To: References: <87lfk04g40.fsf@smart-cactus.org> Message-ID: <87a70e3rwa.fsf@smart-cactus.org> Moritz Angermann writes: > The performance of GH is still better than GL. Reading the code on GH is > faster and easier to navigate than GL. This might be an artifact of my > location? The GL UI feels a lot more sluggish. Though GH is doing their > part with service downtimes recently as well. > > Making a small change on GH to a file is almost comically trivial. Press > Edit, make the change, commit and open the PR. All from within the browser > in a few seconds. Wasn’t this this primary motivation for allowing > documentation PRs on GH? > This same workflow works on GitLab. The decision to allow GitHub merge requests was made when we were still using Phabricator, where this sort of thing was significantly less convenient and required opening a Phabricator account. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From wolfgang-it at jeltsch.info Sat Jul 11 14:42:33 2020 From: wolfgang-it at jeltsch.info (Wolfgang Jeltsch) Date: Sat, 11 Jul 2020 17:42:33 +0300 Subject: Broken links in GHC =?UTF-8?Q?User=E2=80=99s?= Manual Message-ID: <3bb1769e095b662ccecfc695cead59c90a69922d.camel@jeltsch.info> Hi! When I go to https://www.haskell.org/ghc/ and click on “Documentation”, I’m taken to https://downloads.haskell.org/ghc/latest/docs/html/users_guide/ . Clicking there on the entry for Section 9.1, for example, takes me to https://downloads.haskell.org/ghc/latest/docs/html/users_guide/glasgow_exts.html , where there is no page. On the other hand, when I go to https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/index.html (which I have in my bookmarks) and then click on the entry for Section 9.1, I’m taken to https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html , where there is the expected page. Could this perhaps be fixed? Thank you. All the best, Wolfgang From adam at sandbergericsson.se Sat Jul 11 15:04:13 2020 From: adam at sandbergericsson.se (Adam Sandberg Eriksson) Date: Sat, 11 Jul 2020 16:04:13 +0100 Subject: =?UTF-8?Q?Re:_Broken_links_in_GHC_User=E2=80=99s_Manual?= In-Reply-To: <3bb1769e095b662ccecfc695cead59c90a69922d.camel@jeltsch.info> References: <3bb1769e095b662ccecfc695cead59c90a69922d.camel@jeltsch.info> Message-ID: <12038b93-ae9f-4fa9-95f1-add178ca4089@www.fastmail.com> Somewhat related: wouldn't it be better if the latest docs redirected you to the "forever" stable link for the latest release? This would also lessen possible link rot as raised in https://gitlab.haskell.org/ghc/ghc/-/issues/18404. Adam Sandberg Eriksson On Sat, 11 Jul 2020, at 15:42, Wolfgang Jeltsch wrote: > Hi! > > When I go to https://www.haskell.org/ghc/ and click on “Documentation”, > I’m taken to > > https://downloads.haskell.org/ghc/latest/docs/html/users_guide/ . > > Clicking there on the entry for Section 9.1, for example, takes me to > > > https://downloads.haskell.org/ghc/latest/docs/html/users_guide/glasgow_exts.html , > > where there is no page. On the other hand, when I go to > > https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/index.html > > (which I have in my bookmarks) and then click on the entry for > Section 9.1, I’m taken to > > > https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html , > > where there is the expected page. > > Could this perhaps be fixed? Thank you. > > All the best, > Wolfgang > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > From ben at smart-cactus.org Mon Jul 13 14:32:27 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Mon, 13 Jul 2020 10:32:27 -0400 Subject: Broken links in GHC =?utf-8?Q?User=E2=80=99s?= Manual In-Reply-To: <12038b93-ae9f-4fa9-95f1-add178ca4089@www.fastmail.com> References: <3bb1769e095b662ccecfc695cead59c90a69922d.camel@jeltsch.info> <12038b93-ae9f-4fa9-95f1-add178ca4089@www.fastmail.com> Message-ID: <87lfjnfpu2.fsf@smart-cactus.org> "Adam Sandberg Eriksson" writes: > Somewhat related: wouldn't it be better if the latest docs redirected you to the "forever" stable link for the latest release? This would also lessen possible link rot as raised in https://gitlab.haskell.org/ghc/ghc/-/issues/18404. > Yes, that is fair. It's slightly less convenient from a sysadmin perspective but I agree that it's the right thing to do. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From siddu.druid at gmail.com Mon Jul 13 18:24:04 2020 From: siddu.druid at gmail.com (Siddharth Bhat) Date: Mon, 13 Jul 2020 23:54:04 +0530 Subject: Querying Core for information about variables Message-ID: Hello, I'm trying to understand how to query information about `Var`s from a Core plugin. Consider the snippet of haskell: ``` {-# LANGUAGE MagicHash #-} import GHC.Prim fib :: Int# -> Int# fib i = case i of 0# -> i; 1# -> i; _ -> (fib i) +# (fib (i -# 1#)) main :: IO (); main = let x = fib 10# in return () ``` That compiles to the following (elided) GHC Core, dumped right after desugar: ``` Rec { fib [Occ=LoopBreaker] :: Int# -> Int# [LclId] fib = ... end Rec } Main.$trModule :: GHC.Types.Module [LclIdX] Main.$trModule = GHC.Types.Module (GHC.Types.TrNameS "main"#) (GHC.Types.TrNameS "Main"#) -- RHS size: {terms: 7, types: 3, coercions: 0, joins: 0/0} main :: IO () [LclIdX] main = case fib 10# of { __DEFAULT -> return @ IO GHC.Base.$fMonadIO @ () GHC.Tuple.() } -- RHS size: {terms: 2, types: 1, coercions: 0, joins: 0/0} :Main.main :: IO () [LclIdX] :Main.main = GHC.TopHandler.runMainIO @ () main ``` I've been using `occNameString . getOccName` to manipulate names of `Var`s from the Core module. I'm rapidly finding this insufficient, and want more information about a variable. In particular, How to I figure out: 1. When I see the Var with occurence name `fib`, that it belongs to module `Main`? 2. When I see the Var with name `main`, whether it is `Main.main` or `:Main.main`? 3. When I see the Var with name `+#`, that this is an inbuilt name? Similarly for `-#` and `()`. 4. When I see the binder $trModule, that it is added by GHC and has type `GHC.Types.Module`? 5. In general, given a Var, how do I decide where it comes from, and whether it is user-defined or something GHC defined ('wired-in' I believe is the term I am looking for)? 6. When I see a `Var`, how do I learn its type? 7. In general, is there a page that tells me how to 'query' Core/`ModGuts` from within a core plugin? Pointers on how to get this information is much appreciated. Also, pointers on "learning how to learn" --- that is, how I could have figured this out on my own / RTFMing better are also very appreciated! Thanks a lot, ~Siddharth -- https://bollu.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at sandbergericsson.se Mon Jul 13 18:35:41 2020 From: adam at sandbergericsson.se (Adam Sandberg Eriksson) Date: Mon, 13 Jul 2020 19:35:41 +0100 Subject: Querying Core for information about variables In-Reply-To: References: Message-ID: I think some of what you want can be had from the Name type, and it seems you can get a Name from a Var with varName. See https://hackage.haskell.org/package/ghc-8.10.1/docs/Name.html#g:3 for some things that Names contain. For the type of a Var it seems you could use varType (https://hackage.haskell.org/package/ghc-8.10.1/docs/Var.html#v:varType). I really recommend looking at the Haddocks, that's how I figured out what to do with Name's etc. Adam Sandberg Eriksson On Mon, 13 Jul 2020, at 19:24, Siddharth Bhat wrote: > Hello, > > I'm trying to understand how to query information about `Var`s from a > > Core plugin. Consider the snippet of haskell: > > ``` > {-# LANGUAGE MagicHash #-} > import GHC.Prim > fib :: Int# -> Int# > fib i = case i of 0# -> i; 1# -> i; _ -> (fib i) +# (fib (i -# 1#)) > main :: IO (); main = let x = fib 10# in return () > ``` > > That compiles to the following (elided) GHC Core, dumped right after desugar: > > ``` > Rec { > fib [Occ=LoopBreaker] :: Int# -> Int# > [LclId] > fib > = ... > end Rec } > > Main.$trModule :: GHC.Types.Module > [LclIdX] > Main.$trModule > = GHC.Types.Module > (GHC.Types.TrNameS "main"#) (GHC.Types.TrNameS "Main"#) > > -- RHS size: {terms: 7, types: 3, coercions: 0, joins: 0/0} > main :: IO () > [LclIdX] > main > = case fib 10# of { __DEFAULT -> > return @ IO GHC.Base.$fMonadIO @ () GHC.Tuple.() > } > > -- RHS size: {terms: 2, types: 1, coercions: 0, joins: 0/0} > :Main.main :: IO () > [LclIdX] > :Main.main = GHC.TopHandler.runMainIO @ () main > ``` > > I've been using `occNameString . getOccName` to manipulate names of `Var`s from the Core > module. I'm rapidly finding this insufficient, and want more information > about a variable. In particular, How to I figure out: > > 1. When I see the Var with occurence name `fib`, that it belongs to module `Main`? > 2. When I see the Var with name `main`, whether it is `Main.main` or `:Main.main`? > 3. When I see the Var with name `+#`, that this is an inbuilt name? Similarly > for `-#` and `()`. > 4. When I see the binder $trModule, that it is added by GHC and has type `GHC.Types.Module`? > 5. In general, given a Var, how do I decide where it comes from, and whether it is > user-defined or something GHC defined ('wired-in' I believe is the term I am > looking for)? > 6. When I see a `Var`, how do I learn its type? > 7. In general, is there a page that tells me how to 'query' Core/`ModGuts` from within a core plugin? > > Pointers on how to get this information is much appreciated. Also, pointers on > "learning how to learn" --- that is, how I could have figured this out on my own / > RTFMing better are also very appreciated! > > Thanks a lot, > ~Siddharth > > > -- > https://bollu.github.io/ > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Jul 14 10:40:27 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 14 Jul 2020 10:40:27 +0000 Subject: HEAD doesn't build. Totally stalled. Message-ID: I'm getting this failure in a clean HEAD build. Any ideas? I'm totally stalled because I can't build GHC any more. I'm using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs.... -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Jul 14 11:00:24 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 14 Jul 2020 19:00:24 +0800 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: This was my fault. Not sure why this wasn’t caught in CI. It’s due to the addition of the symbols here https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 You should be able to just comment them out. I’ll prepare a proper fix. Cheers, Moritz On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs < ghc-devs at haskell.org> wrote: > I’m getting this failure in a clean HEAD build. Any ideas? I’m totally > stalled because I can’t build GHC any more. > > I’m using Windows Subsystem for Linux (WSL). > > Help help! > > Thanks > > Simon > > /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): > RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' > > collect2: error: ld returned 1 exit status > > `cc' failed in phase `Linker'. (Exit code: 1) > > utils/iserv/ghc.mk:105: recipe for target > 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed > > make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 > > make[1]: *** Waiting for unfinished jobs.... > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Jul 14 11:06:30 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 14 Jul 2020 11:06:30 +0000 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: thanks. What specifically do I comment out? From: Moritz Angermann Sent: 14 July 2020 12:00 To: Simon Peyton Jones Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. This was my fault. Not sure why this wasn’t caught in CI. It’s due to the addition of the symbols here https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 You should be able to just comment them out. I’ll prepare a proper fix. Cheers, Moritz On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs > wrote: I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. I’m using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs.... _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Jul 14 11:13:34 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 14 Jul 2020 19:13:34 +0800 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. Replacing #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) #define RTS_SSP_SYMBOLS \ SymI_NeedsProto(__stack_chk_guard) \ SymI_NeedsProto(__stack_chk_fail) #else #define RTS_SSP_SYMBOLS #endif With just #define RTS_SSP_SYMBOLS Should do. I hope. Currently only on mobile phone :-/ Cheers, Moritz On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones wrote: > thanks. What specifically do I comment out? > > > > *From:* Moritz Angermann > *Sent:* 14 July 2020 12:00 > *To:* Simon Peyton Jones > *Cc:* ghc-devs at haskell.org > *Subject:* Re: HEAD doesn't build. Totally stalled. > > > > This was my fault. Not sure why this wasn’t caught in CI. > > It’s due to the addition of the symbols here > > > > > https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 > > > > > You should be able to just comment them out. I’ll prepare a proper fix. > > > > Cheers, > > Moritz > > > > On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs < > ghc-devs at haskell.org> wrote: > > I’m getting this failure in a clean HEAD build. Any ideas? I’m totally > stalled because I can’t build GHC any more. > > I’m using Windows Subsystem for Linux (WSL). > > Help help! > > Thanks > > Simon > > /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): > RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' > > collect2: error: ld returned 1 exit status > > `cc' failed in phase `Linker'. (Exit code: 1) > > utils/iserv/ghc.mk:105 > : > recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed > > make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 > > make[1]: *** Waiting for unfinished jobs.... > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed Jul 15 14:58:00 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 15 Jul 2020 14:58:00 +0000 Subject: Review for !2377 Haddock and GHC Message-ID: Fellow GHC devs I'd like to ask your help with reviewing a GHC patch !2377: Accumulate Haddock comments in P (#17544, #17561, #8944) It's a substantial patch to the parser, changing the way in which Haddock documentation is parsed. It moves us in a good direction; it makes us more robust for the future; Vlad has invested a lot of effort in it; and I'm sure we should land it. But it's had surprisingly little review, especially for a patch that potentially affects quite a lot of people (incl Haddock folk, IDE folk). So, would any of you be willing to review the code itself? Especially folk involved in Haddock or Haskell IDE. Early versions were not easy to understand, but Vlad has invested a lot of effort in both simplifying it and in documenting it. We'll land it regardless in a couple of weeks. Thanks! Simon Summary of the payload (taken from #17544) Currently Haddock comments are, first and foremost, comments. It's very annoying to incorporate them into the grammar. We can take advantage of an important property: adding a Haddock comment does not change the parse tree in any way other than wrapping some nodes in HsDocTy and the like (and if it does, that's a bug). Here's the design I propose instead: 1. Accumulate Haddock comments with their locations in the P monad. This logic will be very easy to add in the lexer. 2. After parsing, do a pass over the AST to associate Haddock comments with AST nodes using location info. 3. Report the leftover comments to the user as a warning or an error. This means * The happy grammar won't even have the notion of a Haddock comment. * Parser.y becomes quite a bit simpler. * Ultimately, it is conceivable we could move the logic of associating Haddock comments with AST nodes from GHC to Haddock itself A detailed overview of the implementation plan is in comments at the start of compiler/GHC/Parser/PostProcess/Haddock.hs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Wed Jul 15 23:38:31 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 15 Jul 2020 19:38:31 -0400 Subject: [ANNOUNCE] GHC 8.8.4 is now available Message-ID: <87blkgfix7.fsf@smart-cactus.org> Hello everyone, The GHC team is proud to announce the release of GHC 8.8.4. The source distribution, binary distributions, and documentation are available at https://downloads.haskell.org/~ghc/8.8.4 Release notes are also available [1]. This release fixes a handful of issues affecting 8.8.3: - Fixes a bug in process creation on Windows (#17926). Due to this fix we strongly encourage all Windows users to upgrade immediately. - Works around a Linux kernel bug in the implementation of timerfd (#18033) - Fixes a few linking issues affecting ARM - Fixes "missing interface file" error triggered by some uses of Data.Ord.Ordering (#18185) - Fixes an integer overflow in the compact-normal-form import implementation (#16992) - `configure` now accepts a `--enable-numa` flag to enable/disable `numactl` support on Linux. - Fixes potentially lost sharing due to the desugaring of left operator sections (#18151). - Fixes a build-system bug resulting in potential miscompilation by unregisteised compilers (#18024) As always, if anything looks amiss do let us know. Happy compiling! Cheers, - Ben [1] https://downloads.haskell.org/ghc/8.8.4/docs/html/users_guide/8.8.4-notes.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Thu Jul 16 03:14:12 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 15 Jul 2020 23:14:12 -0400 Subject: GHC formally discontinuing 32-bit Windows support? Message-ID: <878sfkf8xs.fsf@smart-cactus.org> tl;dr. Unless someone speaks up, GHC will formally discontinue its (currently-broken) support for 32-bit Windows in 8.12. Hi everyone, As some have noticed, recent GHC releases' support for 32-bit Windows support can be generously described as "unreliable". This has been due to a combination of platform limitations, native toolchain bugs, and a general lack of capacity within the GHC community focusing on Windows support. I won't summarise the concrete issues here (see #17961, and #17700 for the current state-of-play) but let it suffice to say that we are currently stuck due to a bug in GNU binutils. However, I was recently informed that Cygwin and msys have recently discontinued their support for 32-bit Windows. While GHC uses a toolchain from the mingw32-w64 project, it seems only a matter of time before 32-bit builds cease there as well (see [1] for a summary of the relationships between these projects). Furthermore, Microsoft itself has said that 32-bit Windows 10 releases will cease later this year. All of this suggests to me that supporting 32-bit Windows in GHC will be, at best, an up-hill battle. Even worse, it is a battle with little to gained: essentially all Intel-based Windows systems today run on 64-bit-capable systems. I know of no compelling reasons why users would opt to use 32-bit Windows in 2020. Consequently, I suggest that we should formally discontinue 32-bit Windows support in GHC 8.12. In my opinion, GHC's limited engineering capacity on Windows is better spent elsewhere. However, if there are compelling reasons why some users still rely on 32-bit Windows support (despite it being largely unusable for the last two years), please do let me know. I have been consistently surprised by the number of users who have noted the absence of 32-bit Windows builds; I would love to know why they seem to be so popular. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/surviving-windows -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Thu Jul 16 14:25:38 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 16 Jul 2020 14:25:38 +0000 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: Moritz How’s it going getting this patch committed? It’s painful manually applying a fix, but then NOT committing that to master by mistake Thanks s From: Moritz Angermann Sent: 14 July 2020 12:14 To: Simon Peyton Jones Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. Replacing #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) #define RTS_SSP_SYMBOLS \ SymI_NeedsProto(__stack_chk_guard) \ SymI_NeedsProto(__stack_chk_fail) #else #define RTS_SSP_SYMBOLS #endif With just #define RTS_SSP_SYMBOLS Should do. I hope. Currently only on mobile phone :-/ Cheers, Moritz On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones > wrote: thanks. What specifically do I comment out? From: Moritz Angermann > Sent: 14 July 2020 12:00 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. This was my fault. Not sure why this wasn’t caught in CI. It’s due to the addition of the symbols here https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 You should be able to just comment them out. I’ll prepare a proper fix. Cheers, Moritz On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs > wrote: I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. I’m using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs.... _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Thu Jul 16 14:45:19 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 16 Jul 2020 22:45:19 +0800 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones wrote: > Moritz > > How’s it going getting this patch committed? > > It’s painful manually applying a fix, but then NOT committing that to > master by mistake > > > > Thanks > > s > > > > *From:* Moritz Angermann > *Sent:* 14 July 2020 12:14 > *To:* Simon Peyton Jones > *Cc:* ghc-devs at haskell.org > *Subject:* Re: HEAD doesn't build. Totally stalled. > > > > For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and > then the RTS wants __stack_chk symbols. Which it can’t find when linking. > > > > Replacing > > #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) > > #define RTS_SSP_SYMBOLS \ > > SymI_NeedsProto(__stack_chk_guard) \ > > SymI_NeedsProto(__stack_chk_fail) > > #else > > #define RTS_SSP_SYMBOLS > > #endif > > With just > > > > #define RTS_SSP_SYMBOLS > > > > Should do. I hope. > > > > Currently only on mobile phone :-/ > > > > Cheers, > > Moritz > > > > On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones > wrote: > > thanks. What specifically do I comment out? > > > > *From:* Moritz Angermann > *Sent:* 14 July 2020 12:00 > *To:* Simon Peyton Jones > *Cc:* ghc-devs at haskell.org > *Subject:* Re: HEAD doesn't build. Totally stalled. > > > > This was my fault. Not sure why this wasn’t caught in CI. > > It’s due to the addition of the symbols here > > > > > https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 > > > > > You should be able to just comment them out. I’ll prepare a proper fix. > > > > Cheers, > > Moritz > > > > On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs < > ghc-devs at haskell.org> wrote: > > I’m getting this failure in a clean HEAD build. Any ideas? I’m totally > stalled because I can’t build GHC any more. > > I’m using Windows Subsystem for Linux (WSL). > > Help help! > > Thanks > > Simon > > /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): > RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' > > collect2: error: ld returned 1 exit status > > `cc' failed in phase `Linker'. (Exit code: 1) > > utils/iserv/ghc.mk:105 > : > recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed > > make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 > > make[1]: *** Waiting for unfinished jobs.... > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Jul 16 14:47:03 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 16 Jul 2020 14:47:03 +0000 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: I could carry out experiments for you … Regardless, some stop-gap fix would be helpuf. From: Moritz Angermann Sent: 16 July 2020 15:45 To: Ben Gamari ; Simon Peyton Jones Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones > wrote: Moritz How’s it going getting this patch committed? It’s painful manually applying a fix, but then NOT committing that to master by mistake Thanks s From: Moritz Angermann > Sent: 14 July 2020 12:14 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. Replacing #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) #define RTS_SSP_SYMBOLS \ SymI_NeedsProto(__stack_chk_guard) \ SymI_NeedsProto(__stack_chk_fail) #else #define RTS_SSP_SYMBOLS #endif With just #define RTS_SSP_SYMBOLS Should do. I hope. Currently only on mobile phone :-/ Cheers, Moritz On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones > wrote: thanks. What specifically do I comment out? From: Moritz Angermann > Sent: 14 July 2020 12:00 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. This was my fault. Not sure why this wasn’t caught in CI. It’s due to the addition of the symbols here https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 You should be able to just comment them out. I’ll prepare a proper fix. Cheers, Moritz On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs > wrote: I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. I’m using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs.... _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From m at tweag.io Thu Jul 16 14:49:44 2020 From: m at tweag.io (Mathieu Boespflug) Date: Thu, 16 Jul 2020 14:49:44 +0000 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: Perhaps back out the offending patch from master in the meantime? On Thu, Jul 16, 2020 at 16:47:03, Simon Peyton Jones < ghc-devs at haskell.org > wrote: > > > > I could carry out experiments for you … > > > > > Regardless, some stop-gap fix would be helpuf. > > > > > > > > > > *From:* Moritz Angermann < moritz. angermann@ gmail. com ( > moritz.angermann at gmail.com ) > > *Sent:* 16 July 2020 15:45 > *To:* Ben Gamari < bgamari@ gmail. com ( bgamari at gmail.com ) >; Simon > Peyton Jones < simonpj@ microsoft. com ( simonpj at microsoft.com ) > > *Cc:* ghc-devs@ haskell. org ( ghc-devs at haskell.org ) > *Subject:* Re: HEAD doesn't build. Totally stalled. > > > > > > > > > > I’ve tried to reproduce this and it turns out, I fail to. You are somehow > building the rts either with _FORTYFY_SOURCE or __SSP__, but then your > linker ends up not passing -lssp or the equivalent for your tool chain. > > > > > > > > > > > > At this point I’m tempted to add an additional ARM arch guard. While that > would be conceptually wrong, it would reduce the cases where this could go > wrong to a rarely used platform. Maybe @Ben Gamari has an idea? > > > > > > > > > > > On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones < simonpj@ microsoft. com > ( simonpj at microsoft.com ) > wrote: > > > > >> >> >> Moritz >> >> >> >> >> How’s it going getting this patch committed? >> >> >> >> >> It’s painful manually applying a fix, but then NOT committing that to >> master by mistake >> >> >> >> >> >> >> >> >> >> Thanks >> >> >> >> >> >> s >> >> >> >> >> >> >> >> >> >> *From:* Moritz Angermann < moritz. angermann@ gmail. com ( >> moritz.angermann at gmail.com ) > >> *Sent:* 14 July 2020 12:14 >> *To:* Simon Peyton Jones < simonpj@ microsoft. com ( simonpj at microsoft.com >> ) > >> *Cc:* ghc-devs@ haskell. org ( ghc-devs at haskell.org ) >> *Subject:* Re: HEAD doesn't build. Totally stalled. >> >> >> >> >> >> >> >> >> >> For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and >> then the RTS wants __stack_chk symbols. Which it can’t find when linking. >> >> >> >> >> >> >> >> >> >> >> >> Replacing >> >> >> >> #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) >> #define RTS_SSP_SYMBOLS                                \ >>       SymI_NeedsProto(__stack_chk_guard)               \ >>       SymI_NeedsProto(__stack_chk_fail) >> #else >> #define RTS_SSP_SYMBOLS >> #endif >> With just >>   >> #define RTS_SSP_SYMBOLS >> >> >> >> >> >> >> >> >> >> Should do. I hope. >> >> >> >> >> >> >> >> >> >> >> >> Currently only on mobile phone :-/ >> >> >> >> >> >> >> >> >> >> >> >> Cheers, >> >> >> >> >> >> Moritz >> >> >> >> >> >> >> >> >> >> >> On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones < simonpj@ microsoft. com >> ( simonpj at microsoft.com ) > wrote: >> >> >> >> >>> >>> >>> thanks.  What specifically do I comment out? >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *From:* Moritz Angermann < moritz. angermann@ gmail. com ( >>> moritz.angermann at gmail.com ) > >>> *Sent:* 14 July 2020 12:00 >>> *To:* Simon Peyton Jones < simonpj@ microsoft. com ( simonpj at microsoft.com >>> ) > >>> *Cc:* ghc-devs@ haskell. org ( ghc-devs at haskell.org ) >>> *Subject:* Re: HEAD doesn't build. Totally stalled. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> This was my fault. Not sure why this wasn’t caught in CI. >>> >>> >>> >>> >>> >>> It’s due to the addition of the symbols here >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> https:/ / github. com/ ghc/ ghc/ commit/ 686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 >>> ( >>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fghc%2Fghc%2Fcommit%2F686e72253aed3880268dd6858eadd8c320f09e97%23diff-03f5bc5a50fd8ae13e902782c4392c38R1159&data=02%7C01%7Csimonpj%40microsoft.com%7C608d01a0f7ee4bdd103408d82996e3b5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305075328526539&sdata=ZsFoKS2k97fO4RSeSpv%2FYgwx08l68PcFJhigZB9mbNw%3D&reserved=0 >>> ) >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> You should be able to just comment them out. I’ll prepare a proper fix. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Cheers, >>> >>> >>> >>> >>> >>> Moritz >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs < ghc-devs@ >>> haskell. org ( ghc-devs at haskell.org ) > wrote: >>> >>> >>> >>> >>>> >>>> >>>> I’m getting this failure in a clean HEAD build. Any ideas?    I’m totally >>>> stalled because I can’t build GHC any more. >>>> >>>> >>>> >>>> >>>> I’m using Windows Subsystem for Linux (WSL). >>>> >>>> >>>> >>>> >>>> Help help! >>>> >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> Simon >>>> >>>> >>>> >>>> >>>> /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): >>>> RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' >>>> >>>> >>>> >>>> >>>> collect2: error: ld returned 1 exit status >>>> >>>> >>>> >>>> >>>> `cc' failed in phase `Linker'. (Exit code: 1) >>>> >>>> >>>> >>>> >>>> utils/iserv/ ghc. mk:105 ( >>>> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fghc.mk%3A105%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C608d01a0f7ee4bdd103408d82996e3b5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305075328526539&sdata=p%2FOcggYaXjNhiBZCbDTPLNzKPgVS2RilS4rAA3R8jmU%3D&reserved=0 >>>> ) : recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' >>>> failed >>>> >>>> >>>> >>>> >>>> make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 >>>> >>>> >>>> >>>> >>>> make[1]: *** Waiting for unfinished jobs.... >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs@ haskell. org ( ghc-devs at haskell.org ) >>>> http:/ / mail. haskell. org/ cgi-bin/ mailman/ listinfo/ ghc-devs ( >>>> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C608d01a0f7ee4bdd103408d82996e3b5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305075328536535&sdata=0WwLZj6VJJWl2wwQpqpmLmksJqc%2FDeHkDqAtMx47EMg%3D&reserved=0 >>>> ) >>>> >>>> >>> >>> >>> >> >> > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@ haskell. org ( ghc-devs at haskell.org ) > http:/ / mail. haskell. org/ cgi-bin/ mailman/ listinfo/ ghc-devs ( > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs ) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Thu Jul 16 14:53:24 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Thu, 16 Jul 2020 14:53:24 +0000 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: Indeed – but it’s up to Moritz. I don’t want to back out the patch myself, in case that messes up what he’s doing. He’ll best placed to decide the least disruptive way forward S From: Mathieu Boespflug Sent: 16 July 2020 15:50 To: Simon Peyton Jones Cc: ghc-devs at haskell.org; Moritz Angermann ; Ben Gamari Subject: RE: HEAD doesn't build. Totally stalled. Perhaps back out the offending patch from master in the meantime? On Thu, Jul 16, 2020 at 16:47:03, Simon Peyton Jones > wrote: I could carry out experiments for you … Regardless, some stop-gap fix would be helpuf. From: Moritz Angermann > Sent: 16 July 2020 15:45 To: Ben Gamari >; Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones > wrote: Moritz How’s it going getting this patch committed? It’s painful manually applying a fix, but then NOT committing that to master by mistake Thanks s From: Moritz Angermann > Sent: 14 July 2020 12:14 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. Replacing #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) #define RTS_SSP_SYMBOLS \ SymI_NeedsProto(__stack_chk_guard) \ SymI_NeedsProto(__stack_chk_fail) #else #define RTS_SSP_SYMBOLS #endif With just #define RTS_SSP_SYMBOLS Should do. I hope. Currently only on mobile phone :-/ Cheers, Moritz On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones > wrote: thanks. What specifically do I comment out? From: Moritz Angermann > Sent: 14 July 2020 12:00 To: Simon Peyton Jones > Cc: ghc-devs at haskell.org Subject: Re: HEAD doesn't build. Totally stalled. This was my fault. Not sure why this wasn’t caught in CI. It’s due to the addition of the symbols here https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 You should be able to just comment them out. I’ll prepare a proper fix. Cheers, Moritz On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs > wrote: I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. I’m using Windows Subsystem for Linux (WSL). Help help! Thanks Simon /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 make[1]: *** Waiting for unfinished jobs.... _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From lonetiger at gmail.com Thu Jul 16 18:45:16 2020 From: lonetiger at gmail.com (Phyx) Date: Thu, 16 Jul 2020 19:45:16 +0100 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: But, where do you actually check for __SSP__ The guard just checks for not windows and not dynamic https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1157 shouldn't it just be checking for defined(__SSP__) instead? This check is currently only correct if the distro has turned stack protector on by default. Regards, Tamar On Thu, Jul 16, 2020 at 3:46 PM Moritz Angermann wrote: > I’ve tried to reproduce this and it turns out, I fail to. You are somehow > building the rts either with _FORTYFY_SOURCE or __SSP__, but then your > linker ends up not passing -lssp or the equivalent for your tool chain. > > At this point I’m tempted to add an additional ARM arch guard. While that > would be conceptually wrong, it would reduce the cases where this could go > wrong to a rarely used platform. Maybe @Ben Gamari has an idea? > > On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones > wrote: > >> Moritz >> >> How’s it going getting this patch committed? >> >> It’s painful manually applying a fix, but then NOT committing that to >> master by mistake >> >> >> >> Thanks >> >> s >> >> >> >> *From:* Moritz Angermann >> *Sent:* 14 July 2020 12:14 >> *To:* Simon Peyton Jones >> *Cc:* ghc-devs at haskell.org >> *Subject:* Re: HEAD doesn't build. Totally stalled. >> >> >> >> For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and >> then the RTS wants __stack_chk symbols. Which it can’t find when linking. >> >> >> >> Replacing >> >> #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) >> >> #define RTS_SSP_SYMBOLS \ >> >> SymI_NeedsProto(__stack_chk_guard) \ >> >> SymI_NeedsProto(__stack_chk_fail) >> >> #else >> >> #define RTS_SSP_SYMBOLS >> >> #endif >> >> With just >> >> >> >> #define RTS_SSP_SYMBOLS >> >> >> >> Should do. I hope. >> >> >> >> Currently only on mobile phone :-/ >> >> >> >> Cheers, >> >> Moritz >> >> >> >> On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones >> wrote: >> >> thanks. What specifically do I comment out? >> >> >> >> *From:* Moritz Angermann >> *Sent:* 14 July 2020 12:00 >> *To:* Simon Peyton Jones >> *Cc:* ghc-devs at haskell.org >> *Subject:* Re: HEAD doesn't build. Totally stalled. >> >> >> >> This was my fault. Not sure why this wasn’t caught in CI. >> >> It’s due to the addition of the symbols here >> >> >> >> >> https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 >> >> >> >> >> You should be able to just comment them out. I’ll prepare a proper fix. >> >> >> >> Cheers, >> >> Moritz >> >> >> >> On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs < >> ghc-devs at haskell.org> wrote: >> >> I’m getting this failure in a clean HEAD build. Any ideas? I’m totally >> stalled because I can’t build GHC any more. >> >> I’m using Windows Subsystem for Linux (WSL). >> >> Help help! >> >> Thanks >> >> Simon >> >> /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): >> RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' >> >> collect2: error: ld returned 1 exit status >> >> `cc' failed in phase `Linker'. (Exit code: 1) >> >> utils/iserv/ghc.mk:105 >> : >> recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed >> >> make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 >> >> make[1]: *** Waiting for unfinished jobs.... >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Fri Jul 17 02:45:37 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Fri, 17 Jul 2020 10:45:37 +0800 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: Well, we actually *do* test for __SSP__ in HEAD: https://github.com/ghc/ghc/blob/master/rts/RtsSymbols.c#L1170 Which currently lists: #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) && (defined(_FORTIFY_SOURCE) || defined(__SSP__)) But this seems to still be ill conceived. And while Simon is the only one I'm aware of, for whom this breaks we need to find a better solution. As such, we will revert the commits. Why do we do all this symbol nonsense in the RTS to begin with? It has to do with our static linker we have in GHC. Loading arbitrary archives, means we need to be able to resolve all kinds of symbols that objects might refer to. For regular dependencies this will work if the dependencies are listed in the package configuration file, the linker will know which dependencies to link. This get a bit annoying for libraries that the compiler will automagically provide. libgcc, libssp, librt, ... The solution so far was simply to have the RTS depend on these symbols, and keep a list of them around. That way when the linker built the RTS we'd get it to link all these symbols into the RTS, and we could refer to them in the linker. Essentially looking them up in the linked binary (ghc, or iserv). This is a rather tricky problem, and almost all solutions we came up with are annoying in one or more dimensions. After some discussion on IRC last night, we'll go forward trying the following solution: We'll keep a file in the lib folder (similar to the settings, llvm-targets, ...) that is essentially a lookup table of Symbol -> [Library]. If we encounter an unknown symbol, and we have it in our lookup table, we will try to load the named libraries, hoping for them to contain the symbol we are looking for. If everything fails we'll bail. For the example symbols that prompted this issue: (which are emitted when stack smashing protector hardening is enabled, which seems to be the default on most linux distributions today, which is likely why I couldn't reproduce this easily.) [("__stack_chk_guard", ["ssp"])] would tell the compiler to try to locate (through the usual library location means) the library called "ssp", if it encounters the symbol "__stack_chk_guard". Isn't this what the dynamic linker is supposed to solve? Why do we have to do all this on our own? Can't we just use the dynamic linker? Yes, and no. Yes we can use the dynamic linker, and we even do. But not all platforms have a working, or usable linker. iOS for example has a working dynamic linker, but user programs can't use it. muslc reports "Dynamic loading not supported" when calling dlopen on arm. Thus I'm reluctant to drop the static linker outright for the dynamic linker. Cheers, Moritz On Fri, Jul 17, 2020 at 2:45 AM Phyx wrote: > > But, where do you actually check for __SSP__ > > The guard just checks for not windows and not dynamic https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1157 > > shouldn't it just be checking for defined(__SSP__) instead? This check is currently only correct if the distro has turned stack protector on by default. > > > Regards, > Tamar > > On Thu, Jul 16, 2020 at 3:46 PM Moritz Angermann wrote: >> >> I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. >> >> At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? >> >> On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones wrote: >>> >>> Moritz >>> >>> How’s it going getting this patch committed? >>> >>> It’s painful manually applying a fix, but then NOT committing that to master by mistake >>> >>> >>> >>> Thanks >>> >>> s >>> >>> >>> >>> From: Moritz Angermann >>> Sent: 14 July 2020 12:14 >>> To: Simon Peyton Jones >>> Cc: ghc-devs at haskell.org >>> Subject: Re: HEAD doesn't build. Totally stalled. >>> >>> >>> >>> For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. >>> >>> >>> >>> Replacing >>> >>> #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) >>> >>> #define RTS_SSP_SYMBOLS \ >>> >>> SymI_NeedsProto(__stack_chk_guard) \ >>> >>> SymI_NeedsProto(__stack_chk_fail) >>> >>> #else >>> >>> #define RTS_SSP_SYMBOLS >>> >>> #endif >>> >>> With just >>> >>> >>> >>> #define RTS_SSP_SYMBOLS >>> >>> >>> >>> Should do. I hope. >>> >>> >>> >>> Currently only on mobile phone :-/ >>> >>> >>> >>> Cheers, >>> >>> Moritz >>> >>> >>> >>> On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones wrote: >>> >>> thanks. What specifically do I comment out? >>> >>> >>> >>> From: Moritz Angermann >>> Sent: 14 July 2020 12:00 >>> To: Simon Peyton Jones >>> Cc: ghc-devs at haskell.org >>> Subject: Re: HEAD doesn't build. Totally stalled. >>> >>> >>> >>> This was my fault. Not sure why this wasn’t caught in CI. >>> >>> It’s due to the addition of the symbols here >>> >>> >>> >>> https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 >>> >>> >>> >>> You should be able to just comment them out. I’ll prepare a proper fix. >>> >>> >>> >>> Cheers, >>> >>> Moritz >>> >>> >>> >>> On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs wrote: >>> >>> I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. >>> >>> I’m using Windows Subsystem for Linux (WSL). >>> >>> Help help! >>> >>> Thanks >>> >>> Simon >>> >>> /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' >>> >>> collect2: error: ld returned 1 exit status >>> >>> `cc' failed in phase `Linker'. (Exit code: 1) >>> >>> utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed >>> >>> make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 >>> >>> make[1]: *** Waiting for unfinished jobs.... >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From mle+hs at mega-nerd.com Fri Jul 17 04:10:33 2020 From: mle+hs at mega-nerd.com (Erik de Castro Lopo) Date: Fri, 17 Jul 2020 14:10:33 +1000 Subject: Error messages Message-ID: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> Hi all, Although it was many years ago I did spend soem time working on GHC and I do know what a thankless task it is. I made a compliant about GHC error messages on an internal Slack channel and Mortiz encouraged me to repeat it here. I am incredibly happy about the quality of error messges for older more standard parts of Haskell, probably most things in Haskell98 and Haskell2010. By way of contrast, error messages for some newer parts of Haskell are incredibly poor. In fact, for the new parts, the error rmessages are often wrong, just defaulting to error messages for older parts of Haskell. As an example (open source code in a public GH repo): src/Cardano/DbSync/StateQuery.hs:87:44: error: • Data constructor not in scope: QueryAnytime :: QueryHardFork xs0 (Interpreter xs0) -> Query (CardanoBlock TPraosStandardCrypto) (Interpreter (CardanoEras TPraosStandardCrypto)) • Perhaps you want to add ‘QueryAnytime’ to the import list in the import of ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ (src/Cardano/DbSync/StateQuery.hs:49:1-116). | 87 | queryHistoryInterpreter connInfo (point, QueryAnytime GetInterpreter) The suggestion is that I need to import `QueryAnytime` but just 20 line above I have: import Ouroboros.....Query (QueryAnytime (..), QueryHardFork (GetInterpreter)) The problem is that `QueryAnytime` is defined as a pattern synonym. I have only the tinest amount of experience using pattern synonyms and that error message is not really useful. I would like to suggest that a prerequesite for merging of new features would be that it provides good error messages and more importantly does not provide wrong or misleading error messages like the one above. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/ From simonpj at microsoft.com Fri Jul 17 07:36:28 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 17 Jul 2020 07:36:28 +0000 Subject: Error messages In-Reply-To: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> References: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> Message-ID: Thanks Erik. Error message quality is crucial, as you say. But it's a very hard thing to measure, because it's a matter of human judgement. For example, you suggest | I would like to suggest that a prerequesite for merging of new features would | be that it provides good error messages and more importantly does not provide | wrong or misleading error messages like the one above. That is an excellent goal. But I don't know how to verify whether a patch meets that goal. GHC has several thousand regression tests; if the error message changes that is part of the patch, and is subject to code review. But it's harder to review the effect on test cases we'd never thought of, like yours. The only feasible way I know is for GHC's users to submit bug reports, showing a bad error message, with a way to reproduce it in a small test case, and a suggestions for what would be a better one. That way, hopefully one of GHC's contributors will find a way to fix it. Might you do that for your example (which I do not yet understand)? That's would be really helpful. It can take a little work to distil a small test case, but well-characterised bug reports are a major way in which GHC's users can contribute directly to improving GHC's quality. Thanks Simon | -----Original Message----- | From: ghc-devs On Behalf Of Erik de Castro | Lopo | Sent: 17 July 2020 05:11 | To: ghc-devs at haskell.org | Subject: Error messages | | Hi all, | | Although it was many years ago I did spend soem time working on GHC | and I do know what a thankless task it is. I made a compliant about | GHC error messages on an internal Slack channel and Mortiz encouraged | me to repeat it here. | | I am incredibly happy about the quality of error messges for older | more standard parts of Haskell, probably most things in Haskell98 | and Haskell2010. | | By way of contrast, error messages for some newer parts of Haskell are | incredibly poor. In fact, for the new parts, the error rmessages are | often wrong, just defaulting to error messages for older parts of | Haskell. | | As an example (open source code in a public GH repo): | | src/Cardano/DbSync/StateQuery.hs:87:44: error: | • Data constructor not in scope: | QueryAnytime | :: QueryHardFork xs0 (Interpreter xs0) | -> Query | (CardanoBlock TPraosStandardCrypto) | (Interpreter (CardanoEras TPraosStandardCrypto)) | • Perhaps you want to add ‘QueryAnytime’ to the import list | in the import of | ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ | (src/Cardano/DbSync/StateQuery.hs:49:1-116). | | | 87 | queryHistoryInterpreter connInfo (point, QueryAnytime | GetInterpreter) | | The suggestion is that I need to import `QueryAnytime` but just 20 line | above I | have: | | import Ouroboros.....Query (QueryAnytime (..), QueryHardFork | (GetInterpreter)) | | The problem is that `QueryAnytime` is defined as a pattern synonym. I | have only | the tinest amount of experience using pattern synonyms and that error | message | is not really useful. | | I would like to suggest that a prerequesite for merging of new features | would | be that it provides good error messages and more importantly does not | provide | wrong or misleading error messages like the one above. | | Erik | -- | ---------------------------------------------------------------------- | Erik de Castro Lopo | https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mega | - | nerd.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7Cb97ca052f89f4363 | 73c508d82a076974%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63730555861 | 2145985&sdata=BbxTaOA9jhXbEX%2BmgZcwa0AjkZv8H58cgn0sCXgUwCU%3D&re | served=0 | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cb97ca052f89f436373c508d | 82a076974%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305558612145985 | &sdata=yPnFMY3iqnBT8%2FMQTWiVg3aE1Ya%2BWxyqfjszc3XOMLw%3D&reserve | d=0 From rae at richarde.dev Fri Jul 17 08:51:46 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Fri, 17 Jul 2020 08:51:46 +0000 Subject: Error messages In-Reply-To: References: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> Message-ID: <010f01735bf8ea01-47c55532-898e-46ab-a24b-63036ad217c2-000000@us-east-2.amazonses.com> I agree that good error messages are crucial. I also agree that it can be hard for new feature designers to anticipate all the ways things might go wrong, and you've hit several bumps here. I've filed https://gitlab.haskell.org/ghc/ghc/-/issues/18466, which observes how painful this can be and suggests concrete improvements to error messages. I think it will be easy to implement these changes. Thanks for bringing the problem up! Richard > On Jul 17, 2020, at 8:36 AM, Simon Peyton Jones via ghc-devs wrote: > > Thanks Erik. Error message quality is crucial, as you say. > > But it's a very hard thing to measure, because it's a matter of human judgement. For example, you suggest > > | I would like to suggest that a prerequesite for merging of new features would > | be that it provides good error messages and more importantly does not provide > | wrong or misleading error messages like the one above. > > That is an excellent goal. But I don't know how to verify whether a patch meets that goal. GHC has several thousand regression tests; if the error message changes that is part of the patch, and is subject to code review. But it's harder to review the effect on test cases we'd never thought of, like yours. > > The only feasible way I know is for GHC's users to submit bug reports, showing a bad error message, with a way to reproduce it in a small test case, and a suggestions for what would be a better one. That way, hopefully one of GHC's contributors will find a way to fix it. > > Might you do that for your example (which I do not yet understand)? That's would be really helpful. It can take a little work to distil a small test case, but well-characterised bug reports are a major way in which GHC's users can contribute directly to improving GHC's quality. > > Thanks > > Simon > > | -----Original Message----- > | From: ghc-devs On Behalf Of Erik de Castro > | Lopo > | Sent: 17 July 2020 05:11 > | To: ghc-devs at haskell.org > | Subject: Error messages > | > | Hi all, > | > | Although it was many years ago I did spend soem time working on GHC > | and I do know what a thankless task it is. I made a compliant about > | GHC error messages on an internal Slack channel and Mortiz encouraged > | me to repeat it here. > | > | I am incredibly happy about the quality of error messges for older > | more standard parts of Haskell, probably most things in Haskell98 > | and Haskell2010. > | > | By way of contrast, error messages for some newer parts of Haskell are > | incredibly poor. In fact, for the new parts, the error rmessages are > | often wrong, just defaulting to error messages for older parts of > | Haskell. > | > | As an example (open source code in a public GH repo): > | > | src/Cardano/DbSync/StateQuery.hs:87:44: error: > | • Data constructor not in scope: > | QueryAnytime > | :: QueryHardFork xs0 (Interpreter xs0) > | -> Query > | (CardanoBlock TPraosStandardCrypto) > | (Interpreter (CardanoEras TPraosStandardCrypto)) > | • Perhaps you want to add ‘QueryAnytime’ to the import list > | in the import of > | ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ > | (src/Cardano/DbSync/StateQuery.hs:49:1-116). > | | > | 87 | queryHistoryInterpreter connInfo (point, QueryAnytime > | GetInterpreter) > | > | The suggestion is that I need to import `QueryAnytime` but just 20 line > | above I > | have: > | > | import Ouroboros.....Query (QueryAnytime (..), QueryHardFork > | (GetInterpreter)) > | > | The problem is that `QueryAnytime` is defined as a pattern synonym. I > | have only > | the tinest amount of experience using pattern synonyms and that error > | message > | is not really useful. > | > | I would like to suggest that a prerequesite for merging of new features > | would > | be that it provides good error messages and more importantly does not > | provide > | wrong or misleading error messages like the one above. > | > | Erik > | -- > | ---------------------------------------------------------------------- > | Erik de Castro Lopo > | https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mega > | - > | nerd.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7Cb97ca052f89f4363 > | 73c508d82a076974%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63730555861 > | 2145985&sdata=BbxTaOA9jhXbEX%2BmgZcwa0AjkZv8H58cgn0sCXgUwCU%3D&re > | served=0 > | _______________________________________________ > | ghc-devs mailing list > | ghc-devs at haskell.org > | https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has > | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- > | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cb97ca052f89f436373c508d > | 82a076974%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637305558612145985 > | &sdata=yPnFMY3iqnBT8%2FMQTWiVg3aE1Ya%2BWxyqfjszc3XOMLw%3D&reserve > | d=0 > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Fri Jul 17 09:08:16 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 17 Jul 2020 05:08:16 -0400 Subject: Error messages In-Reply-To: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> References: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> Message-ID: On July 17, 2020 12:10:33 AM EDT, Erik de Castro Lopo wrote: >Hi all, > >Although it was many years ago I did spend soem time working on GHC >and I do know what a thankless task it is. I made a compliant about >GHC error messages on an internal Slack channel and Mortiz encouraged >me to repeat it here. > >I am incredibly happy about the quality of error messges for older >more standard parts of Haskell, probably most things in Haskell98 >and Haskell2010. > >By way of contrast, error messages for some newer parts of Haskell are >incredibly poor. In fact, for the new parts, the error rmessages are >often wrong, just defaulting to error messages for older parts of >Haskell. > >As an example (open source code in a public GH repo): > > src/Cardano/DbSync/StateQuery.hs:87:44: error: > • Data constructor not in scope: > QueryAnytime > :: QueryHardFork xs0 (Interpreter xs0) > -> Query > (CardanoBlock TPraosStandardCrypto) > (Interpreter (CardanoEras TPraosStandardCrypto)) > • Perhaps you want to add ‘QueryAnytime’ to the import list > in the import of > ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ > (src/Cardano/DbSync/StateQuery.hs:49:1-116). > | >87 | queryHistoryInterpreter connInfo (point, QueryAnytime >GetInterpreter) > >The suggestion is that I need to import `QueryAnytime` but just 20 line >above I >have: > >import Ouroboros.....Query (QueryAnytime (..), QueryHardFork >(GetInterpreter)) > >The problem is that `QueryAnytime` is defined as a pattern synonym. I >have only >the tinest amount of experience using pattern synonyms and that error >message >is not really useful. > I am not sure that I would call this error wrong or misleading. However, I also suspect that it could do more to help you. GHC is claiming it is looking for a data constructor because that is precisely what it is looking for: Pattern synonyms are supposed to behave identically to data constructors. It sounds to me like Ouroboros.....Query should be exporting the QueryAnytime pattern bundled with the QueryAnytime type. If this were the case then your import would work as you expect. In principle GHC could suggest this (e.g. in the case where a module exports a type `T`, separately a pattern `T`, and the user imports the former but not the latter). However, this is in a way a strange suggestion as it would be suggesting a change in a module other than the module currently being compiled. This could be quite confusing since it may reveal internal implementation details of a library that the user is not supposed to be exposed to. This is really more of a code style lintthan it is a compiler error. >I would like to suggest that a prerequesite for merging of new features >would >be that it provides good error messages and more importantly does not >provide >wrong or misleading error messages like the one above. > Indeed this is something that we consider during review. However, it is also very difficult to predict how features might fail, especially when interacting with other features. As Simon suggests, the best way to improve things here is to open tickets when you see an error that doesn't look as good as it could be. Cheers, - Ben From ryan.gl.scott at gmail.com Fri Jul 17 11:15:15 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Fri, 17 Jul 2020 07:15:15 -0400 Subject: HEAD doesn't build. Totally stalled. Message-ID: > And while Simon is the only > one I'm aware of, for whom this breaks For what it's worth, I am hitting the same issue when building GHC. Ryan S. From lonetiger at gmail.com Fri Jul 17 15:03:43 2020 From: lonetiger at gmail.com (Phyx) Date: Fri, 17 Jul 2020 16:03:43 +0100 Subject: New Windows I/O manager in GHC 8.12 Message-ID: Hi All, In case you've missed it, about 150 or so commits were committed to master yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new I/O manager that is designed for the native Windows I/O subsystem instead of relying on the broken posix-ish compatibility layer that MIO used. This is one of 3 big patches I have been working on for years now.. So before I continue on why WinIO was made I'll add a TL;DR; WinIO adds an internal API break compared to previous GHC releases. That is the internal code was modified to support a completely asynchronous I/O system. What this means is that we have to keep track of the file pointer offset which previously was done by the C runtime. This is because in async I/O you cannot assume the offset to be at any given location. What does this mean for you? Very little. If you did not use internal GHC I/O code. In particular if you haven't used Buffer, BufferIO and RawIO. If you have you will to explicitly add support for GHC 8.12+. Because FDs are a Unix concept and don't behave as you would expect on Windows, the new I/O manager also uses HANDLE instead of FD. This means that any library that has used the internal GHC Fd type won't work with WinIO. Luckily the number of libraries that have seems quite low. If you can please stick to the external Handle interface for I/O functions. The boot libraries have been updated, and in particular process *requires* the version that is shipped with GHC. Please respect the version bounds here! I will be writing a migration guide for those that need to migrate code. The amount of work is usually trivial as Base provides shims to do most of the common things you would have used Fd for. Also if I may make a plea to GHC developers.. Do not add non-trivial implementations in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add them to internal modules (GHC.xxx) and re-export them from the external modules. This allows us to avoid import cycles inside the internal modules :) -- So why WinIO? Over the years a number of hard to fix issues popped up on Windows, including proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. This also allows libraries like Brick to work on Windows without re-inventing the wheel or have to hide their I/O from the I/O manager. In order to attempt to do some of these with MIO layer upon layers of hacks were added. This means that things sometimes worked.., but when it didn't was rather unpredictable. Some of the issues were simply unfixable with MIO. I will be making some posts about how WinIO works (and also archiving them on the wiki don't worry :)) but for now some highlights: WinIO is 3 years of work, First started by Joey Hess, then picked up by Mikhail Glushenkov before landing at my feet. While the majority has been rewritten their work did provide a great jumping off point so thanks! Also thanks to Ben and AndreasK for helping me get it over the line.. As you can imagine I was exhausted by this point :). Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ commits (sorry this was the smallest we could get it while not losing some historical context) and with over 153 files changed not counting the changes to boot libraries. It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 (but only on native windows consoles, so no msys shells) and #806 which is 14 years old! WinIO is a dynamic choice, so you can switch between I/O managers using the RTS flag --io-manager=[native|posix]. On non-Windows native is the same as posix. The chosen Async interface for this implementation is using Completion Ports. The I/O manager uses a new interface added in Windows Vista called GetQueuedCompletionStatusEx which allows us to service multiple request interrupts in one go. Some highlights: * Drops Windows Vista support Vista is out of extended support as of 2017. The new minimum is Windows 7. This allows us to use much more efficient OS provided abstractions. * Replace Events and Monitor locks with much faster and efficient Conditional Variables and SlimReaderWriterLocks. * Change GHC's Buffer and I/O structs to support asynchronous operation by not relying on the OS managing File Offset. * Implement a new command line flag +RTS --io-manager=[native|posix] to control which I/O manager is used. * Implement a new Console I/O interface supporting much faster reads/writes and unicode output correctly. Also supports things like cooked input etc. * In new I/O manager if the user still has their code-page set to OEM, then we use UTF-8 by default. This allows Unicode to work correctly out of the box. * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. * Flush event logs eagerly as to not rely on finalizers running. * A lot of refactoring and more use of hsc2hs to share constants * Control aborts Ctrl+C should be a bit more reliable. * Add a new IOPort primitive that should be only used for these I/O operations. Essentially an IOPort is based on an MVar with the following major differences: - Does not allow multiple pending writes. If the port is full a second write is just discarded. - There is no deadlock avoidance guarantee. If you block on an IOPort and your Haskell application does not have any work left to do the whole application is stalled. In the threaded RTS we just continue idling, in the non-threaded rts the scheduler is blocked. * Support various optimizations in the Windows I/O manager such as skipping I/O Completion if the request finished synchronously etc. * The I/O manager is now agnostic to the handle type. i.e. There is no socket specific code in the manager. This is now all pushed to the network library. Completely de-coupling these. * Unified threaded and non-threaded I/O code. The only major difference is where event loop is driven from and that the non-threaded rts will always use a single OS thread to service requests. We cannot use more as there are no rts locks to make concurrent modifications safe. Cheers, Tamar -------------- next part -------------- An HTML attachment was scrubbed... URL: From iavor.diatchki at gmail.com Fri Jul 17 15:41:55 2020 From: iavor.diatchki at gmail.com (Iavor Diatchki) Date: Fri, 17 Jul 2020 08:41:55 -0700 Subject: Error messages In-Reply-To: References: <20200717141033.a496643784d3010a20e349df@mega-nerd.com> Message-ID: It is interesting that Eric choose this particular example, as it is one of the few types of errors where I usually find GHC's suggestions for fixes to be quite useful---I am talking about the errors where you forgot to import something (or you mistype it's name) and GHC suggests that you import it or use a slightly different name. In general, I think it's better if error messages stick to describing the problem GHC is having, and avoid recommending solutions. The reason I say this is that in general we don't know what someone is trying to do and you need that to suggest the correct fix. In particular, often when I make a mistake it is because I have the wrong understanding of something, be it an API or Haskell itself, and GHC can't know what's in my head. For example, Eric says that he imported `QueryAnitime` but the import has `(..)` after it, which in Haskell means import the *type* `QueryAnytime` and all its constructors, while the problem is that we are missing a *data constructor* or pattern synonym with that name. Now this could be due to a typo, or maybe he doesn't know how exactly that part of the language works, or maybe he was thinking of a similar constructor with a slightly different name, all of which would cause the same error but would require different fixes. One constructive improvement to this error might be to change the words "data constructor" to "data constructor or pattern synonym" as both of these live in the same namespace and they are not completely interchangable (e.g., `..` imports data constructors but not type synonyms). Otoh, I think it'd be quite hard to suggest anything much more sophisticated than what we already do, and for me personally, that particular suggestion is often useful. Iavor On Fri, Jul 17, 2020, 02:09 Ben Gamari wrote: > On July 17, 2020 12:10:33 AM EDT, Erik de Castro Lopo < > mle+hs at mega-nerd.com> wrote: > >Hi all, > > > >Although it was many years ago I did spend soem time working on GHC > >and I do know what a thankless task it is. I made a compliant about > >GHC error messages on an internal Slack channel and Mortiz encouraged > >me to repeat it here. > > > >I am incredibly happy about the quality of error messges for older > >more standard parts of Haskell, probably most things in Haskell98 > >and Haskell2010. > > > >By way of contrast, error messages for some newer parts of Haskell are > >incredibly poor. In fact, for the new parts, the error rmessages are > >often wrong, just defaulting to error messages for older parts of > >Haskell. > > > >As an example (open source code in a public GH repo): > > > > src/Cardano/DbSync/StateQuery.hs:87:44: error: > > • Data constructor not in scope: > > QueryAnytime > > :: QueryHardFork xs0 (Interpreter xs0) > > -> Query > > (CardanoBlock TPraosStandardCrypto) > > (Interpreter (CardanoEras TPraosStandardCrypto)) > > • Perhaps you want to add ‘QueryAnytime’ to the import list > > in the import of > > ‘Ouroboros.Consensus.HardFork.Combinator.Ledger.Query’ > > (src/Cardano/DbSync/StateQuery.hs:49:1-116). > > | > >87 | queryHistoryInterpreter connInfo (point, QueryAnytime > >GetInterpreter) > > > >The suggestion is that I need to import `QueryAnytime` but just 20 line > >above I > >have: > > > >import Ouroboros.....Query (QueryAnytime (..), QueryHardFork > >(GetInterpreter)) > > > >The problem is that `QueryAnytime` is defined as a pattern synonym. I > >have only > >the tinest amount of experience using pattern synonyms and that error > >message > >is not really useful. > > > > I am not sure that I would call this error wrong or misleading. However, I > also suspect that it could do more to help you. GHC is claiming it is > looking for a data constructor because that is precisely what it is looking > for: Pattern synonyms are supposed to behave identically to data > constructors. > > It sounds to me like Ouroboros.....Query should be exporting the > QueryAnytime pattern bundled with the QueryAnytime type. If this were the > case then your import would work as you expect. > > In principle GHC could suggest this (e.g. in the case where a module > exports a type `T`, separately a pattern `T`, and the user imports the > former but not the latter). However, this is in a way a strange suggestion > as it would be suggesting a change in a module other than the module > currently being compiled. This could be quite confusing since it may reveal > internal implementation details of a library that the user is not supposed > to be exposed to. > > This is really more of a code style lintthan it is a compiler error. > > >I would like to suggest that a prerequesite for merging of new features > >would > >be that it provides good error messages and more importantly does not > >provide > >wrong or misleading error messages like the one above. > > > Indeed this is something that we consider during review. However, it is > also very difficult to predict how features might fail, especially when > interacting with other features. As Simon suggests, the best way to improve > things here is to open tickets when you see an error that doesn't look as > good as it could be. > > Cheers, > > - Ben > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From krz.gogolewski at gmail.com Fri Jul 17 16:03:43 2020 From: krz.gogolewski at gmail.com (Krzysztof Gogolewski) Date: Fri, 17 Jul 2020 18:03:43 +0200 Subject: GHC 9.0? Message-ID: Hi, There is an exceptional number of changes stated for the next release. * Better pattern matching coverage detection * New windows IO manager * Linear types * Large-scale typechecker changes - Taming the Kind Inference Monster, simplified subsumption * Better register allocation, improving runtime by 0.8% according to release notes * ghc-bignum * Explicit specificity and eager instantiation * Qualified do * Lexical negation * Perhaps Quick Look will manage to land Should we call it GHC 9.0? I think the name would be deserved. From chessai1996 at gmail.com Fri Jul 17 16:07:37 2020 From: chessai1996 at gmail.com (chessai) Date: Fri, 17 Jul 2020 09:07:37 -0700 Subject: GHC 9.0? In-Reply-To: References: Message-ID: I always thought that we were waiting for -XDependentHaskell before we went to 9. That's just been my impression though; no one has has ever said that, AFAIK. Perhaps it is wrong. On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski wrote: > Hi, > > There is an exceptional number of changes stated for the next release. > > * Better pattern matching coverage detection > * New windows IO manager > * Linear types > * Large-scale typechecker changes - Taming the Kind Inference Monster, > simplified subsumption > * Better register allocation, improving runtime by 0.8% according to > release notes > * ghc-bignum > * Explicit specificity and eager instantiation > * Qualified do > * Lexical negation > * Perhaps Quick Look will manage to land > > Should we call it GHC 9.0? I think the name would be deserved. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allbery.b at gmail.com Fri Jul 17 16:11:38 2020 From: allbery.b at gmail.com (Brandon Allbery) Date: Fri, 17 Jul 2020 12:11:38 -0400 Subject: GHC 9.0? In-Reply-To: References: Message-ID: That's likely to be a fairly long wait, as i understand it. On 7/17/20, chessai wrote: > I always thought that we were waiting for -XDependentHaskell before we went > to 9. That's just been my impression though; no one has has ever said that, > AFAIK. Perhaps it is wrong. > > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski > > wrote: > >> Hi, >> >> There is an exceptional number of changes stated for the next release. >> >> * Better pattern matching coverage detection >> * New windows IO manager >> * Linear types >> * Large-scale typechecker changes - Taming the Kind Inference Monster, >> simplified subsumption >> * Better register allocation, improving runtime by 0.8% according to >> release notes >> * ghc-bignum >> * Explicit specificity and eager instantiation >> * Qualified do >> * Lexical negation >> * Perhaps Quick Look will manage to land >> >> Should we call it GHC 9.0? I think the name would be deserved. >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -- brandon s allbery kf8nh allbery.b at gmail.com From alan.zimm at gmail.com Fri Jul 17 17:08:23 2020 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Fri, 17 Jul 2020 18:08:23 +0100 Subject: GHC 9.0? In-Reply-To: References: Message-ID: I have to admit this thought had crossed my mind too. Alan On Fri, 17 Jul 2020 at 17:11, Brandon Allbery wrote: > That's likely to be a fairly long wait, as i understand it. > > On 7/17/20, chessai wrote: > > I always thought that we were waiting for -XDependentHaskell before we > went > > to 9. That's just been my impression though; no one has has ever said > that, > > AFAIK. Perhaps it is wrong. > > > > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski > > > > wrote: > > > >> Hi, > >> > >> There is an exceptional number of changes stated for the next release. > >> > >> * Better pattern matching coverage detection > >> * New windows IO manager > >> * Linear types > >> * Large-scale typechecker changes - Taming the Kind Inference Monster, > >> simplified subsumption > >> * Better register allocation, improving runtime by 0.8% according to > >> release notes > >> * ghc-bignum > >> * Explicit specificity and eager instantiation > >> * Qualified do > >> * Lexical negation > >> * Perhaps Quick Look will manage to land > >> > >> Should we call it GHC 9.0? I think the name would be deserved. > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >> > > > > > -- > brandon s allbery kf8nh > allbery.b at gmail.com > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Fri Jul 17 22:51:25 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Sat, 18 Jul 2020 06:51:25 +0800 Subject: GHC 9.0? In-Reply-To: References: Message-ID: Can’t dependent haskell be 10? On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman wrote: > I have to admit this thought had crossed my mind too. > > Alan > > On Fri, 17 Jul 2020 at 17:11, Brandon Allbery wrote: > >> That's likely to be a fairly long wait, as i understand it. >> >> On 7/17/20, chessai wrote: >> > I always thought that we were waiting for -XDependentHaskell before we >> went >> > to 9. That's just been my impression though; no one has has ever said >> that, >> > AFAIK. Perhaps it is wrong. >> > >> > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski >> > >> > wrote: >> > >> >> Hi, >> >> >> >> There is an exceptional number of changes stated for the next release. >> >> >> >> * Better pattern matching coverage detection >> >> * New windows IO manager >> >> * Linear types >> >> * Large-scale typechecker changes - Taming the Kind Inference Monster, >> >> simplified subsumption >> >> * Better register allocation, improving runtime by 0.8% according to >> >> release notes >> >> * ghc-bignum >> >> * Explicit specificity and eager instantiation >> >> * Qualified do >> >> * Lexical negation >> >> * Perhaps Quick Look will manage to land >> >> >> >> Should we call it GHC 9.0? I think the name would be deserved. >> >> _______________________________________________ >> >> ghc-devs mailing list >> >> ghc-devs at haskell.org >> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> > >> >> >> -- >> brandon s allbery kf8nh >> allbery.b at gmail.com >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Sat Jul 18 01:01:00 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 17 Jul 2020 21:01:00 -0400 Subject: GHC 9.0? In-Reply-To: References: Message-ID: On July 17, 2020 6:51:25 PM EDT, Moritz Angermann wrote: >Can’t dependent haskell be 10? > >On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman > >wrote: > >> I have to admit this thought had crossed my mind too. >> >> Alan >> >> On Fri, 17 Jul 2020 at 17:11, Brandon Allbery >wrote: >> >>> That's likely to be a fairly long wait, as i understand it. >>> >>> On 7/17/20, chessai wrote: >>> > I always thought that we were waiting for -XDependentHaskell >before we >>> went >>> > to 9. That's just been my impression though; no one has has ever >said >>> that, >>> > AFAIK. Perhaps it is wrong. >>> > >>> > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski >>> > >>> > wrote: >>> > >>> >> Hi, >>> >> >>> >> There is an exceptional number of changes stated for the next >release. >>> >> >>> >> * Better pattern matching coverage detection >>> >> * New windows IO manager >>> >> * Linear types >>> >> * Large-scale typechecker changes - Taming the Kind Inference >Monster, >>> >> simplified subsumption >>> >> * Better register allocation, improving runtime by 0.8% according >to >>> >> release notes >>> >> * ghc-bignum >>> >> * Explicit specificity and eager instantiation >>> >> * Qualified do >>> >> * Lexical negation >>> >> * Perhaps Quick Look will manage to land >>> >> >>> >> Should we call it GHC 9.0? I think the name would be deserved. >>> >> _______________________________________________ >>> >> ghc-devs mailing list >>> >> ghc-devs at haskell.org >>> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> >>> > >>> >>> >>> -- >>> brandon s allbery kf8nh >>> allbery.b at gmail.com >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> No objection from me. Indeed it has been a while since we had a supermajor bump and linear types is quite a significant feature. Cheers, - Ben From takenobu.hs at gmail.com Sat Jul 18 02:55:03 2020 From: takenobu.hs at gmail.com (Takenobu Tani) Date: Sat, 18 Jul 2020 11:55:03 +0900 Subject: GHC 9.0? In-Reply-To: References: Message-ID: There is also major change of the GHC API structure by Sylvain :) Regards, Takenobu On Sat, Jul 18, 2020 at 10:01 AM Ben Gamari wrote: > > On July 17, 2020 6:51:25 PM EDT, Moritz Angermann wrote: > >Can’t dependent haskell be 10? > > > >On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman > > > >wrote: > > > >> I have to admit this thought had crossed my mind too. > >> > >> Alan > >> > >> On Fri, 17 Jul 2020 at 17:11, Brandon Allbery > >wrote: > >> > >>> That's likely to be a fairly long wait, as i understand it. > >>> > >>> On 7/17/20, chessai wrote: > >>> > I always thought that we were waiting for -XDependentHaskell > >before we > >>> went > >>> > to 9. That's just been my impression though; no one has has ever > >said > >>> that, > >>> > AFAIK. Perhaps it is wrong. > >>> > > >>> > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski > >>> > > >>> > wrote: > >>> > > >>> >> Hi, > >>> >> > >>> >> There is an exceptional number of changes stated for the next > >release. > >>> >> > >>> >> * Better pattern matching coverage detection > >>> >> * New windows IO manager > >>> >> * Linear types > >>> >> * Large-scale typechecker changes - Taming the Kind Inference > >Monster, > >>> >> simplified subsumption > >>> >> * Better register allocation, improving runtime by 0.8% according > >to > >>> >> release notes > >>> >> * ghc-bignum > >>> >> * Explicit specificity and eager instantiation > >>> >> * Qualified do > >>> >> * Lexical negation > >>> >> * Perhaps Quick Look will manage to land > >>> >> > >>> >> Should we call it GHC 9.0? I think the name would be deserved. > >>> >> _______________________________________________ > >>> >> ghc-devs mailing list > >>> >> ghc-devs at haskell.org > >>> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >>> >> > >>> > > >>> > >>> > >>> -- > >>> brandon s allbery kf8nh > >>> allbery.b at gmail.com > >>> _______________________________________________ > >>> ghc-devs mailing list > >>> ghc-devs at haskell.org > >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >>> > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >> > > No objection from me. Indeed it has been a while since we had a supermajor bump and linear types is quite a significant feature. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From pi.boy.travis at gmail.com Sat Jul 18 05:38:13 2020 From: pi.boy.travis at gmail.com (Travis Whitaker) Date: Fri, 17 Jul 2020 22:38:13 -0700 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Hello Tamar, Just wanted to send a quick note expressing my gratitude for these improvements! My first professional work in Haskell targeted Windows platforms, and I'm sure there are many others out there who appreciate the tireless work you've done to keep GHC's Windows support chugging along. I've been kicking the tires on your WinIO branch for a few months and this is absolutely a game-changer for Windows support. Thanks! Travis On Fri, Jul 17, 2020 at 8:06 AM Phyx wrote: > > Hi All, > > In case you've missed it, about 150 or so commits were committed to master > yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new I/O > manager that is designed for the native Windows I/O subsystem instead of > relying on the broken posix-ish compatibility layer that MIO used. > > This is one of 3 big patches I have been working on for years now.. > > So before I continue on why WinIO was made I'll add a TL;DR; > > WinIO adds an internal API break compared to previous GHC releases. That is > the internal code was modified to support a completely asynchronous I/O system. > > What this means is that we have to keep track of the file pointer offset which > previously was done by the C runtime. This is because in async I/O you cannot > assume the offset to be at any given location. > > What does this mean for you? Very little. If you did not use internal GHC I/O code. > In particular if you haven't used Buffer, BufferIO and RawIO. If you have you will > to explicitly add support for GHC 8.12+. > > Because FDs are a Unix concept and don't behave as you would expect on Windows, the > new I/O manager also uses HANDLE instead of FD. This means that any library that has > used the internal GHC Fd type won't work with WinIO. Luckily the number of libraries > that have seems quite low. If you can please stick to the external Handle interface > for I/O functions. > > The boot libraries have been updated, and in particular process *requires* the version > that is shipped with GHC. Please respect the version bounds here! I will be writing > a migration guide for those that need to migrate code. The amount of work is usually > trivial as Base provides shims to do most of the common things you would have used Fd for. > > Also if I may make a plea to GHC developers.. Do not add non-trivial implementations > in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add them to internal > modules (GHC.xxx) and re-export them from the external modules. This allows us to avoid > import cycles inside the internal modules :) > > -- > > So why WinIO? Over the years a number of hard to fix issues popped up on Windows, including > proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. This also allows libraries like Brick to work on Windows without re-inventing the wheel or have to hide their I/O from the I/O manager. > > In order to attempt to do some of these with MIO layer upon layers of hacks were added. This means that things sometimes worked.., but when it didn't was rather unpredictable. Some of the issues were simply unfixable with MIO. I will be making some posts about how WinIO works (and also archiving them on the wiki don't worry :)) but for now some highlights: > > WinIO is 3 years of work, First started by Joey Hess, then picked up by Mikhail Glushenkov before landing at my feet. While the majority has been rewritten their work did provide a great jumping off point so thanks! Also thanks to Ben and AndreasK for helping me get it over the line.. As you can imagine I was exhausted by this point :). > > Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ commits (sorry this was the smallest we could get it while not losing some historical context) and with over 153 files changed not counting the changes to boot libraries. > > It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 > (but only on native windows consoles, so no msys shells) and #806 which is 14 years old! > > WinIO is a dynamic choice, so you can switch between I/O managers using the RTS flag --io-manager=[native|posix]. > > On non-Windows native is the same as posix. > > The chosen Async interface for this implementation is using Completion Ports. > > The I/O manager uses a new interface added in Windows Vista called GetQueuedCompletionStatusEx which allows us to service multiple request interrupts in one go. > > Some highlights: > > * Drops Windows Vista support > Vista is out of extended support as of 2017. The new minimum is Windows 7. This allows us to use much more efficient OS provided abstractions. > > * Replace Events and Monitor locks with much faster and efficient Conditional Variables and SlimReaderWriterLocks. > * Change GHC's Buffer and I/O structs to support asynchronous operation by not relying on the OS managing File Offset. > * Implement a new command line flag +RTS --io-manager=[native|posix] to control which I/O manager is used. > * Implement a new Console I/O interface supporting much faster reads/writes and unicode output correctly. Also supports things like cooked input etc. > * In new I/O manager if the user still has their code-page set to OEM, then we use UTF-8 by default. This allows Unicode to work correctly out of the box. > * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. > * Flush event logs eagerly as to not rely on finalizers running. > * A lot of refactoring and more use of hsc2hs to share constants > * Control aborts Ctrl+C should be a bit more reliable. > * Add a new IOPort primitive that should be only used for these I/O operations. Essentially an IOPort is based on an MVar with the following major > differences: > - Does not allow multiple pending writes. If the port is full a second write is just discarded. > - There is no deadlock avoidance guarantee. If you block on an IOPort and your Haskell application does not have any work left to do the whole application is > stalled. In the threaded RTS we just continue idling, in the non-threaded rts the scheduler is blocked. > > * Support various optimizations in the Windows I/O manager such as skipping I/O Completion if the request finished synchronously etc. > * The I/O manager is now agnostic to the handle type. i.e. There is no socket specific code in the manager. This is now all pushed to the network library. Completely de-coupling these. > * Unified threaded and non-threaded I/O code. The only major difference is where event loop is driven from and that the non-threaded rts will always use a single OS thread to service requests. We cannot use more as there are no rts locks to make concurrent modifications safe. > > Cheers, > Tamar > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From lonetiger at gmail.com Sat Jul 18 09:04:11 2020 From: lonetiger at gmail.com (Phyx) Date: Sat, 18 Jul 2020 10:04:11 +0100 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Hi Travis, Thanks for the kind words :) Hopefully once network support gets done and it's the default that all the work pays off :) Cheers, Tamar Sent from my Mobile On Sat, Jul 18, 2020, 06:38 Travis Whitaker wrote: > Hello Tamar, > > Just wanted to send a quick note expressing my gratitude for these > improvements! My first professional work in Haskell targeted Windows > platforms, and I'm sure there are many others out there who appreciate > the tireless work you've done to keep GHC's Windows support chugging > along. I've been kicking the tires on your WinIO branch for a few > months and this is absolutely a game-changer for Windows support. > > Thanks! > > Travis > > On Fri, Jul 17, 2020 at 8:06 AM Phyx wrote: > > > > Hi All, > > > > In case you've missed it, about 150 or so commits were committed to > master > > yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new > I/O > > manager that is designed for the native Windows I/O subsystem instead of > > relying on the broken posix-ish compatibility layer that MIO used. > > > > This is one of 3 big patches I have been working on for years now.. > > > > So before I continue on why WinIO was made I'll add a TL;DR; > > > > WinIO adds an internal API break compared to previous GHC releases. > That is > > the internal code was modified to support a completely asynchronous I/O > system. > > > > What this means is that we have to keep track of the file pointer offset > which > > previously was done by the C runtime. This is because in async I/O you > cannot > > assume the offset to be at any given location. > > > > What does this mean for you? Very little. If you did not use internal > GHC I/O code. > > In particular if you haven't used Buffer, BufferIO and RawIO. If you > have you will > > to explicitly add support for GHC 8.12+. > > > > Because FDs are a Unix concept and don't behave as you would expect on > Windows, the > > new I/O manager also uses HANDLE instead of FD. This means that any > library that has > > used the internal GHC Fd type won't work with WinIO. Luckily the number > of libraries > > that have seems quite low. If you can please stick to the external > Handle interface > > for I/O functions. > > > > The boot libraries have been updated, and in particular process > *requires* the version > > that is shipped with GHC. Please respect the version bounds here! I > will be writing > > a migration guide for those that need to migrate code. The amount of > work is usually > > trivial as Base provides shims to do most of the common things you would > have used Fd for. > > > > Also if I may make a plea to GHC developers.. Do not add non-trivial > implementations > > in the external exposed modules (e.g. System.xxx, Data.xxx) but rather > add them to internal > > modules (GHC.xxx) and re-export them from the external modules. This > allows us to avoid > > import cycles inside the internal modules :) > > > > -- > > > > So why WinIO? Over the years a number of hard to fix issues popped up on > Windows, including > > proper Unicode console I/O, cooked inputs, ability to cancel I/O > requests. This also allows libraries like Brick to work on Windows without > re-inventing the wheel or have to hide their I/O from the I/O manager. > > > > In order to attempt to do some of these with MIO layer upon layers of > hacks were added. This means that things sometimes worked.., but when it > didn't was rather unpredictable. Some of the issues were simply unfixable > with MIO. I will be making some posts about how WinIO works (and also > archiving them on the wiki don't worry :)) but for now some highlights: > > > > WinIO is 3 years of work, First started by Joey Hess, then picked up by > Mikhail Glushenkov before landing at my feet. While the majority has been > rewritten their work did provide a great jumping off point so thanks! Also > thanks to Ben and AndreasK for helping me get it over the line.. As you can > imagine I was exhausted by this point :). > > > > Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ > commits (sorry this was the smallest we could get it while not losing some > historical context) and with over 153 files changed not counting the > changes to boot libraries. > > > > It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, > #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, > #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 > > (but only on native windows consoles, so no msys shells) and #806 which > is 14 years old! > > > > WinIO is a dynamic choice, so you can switch between I/O managers using > the RTS flag --io-manager=[native|posix]. > > > > On non-Windows native is the same as posix. > > > > The chosen Async interface for this implementation is using Completion > Ports. > > > > The I/O manager uses a new interface added in Windows Vista called > GetQueuedCompletionStatusEx which allows us to service multiple request > interrupts in one go. > > > > Some highlights: > > > > * Drops Windows Vista support > > Vista is out of extended support as of 2017. The new minimum is > Windows 7. This allows us to use much more efficient OS provided > abstractions. > > > > * Replace Events and Monitor locks with much faster and efficient > Conditional Variables and SlimReaderWriterLocks. > > * Change GHC's Buffer and I/O structs to support asynchronous operation > by not relying on the OS managing File Offset. > > * Implement a new command line flag +RTS --io-manager=[native|posix] to > control which I/O manager is used. > > * Implement a new Console I/O interface supporting much faster > reads/writes and unicode output correctly. Also supports things like > cooked input etc. > > * In new I/O manager if the user still has their code-page set to OEM, > then we use UTF-8 by default. This allows Unicode to work correctly out of > the box. > > * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. > > * Flush event logs eagerly as to not rely on finalizers running. > > * A lot of refactoring and more use of hsc2hs to share constants > > * Control aborts Ctrl+C should be a bit more reliable. > > * Add a new IOPort primitive that should be only used for these I/O > operations. Essentially an IOPort is based on an MVar with the following > major > > differences: > > - Does not allow multiple pending writes. If the port is full a second > write is just discarded. > > - There is no deadlock avoidance guarantee. If you block on an IOPort > and your Haskell application does not have any work left to do the whole > application is > > stalled. In the threaded RTS we just continue idling, in the > non-threaded rts the scheduler is blocked. > > > > * Support various optimizations in the Windows I/O manager such as > skipping I/O Completion if the request finished synchronously etc. > > * The I/O manager is now agnostic to the handle type. i.e. There is no > socket specific code in the manager. This is now all pushed to the network > library. Completely de-coupling these. > > * Unified threaded and non-threaded I/O code. The only major difference > is where event loop is driven from and that the non-threaded rts will > always use a single OS thread to service requests. We cannot use more as > there are no rts locks to make concurrent modifications safe. > > > > Cheers, > > Tamar > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Sat Jul 18 13:49:24 2020 From: ben at well-typed.com (Ben Gamari) Date: Sat, 18 Jul 2020 09:49:24 -0400 Subject: DreamObjects outage Message-ID: <87eep97x26.fsf@smart-cactus.org> Hi everyone, It seems like our block storage provider is currently experiencing a service interruption. Consequently you may find that accessing CI logs and CI artifacts fails. I have contact the provider and hopefully the issue will be resolved shortly. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From lonetiger at gmail.com Sat Jul 18 21:32:09 2020 From: lonetiger at gmail.com (Phyx) Date: Sat, 18 Jul 2020 22:32:09 +0100 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Oops just a correction here, should be Joseph Adams instead of Joey Hess in the attribution. Regards, Tamar Sent from my Mobile On Fri, Jul 17, 2020, 16:03 Phyx wrote: > Hi All, > > In case you've missed it, about 150 or so commits were committed to master > yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new > I/O > manager that is designed for the native Windows I/O subsystem instead of > relying on the broken posix-ish compatibility layer that MIO used. > > This is one of 3 big patches I have been working on for years now.. > > So before I continue on why WinIO was made I'll add a TL;DR; > > WinIO adds an internal API break compared to previous GHC releases. That > is > the internal code was modified to support a completely asynchronous I/O > system. > > What this means is that we have to keep track of the file pointer offset > which > previously was done by the C runtime. This is because in async I/O you > cannot > assume the offset to be at any given location. > > What does this mean for you? Very little. If you did not use internal GHC > I/O code. > In particular if you haven't used Buffer, BufferIO and RawIO. If you have > you will > to explicitly add support for GHC 8.12+. > > Because FDs are a Unix concept and don't behave as you would expect on > Windows, the > new I/O manager also uses HANDLE instead of FD. This means that any > library that has > used the internal GHC Fd type won't work with WinIO. Luckily the number of > libraries > that have seems quite low. If you can please stick to the external Handle > interface > for I/O functions. > > The boot libraries have been updated, and in particular process *requires* > the version > that is shipped with GHC. Please respect the version bounds here! I will > be writing > a migration guide for those that need to migrate code. The amount of work > is usually > trivial as Base provides shims to do most of the common things you would > have used Fd for. > > Also if I may make a plea to GHC developers.. Do not add non-trivial > implementations > in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add > them to internal > modules (GHC.xxx) and re-export them from the external modules. This > allows us to avoid > import cycles inside the internal modules :) > > -- > > So why WinIO? Over the years a number of hard to fix issues popped up on > Windows, including > proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. > This also allows libraries like Brick to work on Windows without > re-inventing the wheel or have to hide their I/O from the I/O manager. > > In order to attempt to do some of these with MIO layer upon layers of > hacks were added. This means that things sometimes worked.., but when it > didn't was rather unpredictable. Some of the issues were simply unfixable > with MIO. I will be making some posts about how WinIO works (and also > archiving them on the wiki don't worry :)) but for now some highlights: > > WinIO is 3 years of work, First started by Joey Hess, then picked up by > Mikhail Glushenkov before landing at my feet. While the majority has been > rewritten their work did provide a great jumping off point so thanks! Also > thanks to Ben and AndreasK for helping me get it over the line.. As you can > imagine I was exhausted by this point :). > > Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ > commits (sorry this was the smallest we could get it while not losing some > historical context) and with over 153 files changed not counting the > changes to boot libraries. > > It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, > #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, > #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 > (but only on native windows consoles, so no msys shells) and #806 which is > 14 years old! > > WinIO is a dynamic choice, so you can switch between I/O managers using > the RTS flag --io-manager=[native|posix]. > > On non-Windows native is the same as posix. > > The chosen Async interface for this implementation is using Completion > Ports. > > The I/O manager uses a new interface added in Windows Vista called > GetQueuedCompletionStatusEx which allows us to service multiple > request interrupts in one go. > > Some highlights: > > * Drops Windows Vista support > Vista is out of extended support as of 2017. The new minimum is Windows > 7. This allows us to use much more efficient OS provided abstractions. > > * Replace Events and Monitor locks with much faster and efficient > Conditional Variables and SlimReaderWriterLocks. > * Change GHC's Buffer and I/O structs to support asynchronous operation by > not relying on the OS managing File Offset. > * Implement a new command line flag +RTS --io-manager=[native|posix] to > control which I/O manager is used. > * Implement a new Console I/O interface supporting much faster > reads/writes and unicode output correctly. Also supports things like > cooked input etc. > * In new I/O manager if the user still has their code-page set to OEM, > then we use UTF-8 by default. This allows Unicode to work correctly out of > the box. > * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. > * Flush event logs eagerly as to not rely on finalizers running. > * A lot of refactoring and more use of hsc2hs to share constants > * Control aborts Ctrl+C should be a bit more reliable. > * Add a new IOPort primitive that should be only used for these I/O > operations. Essentially an IOPort is based on an MVar with the following > major > differences: > - Does not allow multiple pending writes. If the port is full a second > write is just discarded. > - There is no deadlock avoidance guarantee. If you block on an IOPort > and your Haskell application does not have any work left to do the whole > application is > stalled. In the threaded RTS we just continue idling, in the non-threaded > rts the scheduler is blocked. > > * Support various optimizations in the Windows I/O manager such as > skipping I/O Completion if the request finished synchronously etc. > * The I/O manager is now agnostic to the handle type. i.e. There is no > socket specific code in the manager. This is now all pushed to the network > library. Completely de-coupling these. > * Unified threaded and non-threaded I/O code. The only major difference is > where event loop is driven from and that the non-threaded rts will always > use a single OS thread to service requests. We cannot use more as there are > no rts locks to make concurrent modifications safe. > > Cheers, > Tamar > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhail.glushenkov at gmail.com Sun Jul 19 08:04:41 2020 From: mikhail.glushenkov at gmail.com (Mikhail Glushenkov) Date: Sun, 19 Jul 2020 09:04:41 +0100 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Hi Tamar, Congratulations on getting WinIO merged, this is a really impressive effort. From rae at richarde.dev Sun Jul 19 11:04:21 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Sun, 19 Jul 2020 11:04:21 +0000 Subject: GHC 9.0? In-Reply-To: References: Message-ID: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> Despite having to withdraw my claim about dependent types in GHC 9.0, now does seem like a good time for the bump, for the reasons articulated in this thread. Richard > On Jul 18, 2020, at 3:55 AM, Takenobu Tani wrote: > > There is also major change of the GHC API structure by Sylvain :) > > Regards, > Takenobu > > On Sat, Jul 18, 2020 at 10:01 AM Ben Gamari wrote: >> >> On July 17, 2020 6:51:25 PM EDT, Moritz Angermann wrote: >>> Can’t dependent haskell be 10? >>> >>> On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman >>> >>> wrote: >>> >>>> I have to admit this thought had crossed my mind too. >>>> >>>> Alan >>>> >>>> On Fri, 17 Jul 2020 at 17:11, Brandon Allbery >>> wrote: >>>> >>>>> That's likely to be a fairly long wait, as i understand it. >>>>> >>>>> On 7/17/20, chessai wrote: >>>>>> I always thought that we were waiting for -XDependentHaskell >>> before we >>>>> went >>>>>> to 9. That's just been my impression though; no one has has ever >>> said >>>>> that, >>>>>> AFAIK. Perhaps it is wrong. >>>>>> >>>>>> On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski >>>>>> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> There is an exceptional number of changes stated for the next >>> release. >>>>>>> >>>>>>> * Better pattern matching coverage detection >>>>>>> * New windows IO manager >>>>>>> * Linear types >>>>>>> * Large-scale typechecker changes - Taming the Kind Inference >>> Monster, >>>>>>> simplified subsumption >>>>>>> * Better register allocation, improving runtime by 0.8% according >>> to >>>>>>> release notes >>>>>>> * ghc-bignum >>>>>>> * Explicit specificity and eager instantiation >>>>>>> * Qualified do >>>>>>> * Lexical negation >>>>>>> * Perhaps Quick Look will manage to land >>>>>>> >>>>>>> Should we call it GHC 9.0? I think the name would be deserved. >>>>>>> _______________________________________________ >>>>>>> ghc-devs mailing list >>>>>>> ghc-devs at haskell.org >>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> brandon s allbery kf8nh >>>>> allbery.b at gmail.com >>>>> _______________________________________________ >>>>> ghc-devs mailing list >>>>> ghc-devs at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >> >> No objection from me. Indeed it has been a while since we had a supermajor bump and linear types is quite a significant feature. >> >> Cheers, >> >> - Ben >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From a.pelenitsyn at gmail.com Sun Jul 19 21:47:38 2020 From: a.pelenitsyn at gmail.com (Artem Pelenitsyn) Date: Sun, 19 Jul 2020 17:47:38 -0400 Subject: GHC 9.0? In-Reply-To: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> References: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> Message-ID: Does Quick Look still have chances to make it into the next release? It'd be fascinating if the major version bump got both linear and impredicative types! -- Best, Artem On Sun, 19 Jul 2020 at 07:04, Richard Eisenberg wrote: > Despite having to withdraw my claim about dependent types in GHC 9.0, now > does seem like a good time for the bump, for the reasons articulated in > this thread. > > Richard > > > On Jul 18, 2020, at 3:55 AM, Takenobu Tani > wrote: > > > > There is also major change of the GHC API structure by Sylvain :) > > > > Regards, > > Takenobu > > > > On Sat, Jul 18, 2020 at 10:01 AM Ben Gamari > wrote: > >> > >> On July 17, 2020 6:51:25 PM EDT, Moritz Angermann < > moritz.angermann at gmail.com> wrote: > >>> Can’t dependent haskell be 10? > >>> > >>> On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman > >>> > >>> wrote: > >>> > >>>> I have to admit this thought had crossed my mind too. > >>>> > >>>> Alan > >>>> > >>>> On Fri, 17 Jul 2020 at 17:11, Brandon Allbery > >>> wrote: > >>>> > >>>>> That's likely to be a fairly long wait, as i understand it. > >>>>> > >>>>> On 7/17/20, chessai wrote: > >>>>>> I always thought that we were waiting for -XDependentHaskell > >>> before we > >>>>> went > >>>>>> to 9. That's just been my impression though; no one has has ever > >>> said > >>>>> that, > >>>>>> AFAIK. Perhaps it is wrong. > >>>>>> > >>>>>> On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski > >>>>>> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> There is an exceptional number of changes stated for the next > >>> release. > >>>>>>> > >>>>>>> * Better pattern matching coverage detection > >>>>>>> * New windows IO manager > >>>>>>> * Linear types > >>>>>>> * Large-scale typechecker changes - Taming the Kind Inference > >>> Monster, > >>>>>>> simplified subsumption > >>>>>>> * Better register allocation, improving runtime by 0.8% according > >>> to > >>>>>>> release notes > >>>>>>> * ghc-bignum > >>>>>>> * Explicit specificity and eager instantiation > >>>>>>> * Qualified do > >>>>>>> * Lexical negation > >>>>>>> * Perhaps Quick Look will manage to land > >>>>>>> > >>>>>>> Should we call it GHC 9.0? I think the name would be deserved. > >>>>>>> _______________________________________________ > >>>>>>> ghc-devs mailing list > >>>>>>> ghc-devs at haskell.org > >>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> brandon s allbery kf8nh > >>>>> allbery.b at gmail.com > >>>>> _______________________________________________ > >>>>> ghc-devs mailing list > >>>>> ghc-devs at haskell.org > >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >>>>> > >>>> _______________________________________________ > >>>> ghc-devs mailing list > >>>> ghc-devs at haskell.org > >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >>>> > >> > >> No objection from me. Indeed it has been a while since we had a > supermajor bump and linear types is quite a significant feature. > >> > >> Cheers, > >> > >> - Ben > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lonetiger at gmail.com Mon Jul 20 00:45:53 2020 From: lonetiger at gmail.com (Phyx) Date: Mon, 20 Jul 2020 01:45:53 +0100 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Hi Mikhail, Thanks! And thanks for the jump start :) Cheers, Tamar Sent from my Mobile On Sun, Jul 19, 2020, 09:05 Mikhail Glushenkov wrote: > Hi Tamar, > > Congratulations on getting WinIO merged, this is a really impressive > effort. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eleventynine at gmail.com Mon Jul 20 02:16:41 2020 From: eleventynine at gmail.com (Mike Ledger) Date: Mon, 20 Jul 2020 12:16:41 +1000 Subject: GHC 9.0? In-Reply-To: References: Message-ID: I think we can skip straight to 100 for DependentHaskell. On Sat, Jul 18, 2020 at 8:52 AM Moritz Angermann wrote: > Can’t dependent haskell be 10? > > On Sat, 18 Jul 2020 at 1:09 AM, Alan & Kim Zimmerman > wrote: > >> I have to admit this thought had crossed my mind too. >> >> Alan >> >> On Fri, 17 Jul 2020 at 17:11, Brandon Allbery >> wrote: >> >>> That's likely to be a fairly long wait, as i understand it. >>> >>> On 7/17/20, chessai wrote: >>> > I always thought that we were waiting for -XDependentHaskell before we >>> went >>> > to 9. That's just been my impression though; no one has has ever said >>> that, >>> > AFAIK. Perhaps it is wrong. >>> > >>> > On Fri, Jul 17, 2020, 9:04 AM Krzysztof Gogolewski >>> > >>> > wrote: >>> > >>> >> Hi, >>> >> >>> >> There is an exceptional number of changes stated for the next release. >>> >> >>> >> * Better pattern matching coverage detection >>> >> * New windows IO manager >>> >> * Linear types >>> >> * Large-scale typechecker changes - Taming the Kind Inference Monster, >>> >> simplified subsumption >>> >> * Better register allocation, improving runtime by 0.8% according to >>> >> release notes >>> >> * ghc-bignum >>> >> * Explicit specificity and eager instantiation >>> >> * Qualified do >>> >> * Lexical negation >>> >> * Perhaps Quick Look will manage to land >>> >> >>> >> Should we call it GHC 9.0? I think the name would be deserved. >>> >> _______________________________________________ >>> >> ghc-devs mailing list >>> >> ghc-devs at haskell.org >>> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> >>> > >>> >>> >>> -- >>> brandon s allbery kf8nh >>> allbery.b at gmail.com >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Mon Jul 20 07:46:06 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Mon, 20 Jul 2020 08:46:06 +0100 Subject: Unmerged Patch: 3358 Message-ID: Hi, My patch 3358 needs to get merged before the 8.12 fork. When I finished it (in May), it passed CI and after this point I lacked time to work on it further. Now I have asked 4 times for this patch to get merged and it is still open. The GHC proposal for this patch already took an extortionate amount of time to get accepted. Please can we close this chapter by merging the patch. Cheers, Matt From simonpj at microsoft.com Mon Jul 20 08:08:21 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 20 Jul 2020 08:08:21 +0000 Subject: Unmerged Patch: 3358 In-Reply-To: References: Message-ID: Matthew It looks from https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3358 as if it was blocked on something to do with 'text'. Is that unblocked now? The MR also says "Fast forward merge is not possible". So it sounds as if the steps are: * Check that the change to text, whatever that is, has been done, and fix the text submodule commit on the MR * Rebase * Assign to Marge. If you get stuck with that, do yell. Simon | -----Original Message----- | From: ghc-devs On Behalf Of Matthew | Pickering | Sent: 20 July 2020 08:46 | To: GHC developers | Subject: Unmerged Patch: 3358 | | Hi, | | My patch 3358 needs to get merged before the 8.12 fork. | | When I finished it (in May), it passed CI and after this point I | lacked time to work on it further. Now I have asked 4 times for this | patch to get merged and it is still open. | | The GHC proposal for this patch already took an extortionate amount of | time to get accepted. Please can we close this chapter by merging the | patch. | | Cheers, | | Matt | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.has | kell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc- | devs&data=02%7C01%7Csimonpj%40microsoft.com%7Cc2cb1e33865c4972eaaf08d | 82c810934%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637308279999004717 | &sdata=TZc9lHJMlijCfhhhqWSLk8o46z5tosFz91LWAy9h1b8%3D&reserved=0 From slyich at gmail.com Mon Jul 20 08:28:02 2020 From: slyich at gmail.com (Sergei Trofimovich) Date: Mon, 20 Jul 2020 09:28:02 +0100 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: References: Message-ID: <20200720092750.38941512@sf> On Fri, 17 Jul 2020 10:45:37 +0800 Moritz Angermann wrote: > Well, we actually *do* test for __SSP__ in HEAD: > https://github.com/ghc/ghc/blob/master/rts/RtsSymbols.c#L1170 > Which currently lists: > #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) && > (defined(_FORTIFY_SOURCE) || defined(__SSP__)) I believe it's a https://gitlab.haskell.org/ghc/ghc/-/issues/18442 It breaks for me as well. It triggers if one has gcc compiler with any of 2 properties: 1. gcc is built with --enable-default-ssp (sets __SSP__ for all compilations) 2. gcc defaults to _FORTIFY_SOURCE Note that presence or absence of __stack_chk_guard is indicated by neither of these and instead is present when gcc is built with --enable-libssp (use gcc's __stack_* functions instead gcc's direct TLS instructions with one glibc fallback.) Gentoo does both [1.] and [2.] by default. I believe Debian does at least [2.] by default. I'm surprised gitlab presubmit merge did not detect the build breakage. What do macros [1] and [2.] mean for glibc-linux: - _FORTIFY_SOURCE only affects glibc headers to change memcpy() calls to memcpy_chk() to add overflow checks. It does not affect symbol exports available by libc. __stack_* symbols are always present. Parts of libc or other libraries we link ghc with coult already call __stack_* function as they could already be built with _FORTIFY_SOURCE. Regardless of how ghc is being built: with _FORTIFY_SOURCE or without. - __SSP__ indicates code generation of stack canary placement by gcc (-fstack-protector-* options, or default override with gcc's --enable-default-ssp) If target is not a gcc's libssp target (a.k.a. --disable-libssp), a default for all linux-glibc targets) then gcc never uses -lssp and uses gcc's builtin instructions instead of __stack_chk_guard helpers. In this mode __stack_chk_guard is not present in any libraries installed by gcc or glibc. The only symbol provided by glibc is __stack_chk_fail (which arguably should not be exposed at all as it's an unusual contract between glibc/gcc: https://gcc.gnu.org/PR93509) --enable-libssp for gcc does bring in __stack_chk_guard. Library is present and could use __stack_chk_guard in libraries ghc depends on regardless of -fstack-protector-* options used to build ghc. I believe --enable-libssp is used only on mingw. What I'm trying to say is that presence of __stack_chk_guard is orthogonal to either __SSP__ define or _FORTIFY_SOURCE ghc uses today.. It's rather a function of how gcc toolchain was built: --enable-libssp or not. > But this seems to still be ill conceived. And while Simon is the only > one I'm aware of, for whom this breaks we need to find a better > solution. As such, we will revert the commits. > > Why do we do all this symbol nonsense in the RTS to begin with? It > has to do with our static linker we have in GHC. Loading arbitrary > archives, means we need to be able to resolve all kinds of symbols > that objects might refer to. For regular dependencies this will work > if the dependencies are listed in the package configuration file, the > linker will know which dependencies to link. This get a bit annoying > for libraries that the compiler will automagically provide. libgcc, > libssp, librt, ... > > The solution so far was simply to have the RTS depend on these > symbols, and keep a list of them around. That way when the linker > built the RTS we'd get it to link all these symbols into the RTS, and > we could refer to them in the linker. Essentially looking them up in > the linked binary (ghc, or iserv). > > This is a rather tricky problem, and almost all solutions we came up > with are annoying in one or more dimensions. After some discussion on > IRC last night, we'll go forward trying the following solution: > > We'll keep a file in the lib folder (similar to the settings, > llvm-targets, ...) that is essentially a lookup table of Symbol -> > [Library]. If we encounter an unknown symbol, and we have it in our > lookup table, we will try to load the named libraries, hoping for them > to contain the symbol we are looking for. If everything fails we'll > bail. > > For the example symbols that prompted this issue: (which are emitted > when stack smashing protector hardening is enabled, which seems to be > the default on most linux distributions today, which is likely why I > couldn't reproduce this easily.) > > [("__stack_chk_guard", ["ssp"])] > > would tell the compiler to try to locate (through the usual library > location means) the library called "ssp", if it encounters the symbol > "__stack_chk_guard". > > Isn't this what the dynamic linker is supposed to solve? Why do we > have to do all this on our own? Can't we just use the dynamic linker? > Yes, and no. Yes we can use the dynamic linker, and we even do. But > not all platforms have a working, or usable linker. iOS for example > has a working dynamic linker, but user programs can't use it. muslc > reports "Dynamic loading not supported" when calling dlopen on arm. > > Thus I'm reluctant to drop the static linker outright for the dynamic linker. > > Cheers, > Moritz > > On Fri, Jul 17, 2020 at 2:45 AM Phyx wrote: > > > > But, where do you actually check for __SSP__ > > > > The guard just checks for not windows and not dynamic https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1157 > > > > shouldn't it just be checking for defined(__SSP__) instead? This check is currently only correct if the distro has turned stack protector on by default. > > > > > > Regards, > > Tamar > > > > On Thu, Jul 16, 2020 at 3:46 PM Moritz Angermann wrote: > >> > >> I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. > >> > >> At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? > >> > >> On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones wrote: > >>> > >>> Moritz > >>> > >>> How’s it going getting this patch committed? > >>> > >>> It’s painful manually applying a fix, but then NOT committing that to master by mistake > >>> > >>> > >>> > >>> Thanks > >>> > >>> s > >>> > >>> > >>> > >>> From: Moritz Angermann > >>> Sent: 14 July 2020 12:14 > >>> To: Simon Peyton Jones > >>> Cc: ghc-devs at haskell.org > >>> Subject: Re: HEAD doesn't build. Totally stalled. > >>> > >>> > >>> > >>> For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. > >>> > >>> > >>> > >>> Replacing > >>> > >>> #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) > >>> > >>> #define RTS_SSP_SYMBOLS \ > >>> > >>> SymI_NeedsProto(__stack_chk_guard) \ > >>> > >>> SymI_NeedsProto(__stack_chk_fail) > >>> > >>> #else > >>> > >>> #define RTS_SSP_SYMBOLS > >>> > >>> #endif > >>> > >>> With just > >>> > >>> > >>> > >>> #define RTS_SSP_SYMBOLS > >>> > >>> > >>> > >>> Should do. I hope. > >>> > >>> > >>> > >>> Currently only on mobile phone :-/ > >>> > >>> > >>> > >>> Cheers, > >>> > >>> Moritz > >>> > >>> > >>> > >>> On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones wrote: > >>> > >>> thanks. What specifically do I comment out? > >>> > >>> > >>> > >>> From: Moritz Angermann > >>> Sent: 14 July 2020 12:00 > >>> To: Simon Peyton Jones > >>> Cc: ghc-devs at haskell.org > >>> Subject: Re: HEAD doesn't build. Totally stalled. > >>> > >>> > >>> > >>> This was my fault. Not sure why this wasn’t caught in CI. > >>> > >>> It’s due to the addition of the symbols here > >>> > >>> > >>> > >>> https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 > >>> > >>> > >>> > >>> You should be able to just comment them out. I’ll prepare a proper fix. > >>> > >>> > >>> > >>> Cheers, > >>> > >>> Moritz > >>> > >>> > >>> > >>> On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs wrote: > >>> > >>> I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. > >>> > >>> I’m using Windows Subsystem for Linux (WSL). > >>> > >>> Help help! > >>> > >>> Thanks > >>> > >>> Simon > >>> > >>> /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' > >>> > >>> collect2: error: ld returned 1 exit status > >>> > >>> `cc' failed in phase `Linker'. (Exit code: 1) > >>> > >>> utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed > >>> > >>> make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 > >>> > >>> make[1]: *** Waiting for unfinished jobs.... > >>> > >>> _______________________________________________ > >>> ghc-devs mailing list > >>> ghc-devs at haskell.org > >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > >> > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- Sergei -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 981 bytes Desc: Цифровая подпись OpenPGP URL: From moritz.angermann at gmail.com Mon Jul 20 08:35:10 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Mon, 20 Jul 2020 16:35:10 +0800 Subject: HEAD doesn't build. Totally stalled. In-Reply-To: <20200720092750.38941512@sf> References: <20200720092750.38941512@sf> Message-ID: Ther revert MR is here: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3714 It's kind of ironic that it's stuck in CI limbo, whereas the initial MR wasn't. > I'm surprised gitlab presubmit merge did not detect the build breakage. So am I! As laid out, I believe a better solution is to have a mapping of symbols to potential carrying libraries, and have GHC know about that, when the linker tries to link arbitrary objects and encounters those symbols. Another strategy that Tamar employed to great success on the windows side, is to just increase the set of libraries GHC tries to load by default, and thus get rid of the annoying list of symbols in the RTS. I hope the above MR will pass now (after another rebase); and I can find some time to implement a better solution soon. Cheers, Moritz On Mon, Jul 20, 2020 at 4:28 PM Sergei Trofimovich wrote: > > On Fri, 17 Jul 2020 10:45:37 +0800 > Moritz Angermann wrote: > > > Well, we actually *do* test for __SSP__ in HEAD: > > https://github.com/ghc/ghc/blob/master/rts/RtsSymbols.c#L1170 > > Which currently lists: > > #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) && > > (defined(_FORTIFY_SOURCE) || defined(__SSP__)) > > I believe it's a https://gitlab.haskell.org/ghc/ghc/-/issues/18442 > > It breaks for me as well. > > It triggers if one has gcc compiler with any of 2 properties: > > 1. gcc is built with --enable-default-ssp (sets __SSP__ for all compilations) > 2. gcc defaults to _FORTIFY_SOURCE > > Note that presence or absence of __stack_chk_guard is indicated > by neither of these and instead is present when gcc is built with > --enable-libssp (use gcc's __stack_* functions instead gcc's direct TLS > instructions with one glibc fallback.) > > Gentoo does both [1.] and [2.] by default. I believe Debian does at least > [2.] by default. I'm surprised gitlab presubmit merge did not detect the > build breakage. > > What do macros [1] and [2.] mean for glibc-linux: > > - _FORTIFY_SOURCE only affects glibc headers to change memcpy() > calls to memcpy_chk() to add overflow checks. It does not affect > symbol exports available by libc. __stack_* symbols are always present. > Parts of libc or other libraries we link ghc with coult already call __stack_* > function as they could already be built with _FORTIFY_SOURCE. Regardless > of how ghc is being built: with _FORTIFY_SOURCE or without. > > - __SSP__ indicates code generation of stack canary placement by gcc > (-fstack-protector-* options, or default override with gcc's --enable-default-ssp) > > If target is not a gcc's libssp target (a.k.a. --disable-libssp), a default for all > linux-glibc targets) then gcc never uses -lssp and uses gcc's builtin instructions > instead of __stack_chk_guard helpers. In this mode __stack_chk_guard is not > present in any libraries installed by gcc or glibc. The only symbol provided by glibc > is __stack_chk_fail (which arguably should not be exposed at all as it's an > unusual contract between glibc/gcc: https://gcc.gnu.org/PR93509) > > --enable-libssp for gcc does bring in __stack_chk_guard. Library is present and could > use __stack_chk_guard in libraries ghc depends on regardless of > -fstack-protector-* options used to build ghc. I believe --enable-libssp is used only > on mingw. > > What I'm trying to say is that presence of __stack_chk_guard is orthogonal > to either __SSP__ define or _FORTIFY_SOURCE ghc uses today.. > > It's rather a function of how gcc toolchain was built: --enable-libssp or not. > > > But this seems to still be ill conceived. And while Simon is the only > > one I'm aware of, for whom this breaks we need to find a better > > solution. As such, we will revert the commits. > > > > Why do we do all this symbol nonsense in the RTS to begin with? It > > has to do with our static linker we have in GHC. Loading arbitrary > > archives, means we need to be able to resolve all kinds of symbols > > that objects might refer to. For regular dependencies this will work > > if the dependencies are listed in the package configuration file, the > > linker will know which dependencies to link. This get a bit annoying > > for libraries that the compiler will automagically provide. libgcc, > > libssp, librt, ... > > > > The solution so far was simply to have the RTS depend on these > > symbols, and keep a list of them around. That way when the linker > > built the RTS we'd get it to link all these symbols into the RTS, and > > we could refer to them in the linker. Essentially looking them up in > > the linked binary (ghc, or iserv). > > > > This is a rather tricky problem, and almost all solutions we came up > > with are annoying in one or more dimensions. After some discussion on > > IRC last night, we'll go forward trying the following solution: > > > > We'll keep a file in the lib folder (similar to the settings, > > llvm-targets, ...) that is essentially a lookup table of Symbol -> > > [Library]. If we encounter an unknown symbol, and we have it in our > > lookup table, we will try to load the named libraries, hoping for them > > to contain the symbol we are looking for. If everything fails we'll > > bail. > > > > For the example symbols that prompted this issue: (which are emitted > > when stack smashing protector hardening is enabled, which seems to be > > the default on most linux distributions today, which is likely why I > > couldn't reproduce this easily.) > > > > [("__stack_chk_guard", ["ssp"])] > > > > would tell the compiler to try to locate (through the usual library > > location means) the library called "ssp", if it encounters the symbol > > "__stack_chk_guard". > > > > Isn't this what the dynamic linker is supposed to solve? Why do we > > have to do all this on our own? Can't we just use the dynamic linker? > > Yes, and no. Yes we can use the dynamic linker, and we even do. But > > not all platforms have a working, or usable linker. iOS for example > > has a working dynamic linker, but user programs can't use it. muslc > > reports "Dynamic loading not supported" when calling dlopen on arm. > > > > Thus I'm reluctant to drop the static linker outright for the dynamic linker. > > > > Cheers, > > Moritz > > > > On Fri, Jul 17, 2020 at 2:45 AM Phyx wrote: > > > > > > But, where do you actually check for __SSP__ > > > > > > The guard just checks for not windows and not dynamic https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1157 > > > > > > shouldn't it just be checking for defined(__SSP__) instead? This check is currently only correct if the distro has turned stack protector on by default. > > > > > > > > > Regards, > > > Tamar > > > > > > On Thu, Jul 16, 2020 at 3:46 PM Moritz Angermann wrote: > > >> > > >> I’ve tried to reproduce this and it turns out, I fail to. You are somehow building the rts either with _FORTYFY_SOURCE or __SSP__, but then your linker ends up not passing -lssp or the equivalent for your tool chain. > > >> > > >> At this point I’m tempted to add an additional ARM arch guard. While that would be conceptually wrong, it would reduce the cases where this could go wrong to a rarely used platform. Maybe @Ben Gamari has an idea? > > >> > > >> On Thu, 16 Jul 2020 at 10:25 PM, Simon Peyton Jones wrote: > > >>> > > >>> Moritz > > >>> > > >>> How’s it going getting this patch committed? > > >>> > > >>> It’s painful manually applying a fix, but then NOT committing that to master by mistake > > >>> > > >>> > > >>> > > >>> Thanks > > >>> > > >>> s > > >>> > > >>> > > >>> > > >>> From: Moritz Angermann > > >>> Sent: 14 July 2020 12:14 > > >>> To: Simon Peyton Jones > > >>> Cc: ghc-devs at haskell.org > > >>> Subject: Re: HEAD doesn't build. Totally stalled. > > >>> > > >>> > > >>> > > >>> For some reason, you end up in the defined RTS_SSP_SYMBOLS, I believe and then the RTS wants __stack_chk symbols. Which it can’t find when linking. > > >>> > > >>> > > >>> > > >>> Replacing > > >>> > > >>> #if !defined(mingw32_HOST_OS) && !defined(DYNAMIC) > > >>> > > >>> #define RTS_SSP_SYMBOLS \ > > >>> > > >>> SymI_NeedsProto(__stack_chk_guard) \ > > >>> > > >>> SymI_NeedsProto(__stack_chk_fail) > > >>> > > >>> #else > > >>> > > >>> #define RTS_SSP_SYMBOLS > > >>> > > >>> #endif > > >>> > > >>> With just > > >>> > > >>> > > >>> > > >>> #define RTS_SSP_SYMBOLS > > >>> > > >>> > > >>> > > >>> Should do. I hope. > > >>> > > >>> > > >>> > > >>> Currently only on mobile phone :-/ > > >>> > > >>> > > >>> > > >>> Cheers, > > >>> > > >>> Moritz > > >>> > > >>> > > >>> > > >>> On Tue, 14 Jul 2020 at 7:06 PM, Simon Peyton Jones wrote: > > >>> > > >>> thanks. What specifically do I comment out? > > >>> > > >>> > > >>> > > >>> From: Moritz Angermann > > >>> Sent: 14 July 2020 12:00 > > >>> To: Simon Peyton Jones > > >>> Cc: ghc-devs at haskell.org > > >>> Subject: Re: HEAD doesn't build. Totally stalled. > > >>> > > >>> > > >>> > > >>> This was my fault. Not sure why this wasn’t caught in CI. > > >>> > > >>> It’s due to the addition of the symbols here > > >>> > > >>> > > >>> > > >>> https://github.com/ghc/ghc/commit/686e72253aed3880268dd6858eadd8c320f09e97#diff-03f5bc5a50fd8ae13e902782c4392c38R1159 > > >>> > > >>> > > >>> > > >>> You should be able to just comment them out. I’ll prepare a proper fix. > > >>> > > >>> > > >>> > > >>> Cheers, > > >>> > > >>> Moritz > > >>> > > >>> > > >>> > > >>> On Tue, 14 Jul 2020 at 6:41 PM, Simon Peyton Jones via ghc-devs wrote: > > >>> > > >>> I’m getting this failure in a clean HEAD build. Any ideas? I’m totally stalled because I can’t build GHC any more. > > >>> > > >>> I’m using Windows Subsystem for Linux (WSL). > > >>> > > >>> Help help! > > >>> > > >>> Thanks > > >>> > > >>> Simon > > >>> > > >>> /home/simonpj/code/HEAD-9/rts/dist/build/libHSrts_thr_p.a(RtsSymbols.thr_p_o): RtsSymbols.c:rtsSyms: error: undefined reference to '__stack_chk_guard' > > >>> > > >>> collect2: error: ld returned 1 exit status > > >>> > > >>> `cc' failed in phase `Linker'. (Exit code: 1) > > >>> > > >>> utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed > > >>> > > >>> make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 > > >>> > > >>> make[1]: *** Waiting for unfinished jobs.... > > >>> > > >>> _______________________________________________ > > >>> ghc-devs mailing list > > >>> ghc-devs at haskell.org > > >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > >> > > >> _______________________________________________ > > >> ghc-devs mailing list > > >> ghc-devs at haskell.org > > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > -- > > Sergei From simonpj at microsoft.com Mon Jul 20 14:28:24 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 20 Jul 2020 14:28:24 +0000 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Tamar, I salute you! This is a big piece of work – thank you! Simon From: ghc-devs On Behalf Of Phyx Sent: 17 July 2020 16:04 To: ghc-devs at haskell.org Devs Subject: New Windows I/O manager in GHC 8.12 Hi All, In case you've missed it, about 150 or so commits were committed to master yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new I/O manager that is designed for the native Windows I/O subsystem instead of relying on the broken posix-ish compatibility layer that MIO used. This is one of 3 big patches I have been working on for years now.. So before I continue on why WinIO was made I'll add a TL;DR; WinIO adds an internal API break compared to previous GHC releases. That is the internal code was modified to support a completely asynchronous I/O system. What this means is that we have to keep track of the file pointer offset which previously was done by the C runtime. This is because in async I/O you cannot assume the offset to be at any given location. What does this mean for you? Very little. If you did not use internal GHC I/O code. In particular if you haven't used Buffer, BufferIO and RawIO. If you have you will to explicitly add support for GHC 8.12+. Because FDs are a Unix concept and don't behave as you would expect on Windows, the new I/O manager also uses HANDLE instead of FD. This means that any library that has used the internal GHC Fd type won't work with WinIO. Luckily the number of libraries that have seems quite low. If you can please stick to the external Handle interface for I/O functions. The boot libraries have been updated, and in particular process *requires* the version that is shipped with GHC. Please respect the version bounds here! I will be writing a migration guide for those that need to migrate code. The amount of work is usually trivial as Base provides shims to do most of the common things you would have used Fd for. Also if I may make a plea to GHC developers.. Do not add non-trivial implementations in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add them to internal modules (GHC.xxx) and re-export them from the external modules. This allows us to avoid import cycles inside the internal modules :) -- So why WinIO? Over the years a number of hard to fix issues popped up on Windows, including proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. This also allows libraries like Brick to work on Windows without re-inventing the wheel or have to hide their I/O from the I/O manager. In order to attempt to do some of these with MIO layer upon layers of hacks were added. This means that things sometimes worked.., but when it didn't was rather unpredictable. Some of the issues were simply unfixable with MIO. I will be making some posts about how WinIO works (and also archiving them on the wiki don't worry :)) but for now some highlights: WinIO is 3 years of work, First started by Joey Hess, then picked up by Mikhail Glushenkov before landing at my feet. While the majority has been rewritten their work did provide a great jumping off point so thanks! Also thanks to Ben and AndreasK for helping me get it over the line.. As you can imagine I was exhausted by this point :). Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ commits (sorry this was the smallest we could get it while not losing some historical context) and with over 153 files changed not counting the changes to boot libraries. It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 (but only on native windows consoles, so no msys shells) and #806 which is 14 years old! WinIO is a dynamic choice, so you can switch between I/O managers using the RTS flag --io-manager=[native|posix]. On non-Windows native is the same as posix. The chosen Async interface for this implementation is using Completion Ports. The I/O manager uses a new interface added in Windows Vista called GetQueuedCompletionStatusEx which allows us to service multiple request interrupts in one go. Some highlights: * Drops Windows Vista support Vista is out of extended support as of 2017. The new minimum is Windows 7. This allows us to use much more efficient OS provided abstractions. * Replace Events and Monitor locks with much faster and efficient Conditional Variables and SlimReaderWriterLocks. * Change GHC's Buffer and I/O structs to support asynchronous operation by not relying on the OS managing File Offset. * Implement a new command line flag +RTS --io-manager=[native|posix] to control which I/O manager is used. * Implement a new Console I/O interface supporting much faster reads/writes and unicode output correctly. Also supports things like cooked input etc. * In new I/O manager if the user still has their code-page set to OEM, then we use UTF-8 by default. This allows Unicode to work correctly out of the box. * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. * Flush event logs eagerly as to not rely on finalizers running. * A lot of refactoring and more use of hsc2hs to share constants * Control aborts Ctrl+C should be a bit more reliable. * Add a new IOPort primitive that should be only used for these I/O operations. Essentially an IOPort is based on an MVar with the following major differences: - Does not allow multiple pending writes. If the port is full a second write is just discarded. - There is no deadlock avoidance guarantee. If you block on an IOPort and your Haskell application does not have any work left to do the whole application is stalled. In the threaded RTS we just continue idling, in the non-threaded rts the scheduler is blocked. * Support various optimizations in the Windows I/O manager such as skipping I/O Completion if the request finished synchronously etc. * The I/O manager is now agnostic to the handle type. i.e. There is no socket specific code in the manager. This is now all pushed to the network library. Completely de-coupling these. * Unified threaded and non-threaded I/O code. The only major difference is where event loop is driven from and that the non-threaded rts will always use a single OS thread to service requests. We cannot use more as there are no rts locks to make concurrent modifications safe. Cheers, Tamar -------------- next part -------------- An HTML attachment was scrubbed... URL: From lonetiger at gmail.com Mon Jul 20 23:12:11 2020 From: lonetiger at gmail.com (Phyx) Date: Tue, 21 Jul 2020 00:12:11 +0100 Subject: New Windows I/O manager in GHC 8.12 In-Reply-To: References: Message-ID: Thanks Simon, cheers :) Sent from my Mobile On Mon, Jul 20, 2020, 15:28 Simon Peyton Jones wrote: > Tamar, I salute you! This is a big piece of work – thank you! > > > Simon > > > > *From:* ghc-devs *On Behalf Of *Phyx > *Sent:* 17 July 2020 16:04 > *To:* ghc-devs at haskell.org Devs > *Subject:* New Windows I/O manager in GHC 8.12 > > > > Hi All, > > In case you've missed it, about 150 or so commits were committed to master > yesterday. These commits add WinIO (Windows I/O) to GHC. This is a new > I/O > manager that is designed for the native Windows I/O subsystem instead of > relying on the broken posix-ish compatibility layer that MIO used. > > This is one of 3 big patches I have been working on for years now.. > > So before I continue on why WinIO was made I'll add a TL;DR; > > WinIO adds an internal API break compared to previous GHC releases. That > is > the internal code was modified to support a completely asynchronous I/O > system. > > What this means is that we have to keep track of the file pointer offset > which > previously was done by the C runtime. This is because in async I/O you > cannot > assume the offset to be at any given location. > > What does this mean for you? Very little. If you did not use internal GHC > I/O code. > In particular if you haven't used Buffer, BufferIO and RawIO. If you have > you will > to explicitly add support for GHC 8.12+. > > Because FDs are a Unix concept and don't behave as you would expect on > Windows, the > new I/O manager also uses HANDLE instead of FD. This means that any > library that has > used the internal GHC Fd type won't work with WinIO. Luckily the number of > libraries > that have seems quite low. If you can please stick to the external Handle > interface > for I/O functions. > > The boot libraries have been updated, and in particular process *requires* > the version > that is shipped with GHC. Please respect the version bounds here! I will > be writing > a migration guide for those that need to migrate code. The amount of work > is usually > trivial as Base provides shims to do most of the common things you would > have used Fd for. > > Also if I may make a plea to GHC developers.. Do not add non-trivial > implementations > in the external exposed modules (e.g. System.xxx, Data.xxx) but rather add > them to internal > modules (GHC.xxx) and re-export them from the external modules. This > allows us to avoid > import cycles inside the internal modules :) > > -- > > So why WinIO? Over the years a number of hard to fix issues popped up on > Windows, including > proper Unicode console I/O, cooked inputs, ability to cancel I/O requests. > This also allows libraries like Brick to work on Windows without > re-inventing the wheel or have to hide their I/O from the I/O manager. > > In order to attempt to do some of these with MIO layer upon layers of > hacks were added. This means that things sometimes worked.., but when it > didn't was rather unpredictable. Some of the issues were simply unfixable > with MIO. I will be making some posts about how WinIO works (and also > archiving them on the wiki don't worry :)) but for now some highlights: > > WinIO is 3 years of work, First started by Joey Hess, then picked up by > Mikhail Glushenkov before landing at my feet. While the majority has been > rewritten their work did provide a great jumping off point so thanks! Also > thanks to Ben and AndreasK for helping me get it over the line.. As you can > imagine I was exhausted by this point :). > > Some stats: ~8000 new lines and ~1100 removed ones spread over 130+ > commits (sorry this was the smallest we could get it while not losing some > historical context) and with over 153 files changed not counting the > changes to boot libraries. > > It Fixes #18307, #17035, #16917, #15366, #14530, #13516, #13396, #13359, > #12873, #12869, #11394, #10542, #10484, #10477, #9940, #7593, #7353, #5797, > #5305, #4471, #3937, #3081, #12117, #2408, #10956, #2189 > (but only on native windows consoles, so no msys shells) and #806 which is > 14 years old! > > WinIO is a dynamic choice, so you can switch between I/O managers using > the RTS flag --io-manager=[native|posix]. > > On non-Windows native is the same as posix. > > The chosen Async interface for this implementation is using Completion > Ports. > > The I/O manager uses a new interface added in Windows Vista called > GetQueuedCompletionStatusEx which allows us to service multiple > request interrupts in one go. > > Some highlights: > > * Drops Windows Vista support > Vista is out of extended support as of 2017. The new minimum is Windows > 7. This allows us to use much more efficient OS provided abstractions. > > * Replace Events and Monitor locks with much faster and efficient > Conditional Variables and SlimReaderWriterLocks. > * Change GHC's Buffer and I/O structs to support asynchronous operation by > not relying on the OS managing File Offset. > * Implement a new command line flag +RTS --io-manager=[native|posix] to > control which I/O manager is used. > * Implement a new Console I/O interface supporting much faster > reads/writes and unicode output correctly. Also supports things like > cooked input etc. > * In new I/O manager if the user still has their code-page set to OEM, > then we use UTF-8 by default. This allows Unicode to work correctly out of > the box. > * Add Atomic Exchange PrimOp and implement Atomic Ptr exchanges. > * Flush event logs eagerly as to not rely on finalizers running. > * A lot of refactoring and more use of hsc2hs to share constants > * Control aborts Ctrl+C should be a bit more reliable. > * Add a new IOPort primitive that should be only used for these I/O > operations. Essentially an IOPort is based on an MVar with the following > major > differences: > - Does not allow multiple pending writes. If the port is full a second > write is just discarded. > - There is no deadlock avoidance guarantee. If you block on an IOPort > and your Haskell application does not have any work left to do the whole > application is > stalled. In the threaded RTS we just continue idling, in the non-threaded > rts the scheduler is blocked. > > * Support various optimizations in the Windows I/O manager such as > skipping I/O Completion if the request finished synchronously etc. > * The I/O manager is now agnostic to the handle type. i.e. There is no > socket specific code in the manager. This is now all pushed to the network > library. Completely de-coupling these. > * Unified threaded and non-threaded I/O code. The only major difference is > where event loop is driven from and that the non-threaded rts will always > use a single OS thread to service requests. We cannot use more as there are > no rts locks to make concurrent modifications safe. > > Cheers, > Tamar > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Tue Jul 21 14:21:10 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Tue, 21 Jul 2020 10:21:10 -0400 Subject: Unmerged Patch: 3358 In-Reply-To: References: Message-ID: <87pn8phru5.fsf@smart-cactus.org> Matthew Pickering writes: > Hi, > > My patch 3358 needs to get merged before the 8.12 fork. > Yes, it is on my list. It is in fact one of the last two patches that need to be merged before we can fork. Unfotunately, it has conflicted a number of times and required a fair bit of manual rebasing, hence the wait. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From chessai1996 at gmail.com Tue Jul 21 17:22:52 2020 From: chessai1996 at gmail.com (chessai) Date: Tue, 21 Jul 2020 10:22:52 -0700 Subject: Unmerged Patch: 3358 In-Reply-To: <87pn8phru5.fsf@smart-cactus.org> References: <87pn8phru5.fsf@smart-cactus.org> Message-ID: Ben, I am behind on some details, but if it is blocked on text, is there some way I can help? On Tue, Jul 21, 2020, 7:21 AM Ben Gamari wrote: > Matthew Pickering writes: > > > Hi, > > > > My patch 3358 needs to get merged before the 8.12 fork. > > > Yes, it is on my list. It is in fact one of the last two patches that > need to be merged before we can fork. Unfotunately, it has conflicted a > number of times and required a fair bit of manual rebasing, hence the > wait. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Tue Jul 21 19:07:58 2020 From: ben at well-typed.com (Ben Gamari) Date: Tue, 21 Jul 2020 15:07:58 -0400 Subject: Unmerged Patch: 3358 In-Reply-To: References: <87pn8phru5.fsf@smart-cactus.org> Message-ID: <17BAE71B-6CC4-40C4-81DB-17AED577535F@well-typed.com> On July 21, 2020 1:22:52 PM EDT, chessai wrote: >Ben, > >I am behind on some details, but if it is blocked on text, is there >some >way I can help? > I think we are all clear here; the text branch was merged and I have now merged !3358 to master. Cheers, - Ben -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From ben at smart-cactus.org Wed Jul 22 23:27:35 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 22 Jul 2020 19:27:35 -0400 Subject: GHC 9.0? In-Reply-To: References: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> Message-ID: <87v9ifgmfs.fsf@smart-cactus.org> Artem Pelenitsyn writes: > Does Quick Look still have chances to make it into the next release? > It'd be fascinating if the major version bump got both linear and > impredicative types! > I'm afraid not. Quick Look will need to wait for 9.2. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 483 bytes Desc: not available URL: From alan.zimm at gmail.com Wed Jul 22 23:30:30 2020 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Thu, 23 Jul 2020 00:30:30 +0100 Subject: GHC 9.0? In-Reply-To: <87v9ifgmfs.fsf@smart-cactus.org> References: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> <87v9ifgmfs.fsf@smart-cactus.org> Message-ID: Which sounds like we have agreed on 9.0? Alan On Thu, 23 Jul 2020, 00:28 Ben Gamari, wrote: > Artem Pelenitsyn writes: > > > Does Quick Look still have chances to make it into the next release? > > It'd be fascinating if the major version bump got both linear and > > impredicative types! > > > I'm afraid not. Quick Look will need to wait for 9.2. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu Jul 23 02:55:00 2020 From: ben at well-typed.com (Ben Gamari) Date: Wed, 22 Jul 2020 22:55:00 -0400 Subject: GHC 9.0? In-Reply-To: References: <010f017366bf0588-c953e107-906e-4d02-b892-aa0c5c326a14-000000@us-east-2.amazonses.com> <87v9ifgmfs.fsf@smart-cactus.org> Message-ID: <87pn8ngcu6.fsf@smart-cactus.org> "Alan & Kim Zimmerman" writes: > Which sounds like we have agreed on 9.0? > I've not heard any objections so I am happy to push through the change. However, I'm waiting another day before formally announcing. The earlier email was a bit of a slip on my part. If anyone sees a good reason why with the coming release shouldn't be numbered 9.0, speak now or forever hold your speech. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri Jul 24 15:37:19 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 24 Jul 2020 11:37:19 -0400 Subject: GHC 9.0? In-Reply-To: References: Message-ID: <87blk4hqkz.fsf@smart-cactus.org> Krzysztof Gogolewski writes: > Hi, > > There is an exceptional number of changes stated for the next release. > > * Better pattern matching coverage detection > * New windows IO manager > * Linear types > * Large-scale typechecker changes - Taming the Kind Inference Monster, > simplified subsumption > * Better register allocation, improving runtime by 0.8% according to > release notes > * ghc-bignum > * Explicit specificity and eager instantiation > * Qualified do > * Lexical negation > * Perhaps Quick Look will manage to land > > Should we call it GHC 9.0? I think the name would be deserved. As no one has objected, let's move ahead with this proposal. GHC 8.12.1 shall henceforth be 9.0.1. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mail at joachim-breitner.de Sat Jul 25 19:10:56 2020 From: mail at joachim-breitner.de (Joachim Breitner) Date: Sat, 25 Jul 2020 21:10:56 +0200 Subject: GHC 9.0! In-Reply-To: <87blk4hqkz.fsf@smart-cactus.org> References: <87blk4hqkz.fsf@smart-cactus.org> Message-ID: Am Freitag, den 24.07.2020, 11:37 -0400 schrieb Ben Gamari: > As no one has objected, let's move ahead with this proposal. GHC 8.12.1 > shall henceforth be 9.0.1. Allow me to fix the Subject of this mail thread then ;-) -- Joachim Breitner mail at joachim-breitner.de http://www.joachim-breitner.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: This is a digitally signed message part URL: From simonpj at microsoft.com Mon Jul 27 08:45:50 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 27 Jul 2020 08:45:50 +0000 Subject: How should we treat changes to the GHC API? Message-ID: A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: how we should treat significant changes to the GHC API? Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users - they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.) I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult. Any views? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From m at tweag.io Mon Jul 27 10:11:17 2020 From: m at tweag.io (Mathieu Boespflug) Date: Mon, 27 Jul 2020 10:11:17 +0000 Subject: How should we treat changes to the GHC API? In-Reply-To: References: Message-ID: I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API? It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers. On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones < ghc-devs at haskell.org > wrote: > > > > A recent MR for GHC ( > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3758 ) (adding > machinery for plugins to write data to extensible interface files) made me > wonder: > > > > > how we should treat significant changes to the GHC API? > > > > > Changes to the GHC API, especially to bits used by plugins or by IDEs, are > clearly user-visible to an important class of users – they are not just > internal to GHC itself.   So, how should we review them?  Should they > perhaps be part of the GHC proposals process?  Or some other similar > process?   (The collection of experts on the GHC API, plugins, IDEs etc, > is rather different to the membership of the GHC steering group.) > > > > > I'm asking, not to be obstructive, but because the GHC API deserves to be > thought of as a whole; in the past it has grown incrementally, without > much discussion, and that has not served us well.  But at the moment there > is no process, no group to consult. > > > > > Any views? > > > > > Simon > > > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@ haskell. org ( ghc-devs at haskell.org ) > http:/ / mail. haskell. org/ cgi-bin/ mailman/ listinfo/ ghc-devs ( > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs ) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Mon Jul 27 11:04:06 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 27 Jul 2020 11:04:06 +0000 Subject: How should we treat changes to the GHC API? In-Reply-To: References: Message-ID: What I’m after is a clear opportunity for informed debate, and a clear yes/no decision. That need not be high overhead. It means paying some upfront cost for design changes. But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be. We tend to think of APIs as implementation details. But they are deeply significant, and express key abstractions, just like language designs do. I think we should treat them just as seriously. Simon From: Mathieu Boespflug Sent: 27 July 2020 11:11 To: Simon Peyton Jones Cc: ghc-devs at haskell.org Devs Subject: Re: How should we treat changes to the GHC API? I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API? It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers. On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones > wrote: A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: how we should treat significant changes to the GHC API? Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.) I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult. Any views? Simon _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From iavor.diatchki at gmail.com Mon Jul 27 16:45:51 2020 From: iavor.diatchki at gmail.com (Iavor Diatchki) Date: Mon, 27 Jul 2020 09:45:51 -0700 Subject: How should we treat changes to the GHC API? In-Reply-To: References: Message-ID: In principle, I think we should treat the GHC API like any other library, and try not to break code unnecessarily. However, my impression is that the GHC API grew somewhat organically, so we may want to put some additional work before we stabilize things too much. It's been a while since I used it, so I might be out of date, but last I looked the GHC API was a module exporting some high-level functions from GHC. I think that a single module is too small of an API for a project as large as GHC. In fact, it probably makes sense to define more than one API. For example, each plugin point should probably have its own API, and that's likely different to the GHC API that exposes functionality such as "load and type check this module here", or "parse and evaluate this string". -Iavor On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs < ghc-devs at haskell.org> wrote: > What I’m after is a clear opportunity for informed debate, and a clear > yes/no decision. That need not be high overhead. > > > > It means paying some upfront cost for design changes. But that’s better > than the legacy cost of dealing with things that turn out, in retrospect, > to be less well designed than they could be. > > > > We tend to think of APIs as implementation details. But they are deeply > significant, and express key abstractions, just like language designs do. > I think we should treat them just as seriously. > > > > Simon > > > > *From:* Mathieu Boespflug > *Sent:* 27 July 2020 11:11 > *To:* Simon Peyton Jones > *Cc:* ghc-devs at haskell.org Devs > *Subject:* Re: How should we treat changes to the GHC API? > > > > I would just point out that decision by committee, and in particular the > GHC Proposals process, has a high cost in terms of both total human brain > cycles and latency. The cost is entirely justified when it comes to things > that are a) hard to revert and b) extremely hard to get right the first > time, like new extensions to the language, or c) very sensitive (like > governance, say). For things like breaking changes to API's, it's worth > writing out what the current problems are. Are users complaining that the > API churn is too high? Are they concerned about endemic quality problems > with the API? > > > > It may be enough to make sure to know who the main users of the API are > and tag them as reviewers on these types of changes in GitLab. Or to avoid > extra process but enshrine principles that might be necessary to adopt, > like saying that existing API functions should always be kept as-is during > some deprecation period and new functionality should be exposed in new > additions to the API. Principles to be upheld by reviewers. > > > > On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones > wrote: > > A recent MR for GHC > > (adding machinery for plugins to write data to extensible interface files) > made me wonder: > > how we should treat significant changes to the GHC API? > > Changes to the GHC API, especially to bits used by plugins or by IDEs, are > clearly user-visible to an important class of users – they are not just > internal to GHC itself. So, how should we review them? Should they > perhaps be part of the GHC proposals process? Or some other similar > process? (The collection of experts on the GHC API, plugins, IDEs etc, is > rather different to the membership of the GHC steering group.) > > I'm asking, not to be obstructive, but because the GHC API deserves to be > thought of as a whole; in the past it has grown incrementally, without much > discussion, and that has not served us well. But at the moment there is no > process, no group to consult. > > Any views? > > Simon > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at seidel.io Mon Jul 27 18:07:55 2020 From: eric at seidel.io (Eric Seidel) Date: Mon, 27 Jul 2020 13:07:55 -0500 Subject: How should we treat changes to the GHC API? In-Reply-To: References: Message-ID: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> As another former user of the GHC API, I'd say my two biggest complaints were the relative instability of the API and the lack of documentation. I haven't used the API in at least three years though, so it's possible much has changed since my experience. I remember often having to do significant work to adapt LiquidHaskell to new versions of GHC due to small changes in the API. We often felt like we had to import internal modules (ie other than 'GHC') to get key bits of functionality we needed, which might explain the churn. But it also points to Iavor's point that the public API grew organically and might benefit from a bit of top-down design to make sure it's complete enough for typical use cases. For documentation, the issue was less the API docs but a lack of "How do I do X?" docs and examples. One problem that I remember being particularly vexing was resolving names in a particular scope (in my case it was always module-level scopes, but I can easily imagine clients that would want to resolve names in local scopes). I don't know if the API needs to go through something like the Steering Committee, but a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome! On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote: > In principle, I think we should treat the GHC API like any other > library, and try not to break code unnecessarily. However, my > impression is that the GHC API grew somewhat organically, so we may > want to put some additional work before we stabilize things too much. > It's been a while since I used it, so I might be out of date, but last > I looked the GHC API was a module exporting some high-level functions > from GHC. I think that a single module is too small of an API for a > project as large as GHC. In fact, it probably makes sense to define > more than one API. For example, each plugin point should probably have > its own API, and that's likely different to the GHC API that exposes > functionality such as "load and type check this module here", or "parse > and evaluate this string". > > -Iavor > > > > On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs > wrote: > > What I’m after is a clear opportunity for informed debate, and a clear yes/no decision. That need not be high overhead.____ > > > __ __ > > > It means paying some upfront cost for design changes. But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.____ > > > __ __ > > > We tend to think of APIs as implementation details. But they are deeply significant, and express key abstractions, just like language designs do. I think we should treat them just as seriously.____ > > > __ __ > > > Simon____ > > > __ __ > > > *From:* Mathieu Boespflug > > *Sent:* 27 July 2020 11:11 > > *To:* Simon Peyton Jones > > *Cc:* ghc-devs at haskell.org Devs > > *Subject:* Re: How should we treat changes to the GHC API?____ > > > __ __ > > > I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?____ > > > __ __ > > > It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.____ > > > __ __ > > > On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones wrote:____ > > >> A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: ____ > > >> how we should treat significant changes to the GHC API?____ > > >> Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)____ > > >> I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult.____ > > >> Any views?____ > > >> Simon____ > > >> _______________________________________________ > >> ghc-devs mailing list > >> ghc-devs at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs ____ > > > __ __ > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > From john.ericson at obsidian.systems Mon Jul 27 19:06:59 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Mon, 27 Jul 2020 15:06:59 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> Message-ID: I would say the current API is....absolutely terrible and so we should have no qualms about breaking it. Talking to the people who work on the IDE---i.e. the people who are the most adversely impacted by churn---they also don't like it and not want it stabilized in its current form. I think the only good long term solution is to modularize GHC into multiple packages, giving us conceptually distinct interfaces which break at different rates. This is similar to modularizing base as e.g. GHC.* breaks far more often than e.g. Data.List. How we evaluate changes is less important to me as my main priority is to get the pace of development much higher so we have a chance on fixing all the technical debt to get us out of the current situation. Basically, you could count me in "team heat bath"; I think that should have the right connotations. John On 7/27/20 2:07 PM, Eric Seidel wrote: > As another former user of the GHC API, I'd say my two biggest complaints were the relative instability of the API and the lack of documentation. I haven't used the API in at least three years though, so it's possible much has changed since my experience. > > I remember often having to do significant work to adapt LiquidHaskell to new versions of GHC due to small changes in the API. We often felt like we had to import internal modules (ie other than 'GHC') to get key bits of functionality we needed, which might explain the churn. But it also points to Iavor's point that the public API grew organically and might benefit from a bit of top-down design to make sure it's complete enough for typical use cases. > > For documentation, the issue was less the API docs but a lack of "How do I do X?" docs and examples. One problem that I remember being particularly vexing was resolving names in a particular scope (in my case it was always module-level scopes, but I can easily imagine clients that would want to resolve names in local scopes). > > I don't know if the API needs to go through something like the Steering Committee, but a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome! > > On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote: >> In principle, I think we should treat the GHC API like any other >> library, and try not to break code unnecessarily. However, my >> impression is that the GHC API grew somewhat organically, so we may >> want to put some additional work before we stabilize things too much. >> It's been a while since I used it, so I might be out of date, but last >> I looked the GHC API was a module exporting some high-level functions >> from GHC. I think that a single module is too small of an API for a >> project as large as GHC. In fact, it probably makes sense to define >> more than one API. For example, each plugin point should probably have >> its own API, and that's likely different to the GHC API that exposes >> functionality such as "load and type check this module here", or "parse >> and evaluate this string". >> >> -Iavor >> >> >> >> On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs >> wrote: >>> What I’m after is a clear opportunity for informed debate, and a clear yes/no decision. That need not be high overhead.____ >>> __ __ >>> It means paying some upfront cost for design changes. But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.____ >>> __ __ >>> We tend to think of APIs as implementation details. But they are deeply significant, and express key abstractions, just like language designs do. I think we should treat them just as seriously.____ >>> __ __ >>> Simon____ >>> __ __ >>> *From:* Mathieu Boespflug >>> *Sent:* 27 July 2020 11:11 >>> *To:* Simon Peyton Jones >>> *Cc:* ghc-devs at haskell.org Devs >>> *Subject:* Re: How should we treat changes to the GHC API?____ >>> __ __ >>> I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?____ >>> __ __ >>> It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.____ >>> __ __ >>> On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones wrote:____ >>>> A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: ____ >>>> how we should treat significant changes to the GHC API?____ >>>> Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)____ >>>> I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult.____ >>>> Any views?____ >>>> Simon____ >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs ____ >>> __ __ >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From rae at richarde.dev Mon Jul 27 20:57:12 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Mon, 27 Jul 2020 20:57:12 +0000 Subject: How should we treat changes to the GHC API? In-Reply-To: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> Message-ID: <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> > On Jul 27, 2020, at 2:07 PM, Eric Seidel wrote: > > a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome! I agree with this in the abstract, but I'm not sure about agreeing with it in the concrete. (I don't mean to pick on Eric here -- but this was a nice sentence I could quote.) The problem is that stability in the API has a very real cost. It means that GHC developers maintain some interface indefinitely, even if the implementation drifts in a different direction. Folks doing the internal GHC work often don't have extensive experience with the API, which means that a move toward API stability would mean that GHC devs would be poorly equipped to decide when an interface is important to preserve or unimportant. This leads to the possibility that devs would spend a lot of time maintaining particular behavior that is not needed. And, of course, time spent holding the API stable is time not spent doing other tasks, and thus a move toward stability would slow down development. This might be a small difference -- I'm not predicting calamity -- but it's a real cost. On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC. Having a central project to deal with the API would satisfy clients by projecting a stable interface, and it wouldn't hinder GHC development, as the API volunteer would take on the compatibility burden. Of course, success here would require a dedicated, competent volunteer. To be clear: I'm not against API stability, but I do want to make clear that this choice has a cost. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Mon Jul 27 21:24:00 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Mon, 27 Jul 2020 17:24:00 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: On 7/27/20 4:57 PM, Richard Eisenberg wrote > On the other hand, we could imagine a dedicated GHC API volunteer who > maintains the API on top of the shifting sands of GHC. Having a > central project to deal with the API would satisfy clients by > projecting a stable interface, and it wouldn't hinder GHC development, > as the API volunteer would take on the compatibility burden. Of > course, success here would require a dedicated, competent volunteer. I think that basically exists: https://hackage.haskell.org/package/ghc-lib ? As permanent solution I don't like it, but as a stop gap to relieve any pressure for stability until we have sane interfaces, I *love* it. John From shayne.fletcher.50 at gmail.com Mon Jul 27 23:13:14 2020 From: shayne.fletcher.50 at gmail.com (Shayne Fletcher) Date: Mon, 27 Jul 2020 19:13:14 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: On Mon, Jul 27, 2020 at 5:24 PM John Ericson wrote: > > I think that basically exists: > https://hackage.haskell.org/package/ghc-lib ? As permanent solution I > don't like it, but as a stop gap to relieve any pressure for stability > until we have sane interfaces, I *love* it. > `ghc-lib` is a literal subset of GHC files: `ghc-lib-parser` that set of files sufficient to produce abstract syntax trees (~200 files), `ghc-lib` the remaining set of files enabling Core generation from parsed syntax trees (~300 of those) (more detail in the project README here https://github.com/digital-asset/ghc-lib/blob/master/README.md). There is at this time, no attempt to provide any additional interface, "higher level" or otherwise, in these packages. The key property of `ghc-lib-parser`/`ghc-lib` that makes them useful to tool developers (e.g. HLint) is that they can utilize the GHC API without being bound to a specific compiler version to do so. -- Shayne Fletcher -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at seidel.io Tue Jul 28 01:11:29 2020 From: eric at seidel.io (Eric Seidel) Date: Mon, 27 Jul 2020 20:11:29 -0500 Subject: How should we treat changes to the GHC API? In-Reply-To: <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> References: <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: <48360749-EC8F-443A-B2F1-76D4A4543EA7@seidel.io> Thanks Richard, you raise some important points. API stability does impose a real cost on developers (working at a company with a firm-wide integration build for the past three years has made me keenly aware of that). But API instability also imposes a real cost on clients. I think there are a couple things that could be done though, to help lessen the burden, and to share it between devs and clients. One is to define a clear separation between the public API and internal modules. GHC does this to an extent, but as I mentioned earlier, I think it needs a more comprehensive public API (I did not mean that we should just declare more modules as part of the public API, but provide stable wrappers around that functionality). You’re right that GHC devs are not the best suited to define the public API, I think it should be done as a collaboration between clients and devs. Another option could be to have a transition period for APIs like we do for changes to the language. GHC already commits to waiting a few revisions to use new language features in the name of shortening bootstrapping chains, waiting a few revisions to drop an old API doesn’t feel all that different and would give clients a lot of breathing room. I don’t want to put the burden of API stability all on GHC devs either, but I do think it could probably be balanced better. And this is where having something like the Steering Committee could help. It doesn’t need to be as formal either, perhaps a small group of stakeholders that get tagged on MRs to the public API, or a mailing list where you solicit feedback. Sent from my iPhone > On Jul 27, 2020, at 15:57, Richard Eisenberg wrote: > >  > >> On Jul 27, 2020, at 2:07 PM, Eric Seidel wrote: >> >> a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome! > > I agree with this in the abstract, but I'm not sure about agreeing with it in the concrete. (I don't mean to pick on Eric here -- but this was a nice sentence I could quote.) The problem is that stability in the API has a very real cost. It means that GHC developers maintain some interface indefinitely, even if the implementation drifts in a different direction. Folks doing the internal GHC work often don't have extensive experience with the API, which means that a move toward API stability would mean that GHC devs would be poorly equipped to decide when an interface is important to preserve or unimportant. This leads to the possibility that devs would spend a lot of time maintaining particular behavior that is not needed. And, of course, time spent holding the API stable is time not spent doing other tasks, and thus a move toward stability would slow down development. This might be a small difference -- I'm not predicting calamity -- but it's a real cost. > > On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC. Having a central project to deal with the API would satisfy clients by projecting a stable interface, and it wouldn't hinder GHC development, as the API volunteer would take on the compatibility burden. Of course, success here would require a dedicated, competent volunteer. > > To be clear: I'm not against API stability, but I do want to make clear that this choice has a cost. > > Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Tue Jul 28 02:57:58 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Tue, 28 Jul 2020 02:57:58 +0000 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: <010f0173935af3ac-f59ea1c7-94c6-4dd4-8c94-7463710b1eba-000000@us-east-2.amazonses.com> > On Jul 27, 2020, at 5:24 PM, John Ericson wrote: > > As permanent solution I don't like it, but as a stop gap to relieve any pressure for stability until we have sane interfaces, I *love* it. I guess I'm arguing that ghc-lib (or something like it) should be the permanent solution. GHC will always be in flux internally. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.trstenjak at gmail.com Tue Jul 28 06:37:46 2020 From: daniel.trstenjak at gmail.com (Daniel Trstenjak) Date: Tue, 28 Jul 2020 08:37:46 +0200 Subject: How should we treat changes to the GHC API? In-Reply-To: <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: <20200728063746.GA2913@octa> On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote: > On the other hand, we could imagine a dedicated GHC API volunteer who > maintains the API on top of the shifting sands of GHC. Looking at other compilers that have been successful in having a stable API - like clang with the libclang - that's pretty much how they achieved it. The manpower of GHC devs is already pretty small, so putting another burden on them with a stable API won't work out that well. Also as a compiler dev you want to have the freedom to change your AST in the ways you need it, to incorporate new features. Having workarounds at this level to ensure API stability just seems to be the wrong place and will only increase the complexity in an already quite complex project. However a high level API on top of the AST is the perfect place for such special cases between different compiler versions. Greetings, Daniel From ndmitchell at gmail.com Tue Jul 28 08:18:36 2020 From: ndmitchell at gmail.com (Neil Mitchell) Date: Tue, 28 Jul 2020 09:18:36 +0100 Subject: How should we treat changes to the GHC API? In-Reply-To: <20200728063746.GA2913@octa> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> <20200728063746.GA2913@octa> Message-ID: The API changing regularly is a real hassle for people building on top of it. But, the people building on top of it are also some of the people changing it most. And the API isn't nice and beautiful as it stands, with lots of horrible work-arounds for downstream users. I'd suggest letting people change the API freely. But avoid making bikeshed API changes (those which are minor improvements at best, and really just a different flavour of the same). And perhaps loop in heavy API users when removing functionality. Thanks, Neil On Tue, Jul 28, 2020 at 7:38 AM Daniel Trstenjak wrote: > On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote: > > On the other hand, we could imagine a dedicated GHC API volunteer who > > maintains the API on top of the shifting sands of GHC. > > Looking at other compilers that have been successful in having a stable > API - like clang with the libclang - that's pretty much how they > achieved it. > > The manpower of GHC devs is already pretty small, so putting another > burden on them with a stable API won't work out that well. > > Also as a compiler dev you want to have the freedom to change your AST > in the ways you need it, to incorporate new features. Having workarounds > at this level to ensure API stability just seems to be the wrong place and > will only increase the complexity in an already quite complex project. > However a high level API on top of the AST is the perfect place for such > special cases between different compiler versions. > > Greetings, > Daniel > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shayne.fletcher.50 at gmail.com Tue Jul 28 10:11:03 2020 From: shayne.fletcher.50 at gmail.com (Shayne Fletcher) Date: Tue, 28 Jul 2020 06:11:03 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> Message-ID: On Mon, Jul 27, 2020 at 7:13 PM Shayne Fletcher < shayne.fletcher.50 at gmail.com> wrote: > > > On Mon, Jul 27, 2020 at 5:24 PM John Ericson > wrote: > >> >> I think that basically exists: >> https://hackage.haskell.org/package/ghc-lib ? As permanent solution I >> don't like it, but as a stop gap to relieve any pressure for stability >> until we have sane interfaces, I *love* it. >> > > `ghc-lib` is a literal subset of GHC files: `ghc-lib-parser` that set of > files sufficient to produce abstract syntax trees (~200 files), `ghc-lib` > the remaining set of files enabling Core generation from parsed syntax > trees (~300 of those) (more detail in the project README here > https://github.com/digital-asset/ghc-lib/blob/master/README.md). > > There is at this time, no attempt to provide any additional interface, > "higher level" or otherwise, in these packages. > > [...] > I should mention that the related project `ghc-lib-parser-ex` (README here https://github.com/shayne-fletcher/ghc-lib-parser-ex/blob/master/README.md) IS in large part motivated by the need to provide stable interfaces over `ghc-lib-parser`. -- Shayne Fletcher -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Tue Jul 28 14:38:30 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Tue, 28 Jul 2020 10:38:30 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: <010f0173935af3ac-f59ea1c7-94c6-4dd4-8c94-7463710b1eba-000000@us-east-2.amazonses.com> References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> <010f0173935af3ac-f59ea1c7-94c6-4dd4-8c94-7463710b1eba-000000@us-east-2.amazonses.com> Message-ID: On 7/27/20 10:57 PM, Richard Eisenberg wrote: > I guess I'm arguing that ghc-lib (or something like it) should be the > permanent solution. GHC will always be in flux internally. So to be clear I don't disagree that GHC will always be in flux. But I think things like ghc-lib largely work today because GHC is so imperative. Once it (hopefully!) is more functional, we'll have the problem less that functions change[1], and more that data structures change. I think that will be harder to abstract over---often to migrate an interface properly, data structure will need migrations in both directions *and* that those migrations form an isomorphism, which can be an impossible requirement. It's just a fact of life that rich data structures make migrations harder, but that's no reason we shouldn't have rich data structures! I'm probably too deeply exploring what's only one possible future at this point, so let me rewind out of that exact hypothesis and just make the broader point that if the interfaces change a lot, a lot of organizational stuff can/should too. If the interface changes *don't* engender cultural shifts around working with GHC, we're probably doing them wrong and they are vapid churn after all! John [1]: there should be way fewer functions than today. we currently have a gratuitous combinatorial explosion of little helpers which obscures what the underlying orthogonal concepts are. From compl.yue at icloud.com Thu Jul 30 13:53:57 2020 From: compl.yue at icloud.com (YueCompl) Date: Thu, 30 Jul 2020 21:53:57 +0800 Subject: Fwd: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? References: <8D814D14-3F54-4D2E-A3F1-EC8A2DCB1E31@icloud.com> Message-ID: <6B2609DA-7317-4343-9178-6594499C95CB@icloud.com> Dear GHC Devs, I realize I should seek your helps regarding my current situation. TL;DR the most serious suffering is: After the heap loaded with many TVars with cyclic structures, GC will dominate my CPU utilization with little business progressing. Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for my case, obvious performance degrade starts at about ~350MB RSS, and falls unusable as RSS approaching 1GB. While I expect a GHC compiled process to serve 20~30GB in-mem data practically okay. https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html contains most interesting conversations. I found https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-memory.html much relevant, they managed to keep 100GB per instance, but switching to immutable data structures to reside within compact regions is not immediately feasible for my case, as the schema has to be re-designed, though I have my colleges started evaluating that option. Best regards, Compl > Begin forwarded message: > > From: YueCompl via Haskell-Cafe > Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? > Date: 2020-07-30 at 21:28:31 GMT+8 > To: Haskell Cafe > Reply-To: YueCompl > > For the record, overhead of STM over IO (or other means where manual composition of transactions needed) based concurrency control, is a price I'm willing to pay in my use case, as it's not machine-performance critical in distributing input data + parameters to a cluster of worker nodes, and collecting their results into permanent storage or a data pipeline. But to keep professionally crafting well synced, race-free scheduling code is barely affordable by my org, as shape of datasets, relationship between them and algorithms processing them are varying at fast paces, we have difficulty, or lack the willingness, to hire some workforce specifically to keep each new data pipeline race free, it has to be, but better at cost of machine-hours instead of human head counts. > > While easily compositing stm code, wrapped in scriptable procedures, will enable our analysts to author the scheduling scripts without too much concerns. Then our programmers can focus on performance critical parts of the data processing code, like optimization of tight-loops. > > Only if not in the tight loops, I think it's acceptable by us, that up to 2~3 order of magnitude slower for an stm solution compared to its best rivals, as long as it's scalable. For a (maybe cheating) example, if fully optimized code can return result in 10 ms after an analyst clicked a button, we don't bother if unoptimized stm script needs 10 second, so long as the result is correct. > > In a philosophic thinking, I heard that AT&T had UNIX specifically designed for their Control panel, while their Data panel runs separate software (and on separate hardware obviously), while modern systems have powerful CPUs tempting us to squeeze more performance out of it, and SIMD instructions make it even more tempting, I think we'd better resist it when programming something belong to the Control panel per se, but do it in programming something belong to the Data panel. And appears Data panel programs are being shifted to GPUs nowadays, which feels right. > > Regards, > Compl > > >> On 2020-07-30, at 20:10, YueCompl via Haskell-Cafe > wrote: >> >> Hi Peter, >> >> Great to hear from you! >> >> For the record tskiplist (and stm-containers together) did improve my situation a great lot with respect to scalability at concurrency/parallelism! >> >> I'm still far from the stage to squeeze last drops of performance, currently I'm just making sure performance wise concerns be reasonable during my PoC in correctness and ergonomics of my HPC architecture (an in-memory graph + out-of-core (mmap) array DBMS powered computation cluster, with shared storage), and after parallelism appears acceptable, I seemingly suffer from serious GC issue at up scaling on process working memory size. I'm suspecting it's because of the added more TVars and/or aggressive circular structures of them in my case, and can not find a way to overcome it by far. >> >> Thanks for your detailed information! >> >> Best regards, >> Compl >> >> >>> On 2020-07-30, at 19:19, Peter Robinson > wrote: >>> >>> Hi Compl, >>> >+ This package provides a proof-of-concept implementation of a skip list in STM >>> >>> This has to mean something but I can't figure out yet. >>> >>> Dear Peter Robinson, I hope you can see this message and get in the loop of discussion. >>> >>> >>> The reason for adding this sentence was that tskiplist hasn't been optimized for production use. Later on, I wrote an implementation of a concurrent skip list with atomic operations that performs significantly better, but it's operations work in the IO monad. >>> >>> I'm surprised to hear that you're getting poor performance even when using the stm-container package, which I believe was meant to be used in production. A while ago, I ran some benchmarks comparing concurrent dictionary data structures (such as stm-container) under various workloads. While STMContainers.Map wasn't as fast as the concurrent-hashtable package, the results indicate that the performance doesn't degrade too much under larger workloads. >>> >>> You can find these benchmark results here (10^6 randomly generated insertion/deletion/lookup requests distributed among 32 threads): >>> https://lowerbound.io/blog/bench2-32.html >>> And some explanations about the benchmarks are here: >>> https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html >>> >>> One issue that I came across when implementing the tskiplist package was this: If a thread wants to insert some item into the skip list, it needs to search for the entry point by performing readTVar operations starting at the list head. So, on average, a thread will read O(log n) TVars (assuming a skip list of n items) and, if any of these O(log n) TVars are modified by a simultaneously running thread, the STM runtime will observe a (false) conflict and rerun the transaction. It's not clear to me how to resolve this issue without access to something like unreadTVar (see [1]). >>> >>> Best, >>> Peter >>> >>> [1] UnreadTVar: Extending Haskell Software Transactional Memory for Performance (2007) by Nehir Sonmez , Cristian Perfumo , Srdjan Stipic , Adrian Cristal , Osman S. Unsal , Mateo Valero. >>> >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> To (un)subscribe, modify options or view archives go to: >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >>> Only members subscribed via the mailman list are allowed to post. >> >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post. -------------- next part -------------- An HTML attachment was scrubbed... URL: From compl.yue at icloud.com Thu Jul 30 14:05:04 2020 From: compl.yue at icloud.com (YueCompl) Date: Thu, 30 Jul 2020 22:05:04 +0800 Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? In-Reply-To: <6B2609DA-7317-4343-9178-6594499C95CB@icloud.com> References: <8D814D14-3F54-4D2E-A3F1-EC8A2DCB1E31@icloud.com> <6B2609DA-7317-4343-9178-6594499C95CB@icloud.com> Message-ID: And I have some uneducated guess also in https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132559.html > And per my understanding, GHC's GC doesn't seek free segments within a heap, it instead will copy all live objects to a new heap after then swap the new heap to be the live one, so I assume memory (address space) fragmentation doesn't make much trouble for a GHC process, as for other runtimes. > I suspect the difficulty resides in the detection of circular/cyclic circumstances wrt live data structures within the old heap, especially the circles form with arbitrary number of pointers of indirection. If the GC has to perform some dict lookup to decide if an object has been copied to new heap, that's O(n*log(n)) complexity in best case, where n is number of live objects in the heap. > To efficiently copy circular structures, one optimization I can imagine is to have a `new ptr` field in every heap object, then in copying another object with a pointer to one object, the `new ptr` can be read out and if not nil, assign the pointer field on another object' in the new heap to that value and it's done; or copy one object' to the new heap, and update the field on one object in the old heap pointing to the new heap. But I don't know details of GHC GC and can't imagine even feasibility of this technique. And even the new nonmoving GC may have similar difficulty to jump out of a circle when following pointers. FYI > On 2020-07-30, at 21:53, YueCompl via ghc-devs wrote: > > Dear GHC Devs, > > I realize I should seek your helps regarding my current situation. > > TL;DR the most serious suffering is: After the heap loaded with many TVars with cyclic structures, GC will dominate my CPU utilization with little business progressing. > > Nonmoving GC `-xn` with 8.10.1 helps, but the ceiling is not high enough for my case, obvious performance degrade starts at about ~350MB RSS, and falls unusable as RSS approaching 1GB. While I expect a GHC compiled process to serve 20~30GB in-mem data practically okay. > > https://mail.haskell.org/pipermail/haskell-cafe/2020-July/132556.html contains most interesting conversations. > > I found https://tech.channable.com/posts/2020-04-07-lessons-in-managing-haskell-memory.html much relevant, they managed to keep 100GB per instance, but switching to immutable data structures to reside within compact regions is not immediately feasible for my case, as the schema has to be re-designed, though I have my colleges started evaluating that option. > > Best regards, > Compl > > >> Begin forwarded message: >> >> From: YueCompl via Haskell-Cafe > >> Subject: [Haskell-cafe] For STM to practically serve large in-mem datasets with cyclic structures WAS: STM friendly TreeMap (or similar with range scan api) ? WAS: Best ways to achieve throughput, for large M:N ratio of STM threads, with hot TVar updates? >> Date: 2020-07-30 at 21:28:31 GMT+8 >> To: Haskell Cafe > >> Reply-To: YueCompl > >> >> For the record, overhead of STM over IO (or other means where manual composition of transactions needed) based concurrency control, is a price I'm willing to pay in my use case, as it's not machine-performance critical in distributing input data + parameters to a cluster of worker nodes, and collecting their results into permanent storage or a data pipeline. But to keep professionally crafting well synced, race-free scheduling code is barely affordable by my org, as shape of datasets, relationship between them and algorithms processing them are varying at fast paces, we have difficulty, or lack the willingness, to hire some workforce specifically to keep each new data pipeline race free, it has to be, but better at cost of machine-hours instead of human head counts. >> >> While easily compositing stm code, wrapped in scriptable procedures, will enable our analysts to author the scheduling scripts without too much concerns. Then our programmers can focus on performance critical parts of the data processing code, like optimization of tight-loops. >> >> Only if not in the tight loops, I think it's acceptable by us, that up to 2~3 order of magnitude slower for an stm solution compared to its best rivals, as long as it's scalable. For a (maybe cheating) example, if fully optimized code can return result in 10 ms after an analyst clicked a button, we don't bother if unoptimized stm script needs 10 second, so long as the result is correct. >> >> In a philosophic thinking, I heard that AT&T had UNIX specifically designed for their Control panel, while their Data panel runs separate software (and on separate hardware obviously), while modern systems have powerful CPUs tempting us to squeeze more performance out of it, and SIMD instructions make it even more tempting, I think we'd better resist it when programming something belong to the Control panel per se, but do it in programming something belong to the Data panel. And appears Data panel programs are being shifted to GPUs nowadays, which feels right. >> >> Regards, >> Compl >> >> >>> On 2020-07-30, at 20:10, YueCompl via Haskell-Cafe > wrote: >>> >>> Hi Peter, >>> >>> Great to hear from you! >>> >>> For the record tskiplist (and stm-containers together) did improve my situation a great lot with respect to scalability at concurrency/parallelism! >>> >>> I'm still far from the stage to squeeze last drops of performance, currently I'm just making sure performance wise concerns be reasonable during my PoC in correctness and ergonomics of my HPC architecture (an in-memory graph + out-of-core (mmap) array DBMS powered computation cluster, with shared storage), and after parallelism appears acceptable, I seemingly suffer from serious GC issue at up scaling on process working memory size. I'm suspecting it's because of the added more TVars and/or aggressive circular structures of them in my case, and can not find a way to overcome it by far. >>> >>> Thanks for your detailed information! >>> >>> Best regards, >>> Compl >>> >>> >>>> On 2020-07-30, at 19:19, Peter Robinson > wrote: >>>> >>>> Hi Compl, >>>> >+ This package provides a proof-of-concept implementation of a skip list in STM >>>> >>>> This has to mean something but I can't figure out yet. >>>> >>>> Dear Peter Robinson, I hope you can see this message and get in the loop of discussion. >>>> >>>> >>>> The reason for adding this sentence was that tskiplist hasn't been optimized for production use. Later on, I wrote an implementation of a concurrent skip list with atomic operations that performs significantly better, but it's operations work in the IO monad. >>>> >>>> I'm surprised to hear that you're getting poor performance even when using the stm-container package, which I believe was meant to be used in production. A while ago, I ran some benchmarks comparing concurrent dictionary data structures (such as stm-container) under various workloads. While STMContainers.Map wasn't as fast as the concurrent-hashtable package, the results indicate that the performance doesn't degrade too much under larger workloads. >>>> >>>> You can find these benchmark results here (10^6 randomly generated insertion/deletion/lookup requests distributed among 32 threads): >>>> https://lowerbound.io/blog/bench2-32.html >>>> And some explanations about the benchmarks are here: >>>> https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html >>>> >>>> One issue that I came across when implementing the tskiplist package was this: If a thread wants to insert some item into the skip list, it needs to search for the entry point by performing readTVar operations starting at the list head. So, on average, a thread will read O(log n) TVars (assuming a skip list of n items) and, if any of these O(log n) TVars are modified by a simultaneously running thread, the STM runtime will observe a (false) conflict and rerun the transaction. It's not clear to me how to resolve this issue without access to something like unreadTVar (see [1]). >>>> >>>> Best, >>>> Peter >>>> >>>> [1] UnreadTVar: Extending Haskell Software Transactional Memory for Performance (2007) by Nehir Sonmez , Cristian Perfumo , Srdjan Stipic , Adrian Cristal , Osman S. Unsal , Mateo Valero. >>>> >>>> _______________________________________________ >>>> Haskell-Cafe mailing list >>>> To (un)subscribe, modify options or view archives go to: >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >>>> Only members subscribed via the mailman list are allowed to post. >>> >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> To (un)subscribe, modify options or view archives go to: >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >>> Only members subscribed via the mailman list are allowed to post. >> >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >> Only members subscribed via the mailman list are allowed to post. > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri Jul 31 13:27:57 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 31 Jul 2020 13:27:57 +0000 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> <20200728063746.GA2913@octa> Message-ID: My real question is this: * Who “owns” (as in: takes responsibility for) the GHC API? If no person, or no group of people, feels responsible for it, it’ll just grow in a disorganised fashion, which is why we are where we are. This discussion thread has focused mainly on stability, but there is more to it than that. For example: * What kind of stability strikes a reasonable balance between developers and clients? Personally I think we should have * A public API that is pretty stable, plus * Access to many internal GHC functions that change quite a bit. Heavily used internal functions can get “blessed” and moved to the public API. Someone needs to do the blessing. * What is a good public API? Where is it written down? * What should GHC’s extensibility interface be like? Plugins and all that. What is a good design for (say) extensible interface files? What “hooks” should the GHC API afford? This is more than just “what arguments should this function take”… it’s a matter of fundamental design. But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points. * What is a good design for the plugins story more generally? I’d be quite content with a fairly informal group to act as the “owner”. But I think we would be better served if we had such a group, and knew who they were. Simon From: ghc-devs On Behalf Of Neil Mitchell Sent: 28 July 2020 09:19 To: GHC developers Subject: Re: How should we treat changes to the GHC API? The API changing regularly is a real hassle for people building on top of it. But, the people building on top of it are also some of the people changing it most. And the API isn't nice and beautiful as it stands, with lots of horrible work-arounds for downstream users. I'd suggest letting people change the API freely. But avoid making bikeshed API changes (those which are minor improvements at best, and really just a different flavour of the same). And perhaps loop in heavy API users when removing functionality. Thanks, Neil On Tue, Jul 28, 2020 at 7:38 AM Daniel Trstenjak > wrote: On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote: > On the other hand, we could imagine a dedicated GHC API volunteer who > maintains the API on top of the shifting sands of GHC. Looking at other compilers that have been successful in having a stable API - like clang with the libclang - that's pretty much how they achieved it. The manpower of GHC devs is already pretty small, so putting another burden on them with a stable API won't work out that well. Also as a compiler dev you want to have the freedom to change your AST in the ways you need it, to incorporate new features. Having workarounds at this level to ensure API stability just seems to be the wrong place and will only increase the complexity in an already quite complex project. However a high level API on top of the AST is the perfect place for such special cases between different compiler versions. Greetings, Daniel _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Fri Jul 31 14:16:36 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Fri, 31 Jul 2020 22:16:36 +0800 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> <20200728063746.GA2913@octa> Message-ID: I think this is the core issue here: > What should GHC’s extensibility interface be like? Plugins and all that. What is a good design for (say) extensible interface files? What “hooks” should the GHC API afford? This is more than just “what arguments should this function take”… it’s a matter of fundamental design. But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points. I don't think we know this apriori, and it will be discovered over time as more and more producers and consumers start making use of it. Is the current design the best one? I have my doubts, is it a first approximation? I think so. I think these features are more discovered than designed. It's easier to iterate over concrete implementations in this area than over abstract ideas. I would propose having some EXPERIMENTAL markers for these kinds of features. I do agree that a group of people who feel strongly about this should be listed in the CODEOWNERS file for the respective parts of the codebase, and take an active part in code review. From John.Ericson at Obsidian.Systems Fri Jul 31 18:09:19 2020 From: John.Ericson at Obsidian.Systems (John Cotton Ericson) Date: Fri, 31 Jul 2020 14:09:19 -0400 Subject: How should we treat changes to the GHC API? In-Reply-To: References: <60ba3fbf-e30c-487a-b914-0ca4c8bfd7f0@www.fastmail.com> <010f01739210aa25-cd9bd468-9c5c-4f09-9f4c-5cf88d0e8df6-000000@us-east-2.amazonses.com> <20200728063746.GA2913@octa> Message-ID: On 7/31/20 9:27 AM, Simon Peyton Jones via ghc-devs wrote: > > My real question is this: > > * Who “owns” (as in: takes responsibility for) the GHC API? > I guess one thing very important to me is that the architecture of GHC and it's public interfaces will always be deeply intertwined; or put a different way, efforts to manager the public interface as a thin veneer over a big black box will not work. > * What *is* a good public API?  Where is it written down? > This I agree is important to discuss. I do agree with Moritz that a lot of this stuff can be evolved, but the general direction can be still be discussed. For example, the basic objects I offered are a *huge* departure from what we have today, and any huge change should be discussed up front a bit. > * What should GHC’s extensibility interface be like?   Plugins and > all that.  What is a good design for (say) extensible interface > files?  What “hooks” should the GHC API afford? This is more than > just “what arguments should this function take”… it’s a matter of > fundamental design.   But design questions like this belong in the > GHC-API world (not the core GHC world) because they are all about > extension points. > * What is a good design for the plugins story more generally? > I've mentioned this before but I think plugins/hooks/etc. are a terrible way to reuse GHC for other purposes, and exist as a symptom of the rest of the compiler not being at all modular. It's fine that we continue to support them in the short term, but in the long term I'd really like to have a plan to obviate them completely. (One can search "composition over configuration" and variations on that slogan to find much ink has been spilled on this general principle.) John -------------- next part -------------- An HTML attachment was scrubbed... URL: