From pi.boy.travis at gmail.com Fri May 1 08:18:31 2020 From: pi.boy.travis at gmail.com (Travis Whitaker) Date: Fri, 1 May 2020 01:18:31 -0700 Subject: Justifying sched_yield() in the RTS Message-ID: Hello GHC devs, Through the course of reading some recent material posted by Linus Torvalds ^1 ^2, I remembered stumbling upon GHC Trac #3553 ^3 some years ago. Looking past the harsh tone Linus used in his notes, I think he makes some pretty reasonable points about the problems that can be caused by using spinlocks in userspace. Specifically: - A group of threads yielding while waiting for a spinlock are, in effect, simply trying to coax the scheduler into scheduling the thread that is holding the lock. This is much easier for the scheduler to do correctly with a futex or similar, since the scheduler has some context around which tasks are blocking/waking each other up. - Apparently the Linux scheduler (and maybe other schedulers) has some smarts for preserving cache locality while choosing which physical execution units run which tasks, and reordering threads in the run list in an ad-hoc way with sched_yield() interferes with this mechanism. - Using sched_yield() (or other random interference with run lists) can cause disproportionate havoc on NUMA systems, where jostling around the lock-holding thread by burning time slices and yielding is especially costly. All that said, there is perhaps one concrete advantage (aside from absolute performance) to the current spinlock implementation: the only functionality it requires from the host OS is a yield-like operation. A yielding spinlock occupies a comfortable local optimum, achieving decent performance across platforms with minimal exposure to cross-OS API differences. Revisiting #3553, it seems the only reason that a futex was not used in place of a spinlock with sched_yield() was that, despite all of the points against it, the spinlock code empirically performed better. However, these tests were performed years ago and the futex implementation in the Linux kernel has changed quite a bit. I think there is a healthy amount of empirical evidence from the GHC user community that there are problems afoot with the way parallel GC does locking, and indeed I wonder if the bad cases Linus described are playing a role. Advice like "use -qg" or "use -qnM with small M" is often parroted (this Twitter thread ^4 contains both pieces of advice). Furthermore, I would wager that an RTS using futexes for locking rather than spinning plays nicer with other CPU intensive tasks on the same machine. The "scheduler storm" caused by a large number of threads waiting for a spinlock could have a negative impact on neighboring processes, e.g. a large number of HECs spinning certainly don't do any favors for a busy neighboring DB process. I'm curious if others have thoughts on reviving the futex investigation. Perhaps the futexes provided by modern Linux are more competitive with the current spinlock implementation, or perhaps better scaling on machines with high core counts is worth some cost. In the case that futexes remain handily defeated by yielding spinlocks, it's a worthy endeavor to explain precisely _why_ this happens. Other programs have seen performance issues crop up when the kernel makes subtle changes to how sched_yield() works. ^5 1: https://www.realworldtech.com/forum/?threadid=189711&curpostid=189723 2: https://www.realworldtech.com/forum/?threadid=189711&curpostid=189752 3: https://gitlab.haskell.org/ghc/ghc/issues/3553 4: https://twitter.com/ndm_haskell/status/1076187051610042368?s=20 5: https://lwn.net/Articles/31462/ From jaskiewiczs at icloud.com Fri May 1 09:09:28 2020 From: jaskiewiczs at icloud.com (Sergej Jaskiewicz) Date: Fri, 1 May 2020 12:09:28 +0300 Subject: [RFC] Compiler pipeline timings visualization Message-ID: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> tl;dr: I propose adding a new GHC flag for generating a log that allows visualizing how much time each stage in the compiler pipleline took, similar to Clang's -ftime-trace. Hello, everyone. I'd like to tell you about a feature I've been working on recently, namely, the ability to generate and visualize how long each stage of the compilation pipeline takes to complete. This is basically about the same as the -ftime-trace flag that has landed in Clang several months ago [1]. The initial motivation for this feature was the desire to understand why compilation times are so long when using LLVM as backend. But later I realized that this functionality is useful on its own for end users, not just GHC devs, so it would make sense to add a new flag -ddump-time-trace. Since not only does Haskell have complex type system, but also there is a variety of language extensions, we, the Haskell users, often experience very long compilation times. For instance, the usage of the TypeFaimilies extension can considerably slow down the compilation process [2]. It is useful to understand how you can fix your code so that it compiles faster. There are two options for that right now: - Building GHC from source for profiling and using the just-built GHC for compiling your problem code. - Building the compiler from source with event log support [3]. The former option just doesn't do it, since the output would be "which GHC function calls took how much time", but there'd be no information about which part of the user code was being compiled. The latter option is much closer to what we need. If we link the GHC executable with the -eventlog flag, then various significant events will be written to a special log file. For example, "Parser began parsing the Main.hs file after 5 ms since GHC has started, and ended parsing it 3 ms after that". The resulting log file can be parsed with the ghc-events library [4], and also visualized using the ThreadScope app [5]. Bu this approach has its disadvantages. Firstly, if the user wants visualization, they'd have to install ThreadScope. Some companies' internal policies prohibit installing third-party apps from the internet. It would be good if we could generate a log that could be visualized on any computer with a browser. For that we could use the Chrome Trace Event format [6]. This is an ordinary JSON file with a specific structure that can be viewed in the Chrome browser by going to the chrome://tracing page, or on https://speedscope.app. A file in exactly this format would be generated by Clang if you passed it the -ftime-trace flag. Secondly, the event log contains many events that are not relevant to the end user, for example, thread creation, memory allocation, etc. As an initial proof of concept, I've developed a command line tool for transforming event log files to Chrome Trace Event files [7]. Thirdly, in order for the event log to be generated, you'd still have to build GHC from source. The majority of the GHC users won't go this way. Not only would it require some basic understanding of the GHC building process, but also building itself takes quite some time. It would be great if the needed functionality came with GHC out of the box. This is why I've added support for generating Trace Event files into my fork of GHC [8], and I would like to propose including this feature into the mainline GHC. (Note that my implementation is still a bit buggy, for example, it only works in -j mode. This will be fixed.) You can now execute the following command: ghc -ddump-to-file -ddump-file-prefix="Main." -ddump-time-trace -j -O3 Main.hs And receive this image [9]. Here, we've compiled two modules, one of which depends on the other. One more difference from event log is that now various metadata (like file and module names) are emitted as a separate JSON attribute, instead of being encoded in the name of an event. For example, parsing the Main.hs file and parsing the QSort.hs file in one compilation are events with the same name, but different metadata. We can group them together if we want to know how much time the parsing took overall. The event log format doesn't allow us to do it. So, we can now generate a Trace Event file from a single GHC invocation. However, most projects use build systems that invoke the compiler many times. It would be good if we could form a log for the whole project. This is solved by merging logs. However, there is one subtlety: the events in logs use relative timestamps (the number of microseconds since the process has started). To combine logs from multiple processes, we add a clock synchronization point into each trace log in the form of an additional 'beginningOfTime' JSON attribute that contains the absolute time when the proccess has started. There is a Python script that performs the actual merging [10]. This is how it looks like with multiple processes [11]. Also, I've implemented generation of the 'beginningOfTime' attribute in LLVM [12], so build systems could take advantage of that and combine GHC trace logs with llc/opt trace logs when GHC uses LLVM as backend. Thoughts? Sergej. [1] https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/ [2] https://gitlab.haskell.org/ghc/ghc/issues/8095 [3] https://gitlab.haskell.org/ghc/ghc/-/wikis/event-log [4] http://hackage.haskell.org/package/ghc-events [5] https://wiki.haskell.org/ThreadScope [6] https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit [7] https://github.com/broadwaylamb/ghc-eventlog-chrome [8] https://gitlab.haskell.org/broadwaylamb/ghc/-/commits/timetrace-better-metadata [9] https://user-images.githubusercontent.com/16309982/79079705-39775e00-7d19-11ea-9507-eb0f11581c63.png [10] https://github.com/broadwaylamb/merge_trace_events [11] https://user-images.githubusercontent.com/16309982/79080338-ad673580-7d1c-11ea-9e30-5e6f72e77555.png [12] https://github.com/llvm/llvm-project/commit/2899103108d38215af8aae377cd0e54794278209 From klebinger.andreas at gmx.at Fri May 1 10:04:10 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Fri, 1 May 2020 12:04:10 +0200 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> Message-ID: Hi Sergej, I think this is a good idea in general, and it seems you did some great work there already. Something like this can also help with pinpointing performance issues inside the compiler so would not just be useful to end users. intuitively I would assume that instead of adding another way to produce compiler events we should: * Ship GHC with eventlog enabled by default * Provide an official converter from eventLog to the chrome trace format. The eventlog format is quite flexible, and if it's unsuitable to what you want I would prefer to extend it rather than adding support for additional formats inside GHC itself. This way we: * Continue to have one unified way to dump events from haskell apps (the eventlog) * Users need not go to third party apps, as the converter could reasonably come with GHC (like hp2ps) * We are free to include information that can't (easily) be encoded in the chrome format. The downside is that users will have to invoke ghc, followed by some other tool to get the chrome trace, but to me that seems like a worthwhile tradeoff for keeping the compiler and various visualization formats somewhat separated. The obvious question there is how much enabling the eventlog by default would impact non-logging ghc invocations. I have not measured this and it might rule out this approach if it has a big impact and isn't easily corrected. As a last point I want to encourage you to open a ticket about this. Mailing list threads tend to be harder to follow and find down the line then tickets in my experience. Cheers, Andreas Sergej Jaskiewicz via ghc-devs schrieb am 01.05.2020 um 11:09: > tl;dr: I propose adding a new GHC flag for generating a log that allows > visualizing how much time each stage in the compiler pipleline took, similar > to Clang's -ftime-trace. > > Hello, everyone. > > I'd like to tell you about a feature I've been working on recently, namely, > the ability to generate and visualize how long each stage of the compilation > pipeline takes to complete. This is basically about the same as > the -ftime-trace flag that has landed in Clang several months ago [1]. > > The initial motivation for this feature was the desire to understand why > compilation times are so long when using LLVM as backend. But later I realized > that this functionality is useful on its own for end users, not just GHC devs, > so it would make sense to add a new flag -ddump-time-trace. > > Since not only does Haskell have complex type system, but also there is > a variety of language extensions, we, the Haskell users, often experience > very long compilation times. For instance, the usage of the TypeFaimilies > extension can considerably slow down the compilation process [2]. > It is useful to understand how you can fix your code so that it compiles faster. > > There are two options for that right now: > - Building GHC from source for profiling and using the just-built GHC for > compiling your problem code. > - Building the compiler from source with event log support [3]. > > The former option just doesn't do it, since the output would be > "which GHC function calls took how much time", but there'd be no information > about which part of the user code was being compiled. > > The latter option is much closer to what we need. If we link the GHC executable > with the -eventlog flag, then various significant events will be written to > a special log file. For example, "Parser began parsing the Main.hs file after > 5 ms since GHC has started, and ended parsing it 3 ms after that". > The resulting log file can be parsed with the ghc-events library [4], and also > visualized using the ThreadScope app [5]. > > Bu this approach has its disadvantages. > > Firstly, if the user wants visualization, they'd have to install ThreadScope. > Some companies' internal policies prohibit installing third-party apps from > the internet. It would be good if we could generate a log that could be > visualized on any computer with a browser. For that we could use > the Chrome Trace Event format [6]. This is an ordinary JSON file with a specific > structure that can be viewed in the Chrome browser by going to > the chrome://tracing page, or on https://speedscope.app. A file in exactly this > format would be generated by Clang if you passed it the -ftime-trace flag. > > Secondly, the event log contains many events that are not relevant to the end > user, for example, thread creation, memory allocation, etc. > > As an initial proof of concept, I've developed a command line tool for > transforming event log files to Chrome Trace Event files [7]. > > Thirdly, in order for the event log to be generated, you'd still have to build > GHC from source. The majority of the GHC users won't go this way. Not only > would it require some basic understanding of the GHC building process, but also > building itself takes quite some time. It would be great if the needed > functionality came with GHC out of the box. > > This is why I've added support for generating Trace Event files into my fork > of GHC [8], and I would like to propose including this feature into the mainline > GHC. > > (Note that my implementation is still a bit buggy, for example, it only works > in -j mode. This will be fixed.) > > You can now execute the following command: > > ghc -ddump-to-file -ddump-file-prefix="Main." -ddump-time-trace -j -O3 Main.hs > > And receive this image [9]. Here, we've compiled two modules, one of which > depends on the other. > > One more difference from event log is that now various metadata > (like file and module names) are emitted as a separate JSON attribute, instead > of being encoded in the name of an event. For example, parsing the Main.hs file > and parsing the QSort.hs file in one compilation are events with the same name, > but different metadata. We can group them together if we want to know how much > time the parsing took overall. The event log format doesn't allow us to do it. > > So, we can now generate a Trace Event file from a single GHC invocation. > However, most projects use build systems that invoke the compiler many times. > It would be good if we could form a log for the whole project. > > This is solved by merging logs. However, there is one subtlety: the events in > logs use relative timestamps (the number of microseconds since the process > has started). To combine logs from multiple processes, we add a clock > synchronization point into each trace log in the form of an additional > 'beginningOfTime' JSON attribute that contains the absolute time when > the proccess has started. > > There is a Python script that performs the actual merging [10]. > > This is how it looks like with multiple processes [11]. > > Also, I've implemented generation of the 'beginningOfTime' attribute in > LLVM [12], so build systems could take advantage of that and combine GHC trace > logs with llc/opt trace logs when GHC uses LLVM as backend. > > Thoughts? > > Sergej. > > [1] https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/ > [2] https://gitlab.haskell.org/ghc/ghc/issues/8095 > [3] https://gitlab.haskell.org/ghc/ghc/-/wikis/event-log > [4] http://hackage.haskell.org/package/ghc-events > [5] https://wiki.haskell.org/ThreadScope > [6] https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit > [7] https://github.com/broadwaylamb/ghc-eventlog-chrome > [8] https://gitlab.haskell.org/broadwaylamb/ghc/-/commits/timetrace-better-metadata > [9] https://user-images.githubusercontent.com/16309982/79079705-39775e00-7d19-11ea-9507-eb0f11581c63.png > [10] https://github.com/broadwaylamb/merge_trace_events > [11] https://user-images.githubusercontent.com/16309982/79080338-ad673580-7d1c-11ea-9e30-5e6f72e77555.png > [12] https://github.com/llvm/llvm-project/commit/2899103108d38215af8aae377cd0e54794278209 > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From simonpj at microsoft.com Fri May 1 10:10:29 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 1 May 2020 10:10:29 +0000 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> Message-ID: This sounds great! Event logging needn't add much cost if we don't log much -- eg just compiler phases. It simply depends on how many events you generate! For naïve uses, the log you show in [9] is a bit complicated. Something even simpler might be good. Simon | -----Original Message----- | From: ghc-devs On Behalf Of Andreas | Klebinger | Sent: 01 May 2020 11:04 | To: Sergej Jaskiewicz | Cc: ghc-devs at haskell.org | Subject: Re: [RFC] Compiler pipeline timings visualization | | Hi Sergej, | | I think this is a good idea in general, and it seems you did some great | work there already. | Something like this can also help with pinpointing performance issues | inside the compiler | so would not just be useful to end users. | | intuitively I would assume that instead of adding another way | to produce compiler events we should: | * Ship GHC with eventlog enabled by default | * Provide an official converter from eventLog to the chrome trace format. | | The eventlog format is quite flexible, and if it's unsuitable to what | you want I would | prefer to extend it rather than adding support for additional formats | inside GHC itself. | | This way we: | * Continue to have one unified way to dump events from haskell apps (the | eventlog) | * Users need not go to third party apps, as the converter could | reasonably come with GHC (like hp2ps) | * We are free to include information that can't (easily) be encoded in | the chrome format. | | The downside is that users will have to invoke ghc, followed by some | other tool to get the | chrome trace, but to me that seems like a worthwhile tradeoff for | keeping the compiler and | various visualization formats somewhat separated. | | The obvious question there is how much enabling the eventlog by default | would impact non-logging ghc | invocations. I have not measured this and it might rule out this | approach if it has a big impact and isn't | easily corrected. | | As a last point I want to encourage you to open a ticket about this. | Mailing list threads tend to be harder to follow and find down the line | then tickets in my experience. | | Cheers, | Andreas | | Sergej Jaskiewicz via ghc-devs schrieb am 01.05.2020 um 11:09: | > tl;dr: I propose adding a new GHC flag for generating a log that allows | > visualizing how much time each stage in the compiler pipleline took, | similar | > to Clang's -ftime-trace. | > | > Hello, everyone. | > | > I'd like to tell you about a feature I've been working on recently, | namely, | > the ability to generate and visualize how long each stage of the | compilation | > pipeline takes to complete. This is basically about the same as | > the -ftime-trace flag that has landed in Clang several months ago [1]. | > | > The initial motivation for this feature was the desire to understand | why | > compilation times are so long when using LLVM as backend. But later I | realized | > that this functionality is useful on its own for end users, not just | GHC devs, | > so it would make sense to add a new flag -ddump-time-trace. | > | > Since not only does Haskell have complex type system, but also there is | > a variety of language extensions, we, the Haskell users, often | experience | > very long compilation times. For instance, the usage of the | TypeFaimilies | > extension can considerably slow down the compilation process [2]. | > It is useful to understand how you can fix your code so that it | compiles faster. | > | > There are two options for that right now: | > - Building GHC from source for profiling and using the just-built GHC | for | > compiling your problem code. | > - Building the compiler from source with event log support [3]. | > | > The former option just doesn't do it, since the output would be | > "which GHC function calls took how much time", but there'd be no | information | > about which part of the user code was being compiled. | > | > The latter option is much closer to what we need. If we link the GHC | executable | > with the -eventlog flag, then various significant events will be | written to | > a special log file. For example, "Parser began parsing the Main.hs file | after | > 5 ms since GHC has started, and ended parsing it 3 ms after that". | > The resulting log file can be parsed with the ghc-events library [4], | and also | > visualized using the ThreadScope app [5]. | > | > Bu this approach has its disadvantages. | > | > Firstly, if the user wants visualization, they'd have to install | ThreadScope. | > Some companies' internal policies prohibit installing third-party apps | from | > the internet. It would be good if we could generate a log that could be | > visualized on any computer with a browser. For that we could use | > the Chrome Trace Event format [6]. This is an ordinary JSON file with a | specific | > structure that can be viewed in the Chrome browser by going to | > the chrome://tracing page, or on https://speedscope.app. A file in | exactly this | > format would be generated by Clang if you passed it the -ftime-trace | flag. | > | > Secondly, the event log contains many events that are not relevant to | the end | > user, for example, thread creation, memory allocation, etc. | > | > As an initial proof of concept, I've developed a command line tool for | > transforming event log files to Chrome Trace Event files [7]. | > | > Thirdly, in order for the event log to be generated, you'd still have | to build | > GHC from source. The majority of the GHC users won't go this way. Not | only | > would it require some basic understanding of the GHC building process, | but also | > building itself takes quite some time. It would be great if the needed | > functionality came with GHC out of the box. | > | > This is why I've added support for generating Trace Event files into my | fork | > of GHC [8], and I would like to propose including this feature into the | mainline | > GHC. | > | > (Note that my implementation is still a bit buggy, for example, it only | works | > in -j mode. This will be fixed.) | > | > You can now execute the following command: | > | > ghc -ddump-to-file -ddump-file-prefix="Main." -ddump-time-trace -j -O3 | Main.hs | > | > And receive this image [9]. Here, we've compiled two modules, one of | which | > depends on the other. | > | > One more difference from event log is that now various metadata | > (like file and module names) are emitted as a separate JSON attribute, | instead | > of being encoded in the name of an event. For example, parsing the | Main.hs file | > and parsing the QSort.hs file in one compilation are events with the | same name, | > but different metadata. We can group them together if we want to know | how much | > time the parsing took overall. The event log format doesn't allow us to | do it. | > | > So, we can now generate a Trace Event file from a single GHC | invocation. | > However, most projects use build systems that invoke the compiler many | times. | > It would be good if we could form a log for the whole project. | > | > This is solved by merging logs. However, there is one subtlety: the | events in | > logs use relative timestamps (the number of microseconds since the | process | > has started). To combine logs from multiple processes, we add a clock | > synchronization point into each trace log in the form of an additional | > 'beginningOfTime' JSON attribute that contains the absolute time when | > the proccess has started. | > | > There is a Python script that performs the actual merging [10]. | > | > This is how it looks like with multiple processes [11]. | > | > Also, I've implemented generation of the 'beginningOfTime' attribute in | > LLVM [12], so build systems could take advantage of that and combine | GHC trace | > logs with llc/opt trace logs when GHC uses LLVM as backend. | > | > Thoughts? | > | > Sergej. | > | > [1] https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame- | chart-profiler-for-Clang/ | > [2] https://gitlab.haskell.org/ghc/ghc/issues/8095 | > [3] https://gitlab.haskell.org/ghc/ghc/-/wikis/event-log | > [4] http://hackage.haskell.org/package/ghc-events | > [5] https://wiki.haskell.org/ThreadScope | > [6] https://docs.google.com/document/d/1CvAClvFfyA5R- | PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit | > [7] https://github.com/broadwaylamb/ghc-eventlog-chrome | > [8] https://gitlab.haskell.org/broadwaylamb/ghc/-/commits/timetrace- | better-metadata | > [9] https://user-images.githubusercontent.com/16309982/79079705- | 39775e00-7d19-11ea-9507-eb0f11581c63.png | > [10] https://github.com/broadwaylamb/merge_trace_events | > [11] https://user-images.githubusercontent.com/16309982/79080338- | ad673580-7d1c-11ea-9e30-5e6f72e77555.png | > [12] https://github.com/llvm/llvm- | project/commit/2899103108d38215af8aae377cd0e54794278209 | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs at haskell.org | > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs | | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From mail at nh2.me Fri May 1 12:28:51 2020 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Fri, 1 May 2020 14:28:51 +0200 Subject: Justifying sched_yield() in the RTS In-Reply-To: References: Message-ID: There are more related updates in https://gitlab.haskell.org/ghc/ghc/issues/9221, also including a short discussion of Linus's post. Simon Marlow's overall response was: > I'm very supportive of making this better, but as usual I will require thorough data to back up any changes :) > > Everything I tried in the past made things worse. Including an experiment I did to use futexes directly: https://gitlab.haskell.org/ghc/ghc/issues/3553?cversion=0&cnum_hist=14#note_39009 So it sounds like this topic is currently in the stage of: Somebody needs to take the time to re-do that benchmark done 10 years ago. From simonpj at microsoft.com Fri May 1 15:27:02 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 1 May 2020 15:27:02 +0000 Subject: git prune? Message-ID: I'm getting lots of the messages below. Running 'git prune' makes no difference. Simon simonpj at MSRC-3645512:~/code/HEAD-5/testsuite/tests/th$ git push --set-upstream origin wip/T18121 Counting objects: 13, done. Delta compression using up to 20 threads. Compressing objects: 100% (13/13), done. Writing objects: 100% (13/13), 1.64 KiB | 418.00 KiB/s, done. Total 13 (delta 11), reused 0 (delta 0) remote: remote: To create a merge request for wip/T18121, visit: remote: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/new?merge_request%5Bsource_branch%5D=wip%2FT18121 remote: remote: warning: The last gc run reported the following. Please correct the root cause remote: and remove gc.log. remote: Automatic cleanup will not be performed until the file is removed. remote: remote: warning: There are too many unreachable loose objects; run 'git prune' to remove them. remote: -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Fri May 1 16:01:17 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 01 May 2020 12:01:17 -0400 Subject: git prune? In-Reply-To: References: Message-ID: <87ftcj4pbs.fsf@smart-cactus.org> Simon Peyton Jones via ghc-devs writes: > I'm getting lots of the messages below. > Running 'git prune' makes no difference. > Simon > Indeed, the warning is on the server side (hence the "remote:" prefix). I'll need to take care of this on the server. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Fri May 1 16:22:47 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 01 May 2020 12:22:47 -0400 Subject: Justifying sched_yield() in the RTS In-Reply-To: References: Message-ID: <87bln74obz.fsf@smart-cactus.org> Travis Whitaker writes: > Hello GHC devs, > > Through the course of reading some recent material posted by Linus > Torvalds ^1 ^2, I remembered stumbling upon GHC Trac #3553 ^3 some > years ago. > For what it's worth Simon Marlow and I discussed this a few weeks ago and he agreed that it would be worth re-running the futex experiments. I do suspect there is good money on the table here on larger/busier machines. It would be great if someone could pick this up! Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Fri May 1 16:25:05 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 01 May 2020 12:25:05 -0400 Subject: Justifying sched_yield() in the RTS In-Reply-To: <87bln74obz.fsf@smart-cactus.org> References: <87bln74obz.fsf@smart-cactus.org> Message-ID: <878sib4o81.fsf@smart-cactus.org> Ben Gamari writes: > Travis Whitaker writes: > >> Hello GHC devs, >> >> Through the course of reading some recent material posted by Linus >> Torvalds ^1 ^2, I remembered stumbling upon GHC Trac #3553 ^3 some >> years ago. >> > For what it's worth Simon Marlow and I discussed this a few weeks ago > and he agreed that it would be worth re-running the futex experiments. Whoops. The above should have read "mutex" experiments. Of course, perhaps direct futex usage is also worth evaluating but simpler is better if there is no performance trade-off. Cheerss, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at smart-cactus.org Fri May 1 16:36:16 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 01 May 2020 12:36:16 -0400 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> Message-ID: <875zdf4nph.fsf@smart-cactus.org> Andreas Klebinger writes: > Hi Sergej, > > I think this is a good idea in general, and it seems you did some great > work there already. > Something like this can also help with pinpointing performance issues > inside the compiler > so would not just be useful to end users. > > intuitively I would assume that instead of adding another way > to produce compiler events we should: > * Ship GHC with eventlog enabled by default For what it's worth, I have also been thinking about proposing this. I've been doing a lot of performance characterisation recently where I would have liked to use our binary distributions. Sadly this isn't an option as the ghc executable doesn't support the eventlog. I've not taken measurements on the eventlog overhead in ghc-bin but I suspect that any overhead that *does* exist can be easily eliminated (as I describe in #17949). We just need to ensure that Debug.Trace.{traceEvent,traceMarker} know not to pack their buffers if their respective tracing flags aren't enabled. Frankly, I also question the value of shipping the non-tracing-enabled RTS at all. Enabling tracing by default would allow us to eliminate three whole RTS builds from CI, which I suspect would be worthwhile. The size overhead on a statically-linked executable appears to be fairly small in the grand scheme of things: -rw-r--r-- 1 ben users 6.0M Apr 26 15:11 libHSrts-1.0.a -rw-r--r-- 1 ben users 6.4M Apr 26 15:21 libHSrts-1.0_l.a -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri May 1 16:54:59 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 01 May 2020 12:54:59 -0400 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> Message-ID: <874ksz4muc.fsf@smart-cactus.org> Sergej Jaskiewicz via ghc-devs writes: > tl;dr: I propose adding a new GHC flag for generating a log that allows > visualizing how much time each stage in the compiler pipleline took, similar > to Clang's -ftime-trace. > > Hello, everyone. Hi Sergej, [snip] > The latter option is much closer to what we need. If we link the GHC executable > with the -eventlog flag, then various significant events will be written to > a special log file. For example, "Parser began parsing the Main.hs file after > 5 ms since GHC has started, and ended parsing it 3 ms after that". > The resulting log file can be parsed with the ghc-events library [4], and also > visualized using the ThreadScope app [5]. > I'm a bit short on time but here are a few points in no particular order: Out of curiosity, have you seen Alp's work [1] in this area? This work allows use to the ghc-events-analyze package [2] to visualize (e.g. [3]) the existing withTiming eventlog output in a reasonably easy-to-consume format. That being said, I can certainly see the benefit of using the Chrome tracing infrastructure for this; it would make for a nicer user experience than the static image that ghc-events-analyze spits out. I also know that Matthew Pickering has been working in this area and I'm sure will have something interesting to add. I will admit that I am a bit reluctant to add support for a *particular* tracing format to GHC itself. I think it would make the most sense for GHC to produce a consumer-agnostic trace representation (via our existing eventlog mechanism) and for users to transform this into the format their tools require. Our current withTimings infrastructure is quite ad-hoc and could perhaps expose more detail. It would be great to know what sorts of detail people would find useful. Cheers, - Ben [1] https://www.haskell.org/ghc/blog/20190924-eventful-ghc.html [2] https://hackage.haskell.org/package/ghc-events-analyze [3] https://www.haskell.org/ghc/blog/images/eventful-ghc/ghc-events-viz.svg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From lexi.lambda at gmail.com Fri May 1 21:36:15 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Fri, 1 May 2020 16:36:15 -0500 Subject: Expanding a particular type family in type errors Message-ID: Hi all, As part of my ongoing rework of arrow notation , I have introduced two wired-in type families, which assist in the typechecking process. The details are not terribly important (you can read the proposal if you’d like for those), but the relevant detail is that equalities of the shape ArrowStackTup a ~ b are produced by the typechecker in key places. ArrowStackTup is one of my wired-in type families, and it’s really an implementation detail. Unfortunately, I often end up with type errors that leak this detail, with expected/actual types like these ones: Expected type: ArrowStackTup '[Int] -> Bool Actual type: Int -> String This is quite annoying, as it’s exceedingly rare that these type families actually get stuck, so they can almost always be expanded in type errors. As it happens, `ArrowStackTup '[a]` expands to simply `a`, so the type error I would like to report is this one: Expected type: Int -> Bool Actual type: Int -> String Not technically challenging, but I find myself faced with the question of where this special expansion logic ought to go. It seems like it could go in any of several places: It could be hard-wired into pprType, and perhaps selectively disabled with an -fprint-* flag. This is nice in that it’s universal, but it’s almost certainly a step too far: the casts for ArrowStackTup still end up in Core, and expanding the type synonyms there would be quite confusing. The expansion could happen in tidyType, since it’s called before reporting an error. But this seems probably even worse than putting it in pprType, since it’s still used in too many places, and it isn’t supposed to actually change the type. It could be handled as an extra, ad-hoc preprocessing step in reportWanteds. This is much closer to reasonable, though it feels like quite a hack. A separate error Reporter could catch these errors before the other reporters do and perform the expansion there. But I don’t think this actually makes sense, as the above example demonstrates that ArrowStackTup might be buried inside another type and in fact might not actually be the source of the type error at all! It could be done last-minute in mkEqErr. But I don’t know if this is too late, and ArrowStackTup could leak into an error through some other code path. Of those options, the best one I’ve come up with seems to be option 3, an ad-hoc preprocessing step in reportWanteds. Does that seem reasonable? Or is it too much of a kludge? Thanks, Alexis -------------- next part -------------- An HTML attachment was scrubbed... URL: From lexi.lambda at gmail.com Fri May 1 23:13:01 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Fri, 1 May 2020 18:13:01 -0500 Subject: Expanding a particular type family in type errors In-Reply-To: References: Message-ID: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> Answering my own question: it looks like cleaning this up in GHC.Tc.Errors is rather impractical—there are just too many types buried inside other datatypes that would have to be updated—so it has to be wired into pprType. Fortunately, I can branch on userStyle to decide whether or not to do the expansion. This still seems like a bit of a hack, but I can’t think of a better way. Do tell me if there’s a better approach! > On May 1, 2020, at 16:36, Alexis King wrote: > > Hi all, > > As part of my ongoing rework of arrow notation , I have introduced two wired-in type families, which assist in the typechecking process. The details are not terribly important (you can read the proposal if you’d like for those), but the relevant detail is that equalities of the shape > > ArrowStackTup a ~ b > > are produced by the typechecker in key places. ArrowStackTup is one of my wired-in type families, and it’s really an implementation detail. Unfortunately, I often end up with type errors that leak this detail, with expected/actual types like these ones: > > Expected type: ArrowStackTup '[Int] -> Bool > Actual type: Int -> String > > This is quite annoying, as it’s exceedingly rare that these type families actually get stuck, so they can almost always be expanded in type errors. As it happens, `ArrowStackTup '[a]` expands to simply `a`, so the type error I would like to report is this one: > > Expected type: Int -> Bool > Actual type: Int -> String > > Not technically challenging, but I find myself faced with the question of where this special expansion logic ought to go. It seems like it could go in any of several places: > > It could be hard-wired into pprType, and perhaps selectively disabled with an -fprint-* flag. This is nice in that it’s universal, but it’s almost certainly a step too far: the casts for ArrowStackTup still end up in Core, and expanding the type synonyms there would be quite confusing. > > The expansion could happen in tidyType, since it’s called before reporting an error. But this seems probably even worse than putting it in pprType, since it’s still used in too many places, and it isn’t supposed to actually change the type. > > It could be handled as an extra, ad-hoc preprocessing step in reportWanteds. This is much closer to reasonable, though it feels like quite a hack. > > A separate error Reporter could catch these errors before the other reporters do and perform the expansion there. But I don’t think this actually makes sense, as the above example demonstrates that ArrowStackTup might be buried inside another type and in fact might not actually be the source of the type error at all! > > It could be done last-minute in mkEqErr. But I don’t know if this is too late, and ArrowStackTup could leak into an error through some other code path. > > Of those options, the best one I’ve come up with seems to be option 3, an ad-hoc preprocessing step in reportWanteds. Does that seem reasonable? Or is it too much of a kludge? > > Thanks, > Alexis -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Sat May 2 22:08:18 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sat, 2 May 2020 23:08:18 +0100 Subject: Hadrian build with DWARF information doesn't contain as much debug information as I would expect Message-ID: I followed the instructions on the wiki to enable debug symbols in my build of GHC. (https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian#enabling-dwarf-debug-symbols) So I added these flags to may hadrian.settings file stage1.*.ghc.hs.opts += -g3 stage1.*.cabal.configure.opts += --disable-library-stripping --disable-executable-stripping stage1.ghc-bin.ghc.link.opts += -eventlog The resulting executable has debug information in it for the executable component but not for any of the libraries in including the compiler library. ("../sysdeps/x86_64/start.S",Just 4414944,Just 4414987,139633489318136) ("init.c",Nothing,Nothing,139633489318136) ("../sysdeps/x86_64/crti.S",Nothing,Nothing,139633489318136) ("ghc/Main.hs",Just 4415312,Just 4615455,139633489318136) ("ghc/GHCi/Leak.hs",Just 4615480,Just 4623414,139633489318136) ("ghc/GHCi/UI.hs",Just 4623440,Just 5461990,139633489318136) ("ghc/GHCi/UI/Info.hs",Just 5461992,Just 5571230,139633489318136) ("ghc/GHCi/UI/Monad.hs",Just 5571232,Just 5679695,139633489318136) ("ghc/GHCi/UI/Tags.hs",Just 5679696,Just 5704775,139633489318136) ("ghc/GHCi/Util.hs",Just 24,Just 173,139633489318136) ("../sysdeps/x86_64/crtn.S",Nothing,Nothing,139633489318136) I tried building a project with cabal and the resulting executable had debug information for every file in the dependencies as well as the main project. So how do I convince hadrian to include the correct information? Is it a bug in hadrian? I checked the command line when building the library and `-g3` is passed. Cheers, Matt From adam at sandbergericsson.se Sun May 3 07:19:42 2020 From: adam at sandbergericsson.se (Adam Sandberg Eriksson) Date: Sun, 03 May 2020 08:19:42 +0100 Subject: =?UTF-8?Q?Re:_Hadrian_build_with_DWARF_information_doesn't_contain_as_mu?= =?UTF-8?Q?ch_debug_information_as_I_would_expect?= In-Reply-To: References: Message-ID: I don't know how you read the DWARF info but maybe it's missing info from dynamic libraries? If your GHC is dynamically linked the library DWARF info might be available in their respective .so's. Cheers, Adam Sandberg Eriksson On Sat, 2 May 2020, at 23:08, Matthew Pickering wrote: > I followed the instructions on the wiki to enable debug symbols in my > build of GHC. > (https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian#enabling-dwarf-debug-symbols) > > So I added these flags to may hadrian.settings file > > stage1.*.ghc.hs.opts += -g3 > stage1.*.cabal.configure.opts += --disable-library-stripping > --disable-executable-stripping > stage1.ghc-bin.ghc.link.opts += -eventlog > > The resulting executable has debug information in it for the > executable component but not for any of the libraries in including the > compiler library. > > ("../sysdeps/x86_64/start.S",Just 4414944,Just 4414987,139633489318136) > ("init.c",Nothing,Nothing,139633489318136) > ("../sysdeps/x86_64/crti.S",Nothing,Nothing,139633489318136) > ("ghc/Main.hs",Just 4415312,Just 4615455,139633489318136) > ("ghc/GHCi/Leak.hs",Just 4615480,Just 4623414,139633489318136) > ("ghc/GHCi/UI.hs",Just 4623440,Just 5461990,139633489318136) > ("ghc/GHCi/UI/Info.hs",Just 5461992,Just 5571230,139633489318136) > ("ghc/GHCi/UI/Monad.hs",Just 5571232,Just 5679695,139633489318136) > ("ghc/GHCi/UI/Tags.hs",Just 5679696,Just 5704775,139633489318136) > ("ghc/GHCi/Util.hs",Just 24,Just 173,139633489318136) > ("../sysdeps/x86_64/crtn.S",Nothing,Nothing,139633489318136) > > I tried building a project with cabal and the resulting executable had > debug information for every file in the dependencies as well as the > main project. > > So how do I convince hadrian to include the correct information? Is it > a bug in hadrian? > > I checked the command line when building the library and `-g3` is passed. > > Cheers, > > Matt > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > From matthewtpickering at gmail.com Sun May 3 07:55:58 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sun, 3 May 2020 08:55:58 +0100 Subject: Hadrian build with DWARF information doesn't contain as much debug information as I would expect In-Reply-To: References: Message-ID: Thanks Adam for the tip about dynamic linking, as always. He was right that the debug information was in the relevant .so files and that when I statically linked GHC the information was included (as with cabal). The issue was my program which read the dwarf information did not work properly for dynamically linked executables (and still doesn't, I wonder how gdb knows which shared objects to load and what addresses to use). Cheers, Matt On Sun, May 3, 2020 at 8:20 AM Adam Sandberg Eriksson wrote: > > I don't know how you read the DWARF info but maybe it's missing info from dynamic libraries? If your GHC is dynamically linked the library DWARF info might be available in their respective .so's. > > Cheers, > Adam Sandberg Eriksson > > On Sat, 2 May 2020, at 23:08, Matthew Pickering wrote: > > I followed the instructions on the wiki to enable debug symbols in my > > build of GHC. > > (https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian#enabling-dwarf-debug-symbols) > > > > So I added these flags to may hadrian.settings file > > > > stage1.*.ghc.hs.opts += -g3 > > stage1.*.cabal.configure.opts += --disable-library-stripping > > --disable-executable-stripping > > stage1.ghc-bin.ghc.link.opts += -eventlog > > > > The resulting executable has debug information in it for the > > executable component but not for any of the libraries in including the > > compiler library. > > > > ("../sysdeps/x86_64/start.S",Just 4414944,Just 4414987,139633489318136) > > ("init.c",Nothing,Nothing,139633489318136) > > ("../sysdeps/x86_64/crti.S",Nothing,Nothing,139633489318136) > > ("ghc/Main.hs",Just 4415312,Just 4615455,139633489318136) > > ("ghc/GHCi/Leak.hs",Just 4615480,Just 4623414,139633489318136) > > ("ghc/GHCi/UI.hs",Just 4623440,Just 5461990,139633489318136) > > ("ghc/GHCi/UI/Info.hs",Just 5461992,Just 5571230,139633489318136) > > ("ghc/GHCi/UI/Monad.hs",Just 5571232,Just 5679695,139633489318136) > > ("ghc/GHCi/UI/Tags.hs",Just 5679696,Just 5704775,139633489318136) > > ("ghc/GHCi/Util.hs",Just 24,Just 173,139633489318136) > > ("../sysdeps/x86_64/crtn.S",Nothing,Nothing,139633489318136) > > > > I tried building a project with cabal and the resulting executable had > > debug information for every file in the dependencies as well as the > > main project. > > > > So how do I convince hadrian to include the correct information? Is it > > a bug in hadrian? > > > > I checked the command line when building the library and `-g3` is passed. > > > > Cheers, > > > > Matt > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > From sylvain at haskus.fr Sun May 3 09:28:39 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Sun, 3 May 2020 11:28:39 +0200 Subject: Hadrian build with DWARF information doesn't contain as much debug information as I would expect In-Reply-To: References: Message-ID: <3f16d82d-8733-2321-eaa6-fcdf842ecbfe@haskus.fr> > I wonder how gdb knows which shared objects to load and what addresses to use Perhaps it reads /proc/PID/maps (or /proc/PID/map_files/*)? Cheers, Sylvain On 03/05/2020 09:55, Matthew Pickering wrote: > Thanks Adam for the tip about dynamic linking, as always. > > He was right that the debug information was in the relevant .so files > and that when I statically linked GHC the information was included (as > with cabal). The issue was my program which read the dwarf information > did not work properly for dynamically linked executables (and still > doesn't, I wonder how gdb knows which shared objects to load and what > addresses to use). > > Cheers, > > Matt > > On Sun, May 3, 2020 at 8:20 AM Adam Sandberg Eriksson > wrote: >> I don't know how you read the DWARF info but maybe it's missing info from dynamic libraries? If your GHC is dynamically linked the library DWARF info might be available in their respective .so's. >> >> Cheers, >> Adam Sandberg Eriksson >> >> On Sat, 2 May 2020, at 23:08, Matthew Pickering wrote: >>> I followed the instructions on the wiki to enable debug symbols in my >>> build of GHC. >>> (https://gitlab.haskell.org/ghc/ghc/-/wikis/building/hadrian#enabling-dwarf-debug-symbols) >>> >>> So I added these flags to may hadrian.settings file >>> >>> stage1.*.ghc.hs.opts += -g3 >>> stage1.*.cabal.configure.opts += --disable-library-stripping >>> --disable-executable-stripping >>> stage1.ghc-bin.ghc.link.opts += -eventlog >>> >>> The resulting executable has debug information in it for the >>> executable component but not for any of the libraries in including the >>> compiler library. >>> >>> ("../sysdeps/x86_64/start.S",Just 4414944,Just 4414987,139633489318136) >>> ("init.c",Nothing,Nothing,139633489318136) >>> ("../sysdeps/x86_64/crti.S",Nothing,Nothing,139633489318136) >>> ("ghc/Main.hs",Just 4415312,Just 4615455,139633489318136) >>> ("ghc/GHCi/Leak.hs",Just 4615480,Just 4623414,139633489318136) >>> ("ghc/GHCi/UI.hs",Just 4623440,Just 5461990,139633489318136) >>> ("ghc/GHCi/UI/Info.hs",Just 5461992,Just 5571230,139633489318136) >>> ("ghc/GHCi/UI/Monad.hs",Just 5571232,Just 5679695,139633489318136) >>> ("ghc/GHCi/UI/Tags.hs",Just 5679696,Just 5704775,139633489318136) >>> ("ghc/GHCi/Util.hs",Just 24,Just 173,139633489318136) >>> ("../sysdeps/x86_64/crtn.S",Nothing,Nothing,139633489318136) >>> >>> I tried building a project with cabal and the resulting executable had >>> debug information for every file in the dependencies as well as the >>> main project. >>> >>> So how do I convince hadrian to include the correct information? Is it >>> a bug in hadrian? >>> >>> I checked the command line when building the library and `-g3` is passed. >>> >>> Cheers, >>> >>> Matt >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From matthewtpickering at gmail.com Sun May 3 11:57:27 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sun, 3 May 2020 12:57:27 +0100 Subject: Preview of new profiling mode - profile by info table Message-ID: Hi all, I have hacked together a new profiling mode code named "Profile by Info Table" which uses the address of the info table as the band identity in the profile. The main point of this is that you can then look up the source location of the info table in the dwarf information when rendering the chart. For example, in this profile I have only included THUNK closures, which you can reliably look up in the dwarf information. You can't include things like constructor closures for example as every constructor has the same info table regardless of where it comes from in the source. In the profiles below if you click on the Closure Descs tab you can see the precise source position the band originated from. As a test, I compiled the Cabal library with -O2 and here is the result. Firstly a profile sorted by size of the bands. https://mpickering.github.io/ghc-hi-cabal-o2.eventlog.html I also added a mode to eventlog2html which calculates the gradient of the line so you can more easily spot things which are leaking but not very big themselves which might be retaining other closures. https://mpickering.github.io/ghc-hi-cabal-o2-gradient.eventlog.html We can see clearly here that there are calls to compute unfoldings which remain unevaluated for a long time and therefore retaining reference to core expressions. (See the 0x177ff78 closure for example). The only issue is that at the moment reading the DWARF information takes about 48gb of memory. Cheers, Matt From matthewtpickering at gmail.com Sun May 3 14:43:17 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sun, 3 May 2020 15:43:17 +0100 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: <874ksz4muc.fsf@smart-cactus.org> References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> <874ksz4muc.fsf@smart-cactus.org> Message-ID: Hi Sergej, I definitely agree adding the capability for GHC to emit telemetry would be useful. I also prefer the eventlog like other people has already mentioned. We should at least ship with compiler so it can emit these user events even if it can't emit RTS events. In future we should probably look more into uprobes in order to make tracing more efficient but we have what we have for now. Recently I have been working with Dimity Ivanov on some tooling related to this. He has implemented a standard tracing format and exporters for many common diagnostic tools. https://github.com/ethercrow/opentelemetry-haskell You can see some examples of eventlogs generated by GHC in this issue: https://github.com/ethercrow/opentelemetry-haskell/issues/9 Cheers, Matt On Fri, May 1, 2020 at 5:55 PM Ben Gamari wrote: > > Sergej Jaskiewicz via ghc-devs writes: > > > tl;dr: I propose adding a new GHC flag for generating a log that allows > > visualizing how much time each stage in the compiler pipleline took, similar > > to Clang's -ftime-trace. > > > > Hello, everyone. > > Hi Sergej, > > > [snip] > > > The latter option is much closer to what we need. If we link the GHC executable > > with the -eventlog flag, then various significant events will be written to > > a special log file. For example, "Parser began parsing the Main.hs file after > > 5 ms since GHC has started, and ended parsing it 3 ms after that". > > The resulting log file can be parsed with the ghc-events library [4], and also > > visualized using the ThreadScope app [5]. > > > I'm a bit short on time but here are a few points in no particular order: > > Out of curiosity, have you seen Alp's work [1] in this area? This work > allows use to the ghc-events-analyze package [2] to visualize (e.g. [3]) > the existing withTiming eventlog output in a reasonably easy-to-consume > format. > > That being said, I can certainly see the benefit of using the Chrome > tracing infrastructure for this; it would make for a nicer user > experience than the static image that ghc-events-analyze spits out. > > I also know that Matthew Pickering has been working in this area and I'm > sure will have something interesting to add. > > I will admit that I am a bit reluctant to add support for a *particular* > tracing format to GHC itself. I think it would make the most sense for > GHC to produce a consumer-agnostic trace representation (via our > existing eventlog mechanism) and for users to transform this into the > format their tools require. > > Our current withTimings infrastructure is quite ad-hoc and could perhaps > expose more detail. It would be great to know what sorts of detail > people would find useful. > > Cheers, > > - Ben > > > [1] https://www.haskell.org/ghc/blog/20190924-eventful-ghc.html > [2] https://hackage.haskell.org/package/ghc-events-analyze > [3] https://www.haskell.org/ghc/blog/images/eventful-ghc/ghc-events-viz.svg > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From rae at richarde.dev Mon May 4 09:25:23 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Mon, 4 May 2020 10:25:23 +0100 Subject: Expanding a particular type family in type errors In-Reply-To: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> References: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> Message-ID: <359E1012-39C4-4288-8261-9EFE168F724B@richarde.dev> I'm afraid I like none of (1)-(5). Important background: if we have type family F a where F Int = Bool, then F Int and Bool are *not* equal in Core. We cannot just change one to the other. Instead, a coercion exists between them. 1. pprType seems a very challenging place to do this, given that pprType works on IfaceTypes, which are not amenable to much of any analysis. Maybe you could do the switcho-changeo before converting to IfaceType. Are your type families closed? Then it's just barely conceivable. If they're open, you won't have the instances available in pprType. Still, this seems like far too much processing to be done in pretty-printing. 2. tidyType really cannot change one type to another unequal one. Chaos will ensue. 3. reportWanteds will have a similar problem: if we have -fdefer-type-errors, then reportWanteds may actually happen before desugaring and execution, and we can't have a type magically changed in the process. Doing so will lead to a core-lint error. 4-5. These seem the post plausible. Yes, it's true that calls to ArrowStackTup might leak through, but doing so wouldn't be strictly *wrong*, just suboptimal. So it might be that a best-effort here is good enough. Another possibility: when these ArrowStackTup calls are born, do you already sometimes know the arguments? That is, there must be some code that produces the ArrowStackTup '[a] call -- could it just special-case an '[a] argument and produce `a` instead? Then, the ArrowStackTup family is used only when you have an abstract argument, which might be considerably rarer. If you haven't yet come across it, see the normaliseType function, which evaluates type families as much as possible. I hope this helps! Richard > On May 2, 2020, at 12:13 AM, Alexis King wrote: > > Answering my own question: it looks like cleaning this up in GHC.Tc.Errors is rather impractical—there are just too many types buried inside other datatypes that would have to be updated—so it has to be wired into pprType. Fortunately, I can branch on userStyle to decide whether or not to do the expansion. This still seems like a bit of a hack, but I can’t think of a better way. Do tell me if there’s a better approach! > >> On May 1, 2020, at 16:36, Alexis King > wrote: >> >> Hi all, >> >> As part of my ongoing rework of arrow notation , I have introduced two wired-in type families, which assist in the typechecking process. The details are not terribly important (you can read the proposal if you’d like for those), but the relevant detail is that equalities of the shape >> >> ArrowStackTup a ~ b >> >> are produced by the typechecker in key places. ArrowStackTup is one of my wired-in type families, and it’s really an implementation detail. Unfortunately, I often end up with type errors that leak this detail, with expected/actual types like these ones: >> >> Expected type: ArrowStackTup '[Int] -> Bool >> Actual type: Int -> String >> >> This is quite annoying, as it’s exceedingly rare that these type families actually get stuck, so they can almost always be expanded in type errors. As it happens, `ArrowStackTup '[a]` expands to simply `a`, so the type error I would like to report is this one: >> >> Expected type: Int -> Bool >> Actual type: Int -> String >> >> Not technically challenging, but I find myself faced with the question of where this special expansion logic ought to go. It seems like it could go in any of several places: >> >> It could be hard-wired into pprType, and perhaps selectively disabled with an -fprint-* flag. This is nice in that it’s universal, but it’s almost certainly a step too far: the casts for ArrowStackTup still end up in Core, and expanding the type synonyms there would be quite confusing. >> >> The expansion could happen in tidyType, since it’s called before reporting an error. But this seems probably even worse than putting it in pprType, since it’s still used in too many places, and it isn’t supposed to actually change the type. >> >> It could be handled as an extra, ad-hoc preprocessing step in reportWanteds. This is much closer to reasonable, though it feels like quite a hack. >> >> A separate error Reporter could catch these errors before the other reporters do and perform the expansion there. But I don’t think this actually makes sense, as the above example demonstrates that ArrowStackTup might be buried inside another type and in fact might not actually be the source of the type error at all! >> >> It could be done last-minute in mkEqErr. But I don’t know if this is too late, and ArrowStackTup could leak into an error through some other code path. >> >> Of those options, the best one I’ve come up with seems to be option 3, an ad-hoc preprocessing step in reportWanteds. Does that seem reasonable? Or is it too much of a kludge? >> >> Thanks, >> Alexis > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From lexi.lambda at gmail.com Mon May 4 19:17:06 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Mon, 4 May 2020 14:17:06 -0500 Subject: Expanding a particular type family in type errors In-Reply-To: <359E1012-39C4-4288-8261-9EFE168F724B@richarde.dev> References: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> <359E1012-39C4-4288-8261-9EFE168F724B@richarde.dev> Message-ID: <1700C637-63DA-4AD5-9E71-97F6BDCCFD70@gmail.com> > On May 4, 2020, at 04:25, Richard Eisenberg wrote: > > 1. pprType seems a very challenging place to do this, given that pprType works on IfaceTypes, which are not amenable to much of any analysis. Maybe you could do the switcho-changeo before converting to IfaceType. Are your type families closed? Then it's just barely conceivable. If they're open, you won't have the instances available in pprType. Still, this seems like far too much processing to be done in pretty-printing. You are right—I discovered this after my previous email. But they are more than closed: they are BuiltInSynFams, so they can be expanded easily by isBuiltInSynFamTyCon_maybe + sfMatchFam. > 2. tidyType really cannot change one type to another unequal one. Chaos will ensue. Agreed. > 3. reportWanteds will have a similar problem: if we have -fdefer-type-errors, then reportWanteds may actually happen before desugaring and execution, and we can't have a type magically changed in the process. Doing so will lead to a core-lint error. I don’t totally understand this point—I thought reportWanteds just printed things! It sounds like you’re saying it does something more, but I don’t immediately see what. > 4-5. These seem the post plausible. Yes, it's true that calls to ArrowStackTup might leak through, but doing so wouldn't be strictly *wrong*, just suboptimal. So it might be that a best-effort here is good enough. I think 4 makes sense for situations where the type family is stuck, since in that case we can’t just expand it away, and there is a more useful error message we can report. Maybe 5 makes more sense for the other cases, but I started down that path and it’s quite awkward. > Another possibility: when these ArrowStackTup calls are born, do you already sometimes know the arguments? That is, there must be some code that produces the ArrowStackTup '[a] call -- could it just special-case an '[a] argument and produce `a` instead? Then, the ArrowStackTup family is used only when you have an abstract argument, which might be considerably rarer. I’m afraid not. The entire reason I have the type families in the first place is that the arguments are not, in general, fully known when the equality constraint is emitted. If that weren’t true, I could drop them completely. (The proposal discusses that in a little more detail, as do some Notes in my WIP MR.) It’s all a bit frustrating, because it’s really not like printing things this way is “cheating” or somehow morally wrong—these types are nominally equal, so from the user’s point of view they are completely interchangeable. It only becomes sketchy from the Core point of view, and I’m totally fine to not do any of this magic there! Really, I think this morally belongs in the printer. This is just a display thing, nothing more. I want to be able to say “these type families are not something the user is likely to understand, so when printing error messages, expand them if at all possible.” But it’s very difficult to do that currently given the way pprType goes through IfaceType, since you’ve lost the necessary information by that point. Alexis -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Tue May 5 09:00:06 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Tue, 5 May 2020 10:00:06 +0100 Subject: Expanding a particular type family in type errors In-Reply-To: <1700C637-63DA-4AD5-9E71-97F6BDCCFD70@gmail.com> References: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> <359E1012-39C4-4288-8261-9EFE168F724B@richarde.dev> <1700C637-63DA-4AD5-9E71-97F6BDCCFD70@gmail.com> Message-ID: <938DADC3-A5B1-4CA4-8266-ADBF5A4EB692@richarde.dev> > On May 4, 2020, at 8:17 PM, Alexis King wrote: > >> 3. reportWanteds will have a similar problem: if we have -fdefer-type-errors, then reportWanteds may actually happen before desugaring and execution, and we can't have a type magically changed in the process. Doing so will lead to a core-lint error. > > I don’t totally understand this point—I thought reportWanteds just printed things! It sounds like you’re saying it does something more, but I don’t immediately see what. reportWanteds reports errors, yes, but it also fills in deferred type errors. By the time reportWanteds sees an error, that error takes the form of an unsolved Wanted constraint. Every unsolved Wanted corresponds to an evidence Id (that is, variable) in the desugared program that should contain the evidence. (For example, we might have $dEq which should eventually have a dictionary for Eq Int.) If a Wanted is unsolved by the time it gets to reportWanteds, it will never be solved. Instead, if we are deferring errors, we must let-bind the Id to *something* to produce Core. So we bind it to a call to `error` (roughly) with the same error message that would have been printed if we weren't deferring errors. It's a bit awkward that a function called reportWanteds does this, but it actually makes good sense, because reportWanteds knows the text that should be in the runtime error call. > >> Another possibility: when these ArrowStackTup calls are born, do you already sometimes know the arguments? That is, there must be some code that produces the ArrowStackTup '[a] call -- could it just special-case an '[a] argument and produce `a` instead? Then, the ArrowStackTup family is used only when you have an abstract argument, which might be considerably rarer. > > I’m afraid not. The entire reason I have the type families in the first place is that the arguments are not, in general, fully known when the equality constraint is emitted. You say that they're not, in general, fully known. But are they often known in practice? My suggestion here isn't to remove the type families entirely, but maybe to short-circuit around them. If that's often possible, it takes some pressure off fixing the pretty-printing. > > It’s all a bit frustrating, because it’s really not like printing things this way is “cheating” or somehow morally wrong—these types are nominally equal, so from the user’s point of view they are completely interchangeable. This is a good point -- I don't think I quite absorbed it last time around. I had considered the change-before-printing quite strange. But you're right in that you're interchanging two nominally-equal types, and that's just fine. > Really, I think this morally belongs in the printer. This is just a display thing, nothing more. I want to be able to say “these type families are not something the user is likely to understand, so when printing error messages, expand them if at all possible.” But it’s very difficult to do that currently given the way pprType goes through IfaceType, since you’ve lost the necessary information by that point. I wonder if we should have a preprocessTypeForPrinting :: DynFlags -> Type -> Type function. This could be run *before* the conversion to IfaceType. It could handle both your new idea and the existing idea for -f(no-)print-explicit-runtime-reps. The supplied DynFlags could be gotten from sdocWithDynFlags, I think. I bet that once we have this facility working, we'll find more uses for it. I'm now considerably happier than I was about doing this in the printing pipeline, if you think you see how it could be done. Richard > > Alexis -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreash87 at gmx.ch Tue May 5 09:55:06 2020 From: andreash87 at gmx.ch (Andreas Herrmann) Date: Tue, 5 May 2020 11:55:06 +0200 Subject: ZuriHac 2020 - GHC Track - Call for Contributions Message-ID: Dear GHC developers, As you may have heard this year's ZuriHac will be an online event due to COVID-19 [1]. Nonetheless, we would like to organize a GHC track again this year with the goal to foster contributions to GHC and teach newcomers how to participate in GHC's development. To that end we would like to invite you to hold seminars, workshops, or hack sessions on topics centered around GHC. Please let us know by replying to this email if you would like to contribute to this year's GHC track. ZuriHac will take place from Friday 12 June to Sunday 14 June. We are still working out the technical details. However, we are planning to provide a virtual space for people to collaborate on projects, participate in voice/video chats, and live stream talks. Best, Andreas [1]: https://zfoh.ch/zurihac2020 -------------- next part -------------- An HTML attachment was scrubbed... URL: From lexi.lambda at gmail.com Tue May 5 15:43:40 2020 From: lexi.lambda at gmail.com (Alexis King) Date: Tue, 5 May 2020 10:43:40 -0500 Subject: Expanding a particular type family in type errors In-Reply-To: <938DADC3-A5B1-4CA4-8266-ADBF5A4EB692@richarde.dev> References: <484D65C5-93B8-463C-9012-C68D444BB502@gmail.com> <359E1012-39C4-4288-8261-9EFE168F724B@richarde.dev> <1700C637-63DA-4AD5-9E71-97F6BDCCFD70@gmail.com> <938DADC3-A5B1-4CA4-8266-ADBF5A4EB692@richarde.dev> Message-ID: <4D35E709-BF11-4C03-9CAB-FB6339351E15@gmail.com> > On May 5, 2020, at 04:00, Richard Eisenberg wrote: > > reportWanteds reports errors, yes, but it also fills in deferred type errors. [snip] It's a bit awkward that a function called reportWanteds does this, but it actually makes good sense, because reportWanteds knows the text that should be in the runtime error call. Yes, that does make sense; thank you for the explanation. > You say that they're not, in general, fully known. But are they often known in practice? My suggestion here isn't to remove the type families entirely, but maybe to short-circuit around them. If that's often possible, it takes some pressure off fixing the pretty-printing. You’re right that this might be feasible, but I think it would be awkward, and it probably would break down just often enough to be an unsatisfying solution. > I wonder if we should have a preprocessTypeForPrinting :: DynFlags -> Type -> Type function. This could be run *before* the conversion to IfaceType. It could handle both your new idea and the existing idea for -f(no-)print-explicit-runtime-reps. The supplied DynFlags could be gotten from sdocWithDynFlags, I think. I bet that once we have this facility working, we'll find more uses for it. If you think this is a reasonable idea, I’m happy to give it a shot—it definitely seems like the nicest way to do it to me. I’ll try it out and report back if I run into any trouble. Thanks again! Alexis From compl.yue at gmail.com Tue May 5 17:39:13 2020 From: compl.yue at gmail.com (Compl Yue) Date: Wed, 6 May 2020 01:39:13 +0800 Subject: Connecting the Wolfram Language to Haskell Message-ID: Hi, If you are already running atop the Jupyter technology stack, IHaskel [1] may be very straight forward to be incorporated from technical aspect (while licensing aspect is out of my expertise). But still I would like take this opportunity to have more/further evaluation from you, about [2] , with by far my implementation [3] and application [4]. Once a frontend plugin can be injected into a GHCi session, and hooked up to some interactive UI, a new door is opened to allow much easier setup & use of GHC API interactively, like demonstrated in Hadui [4]. Currently there are too many different ways to setup a GHC API session, HIE has tried hard to unify them, but itself is yet under transition. I see the frontend interface being a proper abstraction even extended to WebUI/GUI applications, making interactions perfectly decoupled from how the underlying artifacts are organized, thus worth further consideration. While IHaskell is more powerful and better established as with Jupyter notebook, I chose the frontend route because IHaskell didn't come to today's maturity when I needed something like Hadui, and ZeroMQ seems bloating to my cases, even for now. Cheers, Compl [1] https://github.com/gibiansky/IHaskell [2] https://gitlab.haskell.org/ghc/ghc/issues/17348 [3] https://gitlab.haskell.org/complyue/ghc/tree/ghc-8.6-ife [4] https://github.com/complyue/hadui On 2020/5/1 上午5:13, Ben Gamari wrote: Simon Peyton Jones via ghc-devs writes: Friends Short summary: can someone familiar with using the GHC API offer advice on getting the Wolfram Language connected to Haskell? Hi Todd, et al., This sounds like a great project. I have fond memories of Mathematica from my studies. ... We have the following basic line of code for evaluating a string of Haskell code: r <- liftIO $ runInterpreter $ do { setImports ["Prelude"]; eval strToEvaluate} The problem is that this is a one-shot evaluation, and we want a long-lived interactive Haskell session, to which a series of inputs can be directed. We have been told that we have to use GHCi for that, but we don't know how to do it. It appears that you are using the `hint` library [1] for evaluation. I'll admit that I've not used hint; it looks quite sensible but I do not know what limitations you might encounter. It looks like its approach to error handling may leave something to be desired. Nevertheless, we can work with it for now; if we run into its limitations then the alternative is to use the GHC API directly, as suggested by Simon. The basic flow of our functionality is as follows: 1) User calls StartExternalSession["LanguageName"] to launch an interpreter for the language. This process remains running and can be used for multiple calls. 2) User calls ExternalEvaluate[session, "some code"] to execute the given code in the external language and return a result converted into native Wolfram Language types (strings, numbers, lists, associations, etc.) Sure. ... We have attached a simple file of Haskell code that one of our engineers has successfully used to get a basic evaluation of Haskell code from the Wolfram Language, but it uses the single-shot evaluation code that was given above, and so is not suitable. We would appreciate any help that you can give us, or developers or resources you can point us at, to assist in integrating Haskell into our ExternalEvaluate system. It looks like you will want to push the `runInterpreter` out of the `forever`. Afterall, you want the interpreter session to persist over multiple requests. Doing this isn't difficult but does require some monad transformer shuffling, which may be unfamiliar to someone coming from another language. I've put up a cleaned up version of your program here [1]; hopefully this is enough to get you started. Do note that this requires a patched version of zeromq4-haskell due to a minor bug [2] which I have fixed [3]. Do note that there is a related effort, iHaskell [4], which provides a Haskell kernel for Jupyter Notebook. This might be a place to draw inspiration from. Let us know how things go and don't hesitate to be in touch if you have questions regarding the GHC API. Cheers, - Ben [1] https://github.com/bgamari/zeromq-hint [2] https://gitlab.com/twittner/zeromq-haskell/-/issues/66 [3] https://gitlab.com/twittner/zeromq-haskell/-/merge_requests/6 [4] https://github.com/gibiansky/IHaskell _______________________________________________ ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Tue May 5 18:12:40 2020 From: ben at well-typed.com (Ben Gamari) Date: Tue, 05 May 2020 14:12:40 -0400 Subject: 8.12 plans Message-ID: <87ftce2qui.fsf@smart-cactus.org> Hi everyone, The time is again upon us to start thinking about release planning for the next major release: GHC 8.12. In keeping with our 6-month release schedule, I propose the following schedule: * Mid-June 2020: Aim to have all major features in the tree * Late-June 2020: Cut the ghc-8.12 branch * June - August 2020: 3 alpha releases * 1 September 2020: beta release * 25 September 2020: Final 8.12.1 release So, if you have any major features which you would like to merge for 8.12, now is the time to start planning how to wrap them up in the next month or so. As always, do let me know if you think this may be problematic and we can discuss options. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From hecate at glitchbra.in Tue May 5 20:32:43 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Tue, 5 May 2020 22:32:43 +0200 Subject: [8.12] Documentation plans Message-ID: Hello ghc-devs For everyone who doesn't know me (yet), I am Hécate, and I coordinate the the Haskell Docs initiative. You may have seen me on various community channels and on social media to recruit lost souls^W^Wmotivated volunteers to help improve the documentation of the `base` library. It can be a daunting task, hence this page that I set up to guide newcomers and track ongoing work[0] The point of my message is the following: If there are any modules you would like to see prioritised for the next release (8.12), please contact us. Current efforts are very organic and stem from personal hardships with the docs, but if you feel that some parts of the docs have been left too long on their own, give us a heads-up. If you want to personally take on a module whose documentation you deem lacking, feel free to join us on #Haskell-docs[1] , so we can coordinate and avoid multiple people working at the same time in an uncoordinated manner towards the same goal. And finally if you do not wish to get involved but still wish to support, we communicate on Twitter[2] and the Fediverse[3]. I would like to end this email by thanking Ben and Carter for their support and guidance. The Core Libraries Committee has been instrumental to the development of these efforts. Cheers, and take care. [0]: https://gitlab.haskell.org/ghc/ghc/issues/17929 [1]: https://webchat.freenode.net/ (enter #Haskell-docs in the 'channel' field) [2]: twitter.com/search?q=%23HaskellDocs [3]: https://search.social/?q=%23haskelldocs -- Hécate IRC: Uniaika Twitter: @TechnoEmpress Fediverse: @hecate at pleroma.fr From ben at smart-cactus.org Tue May 5 21:12:39 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Tue, 05 May 2020 17:12:39 -0400 Subject: [8.12] Documentation plans In-Reply-To: References: Message-ID: <87v9la13y8.fsf@smart-cactus.org> Hécate writes: > Hello ghc-devs > > For everyone who doesn't know me (yet), I am Hécate, and I coordinate > the the Haskell Docs initiative. > Thank you again for taking this on, Hécate! It is invaluable work. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From sandy at sandymaguire.me Wed May 6 05:09:09 2020 From: sandy at sandymaguire.me (Sandy Maguire) Date: Tue, 5 May 2020 22:09:09 -0700 Subject: [8.12] Documentation plans In-Reply-To: <87v9la13y8.fsf@smart-cactus.org> References: <87v9la13y8.fsf@smart-cactus.org> Message-ID: I'm interested in helping, but base has always been a particularly mysterious thing IMO. Where is its codebase? Can I just send PRs at it? Making this an easier thing to do (and find out) I think would greatly help to mobilize the community on the documentation front. On Tue, May 5, 2020 at 2:13 PM Ben Gamari wrote: > Hécate writes: > > > Hello ghc-devs > > > > For everyone who doesn't know me (yet), I am Hécate, and I coordinate > > the the Haskell Docs initiative. > > > Thank you again for taking this on, Hécate! It is invaluable work. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Wed May 6 08:35:27 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 06 May 2020 04:35:27 -0400 Subject: [8.12] Documentation plans In-Reply-To: References: <87v9la13y8.fsf@smart-cactus.org> Message-ID: On May 6, 2020 1:09:09 AM EDT, Sandy Maguire wrote: >I'm interested in helping, but base has always been a particularly >mysterious thing IMO. Where is its codebase? Can I just send PRs at it? >Making this an easier thing to do (and find out) I think would greatly >help >to mobilize the community on the documentation front. > >On Tue, May 5, 2020 at 2:13 PM Ben Gamari wrote: > >> Hécate writes: >> >> > Hello ghc-devs >> > >> > For everyone who doesn't know me (yet), I am Hécate, and I >coordinate >> > the the Haskell Docs initiative. >> > >> Thank you again for taking this on, Hécate! It is invaluable work. >> >> Cheers, >> >> - Ben >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> The source for base located in the ghc repository in the libraries/base directory. I have added a mention of this fact to the documentation tracking ticket. Cheers, - Ben From omeragacan at gmail.com Wed May 6 11:30:23 2020 From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=) Date: Wed, 6 May 2020 14:30:23 +0300 Subject: [RFC] Compiler pipeline timings visualization In-Reply-To: References: <8DC5561D-9E7B-483F-A7E6-D1C1969B7C65@icloud.com> <874ksz4muc.fsf@smart-cactus.org> Message-ID: Hi, While better representation/visualization of the pipeline might be useful, my opinion is that it might not be as useful as you might expect. The problem is lazy evaluation: it assigns the blame to incorrect places. For example, in GHC if I update a record field in pass 1 and use it in pass 2, allocation and runtime costs of generating the field (maybe an analysis result) is assigned to pass 2, which is not what you want most of the time (one of the reasons why I think lazy evaluation is not a good default). Ömer Matthew Pickering , 3 May 2020 Paz, 17:44 tarihinde şunu yazdı: > > Hi Sergej, > > I definitely agree adding the capability for GHC to emit telemetry > would be useful. I also prefer the eventlog like other people has > already mentioned. We should at least ship with compiler so it can > emit these user events even if it can't emit RTS events. > > In future we should probably look more into uprobes in order to make > tracing more efficient but we have what we have for now. > > Recently I have been working with Dimity Ivanov on some tooling > related to this. He has implemented a standard tracing format and > exporters for many common diagnostic tools. > > https://github.com/ethercrow/opentelemetry-haskell > > You can see some examples of eventlogs generated by GHC in this issue: > > https://github.com/ethercrow/opentelemetry-haskell/issues/9 > > Cheers, > > Matt > > > On Fri, May 1, 2020 at 5:55 PM Ben Gamari wrote: > > > > Sergej Jaskiewicz via ghc-devs writes: > > > > > tl;dr: I propose adding a new GHC flag for generating a log that allows > > > visualizing how much time each stage in the compiler pipleline took, similar > > > to Clang's -ftime-trace. > > > > > > Hello, everyone. > > > > Hi Sergej, > > > > > > [snip] > > > > > The latter option is much closer to what we need. If we link the GHC executable > > > with the -eventlog flag, then various significant events will be written to > > > a special log file. For example, "Parser began parsing the Main.hs file after > > > 5 ms since GHC has started, and ended parsing it 3 ms after that". > > > The resulting log file can be parsed with the ghc-events library [4], and also > > > visualized using the ThreadScope app [5]. > > > > > I'm a bit short on time but here are a few points in no particular order: > > > > Out of curiosity, have you seen Alp's work [1] in this area? This work > > allows use to the ghc-events-analyze package [2] to visualize (e.g. [3]) > > the existing withTiming eventlog output in a reasonably easy-to-consume > > format. > > > > That being said, I can certainly see the benefit of using the Chrome > > tracing infrastructure for this; it would make for a nicer user > > experience than the static image that ghc-events-analyze spits out. > > > > I also know that Matthew Pickering has been working in this area and I'm > > sure will have something interesting to add. > > > > I will admit that I am a bit reluctant to add support for a *particular* > > tracing format to GHC itself. I think it would make the most sense for > > GHC to produce a consumer-agnostic trace representation (via our > > existing eventlog mechanism) and for users to transform this into the > > format their tools require. > > > > Our current withTimings infrastructure is quite ad-hoc and could perhaps > > expose more detail. It would be great to know what sorts of detail > > people would find useful. > > > > Cheers, > > > > - Ben > > > > > > [1] https://www.haskell.org/ghc/blog/20190924-eventful-ghc.html > > [2] https://hackage.haskell.org/package/ghc-events-analyze > > [3] https://www.haskell.org/ghc/blog/images/eventful-ghc/ghc-events-viz.svg > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Thu May 7 21:54:31 2020 From: ben at well-typed.com (Ben Gamari) Date: Thu, 07 May 2020 17:54:31 -0400 Subject: Decorating exceptions with backtrace information Message-ID: <87r1vv1kdx.fsf@smart-cactus.org> Hi everyone, After a nice discussion on IRC about the unfortunate state of error reporting in Haskell, I felt compelled to write down some long-lingering thoughts regarding backtraces on exceptions. The result is GHC proposal #330 [1]. I think the approach is viable and perhaps even straightforward. I have the sketch of an implementation here [2]. Please have a look at the proposal and leave your comments. If there is consensus it is possible that we could have this done for 8.12. Cheers, - Ben [1] https://github.com/ghc-proposals/ghc-proposals/pull/330 [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3236 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mgsloan at gmail.com Thu May 7 22:56:22 2020 From: mgsloan at gmail.com (Michael Sloan) Date: Thu, 7 May 2020 16:56:22 -0600 Subject: Decorating exceptions with backtrace information In-Reply-To: <87r1vv1kdx.fsf@smart-cactus.org> References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: Thanks so much for making a proposal for this, Ben!! It's great to see progress here. I'm also glad that there is now a proposal process. I made a fairly similar proposal almost exactly 5 years ago to the libraries list - https://mail.haskell.org/pipermail/libraries/2015-April/025471.html - but without the subtlety of particular backtrace representations. Skimming the ensuing thread may still be informative. In particular, there is one thing I would like to highlight from that old proposal. I think it'd be good to have a standard way to represent a chain of exceptions, and build this into `catch` and `finally`. Python and Java both have a mechanism for this, and both refer to it as a "cause" exception. When an exception is thrown during exception handling, the exception being handled is preserved as its "cause". I find this mechanism to be incredibly useful in Java, it has made the underlying issue much clearer in many cases, and in other cases at least provides helpful context. I have no doubt such a mechanism would have saved me many hours of debugging exceptions in Haskell systems I've worked on in the past. I considered commenting about that directly on the proposal, but I figure this is a better place to suggest expanding the scope of the change :) . Totally understandable if you want to keep this proposal focused on stacktraces, but I think it'd be good to consider this as a potential future improvement. -Michael On Thu, May 7, 2020 at 3:55 PM Ben Gamari wrote: > > Hi everyone, > > After a nice discussion on IRC about the unfortunate state of error > reporting in Haskell, I felt compelled to write down some long-lingering > thoughts regarding backtraces on exceptions. The result is GHC proposal > #330 [1]. I think the approach is viable and perhaps even > straightforward. I have the sketch of an implementation here [2]. > > Please have a look at the proposal and leave your comments. If there is > consensus it is possible that we could have this done for 8.12. > > Cheers, > > - Ben > > > [1] https://github.com/ghc-proposals/ghc-proposals/pull/330 > [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3236 > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From csaba.hruska at gmail.com Fri May 8 13:03:03 2020 From: csaba.hruska at gmail.com (Csaba Hruska) Date: Fri, 8 May 2020 15:03:03 +0200 Subject: Unique invariants: GHC invocation vs scope? Message-ID: Hello, What are the intended invariants of Unique values? So far I thought that it is always gives a unique identity to things during a whole GHC invocation. However this is not true. I.e. eta expansion breaks this assumption. Here is a snippet from the STG IR of GHC.PrimopWrappers: extendInt8# [InlPrag=NOINLINE] :: Int8# -> Int# [GblId, Arity=1, Unf=OtherCon []] = \r [eta_B1] extendInt8# [eta_B1]; ord# [InlPrag=NOINLINE] :: Char# -> Int# [GblId, Arity=1, Unf=OtherCon []] = \r [eta_B1] ord# [eta_B1]; Where eta_B1 is redefined, so it is only unique in the scope. Is this intentional? Thanks, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Fri May 8 13:32:41 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Fri, 8 May 2020 09:32:41 -0400 Subject: Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: ben, could you please email the libraries list with this too? This seems like a core libraries / base change rather than a ghc-the-compiler change On Thu, May 7, 2020 at 6:57 PM Michael Sloan wrote: > Thanks so much for making a proposal for this, Ben!! It's great to see > progress here. > > I'm also glad that there is now a proposal process. I made a fairly > similar proposal almost exactly 5 years ago to the libraries list - > https://mail.haskell.org/pipermail/libraries/2015-April/025471.html - but > without the subtlety of particular backtrace representations. Skimming the > ensuing thread may still be informative. > > In particular, there is one thing I would like to highlight from that old > proposal. I think it'd be good to have a standard way to represent a chain > of exceptions, and build this into `catch` and `finally`. Python and Java > both have a mechanism for this, and both refer to it as a "cause" > exception. When an exception is thrown during exception handling, the > exception being handled is preserved as its "cause". I find this mechanism > to be incredibly useful in Java, it has made the underlying issue much > clearer in many cases, and in other cases at least provides helpful > context. I have no doubt such a mechanism would have saved me many hours > of debugging exceptions in Haskell systems I've worked on in the past. > > I considered commenting about that directly on the proposal, but I figure > this is a better place to suggest expanding the scope of the change :) . > Totally understandable if you want to keep this proposal focused on > stacktraces, but I think it'd be good to consider this as a potential > future improvement. > > -Michael > > On Thu, May 7, 2020 at 3:55 PM Ben Gamari wrote: > >> >> Hi everyone, >> >> After a nice discussion on IRC about the unfortunate state of error >> reporting in Haskell, I felt compelled to write down some long-lingering >> thoughts regarding backtraces on exceptions. The result is GHC proposal >> #330 [1]. I think the approach is viable and perhaps even >> straightforward. I have the sketch of an implementation here [2]. >> >> Please have a look at the proposal and leave your comments. If there is >> consensus it is possible that we could have this done for 8.12. >> >> Cheers, >> >> - Ben >> >> >> [1] https://github.com/ghc-proposals/ghc-proposals/pull/330 >> [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3236 >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Fri May 8 13:33:51 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Fri, 8 May 2020 09:33:51 -0400 Subject: Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: On Fri, May 8, 2020 at 9:32 AM Carter Schonwald wrote: > ben, could you please email the libraries list with this too? This seems > like a core libraries / base change rather than a ghc-the-compiler change > > On Thu, May 7, 2020 at 6:57 PM Michael Sloan wrote: > >> Thanks so much for making a proposal for this, Ben!! It's great to see >> progress here. >> >> I'm also glad that there is now a proposal process. I made a fairly >> similar proposal almost exactly 5 years ago to the libraries list - >> https://mail.haskell.org/pipermail/libraries/2015-April/025471.html - >> but without the subtlety of particular backtrace representations. Skimming >> the ensuing thread may still be informative. >> >> In particular, there is one thing I would like to highlight from that old >> proposal. I think it'd be good to have a standard way to represent a chain >> of exceptions, and build this into `catch` and `finally`. Python and Java >> both have a mechanism for this, and both refer to it as a "cause" >> exception. When an exception is thrown during exception handling, the >> exception being handled is preserved as its "cause". I find this mechanism >> to be incredibly useful in Java, it has made the underlying issue much >> clearer in many cases, and in other cases at least provides helpful >> context. I have no doubt such a mechanism would have saved me many hours >> of debugging exceptions in Haskell systems I've worked on in the past. >> >> I considered commenting about that directly on the proposal, but I figure >> this is a better place to suggest expanding the scope of the change :) . >> Totally understandable if you want to keep this proposal focused on >> stacktraces, but I think it'd be good to consider this as a potential >> future improvement. >> >> -Michael >> >> On Thu, May 7, 2020 at 3:55 PM Ben Gamari wrote: >> >>> >>> Hi everyone, >>> >>> After a nice discussion on IRC about the unfortunate state of error >>> reporting in Haskell, I felt compelled to write down some long-lingering >>> thoughts regarding backtraces on exceptions. The result is GHC proposal >>> #330 [1]. I think the approach is viable and perhaps even >>> straightforward. I have the sketch of an implementation here [2]. >>> >>> Please have a look at the proposal and leave your comments. If there is >>> consensus it is possible that we could have this done for 8.12. >>> >>> Cheers, >>> >>> - Ben >>> >>> >>> [1] https://github.com/ghc-proposals/ghc-proposals/pull/330 >>> [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3236 >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemming at henning-thielemann.de Fri May 8 15:37:36 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Fri, 8 May 2020 17:37:36 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: There seem to be multiple beginnings of the discussion. What is currently discussed? If someone says "exceptions" and "backtrace" in one sentence, I suspect like many times before, that again confusion of the concepts of exceptions and errors is ahead. Errors already support call stacks. Why should exceptions get them, too? Exceptions should carry information that is useful for a user, but a callstack is not useful for a user. I can imagine that it would be helpful for the user to get a stacked exception information like: Parse error on line 42, column 23 while reading file "foo/bar" while traversing directory "blabla" But since you refer to the CallStack feature of GHC, this seems not to be addressed in the proposals. On Fri, 8 May 2020, Carter Schonwald wrote: > I have no doubt such a mechanism would have saved me many hours of > debugging exceptions in Haskell systems I've worked on in the past. If you must debug exceptions, then this sounds like exceptions were abused for programming errors. Ben writes in: http://www.well-typed.com/blog/2020/04/dwarf-3/ "Unfortunately, the untyped nature of Haskell exceptions complicates the migration path for existing code." Actually, it only proves again, that it was wrong from the beginning to hide information about potential exceptions in the IO monad instead of making them explicit via ExceptionalT, ExceptT or the like. From mail at nh2.me Fri May 8 17:27:35 2020 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Fri, 8 May 2020 19:27:35 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> On 5/8/20 5:37 PM, Henning Thielemann wrote: > I can imagine that it would be helpful for the user to get a stacked exception information like: >    Parse error on line 42, column 23 >    while reading file "foo/bar" >    while traversing directory "blabla" That seems to be rather specific use case. It'd be a cool feature but I'm not aware of any programming language following that interpretation so far. I personally would be happy to be able to get the same type of stack trace for exceptions as in other programming langues (and as the proposal suggests). > If you must debug exceptions, then this sounds like exceptions were abused for programming errors. I'd be pretty happy to be able to debug them better; no matter if they were "abused" for anything or not, I must still debug them in practice. Given that they traverse program flow invisibly (e.g. not lexically, like return values) and can become visible in different places than they arose, having a call stack to debug their creation would be useful. > a callstack is not useful for a user. Call stacks have been very useful to me as a user of non-Haskell tools so far, because they are excellent for attaching to bug reports and usually led to developers fixing my problems faster. From lemming at henning-thielemann.de Fri May 8 17:32:54 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Fri, 8 May 2020 19:32:54 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> Message-ID: On Fri, 8 May 2020, Niklas Hambüchen wrote: > On 5/8/20 5:37 PM, Henning Thielemann wrote: > >> a callstack is not useful for a user. > > Call stacks have been very useful to me as a user of non-Haskell tools > so far, because they are excellent for attaching to bug reports and > usually led to developers fixing my problems faster. This confirms that they are not for you, but you only forward them to the developer. Can someone please give me examples where current state lacks and how they are addressed by the proposal(s)? From mail at nh2.me Fri May 8 17:47:03 2020 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Fri, 8 May 2020 19:47:03 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> Message-ID: <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> On 5/8/20 7:32 PM, Henning Thielemann wrote: > This confirms that they are not for you, but you only forward them to the developer. Yes, stack traces are in general for developers. > Can someone please give me examples where current state lacks * Currently stack traces are not printed, so users cannot forward them to the developer, even if both the users and the developers would like that. * Developers cannot easily produce stack traces do debug unintended exceptions. From lemming at henning-thielemann.de Fri May 8 17:52:13 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Fri, 8 May 2020 19:52:13 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> Message-ID: On Fri, 8 May 2020, Niklas Hambüchen wrote: > On 5/8/20 7:32 PM, Henning Thielemann wrote: > >> Can someone please give me examples where current state lacks > > * Currently stack traces are not printed, so users cannot forward them > to the developer, even if both the users and the developers would like > that. We are talking about the HasCallStack stack traces, yes? How is their emission addressed by extending exceptions with stack traces? > * Developers cannot easily produce stack traces do debug unintended > exceptions. What are "unintended exceptions"? What is an example of an "unintended exception"? From mail at nh2.me Fri May 8 18:00:29 2020 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Fri, 8 May 2020 20:00:29 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> Message-ID: <517044fa-4fb2-cde6-3a8c-b1013f7b06f0@nh2.me> On 5/8/20 7:52 PM, Henning Thielemann wrote: > We are talking about the HasCallStack stack traces, yes? > How is their emission addressed by extending exceptions with stack traces? The way I understand the proposal, we may be equally talking about DWARF or profiling cost-center based stack traces. >From a debugging perspective, I guess the developer does not care so much about which implementation is used, as long as the trace points out the code path that led to the creation of the exception. > What are "unintended exceptions"? > What is an example of an "unintended exception"? A recent example from my production server: hPutBuf: resource vanished (Broken pipe) From ben at well-typed.com Fri May 8 18:03:41 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 08 May 2020 14:03:41 -0400 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> Message-ID: <87mu6i1eyw.fsf@smart-cactus.org> Henning Thielemann writes: > On Fri, 8 May 2020, Niklas Hambüchen wrote: > >> On 5/8/20 5:37 PM, Henning Thielemann wrote: >> >>> a callstack is not useful for a user. >> >> Call stacks have been very useful to me as a user of non-Haskell tools >> so far, because they are excellent for attaching to bug reports and >> usually led to developers fixing my problems faster. > > This confirms that they are not for you, but you only forward them to the > developer. > > > Can someone please give me examples where current state lacks and how they > are addressed by the proposal(s)? We can debate whether partial functions like `fromJust` should exist; however, the fact of the matter is that they do exist and they are used. Furthermore, even `base`'s own IO library (e.g. `openFile`) uses synchronous exceptions to report errors. This becomes particularly painful when building large systems: Even if I am careful to avoid such functions in my own code, as my dependency footprint grows it becomes more likely that some transitive dependency will expose a partial interface (perhaps even without my knowledge). This is a problem that industrial users are all too familiar with. Perhaps this helps to shed some light on the motivation? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri May 8 18:18:38 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 08 May 2020 14:18:38 -0400 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> Message-ID: <87k11m1e9z.fsf@smart-cactus.org> Henning Thielemann writes: > On Fri, 8 May 2020, Niklas Hambüchen wrote: > >> On 5/8/20 7:32 PM, Henning Thielemann wrote: >> >>> Can someone please give me examples where current state lacks >> >> * Currently stack traces are not printed, so users cannot forward them >> to the developer, even if both the users and the developers would like >> that. > > We are talking about the HasCallStack stack traces, yes? > How is their emission addressed by extending exceptions with stack > traces? HasCallStack stack traces are one type of backtrace that the proposal supports. However, it's not the only (nor is it even the most useful sort, in my opinion). Other mechanisms include cost center stacks from the cost-center profiler and native stack unwinding. > >> * Developers cannot easily produce stack traces do debug unintended >> exceptions. > > What are "unintended exceptions"? > What is an example of an "unintended exception"? For instance, * Somewhere deep in my code a colleague used `fromJust` due to a miscommunicated invariant * Somewhere in my system a `writeFile "tmp" $ repeat 'a'` failed due to filling the disk * Somewhere in my system I have a partial pattern match in a module which was compiled without -Wall * Somewhere in my system I `div` by zero due to lack of input validation * I use a record selector on a sum. * A logic error results in an assertion failure deep in my program, but it's unclear which path my program took to arrive at the assertion This list could go on and on... Currently the proposal does not cover asynchronous exceptions but it wouldn't be particularly hard to extend it in this direction. This would allow far better reporting of heap/stack overflows and MVar deadlocks (which are particularly hard to debug at the moment). Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From simonpj at microsoft.com Fri May 8 21:07:59 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 8 May 2020 21:07:59 +0000 Subject: Unique invariants: GHC invocation vs scope? In-Reply-To: References: Message-ID: Generally, there is no assumption that binders are unique, anywhere in GHC. The Simplifier does remove gratuitious shadowing (like (\x. \x. e)), but there is really no guarantee at any stage. Simon From: ghc-devs On Behalf Of Csaba Hruska Sent: 08 May 2020 14:03 To: GHC developers Subject: Unique invariants: GHC invocation vs scope? Hello, What are the intended invariants of Unique values? So far I thought that it is always gives a unique identity to things during a whole GHC invocation. However this is not true. I.e. eta expansion breaks this assumption. Here is a snippet from the STG IR of GHC.PrimopWrappers: extendInt8# [InlPrag=NOINLINE] :: Int8# -> Int# [GblId, Arity=1, Unf=OtherCon []] = \r [eta_B1] extendInt8# [eta_B1]; ord# [InlPrag=NOINLINE] :: Char# -> Int# [GblId, Arity=1, Unf=OtherCon []] = \r [eta_B1] ord# [eta_B1]; Where eta_B1 is redefined, so it is only unique in the scope. Is this intentional? Thanks, Csaba Hruska -------------- next part -------------- An HTML attachment was scrubbed... URL: From compl.yue at gmail.com Sat May 9 06:35:17 2020 From: compl.yue at gmail.com (Compl Yue) Date: Sat, 9 May 2020 14:35:17 +0800 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <87k11m1e9z.fsf@smart-cactus.org> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <87k11m1e9z.fsf@smart-cactus.org> Message-ID: This reminds me a joke to put it in a humorous way: > A software QA engineer walks into a bar. He orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. > First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone. LOL, Compl > On 2020-05-09, at 02:18, Ben Gamari wrote: > > Henning Thielemann writes: > >> On Fri, 8 May 2020, Niklas Hambüchen wrote: >> >>> On 5/8/20 7:32 PM, Henning Thielemann wrote: >>> >>>> Can someone please give me examples where current state lacks >>> >>> * Currently stack traces are not printed, so users cannot forward them >>> to the developer, even if both the users and the developers would like >>> that. >> >> We are talking about the HasCallStack stack traces, yes? >> How is their emission addressed by extending exceptions with stack >> traces? > > HasCallStack stack traces are one type of backtrace that the proposal > supports. However, it's not the only (nor is it even the most useful > sort, in my opinion). > > Other mechanisms include cost center stacks from the cost-center > profiler and native stack unwinding. > >> >>> * Developers cannot easily produce stack traces do debug unintended >>> exceptions. >> >> What are "unintended exceptions"? >> What is an example of an "unintended exception"? > > For instance, > > * Somewhere deep in my code a colleague used `fromJust` due to a > miscommunicated invariant > > * Somewhere in my system a `writeFile "tmp" $ repeat 'a'` failed due to > filling the disk > > * Somewhere in my system I have a partial pattern match in a module > which was compiled without -Wall > > * Somewhere in my system I `div` by zero due to lack of input > validation > > * I use a record selector on a sum. > > * A logic error results in an assertion failure deep in my program, but > it's unclear which path my program took to arrive at the assertion > > This list could go on and on... > > Currently the proposal does not cover asynchronous exceptions but it > wouldn't be particularly hard to extend it in this direction. This would > allow far better reporting of heap/stack overflows and MVar deadlocks > (which are particularly hard to debug at the moment). > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From mgajda at mimuw.edu.pl Sat May 9 12:23:57 2020 From: mgajda at mimuw.edu.pl (Michal J Gajda) Date: Sat, 9 May 2020 14:23:57 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information Message-ID: Ben, I agree with you that is a great idea! I can add a few more real world examples: * we get exception from a foreign library that we bound, * we get an exception from a platform (I believe Windows supports throwing exceptions to programs), * user presses CTRL-C and we want to know where our program hanged. * we get infamous <>, because in theory nobody wants non-terminating programs, but in practice everybody gets them sometimes. I also use `ExceptT`, `EitherT` for processing large sets of data, because that allows me to contain the errors efficiently. However from time to time, I get an error to blow up **and I cannot even locate which library was guilty**. It would be nice to extract them automatically and put them into error database before they are prioritized. -- Cheers Michał From matthewtpickering at gmail.com Sun May 10 10:22:06 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sun, 10 May 2020 11:22:06 +0100 Subject: Removal of UniqMap module Message-ID: Hi, I noticed that the UniqMap module was removed from the tree See 1c7c6f1afc8e7f7ba5d256780bc9d5bb5f3e7601 Why was it removed? I needed it today and now I am unsure what the suitable replacement is. As a general point, please can we stop with these annoying refactorings which delete unused code and move around definitions for no clear purpose. The amount of churn on the code base has wasted me a lot of time recently. Cheers, Matt From rae at richarde.dev Sun May 10 12:43:33 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Sun, 10 May 2020 13:43:33 +0100 Subject: Removal of UniqMap module In-Reply-To: References: Message-ID: <3414DAA4-6405-426C-BB4C-063AA0B32BE0@richarde.dev> > On May 10, 2020, at 11:22 AM, Matthew Pickering wrote: > > Hi, > > I noticed that the UniqMap module was removed from the tree > > See 1c7c6f1afc8e7f7ba5d256780bc9d5bb5f3e7601 > > Why was it removed? From the commit message: "This module isn't used anywhere in GHC." That seems like a good reason to remove, to me. While I can understand the frustration at having this disappear when you need it, we can't quite just keep whole unused modules around in the hope that someone someday will use them. > I needed it today and now I am unsure what the > suitable replacement is. A UniqFM whose range includes the domain element would work fine, I think. > > As a general point, please can we stop with these annoying > refactorings which delete unused code I disagree here. A codebase as large and sprawling as GHC's needs constant pruning. The alternative is not to control the sprawl, and that seems considerably worse than refactorings and churn. Richard From klebinger.andreas at gmx.at Sun May 10 12:54:23 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Sun, 10 May 2020 14:54:23 +0200 Subject: Removal of UniqMap module In-Reply-To: <3414DAA4-6405-426C-BB4C-063AA0B32BE0@richarde.dev> References: <3414DAA4-6405-426C-BB4C-063AA0B32BE0@richarde.dev> Message-ID: My two cents: > I needed it today and now I am unsure what the > suitable replacement is. If you need it (inside GHC) just re add it. If you think it should be kept as part of GHC's API re add it and add a comment stating such. Iirc it was removed since no one used it, not because anything was wrong with it design wise iirc. > As a general point, please can we stop with these annoying > refactorings which delete unused code In general I think these kind of refactorings are needed and fine. But it does feel like people became a bit overzealous in that regard recently. Cheers Andreas Richard Eisenberg schrieb am 10.05.2020 um 14:43: > >> On May 10, 2020, at 11:22 AM, Matthew Pickering wrote: >> >> Hi, >> >> I noticed that the UniqMap module was removed from the tree >> >> See 1c7c6f1afc8e7f7ba5d256780bc9d5bb5f3e7601 >> >> Why was it removed? > From the commit message: "This module isn't used anywhere in GHC." That seems like a good reason to remove, to me. While I can understand the frustration at having this disappear when you need it, we can't quite just keep whole unused modules around in the hope that someone someday will use them. > >> I needed it today and now I am unsure what the >> suitable replacement is. > A UniqFM whose range includes the domain element would work fine, I think. > >> As a general point, please can we stop with these annoying >> refactorings which delete unused code > I disagree here. A codebase as large and sprawling as GHC's needs constant pruning. The alternative is not to control the sprawl, and that seems considerably worse than refactorings and churn. > > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Mon May 11 15:48:09 2020 From: ben at well-typed.com (Ben Gamari) Date: Mon, 11 May 2020 11:48:09 -0400 Subject: Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> Message-ID: <87blmu1nih.fsf@smart-cactus.org> Michael Sloan writes: > Thanks so much for making a proposal for this, Ben!! It's great to see > progress here. > > I'm also glad that there is now a proposal process. I made a fairly > similar proposal almost exactly 5 years ago to the libraries list - > https://mail.haskell.org/pipermail/libraries/2015-April/025471.html - but > without the subtlety of particular backtrace representations. Skimming the > ensuing thread may still be informative. > Thanks for the reference, Michael! My feeling is that the proposal in that thread is a bit too dynamic. That being said, I can see the argument for wanting, for instance, a robust way to determine that an exception is asynchronous. > In particular, there is one thing I would like to highlight from that old > proposal. I think it'd be good to have a standard way to represent a chain > of exceptions, and build this into `catch` and `finally`. Python and Java > both have a mechanism for this, and both refer to it as a "cause" > exception. When an exception is thrown during exception handling, the > exception being handled is preserved as its "cause". I find this mechanism > to be incredibly useful in Java, it has made the underlying issue much > clearer in many cases, and in other cases at least provides helpful > context. I have no doubt such a mechanism would have saved me many hours > of debugging exceptions in Haskell systems I've worked on in the past. > > I considered commenting about that directly on the proposal, but I figure > this is a better place to suggest expanding the scope of the change :) . > Totally understandable if you want to keep this proposal focused on > stacktraces, but I think it'd be good to consider this as a potential > future improvement. > Indeed I can see the point. I'll keep this point in the back of my mind. I'm not eager to further expand the scope of the proposal at the moment, but we should be certain that the backtrace design doesn't unintentionally close the door to this use-case. However, one question I would have is whether the exception-chaining use-case *needs* to be handled in SomeException. For instance, you could rather leave this to user code. You might even give this pattern a typeclass. For instance, class HasChainedException e where getChainedException :: e -> Maybe SomeException data MyException = MyException { causedBy :: SomeException } instance HasChainedException MyException where getChainedException = causedBy Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mark.karpov at tweag.io Tue May 12 17:37:34 2020 From: mark.karpov at tweag.io (Karpov, Mark) Date: Tue, 12 May 2020 19:37:34 +0200 Subject: Inclusion of the parser refactor in 8.12 Message-ID: Hello, I'd like to mention that it would be nice if this patch https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2377 could be included in 8.12. Right now we have a few issues that Ormolu users have reported which stem from how Haddocks are parsed by GHC (e.g. that issue when a file starts with haddock it must have a module header). I originally planned to work on those myself, but today I discovered that Vlad is already working on it. Best, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Tue May 12 17:45:24 2020 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Tue, 12 May 2020 18:45:24 +0100 Subject: Inclusion of the parser refactor in 8.12 In-Reply-To: References: Message-ID: This would also help to resolve https://github.com/digital-asset/ghcide/pull/350#discussion_r370878197, where we cannot get ApiAnnotations and Haddock comments at the same time. On Tue, 12 May 2020 at 18:38, Karpov, Mark wrote: > Hello, > > I'd like to mention that it would be nice if this patch > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2377 could be > included in 8.12. Right now we have a few issues that Ormolu users have > reported which stem from how Haddocks are parsed by GHC (e.g. that issue > when a file starts with haddock it must have a module header). I originally > planned to work on those myself, but today I discovered that Vlad is > already working on it. > > Best, > > Mark > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemming at henning-thielemann.de Tue May 12 20:55:08 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 12 May 2020 22:55:08 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <517044fa-4fb2-cde6-3a8c-b1013f7b06f0@nh2.me> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <517044fa-4fb2-cde6-3a8c-b1013f7b06f0@nh2.me> Message-ID: On Fri, 8 May 2020, Niklas Hambüchen wrote: >> What are "unintended exceptions"? >> What is an example of an "unintended exception"? > > A recent example from my production server: > > hPutBuf: resource vanished (Broken pipe) Ok, I lookup the Haddock comment of hPutBuf and it says: "This operation may fail with: * ResourceVanished if the handle is a pipe or socket, and the reading end is closed." That is, ResourceVanished is part of the public interface and in no way unexpected (or what "unintended" may be). I would prefer to make this explicit in the type of hPutBuf: hPutBuf :: (ResourceVanishedException e) => Handle -> Ptr a -> Int -> ExceptT e IO () Now, what do you intend to do with the call-stack? Isn't it something you can attach to the e value? From lemming at henning-thielemann.de Tue May 12 21:06:35 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 12 May 2020 23:06:35 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <87mu6i1eyw.fsf@smart-cactus.org> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <87mu6i1eyw.fsf@smart-cactus.org> Message-ID: On Fri, 8 May 2020, Ben Gamari wrote: > We can debate whether partial functions like `fromJust` should exist; however, > the fact of the matter is that they do exist and they are used. That's not my point. I say: fromJust on Nothing is a programming error, ok. We must debug this. HasCallStack helps here. However, it does not have to do with exceptions or with the proposals as I understand them. > Furthermore, even `base`'s own IO library (e.g. `openFile`) uses > synchronous exceptions to report errors. Right. I say: Such exceptions are part of the public interface and should be expressed in types. If you encounter any problems when not doing this, I would first try to solve the problem with exceptions explicit in the type. E.g. Haddock for openFile says: This operation may fail with: * isAlreadyInUseError ... * isDoesNotExistError ... * isPermissionError ... Thus the type should be: openFile :: (AlreadyInUseException e, DoesNotExistException e, PermissionException e) => FilePath -> IOMode -> ExceptT e IO Handle > Perhaps this helps to shed some light on the motivation? Unfortunately no. I only see the immortal confusion about (programming) errors vs. (IO) exceptions. And I think that part of this confusion is that IO exceptions in 'base' are hidden in the IO type and that there are hybrid functions like 'throw' that can be called like 'error' but they cause IO exceptions that can be caught by 'catch'. From lemming at henning-thielemann.de Tue May 12 21:29:19 2020 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue, 12 May 2020 23:29:19 +0200 (CEST) Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: <87k11m1e9z.fsf@smart-cactus.org> References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <87k11m1e9z.fsf@smart-cactus.org> Message-ID: On Fri, 8 May 2020, Ben Gamari wrote: > Henning Thielemann writes: > >> We are talking about the HasCallStack stack traces, yes? >> How is their emission addressed by extending exceptions with stack >> traces? > > HasCallStack stack traces are one type of backtrace that the proposal > supports. However, it's not the only (nor is it even the most useful > sort, in my opinion). > > Other mechanisms include cost center stacks from the cost-center > profiler and native stack unwinding. Interesting. That's a completely new thing. >>> * Developers cannot easily produce stack traces do debug unintended >>> exceptions. >> >> What are "unintended exceptions"? >> What is an example of an "unintended exception"? > > For instance, > > * Somewhere deep in my code a colleague used `fromJust` due to a > miscommunicated invariant That's a programming error. > * Somewhere in my system a `writeFile "tmp" $ repeat 'a'` failed due to > filling the disk Hm, that's also a programming error, but it ends in an IO exception. If it would not end in an IO exception (e.g. writing to /dev/null) it would go to an infinite loop. Anyway, it is a programming error. However it is an unchecked one. That is, there is no warranty that you can catch it by a debugger. So I do not think you can achieve much with callstacks here. > * Somewhere in my system I have a partial pattern match in a module > which was compiled without -Wall Programming error and btw. before thinking about a GHC extension I would enable -Wall ... > * Somewhere in my system I `div` by zero due to lack of input > validation Programming error > * I use a record selector on a sum. Programming error > * A logic error results in an assertion failure deep in my program, but > it's unclear which path my program took to arrive at the assertion Sounds like Programming error > This list could go on and on... >From your list of examples I deduce that the proposal is about programming errors. But we have HasCallStack for that one. How does the proposal improve or alter the HasCallStack solution? And how does it relate to the IO exception system with hierarchical exceptions and SomeException and so on? > Currently the proposal does not cover asynchronous exceptions but it > wouldn't be particularly hard to extend it in this direction. This would > allow far better reporting of heap/stack overflows and MVar deadlocks > (which are particularly hard to debug at the moment). Hm, what kind of heap or stack overflow are you thinking of? A stack overflow sounds like unlimited recursion and thus like a programming error. In contrast to that, a program must be prepared for a failure of "malloc". Memory exhaustion is an IO exception, it should be explicit in the type. Are MVar deadlocks always detected by the runtime system? From mail at nh2.me Tue May 12 22:14:30 2020 From: mail at nh2.me (=?UTF-8?Q?Niklas_Hamb=c3=bcchen?=) Date: Wed, 13 May 2020 00:14:30 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <517044fa-4fb2-cde6-3a8c-b1013f7b06f0@nh2.me> Message-ID: On 5/12/20 10:55 PM, Henning Thielemann wrote: > "This operation may fail with: > > * ResourceVanished if the handle is a pipe or socket, and the reading end is closed." > > That is, ResourceVanished is part of the public interface and in no way unexpected (or what "unintended" may be). I would prefer to make this explicit in the type of hPutBuf: > > hPutBuf :: >    (ResourceVanishedException e) => >    Handle -> Ptr a -> Int -> ExceptT e IO () > > Now, what do you intend to do with the call-stack? Isn't it something you can attach to the e value? Why is this relevant? The point of debugging is to find programming errors. It does not matter what the Haddocks say; if a programmer uses the function wrong, the exception will occur. hPutBuf does not currently have that type, nor can anybody rewrite all the existing libraries easily. The point of the proposal is to make the RTS help us debug problems in code as it exists today. From hecate at glitchbra.in Wed May 13 10:55:52 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Wed, 13 May 2020 12:55:52 +0200 Subject: A future for the Windows packaging situation. Message-ID: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> Dear GHC devs, dear maintainers, Following a discussion that took place on #ghc, I wish to spread it to the whole mailing-list, in order to receive some feedback, and plan for the future now that it has become clear that the present is rather bleak. As some of you may have seen from the long threads in haskell-cafe@, countless steps of various difficulty for Windows users (excluding power-users) need to be taken in order to have a proper working GHC / Haskell installation on their machine. Moreover, some defiance against Chocolatey has come to our ears, due to the mailing-list registration form that appears when one desires to download this package manager. I shall speak for myself by saying that I do not wish the that the Windows Haskell developers need to become a special combo of Chocolatey maintainers and Windows power users. Some GNU/Linux distributions such as Exherbo have made this their creed, the major difference being that they actually give the tools to make such a thing possible. The point of my email to you all is the following: I suggest that Haskell.org, the 501(c)(3) established in NY which, If I am not mistaken, holds the funds from various individual donations, the Amazon Smile programme and Software in the Public Interest grants, hires a company to establish a strong technological basis regarding Windows packaging. I am not talking of delegating the maintaining task to an external entity, but to provide the foundations upon which volunteers will be able to keep things running. Training in such matters would also be beneficial, so that newcomers can learn on the spot how to best interact with this. Their contract would involve the initial setup of CI tasks able to produce MSIX packages, while the people in charge of the haskell.org landing page would ease the user experience by providing clearer ways to install GHC on various platforms. Ideally we could have a GUI to install libraries easily, like many GNU/Linux package managers offer. That being said, I was also suggested the idea of a grant and/or sponsorship. What we need is less a capitalist framework around that task and more of an incentive to invest a serious amount of work and quality so that it becomes, at last, the non-issue it should have always been. The important thing to keep in mind is that the GNU/Linux and macOS users *cannot* hold the Windows users to the same standards in terms of CLI usability. I cannot weigh in my opinion on the most recent iterations of PowerShell, but Windows XP's cmd.exe was excruciating, to say the least. Now, I know some of you will prefer to have this task handled by competent volunteers, but I am under the moral obligation to say that expecting salvation and better tomorrows from people who have yet to make their presence known in the thirty years of existence of our dear language, is at best mild delusion, at worse folly that will only widen the gap between what is needed to get Haskell up and running smoothly on the Windows platform and the average skill of Windows users. I am not suggesting that my email is The True Way to follow so that everything is fixed forever, and if we can, as a community, arrive to some satisfying workflow that would benefit rather than alienate our Windows user base, this would would be wonderful. Thank you for reading until the end. Cheers, Hécate. PS: I am in no way trying to berate anyone for their implied incompetence, or imply that Windows users are stupid and/or technologically impaired. This would be misinterpreting my words and lead nowhere but to another OS war on another mailing-list. PPS: I am serious. Please stay on-topic. PPPS: I hold no share, no money or any other form of capital in any Windows packaging company we might or might not end up hiring for the task. I am speaking of experience, for my company used an external contractor to work on our landing (non-product) page, while all hands were on deck to support the product development effort. This allowed us to have a strong foundation to iterate on, and bought us countless hours of development time. From ben.franksen at online.de Wed May 13 14:15:17 2020 From: ben.franksen at online.de (Ben Franksen) Date: Wed, 13 May 2020 16:15:17 +0200 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <87k11m1e9z.fsf@smart-cactus.org> Message-ID: Am 12.05.20 um 23:29 schrieb Henning Thielemann: > A stack overflow sounds like unlimited recursion and thus like a > programming error. Perhaps it was just one recursion to many? Computer memory is limited. Heap overflow is also quite possible even with a program that is provably terminating. I have used 'ulimit -v' in the past to force ghc to fail rather than having to reboot my machine :-/ > In contrast to that, a program must be prepared for a > failure of "malloc". I don't see any essential difference between allocation by the runtime and explicit allocation using malloc. I think this is a good thing that in Haskell you /can/ recover from such a condition. > Memory exhaustion is an IO exception, it should be > explicit in the type. Then it must be explicit in all types, since in general all computations may exhaust the available memory. And then what use would that type information have? > Are MVar deadlocks always detected by the runtime system? My guess is that deadlock detection in general is undecidable i.e. with more than one MVar present, but I may be wrong about that. Cheers Ben From david.macek.0 at gmail.com Wed May 13 16:04:19 2020 From: david.macek.0 at gmail.com (David Macek) Date: Wed, 13 May 2020 18:04:19 +0200 Subject: A future for the Windows packaging situation. In-Reply-To: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> References: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> Message-ID: On Wed, May 13, 2020 at 12:56 PM Hécate wrote: > As some of you may have seen from the long threads in haskell-cafe@, > countless steps of various difficulty for Windows users > (excluding power-users) need to be taken in order to have a proper > working GHC / Haskell installation on their machine. Could you (or someone) summarize these issues and steps? I remember one of the big issues was *network* and the likes (which is frankly an issue with other languages as well), but from the sound of this post, I assume there's more. > Their contract would involve the initial setup of CI tasks able to > produce MSIX packages Hopefully that wouldn't become the only way to download GHC. > Ideally we could have a GUI to install libraries easily, like many > GNU/Linux package managers offer. Sounds great, but is it reasonable? GNU/Linux package managers AFAIK don't install Haskell libraries either. > The important thing to keep in mind is that the GNU/Linux and macOS > users *cannot* hold the Windows users to the same standards in > terms of CLI usability. It's true that one-liners and scripting is harder in Cmd, but simple commands (no pipes, no flow control etc.) that are the bread and butter for developers work just fine. I sense I'm missing some context; why is this an issue? -- David Macek From hecate at glitchbra.in Wed May 13 16:58:12 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Wed, 13 May 2020 18:58:12 +0200 Subject: A future for the Windows packaging situation. In-Reply-To: References: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> Message-ID: >Hopefully that wouldn't become the only way to download GHC. That wasn't my intention to suggest it to be the One True Way to download it. >Sounds great, but is it reasonable? GNU/Linux package managers AFAIK don't install Haskell libraries either Some do! On Fedora, the Haskell libraries that are shipped are prefixed with `ghc-`, and Arch Linux has become quite infamous due to the way they ship dynamically-linked Haskell libraries and executables. By GUI I was thinking of something like DNFDragora[0] and its Ubuntu counterpart (whose name I forgot). >It's true that one-liners and scripting is harder in Cmd, but simple commands (no pipes, no flow control etc.) that are the bread and butter for developers work just fine.  I sense I'm missing some context; why is this an issue? cmd.exe is fundamentally a foreign interface to most Windows users, even for developers. IDEs and GUIs have reigned for a long long time in Windows Land, and Microsoft has no will to change this state of fact. WSL is a tool developed to Linux / macOS power-users who incidentally need to deal with Windows, or needed an argument to switch to the platform. In addition to all of this, shipping a .tar.lz format was a terrible mistake, for no tool shipped with Windows is able to properly handle it, but I do not wish to blame anyone, it just needs to be changed to .zip and we can be done with it. [0]: https://github.com/manatools/dnfdragora From compl.yue at gmail.com Thu May 14 09:03:48 2020 From: compl.yue at gmail.com (Compl Yue) Date: Thu, 14 May 2020 17:03:48 +0800 Subject: A future for the Windows packaging situation. In-Reply-To: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> References: <7e236efd-900c-c6cb-fffc-5581674db91e@glitchbra.in> Message-ID: Hey, respectful devs, Just now I see https://coder.com/ (open source at https://github.com/cdr/code-server ), then I realize it may make a good option for windows users to feel home in starting Haskell development. They happen to be talking about releasing a windows binary from current basis that only Linux/macOS are supported - https://github.com/cdr/code-server/issues/1397#issuecomment-627662902 Their mac experience is right about downloading a .zip file from https://github.com/cdr/code-server/releases , unpack it, double-click an executable (security option needs to be tuned as a mac thing), then goto http://localhost:8080/ and voila: Haskell Language Serveralanz.vscode-hie-server can be installed right away I'd think GHC and Cabal-install can be bundled similarly, with such an IDE and batteries of HIE based extensions. Maybe some day HLS can hook ghcup/ghcups/stack up to install GHC wrt per project specification, in a cross-platform way. All the best, Compl > On 2020-05-13, at 18:55, Hécate wrote: > > Dear GHC devs, dear maintainers, > > Following a discussion that took place on #ghc, I wish to spread it to the whole mailing-list, in order to receive some feedback, > and plan for the future now that it has become clear that the present is rather bleak. > > As some of you may have seen from the long threads in haskell-cafe@, countless steps of various difficulty for Windows users > (excluding power-users) need to be taken in order to have a proper working GHC / Haskell installation on their machine. > Moreover, some defiance against Chocolatey has come to our ears, due to the mailing-list registration form that appears > when one desires to download this package manager. I shall speak for myself by saying that I do not wish the that the Windows > Haskell developers need to become a special combo of Chocolatey maintainers and Windows power users. > Some GNU/Linux distributions such as Exherbo have made this their creed, the major difference being that they actually give > the tools to make such a thing possible. > > The point of my email to you all is the following: I suggest that Haskell.org, the 501(c)(3) established in NY which, If I am not mistaken, > holds the funds from various individual donations, the Amazon Smile programme and Software in the Public Interest grants, > hires a company to establish a strong technological basis regarding Windows packaging. I am not talking of delegating the maintaining task > to an external entity, but to provide the foundations upon which volunteers will be able to keep things running. > Training in such matters would also be beneficial, so that newcomers can learn on the spot how to best interact with this. > > Their contract would involve the initial setup of CI tasks able to produce MSIX packages, while the people in charge of the haskell.org > landing page would ease the user experience by providing clearer ways to install GHC on various platforms. > Ideally we could have a GUI to install libraries easily, like many GNU/Linux package managers offer. > > That being said, I was also suggested the idea of a grant and/or sponsorship. What we need is less a capitalist framework around that task > and more of an incentive to invest a serious amount of work and quality so that it becomes, at last, the non-issue it should have always been. > > The important thing to keep in mind is that the GNU/Linux and macOS users *cannot* hold the Windows users to the same standards in > terms of CLI usability. I cannot weigh in my opinion on the most recent iterations of PowerShell, but Windows XP's cmd.exe was > excruciating, to say the least. > > Now, I know some of you will prefer to have this task handled by competent volunteers, but I am under the moral obligation to say > that expecting salvation and better tomorrows from people who have yet to make their presence known in the thirty years of existence > of our dear language, is at best mild delusion, at worse folly that will only widen the gap between what is needed to get Haskell up and > running smoothly on the Windows platform and the average skill of Windows users. > > I am not suggesting that my email is The True Way to follow so that everything is fixed forever, > and if we can, as a community, arrive to some satisfying workflow that would benefit rather than alienate > our Windows user base, this would would be wonderful. > > > Thank you for reading until the end. > > Cheers, > Hécate. > > PS: I am in no way trying to berate anyone for their implied incompetence, or imply that Windows users are stupid and/or technologically impaired. > This would be misinterpreting my words and lead nowhere but to another OS war on another mailing-list. > PPS: I am serious. Please stay on-topic. > PPPS: I hold no share, no money or any other form of capital in any Windows packaging company we might or might not end up hiring for the task. > I am speaking of experience, for my company used an external contractor to work on our landing (non-product) page, while all hands were on deck > to support the product development effort. This allowed us to have a strong foundation to iterate on, and bought us countless hours of development time. > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-1.png Type: image/png Size: 308388 bytes Desc: not available URL: From ben at well-typed.com Thu May 14 14:24:52 2020 From: ben at well-typed.com (Ben Gamari) Date: Thu, 14 May 2020 10:24:52 -0400 Subject: [HIW'20] Second Call for Talks In-Reply-To: <878sk08f4a.fsf@smart-cactus.org> References: <878sk08f4a.fsf@smart-cactus.org> Message-ID: <87h7wizj9u.fsf@smart-cactus.org> Hello everyone, Haskell Implementors Workshop is calling for talk proposals. Co-located with ICFP, HiW is an ideal place to describe a Haskell library, a Haskell extension, compiler, works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. The deadline for submissions is July 2nd 2020. Call for Talks ============== The 12th Haskell Implementors’ Workshop is to be held alongside ICFP 2020 this year. It is a forum for people involved in the design and development of Haskell implementations, tools, libraries, and supporting infrastructure, to share their work and discuss future directions and collaborations with others. Talks and/or demos are proposed by submitting an abstract, and selected by a small program committee. There will be no published proceedings. The workshop will be informal and interactive, with open spaces in the timetable and room for ad-hoc discussion, demos and lightning talks. Scope and Target Audience ------------------------- It is important to distinguish the Haskell Implementors’ Workshop from the Haskell Symposium which is also co-located with ICFP 2020. The Haskell Symposium is for the publication of Haskell-related research. In contrast, the Haskell Implementors’ Workshop will have no proceedings – although we will aim to make talk videos, slides and presented data available with the consent of the speakers. The Implementors’ Workshop is an ideal place to describe a Haskell extension, describe works-in-progress, demo a new Haskell-related tool, or even propose future lines of Haskell development. Members of the wider Haskell community encouraged to attend the workshop – we need your feedback to keep the Haskell ecosystem thriving. Students working with Haskell are specially encouraged to share their work. The scope covers any of the following topics. There may be some topics that people feel we’ve missed, so by all means submit a proposal even if it doesn’t fit exactly into one of these buckets: - Compilation techniques - Language features and extensions - Type system implementation - Concurrency and parallelism: language design and implementation - Performance, optimization and benchmarking - Virtual machines and run-time systems - Libraries and tools for development or deployment Talks ----- We invite proposals from potential speakers for talks and demonstrations. We are aiming for 20-minute talks with 5 minutes for questions and changeovers. We want to hear from people writing compilers, tools, or libraries, people with cool ideas for directions in which we should take the platform, proposals for new features to be implemented, and half-baked crazy ideas. Please submit a talk title and abstract of no more than 300 words. Submissions can be made via HotCRP at https://icfp-hiw20.hotcrp.com/ until July 2nd (anywhere on earth). We will also have lightning talks session. These have been very well received in recent years, and we aim to increase the time available to them. Lightning talks be ~7mins and are scheduled on the day of the workshop. Suggested topics for lightning talks are to present a single idea, a work-in-progress project, a problem to intrigue and perplex Haskell implementors, or simply to ask for feedback and collaborators. Logistics --------- Due to the on-going COVID-19 situation, ICFP (and, consequently, HIW) will be held remotely this year. However, the organizers are still working hard to provide for a great workshop experience. While we are sad that this year will lack the robust hallway track that is often the highlight of HIW, we believe that this remote workshop presents a unique opportunity to include more of the Haskell community in our discussion and explore new modes of communicating with our colleagues. We hope that you will join us in making this HIW as vibrant as any other. Program Committee ----------------- - Andrey Mokhov (Newcastle University) - Ben Gamari (Well-Typed LLP) - Christian Baaij (QBayLogic) - George Karachalias (Tweag I/O) - Klara Marntirosian (KU Leuven) - Matthew Pickering (Univeristy of Bristol) - Ryan G.L. Scott (Indiana University Bloomington) Best wishes, Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From twhitehead at gmail.com Thu May 14 16:06:37 2020 From: twhitehead at gmail.com (Tyson Whitehead) Date: Thu, 14 May 2020 12:06:37 -0400 Subject: [Haskell-cafe] Decorating exceptions with backtrace information In-Reply-To: References: <87r1vv1kdx.fsf@smart-cactus.org> <0c11ba4f-90af-b56e-e21d-6c4b1eba9144@nh2.me> <56a36bd0-3c98-29fd-ad26-8ca4f99d8d66@nh2.me> <87k11m1e9z.fsf@smart-cactus.org> Message-ID: On Tue, 12 May 2020 at 17:30, Henning Thielemann < lemming at henning-thielemann.de> wrote: > From your list of examples I deduce that the proposal is about programming > errors. But we have HasCallStack for that one. How does the proposal > improve or alter the HasCallStack solution? And how does it relate to the > IO exception system with hierarchical exceptions and SomeException and so > on? > As a parallel item, maybe it would be good if incomplete patterns could have a HasCallStack constraint so the current "Non-exaustive patterns in function foo" message could be extended with a helpful backtrace? If a programmer doesn't want this, then they could use complete matches? -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri May 15 14:24:05 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 15 May 2020 14:24:05 +0000 Subject: Strange library git glitch Message-ID: I'm getting this strange output from 'git diff' bash$ git diff diff --git a/libraries/exceptions b/libraries/exceptions --- a/libraries/exceptions +++ b/libraries/exceptions @@ -1 +1 @@ -Subproject commit fe4166f8d23d8288ef2cbbf9e36118b6b99e0d7d +Subproject commit fe4166f8d23d8288ef2cbbf9e36118b6b99e0d7d-dirty No amount of git submodule update makes it go away. Any ideas? Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From hecate at glitchbra.in Fri May 15 14:30:38 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Fri, 15 May 2020 16:30:38 +0200 Subject: Strange library git glitch In-Reply-To: References: Message-ID: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> Hi Simon, I usually manage to get it to disappear by using `git submodule update --recursive`. Is it a flag you've used in your previous attempts? Cheers, Hécate. Le 15/05/2020 à 16:24, Simon Peyton Jones via ghc-devs a écrit : > > No amount of git submodule update makes it go away.  Any ideas? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hecate at glitchbra.in Fri May 15 14:37:11 2020 From: hecate at glitchbra.in (=?UTF-8?Q?H=c3=a9cate?=) Date: Fri, 15 May 2020 16:37:11 +0200 Subject: Strange library git glitch In-Reply-To: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> References: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> Message-ID: <95a09a55-5955-778a-9e27-e2287f634f40@glitchbra.in> My bad, it seems like this is another issue. I found this StackOverflow page quite helpful in explaining the hows and whys: https://stackoverflow.com/questions/4873980/git-diff-says-subproject-is-dirty Cheers, Hécate Le 15/05/2020 à 16:30, Hécate a écrit : > > Hi Simon, > > I usually manage to get it to disappear by using `git submodule update > --recursive`. > Is it a flag you've used in your previous attempts? > > Cheers, > > Hécate. > > Le 15/05/2020 à 16:24, Simon Peyton Jones via ghc-devs a écrit : >> >> No amount of git submodule update makes it go away.  Any ideas? >> > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Fri May 15 15:01:38 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Fri, 15 May 2020 15:01:38 +0000 Subject: Strange library git glitch In-Reply-To: <95a09a55-5955-778a-9e27-e2287f634f40@glitchbra.in> References: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> <95a09a55-5955-778a-9e27-e2287f634f40@glitchbra.in> Message-ID: Thanks. It does have a directory dist-install/ in it - but that's put there by the build system, so if I remove it, it'll just come back. And other libraries (like deepseq) has dist-install too but does not complain. So mysterious. On the other hand git submodule update -recursive *did* fix it. Seems odd. I'm already saying "submodules" and I don' think any submodules have further submodules - or do they? Simon From: ghc-devs On Behalf Of Hécate Sent: 15 May 2020 15:37 To: ghc-devs at haskell.org Subject: Re: Strange library git glitch My bad, it seems like this is another issue. I found this StackOverflow page quite helpful in explaining the hows and whys: https://stackoverflow.com/questions/4873980/git-diff-says-subproject-is-dirty Cheers, Hécate Le 15/05/2020 à 16:30, Hécate a écrit : Hi Simon, I usually manage to get it to disappear by using `git submodule update --recursive`. Is it a flag you've used in your previous attempts? Cheers, Hécate. Le 15/05/2020 à 16:24, Simon Peyton Jones via ghc-devs a écrit : No amount of git submodule update makes it go away. Any ideas? _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.pelenitsyn at gmail.com Fri May 15 15:17:47 2020 From: a.pelenitsyn at gmail.com (Artem Pelenitsyn) Date: Fri, 15 May 2020 11:17:47 -0400 Subject: Strange library git glitch In-Reply-To: References: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> <95a09a55-5955-778a-9e27-e2287f634f40@glitchbra.in> Message-ID: Hey Simon, This is nicely explained by Ryan in the message for the latest commit to exceptions: Add dist-boot to .gitignore This is generated by GHC when building `exceptions` now that it has been made a stage-0 boot library (as of commitghc/ghc at 3027241 ). https://github.com/ekmett/exceptions/commit/23c0b8a50d7592af37ca09beeec16b93080df98f Best, Artem On Fri, May 15, 2020, 11:02 AM Simon Peyton Jones via ghc-devs < ghc-devs at haskell.org> wrote: > Thanks. It does have a directory dist-install/ in it – but that’s put > there by the build system, so if I remove it, it’ll just come back. And > other libraries (like deepseq) has dist-install too but does not complain. > So mysterious. > > > > On the other hand git submodule update –recursive **did** fix it. Seems > odd. I’m already saying “submodules” and I don’ think any submodules have > further submodules – or do they? > > > > Simon > > > > *From:* ghc-devs *On Behalf Of *Hécate > *Sent:* 15 May 2020 15:37 > *To:* ghc-devs at haskell.org > *Subject:* Re: Strange library git glitch > > > > My bad, it seems like this is another issue. > > I found this StackOverflow page quite helpful in explaining the hows and > whys: > > https://stackoverflow.com/questions/4873980/git-diff-says-subproject-is-dirty > > Cheers, > > Hécate > > Le 15/05/2020 à 16:30, Hécate a écrit : > > Hi Simon, > > I usually manage to get it to disappear by using `git submodule update > --recursive`. > Is it a flag you've used in your previous attempts? > > Cheers, > > Hécate. > > Le 15/05/2020 à 16:24, Simon Peyton Jones via ghc-devs a écrit : > > No amount of git submodule update makes it go away. Any ideas? > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Fri May 15 15:18:14 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 15 May 2020 16:18:14 +0100 Subject: Strange library git glitch In-Reply-To: References: <394e28b3-b3c2-f052-8411-746546ea8cca@glitchbra.in> <95a09a55-5955-778a-9e27-e2287f634f40@glitchbra.in> Message-ID: Simon, I think the issue was a missing entry in the .gitignore file. Ryan fixed something to with that in this MR (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3222) but he says it was the dist-boot rather than dist-install directory. Matt On Fri, May 15, 2020 at 4:03 PM Simon Peyton Jones via ghc-devs wrote: > > Thanks. It does have a directory dist-install/ in it – but that’s put there by the build system, so if I remove it, it’ll just come back. And other libraries (like deepseq) has dist-install too but does not complain. So mysterious. > > > > On the other hand git submodule update –recursive *did* fix it. Seems odd. I’m already saying “submodules” and I don’ think any submodules have further submodules – or do they? > > > > Simon > > > > From: ghc-devs On Behalf Of Hécate > Sent: 15 May 2020 15:37 > To: ghc-devs at haskell.org > Subject: Re: Strange library git glitch > > > > My bad, it seems like this is another issue. > > I found this StackOverflow page quite helpful in explaining the hows and whys: > https://stackoverflow.com/questions/4873980/git-diff-says-subproject-is-dirty > > Cheers, > > Hécate > > Le 15/05/2020 à 16:30, Hécate a écrit : > > Hi Simon, > > I usually manage to get it to disappear by using `git submodule update --recursive`. > Is it a flag you've used in your previous attempts? > > Cheers, > > Hécate. > > Le 15/05/2020 à 16:24, Simon Peyton Jones via ghc-devs a écrit : > > No amount of git submodule update makes it go away. Any ideas? > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From juhpetersen at gmail.com Sun May 17 02:18:27 2020 From: juhpetersen at gmail.com (Jens Petersen) Date: Sun, 17 May 2020 10:18:27 +0800 Subject: [ANNOUNCE] Glasgow Haskell Compiler 8.10.1 released In-Reply-To: <87zhc5aimf.fsf@smart-cactus.org> References: <87369xbxuo.fsf@smart-cactus.org> <87zhc5aimf.fsf@smart-cactus.org> Message-ID: Hi, late question, I just noticed: Is it intentional that the text library in 8.10.1 is older than the one in 8.8? (1.2.3.2 vs 1.2.4.0 ) Thanks, Jens -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshmeredith2008 at gmail.com Sun May 17 16:54:39 2020 From: joshmeredith2008 at gmail.com (Josh Meredith) Date: Mon, 18 May 2020 02:54:39 +1000 Subject: Iface loading type reflection module bindings Message-ID: Hi, In attempting to implement an extensible interface field for Core bindings based on my previous interfaces patch, I've run into problems with deserialising the special type reflection `$trModule` bindings generated by `GHC.Tc.Instance.Typeable.mkTypeableBinds`. These bindings are then stored in the `ModGuts.mg_binds` along with the real exported top-level bindings of the module. Specifically, attempting to load the iface right-hand side expressions with `tcIfaceExpr` results in the error "Iface id out of scope" from `GHC.Iface.Env.tcIfaceLclId`. My understanding is that this may be because the binding is attempting to look itself up within the interface loading environment, without being bound yet - though it's unclear to me whether this is the correct behaviour for these special type reflection bindings, or if there's some special treatment that should be instead applied to load these. Any advice on how I should proceed would be greatly appreciated. Cheers, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewtpickering at gmail.com Sun May 17 17:04:54 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Sun, 17 May 2020 18:04:54 +0100 Subject: Iface loading type reflection module bindings In-Reply-To: References: Message-ID: So we are clear here you are trying to store additional core bindings other than the ones stored normally in the interface files? It sounds like to me you might need to use `forkM` to make your custom loading lazier. On Sun, May 17, 2020 at 5:55 PM Josh Meredith wrote: > > Hi, > > In attempting to implement an extensible interface field for Core bindings based > on my previous interfaces patch, I've run into problems with deserialising the > special type reflection `$trModule` bindings generated by `GHC.Tc.Instance.Typeable.mkTypeableBinds`. > These bindings are then stored in the `ModGuts.mg_binds` along with the real exported > top-level bindings of the module. > > Specifically, attempting to load the iface right-hand side expressions with > `tcIfaceExpr` results in the error "Iface id out of scope" from `GHC.Iface.Env.tcIfaceLclId`. > My understanding is that this may be because the binding is attempting to look > itself up within the interface loading environment, without being bound yet - > though it's unclear to me whether this is the correct behaviour for these special > type reflection bindings, or if there's some special treatment that should be > instead applied to load these. > > Any advice on how I should proceed would be greatly appreciated. > > Cheers, > Josh > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From joshmeredith2008 at gmail.com Sun May 17 17:12:28 2020 From: joshmeredith2008 at gmail.com (Josh Meredith) Date: Mon, 18 May 2020 03:12:28 +1000 Subject: Iface loading type reflection module bindings In-Reply-To: References: Message-ID: Yes, based on Michael's reasoning in https://gitlab.haskell.org/ghc/ghc/-/wikis/Core-interface-section, I want to round-trip the entire ModGuts, and this part of the mg_binds is giving me trouble. On Mon, 18 May 2020 at 03:05, Matthew Pickering wrote: > So we are clear here you are trying to store additional core bindings > other than the ones stored normally in the interface files? > > It sounds like to me you might need to use `forkM` to make your custom > loading lazier. > > On Sun, May 17, 2020 at 5:55 PM Josh Meredith > wrote: > > > > Hi, > > > > In attempting to implement an extensible interface field for Core > bindings based > > on my previous interfaces patch, I've run into problems with > deserialising the > > special type reflection `$trModule` bindings generated by > `GHC.Tc.Instance.Typeable.mkTypeableBinds`. > > These bindings are then stored in the `ModGuts.mg_binds` along with the > real exported > > top-level bindings of the module. > > > > Specifically, attempting to load the iface right-hand side expressions > with > > `tcIfaceExpr` results in the error "Iface id out of scope" from > `GHC.Iface.Env.tcIfaceLclId`. > > My understanding is that this may be because the binding is attempting > to look > > itself up within the interface loading environment, without being bound > yet - > > though it's unclear to me whether this is the correct behaviour for > these special > > type reflection bindings, or if there's some special treatment that > should be > > instead applied to load these. > > > > Any advice on how I should proceed would be greatly appreciated. > > > > Cheers, > > Josh > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From juhpetersen at gmail.com Mon May 18 05:13:00 2020 From: juhpetersen at gmail.com (Jens Petersen) Date: Mon, 18 May 2020 13:13:00 +0800 Subject: [ANNOUNCE] Glasgow Haskell Compiler 8.10.1 released In-Reply-To: References: <87369xbxuo.fsf@smart-cactus.org> <87zhc5aimf.fsf@smart-cactus.org> Message-ID: On Sun, 17 May 2020 at 10:18, Jens Petersen wrote: > Is it intentional that the text library in 8.10.1 is older than the one in > 8.8? > Ben pointed me at https://gitlab.haskell.org/ghc/ghc/issues/17956 in irc. Thanks, Jens -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Tue May 19 10:42:44 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Tue, 19 May 2020 11:42:44 +0100 Subject: GitLab is down ? Message-ID: Hi devs, gitlab.haskell.org is reporting an error 500. :( Help? Thanks! Richard From rae at richarde.dev Tue May 19 10:55:29 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Tue, 19 May 2020 11:55:29 +0100 Subject: GitLab is down ? In-Reply-To: References: Message-ID: and.... it's back! Yay! Richard > On May 19, 2020, at 11:42 AM, Richard Eisenberg wrote: > > Hi devs, > > gitlab.haskell.org is reporting an error 500. :( > > Help? > > Thanks! > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Tue May 19 10:56:21 2020 From: ben at well-typed.com (Ben Gamari) Date: Tue, 19 May 2020 06:56:21 -0400 Subject: Brief GitLab reboot Message-ID: <87ftbwxkfm.fsf@smart-cactus.org> Hi all, GitLab will be down for about 10 minutes for a quick reboot. Apologies for the inconvenience. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From dyaitskov at gmail.com Wed May 20 17:48:41 2020 From: dyaitskov at gmail.com (Daneel Yaitskov) Date: Wed, 20 May 2020 10:48:41 -0700 Subject: build ghc with make (ghc-boot-th broken deps) Message-ID: Hi List, I pulled ghc repo (568d7279a) and trying to build with following env: Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin ./boot is passing well, but ./configure breaks on ghc-boot step. I have no idea how could I fix this issue. I manually modified dep versions in to what I have for ghc-8.8.3 and issue with ghc-boot-th disappeared and appeared parsec libraries/bootstrapping.conf/binary-0.8.7.0.conf depends: array-0.5.4.0 base-4.13.0.0 bytestring-0.10.10.0 containers-0.6.2.1 but there was depends: array-0.5.3.0 base-4.12.0.0 bytestring-0.10.8.2 containers-0.6.0.1 Where is actual problem? Old cabal? 10:45$ cabal --version cabal-install version 1.24.0.2 compiled using version 1.24.2.0 of the Cabal library Configuring ghc-boot-th-8.11.0.20200520... "/usr/bin/ghc-pkg" update -v0 --force --package-db=libraries/bootstrapping.conf libraries/ghc-boot-th/dist-boot/inplace-pkg-config "inplace/bin/ghc-cabal" configure libraries/ghc-boot dist-boot --with-ghc="/usr/bin/ghc" --with-ghc-pkg="/usr/bin/ghc-pkg" --package-db=/home/dan/demo/haskell/compiler/ghc/libraries/bootstrapping.conf --disable-library-for-ghci --enable-library-vanilla --enable-library-for-ghci --disable-library-profiling --disable-shared --with-hscolour="/home/dan/.cabal/bin/HsColour" --configure-option=CFLAGS="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot" --configure-option=LDFLAGS=" " --configure-option=CPPFLAGS=" " --gcc-options="-Wall -Werror=unused-but-set-variable -Wno-error=inline -iquote /home/dan/demo/haskell/compiler/ghc/libraries/ghc-boot " --constraint "binary == 0.8.7.0" --constraint "transformers == 0.5.6.2" --constraint "mtl == 2.2.2" --constraint "hpc == 0.6.1.0" --constraint "ghc-boot-th == 8.11.0.20200520" --constraint "ghc-boot == 8.11.0.20200520" --constraint "template-haskell == 2.17.0.0" --constraint "text == 1.2.4.0" --constraint "parsec == 3.1.14.0" --constraint "Cabal == 3.3.0.0" --constraint "ghc-heap == 8.11.0.20200520" --constraint "exceptions == 0.10.4" --constraint "ghci == 8.11.0.20200520" --constraint "terminfo == 0.4.1.4" --with-gcc="gcc" --with-ld="ld.gold" --with-ar="ar" --with-alex="/home/dan/.cabal/bin/alex" --with-happy="/home/dan/.cabal/bin/happy" Configuring ghc-boot-8.11.0.20200520... Error: The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used. installed package binary-0.8.7.0 is broken due to missing package array-0.5.3.0, base-4.12.0.0, bytestring-0.10.8.2, containers-0.6.0.1 libraries/ghc-boot/ghc.mk:3: recipe for target 'libraries/ghc-boot/dist-boot/package-data.mk' failed make[1]: *** [libraries/ghc-boot/dist-boot/package-data.mk] Error 1 Makefile:123: recipe for target 'all' failed make: *** [all] Error 2 -- Best regards, Daniil Iaitskov From ben at smart-cactus.org Thu May 21 00:32:46 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 20 May 2020 20:32:46 -0400 Subject: build ghc with make (ghc-boot-th broken deps) In-Reply-To: References: Message-ID: <87367uxh3r.fsf@smart-cactus.org> Daneel Yaitskov writes: > Hi List, > > I pulled ghc repo (568d7279a) and trying to build with following env: > Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin > Hi Daneel, > ./boot is passing well, but ./configure breaks on ghc-boot step. > I have no idea how could I fix this issue. > Can you offer more detail here? How precisely does it break? Can you paste the full output from ./configure? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From dyaitskov at gmail.com Thu May 21 02:01:36 2020 From: dyaitskov at gmail.com (Daneel Yaitskov) Date: Wed, 20 May 2020 19:01:36 -0700 Subject: build ghc with make (ghc-boot-th broken deps) In-Reply-To: <87367uxh3r.fsf@smart-cactus.org> References: <87367uxh3r.fsf@smart-cactus.org> Message-ID: Hi Ben, Thanks for bothering. I solved issue. That was ancient cabal. I build ghc successfully once I installed Cabal-3. I thought that configure script will warn me about incompatible cabal :) Btw could you shed some light on RTS locking? I am interested in: ccall traceUserBinaryMsg(MyCapability() "ptr", msg "ptr", len); because it looks slower on smaller chunks. Writing data in 64 bytes chunks vs 4k (ghc 8.11) is twice slower. https://github.com/yaitskov/eventslog-benchmark benchmark [ LoadWithBinary-b1048576-t001-c0064] lasted for cpu 4.23 ms real 4.24 ms benchmark [ LoadWithBinary-b1048576-t001-c0128] lasted for cpu 3.97 ms real 3.91 ms benchmark [ LoadWithBinary-b1048576-t001-c1024] lasted for cpu 2.73 ms real 2.61 ms benchmark [ LoadWithBinary-b1048576-t001-c2048] lasted for cpu 2.34 ms real 2.23 ms benchmark [ LoadWithBinary-b1048576-t001-c4096] lasted for cpu 2.16 ms real 2.20 ms Eventlog messages tend to be small. So application developer has to think about bufferization to speed up. OpenTelemetry gains 6x speed improvement with following hack. mm :: IO (IORef (Int, Builder)) mm = newIORef (0, mempty) traceBuilder :: MonadIO m => (Int, Builder) -> m () traceBuilder !sizedBuilder@(size, builder) = do liftIO $! do tls <- G.mkTLS mm ref <- G.getTLS tls (!bufferedSize, !bufferedBuilders) <- readIORef ref if bufferedSize + size > 1024 then do traceBinaryEventIO . LBS.toStrict $! toLazyByteString bufferedBuilders writeIORef ref sizedBuilder else do writeIORef ref (bufferedSize + size, builder <> bufferedBuilders) I see that some EventLog methods use mutexes, but user message function don't. It just appends to buffer and can flush if it is full. So I don't see where locking happening. MyCapability is just arithmetic #define MyCapability() (BaseReg - OFFSET_Capability_r) I guess capability locking happens somewhere upper - before C. How to register RTS function, which would don't acquire capability lock? I think locking for this case could be replaced with atomic variable. C11 standard provides such IPC abstraction. Thanks, Daniil On 5/20/20, Ben Gamari wrote: > Daneel Yaitskov writes: > >> Hi List, >> >> I pulled ghc repo (568d7279a) and trying to build with following env: >> Ghc from stack (8.8.3). I made links to ghc and ghc-pkg in /usr/bin >> > Hi Daneel, > >> ./boot is passing well, but ./configure breaks on ghc-boot step. >> I have no idea how could I fix this issue. >> > Can you offer more detail here? How precisely does it break? Can you > paste the full output from ./configure? > > Cheers, > > - Ben > > -- Best regards, Daniil Iaitskov From ben at smart-cactus.org Thu May 21 03:31:00 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Wed, 20 May 2020 23:31:00 -0400 Subject: build ghc with make (ghc-boot-th broken deps) In-Reply-To: References: <87367uxh3r.fsf@smart-cactus.org> Message-ID: <87zha2vuat.fsf@smart-cactus.org> Daneel Yaitskov writes: > Hi Ben, > > Thanks for bothering. > I solved issue. > That was ancient cabal. > I build ghc successfully once I installed Cabal-3. > I thought that configure script will warn me about incompatible cabal :) > Hmm, interesting. I'm actually rather surprised that the problem was cabal-install. Afterall, GHC's build system doesn't use cabal-install apart from building Hadrian. > Btw could you shed some light on RTS locking? > > I am interested in: > > ccall traceUserBinaryMsg(MyCapability() "ptr", msg "ptr", len); > > because it looks slower on smaller chunks. > Writing data in 64 bytes chunks vs 4k (ghc 8.11) is twice slower. > > https://github.com/yaitskov/eventslog-benchmark > benchmark [ LoadWithBinary-b1048576-t001-c0064] lasted for cpu > 4.23 ms real 4.24 ms > benchmark [ LoadWithBinary-b1048576-t001-c0128] lasted for cpu > 3.97 ms real 3.91 ms > benchmark [ LoadWithBinary-b1048576-t001-c1024] lasted for cpu > 2.73 ms real 2.61 ms > benchmark [ LoadWithBinary-b1048576-t001-c2048] lasted for cpu > 2.34 ms real 2.23 ms > benchmark [ LoadWithBinary-b1048576-t001-c4096] lasted for cpu > 2.16 ms real 2.20 ms > > Eventlog messages tend to be small. > > So application developer has to think about bufferization to speed up. > OpenTelemetry gains 6x speed improvement with following hack. > ... Interesting. Please do open a GitLab ticket to track this. You might also be interested in #17949, which is covers the case of traceEvent being expensive when tracing is disabled. > I see that some EventLog methods use mutexes, but user message function don't. > It just appends to buffer and can flush if it is full. > So I don't see where locking happening. > MyCapability is just arithmetic > > #define MyCapability() (BaseReg - OFFSET_Capability_r) > > I guess capability locking happens somewhere upper - before C. > There is no locking necessary. Capabilities are owned by a particular operating system thread. The capability is held until the program yields it (which generally only happens during a GC or foreign function call) and during this time the program has full access to all of the state in the Capability structure without the need for a lock. This is why each Capability has its own eventlog buffer; it allows lock-free tracing. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From facundo.dominguez at tweag.io Thu May 21 19:05:48 2020 From: facundo.dominguez at tweag.io (=?UTF-8?Q?Dom=C3=ADnguez=2C_Facundo?=) Date: Thu, 21 May 2020 16:05:48 -0300 Subject: 8.12 plans In-Reply-To: <87ftce2qui.fsf@smart-cactus.org> References: <87ftce2qui.fsf@smart-cactus.org> Message-ID: Hello Ben, Matthías Páll and I are working on an implementation of QualifiedDo [1] with the aim to get it merged on time. We are trying to get a small pull request submitted in the first week of June. Cheers, Facundo [1] https://github.com/ghc-proposals/ghc-proposals/pull/216 On Tue, May 5, 2020 at 3:13 PM Ben Gamari wrote: > Hi everyone, > > The time is again upon us to start thinking about release planning for > the next major release: GHC 8.12. In keeping with our 6-month release > schedule, I propose the following schedule: > > * Mid-June 2020: Aim to have all major features in the tree > * Late-June 2020: Cut the ghc-8.12 branch > * June - August 2020: 3 alpha releases > * 1 September 2020: beta release > * 25 September 2020: Final 8.12.1 release > > So, if you have any major features which you would like to merge for > 8.12, now is the time to start planning how to wrap them up in the next > month or so. As always, do let me know if you think this may be > problematic and we can discuss options. > > Cheers, > > - Ben > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Thu May 21 19:48:19 2020 From: ben at well-typed.com (Ben Gamari) Date: Thu, 21 May 2020 15:48:19 -0400 Subject: 8.12 plans In-Reply-To: References: Message-ID: <87wo55vzlu.fsf@smart-cactus.org> Domínguez, Facundo writes: > Hello Ben, > > Matthías Páll and I are working on an implementation of QualifiedDo [1] > with the aim to get it merged on time. We are trying to get a small pull > request submitted in the first week of June. > Thanks for the heads-up, Facundo! I've added this to the milestone. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From matthewtpickering at gmail.com Fri May 22 08:40:10 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 22 May 2020 09:40:10 +0100 Subject: 8.12 plans In-Reply-To: <87wo55vzlu.fsf@smart-cactus.org> References: <87wo55vzlu.fsf@smart-cactus.org> Message-ID: Patches I would like merged: * Short ByteString Patch - seems stalled again * My patches to allow compacting a ModIface - Need to put these up for review * Make Q (TExp a) a newtype (Proposal 195 -https://github.com/ghc-proposals/ghc-proposals/pull/195) - Not yet written but will not be a big effort Cheers, Matt On Thu, May 21, 2020 at 8:48 PM Ben Gamari wrote: > > Domínguez, Facundo writes: > > > Hello Ben, > > > > Matthías Páll and I are working on an implementation of QualifiedDo [1] > > with the aim to get it merged on time. We are trying to get a small pull > > request submitted in the first week of June. > > > Thanks for the heads-up, Facundo! I've added this to the milestone. > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From matthewtpickering at gmail.com Fri May 22 11:31:36 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 22 May 2020 12:31:36 +0100 Subject: How to use the compact regions API to write a compact to a file? Message-ID: Dear devs, I have been under the impression for a while that you can write a compact region to a file and the functions in GHC.Compact.Serialized also seem to suggest this is possible. https://hackage.haskell.org/package/ghc-compact-0.1.0.0/docs/GHC-Compact-Serialized.html It seems like there are some functions missing from the API. There is "importCompactByteStrings" but no function in the API which produces ByteStrings. So in order to use this function you have to convert a SerializedCompact into a ByteString, but in what specific way? Has anyone got any code where they have tried to do this? Cheers, Matt From cheng.shao at tweag.io Fri May 22 11:51:10 2020 From: cheng.shao at tweag.io (Shao, Cheng) Date: Fri, 22 May 2020 13:51:10 +0200 Subject: How to use the compact regions API to write a compact to a file? In-Reply-To: References: Message-ID: Hi Matthew, It's possible to use Data.Compact.Serialize to write a compact to a file or read it back. Directly serializing via ByteStrings has also been discussed before, see link below. Hope this helps! https://hackage.haskell.org/package/compact https://github.com/ezyang/compact/issues/3 On Fri, May 22, 2020 at 1:32 PM Matthew Pickering wrote: > > Dear devs, > > I have been under the impression for a while that you can write a > compact region to a file and the functions in GHC.Compact.Serialized > also seem to suggest this is possible. > > https://hackage.haskell.org/package/ghc-compact-0.1.0.0/docs/GHC-Compact-Serialized.html > > It seems like there are some functions missing from the API. There is > "importCompactByteStrings" > but no function in the API which produces ByteStrings. So in order to > use this function you have to convert a SerializedCompact into a > ByteString, but in what specific way? > > Has anyone got any code where they have tried to do this? > > Cheers, > > Matt > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From matthewtpickering at gmail.com Fri May 22 11:56:31 2020 From: matthewtpickering at gmail.com (Matthew Pickering) Date: Fri, 22 May 2020 12:56:31 +0100 Subject: How to use the compact regions API to write a compact to a file? In-Reply-To: References: Message-ID: Ah thanks, those links are useful. I was looking in `ghc-compact` not in the `compact` library. Cheers, Matt On Fri, May 22, 2020 at 12:51 PM Shao, Cheng wrote: > > Hi Matthew, > > It's possible to use Data.Compact.Serialize to write a compact to a > file or read it back. Directly serializing via ByteStrings has also > been discussed before, see link below. Hope this helps! > > https://hackage.haskell.org/package/compact > https://github.com/ezyang/compact/issues/3 > > On Fri, May 22, 2020 at 1:32 PM Matthew Pickering > wrote: > > > > Dear devs, > > > > I have been under the impression for a while that you can write a > > compact region to a file and the functions in GHC.Compact.Serialized > > also seem to suggest this is possible. > > > > https://hackage.haskell.org/package/ghc-compact-0.1.0.0/docs/GHC-Compact-Serialized.html > > > > It seems like there are some functions missing from the API. There is > > "importCompactByteStrings" > > but no function in the API which produces ByteStrings. So in order to > > use this function you have to convert a SerializedCompact into a > > ByteString, but in what specific way? > > > > Has anyone got any code where they have tried to do this? > > > > Cheers, > > > > Matt > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Fri May 22 14:16:42 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 22 May 2020 10:16:42 -0400 Subject: 8.12 plans In-Reply-To: References: <87wo55vzlu.fsf@smart-cactus.org> Message-ID: <87tv08vyv0.fsf@smart-cactus.org> Matthew Pickering writes: > Patches I would like merged: > > * Short ByteString Patch - seems stalled again Indeed I think dxld was going to investigate the unregisterised regression but frankly I'm happy enough just to merge as-is. I strongly suspect that the issue is just a change in GC timing. > * My patches to allow compacting a ModIface - Need to put these up for review > * Make Q (TExp a) a newtype (Proposal 195 > -https://github.com/ghc-proposals/ghc-proposals/pull/195) - Not yet > written but will not be a big effort > Thanks Matt! Added to the status page [1]. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/status/ghc-8.12.1 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From moritz.angermann at gmail.com Fri May 22 14:21:32 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Fri, 22 May 2020 22:21:32 +0800 Subject: 8.12 plans In-Reply-To: <87ftce2qui.fsf@smart-cactus.org> References: <87ftce2qui.fsf@smart-cactus.org> Message-ID: I have a few aarch64 linker patches I'll need to open MRs for. They are mostly 5-10 line changes, but will give us a working linker :-) On Wed, May 6, 2020 at 2:13 AM Ben Gamari wrote: > > Hi everyone, > > The time is again upon us to start thinking about release planning for > the next major release: GHC 8.12. In keeping with our 6-month release > schedule, I propose the following schedule: > > * Mid-June 2020: Aim to have all major features in the tree > * Late-June 2020: Cut the ghc-8.12 branch > * June - August 2020: 3 alpha releases > * 1 September 2020: beta release > * 25 September 2020: Final 8.12.1 release > > So, if you have any major features which you would like to merge for > 8.12, now is the time to start planning how to wrap them up in the next > month or so. As always, do let me know if you think this may be > problematic and we can discuss options. > > Cheers, > > - Ben > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From sylvain at haskus.fr Sat May 23 10:49:49 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Sat, 23 May 2020 12:49:49 +0200 Subject: 8.12 plans In-Reply-To: <87ftce2qui.fsf@smart-cactus.org> References: <87ftce2qui.fsf@smart-cactus.org> Message-ID: <554211d6-0aa4-1036-885c-8bc6bba0adc3@haskus.fr> Hi Ben, ghc-bignum (!2231) is ready to be merged for 8.12. We are just waiting for bytestring/text maintainers to merge 2 simple patches. Thanks, Sylvain On 05/05/2020 20:12, Ben Gamari wrote: > Hi everyone, > > The time is again upon us to start thinking about release planning for > the next major release: GHC 8.12. In keeping with our 6-month release > schedule, I propose the following schedule: > > * Mid-June 2020: Aim to have all major features in the tree > * Late-June 2020: Cut the ghc-8.12 branch > * June - August 2020: 3 alpha releases > * 1 September 2020: beta release > * 25 September 2020: Final 8.12.1 release > > So, if you have any major features which you would like to merge for > 8.12, now is the time to start planning how to wrap them up in the next > month or so. As always, do let me know if you think this may be > problematic and we can discuss options. > > Cheers, > > - Ben > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Sat May 23 12:58:30 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Sat, 23 May 2020 13:58:30 +0100 Subject: profiling a constructor? Message-ID: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Hi devs, Is there a way to count the number of times a particular constructor is allocated? I might want to know, say, the total number of cons cells allocated over the course of a program, or (in my actual case) the total number of FunTy cells allocated over the course of a program. I can try to do this with an SCC. But it's very clunky: * The constructor may be used in a number of places; each such place would need the SCC. * SCCs can interfere with optimizations. In my use case, this would negate the usefulness of the exercise entirely, as I think some of the structures I wish to observe should never come into being, due to optimizations (e.g. case-of-known-constructor after inlining). Surely, the RTS must be able to count the number of times a constructor is used. But is there any way to access such a feature? If others agree that this would sometimes be helpful, perhaps we can build the feature. Now is not the first time I've wanted this. Thanks! Richard From ben at well-typed.com Sat May 23 13:21:20 2020 From: ben at well-typed.com (Ben Gamari) Date: Sat, 23 May 2020 09:21:20 -0400 Subject: profiling a constructor? In-Reply-To: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Message-ID: Yes, you can use +RTS -hT, which is available even in the non-profiled runtime. This breaks down the heap by closure type. Of course, you don't have visibility into the source of the allocation, but this is often still quite useful. — Ben On May 23, 2020 8:58:30 AM EDT, Richard Eisenberg wrote: >Hi devs, > >Is there a way to count the number of times a particular constructor is >allocated? I might want to know, say, the total number of cons cells >allocated over the course of a program, or (in my actual case) the >total number of FunTy cells allocated over the course of a program. > >I can try to do this with an SCC. But it's very clunky: >* The constructor may be used in a number of places; each such place >would need the SCC. >* SCCs can interfere with optimizations. In my use case, this would >negate the usefulness of the exercise entirely, as I think some of the >structures I wish to observe should never come into being, due to >optimizations (e.g. case-of-known-constructor after inlining). > >Surely, the RTS must be able to count the number of times a constructor >is used. But is there any way to access such a feature? If others agree >that this would sometimes be helpful, perhaps we can build the feature. >Now is not the first time I've wanted this. > >Thanks! >Richard >_______________________________________________ >ghc-devs mailing list >ghc-devs at haskell.org >http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klebinger.andreas at gmx.at Sat May 23 13:32:32 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Sat, 23 May 2020 15:32:32 +0200 Subject: profiling a constructor? In-Reply-To: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Message-ID: <1d7632ab-9c75-2a39-40d2-5ef2367ecd8d@gmx.at> > Surely, the RTS must be able to count the number of times a constructor is used. It can't. For one there are different kinds of "uses" for constructors. * Allocations - They might be dead by the time we gc the nursery, so the RTS never get's to see them. * Accessing the Constructor? That's even harder to track. * The constructor being present during GC? One can do this using heap profiling (as ben described). There are also top level constructors which currently don't generate code at all (just static data). So currently there is no such feature. For allocations in particular we could implement one on top of the ticky profiler. It operates on the STG => Cmm boundry so doesn't affect core optimizations. There we could for every runtime constructor allocation emit code which will bump a counter for that specific constructor. I imagine this wouldn't that hard either. But it does require modification of the compiler. Cheers, Andreas Richard Eisenberg schrieb am 23.05.2020 um 14:58: > Hi devs, > > Is there a way to count the number of times a particular constructor is allocated? I might want to know, say, the total number of cons cells allocated over the course of a program, or (in my actual case) the total number of FunTy cells allocated over the course of a program. > > I can try to do this with an SCC. But it's very clunky: > * The constructor may be used in a number of places; each such place would need the SCC. > * SCCs can interfere with optimizations. In my use case, this would negate the usefulness of the exercise entirely, as I think some of the structures I wish to observe should never come into being, due to optimizations (e.g. case-of-known-constructor after inlining). > > Surely, the RTS must be able to count the number of times a constructor is used. But is there any way to access such a feature? If others agree that this would sometimes be helpful, perhaps we can build the feature. Now is not the first time I've wanted this. > > Thanks! > Richard > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Sat May 23 13:50:20 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sat, 23 May 2020 09:50:20 -0400 Subject: profiling a constructor? In-Reply-To: <1d7632ab-9c75-2a39-40d2-5ef2367ecd8d@gmx.at> References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> <1d7632ab-9c75-2a39-40d2-5ef2367ecd8d@gmx.at> Message-ID: <46A0C321-7FC3-4325-9148-DC1E5CAF54C1@smart-cactus.org> On May 23, 2020 9:32:32 AM EDT, Andreas Klebinger wrote: >> Surely, the RTS must be able to count the number of times a >constructor is used. >It can't. For one there are different kinds of "uses" for constructors. This is a fair point. I had assumed that Richard was debugging a residency issue but i suppose that was an ungrounded assumption on my part. >* Allocations - They might be dead by the time we gc the nursery, so >the >RTS never get's to see them. Indeed I thought that the ticky-ticky profiler could do precisely this but it seems I was mistaken; it can only report thunks. Indeed it doesn't seem like it would be hard to introduce this. >* Accessing the Constructor? That's even harder to track. Indeed, this is not possible as written. However, the LDV profiler allows a slightly weaker version on this (reporting whether the closure as been read at all). Allowing full tracking of closure reads this would incur a significant overhead. From simonpj at microsoft.com Sat May 23 21:11:29 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Sat, 23 May 2020 21:11:29 +0000 Subject: profiling a constructor? In-Reply-To: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Message-ID: This would be a sensible thing to be able to count with -ticky. Not hard to add either. (Richard is asking for number of allocations, not what is live at any moment.) Simon | -----Original Message----- | From: ghc-devs On Behalf Of Richard | Eisenberg | Sent: 23 May 2020 13:59 | To: GHC developers | Subject: profiling a constructor? | | Hi devs, | | Is there a way to count the number of times a particular constructor is | allocated? I might want to know, say, the total number of cons cells | allocated over the course of a program, or (in my actual case) the total | number of FunTy cells allocated over the course of a program. | | I can try to do this with an SCC. But it's very clunky: | * The constructor may be used in a number of places; each such place | would need the SCC. | * SCCs can interfere with optimizations. In my use case, this would | negate the usefulness of the exercise entirely, as I think some of the | structures I wish to observe should never come into being, due to | optimizations (e.g. case-of-known-constructor after inlining). | | Surely, the RTS must be able to count the number of times a constructor | is used. But is there any way to access such a feature? If others agree | that this would sometimes be helpful, perhaps we can build the feature. | Now is not the first time I've wanted this. | | Thanks! | Richard | _______________________________________________ | ghc-devs mailing list | ghc-devs at haskell.org | http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From rae at richarde.dev Sat May 23 21:17:02 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Sat, 23 May 2020 22:17:02 +0100 Subject: profiling a constructor? In-Reply-To: References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Message-ID: Thanks for all the responses. Yes, I'm looking for the number of allocations of a constructor; this is an allocations issue, not a retention/liveness issue. I'm not concerned about accesses (not even sure what that would mean). Adding this to -ticky would be very helpful -- and not just for me. For example, it would help us to know more precisely why !2249 (implementing BoxedRep) is mired in performance trouble: we could see how many more TyConApp nodes are created. If this would be easy, is there someone who can guide me how to implement it? I am almost as clueless as someone who has never before gazed on GHC's code in these areas. (Or, I would be grateful for someone else to implement it, too.) Thanks, Richard > On May 23, 2020, at 10:11 PM, Simon Peyton Jones wrote: > > This would be a sensible thing to be able to count with -ticky. Not hard to add either. > > (Richard is asking for number of allocations, not what is live at any moment.) > > Simon > > | -----Original Message----- > | From: ghc-devs On Behalf Of Richard > | Eisenberg > | Sent: 23 May 2020 13:59 > | To: GHC developers > | Subject: profiling a constructor? > | > | Hi devs, > | > | Is there a way to count the number of times a particular constructor is > | allocated? I might want to know, say, the total number of cons cells > | allocated over the course of a program, or (in my actual case) the total > | number of FunTy cells allocated over the course of a program. > | > | I can try to do this with an SCC. But it's very clunky: > | * The constructor may be used in a number of places; each such place > | would need the SCC. > | * SCCs can interfere with optimizations. In my use case, this would > | negate the usefulness of the exercise entirely, as I think some of the > | structures I wish to observe should never come into being, due to > | optimizations (e.g. case-of-known-constructor after inlining). > | > | Surely, the RTS must be able to count the number of times a constructor > | is used. But is there any way to access such a feature? If others agree > | that this would sometimes be helpful, perhaps we can build the feature. > | Now is not the first time I've wanted this. > | > | Thanks! > | Richard > | _______________________________________________ > | ghc-devs mailing list > | ghc-devs at haskell.org > | http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at smart-cactus.org Sun May 24 04:24:40 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sun, 24 May 2020 00:24:40 -0400 Subject: profiling a constructor? In-Reply-To: References: <2089CDC1-3CFF-418D-A1FB-A5FF6287B8D1@richarde.dev> Message-ID: <871rnaq7st.fsf@smart-cactus.org> Richard Eisenberg writes: > Thanks for all the responses. Yes, I'm looking for the number of > allocations of a constructor; this is an allocations issue, not a > retention/liveness issue. I'm not concerned about accesses (not even > sure what that would mean). Adding this to -ticky would be very > helpful -- and not just for me. For example, it would help us to know > more precisely why !2249 (implementing BoxedRep) is mired in > performance trouble: we could see how many more TyConApp nodes are > created. > > If this would be easy, is there someone who can guide me how to > implement it? I am almost as clueless as someone who has never before > gazed on GHC's code in these areas. (Or, I would be grateful for > someone else to implement it, too.) > Actually, looking more closely at the implementation of ticky, it looks like we *do* track DataCon allocations (the giveaway was the mention of withNewTickyCounterCon in GHC.StgToCmm.Bind.cgRhs). For instance, let con = Just x in ... would produce a ticky counter named "con". The problem is that there is no convenient way to get from the Id name (e.g. `con`) back to the datacon that it allocates (e.g. `Just`). You could of course try to parse this out of -ddump-stg output but... yuck. Perhaps the ticker name should preserve this information. I've pushed a completely un-build-tested attempt at this as !3340. Like most of ticky's output, it won't be the easiest thing to mechanically parse but it should be possible. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From john.ericson at obsidian.systems Sun May 24 13:10:27 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Sun, 24 May 2020 09:10:27 -0400 Subject: 8.12 plans In-Reply-To: <554211d6-0aa4-1036-885c-8bc6bba0adc3@haskus.fr> References: <87ftce2qui.fsf@smart-cactus.org> <554211d6-0aa4-1036-885c-8bc6bba0adc3@haskus.fr> Message-ID: <9701a5c2-4dad-c863-b8fc-6003f026c152@obsidian.systems> In that case I should perhaps try to get in !1102, which makes all the fixed-sized prim types always available, in too? I'm fine either way, but figure it's nice for any CPP to deal with the fixed and bignum numeric representation changing across a single release, and the schedule gives us time to patch important packages that need it between mid-June and the end of September. Cheers, John On 5/23/20 6:49 AM, Sylvain Henry wrote: > > Hi Ben, > > ghc-bignum (!2231) is ready to be merged for 8.12. We are just waiting > for bytestring/text maintainers to merge 2 simple patches. > > Thanks, > Sylvain > > > On 05/05/2020 20:12, Ben Gamari wrote: >> Hi everyone, >> >> The time is again upon us to start thinking about release planning for >> the next major release: GHC 8.12. In keeping with our 6-month release >> schedule, I propose the following schedule: >> >> * Mid-June 2020: Aim to have all major features in the tree >> * Late-June 2020: Cut the ghc-8.12 branch >> * June - August 2020: 3 alpha releases >> * 1 September 2020: beta release >> * 25 September 2020: Final 8.12.1 release >> >> So, if you have any major features which you would like to merge for >> 8.12, now is the time to start planning how to wrap them up in the next >> month or so. As always, do let me know if you think this may be >> problematic and we can discuss options. >> >> Cheers, >> >> - Ben >> >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Mon May 25 22:33:39 2020 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Mon, 25 May 2020 23:33:39 +0100 Subject: in-tree API Annotations Message-ID: I am trying to get my head around the best way to put the API annotations into the GHC ParsedSource directly, so ghc-exactprint can move in to GHC and make the development/testing process easier. Prior to API Annotations, there were very few locations in the AST, as they were added as anchor points for the annotations, being separate from the AST and indexed to it by the SrcSpan. I am intending to strip out the superfluous ones, in particular all the (Located RdrName), replacing them with something like data ApiAnnName a = N ApiAnn a instead. So they would still be wrapped, but the annotations would be only ones that can occur for a RdrName, such a backticks, parens, and so on. Does anything actually need these locations now they are there? I am thinking specifically of the .hie files, as the `toHie` instances seem to make use of locations extensively. Regards Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From alan.zimm at gmail.com Tue May 26 07:33:59 2020 From: alan.zimm at gmail.com (Alan & Kim Zimmerman) Date: Tue, 26 May 2020 08:33:59 +0100 Subject: in-tree API Annotations In-Reply-To: References: Message-ID: And to reply to my own question, yes we do, we use them to tie up the ParsedSource RdrName to the renamed Name when considering refactorings. On Mon, 25 May 2020 at 23:33, Alan & Kim Zimmerman wrote: > I am trying to get my head around the best way to put the API annotations > into the GHC ParsedSource directly, so ghc-exactprint can move in to GHC > and make the development/testing process easier. > > Prior to API Annotations, there were very few locations in the AST, as > they were added as anchor points for the annotations, being separate from > the AST and indexed to it by the SrcSpan. > > I am intending to strip out the superfluous ones, in particular all the > (Located RdrName), replacing them with something like > > data ApiAnnName a = N ApiAnn a > > instead. So they would still be wrapped, but the annotations would be only > ones that can occur for a RdrName, such a backticks, parens, and so on. > > Does anything actually need these locations now they are there? I am > thinking specifically of the .hie files, as the `toHie` instances seem to > make use of locations extensively. > > Regards > Alan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at haskus.fr Tue May 26 17:04:02 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Tue, 26 May 2020 19:04:02 +0200 Subject: keepAlive# primop In-Reply-To: <875ze39suw.fsf@smart-cactus.org> References: <04361b3f-4323-7610-1bc1-9a6a90abec2b@haskus.fr> <875ze39suw.fsf@smart-cactus.org> Message-ID: <45e2c2f1-b6c2-9d9b-5cf0-6f5e3317c4f5@haskus.fr> Hi ghc-devs, After a discussion today about `keepAlive#`, I think Option E [1] is even more appealing. To recap the idea was to have keep-alive variable sets attached to case-expressions in Core. E.g. `case {k} x of ...` 1. One of the issue was the semantics of `keepAlive#`. `keepAlive#` is very intricate with evaluation. As Simon mentioned we want a semantics like: keepAlive# k x ==> x `seq` touch# k `seq` x with potentially a special `seq` to deal with diverging `x`. With `case {k} x of ...` we have this semantics. Even if we throw the case alternatives if `x` diverges, we don't throw the keep-alive set so we're good. 2. Simon wanted to push `keepAlive#` into case-expressions. With this approach we should only have to fix case-of-case to take keep-alive sets into account. case {k} (case {k2} x of .. -> a; ... -> b) of C0 .. -> e1 C1 .. -> e2 ===> case {k,k2} x of ... -> case {k} a of { C0 .. -> e1; C1 ... -> e2 } ... -> case {k} b of { C0 .. -> e1; C1 ... -> e2 } 3. Compared to other approaches: we don't have to use stack frames (good for performance) and we don't have to deal with a continuation (good for Core optimizations, hence perf). 4. Implementing this approach is quite straightforward even if it modifies Core. I did it last month in [2]. This patch doesn't fully work yet with `-O` because some transformation (related to join points IIRC) doesn't take keep-alive sets into account but it should be pretty easy to fix if we want to use this approach. Given how hard it is to come up with a good design/implementation of other approaches, this one strikes me as probably the more principled we have and yet it is relatively easy to implement. What do you think? Cheers, Sylvain [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables [2] https://gitlab.haskell.org/hsyl20/ghc/-/commits/hsyl20-keepalive On 13/04/2020 19:51, Ben Gamari wrote: > Ccing ghc-devs@ since this discussion is something of general interest > to the community. > > > Sylvain Henry writes: > >> Simon, Ben, >> >> I've been reading and thinking about `readRW#` issues which are very >> related to issues we have with `keepAlive#` primop. >> >> To recap, the problem is that we want some transformations (that Simon >> has listed in [1]) to consider: >> >> ``` >> case runRW# f of ... >> >> case keepAlive# k a of ... >> ``` >> >> as if they were really: >> >> ``` >> case f realWorld# of ... >> >> case a of ... >> ``` >> >> BUT without breaking the semantics of runRW# and keepAlive#. >> >> I have been thinking about a solution that I have described on the wiki: >> https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables >> >> The idea is to keep a set of variable names in each Core case-expression >> that are kept alive during the evaluation of the scrutinee. >> >> I think it would work very nicely with your `newState#` primop described >> in [2], both for `runST` and for `unsafeDupablePerformIO` (details on >> the wiki). >> >> It requires a little more upfront work to adapt the code involving >> case-expressions. But it will force us to review all transformations to >> check if they are sound when keep-alive sets are not empty, which we >> would have to do anyway if we implemented another option. We could start >> by disabling transformations involving non-empty keep-alive sets and >> iterate to enable the sound ones. >> >> I would like your opinions on the approach. I may have totally missed >> something. > Thanks for writing this down! > > Indeed it is an interesting idea. However, as expressed on IRC, I > wonder whether this problem rises to the level where it warrants an > adaptation to our Core representation. It feels a bit like the > tail is wagging the dog here, especially given how the "tail" here > merely exists to support FFI. > > That being said, this is one of the few options which remain on the > table that doesn't require changes to user code. Moreover, the > applicability to runRW# is quite intriguing. > > Another (admittedly, more ad-hoc) option that would avoid modifying Core > would be to teach the simplifier about the class of > "continuation-passing" primops (e.g. `keepAlive#` and `runRW#`), allowing it > to push case analyses into the continuation argument. That is, > > case keepAlive# x expr of pat -> rhs > > ~> > > keepAlive# x (case expr of pat -> rhs) > > Of course, doing this is a bit tricky since one must rewrite the > application of keepAlive# to ensure that the resulting application is > well-typed. Admittedly, this doesn't help the runRW# case (although this > could presumably be accommodated by touch#'ing the final state token in > the runRW# desugaring emitted by CorePrep). > > On the whole, I'm not a fan of this ad-hoc option. It increases the > complexity of the simplifier all to support a single operation. By > comparison, the Core extension looks somewhat appealing. > > Cheers, > > - Ben > > > [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables > > P.S. A minor note: the keepAlive# "pseudo-instruction" mentioned on the > Wiki [1] is precisely the touch# operation we have today. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed May 27 11:03:20 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 27 May 2020 11:03:20 +0000 Subject: keepAlive# primop In-Reply-To: <45e2c2f1-b6c2-9d9b-5cf0-6f5e3317c4f5@haskus.fr> References: <04361b3f-4323-7610-1bc1-9a6a90abec2b@haskus.fr> <875ze39suw.fsf@smart-cactus.org> <45e2c2f1-b6c2-9d9b-5cf0-6f5e3317c4f5@haskus.fr> Message-ID: I’m pretty reluctant to change the Core Language to support one very dark corner use-case. Then every Core consumer and producer must understand the semantics of this new field, and respect that semantics. As I understand it, our definition keepAlive# k x s = case nodiv x s of (# s1, r #) -> case touch# k s1 of (# s2, _ #) -> (# s2, r #) does just what we want, except perhaps efficiency, without changing the language. If that’s true, let’s convince ourselves that * The efficiency cannot be recovered * The loss of efficiency matters a lot before taking more drastic steps. For completeness, nodiv :: IO a -> IO a behaves like the identity function except that (nodiv e) never says “guarantees to diverges” Simon From: Sylvain Henry Sent: 26 May 2020 18:04 To: Ben Gamari ; Simon Peyton Jones Cc: GHC developers Subject: Re: keepAlive# primop Hi ghc-devs, After a discussion today about `keepAlive#`, I think Option E [1] is even more appealing. To recap the idea was to have keep-alive variable sets attached to case-expressions in Core. E.g. `case {k} x of ...` 1. One of the issue was the semantics of `keepAlive#`. `keepAlive#` is very intricate with evaluation. As Simon mentioned we want a semantics like: keepAlive# k x ==> x `seq` touch# k `seq` x with potentially a special `seq` to deal with diverging `x`. With `case {k} x of ...` we have this semantics. Even if we throw the case alternatives if `x` diverges, we don't throw the keep-alive set so we're good. 2. Simon wanted to push `keepAlive#` into case-expressions. With this approach we should only have to fix case-of-case to take keep-alive sets into account. case {k} (case {k2} x of .. -> a; ... -> b) of C0 .. -> e1 C1 .. -> e2 ===> case {k,k2} x of ... -> case {k} a of { C0 .. -> e1; C1 ... -> e2 } ... -> case {k} b of { C0 .. -> e1; C1 ... -> e2 } 3. Compared to other approaches: we don't have to use stack frames (good for performance) and we don't have to deal with a continuation (good for Core optimizations, hence perf). 4. Implementing this approach is quite straightforward even if it modifies Core. I did it last month in [2]. This patch doesn't fully work yet with `-O` because some transformation (related to join points IIRC) doesn't take keep-alive sets into account but it should be pretty easy to fix if we want to use this approach. Given how hard it is to come up with a good design/implementation of other approaches, this one strikes me as probably the more principled we have and yet it is relatively easy to implement. What do you think? Cheers, Sylvain [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables [2] https://gitlab.haskell.org/hsyl20/ghc/-/commits/hsyl20-keepalive On 13/04/2020 19:51, Ben Gamari wrote: Ccing ghc-devs@ since this discussion is something of general interest to the community. Sylvain Henry writes: Simon, Ben, I've been reading and thinking about `readRW#` issues which are very related to issues we have with `keepAlive#` primop. To recap, the problem is that we want some transformations (that Simon has listed in [1]) to consider: ``` case runRW# f of ... case keepAlive# k a of ... ``` as if they were really: ``` case f realWorld# of ... case a of ... ``` BUT without breaking the semantics of runRW# and keepAlive#. I have been thinking about a solution that I have described on the wiki: https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables The idea is to keep a set of variable names in each Core case-expression that are kept alive during the evaluation of the scrutinee. I think it would work very nicely with your `newState#` primop described in [2], both for `runST` and for `unsafeDupablePerformIO` (details on the wiki). It requires a little more upfront work to adapt the code involving case-expressions. But it will force us to review all transformations to check if they are sound when keep-alive sets are not empty, which we would have to do anyway if we implemented another option. We could start by disabling transformations involving non-empty keep-alive sets and iterate to enable the sound ones. I would like your opinions on the approach. I may have totally missed something. Thanks for writing this down! Indeed it is an interesting idea. However, as expressed on IRC, I wonder whether this problem rises to the level where it warrants an adaptation to our Core representation. It feels a bit like the tail is wagging the dog here, especially given how the "tail" here merely exists to support FFI. That being said, this is one of the few options which remain on the table that doesn't require changes to user code. Moreover, the applicability to runRW# is quite intriguing. Another (admittedly, more ad-hoc) option that would avoid modifying Core would be to teach the simplifier about the class of "continuation-passing" primops (e.g. `keepAlive#` and `runRW#`), allowing it to push case analyses into the continuation argument. That is, case keepAlive# x expr of pat -> rhs ~> keepAlive# x (case expr of pat -> rhs) Of course, doing this is a bit tricky since one must rewrite the application of keepAlive# to ensure that the resulting application is well-typed. Admittedly, this doesn't help the runRW# case (although this could presumably be accommodated by touch#'ing the final state token in the runRW# desugaring emitted by CorePrep). On the whole, I'm not a fan of this ad-hoc option. It increases the complexity of the simplifier all to support a single operation. By comparison, the Core extension looks somewhat appealing. Cheers, - Ben [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/proposal/with-combinator#option-e-tag-core-case-expression-with-kept-alive-variables P.S. A minor note: the keepAlive# "pseudo-instruction" mentioned on the Wiki [1] is precisely the touch# operation we have today. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dyaitskov at gmail.com Thu May 28 21:18:20 2020 From: dyaitskov at gmail.com (Daneel Yaitskov) Date: Thu, 28 May 2020 14:18:20 -0700 Subject: STG/CMM questions Message-ID: Hi, There are 2 questions: 1. How to decipher myThreadId implementation in STG? What is the principle behind STG version? Is it matching first field from TSO by type? "id" is not mentioned anyhow and there is no offset to id field. The whole TSO struct is just mentioned. emitAssign and CmmLocal are used everywhere, so they don't express logic particular to myThreadId function. was: return tso->id; now: currentTSOExpr = CmmReg currentTSOReg MyThreadIdOp -> \[] -> opAllDone $ \[res] -> do emitAssign (CmmLocal res) currentTSOExpr 2. Cmm macro for new TSO field is not found. I added to deriveConstants/Main.hs: ,closureField C "StgTSO" "trace_id" TSO.h: typedef struct StgTSO_ { ... StgClosure* trace_id; PrimsOps.cmm: gcptr p; p = StgTSO_trace_id(CurrentTSO); Linking error happens after make clean && ./boot && ./configure && make -j8 so I guess macros is not defined. Looks like such macros are not written to any file and it is hard to grep them. collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:101: recipe for target 'utils/iserv/stage2/build/tmp/ghc-iserv' failed make[1]: *** [utils/iserv/stage2/build/tmp/ghc-iserv] Error 1 /home/dan/demo/haskell/compiler/ghc/rts/dist/build/libHSrts_thr_p.a(PrimOps.thr_p_o):function stg_getAdamTraceIdzh: error: undefined reference to 'StgTSO_trace_id' /home/dan/demo/haskell/compiler/ghc/rts/dist/build/libHSrts_thr_p.a(PrimOps.thr_p_o):function stg_getAdamTraceIdzh: error: undefined reference to 'n' collect2: error: ld returned 1 exit status `cc' failed in phase `Linker'. (Exit code: 1) utils/iserv/ghc.mk:105: recipe for target 'utils/iserv/stage2_p/build/tmp/ghc-iserv-prof' failed make[1]: *** [utils/iserv/stage2_p/build/tmp/ghc-iserv-prof] Error 1 Makefile:123: recipe for target 'all' failed -- Best regards, Daniil Iaitskov From ben at smart-cactus.org Fri May 29 15:22:00 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Fri, 29 May 2020 11:22:00 -0400 Subject: STG/CMM questions In-Reply-To: References: Message-ID: <87sgfipy0e.fsf@smart-cactus.org> Daneel Yaitskov writes: > Hi, > > There are 2 questions: > > 1. How to decipher myThreadId implementation in STG? > What is the principle behind STG version? > Is it matching first field from TSO by type? > "id" is not mentioned anyhow and there is no offset to id field. > The whole TSO struct is just mentioned. > See the Haddocks for GHC.Conc.Sync.ThreadId. In short, ThreadId# (which is what the myThreadId# primop returns) is an opaque thread identifier which is represented by a pointer to a TSO. The Show instance for ThreadId prints the `id` field of the TSO. However, we don't directly expose the `id` field to the user; you merely get Eq and Ord instances. Admittedly, this may be a problem for users wanting to use, e.g., a HashMap keyed on ThreadId. Perhaps we could expose the GHC.Conc.Sync.getThreadId to the user. However, using this correctly would take great care. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ml at stefansf.de Sat May 30 17:15:48 2020 From: ml at stefansf.de (Stefan Schulze Frielinghaus) Date: Sat, 30 May 2020 19:15:48 +0200 Subject: [hadrian] Use LLVM for test and validate target Message-ID: <20200530171548.GA6854@localhost.localdomain> Hi all, Is it possible to force Hadrian to use the LLVM code generator for the test and validate targets? For example, while using the following commands it seems to me that only the native code generator (if available) is used: $ ./hadrian/build -c -j --flavour=quick-llvm test $ ./hadrian/build -c -j --flavour=validate Cheers, Stefan From ben at smart-cactus.org Sat May 30 18:56:52 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Sat, 30 May 2020 14:56:52 -0400 Subject: [hadrian] Use LLVM for test and validate target In-Reply-To: <20200530171548.GA6854@localhost.localdomain> References: <20200530171548.GA6854@localhost.localdomain> Message-ID: <87lfl9p7yp.fsf@smart-cactus.org> Stefan Schulze Frielinghaus writes: > Hi all, > > Is it possible to force Hadrian to use the LLVM code generator for the > test and validate targets? For example, while using the following > commands it seems to me that only the native code generator (if > available) is used: > > $ ./hadrian/build -c -j --flavour=quick-llvm test I think you want --test-speed=slow. This enables all testsuite ways (some of which use -fllvm). > $ ./hadrian/build -c -j --flavour=validate > Note that this isn't a target but rather a build flavour. I don't believe that there is an analogous flavour which bootstraps with -fllvm, but you could easily edit its definition (either via hadrian/UserSettings.hs or by editing hadrian/src/Settings/Flavours/Validate.hs). Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: