From austin at well-typed.com Mon Sep 14 13:47:40 2015 From: austin at well-typed.com (Austin Seipp) Date: Mon, 14 Sep 2015 08:47:40 -0500 Subject: HEADS UP (devs, users): 8.0.1 Roadmap Message-ID: Hi *, I've returned from vacation, and last week Simon, Simon and I met up again after a long break, and talked a bit about the upcoming release. The good news is that it is going to be an exciting one! The flip side is, there's a lot of work to be done! The current plan we'd roughly like to follow is... - November: Fork the new `ghc-8.0` STABLE branch - At this point, `master` development will probably slow as we fix bugs. - This gives us 2 months or so until branch, from Today. - This is nice as the branch is close to the first RC. - December: First release candidate - Mid/late February: Final release. Here's our current feature roadmap (in basically the same place as all our previous pages): https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-8.0.1 As time goes on, this page will be updated to reflect Reality? and track it as closely as possible. So keep an eye on it! It's got the roadmap (near top) and large bug list (bottom). Now, there are some things we need, so depending on who you are, please... - *Users*: please look over the bug list! If there's a bug you need fixed that isn't there, set it to the 8.0.1 milestone (updated in Trac). If this bug is critical to you, please let us know! You can bump the priority (if we disagree, or it's workaround-able, it might get bumped down). We just need a way to see what you need, so please let us know somehow! As a reminder, our regular policy is this: if a bug is NOT marked highest or high priority, it is essentially 100% the case we will not look at it. So please make sure this is accurate. Or if you can, write a patch yourself! - *Developers*: double check the roadmap list, _and if you're responsible for something, make sure it is accurate!_ There are some great things planned to land in HEAD, but we'll have to work for it. Onward! - A better LLVM backend for Tier-1 platforms - Types are kinds and kind equality - Overloaded record fields! - Enhancements to DWARF debugging - ApplicativeDo - ... and many more... Thanks everyone! -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ From austin at well-typed.com Mon Sep 14 13:53:38 2015 From: austin at well-typed.com (Austin Seipp) Date: Mon, 14 Sep 2015 08:53:38 -0500 Subject: HEADS UP: Need 7.10.3? Message-ID: Hi *, (This is an email primarily aimed at users reading this list and developers who have any interest). As some of you may know, there's currently a 7.10.3 milestone and status page on our wiki: https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.10.3 The basic summary is best captured on the above page: "We have not yet decided when, or even whether, to release GHC 7.10.3. We will do so if (but only if!) we have documented cases of "show-stoppers" in 7.10.2. Namely, cases from users where - You are unable to use 7.10.2 because of some bug - There is no reasonable workaround, so you are truly stuck - We know how to fix it - The fix is not too disruptive; i.e. does not risk introducing a raft of new bugs" That is, we're currently not fully sold on the need for a release. However, the milestone and issue page serve as a useful guide, and also make it easier to keep track of smaller, point-release worthy issues. So in the wake of the 8.0 roadmap I just sent: If you *need* 7.10.3 because the 7.10.x series has a major regression or problem you can't work around, let us know! - Find or file a bug in Trac - Make sure it's highest priority - Assign it to the 7.10.3 milestone - Follow up on this email if possible, or edit it on the status page text above - it would be nice to get some public feedback in one place about what everyone needs. Currently we have two bugs on the listed page in the 'show stopper category', possibly the same bug, which is a deal-breaker for HERMIT I believe. Knowing of anything else would be very useful. Thanks all! -- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/ From ruben.moor at gmail.com Wed Sep 16 11:38:52 2015 From: ruben.moor at gmail.com (Ruben Moor) Date: Wed, 16 Sep 2015 13:38:52 +0200 Subject: HEADS UP: Need 7.10.3? Message-ID: Hello, I want to support the case of a GHC 7.10.3 release. The show-stopper for our company in 7.10.2 consists of ticket #10726 https://ghc.haskell.org/trac/ghc/ticket/10726 and ticket #10795 https://ghc.haskell.org/trac/ghc/ticket/10795 I.e. I am not able to compile my code on MS Windows and reached a point where my workaround is exhausted (reducing overall file path length in the project dependency). Luckily the fix has already been implemented and merged (into the 7.10 branch and HEAD). Cf. this commit https://github.com/ghc/ghc/commit/296bc70b5ff6c853f2782e9ec5aa47a52110345e and this stack issue: https://github.com/commercialhaskell/stack/issues/795#issuecomment-140618684 Thanks! Ruben Moor -------------- next part -------------- An HTML attachment was scrubbed... URL: From afarmer at ittc.ku.edu Wed Sep 16 14:50:10 2015 From: afarmer at ittc.ku.edu (Andrew Farmer) Date: Wed, 16 Sep 2015 07:50:10 -0700 Subject: HEADS UP: Need 7.10.3? In-Reply-To: References: Message-ID: As you mentioned, the two show stoppers for HERMIT are #10528 (specifically SPJs commit in comment:15 - see [1]) and #10829 (see D1246). The first disables inlining/rule application in the LHS of rules, the second does the same in the RHS. nofib results for the latter are on the ticket. I've set both to 7.10.3 milestone and high priority... thanks for merging them! [1] bc4b64ca5b99bff6b3d5051b57cb2bc52bd4c841 On Mon, Sep 14, 2015 at 6:53 AM, Austin Seipp wrote: > Hi *, > > (This is an email primarily aimed at users reading this list and > developers who have any interest). > > As some of you may know, there's currently a 7.10.3 milestone and > status page on our wiki: > > https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.10.3 > > The basic summary is best captured on the above page: > > "We have not yet decided when, or even whether, to release GHC 7.10.3. > We will do so if (but only if!) we have documented cases of > "show-stoppers" in 7.10.2. Namely, cases from users where > > - You are unable to use 7.10.2 because of some bug > - There is no reasonable workaround, so you are truly stuck > - We know how to fix it > - The fix is not too disruptive; i.e. does not risk introducing a > raft of new bugs" > > That is, we're currently not fully sold on the need for a release. > However, the milestone and issue page serve as a useful guide, and > also make it easier to keep track of smaller, point-release worthy > issues. > > So in the wake of the 8.0 roadmap I just sent: If you *need* 7.10.3 > because the 7.10.x series has a major regression or problem you can't > work around, let us know! > > - Find or file a bug in Trac > - Make sure it's highest priority > - Assign it to the 7.10.3 milestone > - Follow up on this email if possible, or edit it on the status page > text above - it would be nice to get some public feedback in one place > about what everyone needs. > > Currently we have two bugs on the listed page in the 'show stopper > category', possibly the same bug, which is a deal-breaker for HERMIT I > believe. Knowing of anything else would be very useful. > > Thanks all! > > -- > Regards, > > Austin Seipp, Haskell Consultant > Well-Typed LLP, http://www.well-typed.com/ > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > From svenpanne at gmail.com Thu Sep 17 16:46:01 2015 From: svenpanne at gmail.com (Sven Panne) Date: Thu, 17 Sep 2015 18:46:01 +0200 Subject: HEADS UP: Need 7.10.3? In-Reply-To: References: Message-ID: Building Haddock documentation on Windows for larger packages (e.g. OpenGLRaw) is broken in 7.10.2, similar to linking: The reason is once again the silly Windows command line length limitation, so we need response files here, too. Haddock 2.16.1 already has support for this, but this seems to be broken (probably https://github.com/haskell/haddock/commit/9affe8f6b3a9b07367c8c14162aecea8b15856a6 is missing), see the corresponding check in cabal ( https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Haddock.hs#L470 ). So in a nutshell: We would need a new Haddock release (bundled with GHC 7.10.3) and a new cabal release with support for Haddock response files (in cabal's HEAD, but not yet released). Would this be possible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hvr at gnu.org Wed Sep 23 11:17:22 2015 From: hvr at gnu.org (Herbert Valerio Riedel) Date: Wed, 23 Sep 2015 13:17:22 +0200 Subject: ANN: CfN for new Haskell Prime language committee Message-ID: <87k2rh8jwd.fsf@gnu.org> Dear Haskell Community, In short, it's time to assemble a new Haskell Prime language committee. Please refer to the CfN at https://mail.haskell.org/pipermail/haskell-prime/2015-September/003936.html for more details. Cheers, hvr -- PGP fingerprint: 427C B69A AC9D 00F2 A43C AF1C BA3C BA3F FE22 B574 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 818 bytes Desc: not available URL: From me at willsewell.com Mon Sep 28 16:08:33 2015 From: me at willsewell.com (Will Sewell) Date: Mon, 28 Sep 2015 17:08:33 +0100 Subject: Removing latency spikes. Garbage collector related? Message-ID: Hi, I was told in the #haskell IRC channel that this would be a good place to ask this question, so here goes! We?re writing a low-latency messaging system. The problem is we are getting a lot of latency spikes. See this image: http://i.imgur.com/GZ0Ek98.png (yellow to red is the 90th percentile), which shows end-to-end latency of messages through the system. I have tried to eliminate the problem by removing parts of the system that I suspected to be expensive, but the spikes are still there. I?m now thinking that it?s the GC. As you can see in this output from ghc-events-analyze, work on the GC thread (red) seems to be blocking work on the main program thread (green) http://i.imgur.com/4YO5q4U.png (x axis is time, darkness of buckets is % CPU time). *Note: the graphs are not of the same run, but are typical* Do you think the GC is the most likely culprit? Is there anything I can do to confirm this hypothesis? (I looked into turning off the GC, but this seems tricky) If it is the GC, then is there anything that can be done about it? Thanks! Will From greg at gregorycollins.net Mon Sep 28 18:02:35 2015 From: greg at gregorycollins.net (Gregory Collins) Date: Mon, 28 Sep 2015 11:02:35 -0700 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: > If it is the GC, then is there anything that can be done about it? - Increase value of -A (the default is too small) -- best value for this is L3 cache size of the chip - Increase value of -H (total heap size) -- this will use more ram but you'll run GC less often - This will sound flip, but: generate less garbage. Frequency of GC runs is proportional to the amount of garbage being produced, so if you can lower mutator allocation rate then you will also increase net productivity. Built-up thunks can transparently hide a lot of allocation so fire up the profiler and tighten those up (there's an 80-20 rule here). Reuse output buffers if you aren't already, etc. G -- Gregory Collins -------------- next part -------------- An HTML attachment was scrubbed... URL: From jwlato at gmail.com Mon Sep 28 18:31:39 2015 From: jwlato at gmail.com (John Lato) Date: Mon, 28 Sep 2015 18:31:39 +0000 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: Try Greg's recommendations first. If you still need to do more investigation, I'd recommend that you look at some samples with either threadscope or dumping the eventlog to text. I really like ghc-events-analyze, but it doesn't provide quite the same level of detail. You may also want to dump some of your metrics into the eventlog, because then you'll be able to see exactly how high latency episodes line up with GC pauses. On Mon, Sep 28, 2015 at 1:02 PM Gregory Collins wrote: > > On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: > >> If it is the GC, then is there anything that can be done about it? > > > - Increase value of -A (the default is too small) -- best value for > this is L3 cache size of the chip > - Increase value of -H (total heap size) -- this will use more ram but > you'll run GC less often > - This will sound flip, but: generate less garbage. Frequency of GC > runs is proportional to the amount of garbage being produced, so if you can > lower mutator allocation rate then you will also increase net productivity. > Built-up thunks can transparently hide a lot of allocation so fire up the > profiler and tighten those up (there's an 80-20 rule here). Reuse output > buffers if you aren't already, etc. > > G > > -- > Gregory Collins > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at willsewell.com Tue Sep 29 09:03:58 2015 From: me at willsewell.com (Will Sewell) Date: Tue, 29 Sep 2015 10:03:58 +0100 Subject: Fwd: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: Thanks for the reply Greg. I have already tried tweaking these values a bit, and this is what I found: * I first tried -A256k because the L2 cache is that size (Simon Marlow mentioned this can lead to good performance http://stackoverflow.com/a/3172704/1018290) * I then tried a value of -A2048k because he also said "using a very large young generation size might outweigh the cache benefits". I don't exactly know what he meant by "a very large young generation size", so I guessed at this value. Is it in the right ballpark? * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m But all lead to worse performance over the defaults (and -H didn't really have much affect at all). I will try your suggestion of setting -A to the L3 cache size. Are there any other values I should try setting these at? As for your final point, I have run space profiling, and it looks like >90% of the memory is used for our message index, which is a temporary store of messages that have gone through the system. These messages are stored in aligned chunks in memory that are merged together. I initially though this was causing the spikes, but they were still there even after I removed the component. I will try and run space profiling in the build with the message index. Thanks again. On 28 September 2015 at 19:02, Gregory Collins wrote: > > On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >> >> If it is the GC, then is there anything that can be done about it? > > Increase value of -A (the default is too small) -- best value for this is L3 > cache size of the chip > Increase value of -H (total heap size) -- this will use more ram but you'll > run GC less often > This will sound flip, but: generate less garbage. Frequency of GC runs is > proportional to the amount of garbage being produced, so if you can lower > mutator allocation rate then you will also increase net productivity. > Built-up thunks can transparently hide a lot of allocation so fire up the > profiler and tighten those up (there's an 80-20 rule here). Reuse output > buffers if you aren't already, etc. > > G > > -- > Gregory Collins From me at willsewell.com Tue Sep 29 09:15:41 2015 From: me at willsewell.com (Will Sewell) Date: Tue, 29 Sep 2015 10:15:41 +0100 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: Thanks for the reply John. I will have a go at doing that. What do you mean exactly by dumping metrics, do you mean measuring the latency within the program, and dumping it if it exceeds a certain threshold? And from the answers I'm assuming you believe it is the GC that is most likely causing these spikes. I've never profiled Haskell code, so I'm not used to seeing what the effects of the GC actually are. On 28 September 2015 at 19:31, John Lato wrote: > Try Greg's recommendations first. If you still need to do more > investigation, I'd recommend that you look at some samples with either > threadscope or dumping the eventlog to text. I really like > ghc-events-analyze, but it doesn't provide quite the same level of detail. > You may also want to dump some of your metrics into the eventlog, because > then you'll be able to see exactly how high latency episodes line up with GC > pauses. > > On Mon, Sep 28, 2015 at 1:02 PM Gregory Collins > wrote: >> >> >> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >>> >>> If it is the GC, then is there anything that can be done about it? >> >> Increase value of -A (the default is too small) -- best value for this is >> L3 cache size of the chip >> Increase value of -H (total heap size) -- this will use more ram but >> you'll run GC less often >> This will sound flip, but: generate less garbage. Frequency of GC runs is >> proportional to the amount of garbage being produced, so if you can lower >> mutator allocation rate then you will also increase net productivity. >> Built-up thunks can transparently hide a lot of allocation so fire up the >> profiler and tighten those up (there's an 80-20 rule here). Reuse output >> buffers if you aren't already, etc. >> >> G >> >> -- >> Gregory Collins >> _______________________________________________ >> Glasgow-haskell-users mailing list >> Glasgow-haskell-users at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users From semanticphilosopher at gmail.com Tue Sep 29 09:16:09 2015 From: semanticphilosopher at gmail.com (Neil Davies) Date: Tue, 29 Sep 2015 10:16:09 +0100 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: <870FDAF7-3E2C-4C5E-B0DC-A3A3668DDE76@gmail.com> Will is your issue with the spikes i response time, rather than the mean values? If so, once you?ve reduced the amount of unnecessary mutation, you might want to take more control over when the GC is taking place. You might want to disable GC on timer (-I0) and force GC to occur at points you select - we found this useful. Lastly, is the arrival pattern (and distribution pattern) of messages constant or variable? just making sure that you are not trying to fight basic queueing theory here. Neil On 29 Sep 2015, at 10:03, Will Sewell wrote: > Thanks for the reply Greg. I have already tried tweaking these values > a bit, and this is what I found: > > * I first tried -A256k because the L2 cache is that size (Simon Marlow > mentioned this can lead to good performance > http://stackoverflow.com/a/3172704/1018290) > * I then tried a value of -A2048k because he also said "using a very > large young generation size might outweigh the cache benefits". I > don't exactly know what he meant by "a very large young generation > size", so I guessed at this value. Is it in the right ballpark? > * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m > > But all lead to worse performance over the defaults (and -H didn't > really have much affect at all). > > I will try your suggestion of setting -A to the L3 cache size. > > Are there any other values I should try setting these at? > > As for your final point, I have run space profiling, and it looks like >> 90% of the memory is used for our message index, which is a temporary > store of messages that have gone through the system. These messages > are stored in aligned chunks in memory that are merged together. I > initially though this was causing the spikes, but they were still > there even after I removed the component. I will try and run space > profiling in the build with the message index. > > Thanks again. > > On 28 September 2015 at 19:02, Gregory Collins wrote: >> >> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >>> >>> If it is the GC, then is there anything that can be done about it? >> >> Increase value of -A (the default is too small) -- best value for this is L3 >> cache size of the chip >> Increase value of -H (total heap size) -- this will use more ram but you'll >> run GC less often >> This will sound flip, but: generate less garbage. Frequency of GC runs is >> proportional to the amount of garbage being produced, so if you can lower >> mutator allocation rate then you will also increase net productivity. >> Built-up thunks can transparently hide a lot of allocation so fire up the >> profiler and tighten those up (there's an 80-20 rule here). Reuse output >> buffers if you aren't already, etc. >> >> G >> >> -- >> Gregory Collins > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users From me at willsewell.com Tue Sep 29 09:35:55 2015 From: me at willsewell.com (Will Sewell) Date: Tue, 29 Sep 2015 10:35:55 +0100 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: <870FDAF7-3E2C-4C5E-B0DC-A3A3668DDE76@gmail.com> References: <870FDAF7-3E2C-4C5E-B0DC-A3A3668DDE76@gmail.com> Message-ID: Thank you for the reply Neil. The spikes are in response time. The graph I linked to shows the distribution of response times in a given window of time (darkness of the square is the number of messages in a particular window of response time). So the spikes are in the mean and also the max response time. Having said that I'm not exactly sure what you mean by "mean values". I will have a look into -I0. Yes the arrival of messages is constant. This graph shows the number of messages that have been published to the system: http://i.imgur.com/ADzMPIp.png On 29 September 2015 at 10:16, Neil Davies wrote: > Will > > is your issue with the spikes i response time, rather than the mean values? > > If so, once you?ve reduced the amount of unnecessary mutation, you might want > to take more control over when the GC is taking place. You might want to disable > GC on timer (-I0) and force GC to occur at points you select - we found this useful. > > Lastly, is the arrival pattern (and distribution pattern) of messages constant or > variable? just making sure that you are not trying to fight basic queueing theory here. > > > Neil > > On 29 Sep 2015, at 10:03, Will Sewell wrote: > >> Thanks for the reply Greg. I have already tried tweaking these values >> a bit, and this is what I found: >> >> * I first tried -A256k because the L2 cache is that size (Simon Marlow >> mentioned this can lead to good performance >> http://stackoverflow.com/a/3172704/1018290) >> * I then tried a value of -A2048k because he also said "using a very >> large young generation size might outweigh the cache benefits". I >> don't exactly know what he meant by "a very large young generation >> size", so I guessed at this value. Is it in the right ballpark? >> * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m >> >> But all lead to worse performance over the defaults (and -H didn't >> really have much affect at all). >> >> I will try your suggestion of setting -A to the L3 cache size. >> >> Are there any other values I should try setting these at? >> >> As for your final point, I have run space profiling, and it looks like >>> 90% of the memory is used for our message index, which is a temporary >> store of messages that have gone through the system. These messages >> are stored in aligned chunks in memory that are merged together. I >> initially though this was causing the spikes, but they were still >> there even after I removed the component. I will try and run space >> profiling in the build with the message index. >> >> Thanks again. >> >> On 28 September 2015 at 19:02, Gregory Collins wrote: >>> >>> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >>>> >>>> If it is the GC, then is there anything that can be done about it? >>> >>> Increase value of -A (the default is too small) -- best value for this is L3 >>> cache size of the chip >>> Increase value of -H (total heap size) -- this will use more ram but you'll >>> run GC less often >>> This will sound flip, but: generate less garbage. Frequency of GC runs is >>> proportional to the amount of garbage being produced, so if you can lower >>> mutator allocation rate then you will also increase net productivity. >>> Built-up thunks can transparently hide a lot of allocation so fire up the >>> profiler and tighten those up (there's an 80-20 rule here). Reuse output >>> buffers if you aren't already, etc. >>> >>> G >>> >>> -- >>> Gregory Collins >> _______________________________________________ >> Glasgow-haskell-users mailing list >> Glasgow-haskell-users at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > From semanticphilosopher at gmail.com Tue Sep 29 10:45:23 2015 From: semanticphilosopher at gmail.com (Neil Davies) Date: Tue, 29 Sep 2015 11:45:23 +0100 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: <870FDAF7-3E2C-4C5E-B0DC-A3A3668DDE76@gmail.com> Message-ID: Will I was trying to get a feeling for what those coloured squares actually denoted - typically we examine this sort of performance information as CDFs (cumulative distribution functions[1]) trying to pull apart the issues that ?mean? effecting (i.e typical path through code/system) and those that are ?tail? effecting (i.e exceptions - and GC running could be seen as an ?exception? - one that you can manage and time shift in the relative timing). I?m assuming that messages have a similar ?cost? (i.e similar work to complete) - so that a uniform arrival rate equates to a uniform rate of work to be done arriving. Neil [1] We plot the CDF?s in two ways, the ?usual? way for the major part of the probability mass and then as a (1-CDF) on a log log scale to expose the tail behaviour. On 29 Sep 2015, at 10:35, Will Sewell wrote: > Thank you for the reply Neil. > > The spikes are in response time. The graph I linked to shows the > distribution of response times in a given window of time (darkness of > the square is the number of messages in a particular window of > response time). So the spikes are in the mean and also the max > response time. Having said that I'm not exactly sure what you mean by > "mean values". > > I will have a look into -I0. > > Yes the arrival of messages is constant. This graph shows the number > of messages that have been published to the system: > http://i.imgur.com/ADzMPIp.png > > On 29 September 2015 at 10:16, Neil Davies > wrote: >> Will >> >> is your issue with the spikes i response time, rather than the mean values? >> >> If so, once you?ve reduced the amount of unnecessary mutation, you might want >> to take more control over when the GC is taking place. You might want to disable >> GC on timer (-I0) and force GC to occur at points you select - we found this useful. >> >> Lastly, is the arrival pattern (and distribution pattern) of messages constant or >> variable? just making sure that you are not trying to fight basic queueing theory here. >> >> >> Neil >> >> On 29 Sep 2015, at 10:03, Will Sewell wrote: >> >>> Thanks for the reply Greg. I have already tried tweaking these values >>> a bit, and this is what I found: >>> >>> * I first tried -A256k because the L2 cache is that size (Simon Marlow >>> mentioned this can lead to good performance >>> http://stackoverflow.com/a/3172704/1018290) >>> * I then tried a value of -A2048k because he also said "using a very >>> large young generation size might outweigh the cache benefits". I >>> don't exactly know what he meant by "a very large young generation >>> size", so I guessed at this value. Is it in the right ballpark? >>> * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m >>> >>> But all lead to worse performance over the defaults (and -H didn't >>> really have much affect at all). >>> >>> I will try your suggestion of setting -A to the L3 cache size. >>> >>> Are there any other values I should try setting these at? >>> >>> As for your final point, I have run space profiling, and it looks like >>>> 90% of the memory is used for our message index, which is a temporary >>> store of messages that have gone through the system. These messages >>> are stored in aligned chunks in memory that are merged together. I >>> initially though this was causing the spikes, but they were still >>> there even after I removed the component. I will try and run space >>> profiling in the build with the message index. >>> >>> Thanks again. >>> >>> On 28 September 2015 at 19:02, Gregory Collins wrote: >>>> >>>> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >>>>> >>>>> If it is the GC, then is there anything that can be done about it? >>>> >>>> Increase value of -A (the default is too small) -- best value for this is L3 >>>> cache size of the chip >>>> Increase value of -H (total heap size) -- this will use more ram but you'll >>>> run GC less often >>>> This will sound flip, but: generate less garbage. Frequency of GC runs is >>>> proportional to the amount of garbage being produced, so if you can lower >>>> mutator allocation rate then you will also increase net productivity. >>>> Built-up thunks can transparently hide a lot of allocation so fire up the >>>> profiler and tighten those up (there's an 80-20 rule here). Reuse output >>>> buffers if you aren't already, etc. >>>> >>>> G >>>> >>>> -- >>>> Gregory Collins >>> _______________________________________________ >>> Glasgow-haskell-users mailing list >>> Glasgow-haskell-users at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users >> From me at willsewell.com Tue Sep 29 12:37:33 2015 From: me at willsewell.com (Will Sewell) Date: Tue, 29 Sep 2015 13:37:33 +0100 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: <870FDAF7-3E2C-4C5E-B0DC-A3A3668DDE76@gmail.com> Message-ID: That's interesting. I have not done this kind of work before, and had not come across CDFs. I can see why it make sense to look at the mean and tail. Your assumption is correct. The messages have a similar cost, which is why the graph I posted is relatively flat most of the time. The spikes suggest to me that it is a tail affecting issue because the messages are following the same code path as when it is running normally. On 29 September 2015 at 11:45, Neil Davies wrote: > Will > > I was trying to get a feeling for what those coloured squares actually > denoted - typically we examine this sort of performance information > as CDFs (cumulative distribution functions[1]) trying to pull apart the > issues that ?mean? effecting (i.e typical path through code/system) and > those that are ?tail? effecting (i.e exceptions - and GC running could > be seen as an ?exception? - one that you can manage and time shift > in the relative timing). > > I?m assuming that messages have a similar ?cost? (i.e similar work > to complete) - so that a uniform arrival rate equates to a uniform > rate of work to be done arriving. > > Neil > [1] We plot the CDF?s in two ways, the ?usual? way for the major part > of the probability mass and then as a (1-CDF) on a log log scale to > expose the tail behaviour. > > On 29 Sep 2015, at 10:35, Will Sewell wrote: > >> Thank you for the reply Neil. >> >> The spikes are in response time. The graph I linked to shows the >> distribution of response times in a given window of time (darkness of >> the square is the number of messages in a particular window of >> response time). So the spikes are in the mean and also the max >> response time. Having said that I'm not exactly sure what you mean by >> "mean values". >> >> I will have a look into -I0. >> >> Yes the arrival of messages is constant. This graph shows the number >> of messages that have been published to the system: >> http://i.imgur.com/ADzMPIp.png >> >> On 29 September 2015 at 10:16, Neil Davies >> wrote: >>> Will >>> >>> is your issue with the spikes i response time, rather than the mean values? >>> >>> If so, once you?ve reduced the amount of unnecessary mutation, you might want >>> to take more control over when the GC is taking place. You might want to disable >>> GC on timer (-I0) and force GC to occur at points you select - we found this useful. >>> >>> Lastly, is the arrival pattern (and distribution pattern) of messages constant or >>> variable? just making sure that you are not trying to fight basic queueing theory here. >>> >>> >>> Neil >>> >>> On 29 Sep 2015, at 10:03, Will Sewell wrote: >>> >>>> Thanks for the reply Greg. I have already tried tweaking these values >>>> a bit, and this is what I found: >>>> >>>> * I first tried -A256k because the L2 cache is that size (Simon Marlow >>>> mentioned this can lead to good performance >>>> http://stackoverflow.com/a/3172704/1018290) >>>> * I then tried a value of -A2048k because he also said "using a very >>>> large young generation size might outweigh the cache benefits". I >>>> don't exactly know what he meant by "a very large young generation >>>> size", so I guessed at this value. Is it in the right ballpark? >>>> * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m >>>> >>>> But all lead to worse performance over the defaults (and -H didn't >>>> really have much affect at all). >>>> >>>> I will try your suggestion of setting -A to the L3 cache size. >>>> >>>> Are there any other values I should try setting these at? >>>> >>>> As for your final point, I have run space profiling, and it looks like >>>>> 90% of the memory is used for our message index, which is a temporary >>>> store of messages that have gone through the system. These messages >>>> are stored in aligned chunks in memory that are merged together. I >>>> initially though this was causing the spikes, but they were still >>>> there even after I removed the component. I will try and run space >>>> profiling in the build with the message index. >>>> >>>> Thanks again. >>>> >>>> On 28 September 2015 at 19:02, Gregory Collins wrote: >>>>> >>>>> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: >>>>>> >>>>>> If it is the GC, then is there anything that can be done about it? >>>>> >>>>> Increase value of -A (the default is too small) -- best value for this is L3 >>>>> cache size of the chip >>>>> Increase value of -H (total heap size) -- this will use more ram but you'll >>>>> run GC less often >>>>> This will sound flip, but: generate less garbage. Frequency of GC runs is >>>>> proportional to the amount of garbage being produced, so if you can lower >>>>> mutator allocation rate then you will also increase net productivity. >>>>> Built-up thunks can transparently hide a lot of allocation so fire up the >>>>> profiler and tighten those up (there's an 80-20 rule here). Reuse output >>>>> buffers if you aren't already, etc. >>>>> >>>>> G >>>>> >>>>> -- >>>>> Gregory Collins >>>> _______________________________________________ >>>> Glasgow-haskell-users mailing list >>>> Glasgow-haskell-users at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users >>> > From greg at gregorycollins.net Tue Sep 29 15:33:06 2015 From: greg at gregorycollins.net (Gregory Collins) Date: Tue, 29 Sep 2015 08:33:06 -0700 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 2:03 AM, Will Sewell wrote: > * I then tried a value of -A2048k because he also said "using a very > large young generation size might outweigh the cache benefits". I > don't exactly know what he meant by "a very large young generation > size", so I guessed at this value. Is it in the right ballpark? > I usually use 2-8M for this value, depending on the chip. Most values in the young generation are going to be garbage, and collection is O(num_live_objects), so as long as you can keep this buffer and your working set (i.e. the long-lived stuff that doesn't get GC'ed) in L3 cache, higher values are better. I expect there is another such phase transition as you set -A around the L2 cache size, but everything depends on what your program is actually doing. Keeping a smaller young generation will mean that those cache lines are hotter than they would be if you set it larger, and that means increasing L2 cache pressure and potentially evicting working set, so maybe you make average GC pause time faster (helping with tail latency) at the expense of doing GC more often and maybe reducing the amount of L2 cache available. * With -H, I tried values of -H8m, -H32m, -H128m, -H512m, -H1024m > > But all lead to worse performance over the defaults (and -H didn't > really have much affect at all). > What you should expect to see as you increase -H is that major GC pauses become more infrequent, but average GC times go up. Dumping +RTS -S for us will also help us understand your GC behaviour, since I wouldn't expect to see 1s pauses on any but the largest heaps. Are you using large MutableArrays? -- Gregory Collins -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdrexl at fastmail.fm Tue Sep 29 15:39:56 2015 From: mdrexl at fastmail.fm (Moritz Drexl) Date: Tue, 29 Sep 2015 17:39:56 +0200 Subject: HEADS UP: Need 7.10.3? In-Reply-To: <1443540992.3162169.396622561.1B00C2B8@webmail.messagingengine.com> References: <1443540992.3162169.396622561.1B00C2B8@webmail.messagingengine.com> Message-ID: <1443541196.3162452.396631361.2BA282CA@webmail.messagingengine.com> Hi all, For me, a 7.10.3 release is also urgently needed. Showstopper is the problem regarding invocation of GCC under Windows, with response files as an already implemented solution. See: https://ghc.haskell.org/trac/ghc/ticket/10726 https://ghc.haskell.org/trac/ghc/ticket/10795 https://github.com/ghc/ghc/commit/296bc70b5ff6c853f2782e9ec5aa47a52110345e I am unable to compile a working GHC version of the patched 7.10 branch. I exhausted all other avenues and am unable to make releases for customers using Windows. Thanks! Regards Moritz Drexl From jwlato at gmail.com Tue Sep 29 15:47:20 2015 From: jwlato at gmail.com (John Lato) Date: Tue, 29 Sep 2015 15:47:20 +0000 Subject: Removing latency spikes. Garbage collector related? In-Reply-To: References: Message-ID: By dumping metrics, I mean essentially the same as the ghc-events-analyze annotations but with any more information that is useful for the investigation. In particular, if you have a message id, include that. You may also want to annotate thread names with GHC.Conc.labelThread. You may also want to add more annotations to drill down if you uncover a problem area. If I were investigating, I would take e.g. the five largest outliers, then look in the (text) eventlog for those message ids, and see what happened between the start and stop. You'll likely want to track the thread states (which is why I suggested you annotate the thread names). I'm not convinced it's entirely the GC, the latencies are larger than I would expect from a GC pause (although lots of factors can affect that). I suspect that either you have something causing abnormal GC spikes, or there's a different cause. On 04:15, Tue, Sep 29, 2015 Will Sewell wrote: > Thanks for the reply John. I will have a go at doing that. What do you > mean exactly by dumping metrics, do you mean measuring the latency > within the program, and dumping it if it exceeds a certain threshold? > > And from the answers I'm assuming you believe it is the GC that is > most likely causing these spikes. I've never profiled Haskell code, so > I'm not used to seeing what the effects of the GC actually are. > > On 28 September 2015 at 19:31, John Lato wrote: > > Try Greg's recommendations first. If you still need to do more > > investigation, I'd recommend that you look at some samples with either > > threadscope or dumping the eventlog to text. I really like > > ghc-events-analyze, but it doesn't provide quite the same level of > detail. > > You may also want to dump some of your metrics into the eventlog, because > > then you'll be able to see exactly how high latency episodes line up > with GC > > pauses. > > > > On Mon, Sep 28, 2015 at 1:02 PM Gregory Collins > > > wrote: > >> > >> > >> On Mon, Sep 28, 2015 at 9:08 AM, Will Sewell wrote: > >>> > >>> If it is the GC, then is there anything that can be done about it? > >> > >> Increase value of -A (the default is too small) -- best value for this > is > >> L3 cache size of the chip > >> Increase value of -H (total heap size) -- this will use more ram but > >> you'll run GC less often > >> This will sound flip, but: generate less garbage. Frequency of GC runs > is > >> proportional to the amount of garbage being produced, so if you can > lower > >> mutator allocation rate then you will also increase net productivity. > >> Built-up thunks can transparently hide a lot of allocation so fire up > the > >> profiler and tighten those up (there's an 80-20 rule here). Reuse output > >> buffers if you aren't already, etc. > >> > >> G > >> > >> -- > >> Gregory Collins > >> _______________________________________________ > >> Glasgow-haskell-users mailing list > >> Glasgow-haskell-users at haskell.org > >> http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Tue Sep 29 16:04:42 2015 From: ben at well-typed.com (Ben Gamari) Date: Tue, 29 Sep 2015 18:04:42 +0200 Subject: HEADS UP: Need 7.10.3? In-Reply-To: <1443541196.3162452.396631361.2BA282CA@webmail.messagingengine.com> References: <1443540992.3162169.396622561.1B00C2B8@webmail.messagingengine.com> <1443541196.3162452.396631361.2BA282CA@webmail.messagingengine.com> Message-ID: <87eghh6wkl.fsf@smart-cactus.org> Moritz Drexl writes: > Hi all, > > For me, a 7.10.3 release is also urgently needed. > > Showstopper is the problem regarding invocation of GCC under Windows, > with response files as an already implemented solution. See: > > https://ghc.haskell.org/trac/ghc/ticket/10726 > https://ghc.haskell.org/trac/ghc/ticket/10795 > https://github.com/ghc/ghc/commit/296bc70b5ff6c853f2782e9ec5aa47a52110345e > > I am unable to compile a working GHC version of the patched 7.10 branch. > I exhausted all other avenues and am unable to make releases for > customers using Windows. > Oh dear, this does indeed sound bad. Thanks for bringing this up! We're currently working through merging tickets back to ghc-7.10 and will hopefully have a release in several of weeks. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 472 bytes Desc: not available URL: From svenpanne at gmail.com Wed Sep 30 18:04:33 2015 From: svenpanne at gmail.com (Sven Panne) Date: Wed, 30 Sep 2015 20:04:33 +0200 Subject: Typing pattern synonyms Message-ID: The type of a pattern synonym like pattern FOO = 1234 seems to be '(Eq a, Num a) => a', which makes partially makes sense, although it's not immediately clear to me where the 'Eq a' part comes from. But probably that would be clear if I read the desugaring rules closely enough. ;-) My real question is about: pattern FOO = 1234 :: Int This doesn't compile out of the box, GHC seems to require ScopedTypeVariables, too: Illegal type signature: `Int' Perhaps you intended to use ScopedTypeVariables In a pattern type-signature Why is this the case? From a user perspective, the error is totally bogus, there are no visible type variables at all. Can GHC be fixed to avoid enabling ScopedTypeVariables? Cheers, S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Wed Sep 30 18:10:59 2015 From: david.feuer at gmail.com (David Feuer) Date: Wed, 30 Sep 2015 14:10:59 -0400 Subject: Typing pattern synonyms In-Reply-To: References: Message-ID: The Eq constraint is needed to support pattern matching, the raison d??tre of pattern synonyms. I'm pretty sure the reason you need ScopedTypeVariables for your second example is that GHC only allows pattern signatures with that extension enabled. Once upon a time there was a separate PatternSignatures extension, I believe, but not any more. On Sep 30, 2015 2:04 PM, "Sven Panne" wrote: > The type of a pattern synonym like > > pattern FOO = 1234 > > seems to be '(Eq a, Num a) => a', which makes partially makes sense, > although it's not immediately clear to me where the 'Eq a' part comes from. > But probably that would be clear if I read the desugaring rules closely > enough. ;-) My real question is about: > > pattern FOO = 1234 :: Int > > This doesn't compile out of the box, GHC seems to require > ScopedTypeVariables, too: > > Illegal type signature: `Int' > Perhaps you intended to use ScopedTypeVariables > In a pattern type-signature > > Why is this the case? From a user perspective, the error is totally bogus, > there are no visible type variables at all. Can GHC be fixed to avoid > enabling ScopedTypeVariables? > > Cheers, > S. > > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spam at scientician.net Wed Sep 30 18:26:40 2015 From: spam at scientician.net (Bardur Arantsson) Date: Wed, 30 Sep 2015 20:26:40 +0200 Subject: Typing pattern synonyms In-Reply-To: References: Message-ID: On 09/30/2015 08:10 PM, David Feuer wrote: > The Eq constraint is needed to support pattern matching, the raison d??tre > of pattern synonyms. I'm pretty sure the reason you need > ScopedTypeVariables for your second example is that GHC only allows pattern > signatures with that extension enabled. Once upon a time there was a > separate PatternSignatures extension, I believe, but not any more. Perhaps ScopedTypeVariables by default (or only?) would be relevant for the revamped[1] Haskell' committee? Would there be any actual downsides? It's always seemed to me that ScopedTypeVariables was the way it should have been from the start (given perfect foresight). Regards, [1] http://permalink.gmane.org/gmane.comp.lang.haskell.prime/3965 From david.feuer at gmail.com Wed Sep 30 18:33:59 2015 From: david.feuer at gmail.com (David Feuer) Date: Wed, 30 Sep 2015 14:33:59 -0400 Subject: Typing pattern synonyms In-Reply-To: References: Message-ID: Edward Kmett seems to have some concerns about its implications for type checker implementations. Personally, I have little interest in programming in a Haskell that doesn't have a feature with the power to express what ScopedTypeVariables can. That said, there are some weird corners in the design relating to pattern signatures and in particular to pattern bindings. On Sep 30, 2015 2:26 PM, "Bardur Arantsson" wrote: > On 09/30/2015 08:10 PM, David Feuer wrote: > > The Eq constraint is needed to support pattern matching, the raison > d??tre > > of pattern synonyms. I'm pretty sure the reason you need > > ScopedTypeVariables for your second example is that GHC only allows > pattern > > signatures with that extension enabled. Once upon a time there was a > > separate PatternSignatures extension, I believe, but not any more. > > Perhaps ScopedTypeVariables by default (or only?) would be relevant for > the revamped[1] Haskell' committee? > > Would there be any actual downsides? It's always seemed to me that > ScopedTypeVariables was the way it should have been from the start > (given perfect foresight). > > Regards, > > [1] http://permalink.gmane.org/gmane.comp.lang.haskell.prime/3965 > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From svenpanne at gmail.com Wed Sep 30 20:59:52 2015 From: svenpanne at gmail.com (Sven Panne) Date: Wed, 30 Sep 2015 22:59:52 +0200 Subject: Typing pattern synonyms In-Reply-To: References: Message-ID: 2015-09-30 20:10 GMT+02:00 David Feuer : > The Eq constraint is needed to support pattern matching, the raison d??tre > of pattern synonyms. > I was just a bit confused by the fact that normally you don't need an 'Eq a' constraint for pattern matching. But looking at the Haskell report, one can see that matching against numeric/char/string literals involves (==), so: \x -> case x of 1234 -> undefined :: (Eq a, Num a) => a -> r This makes sense now, and there is no magic involved. :-] > I'm pretty sure the reason you need ScopedTypeVariables for your second > example is that GHC only allows pattern signatures with that extension > enabled. > My main question is: Why is this the case? Is it really necessary in our simple example? The GHC docs should probably mention the magic behind this in the patter synonym section. > Once upon a time there was a separate PatternSignatures extension, I > believe, but not any more. > Never heard of that one... -------------- next part -------------- An HTML attachment was scrubbed... URL: