<div dir="ltr">Hi KC,<div><br></div><div>if blackholes only appear during thunk evaluation, could the problem you describe below be worked around by simply imposing that the scheduler never creates black holes? Say by leveraging GHC's new -XStrict language extension?</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature">--<br>Mathieu Boespflug<br>Founder at <a href="http://tweag.io" target="_blank">http://tweag.io</a>.</div></div>
<br><div class="gmail_quote">On 4 January 2017 at 13:54, Simon Peyton Jones via ghc-devs <span dir="ltr"><<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div class="m_-4941171629830869293WordSection1">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">Reply from KC… see below.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">Simon<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> KC Sivaramakrishnan [mailto:<a href="mailto:sk826@cam.ac.uk" target="_blank">sk826@cam.ac.uk</a>]
<br>
<b>Sent:</b> 04 January 2017 11:29<br>
<b>To:</b> Simon Peyton Jones <<a href="mailto:simonpj@microsoft.com" target="_blank">simonpj@microsoft.com</a>><br>
<b>Cc:</b> Daniel Bennet <<a href="mailto:daniel.bennet83@gmail.com" target="_blank">daniel.bennet83@gmail.com</a>>; <a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>; KC (<a href="mailto:sk826@hermes.cam.ac.uk" target="_blank">sk826@hermes.cam.ac.uk</a>) <<a href="mailto:sk826@hermes.cam.ac.uk" target="_blank">sk826@hermes.cam.ac.uk</a>><br>
<b>Subject:</b> Re: Lightweight Concurrency Branch<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
Hi Simon, David,<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
Indeed as Simon mentioned it, the work was never finished. The implementation is(was) at a stage where we can run some small non-trivial benchmarks (Section 7 of the JFP paper). The interactions with FFI, MVars, STM, asynchronous exceptions worked well (though
we most probably do things a little differently now), until we encountered interaction with blackholes mechanism. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
The crux of the problem is that the blackholing mechanism interacts with the scheduler and if the scheduler functionality itself is written in Haskell, then we have the potential for a deadlock. Some of the details are presented in Section 6.5, but we never
got around to a clean solution that I could get working properly. Hence the reason for not formalizing it completely in the paper, and I am sure there are some edge cases that I hadn't thought about. This is a particularly tricky issue, and the current design
unfortunately does not lend itself to a clean solution.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
Although the asynchronous exception mechanism interaction was not formalized, it works well and passes the tests. Not much effort was put in to make the implementation go particularly fast. While the performance was comparable on average, but varied quite a
bit on edge cases; some of the results in the paper show this. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
If one were to revive the project, I would suggest starting from the design, using the existing code as a prototype, but write code from scratch; pleasantly there isn't much new code in this branch. The project does need substantial amount of work to make it
upstream with the newer RTS mechanisms. I am very happy to provide more details and eager to assist with the work, but my time commitments mean that I cannot lead this effort.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
Kind Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
KC<u></u><u></u></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
<u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6.0pt;margin-left:36.0pt">
On Wed, Jan 4, 2017 at 10:38 AM, Simon Peyton Jones <<a href="mailto:simonpj@microsoft.com" target="_blank">simonpj@microsoft.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">David</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">KC never finished work on this stuff. I’m copying him because I’m sure he’d be happy to help.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">KC: can you summarise where you left it?</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">I think it’s very interesting work, and has the potential to make GHC’s RTS much more malleable, by moving more of it into Haskell libraries instead of deeply-magic C code.
</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">But it’s not easy, because we are reluctant to lose performance, and because there are interactions with STM, weak pointers, foreign function calls, etc. I think it’d require a bit of commitment to make a go
of it.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif">Simon</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt">
<span style="font-family:"Calibri",sans-serif"> </span><u></u><u></u></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt">
<b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> ghc-devs [mailto:<a href="mailto:ghc-devs-bounces@haskell.org" target="_blank">ghc-devs-bounces@<wbr>haskell.org</a>]
<b>On Behalf Of </b>Daniel Bennet<br>
<b>Sent:</b> 28 December 2016 17:23<br>
<b>To:</b> <a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a><br>
<b>Subject:</b> Lightweight Concurrency Branch</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
The lightweight concurrency branch is highly interesting and relevant to my interests, however, the ghc-lwc2 branch hasn't been updated in several years even though it's listed as an active branch at
<a href="https://ghc.haskell.org/trac/ghc/wiki/ActiveBranches" target="_blank">https://ghc.haskell.org/trac/<wbr>ghc/wiki/ActiveBranches</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
The wiki page for the work hasn't been updated in almost two years either, <a href="https://ghc.haskell.org/trac/ghc/wiki/LightweightConcurrency" target="_blank">
https://ghc.haskell.org/trac/<wbr>ghc/wiki/<wbr>LightweightConcurrency</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
Relevant papers:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
Composable Scheduler Activations for Haskell (2014)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftimharris.uk%2Fpapers%2F2014-composable-tr.pdf&data=02%7C01%7Csimonpj%40microsoft.com%7Cd132a48641a54f732ffc08d42f463e8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636185426136080161&sdata=YG%2B0xfi%2Bd74Vi9GBI8iTrr4GEMzjS2WwrI182%2BcEgqM%3D&reserved=0" target="_blank">https://timharris.uk/papers/<wbr>2014-composable-tr.pdf</a> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
Composable Scheduler Activations for Haskell (2016)<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fkcsrk.info%2Fpapers%2Fschedact_jfp16.pdf&data=02%7C01%7Csimonpj%40microsoft.com%7Cd132a48641a54f732ffc08d42f463e8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636185426136080161&sdata=mH79Y3U%2BdDvcw2PA8lGcFaPMJw6DgCIXbu%2F3WGvyGaU%3D&reserved=0" target="_blank">http://kcsrk.info/papers/<wbr>schedact_jfp16.pdf</a><u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">
What remains for integrating this branch into GHC?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a><br>
<a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs" rel="noreferrer" target="_blank">http://mail.haskell.org/cgi-<wbr>bin/mailman/listinfo/ghc-devs</a><br>
<br></blockquote></div><br></div>