Lazy ST vs concurrency

[David Feuer]

I forgot to CC ghc-devs the first time, so here's another copy.

I was working on #11760 this weekend, which has to do with concurrency 
breaking lazy ST. I came up with what I thought was a pretty decent solution ( 
https://phabricator.haskell.org/D3038 ). Simon Peyton Jones, however, is quite 
unhappy about the idea of sticking this weird unsafePerformIO-like code 
(noDup, which I originally implemented as (unsafePerformIO . evaluate), but 
which he finds ugly regardless of the details) into fmap and (>>=).  He's also 
concerned that the noDuplicate# applications will kill performance in the 
multi-threaded case, and suggests he would rather leave lazy ST broken, or 
even remove it altogether, than use a fix that will make it slow sometimes, 
particularly since there haven't been a lot of reports of problems in the 
wild.

My view is that leaving it broken, even if it only causes trouble 
occasionally, is simply not an option. If users can't rely on it to always 
give correct answers, then it's effectively useless. And for the sake of 
backwards compatibility, I think it's a lot better to keep it around, even if 
it runs slowly multithreaded, than to remove it altogether.

Note to Simon PJ: Yes, it's ugly to stick that noDup in there. But lazy ST has 
always been a bit of deep magic. You can't *really* carry a moment of time 
around in your pocket and make its history happen only if necessary. We can 
make it work in GHC because its execution model is entirely based around graph 
reduction, so evaluation is capable of driving execution. Whereas lazy IO is 
extremely tricky because it causes effects observable in the real world, lazy 
ST is only *moderately* tricky, causing effects that we have to make sure 
don't lead to weird interactions between threads. I don't think it's terribly 
surprising that it needs to do a few more weird things to work properly.

David