[Haskell-cafe] IOSim on Hackage

Sat Aug 26 10:58:52 UTC 2023

If it comes to exception handling, we follow `base` rather than `safe-execeptions` package (there's a trade-off between async exceptions vs deadlocks, as safe exceptions is using `uninterruptibleMask`).  But one can always create a type class which provide the same semantics as `safe-exceptions` do (think of `io-classes` as a description of interface which `base` provides).

We provide instances for `ReaderT` transformer in `io-classes`,  instances for some other transformers are provided in a more experimental package `io-classes-mtl` (we don't use it in production).

> All in all, dejafu expects (or simulates) MonadConc according to > the base behavior (unsurprisingly). So MonadConc would need to be > banished into some internal layer of unliftio (in some unlfitio-concurrency package?), which sounds like a rather tedious work (but > maybe not impossible).

Some monad transformers are not really well suited for concurrency, e.g. `StateT` is one example.  One would need a strategy for merging state when something like `concurrently` finishes.  There's `lifted-async` on Hackage, but we were not convinced that it's the right choice. I did experiment with this a bit, but unfortunately I wasn't able to find the branch now.

> A follow-up question: what is the approach with using third-party libraries that operate in terms of plain IO?
One would need to generalise the library to be polymorphic over `IO`.  One example we have is the `MonadST` (https://hackage.haskell.org/package/io-classes-1.2.0.0/docs/Control-Monad-Class-MonadST.html#t:MonadST).  So for MonadUnliftIO this could be:
```
class MonadUnliftIO m io where ...
```
where `m` has the same meaning but `io` is either `IO` or `IOSim`; Alternatively `io` could be captured as an associated type family.

Cheers,
Marcin

Sent with Proton Mail secure email.

------- Original Message -------
On Saturday, August 26th, 2023 at 00:30, Robin Palotai <palotai.robin at gmail.com> wrote:

> A follow-up question: what is the approach with using third-party libraries that operate in terms of plain IO? Without a MonadIO constraint, lifting is not possible (even less MonadBaseControl or UnliftIO-style callback lifting... which is a great thing in UnliftIO by the way).
> Given that being able to simulate is a goal, do I assume right that the approch would be to rewrite those third-party libs in terms of io-classes? Or maybe to inject some mockable behavior (like a monad-parameterized Handle pattern, or just plain monadic functions), and use a fake during simulation, while the true implementation otherwise?
> 

> Thanks,
> Robin
> 

> Robin Palotai <palotai.robin at gmail.com> ezt írta (időpont: 2023. aug. 26., Szo, 0:15):
> 

> > Thank you Marcin for the overview! The dejafu issue is indeed insightful as well.
> > Let me recap my understanding of unliftio - you are probably aware, but let's have it down for future reference.
> > 

> > (Monad)UnliftIO, the typeclass, itself is "just" an IO abstraction similar to previous ones like MonadIO or MonadBaseControl, but more restrictive: Only those instances are admitted, that play nicely with running them in plain IO, also with concurrenty. This in practice mostly means ReaderT IO and things isomorphic to it. So excluded is StateT etc - because it gets hairy to forkIO a StateT IO computation - the state is practically discarded (hm, one might be able to do something funky with async and StateT, though it would still operate on some forked state, and one would need to manually merge back isn't it..) Anyway UnliftIO's take in this case, is to be honest and use explicit mutable state (like MVars) in a concurrent setting.
> > 

> > Unliftio, the ecosystem (well, mostly the 'unlftio' package), is akin to lifted-base and lifted-whatever, providing operations with UnliftIO constraint.
> > 

> > Now, about unliftio and dejafu.. I pondered using them together (some thread lingering in https://github.com/barrucadu/dejafu/issues/298). But one can't just program against some newtyped monad that has both a MonadConc instance (for dejafu's sake) and also an UnliftIO instance (for production running), because - and let's shelve API differences - the semantics on exception handling and cleanup differs: UnliftIO follows the safe-exceptions tradition, and a) the default catch doesn't catch async exceptions (so, as long as your own code is concerned, you can promptly kill a thread without accidentally catching the async exception with a sync-intended handler), also b) the default cleanup actions run under uninterruptibleMask (of which there's a mega-issue-thread, but the intention is to rather be correct and maybe deadlock, rather than leave resources in uncleaned state, in case the cleanup handler gets interrupted itself too).*
> > 

> > All in all, dejafu expects (or simulates) MonadConc according to the base behavior (unsurprisingly). So MonadConc would need to be banished into some internal layer of unliftio (in some unlfitio-concurrency package?), which sounds like a rather tedious work (but maybe not impossible).
> > 

> > But having this, at least when unliftio and dejafu is concerned, would be desirable - it would be worth a separate story the subtle differences I ran into when using unliftio's supposedly "just lifted" versions of some async operations, that subtly differ in exception handling / masking behavior compared to the original, and can lead to some "interesting" results. Not having dejafu available, I have to resort to random threadDelay injections when testing, which is not very nice (or fast, or comprehensive).
> > 

> > Having this said, I certainly see the utility of an abstraction that provides seamless mocking and testable concurrency. Mockable time sounds especially useful. And one, if really wants, can implement the safe-exceptions behavior on top of any abstraction (hm.. I guess), and avoid using oddly-behaving instances.
> > 

> > I hope one day I can try io-classes / io-sim. Good luck with it & thanks again!
> > 

> > *: I'm rather content with unliftio's choices about exception and cleanup behavior, just sad it creates such non-uniformity in tooling / mindset needed.
> > 

> > <coot at coot.me> ezt írta (időpont: 2023. aug. 25., P, 19:41):
> > 

> > > Hi Robin,
> > > 

> > > Some time ago we received a question about Dejafu on GitHub. Duncan provided a longer answer so let me share the link:
> > > 

> > > https://github.com/input-output-hk/io-sim/issues/53
> > > (tldr: `io-sim` simulates time, while dejafu might provide nicer partial order reduction).
> > > 

> > > I haven't thoroughly reviewed `unliftio`, so my answer can only be partial. My impression is that a subset of `unliftio` can be supported by `io-sim`, but not things that require `OS` support or cross language barrier, e.g. disk access, spawning processes, foreign calls. With `io-sim` one has to implement such things as `IO` is not available in `IOSim` monad :smile: - but this isn't that hard as it sounds and gives some interesting opportunities. One of teams in IOG simulates file system API with `fs-sim` (https://github.com/input-output-hk/fs-sim) - they're planning to publish it on Hackage. We implemented the Berkeley Socket interface (actually something a bit more general as we needed to support some Windows stuff too), which we also plan to publish on Hackage at some point. The advantage of owning an implementation is that one can do failure injection, something that otherwise requires a middleman between the process and OS (or a special kernel driver) - which isn't that easy as writing something in Haskell.
> > > 

> > > To be honest I have too little experience with working with Dejafu to be able to give a very good comparison. I spent a fair bit of time debugging deadlocks and race conditions with `io-sim`, and that's how `io-sim` support for debugging such bugs got quite decent. In particular all threads, `TVar`'s and `MVar`s can be named, one can emit traces when `TVar`s are committed - still debugging race conditions or deadlocks is fun :wink:.
> > > This is also useful for writing quickcheck property style tests, which have access to internal state changes, which becomes independent of thread scheduling (unlike `putStrLn` or something similar).
> > > 

> > > Two outstanding issues are:
> > > * add support for `IORef`s: for `io-sim` this is streightforward, for `io-sim-por` (the partial reduction algorightm we have) probably complex, just because POR is complex in itself;
> > > * add support for `MVar` inspection (as we do with `TVar`s; the simulated `MVar`s are implemented with `TVar`s).
> > > There are probably some places were we missing some APIs from one of the packages which we want to support (`base`, `async`, `stm` & `time`). We want to keep up with their evolution. The list of issues is quite short now (we churned quite a number of issues in the past!):
> > > https://github.com/orgs/input-output-hk/projects/19/views/24
> > > 

> > > 

> > > 

> > > Best regards,
> > > Marcin
> > > 

> > > ------- Original Message -------
> > > On Monday, August 21st, 2023 at 15:30, Robin Palotai <palotai.robin at gmail.com> wrote:
> > > 

> > > 

> > > > Nice! Always good to see more production tooling to emerge.
> > > > A question, not that it is your responsibility to consider, but in case you have thoughts, would be keen to hear:
> > > >
> > > 

> > > > Are there any long term integrative plan on bringing the various Io/exception/concurrency abstractions on a compatible basis? Currently there's for example unliftio/Rio, which is already not compatible with dejafu. Dejafu is class based, but might not play well with these io-classes. Also, the emulated semantics might differ. Etc.
> > > >
> > > 

> > > > To a player who wants "just" some robust working ecosystem, all these choices and turbulence is quite hard to navigate or make choices about. And input about observed pros/cons and long term plans appreciated.
> > > >
> > > 

> > > > Thank you,
> > > > Robin
> > > >
> > > 

> > > > On Sun, Aug 20, 2023, 23:02 <coot at coot.me> wrote:
> > > >
> > > 

> > > > > Hello dear Haskell-Cafe,
> > > > >
> > > 

> > > > > I realised that I haven't shared here that we (the networking team of IOG) released io-sim on Hackage. We found and fixed countless bugs thanks to it, including some concurrent ones. Here's a blog post which I wrote some time ago:
> > > > > https://engineering.iog.io/2023-04-14-io-sim-annoucement
> > > > >
> > > 

> > > > > https://hackage.haskell.org/package/io-sim
> > > > > https://hackage.haskell.org/package/io-classes
> > > > >
> > > 

> > > > > Cheers,
> > > > > Marcin Szamotulski
> > > > > _______________________________________________
> > > > > Haskell-Cafe mailing list
> > > > > To (un)subscribe, modify options or view archives go to:
> > > > > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> > > > > Only members subscribed via the mailman list are allowed to post.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 509 bytes
Desc: OpenPGP digital signature
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20230826/9134b1fa/attachment-0001.sig>