[Haskell-cafe] IOSim on Hackage

Fri Aug 25 22:15:57 UTC 2023

Thank you Marcin for the overview! The dejafu issue is indeed insightful as
well.

Let me recap my understanding of unliftio - you are probably aware, but
let's have it down for future reference.

(Monad)UnliftIO, the typeclass, itself is "just" an IO abstraction similar
to previous ones like MonadIO or MonadBaseControl, but more restrictive:
Only those instances are admitted, that play nicely with running them in
plain IO, also with concurrenty. This in practice mostly means ReaderT IO
and things isomorphic to it. So excluded is StateT etc - because it gets
hairy to forkIO a StateT IO computation - the state is practically
discarded (hm, one might be able to do something funky with async and
StateT, though it would still operate on some forked state, and one would
need to manually merge back isn't it..) Anyway UnliftIO's take in this
case, is to be honest and use explicit mutable state (like MVars) in a
concurrent setting.

Unliftio, the ecosystem (well, mostly the 'unlftio' package), is akin to
lifted-base and lifted-whatever, providing operations with UnliftIO
constraint.

Now, about unliftio and dejafu.. I pondered using them together (some
thread lingering in https://github.com/barrucadu/dejafu/issues/298). But
one can't just program against some newtyped monad that has both a
MonadConc instance (for dejafu's sake) and also an UnliftIO instance (for
production running), because - and let's shelve API differences - the
semantics on exception handling and cleanup differs: UnliftIO follows the
safe-exceptions tradition, and a) the default catch doesn't catch async
exceptions (so, as long as your own code is concerned, you can promptly
kill a thread without accidentally catching the async exception with a
sync-intended handler), also b) the default cleanup actions run under
uninterruptibleMask (of which there's a mega-issue-thread, but the
intention is to rather be correct and maybe deadlock, rather than leave
resources in uncleaned state, in case the cleanup handler gets interrupted
itself too).*

All in all, dejafu expects (or simulates) MonadConc according to the base
behavior (unsurprisingly). So MonadConc would need to be banished into some
internal layer of unliftio (in some unlfitio-concurrency package?), which
sounds like a rather tedious work (but maybe not impossible).

But having this, at least when unliftio and dejafu is concerned, would be
desirable - it would be worth a separate story the subtle differences I ran
into when using unliftio's supposedly "just lifted" versions of some async
operations, that subtly differ in exception handling / masking behavior
compared to the original, and can lead to some "interesting" results. Not
having dejafu available, I have to resort to random threadDelay injections
when testing, which is not very nice (or fast, or comprehensive).

Having this said, I certainly see the utility of an abstraction that
provides seamless mocking and testable concurrency. Mockable time sounds
especially useful. And one, if really wants, can implement the
safe-exceptions behavior on top of any abstraction (hm.. I guess), and
avoid using oddly-behaving instances.

I hope one day I can try io-classes / io-sim. Good luck with it & thanks
again!

*: I'm rather content with unliftio's choices about exception and cleanup
behavior, just sad it creates such non-uniformity in tooling / mindset
needed.

<coot at coot.me> ezt írta (időpont: 2023. aug. 25., P, 19:41):

> Hi Robin,
>
> Some time ago we received a question about Dejafu on GitHub. Duncan
> provided a longer answer so let me share the link:
>
> https://github.com/input-output-hk/io-sim/issues/53
> (tldr: `io-sim` simulates time, while dejafu might provide nicer partial
> order reduction).
>
> I haven't thoroughly reviewed `unliftio`, so my answer can only be
> partial.  My impression is that a subset of `unliftio` can be supported by
> `io-sim`, but not things that require `OS` support or cross language
> barrier, e.g. disk access, spawning processes, foreign calls.  With
> `io-sim` one has to implement such things as `IO` is not available in
> `IOSim` monad :smile: - but this isn't that hard as it sounds and gives
> some interesting opportunities. One of teams in IOG simulates file system
> API with `fs-sim` (https://github.com/input-output-hk/fs-sim) - they're
> planning to publish it on Hackage.  We implemented the Berkeley Socket
> interface (actually something a bit more general as we needed to support
> some Windows stuff too), which we also plan to publish on Hackage at some
> point.  The advantage of owning an implementation is that one can do
> failure injection, something that otherwise requires a middleman between
> the process and OS (or a special kernel driver) - which isn't that easy as
> writing something in Haskell.
>
> To be honest I have too little experience with working with Dejafu to be
> able to give a very good comparison.  I spent a fair bit of time debugging
> deadlocks and race conditions with `io-sim`, and that's how `io-sim`
> support for debugging such bugs got quite decent. In particular all
> threads, `TVar`'s and `MVar`s can be named, one can emit traces when
> `TVar`s are committed - still debugging race conditions or deadlocks is fun
> :wink:.
>  This is also useful for writing quickcheck property style tests, which
> have access to internal state changes, which becomes independent of thread
> scheduling (unlike `putStrLn` or something similar).
>
> Two outstanding issues are:
> * add support for `IORef`s: for `io-sim` this is streightforward, for
> `io-sim-por` (the partial reduction algorightm we have) probably complex,
> just because POR is complex in itself;
> * add support for `MVar` inspection (as we do with `TVar`s; the simulated
> `MVar`s are implemented with `TVar`s).
> There are probably some places were we missing some APIs from one of the
> packages which we want to support (`base`, `async`, `stm` & `time`). We
> want to keep up with their evolution.  The list of issues is quite short
> now (we churned quite a number of issues in the past!):
> https://github.com/orgs/input-output-hk/projects/19/views/24
>
>
>
> Best regards,
> Marcin
>
> ------- Original Message -------
> On Monday, August 21st, 2023 at 15:30, Robin Palotai <
> palotai.robin at gmail.com> wrote:
>
>
> > Nice! Always good to see more production tooling to emerge.
> > A question, not that it is your responsibility to consider, but in case
> you have thoughts, would be keen to hear:
> >
>
> > Are there any long term integrative plan on bringing the various
> Io/exception/concurrency abstractions on a compatible basis? Currently
> there's for example unliftio/Rio, which is already not compatible with
> dejafu. Dejafu is class based, but might not play well with these
> io-classes. Also, the emulated semantics might differ. Etc.
> >
>
> > To a player who wants "just" some robust working ecosystem, all these
> choices and turbulence is quite hard to navigate or make choices about. And
> input about observed pros/cons and long term plans appreciated.
> >
>
> > Thank you,
> > Robin
> >
>
> > On Sun, Aug 20, 2023, 23:02 <coot at coot.me> wrote:
> >
>
> > > Hello dear Haskell-Cafe,
> > >
>
> > > I realised that I haven't shared here that we (the networking team of
> IOG) released io-sim on Hackage. We found and fixed countless bugs thanks
> to it, including some concurrent ones. Here's a blog post which I wrote
> some time ago:
> > > https://engineering.iog.io/2023-04-14-io-sim-annoucement
> > >
>
> > > https://hackage.haskell.org/package/io-sim
> > > https://hackage.haskell.org/package/io-classes
> > >
>
> > > Cheers,
> > > Marcin Szamotulski
> > > _______________________________________________
> > > Haskell-Cafe mailing list
> > > To (un)subscribe, modify options or view archives go to:
> > > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> > > Only members subscribed via the mailman list are allowed to post.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20230826/9d11bcce/attachment.html>