[Haskell-cafe] IOSim on Hackage

Fri Aug 25 22:30:46 UTC 2023

A follow-up question: what is the approach with using third-party libraries
that operate in terms of plain IO? Without a MonadIO constraint, lifting is
not possible (even less MonadBaseControl or UnliftIO-style callback
lifting... which is a great thing in UnliftIO by the way).

Given that being able to simulate is a goal, do I assume right that the
approch would be to rewrite those third-party libs in terms of io-classes?
Or maybe to inject some mockable behavior (like a monad-parameterized
Handle pattern, or just plain monadic functions), and use a fake during
simulation, while the true implementation otherwise?

Thanks,
Robin

Robin Palotai <palotai.robin at gmail.com> ezt írta (időpont: 2023. aug. 26.,
Szo, 0:15):

> Thank you Marcin for the overview! The dejafu issue is indeed insightful
> as well.
>
> Let me recap my understanding of unliftio - you are probably aware, but
> let's have it down for future reference.
>
> (Monad)UnliftIO, the typeclass, itself is "just" an IO abstraction similar
> to previous ones like MonadIO or MonadBaseControl, but more restrictive:
> Only those instances are admitted, that play nicely with running them in
> plain IO, also with concurrenty. This in practice mostly means ReaderT IO
> and things isomorphic to it. So excluded is StateT etc - because it gets
> hairy to forkIO a StateT IO computation - the state is practically
> discarded (hm, one might be able to do something funky with async and
> StateT, though it would still operate on some forked state, and one would
> need to manually merge back isn't it..) Anyway UnliftIO's take in this
> case, is to be honest and use explicit mutable state (like MVars) in a
> concurrent setting.
>
> Unliftio, the ecosystem (well, mostly the 'unlftio' package), is akin to
> lifted-base and lifted-whatever, providing operations with UnliftIO
> constraint.
>
> Now, about unliftio and dejafu.. I pondered using them together (some
> thread lingering in https://github.com/barrucadu/dejafu/issues/298). But
> one can't just program against some newtyped monad that has both a
> MonadConc instance (for dejafu's sake) and also an UnliftIO instance (for
> production running), because - and let's shelve API differences - the
> semantics on exception handling and cleanup differs: UnliftIO follows the
> safe-exceptions tradition, and a) the default catch doesn't catch async
> exceptions (so, as long as your own code is concerned, you can promptly
> kill a thread without accidentally catching the async exception with a
> sync-intended handler), also b) the default cleanup actions run under
> uninterruptibleMask (of which there's a mega-issue-thread, but the
> intention is to rather be correct and maybe deadlock, rather than leave
> resources in uncleaned state, in case the cleanup handler gets interrupted
> itself too).*
>
> All in all, dejafu expects (or simulates) MonadConc according to the base
> behavior (unsurprisingly). So MonadConc would need to be banished into some
> internal layer of unliftio (in some unlfitio-concurrency package?), which
> sounds like a rather tedious work (but maybe not impossible).
>
> But having this, at least when unliftio and dejafu is concerned, would be
> desirable - it would be worth a separate story the subtle differences I ran
> into when using unliftio's supposedly "just lifted" versions of some async
> operations, that subtly differ in exception handling / masking behavior
> compared to the original, and can lead to some "interesting" results. Not
> having dejafu available, I have to resort to random threadDelay injections
> when testing, which is not very nice (or fast, or comprehensive).
>
> Having this said, I certainly see the utility of an abstraction that
> provides seamless mocking and testable concurrency. Mockable time sounds
> especially useful. And one, if really wants, can implement the
> safe-exceptions behavior on top of any abstraction (hm.. I guess), and
> avoid using oddly-behaving instances.
>
> I hope one day I can try io-classes / io-sim. Good luck with it & thanks
> again!
>
> *: I'm rather content with unliftio's choices about exception and cleanup
> behavior, just sad it creates such non-uniformity in tooling / mindset
> needed.
>
> <coot at coot.me> ezt írta (időpont: 2023. aug. 25., P, 19:41):
>
>> Hi Robin,
>>
>> Some time ago we received a question about Dejafu on GitHub. Duncan
>> provided a longer answer so let me share the link:
>>
>> https://github.com/input-output-hk/io-sim/issues/53
>> (tldr: `io-sim` simulates time, while dejafu might provide nicer partial
>> order reduction).
>>
>> I haven't thoroughly reviewed `unliftio`, so my answer can only be
>> partial.  My impression is that a subset of `unliftio` can be supported by
>> `io-sim`, but not things that require `OS` support or cross language
>> barrier, e.g. disk access, spawning processes, foreign calls.  With
>> `io-sim` one has to implement such things as `IO` is not available in
>> `IOSim` monad :smile: - but this isn't that hard as it sounds and gives
>> some interesting opportunities. One of teams in IOG simulates file system
>> API with `fs-sim` (https://github.com/input-output-hk/fs-sim) - they're
>> planning to publish it on Hackage.  We implemented the Berkeley Socket
>> interface (actually something a bit more general as we needed to support
>> some Windows stuff too), which we also plan to publish on Hackage at some
>> point.  The advantage of owning an implementation is that one can do
>> failure injection, something that otherwise requires a middleman between
>> the process and OS (or a special kernel driver) - which isn't that easy as
>> writing something in Haskell.
>>
>> To be honest I have too little experience with working with Dejafu to be
>> able to give a very good comparison.  I spent a fair bit of time debugging
>> deadlocks and race conditions with `io-sim`, and that's how `io-sim`
>> support for debugging such bugs got quite decent. In particular all
>> threads, `TVar`'s and `MVar`s can be named, one can emit traces when
>> `TVar`s are committed - still debugging race conditions or deadlocks is fun
>> :wink:.
>>  This is also useful for writing quickcheck property style tests, which
>> have access to internal state changes, which becomes independent of thread
>> scheduling (unlike `putStrLn` or something similar).
>>
>> Two outstanding issues are:
>> * add support for `IORef`s: for `io-sim` this is streightforward, for
>> `io-sim-por` (the partial reduction algorightm we have) probably complex,
>> just because POR is complex in itself;
>> * add support for `MVar` inspection (as we do with `TVar`s; the simulated
>> `MVar`s are implemented with `TVar`s).
>> There are probably some places were we missing some APIs from one of the
>> packages which we want to support (`base`, `async`, `stm` & `time`). We
>> want to keep up with their evolution.  The list of issues is quite short
>> now (we churned quite a number of issues in the past!):
>> https://github.com/orgs/input-output-hk/projects/19/views/24
>>
>>
>>
>> Best regards,
>> Marcin
>>
>> ------- Original Message -------
>> On Monday, August 21st, 2023 at 15:30, Robin Palotai <
>> palotai.robin at gmail.com> wrote:
>>
>>
>> > Nice! Always good to see more production tooling to emerge.
>> > A question, not that it is your responsibility to consider, but in case
>> you have thoughts, would be keen to hear:
>> >
>>
>> > Are there any long term integrative plan on bringing the various
>> Io/exception/concurrency abstractions on a compatible basis? Currently
>> there's for example unliftio/Rio, which is already not compatible with
>> dejafu. Dejafu is class based, but might not play well with these
>> io-classes. Also, the emulated semantics might differ. Etc.
>> >
>>
>> > To a player who wants "just" some robust working ecosystem, all these
>> choices and turbulence is quite hard to navigate or make choices about. And
>> input about observed pros/cons and long term plans appreciated.
>> >
>>
>> > Thank you,
>> > Robin
>> >
>>
>> > On Sun, Aug 20, 2023, 23:02 <coot at coot.me> wrote:
>> >
>>
>> > > Hello dear Haskell-Cafe,
>> > >
>>
>> > > I realised that I haven't shared here that we (the networking team of
>> IOG) released io-sim on Hackage. We found and fixed countless bugs thanks
>> to it, including some concurrent ones. Here's a blog post which I wrote
>> some time ago:
>> > > https://engineering.iog.io/2023-04-14-io-sim-annoucement
>> > >
>>
>> > > https://hackage.haskell.org/package/io-sim
>> > > https://hackage.haskell.org/package/io-classes
>> > >
>>
>> > > Cheers,
>> > > Marcin Szamotulski
>> > > _______________________________________________
>> > > Haskell-Cafe mailing list
>> > > To (un)subscribe, modify options or view archives go to:
>> > > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>> > > Only members subscribed via the mailman list are allowed to post.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20230826/bb13e501/attachment.html>