[Haskell-cafe] Testing Msgpack system

David Turner dct25-561bs at mythic-beasts.com
Thu Jun 16 07:21:37 UTC 2016


That's very kind of you, thanks. Not sure I deserve it, but glad I could
help. I'll look forward to the video.

One of my colleagues stumbled across this page which also suggests
injecting the current time as a periodic Tick message rather than
sprinkling timeouts throughout your code:

http://yager.io/Distributed/Distributed.html

Cheers,

David
On 15 Jun 2016 23:18, "Jonn Mostovoy" <jm at memorici.de> wrote:

> Dear David,
>
> we were inspired by your suggestions and our employee, Konstantin
> Ivanov, implemented a very elegant emulator, which we will eventually
> carry out into a separate library.
>
> While presenting the project at Blockchain Summer School, we have
> credited you for pointing us in the right direction. Konstantin, of
> course, was credited for implementation of the solution. In case you
> are interested, video of the talk will be available in late June /
> early July.
>
> Entire source tree is public and hosted on Github:
> https://github.com/input-output-hk/rscoin-haskell
>
> Thank you once again for inspiration!
>
> On Apr 1, 2016 6:57 PM, "Jonn Mostovoy" <jm at memorici.de> wrote:
> >
> > Dear David,
> >
> > thank you very much! Your answers were extremely insightful.
> >
> > To adapt this approach we will need to refactor our types quite a bit,
> but that will be well worth it.
> >
> > —
> > Kindest regards,
> > jm
> >
> > On Mar 31, 2016 10:10 AM, "David Turner" <dct25-561bs at mythic-beasts.com>
> wrote:
> >>
> >> Hi Jonn,
> >>
> >> I work on a similar-sounding system. We have arranged things so that
> each node is a pure state machine, with outputs that are a pure function of
> its inputs, with separate (and simple, obviously correct) machinery for
> connecting these state machines over the network. This makes it rather
> simple to run a bunch of these state machines in a test harness that
> simulates all sorts of network nastiness (disconnections, dropped or
> reordered messages, delays, corruption etc.) on a single thread.
> >>
> >> One trick that proved useful was to feed in the current time as an
> explicit input message. This makes it possible to test things like timeouts
> without having to actually wait for the time to pass, which speeds things
> up immensely. We also make use of ContT somewhere in the tests to
> interleave processing and assertions, and to define a 'hypothetically'
> operator that lets a test run a sequence of actions and then backtrack.
> >>
> >> I think this idea was inspired by
> https://github.com/NicolasT/paxos/blob/master/bin/synod.hs, at least the
> network nastiness simulator thing. He uses threads for that demo but the
> nodes' behaviour itself is pure:
> https://github.com/NicolasT/paxos/blob/master/src/Network/Paxos/Synod/Proposer.hs
> for example.
> >>
> >> We also have proved certain key properties of the network are implied
> by certain local invariants, which reduces the testing problem down to one
> of checking properties on each node separately. This was time consuming,
> but highlighted certain important corner cases that it's unlikely we would
> have found by random testing.
> >>
> >> If you're interested in Byzantine behaviour (the 'evil node' test) then
> you may enjoy reading James Mickens' article on the subject:
> https://www.usenix.org/publications/login-logout/may-2013/saddest-moment
> >>
> >> Hope that helps,
> >>
> >> David
> >>
> >> PS a double apology: firstly for the double message (my first attempt
> was sent from the wrong address) and secondly for spelling your name wrong
> in that message!
> >>
> >>
> >> On 31 March 2016 at 00:41, Jonn Mostovoy <jm at memorici.de> wrote:
> >>>
> >>> Dear friends,
> >>>
> >>> we have a distributed system written in Haskell, consisting of three
> >>> types of nodes with dozen of instances of each of two types and a
> >>> central node of the third type.
> >>>
> >>> Each node is started by executing a binary which sets up acid-state
> >>> persistence layer and sockets over which msgpack messages are sent
> >>> around.
> >>>
> >>> It is trivial to write low level functionality quickcheck test suites,
> >>> which test properties of functions.
> >>>
> >>> We would like, however, to have a quickcheck-esque suite which sets up
> >>> the network, then gets it to an arbitrary valid state (sending correct
> >>> messages between nodes), then rigorously testing it for three
> >>> high-level properties:
> >>>
> >>> 1. Chaos monkey test (disable random node, see if certain invariants
> hold);
> >>> 2. Evil node test (make several nodes work against the system, see if
> >>> certain properties hold);
> >>> 3. Rigorous testing of network-wide invariants, if all the nodes
> >>> operate correctly.
> >>>
> >>> The problem we're facing is the following — if we want to inspect
> >>> state of nodes in Haskell-land, we have to write a huge machinery
> >>> which emulates every launch of node via thread. There will be bugs in
> >>> this machinery, so we won't be able to get quality testing information
> >>> before we fix those; if we want to run things as processes, then the
> >>> best thing we can do is to inspect either acid-state dbs of each node
> >>> (it poses resource locking problems and forces us to dump the state on
> >>> every change, which is undesirable), or make an observer node, which
> >>> dumps the consensus as Text and then parsing the data into Haskell
> >>> terms, making decisions about the required properties based on that
> >>> (so far, it feels like the best option).
> >>>
> >>> Am I missing something? How is something like this achieved in
> >>> culture? How would you approach such a problem?
> >>>
> >>> Links to source files of test suites which do something similar are
> >>> highly appreciated.
> >>> _______________________________________________
> >>> Haskell-Cafe mailing list
> >>> Haskell-Cafe at haskell.org
> >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160616/094e33ac/attachment.html>


More information about the Haskell-Cafe mailing list