[Haskell-cafe] Testing Msgpack system

David Turner dct25-561bs at mythic-beasts.com
Thu Mar 31 08:10:46 UTC 2016

Hi Jonn,

I work on a similar-sounding system. We have arranged things so that each
node is a pure state machine, with outputs that are a pure function of its
inputs, with separate (and simple, obviously correct) machinery for
connecting these state machines over the network. This makes it rather
simple to run a bunch of these state machines in a test harness that
simulates all sorts of network nastiness (disconnections, dropped or
reordered messages, delays, corruption etc.) on a single thread.

One trick that proved useful was to feed in the current time as an explicit
input message. This makes it possible to test things like timeouts without
having to actually wait for the time to pass, which speeds things up
immensely. We also make use of ContT somewhere in the tests to interleave
processing and assertions, and to define a 'hypothetically' operator that
lets a test run a sequence of actions and then backtrack.

I think this idea was inspired by
https://github.com/NicolasT/paxos/blob/master/bin/synod.hs, at least the
network nastiness simulator thing. He uses threads for that demo but the
nodes' behaviour itself is pure:

We also have proved certain key properties of the network are implied by
certain local invariants, which reduces the testing problem down to one of
checking properties on each node separately. This was time consuming, but
highlighted certain important corner cases that it's unlikely we would have
found by random testing.

If you're interested in Byzantine behaviour (the 'evil node' test) then you
may enjoy reading James Mickens' article on the subject:

Hope that helps,


PS a double apology: firstly for the double message (my first attempt was
sent from the wrong address) and secondly for spelling your name wrong in
that message!

On 31 March 2016 at 00:41, Jonn Mostovoy <jm at memorici.de> wrote:

> Dear friends,
> we have a distributed system written in Haskell, consisting of three
> types of nodes with dozen of instances of each of two types and a
> central node of the third type.
> Each node is started by executing a binary which sets up acid-state
> persistence layer and sockets over which msgpack messages are sent
> around.
> It is trivial to write low level functionality quickcheck test suites,
> which test properties of functions.
> We would like, however, to have a quickcheck-esque suite which sets up
> the network, then gets it to an arbitrary valid state (sending correct
> messages between nodes), then rigorously testing it for three
> high-level properties:
> 1. Chaos monkey test (disable random node, see if certain invariants hold);
> 2. Evil node test (make several nodes work against the system, see if
> certain properties hold);
> 3. Rigorous testing of network-wide invariants, if all the nodes
> operate correctly.
> The problem we're facing is the following — if we want to inspect
> state of nodes in Haskell-land, we have to write a huge machinery
> which emulates every launch of node via thread. There will be bugs in
> this machinery, so we won't be able to get quality testing information
> before we fix those; if we want to run things as processes, then the
> best thing we can do is to inspect either acid-state dbs of each node
> (it poses resource locking problems and forces us to dump the state on
> every change, which is undesirable), or make an observer node, which
> dumps the consensus as Text and then parsing the data into Haskell
> terms, making decisions about the required properties based on that
> (so far, it feels like the best option).
> Am I missing something? How is something like this achieved in
> culture? How would you approach such a problem?
> Links to source files of test suites which do something similar are
> highly appreciated.
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160331/0d5375a2/attachment.html>

More information about the Haskell-Cafe mailing list