[Haskell-cafe] Type System vs Test Driven Development

[Evan Laforge]

On Wed, Jan 5, 2011 at 7:31 PM, Chung-chieh Shan
<ccshan at post.harvard.edu> wrote:
> Besides those example inputs and expected outputs, what about:
> If two signals are (in)compatible then after applying some simple
> transformations to both they remain (in)compatible?  A certain family of
> signals is always compatible with another family of signals?  Silence is
> compatible with every signal?  Every non-silent signal is (in)compatible
> with itself (perhaps after applying a transformation)?

Well, signals are never transformed.  Silence is, in fact, not
specially compatible.  The most I can say is that signals that don't
overlap are always compatible.  So you're correct in that it's
possible to extract properties.  However, this particular property,
being simple, is also expressed in a simple way directly in the code,
so past a couple tests to make sure I didn't reverse any (>)s, I don't
feel like it needs the exhaustive testing that quickcheck brings to
bear.  And basically it's just reimplementing a part of the original
function, in this case the first guard... I suppose you could say if I
typoed the (>)s in the original definition, maybe I won't in the test
version.  But this is too low level, what I care about is if the whole
thing has the conceptually simple but computationally complex result
that I expect.  The interesting bug is when the first guard shadows an
exception later on, so it turns out it's *not* totally true that
non-overlapping signals must be compatible, or maybe my definition of
"overlapping" is not sufficiently defined, or defined different ways
in different places, or needs to be adjusted, or....  I suppose input
fuzzing should be able to flush out things like fuzzy definitions of
overlapping

I can also say weak things about complex outputs, that they will be
returned in sorted order, that they won't overlap, etc.  But I those
are rarely the interesting complicated things that I really want to
test.  Even my "signal compatibility" example is relatively amenable
to extracting properties, picking some other examples:

- Having a certain kind of syntax error will result in a certain kind
of error msg, and surrounding expressions will continue to be included
in the output.  The error msg will include the proper location.  So
I'd need an Arbitrary to generate the right structure with an error or
two and then have code to figure out the reported location from the
location in the data structure, and debug all that.  There's actually
a fair amount of stuff that wants to look for a log msg, like "hit a
cache, the only sign of which is a log msg of a certain format".
Certainly caches can be tested by asserting that you get the same
results with the cache turned off, that's an easy property.

- Hitting a certain key sequence results in certain data being entered
in the UI.  There's nothing particularly "property" like about this,
it's too ad-hoc...  this seems to apply for all UI-level tests.

- There's also a large class of "integration" type tests: I've tested
the signal compatibility function separately, but the final proof is
that the high-level user input of this shape results in this output,
due to do signal compatibility.  These are the ones whose failure is
the most valuable because they test the emergent behaviour of a set of
interacting systems, and that's ultimately the user-visible behaviour
and also the place where the results are the most subtle.  But those
are also the ones that have huge state spaces and, similar to the UI
tests, basically ad-hoc relationships between input and output.

- Testing for laziness of course doesn't work either.  Or timed
things.  As far as performance goes, some can be tested with tests
("taking the first output doesn't force the entire input" or "a new
key cancels the old threads and starts new ones") but some must be
tested with profiling and eyeballing the results.

QuickCheck seems to fit well when you have small input and output
spaces, but complicated stuff in the middle, but still simple
relations between the input and output.  I think that's why data
structures are so easy to QuickCheck.  I suppose I should look around
for more use of QuickCheck for non-data structures... the examples
I've seen have been trivial stuff like 'reverse . reverse = id'.