working on a fix for ticket #212

[Duncan Coutts]

On Wed, 2009-05-13 at 22:32 +0200, Laszlo Nagy wrote:
> Hi All,
> 
> Trying to create unit test suite.. work in progress. Comments are more
> than welcome!
> 
> http://sites.google.com/site/rizsotto/patches/cabal-ticket-212-v1.darcs-send

Hi Laszlo,

I'd like to get your tests integrated, in addition to the ones Stephen
added recently. We can sort out the framework to use after getting them
in. I expect Stephen is right that in the end we should use
test-framework so that we can handle QC and HUnit based tests. It also
looks like a reasonable framework generally.

However the first thing to do is to get your existing tests in. Can I
ask you to make a couple minor organisational changes?

Instead of one massive tests/Test/Properties.hs file containing all the
tests for the Distribution/* modules, could we use one module of
properties for each module we're testing?

So for example you integrated the existing
tests/Test/Distribution/Version.hs
into your 
tests/Test/Properties.hs
but perhaps we could do the reverse and split it out into
tests/Test/Distribution/License.hs
tests/Test/Distribution/Package.hs
tests/Test/Distribution/Compiler.hs
tests/Test/Distribution/System.hs
tests/Test/Distribution/ModuleName.hs
tests/Test/Distribution/PackageDescription.hs

That would make it much easier for someone to go and find the properties
they're looking for. I realise that many tests will share the same
utility functions and generators etc. That's fine. For example the 
Test.Distribution.System module would define the generator for the OS
and Arch types which are needed eg in
Test.Distribution.PackageDescription. So just import one in the other.
For things which do not belong to a specific module, we can just use
shared utility modules like the existing
tests/Test/QuickCheck/Utils.hs
or any other name that seems appropriate.

> I wrote simple properties like this, for all data type
> 
> > prop_parse x = x == parse $ display x
> 
> I did merged the existing QuickCheck test cases into one file. I found
> it easier to reuse the generator functions for different types.
> 
> My findings:
>  - The System.Platform did not passed parse tests. The reason: both
> System.OS and System.Arch allowed the dash character in the name. But
> System.Platform text representation was concatenate the two strings
> separated by a dash. So I just simple removed the dash from them.
> Solved! ;)

Great. Could that be made into a separate patch from the addition of the
tests? The commit message can say exactly what you just said in
explanation of the change.

> - VersionRange does not passes parse tests, iff more than two ranges
> is present. As far as I could see, the parse method is wired to handle
> only two ranges. Not corrected, because found no good solution. Need
> help on this!

Right, currently the parser only allows one conjunction and no brackets.
There's an open ticket to allow more general expressions.
http://hackage.haskell.org/trac/hackage/ticket/484
For the moment then it's ok for this test to fail. We can expect it'll
be fixed when ticket #484 is completed.

> - Package.Dependency also not passes parse tests. (It is depend on
> VersionRange success.)

Right.

> - Compat.ReadR gather property is obsolete and does not compile. Can't
> figure it out how could I rewrite it. Need help!

Easy solution! The gather function is not used. We can just remove
it :-)

> - PackageDescription does not passes simple parse test. I guess the
> problem is with the test itself! ;) I used showPackageDescription
> method to generate string representation of this type and parse it
> back. I could not judge it is an expected behaviour or a bug. Need
> some explanation.

The showPackageDescription is totally borked. It really doesn't work at
all. Again it's ok for the moment if this test just fails. I'll be
motivation for someone to fix showPackageDescription.

> My problem is mainly how to run these tests?

Lets get the tests in first and then sort this out.

> As unit tests they are pure, does not manipulate any files on disks,
> does not open socket or anything. (Michael Feathers use the unit test
> word as strict as this.) So, I expected to run them by anyone, who
> could compile the sources.

Yes, pure tests are preferable wherever possible.

> #1 'cabal test' could be the best way to exercise them. But the user
> hook is so complicated for a simple 'compile and run'. Found others
> ignoring the arguments and call System.Cmd, execute _the compiler_.
> The compiler name is wired and dependencies also not visible enough,
> not to mention the extension handling. Basically compile by hand is
> against cabal! :) (Pro: the best way to invoke; Con: very hard to do)

Yeah the test hook is pretty much useless. We've got a design for an
improved test system but it may be a while before we implement it.

So that would be the long term solution, and where better to try it out
that Cabal's own tests. In the mean time we can sort out something, even
if it's a little hacky. I can do that bit.

Duncan