[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Gregory Crosswhite]

On Apr 6, 2010, at 5:28 PM, Thomas Tuegel wrote:

> On Tue, Apr 6, 2010 at 7:43 PM, Rogan Creswick <creswick at gmail.com> wrote:
>> test-framework and test-runner both address the second problem, and
>> those solutions can be kept separate, at least for now.  Figuring out
>> the best way to specify test commands, dependencies, build/execution
>> order, etc. is going to take some substantial effort, and I think that
>> should be the first goal of the project.
> 
> Ok, this is the bottom-line that I didn't understand after our first
> exchange, but I think now I do: I should entirely scrap the second
> aspect of my proposal and focus exclusively on making Cabal build and
> run test programs.

I concur with this conclusion.

>> Cabal allows for multiple executable sections -- are multiple test
>> sections allowed? If so, how are they handled when 'cabal test' is
>> invoked?  If not, will there be any support for running multiple test
>> suites? (more on this below).
>> 
>> While the test executable could be configured to run different sets of
>> tests (at runtime? hm.. we may need more flags to 'cabal test'), there
>> are some situations it's necessary to build multiple test suites
>> because of odd library dependencies.  (for example, testing certain
>> combinations of libraries--imagine supporting multiple versions of
>> ghc.)
> 
> I had intended to allow multiple 'Executable' sections; that's what I
> meant by "this is really an 'Executable' stanza by another name".  I
> think that takes care of your concern about building multiple test
> suites with different dependencies, also.
> 
> My thinking is that 'cabal test' should run all the tests that were
> built, and 'cabal test foo' should run only the test named 'foo'.

This sounds like a good idea to me.

>> The existing Executable sections may serve the need fine, if we could
>> specify how to run the tests in a different way.  Perhaps a list of
>> test commands could be specified instead, eg:
>> 
>>> TestCommands: foo-test-ghc6.6,
>>>                            foo-test-ghc6.8,
>>>                            foo-props --all
>> 
>> Anyhow, just food for thought.
> 
> One of the reasons I prefer implementing a dedicated 'Test' stanza is
> that it makes it easier to tell which executables to install.  There
> may be situations where we want to install test programs, but there
> will _always_ be situations where we _don't_ want to.  (Maybe
> '--{en,dis}able-tests' passed to 'cabal install' should control this?)
> Could we take a 'TestCommands' list and parse out the options to get
> the executable names?  Yes, but relying on that to work makes me
> uneasy.  I think the more robust way to specify executable-specific
> options instead would be to add a field to the 'Test' section
> ('run-options' seems to be consistent with the naming scheme) that
> doesn't exist in 'Executable'.
> 
> (I've snipped some of the comments here; if forget about test
> frameworks for this proposal, it's all tangential.)

I agree with your reasoning here about having a separate Test stanza so that tests can be kept separate from the rest of the package.

- Greg