[Haskell-cafe] "doctest" for haskell -- a good project?

[Shaun Cutts]

Thanks for the quick responses. To elaborate on how doctests work in python:

(Background: in python functions, classes & modules are all "objects" --
having attributes as well as (for functions) the ability to run them.  They
all have, in particular a documentation string associated with them. To get
at these doctest module just loads the module, then examines its contents,
looking for doc strings of contained functions & classes... so the language
itself provides introspection capabilities: a strong point of python. Maybe
Haskell should also have this :).)

Doctests in python are bits of code in documentation strings specially
annotated. In the docstring for a function "f", we might have (with
apologies for python function call syntax :)):

  >>> f( foo )
  bar

This means: run f( foo ) and check that (the string representation of ) the
output is "bar".

You can put any code in docstring. In effect, the doctest module turns what
you have written into a little module, together with imports, etc.

For continuation lines on blocks of code the convention is:

  >>> f(
  ...     foo )

A blank line after a ">>>" line means that no output is expected (you
defined a function, for example).
If a blank line is part of the string representation of output, then it is
represented, e.g.:

  >>> f( foo )
  <BLANKLINE>
  bar

If result has lots of text, you can use

  >>> f( foo )
  <ELLIPSIS>
  bar

This is helpful, e.g. if f is a helper function that catches an exception
then prints it out; you don't want to check the resulting stack-trace --
just the part at the end about what exception was caught.

There are lots of other switches & options, but no need to copy all of the
details. (Anyway, by the time I get to worrying about them, I'll have my own
opinions. :))

---------------------------------------

So Paul Johnson writes:

> You need to support QuickCheck and Hunit

In fact, at least in python, doctests are even more generic... They support
"anything" -- though
QuickCheck tests are also allowed. Does this approach seem ok/desirable here
too?

A bit worrisome info from Paul:

> Haddock does not contain a full Haskell parser, which makes it difficult
to extend in a principled way (or maybe I'm just not a good enough
programmer). 

> I also looked at using the Haskell parser in the libraries, but that
doesn't capture comments.  
> Associating a comment with a particular construct is not that simple
either.

Hmm... this makes me think that the first thing would be for me to put in an
option to capture comments in the parse tree in the parser. Then Paul and
crew would hack the parser's use into Haddock. Then I'd extend Haddock to
support doctests!

How about that? 

Or actually I think its overkill. If I can hook into the processing of the
comment (presumably only extended comments), I can piece together a fake
"module". The "..." convention means I should know how to stitch together
expressions without having to understand them. One problem is checking the
"output" of an expression. If an expression has output, it would seem I
should throw in a test (using QuickCheck?). Hmmm.

- Shaun


> -----Original Message-----
> From: Thomas Schilling [mailto:nominolo at googlemail.com] 
> Sent: Saturday, March 22, 2008 9:54 AM
> To: Neil Mitchell
> Cc: Don Stewart; Shaun Cutts; haskell-cafe at haskell.org
> Subject: Re: [Haskell-cafe] "doctest" for haskell -- a good project?
> 
> 
> On 22 mar 2008, at 13.17, Neil Mitchell wrote:
> 
> > Hi
> >
> >>  One idea that does strike me is that it would be super useful to 
> >> have  the ability in ghci to extract the haddocks 
> associated with a 
> >> function.
> >>
> >>> :doc map
> >>
> >>  would result in:
> >>
> >>      -- | 'map' @f xs@ is the list obtained by applying 
> @f@ to each 
> >> element
> >>      -- of @xs@, i.e.,
> >>      --
> >>      -- > map f [x1, x2, ..., xn] == [f x1, f x2, ..
> >>      -- > map f [x1, x2, ...] == [f x1, f x2, ...]
> >
> > That will be in Hoogle 4, as it does already directly 
> relate to what 
> > Hoogle has to do.
> >
> > As for doctest, i think that could be implemented as a 
> --warn flag to 
> > Haddock, and might be a useful thing to have. For example 
> "90% of the 
> > base library has documentation, please go add documentation to 
> > 'sinh'", as a warning message.
> >
> 
> 
> Though that would be a nice feature, too, I think you 
> misunderstood what a doctest is.
> 
> A doctest is a unit test in the documentation.  This way it 
> serves as  
> a usage example as well as a formal specification of the semantics.   
> I think we already have some way to run quickchecks inside 
> documentation, but I can't remember what it was that could do that.
> 
> / Thomas
>