[Haskell-cafe] [ANN] Laborantin: experimentation framework

[adam vogt]

Hello Lucas,

Am I correct to say that laborantin only does full factorial
experiments? Perhaps there is a straightforward way for users to
specify which model parameters should be confounded in a fractional
factorial design. Another extension would be to move towards
sequential designs, where the trials to run depend on the results so
far. Then more time is spent on the "interesting" regions of the
parameter space.

I think getVar/param could be re-worked to give errors at compile
time. Now you get a runtime error if you typo a parameter or get the
type wrong. Another mistake is to include parameters in the experiment
that do not have any effect on the `run` action, unless those
parameters are there for doing replicates.

Those might be addressed by doing something like:

    a <- parameter "destination" $ do ...
    run $ print =<< param a

Where the types are something like:

  param :: Data.Tagged.Tagged a Text -> M a
  values :: [T a] -> M (Tagged a Text)
  str :: Text -> T Text
  num :: Double -> T Double

with M being whatever state monad you currently use, and param does
the same thing it always has, except now it knows which type you put
in the values list, and it cannot be called with any string. The third
requirement might be met by requiring -fwarn-unused-matches.

An alternative strategy is to change your type Step, into an algebraic
data type with a function to convert it into what it is currently.
Before the experiment happens, you can have a function go through that
data to make sure it will succeed with it's getVar/param. This is
called a deep embedding:
<http://www.haskell.org/haskellwiki/Embedded_domain_specific_language>.

Regards,
Adam

On Mon, Dec 23, 2013 at 4:27 AM, lucas di cioccio
<lucas.dicioccio at gmail.com> wrote:
> Dear all,
>
> I am happy to announce Laborantin. Laborantin is a Haskell library and DSL
> for
> running and analyzing controlled experiments.
>
> Repository: https://github.com/lucasdicioccio/laborantin-hs
> Hackage page: http://hackage.haskell.org/package/laborantin-hs
>
> Laborantin's opinion is that running proper experiments is a non-trivial and
> often overlooked problem. Therefore, we should provide good tools to assist
> experimenters. The hope is that, with Laborantin, experimenters will spend
> more
> time on their core problem while racing through the menial tasks of editing
> scripts because one data point is missing in a plot. At the same time,
> Laborantin is also an effort within the broad open-science movement. Indeed,
> Laborantin's DSL separates boilerplate from the actual experiment
> implementation. Thus, Laborantin could reduce the friction for code and
> data-reuse.
>
> One family of experiments that fit well Laborantin are benchmarks with
> tedious
> setup and teardown procedures (for instance starting, configuring, and
> stopping
> remote machines). Analyses that require measurements from a variety of data
> points in a multi-dimensional parameter space also fall in the scope of
> Laborantin.
>
> When using Laborantin, the experimenter:
>
> * Can express experimental scenarios using a readable and familiar DSL.
>   This feature, albeit subjective, was confirmed by non-Haskeller
> colleagues.
> * Saves time on boilerplate such as writing command-line parsers or
>   encoding dependencies between experiments and analysis results in a
> Makefile.
> * Benefits from auto-documentation and result introspection features when
> one
>   comes back to a project, possibly months or weeks later.
> * Harnesses the power of Haskell type-system to catch common errors at
> compile time
>
> If you had to read one story to understand the pain points that Laborantin
> tries to address, it should be Section 5 of "Strategies for Sound Internet
> Measurement" (V. Paxson, IMC 2004).
>
> I'd be glad to take question and comments (or, even better, code reviews and
> pull requests).
>
> Kind regards,
> --Lucas DiCioccio (@lucasdicioccio on GitHub/Twitter)
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>