M.C. Escher - Metamorphosis II

Irulan: High Coverage Testing of Haskell Programs

About

Irulan is a black box testing tool for Haskell. It uses the GHC API to automatically load Haskell modules and then automatically constructs and runs expressions to test the functions found therein. It can be used for systematic crash testing, property testing or test-suite generation and comparison for regression testing.

Irulan was developed by me, Tristan Allwood, with mentoring by Lecturer Cristian Cadar and my PhD supervisor Prof. Susan Eisenbach.

Resources

Source Releases:

The sources for Irulan can be downloaded as a source tarball, or the development version is available from Irulan's public monotone repository description source browser.mtn clone irulan.mtn-host.prjek.net uk.co.zonetora.toral.irulan.6.12.3 irulan Note the development version may have many new cool features not (yet) documented and behave differently to what is in the technical reports below, also it is not always guaranteed to build.

Papers

Usage

Building:

Irulan should be built using the standard Haskell Cabal infrastructure. cabal configure && cabal build && cabal install in an unpacked source tarball is your friend.

Note: Irulan has only been tested on x86 and x86_64 Linux systems and is currently only supported with ghc 6.12.3.

Running:

irulan --help to see all the options.

Simplest use: $ irulan source Foo.

If you want code coverage information use: irulan --with-stats-output=foo.stats source Foo or irulan source Foo --ghc-options='-fforce-recomp -fhpc'

At the end you'll get an irulan.tix that the hpc tool can use to report or markup your sources. The -fforce-recomp isn't always necessary, but stops ghc getting confused by stale mix/tix files.

Note, the example commandline and output in the papers have been prettified for presentation.

Use examples:

Experimental Results

Nofib

As described in the most recent paper, we ran Irulan on a filtered version of the real and spectral suites from nofib. The experimental results from all runs (1,10,60 and 300 seconds per module, with (C) and without (N) using case statment generation) are below.

Property Testing

We have compared Irulan to the other property testing libraries (QuickCheck, SmallCheck and Lazy SmallCheck) on the benchmark available from here. Full details will hopefully be published in the next few months in my thesis.

Regression Testing

In the most recent draft paper we discuss several case studies based on using Irulan for regression or behaviour change testing.

M.C. Escher - Metamorphose