This is verbatim and this is
code.
ocamltest is a test-driver, that is, a program that can run tests and report their results so that they can be used by a test infrastructure.
Originally, the tool has been designed specifically to run the integration tests of the OCaml compiler's test suite. However, it has been designed with extensibility in mind and thus has a plugin mechanism that makes it possible to extend it with other tests.
For a start, one may wonder in which language a test-driver for a compiler should be written. It may indeed seem odd to write a test-driver for a compiler in the language it compiles, since the compiler itself is yet untested and thus not trustworthy.
It can however be observed that the OCaml compiler is
bootstrapped, meaning that it is itself written in OCaml. A
newer version of the compiler can thus be produced from an existing one
and the (OCaml) source code of that newer version. Practically, this
means that the compiler works at least well enough to recompile itself.
This is why we consider that it is okay to write a test-driver like
ocamltest in OCaml, as long as it uses only code that has been used to
bootstrap the compiler. In particular, this is why we prefer not to rely
on any external dependency, not even the libraries included in the
compiler distribution such as the Unix
library.
As has been noted above, ocamltest has been developed to run the already existing integration tests of the OCaml compiler's test suite, which were previously run by a set of makefiles. This context explains several design decisions which could otherwise seem rather arbitrary.
For example, the reason why ocamltest has no support for running unit tests is that there were no such tests in the OCaml compiler's test suite.
Indeed, the OCaml compiler's test suite is composed mainly of complete programs. In this context, the most usual meaning of "testing" a program is that the program needs to be compiled and executed. The test will be considered successful if the program compiles as expected and, when run, returns the expected value.
Since this scenario is the most frequent one, it was of particular importance to make writing tests of this form as simple as possible.
However, not all tests fall into the previously described category, so it is also necessary to support not only variations on the previous scenario (compile but do not run, compile with certain options, etc.) but also completely different tests, such as REPL tests, debugger tests, etc.
To fulfill these requirements and make it as easy as possible to turn
a program into a test, it has been chosen to design a Domain-Specific
Language (DSL) used to annotate the test program with a (* TEST *) block at its top. This block
specifies how the test should be performed.
The next chapter explains through examples how to write simple tests. We then introduce the key concepts used by ocamltest to provide a better understanding of how it works and can be used to write more complex tests. The last two chapters give an in-depth description of the built-in tests and actions and of the tests and actions that are specific to the OCaml compiler.
This chapter is a tutorial. It explains how to write simple test programs and also tries to give insights about how ocamltest works. These insights will be deepened in chapter #concepts where ocamltest is presented in a more abstract and conceptual way.
We start by explaining how to set-up a proper environment for writing tests. We then show how to turn the traditional "Hello, world!" program into a test and explain how to run it with ocamltest. We continue with a few variations on this test and conclude this chapter with a few other useful tests.
Writing tests requires that the sources of the OCaml compiler for which one wants to write them are downloaded and compiled. The compiler does not need to be installed.
The sources can be downloaded either as an archive, or directly
cloned through git, which seems more appropriate in the context of
writing ones own tests. Refer to INSTALL.adoc (and also to README.win32.adoc if you are on Windows) to
learn how to get the sources of the OCaml compiler and to compile
them.
In the remainder of this manual, we will assume that the sources of
the OCaml compiler have been extracted in the ${OCAMLSRCDIR} directory (for instance ${HOME}/src/ocaml) and that you have
successfully configured and compiled them as described in INSTALL.adoc or README.win32.adoc, according to your operating
system. The tools and libraries necessary for running tests should also
be built. This can be achieved by running the following command from
${OCAMLSRCDIR}:
make -C testsuite lib tools
We will also assume that an ocamltest
command is available in your PATH.
Although this is not strictly necessary, it is strongly recommended that
you set this up because this will simplify test development a lot. This
can be achieved e.g. by creating a symbolic link to ${OCAMLSRCDIR}/ocamltest/ocamltest (or its
native counterpart ${OCAMLSRCDIR}/ocamltest/ocamltest.opt) in a
directory that is already in your PATH,
like ~/bin.
Consider the following OCaml implementation of the classical "Hello,
world!" program written to a hello.ml
file:
let _ = print_endline "Hello, world!"
Now assume we would like to make sure that the OCaml compiler can compile this program and that the resulting executable indeed prints the expected output. Here are the required steps to turn the program above into a test usable by ocamltest to verify this:
First, we add a special comment at the very beginning of our
hello.ml file to make it explicit that it
is a test:
(* TEST *)
let _ = print_endline "Hello, world!"
We then need to say what the expected outputs are. In our case, we expect that compiling the test produces no output at all and that its execution produces one single line:
Hello, world!
To let ocamltest know about this, we create a hello.reference file containing the program's
expected output – the line mentioned above. There is nothing special to
do for silent compilations since this is what is expected by default and
a non-silent compilation would actually cause a test failure.
We can now ask ocamltest to run our test program with the following command:
ocamltest hello.ml
Running this would produce an output similar to this one:
... testing 'hello.ml' with 1 (native) => passed
... testing 'hello.ml' with 2 (bytecode) => passed
In addition to this output, it may be noticed that the previous
command has also created an _ocamltest
directory whose content will be examined in the next
sub-section.
Finally, there is one extra step required if we want our newly
created test to be run automatically as part of the OCaml compiler's
test suite. We need to move hello.ml and
hello.reference to a directory (say newtest) located somewhere below testsuite/tests in the compiler's source tree.
Once this is done, the command
make all
executed in the testsuite directory of
the OCaml compiler' source tree will run all the test suite, which now
also includes our own test.
The only thing we know from ocamltest's output when run on hello.ml is that it is running two tests named
bytecode and native and that the two of them succeed. This
can seem rather uninformative, and in a way it is, but it has to be kept
in mind that this information is the one passed by the test-driver
(ocamltest) to the test infrastructure. In that respect, this is enough.
For us users, though, it is not. That's why ocamltest logs much more
details about what is going on in a per-test log file, which should be
located in the _ocamltest/hello/hello.log
file found in the directory where hello.ml
is.
Before looking at this log file, notice that it has been created in a
test-specific directory. ocamltest creates such a directory for each
file it tests and makes sure every file produced as a result of testing
this file will be placed in this directory, either directly, or in one
of its sub-directories. The latter happens if the test has to be
compiled several times, with the same compiler and different
command-line options, or with different compilers. In particular, in
order to better understand what follows, it may be helpful to remember
that OCaml actually consists in not less
than four compilers: ocamlc.byte and ocamlc.opt which are the bytecode and native
flavors of the bytecode compiler and ocamlopt.byte and ocamlopt.opt which are the bytecode and native
flavors of the native compiler. So, as we will see, ''testing the
bytecode compiler'' actually involves testing two compilers, and the
same goes for ''testing the native compiler''.
Now that all this has been spelled out, let's examine the log file produced by the test. Although it is too long to be reproduced here, it is recommended to go through it quickly to get an idea of its structure. Here is how it starts:
Specified modules: hello.ml
Source modules: hello.ml
The first line lists the names of the modules the test consists of.
The second line is almost similar but if some modules had separate
interface files, they would be listed here, too, without the user having
to specify them in the list of modules (for each specified .ml file, ocamltest looks whether a
corresponding .mli file exists and, if so,
adds it to the list of files to consider).
The rest of the log file can be split into two parts which are very
similar to each other: one for the native
test and one for the bytecode test. Among
other things, we learn that each of these tests is composed of nine
actions. Before diving into the details of what each of these actions
does, let us take this opportunity to introduce a bit of ocamltest
terminology. An action is anything that can pass, skip or
fail. A test
is a sequence of such actions. Running a test thus means running each of
its actions, in sequence, until all the actions have been run or one of
them returns fail or skip. Whatever the last run action returns, this
value will be the result of the whole test.
To give concrete examples of actions, let's briefly go over the nine
ones involved in the bytecode test (those
for the native test are quite
similar):
setup-ocamlc.byte-build-env:: as
its name suggests, this action creates a build environment where a
program can be compiled and executed using the ocamlc.byte compiler. More precisely, this
involves creating a dedicated directory under the test-file specific
directory and populating it with the files required by subsequent
actions. Depending on what the underlying operating system supports, the
files will be either symlinked or copied from the test source
directory.
ocamlc.byte:: invokes the ocamlc.byte compiler in various ways. Here, the
test program is compiled and linked, but as we will see later, different
behaviors are possible depending on ocamltest
variables.
check-ocamlc.byte-output:: this
action compares the compiler's output to a reference file, if one
exists. As has been mentioned earlier, the absence of such a reference
file specifies that the compiler's output is expected to be empty – if
it is not, this causes a failure of this action and thus of the whole
bytecode test.
run:: now that the program has been
successfully compiled, it is run with its standard output and error
streams saved to a file.
check-program-output:: this time it
is the output of the program which is compared to a reference file,
namely the hello.reference file created
earlier. So far this comparison succeeds, because the output of the
program is identical to the reference file but, as an exercise, one may
try to modify the reference file to see how this causes the failure of
this action and of the whole bytecode
test.
This action concludes the test of the ocamlc.byte compiler. We now know that it is
able to successfully compile our test program and that the resulting
executable runs as expected. The four remaining actions are going to
test the ocamlc.opt compiler in a similar
but not identical way:
setup-ocamlc.opt-build-env:: this
action is the counterpart of action 1 for the ocamlc.opt compiler.
ocamlc.opt:: like action 2, this
action compiles the test program but with the ocamlc.opt compiler.
check-ocamlc.opt-output:: again,
this action is similar to action 3.
compare-bytecode-programs:: here we
make sure that the generated executable is correct, but in a different
way than for the ocamlc.byte compiler.
Rather than running it and checking its output, we compare it to the one
produced in action 2. Such a check may seem strange, because what it
requires is that ocamlc.byte and ocamlc.opt produce exactly the same binary and
not two binaries that perform similarly when they are run, but it has
proven useful in the past and has permitted to detect a subtle bug in
the compiler.
As has been briefly mentioned, the precise behavior of actions (and
thus of tests) may depend on variables whose value can be
adjusted in the (* TEST ... *) blocks. In
ocamltest, all the values of variables are strings. Here are a few
examples of things that can be achieved just by defining the appropriate
variables. The complete description of the actions provided by ocamltest
and the variables they use will be given in chapters #builtins and #ocaml-specific.
Assume our hello.ml example is modified
as follows:
(* TEST *)
open Format
let _ = print_endline "Hello, world!"
As may be verified, this program still passes the default tests. It
is however not as minimal as our previous version, because the Format module is opened but not used.
Fortunately, OCaml has a warning to detect such unused open directives, namely warning 33, which is
disabled by default. We could thus add this version of hello.ml to the test suite, not so much to
verify that the program compiles and runs as expected (we verified this
already), but rather to make sure the compiler does indeed trigger the
expected warning. Here are the required steps to achieve this:
We slightly modify the test block in hello.ml, as follows:
(* TEST
flags = "-w +33";
*)
Since we now expect a non-empty output for the compilers, we need
to store the expected output in a file, namely hello.compilers.reference beside hello.ml and hello.reference. To figure out what this file
shall contain, we can run ocamltest even before it has been created. Of
course, the action that checks compiler output will fail, but in this
way we will get the compiler's output which we will just have to check
(to make sure it is what we expect) and to move to the reference file.
Thus, we do:
$ ocamltest hello.ml
which fails, unsurprisingly, and shows us the paths to the file containing the output produced by the compiler and the path to the expected reference file. We also see what the compiler produced as output but we can double-check that the output is what we expect as a reference:
$ cat _ocamltest/hello/ocamlc.byte/ocamlc.byte.output
which shows the warning we expect from the compiler. We can thus move this file to the reference file:
$ mv _ocamltest/hello/ocamlc.byte/ocamlc.byte.output hello.compilers.reference
and if we now run ocamltest again, all the tests pass.
Two remarks are due. First, we have used the flags variable, to pass extra flags to all the
compilers. There are two other variables one can use, namely ocamlc_flags and ocamlopt_flags, to pass flags to the bytecode or
native compilers. Second, in this test all the compilers have the same
output so one reference file is enough for all of them. There are
situations, though, where the compiler's output is back-end-specific (it
depends whether we compile to bytecode or to native code) or even
compiler-specific. ocamltest is clever enough to know how to deal with
such situations, provided that the reference files are named
appropriately. It will indeed first lookup the test source directory for
a compiler-specific reference file, e.g. hello.ocamlc.byte.reference. If no such file
exists, a back-end-specific reference file is searched, e.g. hello.ocamlc.reference for a reference common to
both ocamlc.byte and ocamlc.opt. If this file does not exist either,
ocamltest falls back to looking for hello.compilers.reference as we have seen in
this example, the absence of which meaning that the compiler's output is
expected to be empty.
Let's start with our original hello.ml
test program and extract the greeting logic into a distinct greet.ml module:
let greet guest = Printf.printf "Hello, %s!\n" guest
Let's also write an interface, greet.mli:
val greet : string -> unit
Our hello.ml test program can then be
rewritten as follows:
(* TEST
modules = "greet.ml";
*)
let _ = Greet.greet "world"
Provided that the hello.compilers.reference file previously used
to test warnings is deleted, running ocamltest on hello.ml should work. It will also be worth
looking at the first two lines of the log file generated while running
the test. It says:
Specified modules: greet.ml hello.ml
Source modules: greet.mli greet.ml hello.ml
The first line shows that the modules
variable has been taken into account. On the second line, it can be seen
that the greet.mli file appears, right
before greet.ml. It is ocamltest that has
added it, because it has been recognized as an interface for one of the
specified modules.
To sum up, if a test consists in several modules, it is enough to
list their implementations (except the one of the main test program
which is implicit) in the modules
variable, in linking order. There is no need to worry about their
interfaces, which will be added automatically by ocamltest, if they
exist.
Assume we want to use the following program to make sure regular
expressions as implemented by the Str
library work as expected:
let hello_re = Str.regexp "^Hello, .+!$"
let hello_str = "Hello, world!"
let _ =
if not (Str.string_match hello_re hello_str 0) then
begin
Printf.eprintf "There is a problem!\n";
exit 2
end
This test terminates silently if everything goes well and prints a
message on its standard error only if something goes wrong, which means
we won't have anything special to do so that ocamltest checks for an
empty output after the program has run. However, to be able to compile
and link this test, there are several things we need to do so that it
finds the Str library it uses. More
precisely, we need to add the -I option
pointing to the right directory and, at link time, to give the name of
the appropriate library file. To make our life a bit simpler, ocamltest
has a few variables where directories and libraries can be listed. Once
they are there, it is ocamltest which will take care of adding the -I option for each directory and for adding the
right library file depending on whether we are producing bytecode or
native programs. So, here is how the previous program can be annotated
so that it becomes a test:
(* TEST
directories += " ${ocamlsrcdir}/otherlibs/str ";
libraries += " str ";
*)
With these annotations, it becomes possible to run re.ml as an ocamltest test program and, doing
so, one may notice that the two tests pass. There are however a few
other things worth pointing out here regarding the ocamltest DSL. For a
start, the notation ${variable} inside a
string means to replace variable by its
value, as happens in many other languages, like the bash shell.
Moreover, it is the first time we meet the += operator
which concatenates a value to a variable. More precisely,
foo += "bar"
is equivalent to
foo = "${foo}bar"
and not to
foo = "${foo} bar"
as it may happen in other languages such as makefiles.
In other words, the += operator concatenates two strings
without inserting any implicit space between them as e.g. make would do.
This is because in some cases such a behavior is required and could not
be achieved if spaces were implicitly added, whereas with a literal
concatenation it is always possible to include spaces explicitly. This
is exactly what happens in the ocamltest annotation block above, where
the strings added to the libraries and
directories variables are surrounded by
spaces. As should be clear to the reader by now, these spaces are
mandatory. Without them, the added values would be glued to the last
word of the variable and would thus be misinterpreted.
Finally, one may notice that, although ocamltest does make it
possible to link a test program with a library, it does not really make
it easy or convenient to do so. In particular, what if we want to write
several, perhaps many test programs that need to be linked with Str? Will we have to repeat these lines
everywhere, thus creating code that is going to be tedious to maintain?
Well, fortunately not. Actually, ocamltest has a much more elegant way
to deal with such issues, namely environment modifiers. As will
be explained in chapter #concepts, an
environment modifier is an object that gathers several variable
definitions that can then be included in an ocamltest block at once.
Environment modifiers have to be defined in ocamltest itself and can
then be used with the include directive.
For instance, the previous test block is actually written as
follows:
(* TEST
include str;
*)
So far, we have been able to fulfill our requirements just by
assigning the right values to variables and relying on the bytecode and native tests ocaml runs by default. There are
however situations where this is not enough and where one needs the
ability to run other tests. One example of such a situation is when a
test needs to be performed only on one given operating system, e.g.
because it uses a feature which is present only on that operating
system. On an other operating system, the test should be skipped because
it is irrelevant. To illustrate this, here is how our original hello.ml test program should be annotated so
that it is run only on Unix platforms:
(* TEST
unix;
{
bytecode;
}
{
native;
}
*)
As can be understood from this example, tests are organised in a tree
nested blocks. Each block begins with a brace and a list of tests and
environment statements that are executed in sequence. Then the block
contains a set of sub-blocks that are executed independently of each
other (i.e. their environments are independent and they are run
regardless of the success or failure of their siblings). Here for
instance, bytecode and native are sub-tests that will be run only if
the unix test passes and will not be
started if it fails or skips.
With this information in mind, it can be seen that the smallest test block
(* TEST *)
is actually equivalent to
(* TEST
{
bytecode;
}
{
native;
}
*)
One common error when designing tests is to believe that a block like
(* TEST
unix;
*)
means to execute the unix test that
verifies that the OS is indeed Unix and then to execute the default
tests. This is actually not the case. The only situation in which the
default tests are considered is when the test block contains absolutely
no test statement. As soon as there is a test statement, the default
tests are ignored completely and one needs to be totally explicit about
which tests to run. So the correct way to write the erroneous block
above is the use shown at the beginning of this section, namely:
(* TEST
unix;
{
bytecode;
}
{
native;
}
*)
The braces make explicit the scope of variable assignments: an assignment modifies a variable for the rest of its block and for all sub-blocks (unless overridden at some point).
For instance, given the following blocks:
(* TEST
foo = "abc";
{
bar = "def";
test1;
{
baz = "hij";
subtest1;
}
{
subtest2;
}
}
{
test2;
}
*)
The definition of foo is visible in
all the tests
The definition of bar is visible in
all the tests except test2.
The definition of baz is visible
only in subtest1.
This section introduces three tests provided by ocamltest and that can be of particular
interest. A brief description of each available test is shown by the
option `-show-tests` of ocamltest. A complete list of available tests
and actions and their detailed descriptions are given in chapters #builtins and #ocaml-specific.
toplevel and expect testsTwo tests are provided to make sure that the OCaml top-level behaves
as expected: toplevel and expect. These tests are similar in that they
both allow testing how the OCaml top-level reacts to some user input,
but they are different in the way one specifies the expected output and
also in what they can test. The toplevel
test behaves in a spirit similar to the compiler tests described above,
meaning that the expected output has to be stored in its own, separate
file. Since this test invokes the real OCaml toplevel, it is useful to
test advanced features like the behavior of the toplevel when its input
is a file rather than a terminal, or similar things. In the expect test,
on the contrary, the input and the output it is expected to produce can
be written in the same file, close to each other. However, this test
uses the OCaml toplevel as a library, rather than calling it as an
external program. So this test is actually not testing the complete real
OCaml toplevel, but for testing language features it remains perfectly
valid and is actually what is needed in most of the cases. We thus give
below an example of an expect test and will describe the toplevel test in chapter #ocaml-specific.
So, here is a toy example of an expect
test:
(* TEST
expect;
*)
type point = { x : int; y : int };;
[%%expect{|
type point = { x : int; y : int; }
|}];;
The first line after the test block is the input phrase, while the
line that appears between [%%expect{| and
|}];; is the corresponding expected
output. The expect test can also be used
to test the output in presence of the -principal command-line flag. In such cases, the
expected output should be written in a |}, Principal{| block (to be improved).
script testIt may happen that a needed test is not provided by ocamltest. Of
course, if it turns out that this test would be helpful to test several
source files, then the best solution is to add it to ocamltest itself.
Some tests are however so specific that it is easier to write them as
shell scripts. Such tests can be run by the script test, their name being defined by the
script variable. In this case, the script
is run in an environment where all the variables defined in ocamltest
have been exported. The script uses its exit status to report its result
and can write a response to a dedicated file to modify its environment
or explain why it failed or skipped, as will be explained in chapter #builtins. For the moment, let's see how to use a
script to "test" our original hello.ml
example. Our annotated program would look as follows:
(* TEST
script = "${test_source_directory}/faketest.sh";
script;
*)
let _ = print_endline "Hello, world!"
And here is faketest.sh, make sure it
is executable:
#!/bin/sh
exit ${TEST_PASS}
This should be enough for the following command to work:
ocamltest hello.ml
This of course tests nothing and a real test script should actually do something before returning its result. Let's however see how we can make the script test fail gracefully:
#!/bin/sh
echo Why should this pass in the first place > ${ocamltest_response}
exit ${TEST_FAIL}
Running ocamltest on our hello.ml
program again produces the following output:
... testing 'hello.ml' with 1 (script) => failed (Why should this pass in the first place)
In addition, the script can write lines of the form
variable="string", or variable+="string", or
-variable, to file ${ocamltest_response}, to
set, append to, and unset ocamltest environment variables,
respectively.
The list of builtin variables can be obtained by running ocamltest
-show-variables.
Environment variables for a test can be set using:
set VARIABLE_NAME="value";
in the test header (the quotes are mandatory).
On the contrary, you can ensure that an environment variable is not set when the test runs with:
unset VARIABLE_NAME;
The list of builtin and OCaml-specific actions and tests can be
obtained by running ocamltest -show-actions.