Testing

Note: Not built: a failure that names the expression it checked and its line; test functions marked in source and a test build; a test that reaches a package's private submodules; fixtures, setup and teardown beyond a program's own; mocks; running a package's examples (they are built, not run).

A Cone test is an ordinary program. It lives in the tests folder of the package it tests, imports that package by name as any user would, makes some checks, and prints what it finds. congo test builds and runs every test of a package and compares what each one printed, and the status it exited with, against what the test's files say they should be. The testing package supplies the checks. Nothing in either is known to the compiler: both are library and tool code.

Writing a test#

Suppose a package mylib has three functions:

mod mylib;

pub fn sum(a i64, b i64) i64 {
  a + b;
}

pub fn mean(a f64, b f64) f64 {
  (a + b) / 2.;
}

pub fn isEven(n i64) bool {
  n % 2i64 == 0i64;
}

Its folder holds the manifest, the source, and the tests (Packages and Congo describes the folder):

mylib/
    congo.toml
    src/mylib.cone
    tests/arith.cone       one test: a program
    tests/arith.out        what it must print
    examples/              programs showing the package's use; built by congo test, not run

A test is a lone file with its own mod line and a main. It imports the package under test and the testing package, and returns what done gives:

mod arith;

import testing use *;
import mylib;

fn main() i32 {
  expectInt(mylib.sum(2, 3), 5, "sum of 2 and 3");
  expectFloat(mylib.mean(1., 2.), 1.5, 0.000001, "mean of 1 and 2");
  expect(mylib.isEven(4), "4 is even");
  done();
}

Every check that passes prints nothing, so the test prints only its summary. That summary is the whole of tests/arith.out:

3 checks passed

Because the test imports mylib the way any other program does, it reaches only what the package makes pub.

The testing package: checks#

The package is imported with import testing use *;, and its checks are functions. Each kind of value has a check of its own name, in two forms: an expect check, which records a failure and lets the test go on, and a require twin, which stops the test when it fails.

Check	Passes when
expect(cond, label)	cond is true
expectInt(got, want, label)	two signed integers are equal; an i32, i16 or i8 value is widened to i64
expectUInt(got, want, label)	two unsigned integers are equal; a usize, u32, u16 or u8 value is widened to u64
expectFloat(got, want, within, label)	the two differ by no more than within (the bound is inclusive); an f32 is widened to f64; a NaN is never within any tolerance of anything
expectStr(got, want, label)	two strings hold the same bytes
expectBool(got, want, label)	two bool values are equal

The label is a string saying what the check is about. Each check returns true when it passed, for a test that wants to branch on it. The require versions are named require, requireInt, requireUInt, requireFloat, requireStr and requireBool, with the same parameters.

A failing check prints one line: the label, then what was expected against what was got. The test goes on, so one run reports every failing check. At the end, done() prints the summary, 3 checks passed or 3 checks, 1 failed, and returns the exit status for main to return: 0 when every check passed, 1 when any failed. A test whose .out file holds the summary therefore pins how many checks ran, as well as that they passed.

mod standalone;

import testing use *;

// The code under test
fn sum(a i64, b i64) i64 {
  a + b;
}

// Wrong on purpose: it drops the fractional part
fn mean3(a f64, b f64, c f64) f64 {
  f64.from(i64.from((a + b + c) / 3.));
}

fn main() i32 {
  expectInt(sum(2, 3), 5, "sum of 2 and 3");
  expectFloat(mean3(1., 2., 3.), 2., 0.0001, "mean of 1 and 2 and 3");
  expectFloat(mean3(1., 2., 4.), 2.333333, 0.0001, "mean of 1 and 2 and 4");
  expectStr("cone ", "cone", "a trailing space shows");
  expectBool(sum(1, 1) == 2, true, "one and one is two");
  // A check whose failure would make the rest meaningless stops the test
  require(sum(0, 0) == 0, "zero is the identity");
  done();
}

Run as a program, that prints:

mean of 1 and 2 and 4: expected 2.333333 within 0.0001, got 2.000000
a trailing space shows: expected "cone", got "cone "
6 checks, 2 failed

and exits with status 1. A test written to fail like this one would pin that status in a .exit file (below).

Stopping a test#

A require check is for the place where going on would make no sense, as when a later check reads something this one found missing. When it fails it prints its failure line, then stopped: and the summary, and ends the program with exit status 1; the statements after it never run, and neither does done. Each test is its own program, so stopping one stops no other. A require that passes goes on.

mod stopper;

import testing use *;

fn main() i32 {
  expectInt(1, 2, "first");
  requireInt(3, 4, "second");
  expectInt(5, 6, "never runs");
  done();
}

first: expected 2, got 1
second: expected 4, got 3
stopped: 2 checks, 2 failed

How values print#

• Integers print exactly, and an unsigned value prints unsigned.
• Floats print to six decimal places, rounded, so the noise in the last digits of an f32 does not show (a tiny negative value prints as 0.000000, without a sign). NaN and the infinities print as nan, inf and -inf; a value of nine million million or more prints as the C library's %g does, 1e+20. The tolerance prints as %g does, so a tolerance below a millionth can print a failing expected value and got value alike.
• Strings print as given in double quotes, so leading and trailing spaces show.
• bool prints as true or false.

A float literal with no suffix is an f32, and carries an f32's rounding into the f64 a check takes. Against an f64 result, with a tolerance finer than about a ten-millionth of the value, write the expected value as an f64: 0.1d.

Running tests: congo test#

congo test, run in a package's folder or any folder below it, does four things:

1. Builds the package as congo build does. A package that does not build is reported, and none of it is tested.
2. Runs each test, tests/name.cone. It is compiled against the package's generated include file and linked with the package's object, the path every importer takes. The package under test is found before any other package of its name, so a package in no registry can be tested where it stands. Each test is built in build/debug/tests/name/ and run there, with no arguments and no input, for at most 60 seconds.
3. Compares what the test printed with tests/name.out, line ends and trailing blank lines aside, and the exit status with 0, or with the number in tests/name.exit where there is one. A mismatch prints a difference.
4. Builds each example, examples/name.cone, and does not run it. An example that does not build is a failure.

Each test reports as test name ... ok or FAILED, followed by a summary line, and congo test exits 1 if anything failed. For mylib:

        test arith ... ok
      Result mylib: 1 test: 1 passed, 0 failed; 0 examples: 0 built, 0 failed to build

A test that is meant to fail says so in its files. A copy of the test above that expects sum(2, 3) to be 6, and so fails:

mod broken;

import testing use *;
import mylib;

fn main() i32 {
  expectInt(mylib.sum(2, 3), 6, "sum of 2 and 3");
  expect(mylib.isEven(4), "4 is even");
  done();
}

pins what it prints in tests/broken.out:

sum of 2 and 3: expected 6, got 5
2 checks, 1 failed

and its status in tests/broken.exit, which holds the single line 1.

The files beside a test#

• .out: what the test must print. Required.
• .exit: the exit status, where it is not 0.
• .err: what the test must write to standard error. Written by hand, for a test that provokes a panic on purpose: the file holds the panic's line, and the .exit holds the status an aborted program has (3221226505 on Windows). Without a .err file, standard error is not compared.
• .debug: marks a test of a check made only in a debug build (an assertDebug). What the file holds does not matter. A debug build panics on the broken check, which the test expects; a release build leaves the check out, so congo test --release does not build or run such a test, and reports it as skipped, which is not a failure.

Choosing and blessing tests#

congo test arith runs only the tests and examples whose file name contains the text; text that matches nothing is an error.

congo test --bless writes the .out (and .exit, if the status is not 0) of a test that has none, from a run. It never rewrites a file that exists: to change an expected file, edit it, or delete it and bless again. Blessed output is a claim that the package is right, so read each line of it against what you know the package should do before you keep it. A blessed file is only as good as the run it was copied from.

Run in a folder of packages instead of inside one, congo test tests each package in turn and ends with a summary of them all. A test of an executable package cannot import it, since only a library can be imported, so congo test says so and fails the test.