Testing patterns

The recipes on this page are run with oxl test file.oxl. Blocks with test are verified on the site build by running oxl test; the kinds sim and fuzz are selected with the --kind flag (oxl test).

A table-driven test#

Task. Check a function on many inputs so that adding a case costs one line.

oxl✓ checked in CI

fn slug(title: str) -> str  pure {
  var out = String.new()
  var dash = false
  for c in title.lower().chars() {
    if c.isAlpha() || c.isDigit() {
      out.pushChar(c)
      dash = false
    } else if !dash && out.len() > 0 {
      out.pushChar('-')
      dash = true
    }
  }
  out.freeze().trimSuffix("-")
}

test "slug: table of cases" {
  let cases = [
    ("Hello, World!", "hello-world"),
    ("  many   spaces ", "many-spaces"),
    ("Already-a-slug", "already-a-slug"),
    ("", ""),
    ("2024: the year", "2024-the-year"),
  ]
  for (input, want) in cases {
    assert_eq(slug(input), want)  // on failure both the input and both sides of the comparison are shown
  }
}

test "slug is idempotent" {
  for s in ["Some Title", "x", "A  B"] {
    assert_eq(slug(slug(s)), slug(s))
  }
}

The table is an ordinary list of tuples; the loop for (input, want) in cases unrolls it. No special syntax or parameterization is needed.
On failure assert_eq(a, b) prints both sides, so you can see which case failed.
The second test checks a property (idempotence) on several values; the recipe below does the same on generated ones.

Fixtures#

Task. Prepare data for a test and clean up after it for certain.

oxl✓ checked in CI

data Settings { retries: i32; name: str }

fixture settings() -> Settings {  // the value before yield goes to the test
  let s = Settings{ retries: 3, name: "test" }
  yield s
  print("fixture cleanup")  // code after yield is the clean-up after the test
}

fixture names() -> List<str> {
  yield ["ann", "bob"]
}

test "a test receives the fixture value" with settings as s {
  assert_eq(s.retries, 3)
  assert_eq(s.name, "test")
}

test "fixtures can be combined" with settings as s, names as ns {
  assert_eq(ns.len(), 2)
  assert_ne(s.name, ns[0])
}

fixture name() -> T { …; yield value; …cleanup } - the code before yield prepares the value, the code after it cleans up (it runs even if the test failed).
with settings as s, names as ns attaches several fixtures; every test gets a fresh value.
The fixture value is immutable for the test; if you need a mutable resource (a database, a temp directory), the fixture returns it and the test works with it through methods.

A service mock#

Task. Test code that calls a payment service without starting the service itself - and reproduce failures.

oxl✓ checked in CI

@error data Declined { why: str }
data Receipt { id: i64 }

service Payments {
  @public rpc charge(amount: i64) -> Receipt | Declined
}

impl Payments {
  rpc charge(amount) { Receipt{ id: 0 } }
}

fn checkout(amount: i64) -> str {
  match Payments.charge(amount) {
    Receipt r => "paid #{r.id}"
    Declined e => "declined: {e.why}"
    NetError => "network error"
  }
}

test "retry after a timeout: the mock plays the answers in order" {
  mock Payments {
    charge(_) => fail Timeout once, then Receipt{ id: 7 }  // the first call times out, later calls get the receipt
  }
  assert_eq(checkout(100), "network error")
  assert_eq(checkout(100), "paid #7")
  assert_eq(Payments.calls(.charge).count, 2)  // every call is recorded
}

test "a mock can reject by argument" {
  mock Payments {
    charge(a) if a > 500 => fail Declined{ why: "limit" } always
    charge(_) => Receipt{ id: 1 } always
  }
  assert_eq(checkout(1000), "declined: limit")
  assert_eq(checkout(10), "paid #1")
}

mock Payments { charge(_) => fail Timeout once, then Receipt{ id: 7 } } is a script of answers: the first call times out, then it succeeds (the last answer repeats).
An argument pattern may have a guard: charge(a) if a > 500 => … always.
Payments.calls(.charge).count tells how many times and with what the service was called; this is how you check that the code really retried.
A call that matches no branch fails the test with a description of the call - there are no silent "stubs".

A snapshot#

Task. Fix the output of complex formatting and notice any accidental change.

oxl✓ checked in CI

data Invoice { id: i64; lines: List<str>; total: i64 }

fn render(inv: &Invoice) -> str {
  var out = "invoice #{inv.id}\n"
  for l in inv.lines.iter() { out = "{out}  - {l}\n" }
  "{out}total: {inv.total}"
}

test "invoice layout does not change by accident" {
  let inv = Invoice{ id: 12, lines: ["pen x2", "ink x1"], total: 21 }
  assert_snapshot(&render(&inv))  // the first run creates a file in snapshots/, later runs compare
}

assert_snapshot(&value) compares "{value}" with a file in snapshots/. The first run does not pass: the value is written to ….snap.new.
Review the file and accept it: oxl test --update-snapshots; commit the .snap file. After that any difference is a failure with a diff.
Snapshots are good for reports, templates and serialization; for numbers and simple strings assert_eq is more reliable.
(The example is verified with oxl check: the test has no previously accepted snapshot.)

Virtual time: simulation#

Task. Check logic with clocks (TTL, timeouts) without really waiting.

oxl✓ checked in CI

use std.test { clock }
use std.time { sleep, now }

fn expiresAt(ttl: Duration) -> Time  io, time => now() + ttl

test sim "a one-hour TTL passes instantly in virtual time" seeds 1..=5 {
  let deadline = expiresAt(1h)
  clock.advance(59m)
  assert now() < deadline
  sleep(2m)  // sleep also runs on the virtual clock
  assert now() > deadline
}

test sim "…" seeds 1..=5 runs the body for every seed in a deterministic environment: a virtual clock, scheduler and network.
clock.advance(59m) and even sleep(2m) take microseconds of real time; the result depends only on the seed.
A failed seed is reproduced: oxl test --kind sim --seed 3. The simulation can also cut the network (net.partition), kill nodes (node.kill), delay the disk.

Properties: fuzzing#

Task. Check a regularity on hundreds of automatically generated inputs.

oxl✓ checked in CI

fn reverse(xs: List<i64>) -> List<i64>  pure {
  var out: List<i64> = []
  for i in 0..xs.len() { out.push(xs[xs.len() - 1 - i]) }
  out
}

test fuzz "reversing twice gives the original" (xs: List<i64>) {
  assert_eq(reverse(reverse(xs.clone())), xs)  // the property must hold for any list
}

test fuzz "reverse keeps the length" (xs: List<i64>) {
  let n = xs.len()
  assert_eq(reverse(xs).len(), n)
}

test fuzz "…" (xs: List<i64>) - the parameters are generated by the runtime from their types (respecting where … refinements), guided by code coverage.
State properties, not examples: "a list reversed twice equals the original", "the length is preserved", "serialization and parsing return the same value".
A failing input is saved and reproduced on the next run; details are in the fuzzing guide.

Checking errors#

Task. Make sure a function returns exactly that error.

oxl✓ checked in CI

@error data TooShort { min: usize }
@error data NoDigit {}

fn validate(pw: str) -> () | TooShort | NoDigit {
  if pw.len() < 8 { fail TooShort{ min: 8 } }
  if !pw.chars().iter().any(c => c.isDigit()) { fail NoDigit{} }
}

test "validate reports the right error" {
  assert_fails(validate("abc"), TooShort)  // assert_fails checks the error type
  assert_fails(validate("longpassword"), NoDigit)
  validate("longpass1")!  // no error: an ordinary () value
}

assert_fails(expression, ErrorType) checks the type; a value without an error, validate("longpass1")!, simply gives ().
Errors are values, so a test uses no exception catching: match or is is enough.

Back: cookbook.

Testing patterns

A table-driven test#

Fixtures#

A service mock#

A snapshot#

Virtual time: simulation#

Properties: fuzzing#

Checking errors#

See also