error ab-test-decl
What it means#
test ab "name" { … } declares an A/B experiment (§18.5): its body is a list of clauses, checked by the compiler (D-6850..D-6855):
- exactly two
variant name = fclauses, the control first (an experiment is a dynamicFlag, which is on or off — D-6852);fmust name something of the program; split by .User|.Tenant|.Node a/bwith one share per variant, adding up to 100 (aFlagsplits by these dimensions only, §14.2);- every
metric/guardmetric is built in (latency.p50,latency.p95,latency.p99,latency.mean,error_rate) or the name of a std.obs metric (no dots); traffic recorded("file.oxlrec")names the recorded traffic (relative to the package root, D-6853).
Example#
fn oldSearch(q: str) -> str pure => q
fn newSearch(q: str) -> str pure => q
fn otherSearch(q: str) -> str pure => q
test ab "search v2" {
variant a = oldSearch
variant b = newSearch
variant c = otherSearch // <- ab-test-decl: exactly two variants
split by .Request 40/30/30 // <- ab-test-decl: a Flag splits by .User, .Tenant or .Node
metric latency.p42 // <- ab-test-decl: unknown metric
} // <- ab-test-decl: no `traffic recorded(...)`error
ab-test-decl
How to fix#
test ab "search v2" {
variant a = oldSearch
variant b = newSearch
split by .User 50/50
metric ctr, latency.p95
guard latency.p95 < SEARCH_P95_LIMIT
traffic recorded("traffic/search.oxlrec")
}