OxL Reference
EnglishРусский

18.5 A/B tests

oxl
test ab "search v2" {
  variant a = oldSearch
  variant b = newSearch
  split by .User 50/50
  metric ctr, latency.p95
  guard latency.p95 < SEARCH_P95_LIMIT
  traffic recorded("traffic/search.oxlrec")
}
  • In oxl test the experiment is run on recorded traffic and compares the metrics of the variants with statistical significance.
  • In a cluster oxl ab start "search v2" creates a dynamic Flag, splits traffic and collects metrics; a guard violation automatically rolls the flag back.
  • The body of test ab is a set of clauses variant, split by, metric, guard, traffic, not a block of statements (D-8052; DIV-001; D-6850). From the code of the function under test the variant of a request is reported by ab.variant(…) from std.ab (an implementation extension, App. B).

Implementation-defined. Limits of A/B tests (D-8056; DIV-038; D-6851). This implementation: user metrics of std.obs are not split by variants in a cluster; streaming rpcs are not journalled; replay reproduces only recorded traffic.

ab-test-decl

See also

Source: docs/spec/oxl-standard.en.md