.oxlrec — the request journal

oxl trace --record ([С III.1], D-6860) journals the unary rpcs a cluster receives over the network; test ab … traffic recorded("f.oxlrec") ([С §18.5]) replays such a journal against the variants of an experiment, and oxl replay f.oxlrec sends it to a node again and compares the responses. Implementation: runtime/src/ab/ab.h (encodeFile / decodeFile), a Python reader/writer: scripts/oxlrec.py.

Binary form (what oxl trace --record writes)#

"OXLREC" 0x00 <version: 1 byte, currently 1>
<varint length> Header      (protobuf message)
<varint length> Record      (protobuf message), repeated until the end of the file

Every message is length-delimited like writeDelimitedTo of protobuf (a base-128 varint length, then the bytes). A reader refuses a version it does not know (unsupported .oxlrec version N); within one version new fields may be added, readers skip fields they do not know.

message Header {
  uint32 version         = 1;   // = the version byte
  int64  created_unix_ms = 2;
  string source          = 3;   // the address the tool recorded from (`--cluster`)
  string service         = 4;   // the `--service` filter, "" = all
  string tool            = 5;   // "oxl trace --record"
}

message Record {
  uint64 seq             = 1;   // 1.. in the order the calls started
  string method          = 2;   // full gRPC path "/<package>.<Service>/<rpc>"
  bytes  request         = 3;   // the request message as received (the rpc parameters, OxL binary = protobuf encoding)
  repeated Meta meta     = 4;   // the ctx metadata as received after authentication (without credentials, see below)
  string user            = 5;   // ctx.caller (x-oxl-user) — the split key of `split by .User`
  string tenant          = 6;   //                          — the split key of `split by .Tenant`
  string trace           = 7;   // trace id (32 hex digits) — `oxl replay <traceId>`
  string node            = 8;   // the node that served it  — the split key of `split by .Node`
  int64  start_unix_ns   = 9;
  int64  duration_ns     = 10;  // handler time on the node
  int32  status          = 11;  // gRPC status code, 0 = OK
  bytes  response        = 12;  // the response message (status OK)
  string error           = 13;  // the error: detail type (`pkg.NotFound`) or the status message
}
message Meta { string key = 1; string value = 2; }

What is not recorded (D-6860): the authorization, cookie and x-oxl-token headers; streaming rpcs; calls between handlers of one process (only network ingress is journaled); requests answered from the idempotency store without running the handler.

Text form (fixtures)#

The readers also accept JSON lines: the first line is the header, every other line one record; request / response are base64, meta is a list of [key, value] pairs. oxl trace --record f.json writes this form; scripts/oxlrec.py to-binary|to-text converts.

{"oxlrec": 1, "created_unix_ms": 1793608644508, "source": "127.0.0.1:7000", "service": "", "tool": "oxl trace --record"}
{"seq": 1, "method": "/shop.Shop/search", "request": "CgVzaG9lcw==", "meta": [["x-oxl-user", "17"]], "user": "17", "node": "n0", "start_unix_ns": 1793608644509424192, "duration_ns": 81234, "status": 0, "response": "CgNoYXQ="}

Recording#

oxl trace --record traffic/search.oxlrec --cluster 127.0.0.1:7000 --for 2m --service shop.Shop

The tool asks every node (inspector action record.start, right debug.enable because the journal holds request payloads) to journal its incoming unary calls for the period (a ring of 64 MiB per node), pulls the records every 500 ms (inspector section records), stops the recording and writes the file sorted by start time. Credentials: the same as oxl inspect --cluster (--dev-pki-dir, --cluster-cert/-key/-ca, --token).

Replaying#

• oxl replay f.oxlrec --cluster a sends every record to the node a (method, request, recorded metadata) and compares status and response with the recorded ones: same, or DIFFERS: status 0 -> 5 / field 2: "x" -> "y" (protobuf fields of the response). The exit code is 1 when anything differs. A caller identity in the metadata is only trusted by the node with credentials (--token).
• oxl replay <traceId> --cluster a takes the records of that trace from the nodes' journals (it must have been recorded).
• test ab replays the file in the test process through the program's own rpc handlers (oxl test --kind ab), see [С §18.5] and docs/decisions.md D-6850..D-6858.

Source: docs/oxlrec.md