Benchmarks

Performance targets of TZ 3.2 and how they are measured. Run scripts/bench.py (see its --help); the tables between the markers below are regenerated by it, everything else is maintained by hand.

• benchmarks/<name>.oxl and <name>.cpp do identical work and print the same checksum.
• scripts/bench.py builds OxL with --profile prod and C++ with clang++ -O2, runs each N times (default 5), compares the median CPU times and the checksums. --update-baseline stores the ratios in benchmarks/baseline.json; --check fails when a ratio grew by more than 5 % (TZ: a regression above 5 % blocks the merge).
• Methodology and the changes made for it: D-1700..D-1705 in docs/decisions.md.

Generated by scripts/bench.py (median CPU time (user+sys) of 11 runs, prod profile vs clang++ -O2).

Machine: Intel(R) Xeon(R) Processor @ 2.80GHz, 4 cores, 15.7 GiB RAM; Linux-6.18.44-fc-v230-x86_64-with-glibc2.39; Ubuntu clang version 18.1.3 (1ubuntu1); measured 2026-11-03.

Compute benchmarks (TZ 3.2: ratio must be <= 1.15)#

Geometric mean ratio: 0.92; benchmarks within target: 8 of 10.

Other TZ 3.2 numbers#

Runtime gtest output (test_runtime --gtest_filter='*Bench*:*MillionTasks*:*ManySuspendedTasksStayCheap*'):

1M tasks spawned+joined in 771 ms (771 ns/task); RSS before 13388 KiB, peak 288176 KiB, after 280364 KiB
10000 started+suspended tasks: 4096 bytes of RSS per task
all workers:   create 131 ns, switch 49 ns per context switch (98 ns per 2-task yield), solo yield 96 ns
one worker:    create 120 ns, switch 47 ns per context switch (94 ns per 2-task yield), solo yield 93 ns

Accepted deviation from TZ 3.2: bintrees (D-8258)#

By the customer's decision of session 6, bintrees (ratio about 1.2 against the 1.15 target) is a known, accepted deviation for 1.0: the remaining gap is the memory layout of nested lists (see the table below), and the fix (by-value List headers in the parent or a proven-sound arena) changes the layout of every collection type. Planned for 1.1. The extra row nbodylist (lists instead of fixed arrays, about 1.23) is informative only and has the same cause; the TZ row nbody meets the target on fixed arrays ([Body; 5] against std::array).

Where OxL is still slower than 115 % of C++ -O2, and why#

Measured with a Release build of the toolchain (the runtime library is linked into every program; a Debug runtime made string/map benchmarks several times slower and must not be used for these numbers, D-7701), pinned to one core with taskset, oxl and C++ runs alternating, medians of 11 runs (scripts/bench.py -n 11). The machine is shared (load average 6-9 during the final run, 10-45 at other times of the session), so single ratios move by +-10 %; the table above is the last such run.

Stage 6 start -> after agent E (D-7680..D-7701) -> after agent E2 (D-7702..D-7712):

Hello world (fn main() io { print("hello") }, --profile prod): 6.10 MiB (dynamic libstdc++) / 16.05 MiB (-static) before D-7702, 0.19 MiB / 1.15 MiB after (see the notes below).

Within target: nbody (arrays), intloop, sieve, strings, fib, sort, closures (the C++ twin pays for std::vector reallocations in push_back), wordcount (at the limit).

nbody uses fixed-size arrays on both sides since D-7711 ([С §3.3]: [T; N] with a comptime-constant N; C++: std::array<Body, 5>): same work, same checksum, and the case in which the C++ optimiser knows the length is exactly the case the array type gives OxL's optimiser. The list version is kept as nbodylist so that the cost of the list indirection stays visible.

What was optimised in this stage#

• D-1701: preemption poll inline (volatile load, cold call) instead of a call per iteration.
• D-1702: List index / len / push / for expanded inline in IR (nbody 5.5 -> 1.5, sieve 4.2 -> 1.3, closures 1.3 -> 0.5).
• D-1703: List<intN>.sort() with an inlined std::sort (1.8 -> 0.96).
• D-1704: cheaper oxl_alloc, no zeroing on list/string growth, plain integer formatting.
• D-7680/D-7681: strip-mined poll: an innermost counted loop polls once per 1024 iterations (bound documented in the decision), the poll call does not clobber list headers. intloop 1.39 -> about 1.0.
• D-7682: TBAA for list headers/slots/elements, data loaded with len.
• D-7683: interpolation straight into one buffer; D-7684: Map<str,V> inlined hash/eq, borrowed keys (no atomic retain/release pair per lookup).
• D-7685/D-7686: size-class allocator with per-thread free lists, smaller first list buffer (bintrees 3.9 -> about 1.2).
• D-7702/D-7703: the program entry no longer refers to the cluster, ORM, HTTP/2, TLS, ... directly (weak hooks), --gc-sections + -ffunction-sections, --as-needed: hello world 6.1 MiB -> 0.19 MiB, --static 16 -> 1.15 MiB; oxl build --static.
• D-7704: the SLP vectorizer is on in the driver's pipeline as in clang -O2 (nbody on arrays 1.36 -> 1.11).
• D-7710: lists of small elements carry their first 16 bytes in the header block (one allocation per tree node instead of two).
• D-7712: the variable of for x in xs borrows the element when that is provably unobservable (a list of 100 000 strings, 300 rounds: 538 -> 67 ms; the C++ twin with const auto&: 52 ms).
• Not done: @arena (D-7700, unsound without escape analysis), a borrowed loop variable for call results / patterns / maps.

Notes on the other TZ 3.2 numbers#

• Compile speed: measured on a generated file of 20 000 small functions (about 160 000 lines), dev profile, one compiler thread (the driver is single-threaded), including linking: about 90 000 lines/s, above the 50 000 lines/s (8 cores) target even on one core of a 4-core VM. Real code with more generics/closures will be slower; the figure is for a synthetic file.
• Hello world: the default prod link (dynamic libstdc++, with the line tables of --debug-info line) 0.19 MiB, 0.14 MiB stripped; oxl build --static 1.15 MiB (glibc, libstdc++ and the used part of the runtime inside), -static-libstdc++ -static-libgcc 0.32 MiB; all below the 3 MiB of the TZ. At stage 1 it was 1.25 MiB static, at the start of stage 6 6.1 MiB / 16 MiB: the program entry referred to every optional subsystem (D-7702). The ctest hello_size keeps the three variants below 3 MiB and checks that a hello world links none of the cluster / ORM / TLS / HTTP/2 code. A prod program that serves nothing has no inspector, debugger or OTLP exporter in it (D-7702).
• Cold start (hello world, spawn to exit): about 3.6 ms dynamic / 2.3 ms static, far below 100 ms.
• Scheduler (gtest Sched.BenchCreateAndSwitch, MillionTasks, ManySuspendedTasksStayCheap): with one worker create 95 ns and switch 46 ns (targets < 200 ns and < 50 ns; on this VM switch hovers around the 50 ns limit, 46-61 ns between runs). With all four workers contending the numbers are worse and noisy (create 190-730 ns, switch 60-150 ns, "all workers" mode measures cross-thread wake-ups on a shared VM). An idle suspended task costs about 4.4 KiB of RSS (target <= 8 KiB). 1M tasks are spawned and joined in 0.6-1.2 s with about 165 MiB peak RSS. The create target is therefore met only in the single-worker configuration on this machine; the multi-worker figures should be re-taken on the 8-16 core reference hardware of the TZ.

rpc targets of TZ 3.2 (D-3800..D-3804)#

Run scripts/bench.py --rpc (also part of a full run). Sources: benchmarks/rpc/{inproc,server,client}.oxl.

• Unary rpc in the process (inproc.oxl): a service in the same program, 1M calls of Bench.ping(i) (handler x + 1), after a 20k warm-up, on one core; the program prints picoseconds per call. The call is the full oxl_rpc_run path (route, node drain check, ctx, deadline, panic containment), no serialization.
• Throughput over the network (server.oxl + client.oxl): the server is taskset to the first S cores and listens on 127.0.0.1:0; the load generators are separate OxL processes (client.oxl), one per remaining core, each taskset to its core, each with its own h2c connection and 16 callers multiplexing unary streams on it (the runtime keeps one HTTP/2 connection per target string, so all callers of a process share one). Each caller does 5000 calls; a warm-up pass runs first. Reported: summed calls/s of all generators, the server's CPU seconds (/proc/<pid>/stat) during the run, and rpc per server-CPU-second. Only 4 shared cores are available, so the 500k rpc/s on 16 cores cannot be measured directly. The honest figure is the per-core one: with the server on one core the generators saturate it (about 0.98 cores used), and the 16-core number is a linear extrapolation (16 x per-core), i.e. it assumes no contention between workers and 16 connections (one HTTP/2 connection is served by one task, so a real 16-core run needs at least 16 connections). The row with two server cores is limited by the two remaining client cores (the client costs about as much per call as the server).
• Cold start (server.oxl): wall time from fork/exec of the node to the first grpc.health.v1.Health/Check reply SERVING over h2c (a minimal HTTP/2 client in bench.py), median of at least 10 starts after 2 warm-ups. Includes dynamic linking, runtime start, service registration, the listener and one request.
• Noise: the machine is shared with other builds (load average 5-7 during the measurements; the figures vary by +-30 %); medians of 5 runs, min-max printed.

Before / after (same machine, same noisy conditions; "before" = commit 42845fe):

What was changed (all behaviour-preserving, see the decisions):

• D-3800: a handler runs on the caller's stack when it needs no task of its own (no deadline, caller not cancelled / shielded, stack room >= 192 KiB, small result); otherwise the previous spawn path is used. This removes the task spawn, the scope, two context switches and a few allocations from every local call, and the per-request task of a gRPC call (the network server already runs each request in its own task).
• D-3801: cheaper ctx (trace id without snprintf, node id cached), lock-free routing (oxl_rpc_route), the in-flight list without allocation and sharded.
• D-3802: the scheduler creates one worker per CPU the process may run on (sched_getaffinity), not per CPU of the machine.
• D-3803: TCP_NODELAY on accepted gRPC connections (the clients already set it).
• D-3804: HTTP/2 header vector reserve.

Remaining cost on the server side (callgrind of the server, about 25k instructions per request): nghttp2 frame/HPACK processing about 30 %, malloc/free about 25 % (header strings, per-stream objects, message buffers), ctx; the next steps would be a per-stream arena for header strings and std::string_view headers.

Where the half-load latency goes (D-8400..D-8402)#

The TZ target "p99 < 1 ms at 50 % load on top of the network latency" is not reached on this 4-core shared VM; the measured p50 is 0.30-0.40 ms and p99 1.5-4 ms (quiet VM to a VM with other jobs; the generated row below is one such run). Findings:

• The VM sets the floor. A C program that only does nanosleep(200 us) wakes late by a median of 83-98 us and by more than 1 ms in 0.2-0.3 % of the sleeps, on all cores at once, quiet or loaded. A plain C blocking TCP ping-pong over loopback between two cores (no scheduler, no HTTP/2) has p50 65-110 us and p99 0.37 ms on the quiet VM and 1.1-2.0 ms while other jobs run (the case during most of the measurements: load average 3.5-4.5 on 4 cores). Only a spinning ping-pong avoids it (p50 8-31 us).
• An OxL rpc has four such blocking wake-ups on its critical path: client completion thread (io_uring), client worker (futex), server completion thread, server worker. One idle caller sees p50 about 200 us (C nghttp2 client against the same server: 180 us), 27-30k rpc/s p50 350-400 us.
• The tail is the server worker. With 16 callers per connection one stalled wake-up of the single server core delays all calls in flight: the slow calls of all callers of all three load processes line up at the same instants, and p99 is 1.5-2.5 ms on a quiet VM, 2-4 ms on a loaded one. One idle caller has the random single-hop tail instead (0.8 % of the calls above 1 ms).
• The measurement is sound: a C nghttp2 client (open loop) against the same server gives the same or a worse p50/p99, fewer callers per process or more processes only move the p50, the load processes are not saturated (one core each, 10k rpc/s).
• Server side at 50 % load (/proc/<pid>/task): about 0.46 context switches per rpc and 16 us of CPU per rpc, the worker of the single-core server is busy 95 % of the wall time because an idle worker spins (48 pause + 32 sched_yield) before it parks on the condition variable.
• Done: D-8400 (a send / recv that can complete at once does not go through the ring: server context switches per rpc 0.46 -> 0.17, CPU per rpc -2..-5 %, latency unchanged). Tried and dropped: longer worker / completion-thread spinning (worse p50, same p99, a core burnt), notify_one after the unlock of idleMu, pause instead of sched_yield. The remaining lever is removing the completion thread from the critical path (workers reaping the ring themselves): a redesign of the idle loop, not done.

Generated by scripts/bench.py --rpc (prod profile, 3 runs per figure, median; min-max in brackets). Machine: Intel(R) Xeon(R) Processor @ 2.80GHz, 4 cores (shared VM, other jobs may run: treat the numbers as noisy), Ubuntu clang version 18.1.3 (1ubuntu1); measured 2026-11-03.

Storage engine (D-4040)#

OXL_STORE_BENCH=1 build-rel/tests/unit/test_runtime --gtest_filter='StoreBench.*' (Release; OXL_STORE_BENCH_N sets the key count, default 200000). Real file system (ext4 on a virtio disk), default Options (4 MiB memtable, 2 MiB files, 10 bits/key bloom, 8 MiB block cache), 100-byte values, 17-byte keys, one thread, 4 shared cores (other jobs may run: treat as noisy). Machine: Intel Xeon @ 2.80 GHz. Measured 2026-10-30, single run.

Notes: the fsync figure is the disk's fdatasync latency (~3.4 ms here), not the engine; concurrent writers share one fsync (group commit, covered by StoreDbPosix.GroupCommitFromManyThreadsAndReopen). Write amplification in this run: 62 MB compacted for 40 MB of user data. Nothing here is tuned (single flush/compaction thread, no compression, software CRC32C); correctness and recoverability came first.

Source: docs/benchmarks.md