OxwaldLink — architecture

Implementation of the OxL language (standard: docs/spec/oxl-standard.md, cited as [С §N]; implementation brief: docs/spec/tz.txt, requirement ids K1…, P1…, И1…).

Scope of the current effort: stages 1–2 of the TZ (language core + concurrency). Distributed features (services, cluster, storage, …) are parsed but rejected by sema with not-implemented.

Layout#

Pipeline#

source ─► Lexer ─► tokens ─► Parser ─► AST ─► Sema ─────────────► typed AST ─► Codegen ─► LLVM IR ─► object ─► link (clang) + liboxlrt.a
                                             ├ Resolve (modules, imports, names)
                                             ├ TypeCheck (types, inference, overloads, match exhaustiveness)
                                             ├ Effects (§6.2)  ├ Errors (§6.5)  ├ Ownership (§7, @must_close)
                                             └ Lints (ч. III.4)

• Comptime reflection (reflect, inline for, x.[f]) is expanded in sema right before a body is type-checked (sema/Reflect.cpp, D-1900..D-1914); generic functions that reflect on their type parameters are checked per instantiation and codegen calls the clone (Sema::reflectCloneFor).
• Diagnostics are never printed directly; everything goes through DiagEngine.
• Each pass is a separate translation unit with its own unit tests.

Value model (codegen ↔ runtime ABI)#

Every type used in a collection gets an OxlTypeDesc constant emitted by codegen (size, align, drop, clone, eq, hash, cmp, show).

Panics unwind with the C++ exception ABI (__gxx_personality_v0); functions that own values needing drop or have defer use invoke + cleanup landing pads.

Lazy iterators (D-2000..D-2005)#

iter<T> is a synchronous, lazy, single-pass sequence: a pull closure fn() -> T? (the representation of a function value, BuiltinKind::Iter lowers to closureType()), so the whole chain lives in one reference-counted environment and drop releases every captured source. The library is plain OxL: std/core/iter.oxl (adapters and consumers written with fromFn and next), std/core/iter_numbers.oxl (generated by scripts/gen_std_iter_numbers.py: sum/product), four intrinsics in compiler/src/codegen/CGQuery.cpp.

• Ownership: an iter is not Copy, adapters and consumers move it, functions passed to adapters are escaping; list.iter() copies the list, so an iterator never borrows (nothing to escape). for x in it borrows and advances the iterator (CodeGen::emitFor, kind IterK); the temporaries of the iterable belong to the loop scope.
• for also accepts user types with next() -> T? or iter(), and I: Iterable<T> type parameters (Sema::elementTypeOf).
• Query expressions (§5.9) are lowered by Sema::checkQuery into adapter calls (QueryExpr::lowered); the clause expressions are moved into lambdas, the binding pattern is copied per clause, so the usual type, effect and ownership checks apply. order by/group by are barriers that drain their input at the first pull.

Profiles (ч. III.2)#

debug-info is full (DWARF 5 with variables and types), line (line tables only) or none; override with [profile.<name>] debug-info = "…" or oxl build --debug-info none|line|full. [profile.<name>] lto = true links with -flto (bitcode objects, lld). Builds are reproducible (byte-identical). See D-1400..D-1403.

Concurrency runtime (stage 2)#

M:N stackful tasks on a work-stealing pool (one worker per core), context switch in assembly (x86-64, aarch64), stacks = reserved virtual memory with a guard page and on-demand commit. File I/O through one process-wide io_uring plus a completion thread (Linux, D-1600..1604; blocking-pool fallback, kqueue not done). Preemption of tasks running >10 ms via a monitor thread + signal. scope, cancellation, deadlines, channels, select, local actors.

Serialization (stage 3, [С §9], D-2100..D-2114)#

Codecs are descriptor driven: instead of one encoder/decoder function per type the compiler emits a constant graph of OxlSerNodes (runtime/include/oxl_ser.h) and liboxlrt (runtime/src/ser_{bin,json,common,api}.cpp) walks it against a value in its lowered memory layout. Offsets and sizes come from the LLVM data layout, so the runtime never guesses. One implementation serves both formats (binary = protobuf 3 wire encoding, JSON = canonical protobuf JSON), the node graph is the inspectable schema, and decoding is all-or-nothing (the partially built value is dropped on error).

How a later task uses it (rpc, topics, storage): registerWireType(T) so the type is persisted in the lock, isSerializable for the compile error, then call emitEncode(T) / emitDecode(T) from generated code. For a type with a node graph ser::schemaOf(sym).fields[i].tag is the field number a .proto exporter must print; CodeGen::serNode shows the exact kinds (message, enum, oneof, union, repeated, map, wrap, repr, codec). The node ABI is pinned by tests/unit/codegen/serial_test.cpp.

Memory facts the runtime depends on (all in the node): a union is { i32 tag, payload at +8 } with the tag a type id (none has the id of none, none_discr); an enum with fields is { i32 variant index, payload at +8 }, a fieldless enum an i32 discriminant (an @open enum has the automatic Unknown(tag: u32) and therefore a payload); str/bytes are OxlStr*, List OxlList*, Map/Set OxlMap*. Custom codecs (@codec) and @repr are thunks the codegen emits next to the node (CODEC / REPR kinds).

oxl.schema.lock is handled by the driver (Compilation::check): read next to oxl.toml, updated by oxl build/run, --locked and --against options; sema builds the lock in finishSerialization and compares it with the previous one (schema-break). Interop with other implementations is tested against protoc in tests/ser_interop/ (CTest ser_interop). MessagePack and CBOR (D-7600) are one walker in runtime/src/ser_doc.cpp (a Writer per format on the way out, a parsed document tree on the way in); the same walker drives level-3 user formats (D-7620): the compiler passes the dyn Encoder of Format.encoder() (data pointer + vtable) to oxl_ser_encode_custom, which calls the vtable slots in the order of std/ser/ser.oxl (OXL_ENC_*). @preserve_unknown (D-7610) adds a hidden OxlStr* slot to the lowered struct (CodeGen::hasUnknownSlot), flagged OXL_SN_PRESERVE in the node. Not done: decoding with a user format (D-7621), decode_view, ActorRef addresses, oxl export itself.

Network I/O (stage 3, D-2200..D-2207)#

Layers, bottom up:

C++ API contract (#include "net/net.h", all in oxl::rt::net):

• A socket is an int fd; Socket is a move-only owner (closes with net::close). Calls return >= 0 or -errno (name lookup: -(kDnsErrBase + EAI_*)); errorText(err) / classify(err) give text and the NetError class.
• socketOpen, bind, listen, setOption(NoDelay|ReuseAddr|ReusePort|KeepAlive|V6Only|SendBuf|RecvBuf), localAddr, peerAddr, shutdown, close, adopt(fd) never park. accept(fd, &peer, deadline), connect(fd, addr, deadline), readSome, writeSome, writeAll, readExact, resolve(host, port, out, deadline) park only the calling task (a plain thread blocks) and are cancellation points. Convenience: listenTcp, listenUnix, connectTcp, connectUnix, connectHost.
• deadline is absolute monotonic ns (deadlineAfter(timeoutNs); 0 = none). Pending at the deadline -> -ETIMEDOUT; an expired deadline fails at once. A cancelled task (or one past its chain deadline, outside shield) gets Cancelled thrown from the call; RAII Sockets and defers unwind normally.
• Addr wraps sockaddr_storage (parse, parseIp, unixPath, host(), port(), str()); readSome returns 0 at end of stream; sends use MSG_NOSIGNAL. A descriptor must not be closed while another task waits on it (ring mode: shutdown first).
• For TLS: read/write the raw fd through readSome/writeSome from the BIO callbacks; they are safe to call from any task and from the blocking pool (plain threads use poll).

Static configuration and secret<T> (stage 3, D-2700..D-2710)#

config Name { field: T = default … profile p { field = v } } is a record type plus Name.get() -> Arc<frozen Name>:

Sources, precedence, naming and text forms: D-2701..D-2705. Errors (exit code 78): D-2706. get() ordering: D-2707.

Динамический конфиг и флаги (stage 4, wave 6 D, D-5190..D-5203)#

dynamic Limits { rps: u32 = 100 newSearch: Flag = Flag.off } ([С §14.2]) — значения, которые живут в кластере и меняются без перезапуска. Весь путь:

Цель распространения ≤ 1 с, best effort: на нагруженном 4-ядерном хосте с 3 процессами измерено 0,2–0,7 с до значения на всех узлах. Тесты: tests/unit/runtime/dynconf_test.cpp (иерархия, детерминизм хеша флагов, отказ уточнению, история и откат, хранилище, очередь запросов, control-строки), conformance/§14.2/dynamic-*.oxl (язык), tests/dynconf/test_dynconf_cluster.py (ctest cluster_dynconf: oxl dev --nodes 3, set на одном узле виден на всех, .changes() срабатывает, откат, отказ по политике с аудитом, флаги). Решения и пробелы: D-5190..D-5203.

static globals (stage 3, D-3480..D-3483)#

static NAME: T = e is a lazily, once-initialized global (a similar lazily-initialized global to the config box, but per static and initialized on first access). Sema: Sema::isStaticSafe (global-mutable for non-frozen/non-sync T, D-3481), assignment is not-assignable. Codegen (codegen/CGStatic.cpp): per static a slot global, a once-cell { i64 state, i64 owner } and an oxl.static.NAME.init function; an access is load acquire state == 2 ? slot : oxl_static_once(cell, init, name, loc) and yields the slot address (emitPlace) or a clone of it (emitStaticValue). Runtime (runtime/src/static_once.cpp): CAS UNINIT -> RUNNING, the winner runs the initializer, others oxl_task_yield() until DONE; the same task re-entering panics (recursion), a panicking initializer poisons the cell. The value is never dropped. Literal initializers skip all of this (CodeGen::isTrivialStaticInit).

TLS (stage 3, D-2400..D-2406)#

TLS 1.2/1.3 sessions layered on oxl::rt::net sockets with memory BIOs, so socket I/O stays task-aware (io_uring or the epoll poller), honours deadlines and is a cancellation point (D-2202). The library is the OpenSSL-compatible libssl API; CMake option OXL_TLS_BACKEND=openssl|awslc (default openssl >= 3.0; awslc + OXL_AWSLC_ROOT compiles with OXL_TLS_AWSLC guards, untested).

C++ API contract (#include "net/tls.h"):

• Context::create(Options, &error) loads trust anchors (system roots or caFile/caPem), identity (certFile|certPem leaf + intermediates, keyFile|keyPem, keyPassword), ALPN list, versions (minVersion/maxVersion = 0x0303/0x0304), clientAuth (server mTLS), pinSha256, verifyPeer (false = insecure, tests), verifyHostname. It is immutable and shareable between sessions and threads.
• Session(ctx, net::Socket, serverName) owns the connected socket. handshake(deadline), read, readExact, write (all bytes), shutdown (close_notify, half close), closeNow (never parks), then alpn() (e.g. "h2"), version(), cipher(), sni() (server), peerSubject(), peerCertDer(), peerCertSha256(), verifyResult(), lastError()/lastErr(). Results are >= 0 or -errno / -(200000 + Err) (tls::code(Err::X)); tls::errorText, tls::classify, tls::isTlsError.
• Concurrency: one reader plus one writer concurrently (separate tasks), more of either queue; read returns 0 only for a received close_notify (Err::Truncated otherwise); a read timeout is recoverable, a failed write or handshake poisons the session.
• For HTTP/2: offer alpn = {"h2", "http/1.1"} (client) or support them in preference order (server) and check session.alpn() == "h2"; TLS 1.2 with h2 relies on libssl's default cipher list (no cipher restriction is applied).
• Tests: tests/unit/runtime/tls_test.cpp (in-memory CA from tls_test_certs.h; both io backends; interop with openssl s_client/s_server when installed), tests/unit/codegen/tls_test.cpp, conformance §II.1/tls-*.oxl (fixtures in tests/fixtures/tls, test only).

Standard library MVP (stage 3, W3-L, D-3100..D-3160)#

How the new std modules bind to the runtime, and where each piece lives:

rt: intrinsics (D-3100). @intrinsic("rt:oxl_xyz") on a body-less OxL function calls the C function oxl_xyz directly; the OxL signature fixes the C one (integers/bool as int64_t, floats as double, str/bytes as borrowed OxlStr*, &var T as a pointer, results int64_t/double/owned OxlStr*/void). Adding a runtime function needs no compiler change: declare it in the module and write the C++ (CGRtCall.cpp). Anything richer (errors, optionals, records) is built in OxL, as std.net does.

ORM tables. CGOrm.cpp emits one constant OxlOrmTable per @table type used by a program: SQL name, the type's OxlSerNode (std.ser: field names, offsets, kinds), the column flags (@pk, @auto) and the indexes. The runtime reads and writes a row through the node, so scalars become columns and everything else a serialized column without per-type generated code. Queries over db.T are translated in sema (checkTableQuery) to SQL text with ?N parameters; the parameters are ordinary expressions bound at run time.

Dependencies added to every OxL program: libpcre2-8 and libsqlite3 (scripts/install_deps.sh); libssl/libcrypto were already linked for TLS.

Streams and Broadcast (stage 3, [С §10.3, §10.4], D-2600..D-2619)#

stream<T, E = never> is a single-consumer, affine, counted handle (OxlStream*) of an asynchronous sequence that ends normally or with one error value of E. Local streams pass elements without serialization; the same object type is meant to carry remote streams (wave 3).

Generator. fn gen(a, b) -> stream<T, E> { … yield x … } compiles to a closure body (emitClosureFunction, captures = the moved parameters) and an entry thunk that runs it and hands an error result to oxl_stream_fail. The function itself only calls oxl_stream_new_gen(elemDesc, errDesc, entry, env, oxl_env_release); nothing runs until the first oxl_stream_next, which spawns the body as a producer task (node background scope). Protocol (demand driven, buffer = one element): a pull sets want; the producer runs to its next yield, which stores the element in the slot and suspends until the slot was emptied and a new pull arrived. next returns OXL_STREAM_ITEM, OXL_STREAM_ERROR (once, then END), OXL_STREAM_END, and next_within also OXL_STREAM_TIMEOUT (the demand stays). Every wait is a cancellation point. oxl_stream_close (drop glue; the last of the handles oxl_stream_retain made) cancels the producer, waits until its task is finished (defers and drops have run), drops an undelivered element and releases the environment. A producer panic is re-raised in the consumer's pull.

Source streams: the hook for wave 3. oxl_stream_new_source(elemDesc, errDesc, &vtable, ctx) makes a stream whose elements come from callbacks that run on the consumer's task and may park it:

typedef struct OxlStreamSource {
  int32_t (*pull)(void* ctx, void* elem, void* err);   // ITEM / ERROR (once, then finished) / END
  void    (*close)(void* ctx);                          // exactly once, when the last handle is dropped
  void    (*cancel)(void* ctx);                         // optional: oxl_stream_cancel
  uint64_t (*lagged)(void* ctx);                        // optional
  int32_t (*pull_within)(void* ctx, void* elem, void* err, int64_t deadline_ns);  // optional: TIMEOUT
} OxlStreamSource;

A remote server-stream rpc is such a source: pull reads the next message (parking the task, deadlines and cancellation as in D-2202) and decodes it straight into elem (lowered layout of T); a transport failure or trailing status is written to err as E | NetError (the compiler passes the descriptor of that union); close cancels the call. A client-stream/bidi parameter input: stream<A> is consumed by the server glue with oxl_stream_next in a task that encodes onto the wire.

Broadcast (oxl_bcast_*): a hub with one ring of Shared<T> values (Arc<frozen T>, one pointer shared by reference count) and a cursor per subscription. publish consumes the element; .Block parks the publisher while a live subscriber is full, .DropOldest advances a full subscriber's cursor at publish time (lagged is exact), .Disconnect lets it deliver what it holds, then ends with Lagged { missed }. A subscription is a source stream. One mutex per hub; publishers and subscribers may be tasks on any worker or plain threads. Network fan-out should subscribe one local stream per client and encode each element once, keyed by the Arc pointer. Pipelines (x |> map(f) |> take(3)) are chains of generators; dropping the end of the chain cancels the stages one after the other.

Services and rpc (stage 3, D-2900..D-2914)#

service / rpc / impl / dep / provide / cluster / policy are compiled by sema/ServiceSema.cpp + ServiceCheck.cpp (data for codegen: oxl/sema/Service.h), codegen/CGService.cpp, and the runtime runtime/src/rpc/rpc.cpp (C ABI and descriptors: runtime/include/oxl_rpc.h). std side: std/core/rpc.oxl, std/core/node.oxl (prelude), std/rpc, std/cluster.

Per rpc codegen emits (names oxl.rpc.<pkg.S>.<m>.*): tramp(args, result) (the handler task body: clones the request block, calls the handler), drop, res_clone/res_drop/res_is_error, key* (cache/idempotency/rate-limit keys), call (client side: route, run, retry, outcome → NetError), wire (server side of a transport) and the OxlRpcMethod descriptor.

Seam for the network transport (W3-J2)#

• Server. oxl_rpc_find("/pkg.Service/method") gives the OxlRpcMethod: kind (unary / server-stream / client-stream / bidi), flags (readonly, idempotent, public, internal, resumable), timeout_ns, request / response = OxlSerNode message nodes (field 1 of the response is the value; ser::schemaOf(S.requestType) is what an oxl export .proto generator reads). m->wire(&inbound, bytes, len, &reply) decodes the request, runs the full call path (oxl_rpc_run: ctx from inbound — trace id, metadata, x-oxl-user, grpc-timeout as deadline_ns) and fills OxlRpcReply: outcome, gRPC status, body (encoded response) or detail_type / detail (typed error detail), retry_after_ns. Streaming kinds go through OxlRpcStreamOps instead (see below). Call oxl_node_set_listening(true) once the listener accepts clients (a served service then keeps the node alive after main, D-2907).
• Client. A call site asks oxl_rpc_route(m, clientKind, target) (0 local, 1 transport, 2 nobody → Unavailable; kind 0 = S.m(), 1 = S.at(node), 2 = S.connect(addr)). Route 1 encodes the request message, calls oxl_rpc_transport_call(m, kind, target, req, &reply, &detailType, &detail) and decodes the reply (OK → response message; APP_ERROR → decode detail as the error type named detail_type; other outcomes → NetError). The transport installs itself with oxl_rpc_set_transport(fn); without one the route is Unavailable.
• Test seam. OXL_RPC_LOOPBACK=1 installs a transport that calls wire in process and routes local calls through it; std.rpc.callRaw / callStatus / callDetails call wire directly. tests/unit/codegen/service_test.cpp runs the same program both ways.

Runtime notes: handlers are tasks of a private scope of the call, so cancelling the caller (or its deadline) cancels the handler once and the call returns after it finished; ctx is a refcounted block in Task::rpcCtx inherited by children (SpawnInit lets a call give its handler its own); the node state (roles, drain flag, registered services, in-flight set) is process wide. Open: streaming rpcs, middleware attributes, enforcement of policy / @allow, the built-in storage behind @idempotent (stage 4).

Services over the network and oxl export (stage 3, W3-J2, D-3000..D-3042)#

Stream parameters, @resumable, encode once (wave 1 A, D-3600..D-3604)#

oxl export --proto [path] [--out dir] [--all] writes one file per module (shop/catalog.proto for module catalog of package shop, package shop.catalog) and oxl/types.proto for Date, LocalDateTime, ActorRef. Not done: --openapi, at(node) / cluster routing, ACME, mTLS between nodes (stream parameters, @resumable and encode-once are in the next section).

HTTP/2 and gRPC (stage 3, D-2500..D-2550)#

HTTP/2 over net sockets or TLS sessions with nghttp2 for framing/HPACK, and gRPC on top of it (framing, status, metadata, deadlines). Payloads are opaque protobuf bytes: the compiler's service/rpc codegen encodes and decodes, this layer moves them.

Model#

• One h2::Conn per connection: a reader task (transport -> nghttp2_session_mem_recv) and a writer task (mem_send -> transport), so writes are serialised by construction. All nghttp2/stream state is behind one mutex never held across a parking call; waits use h2::Cond. Connection tasks live in the background scope with a cleared deadline.
• Flow control = backpressure (D-2502, D-2508): receive windows are returned to the peer only when the application reads (Stream::read); Stream::write / gRPC send park while more than 256 KiB are queued behind the peer's window. A slow consumer therefore suspends the producer; other streams of the connection keep flowing (connection window 4 MiB). @window(n) (default 64) is a message-credit window on top: between two OxL runtimes (SETTINGS 0x4f58, frame type 0xf0, ignored by every other implementation) the producer parks before message n+1 until the consumer's application has taken one out, so it is never more than n messages ahead; towards foreign peers it degrades to "n messages queued".
• Cancellation (D-2503): RST_STREAM from the peer, a lost connection or a drain that gives up cancels the handler task bound to the stream; a gRPC server call runs with the client's grpc-timeout as the chain deadline of its task (§8.6), so every wait inside the handler is bounded and a passed deadline ends the call with DEADLINE_EXCEEDED. A client task that is cancelled (or past its deadline) resets its call's stream.
• Drain (D-2504, D-2522): Server::shutdown(grace) / Conn::drain = GOAWAY(2^31-1), PING, GOAWAY(last), wait, reset leftovers, close. Needed for SIGTERM.

• Status mapping (D-2511, §10.1) — grpc::codeFromNet / netClassFromCode, C: oxl_grpc_code_from_net / oxl_grpc_net_from_code:

  NetError	gRPC status	back
  Timeout	DEADLINE_EXCEEDED (4)	4 -> Timeout
  Unavailable	UNAVAILABLE (14)	14 -> Unavailable
  Cancelled	CANCELLED (1)	1 -> Cancelled
  AccessDenied	PERMISSION_DENIED (7) or UNAUTHENTICATED (16), caller's choice	both -> AccessDenied
  ProtocolError	INTERNAL (13)	every other code (2, 3, 5, 6, 8..13, 15) -> ProtocolError

  Typed service errors are not NetError: the codegen sends @grpc_status(code) (default FAILED_PRECONDITION) with the error value as a detail (Detail = type URL + serialized message inside grpc-status-details-bin, a standard google.rpc.Status).

C++ API (what the C ABI wraps)#

grpc::Server srv(opts);                                   // opts.tls null = h2c
srv.addMethod("/shop.Orders/Get", grpc::Kind::Unary, handler, user, {.maxTimeoutNs = 2s /*@timeout*/, .window = 0});
srv.listen(addr);  /* port 0 -> srv.localAddr().port() */  srv.shutdown(30s);  // GOAWAY drain, parks
void handler(grpc::ServerCall* c, void* user) {           // fresh task; chain deadline = call deadline
  std::string req; c->recv(&req);                         // 1 message, 0 = client done, <0 = cancelled / malformed
  c->meta("x-oxl-user"); c->deadline(); c->setHeader(..); c->setTrailer(..);
  c->send(resp);                                          // parks under backpressure
  c->finish(Status{Code::FailedPrecondition, "msg", {{typeUrl, bytes}}});   // or return: OK
}
auto ch = grpc::Channel::create("10.0.0.5:7000", {.tls = ctx, .serverName = "orders"}, &err);
grpc::CallOptions co; co.timeoutNs = 2s; co.add("x-oxl-user", "42"); co.inherit(*serverCall);   // ctx propagation
Status st = ch->unary("/shop.Orders/Get", req, &resp, co);
auto call = ch->start(path, co, &st);                     // call->send / closeSend / recv / initialMetadata / status / cancel

All four rpc kinds use the same stream API; the Kind given at registration makes the framework pre-read the single request of unary/server-stream calls and enforce one response for unary/client-stream.

C ABI and a unary call#

oxl_grpc_* in oxl_rt.h (documented there with the same example). Sequence of Users.connect(addr).get(id) with a handler on the other side:

client task (generated code)        client reader/writer       server reader/writer          server call task (handler)
 oxl_grpc_channel_new(target,tls)
 call = oxl_grpc_call_new(ch,"/Users/get")
 oxl_grpc_call_set_timeout / add_meta
 oxl_grpc_call_unary(call, req, n)
   Channel::getConn -> h2::dial: TCP connect, TLS ALPN h2,
     SETTINGS exchange (awaitReady)
   open(HEADERS :path, grpc-timeout=1999999u,
        content-type, x-oxl-user) ----writer----> HEADERS ------reader-----> new stream, spawn task
   write(0x00 len32 payload) + END_STREAM ----> DATA(END_STREAM) --------> stream buffer          runCall: parse grpc-timeout,
   recv() parks (deadline = call deadline)                                                        oxl_deadline_push, find method,
                                                                                                  pre-read request, call handler
                                                                                                  oxl_grpc_call_recv -> request bytes
                                                                                                  oxl_grpc_call_send -> HEADERS(200)+DATA
                                      <-------------- HEADERS, DATA ------- writer <------------ (flow control: parks if the window is full)
   wakes: 1 message
   status() drains to the end          <-------------- HEADERS(END_STREAM: grpc-status=0 [, message, details-bin, trailers])
                                                                                                  handler returns -> finish(OK)
 returns the status code; oxl_grpc_call_response -> response bytes; oxl_grpc_call_free

On an error: the handler calls oxl_grpc_call_add_detail(c, "type.googleapis.com/shop.OutOfStock", bytes, n) and oxl_grpc_call_finish(c, 9, "msg"); the client gets code 9 from oxl_grpc_call_unary and reads oxl_grpc_call_detail_at(call, 0, &url, &bytes, &n). Transport outcomes map through oxl_grpc_net_from_code.

Gaps (D-2550)#

HTTP/1.1 client/server for std.http (h2 only, via oxl_h2_request), gRPC compression, reflection, keepalive/idle/max-age, retries/hedging and load balancing, interceptors, plain-HTTP routes on the server, HTTP/3.

Хранилище (stage 4, S1, [С §12.4, §12.8], D-4000..D-4099)#

Built-in storage of a node: our own LSM engine in runtime/src/store/ (no RocksDB/LevelDB), C ABI in runtime/include/oxl_store.h. Today it is a single-node engine; Raft (one group, then multi-Raft by shard) will sit above it as the replication of a state machine whose storage this is.

                       DB (db.cpp, db_impl.h)                    callers: C ABI (capi.cpp) -> std.kv, actors, journals, coordinator
   Write(batch, sync) ──► writer queue ──► leader: group commit ──► LogWriter ──► NNNNNN.log   (WAL, fsync per group if any member wants it)
                                              │                                      │
                                              └─► MemTable (skiplist, MVCC) ◄── Get/Iterator (lock-free readers, snapshot = sequence number)
   memtable full ─► switch: sync old log, new log, fsync dir ─► immutable memtable ─► background flush ─► NNNNNN.sst (L0)
   background (Env::Schedule): flush  >  manual CompactRange  >  highest-score level ─► leveled compaction L0 ─► L1 ─► … ─► L6
   VersionSet: immutable Version (files per level) + MANIFEST-N (edit log) + CURRENT (atomic pointer)

   Env (env.h): PosixEnv | MemEnv (deterministic, crash + fault model)      <- the only way the engine touches files, time, threads

Layers (each in its own file, each unit-tested):

Consistency model (what the tests check)#

• Atomic batches: all operations of a WriteBatch become visible together, are logged as one record and recover all-or-nothing.
• Durability: sync=true → on disk when the call returns. sync=false → a power loss may drop a suffix of the acknowledged writes, never a middle part (recovered state = a prefix of the write sequence). A write that returned an error may be applied or not (atomically), and the store turns read-only until reopened (the log may end in a torn record; appending behind it would hide later writes).
• Snapshots/iterators see exactly the sequence number they were created at, independent of flushes and compactions; compaction keeps every version a live snapshot can see and drops tombstones only when nothing older can resurface.
• Corruption (bad CRC, malformed block, missing file) is reported as Corruption for the affected read; checksummed SST data is never returned wrong.

Crash safety, concretely#

1. A new file is fsynced, then the directory is fsynced, only then is it referenced (new log before use, SST before its manifest edit, manifest before CURRENT).
2. CURRENT changes only by write tmp → fsync → rename → fsync dir; the previous manifest is deleted after that.
3. A memtable switch syncs the old log before creating the next one, so a torn record can only be at the end of the newest log with data.
4. Recovery replays logs ≥ log_number, writes the recovered tail to L0 and starts a fresh log in the same manifest edit that retires the old ones; it can be interrupted at any point and re-run.
5. Files that a failed manifest update may reference are kept; everything unreferenced is swept at the next open.

Deterministic simulation (test sim, [С §18.4])#

All file, directory, clock and background-thread access goes through Env. MemEnv models power loss (unsynced file tails and unsynced directory operations are partly lost, torn sectors), EIO on append/sync/read/directory operations, torn failed appends, disk full, a virtual clock, and runs background jobs inline or from a queue (Pump()), so a simulator can interleave crashes between flush/compaction steps. Randomness is a seeded splitmix64. The C ABI exposes the same knobs (oxl_store_env_*). tests/unit/runtime/store_model_test.cpp is the reference client: random operations against std::map with crashes and fault windows.

Tuning knobs (Options)#

write_buffer_size (4 MiB), block_size (4 KiB), bloom_bits_per_key (10), block_cache_bytes (8 MiB), target_file_size (2 MiB), level1_max_bytes (10 MiB, ×10 per level), level0_compaction_trigger (4) / level0_stop_trigger (12), max_group_commit_bytes (1 MiB), seed.

For the Raft/shard layers (not built yet)#

• One DB per shard replica (or one per node with a shard prefix in the key); the applied Raft index goes into the same WriteBatch as the applied command.
• The Raft log itself is a different access pattern (append, truncate prefix/suffix): plan a dedicated log store on the same Env rather than an LSM.
• Snapshots for follower catch-up: iterate a Snapshot (the engine has no checkpoint API yet, D-4099).

Gaps (D-4099)#

Reverse iteration, compression, range delete/TTL/merge, parallel background jobs, rate limiting, backup/checkpoint, mid-file corruption detection in the WAL/manifest (the first bad record ends the log), prefix bloom, wiring Db.open("builtin://…") of std.orm onto the engine (D-4050).

Симуляция (stage 4, [С §18.4], D-4100..D-4110)#

Details and the protocol-author guide: docs/testing/simulation.md.

• Kernel runtime/src/sim/: World is a single-threaded discrete-event simulator (virtual nanoseconds, events ordered by (time, seeded tie-break, seq)) with N nodes, a Network (latency/jitter, drop, duplicate, reorder, per-link overrides, symmetric / one-way / timed partitions), per-node Disks that survive crashes, kill/restart with incarnation numbers, clock skew, invariants, a Trace with a running hash (same seed -> same hash), a History + register linearizability checker, and a seed sweep runner. Protocol code (Raft, SWIM) is C++ Process classes using only NodeCtx (clock, rng, send, timers, disk).
• Why a separate executor (D-4100): the OxL scheduler is M:N, work-stealing, with a timer thread, io_uring and preemption by signal; making that deterministic means rewriting it. Cluster protocols are event-driven state machines anyway, so they run on the kernel; OxL code (the test body) stays on the real scheduler and drives virtual time through std.test.
• Seam to the OxL runtime (sim/session.h): while OXL_TEST_SEED is set (or after oxl_sim_begin) oxl_time_now_nanos, oxl_instant_now_nanos, oxl_rand_u64 and oxl_sleep_nanos use the session world (clock + seeded RNG; sleep = advance virtual time). Everything else of the runtime is untouched when no session exists (one atomic load).
• C ABI runtime/include/oxl_sim.h: oxl_sim_* (explicit world: nodes with a callback vtable, faults, run, trace hash) and oxl_simt_* (the implicit session world, bound to std/test/test.oxl by @intrinsic("rt:...") - no compiler changes).
• oxl test: sim tests are runnable; the runner starts one process per seed (OXL_TEST_SEED), stops at the first failure, re-runs it with OXL_SIM_REPORT=1 to confirm reproducibility and prints oxl test <pkg> --exact '<name>' --seed N. --seed N / --seeds a..b as in the Standard.
• Storage: Disk is the seam; SimDisk (own tiny file model) and MemEnvDisk (adapter over the LSM store's fault-injecting MemEnv).

К14: размещение и граф вызовов (stage 4, [С §12.1, §12.7, §13.4], D-4460..D-4479)#

Compiler-only part of the cluster (no cluster runtime needed). Code: compiler/src/sema/Placement.cpp, CallGraph.cpp, runtime/src/cluster/callgraph.cpp.

• Declaration (Sema::checkClusterDecl). Every clause of cluster { … } is read into ClusterInfo / ClusterRole (include/oxl/sema/Service.h): seeds (static[…], k8s(label:), dns(…)), port, auth (jwt, oidc, apikey, session, roles from claim(…) as Enum), defaults, assign by and per role serves, hosts, runs sagas / runs leader(n), journal, shards, replication, replicas, needs, expose … tls(cert:, key:) / tls(acme:). Unknown clauses and keys are errors. Two parser touches: seeds static[…] (a keyword) is kept as the call static([…]), and replicas 2.. as a range whose end is an integer literal with suffix open. assign by label("k") is rewritten to env("OXL_LABEL_K") (D-4464).
• Placement (Sema::checkPlacement), all errors with their own page in docs/diagnostics: service-not-served, actor-not-hosted (sharded actors), topic-without-journal, saga-without-role (runs sagas + journal), unknown-role (on role, provide … for role, literal assign by), missing-provide (also for the dep of a sharded actor, per hosting role), placement-conflict (shards/replication without journal, replication above replicas, two values of a label, one expose port in two roles, tls(acme) + tls(cert)), cluster-seeds, cluster-needs, cluster-auth, cluster-decl. Sagas and pipelines are recorded (ServiceState::sagas/pipelines) for these checks and then lowered by SagaSema.cpp (see "Саги и пайплайны").
• Call graph (Sema::buildCallGraph). FnSymbol::graphCalls records every call of a body, closures included (callSites skips closures). For each role the builder walks from the code that runs there — handlers of served services, messages of hosted actors, provide factories, main (every node), the generated bodies of sagas (roles with runs sagas) and pipelines (the calls of a step with on role only for that role) — through plain functions and actor message bodies; a call of a service rpc becomes an edge role -> pkg.Service.rpc, a message role -> pkg.Actor.msg. Conservative: no pruning by hasRole, interface calls and function values are not followed. A program without cluster has the role all.
• Allow table. Codegen embeds the edges as OxlCallEdge[] and calls oxl_callgraph_register at node start (runtime/include/oxl_callgraph.h: oxl_callgraph_allows(role, target), _callers, _edge, _count). Enforcement (mTLS role -> table lookup) belongs to the policy agent. missing-policy now means: not allowed to external principals, not @internal, and called by no role (test builds also count calls from tests).
• oxl export --call-graph [path] [--out file] prints the roles, edges (with the calling functions), the per-role allow lists and, per rpc, its permissions and the roles that call it, as JSON (Sema::callGraphJson).
• Routing hooks. S.at(node) already type-checks and lowers to Client { kind 1, node } (rpc.at); the runtime answers Unavailable for a foreign node until the coordinator exists. ActorRef<A> is an address in the wire schema (oxl.ActorRef, oxl/types.proto).

Raft (stage 4, wave 3 R, [С §12.4, §12.8], D-4300..D-4379)#

Our own Raft (customer decision: no NuRaft/etcd-raft; design ideas only). Code runtime/src/raft/ (namespace oxl::raft, part of liboxlrt), tests tests/unit/raft/ (binary test_raft), the API contract is the header comment of runtime/src/raft/host.h.

 K (std.kv, shard automaton)  ->  RaftGroup  (propose / read_barrier / membership / leader info)
                                      |
 F (cluster transport, SWIM)  <-> RaftHost  (many groups, one tick, one log store, coalesced frames)  -- HostEnv {self, send, defer}
                                      |
                       RaftCore  (pure: events in -> Ready out)        LogStore (segmented files on store::Env)
                                                                         LsmStateMachine (data + applied index in ONE WriteBatch)

• RaftCore (core.h/.cpp): one group, no clock, no I/O. Events: tick(), step(Message), propose, propose_conf_change, read_index, transfer_leader, campaign, compact, restore, report_snapshot. Effects: take_ready() returns a Ready (hard state, entries to persist, snapshot to persist, committed entries to apply, messages, read states, snapshot requests); the host persists, fsyncs, sends, applies, then advance(rd). Leader appends/heartbeats (early_msgs) may leave before the fsync, everything else (votes, append acks) only after it. The same object runs under the simulation (NodeCtx) and in production.
• Elections: pre-vote (a node that cannot win never raises the term), check-quorum (a leader that sees fewer than a majority within an election timeout steps down), leader lease for votes (a node that heard from a live leader recently ignores vote requests unless they carry the leader-transfer flag), randomised timeouts from a seed. Learners and unconfigured nodes never campaign or vote.
• Replication: per-follower Progress in Probe / Replicate / Snapshot. Appends are produced lazily at take_ready() so proposals of one turn share messages (batching); in Replicate the leader pipelines with an inflight window (max_inflight messages) and a byte limit per message; the uncommitted-bytes limit makes propose return Busy. Rejections carry a (hint_index, hint_term) so the leader skips whole conflicting terms. A new leader commits a no-op of its term first (entries of older terms are committed only through it - the figure-8 rule).
• Membership (single-step, no joint consensus): AddLearner -> catch up (PromoteVoter is refused with NotCaughtUp while the learner is more than promote_slack entries behind) -> PromoteVoter; Remove for anything. One change at a time, only after the leader committed an entry of its own term (the single-server-change fix). The configuration takes effect when the entry is appended and is recomputed from the log base on truncation. A group is bootstrapped by identical AddVoter entries (index 1..N, term 1) on every initial member, so a later joiner replays the membership from the log. A removed leader steps down once the removal commits; the leader keeps nudging a removed member for a while, the coordinator destroys the replica anyway.
• ReadIndex: reads are batched; one heartbeat round with the context of the newest read confirms leadership; the barrier completes when the state machine applied the commit index recorded at the time of the request. A new leader holds reads until its no-op committed. Reads are leader-only (no follower forwarding, no leases): customer answer 1.
• Snapshots: the core only asks (snapshots_to_send) and later restores (restore(meta): log reset to the snapshot, commit = applied = index). The host streams chunks of a SnapshotReader (stop-and-wait, snapshot_chunk_bytes, resend after snapshot_ack_ticks, three tries) and installs them through a SnapshotWriter. Order on the follower: all chunks written -> restore -> the snapshot record is made durable -> finish() (marker cleared + applied index set in one batch) -> done-ack. A crash in between leaves the state machine "installing" (applied index reads 0) while the log is already compacted: at open the host detects applied < log base and resets the replica's log (keeping term and vote), which makes the leader send a fresh snapshot. Compaction: after applied - base >= compact_threshold the host calls sm.sync() (applied state durable) and then drops the prefix, keeping retain_entries for slow followers.
• Quiescing: a leader whose group has been idle for quiesce_ticks with every follower caught up sends a heartbeat with a quiesce flag; the followers stop their election timers. Wake-ups: any message but the quiesced heartbeat, a local proposal/read/membership op, a node that stops sending node beats (node_suspect_ticks) for the groups it leads, a peer whose beat carries a new incarnation (it rebooted). Node beats are one tiny frame per peer node per node_beat_ticks, independent of the number of groups.
• LogStore (log_store.h/.cpp): ONE shared sequence of segment files raft-<n>.log per node for all groups, written through store::Env (so MemEnv's crash model applies: unsynced tails vanish or get garbled, directory entries need SyncDir). Records: HardState, Entries ("delete >= first.index, append"), Snapshot (reset), Compact, Create/Destroy (replica id, tombstone). commit(sync) writes everything staged by all groups with one append and one fsync (group commit across groups). Each new segment starts with a checkpoint (create + hard state + base of every group, tombstones) so old segments can be deleted once every group compacted past their entries - but only after the compaction record is durable; the previous segment is synced before a new one is created (no hole in the middle of the log). Recovery cuts a torn tail of the newest segment (atomic rewrite of the valid prefix), treats damage anywhere else as corruption, accepts a hole only if a Compact covers it. An I/O error fails the store for good (fail-stop; the host stops and the process restarts from disk).
• RaftHost / multi-Raft (host.h/.cpp): groups keyed by id, deterministic (sorted) iteration. One tick() drives every group; drive() collects the Ready of all groups, stages one log batch, one fsync, then sends, applies, advances. Outgoing messages are packed per destination node into frames ([version, kind, count, (group, len, message)*]), so the heartbeats of thousands of groups between two nodes share frames. A message for an unknown group creates it when group_factory accepts and the replica id is not tombstoned. destroy_group drops the log, writes a tombstone for (group, replica) and calls on_destroy() of the state machine. Replica ids are unique per (group, node) incarnation: a node that leaves and comes back gets a new id.
• Transport seam (HostEnv): self(), send(Addr, frame) (best effort), defer(fn) (run after the current handler; batches proposals into one fsync). F plugs its channel in by implementing these three calls and calling host.deliver(from, frame); under the simulation the same three calls go to NodeCtx.
• API for K (RaftGroup): propose(data, cb) (cb after commit AND apply on this node, with the state machine's reply; statuses Ok, NotLeader (+hint), Superseded (definitely not applied), Timeout/Stopped (unknown), Busy, Removed), read_barrier(cb), leader(), membership(), add_learner, promote, remove, add_voter (the whole learner dance with retries), transfer_leader, campaign. Callbacks never run inside the call that registered them.
• State machine on the LSM (lsm_sm.h): LsmStateMachine applies each entry with its effects and the applied index in one WriteBatch (D-4000); snapshots are streams of key/value records of the group's prefix taken from an LSM snapshot; install marks the group "installing", deletes the old range key by key (no DeleteRange yet), writes the chunks, then clears the mark and sets the applied index atomically.
• Storage-agnostic state machine: StateMachine knows nothing about the LSM; the SQLite-per-shard machine for builtin:// tables (spec §12.8: Raft replicates row changesets) implements the same five calls (applied index in the same SQLite transaction, snapshot = consistent copy of the database file in chunks, atomic swap in finish()), D-4325.
• Profiles: Profile::prod() (tick 50 ms, heartbeat 150 ms, election 1-2 s, quiesce after 10 s), Profile::test() (tick 10 ms, heartbeat 50 ms, election 300-600 ms).

Tests (deterministic simulation from the first commit)#

• tests/unit/raft/raft_core_test.cpp - the core on a synchronous message pump (through the codec): elections, pre-vote, check-quorum, conflicts, transfer, learners, removal, ReadIndex, compaction, quiescing, recovery.
• raft_log_test.cpp - codec round trips/garbage, log store: recovery, torn tails over many power-loss seeds, rolls, purge, tombstones, corruption.
• raft_sim_test.cpp with raft_harness.h/raft_scenario.h - 5 RaftNode processes (RaftHost + LsmStateMachine on a MemEnvDisk), a KV with put/get/cas, clients that record a history, a fault schedule drawn from the world's rng: partitions (isolate / split / one-way), loss, duplication, reordering, latency spikes, crashes and whole-cluster power loss with torn unsynced tails, disk write/sync failures, clock skew and drift, membership changes (add, remove, re-add under a new replica id), leader transfers, tiny snapshot thresholds (snapshots all the time). Invariants after EVERY event: election safety (<= 1 leader per group and term, ever), committed agreement (one (term, data) per committed index on all nodes), leader completeness, state machine safety (the same entry applied at the same index everywhere), applied <= commit <= last; periodic full log matching. At the end the cluster is healed, every node restarts, and every voter must have a leader, the same applied index and the same data; the client history must be linearizable (raft_kv_check.h: per-key search for put/get/cas with unknown-outcome operations).
• Sweeps: four configurations x 60 seeds = 240 seeds in ctest (about one minute in Debug); OXL_RAFT_SEEDS=N and OXL_RAFT_FIRST_SEED=K run more.
• Injected bugs (RaftBugs.*): reads served without ReadIndex (caught by the linearizability check), log entries acknowledged before the fsync (caught as an election-safety violation after a power loss); test_break_commit_rule (commit of old-term entries by counting) is a hook for further experiments.

Gaps (D-4379)#

Follower reads and leader leases (1.0), joint consensus, forwarding of proposals/reads from followers, a bound on the memory of the in-memory log of a group with a very lagging member (only compaction bounds it), parallel snapshot streams per peer (stop-and-wait), per-group log files, asynchronous fsync (the host is synchronous), fsync latency in the simulation (kill points are between events), apply on a separate thread, coordinator-level cleanup of removed replicas (the core only nudges them).

Кластер: транспорт и членство (stage 4, F «ткань», [С §12.1–12.3, §13.1], D-4200..D-4259)#

Everything is in runtime/src/cluster/. A node of a cluster has one TLS port; on it ALPN h2 is the existing gRPC stack and oxl-cluster/1 the cluster protocol (SWIM now; Raft, snapshots and saga takeover later register protocols on the same connections). SWIM runs over TCP inside that channel (datagram frames), never over UDP.

 :7000 (TLS 1.3, mTLS)                        process                                             (OXL_SEEDS / --seeds enables it)
   │ ALPN h2 ─────────────► grpc::Server::serveSession ─► services, rpc                          oxl_rt_start ─► cluster::autostart()
   │ ALPN oxl-cluster/1 ──► Transport::handleInbound                                             SIGTERM ─► LEFT gossip, then gRPC drain
   │                          │ HELLO / REJECT (cluster id, versions, node id == certificate SAN)
   │                          ▼
   │                        Mux  ── streams (proto name, messages, credit flow control) ─► setStreamHandler("raft"|…)
   │                          │  └─ DGRAM (unreliable, bounded) ─► ClusterNode::onDatagram ─► Swim::onMessage
   │                          └─ PING keepalive, FaultInjector hook (delay / drop / blackhole per frame, per direction)
   └ dial side: Transport::addPeer(addr) ─► backoff dial loop ─► same handshake ─► duplicate resolution (smaller dialer id wins)

 ClusterNode: driver task ─► Swim::tick() / nextWakeup() ─► reconcilePeers() (members' addresses ─► addPeer/removePeer)
              registry = Swim member table ─► cluster.nodes()/members()/selfInfo() (std), control socket (members, fault, partition, …)

SWIM (D-4205): probe every 500 ms (one shuffled round-robin target), 250 ms for the direct ack, then 3 indirect probes, SUSPECT after the interval, DEAD after 1.5 s without a refutation; test profile x3. Incarnation numbers (only the node raises its own), refutation, piggybacked gossip plus a 125 ms push of fresh updates to 3 random members, suspect nudging, full-table push-pull every 5 s, seed JOIN, re-probing of dead members (heals partitions), LEFT for graceful leave. Swim never reads a clock: tick() does everything that is due at Env::now(), nextWakeup() says when to call again. Detection on the 4-core container (production profile, 5 nodes): first suspicion 0.6-1.2 s, all survivors DEAD 2.1-3.0 s after the failure (the Standard asks for 1-2 s for suspicion).

Registry (D-4206): every member publishes id, roles, zone, labels, address, binary version, schema hash and load; cluster.nodes() / cluster.members() / cluster.selfInfo() return ClusterMember values (std/core/node.oxl, runtime ABI oxl_cluster_members_str). Outside a cluster the registry holds this node only.

Configuration (flag, then environment): --seeds/OXL_SEEDS (enables clustering), --listen/OXL_LISTEN (default 0.0.0.0:7000), --advertise, --cluster-id/OXL_CLUSTER_ID, --zone/OXL_ZONE, --label k=v/OXL_LABELS, OXL_ROLE, OXL_NODE_ID, --data-dir/OXL_DATA_DIR, OXL_CONTROL_SOCK, OXL_SWIM_PROFILE=test, OXL_BINARY_VERSION, OXL_SCHEMA_HASH, credentials OXL_CLUSTER_CERT|KEY|CA or OXL_DEV_PKI_DIR, faults OXL_FAULT_*. The cluster { seeds ... port ... } declaration reaches the runtime (oxl_cluster_declare, D-4573) and a declared seeds list starts clustering by itself (D-4603).

Control socket (<data-dir>/control.sock, mode 0600, one line command per connection, reply ok\n... / error: ...): members, self, peers, stats, events [n], fault <rule|list|clear>, partition a,b|c,d, heal, leave, label k=v, roles a,b. The real-cluster Jepsen tests drive faults through it.

oxl dev --nodes N [path] [--base-port 7000] [--roles a,b,c] [--zones x,y] [--data-dir d] [--swim-profile test] (D-4208): builds once, starts N nodes (n0..), restarts crashed ones unless <data-dir>/n<i>/hold exists, prints the status table (including how many members each node sees alive), stops on Ctrl-C/SIGTERM with a graceful leave.

Tests: cluster_mux_test (frames, handshake, streams, flow control, backpressure, violations, keepalive, fault layer), cluster_transport_test (real mTLS: dedup, reconnect, identity binding, ALPN, partition/heal, 5-node mesh), cluster_node_test (members in one process, control socket, gRPC and cluster on the same port), swim_test (virtual-time world, fixed seeds), swim_sim_test (the production state machine on oxl::sim::World), ctest cluster_dev5 (five real processes: kill -9, restart, asymmetric fault, partition, heal, graceful stop), conformance §12.3/members-single-node.oxl.

Not done (see D-4259): plain-TCP dev mode, k8s seed resolution, Lifeguard-style dynamic suspicion, membership change subscriptions in std, passing the cluster declaration to the runtime.

import proto и oxl gen (stage 4, wave 4 I, [С §16.3, §5.9, §12.8], D-4740..D-4799)#

Three pieces that connect OxL to other languages and to relational data; decisions in docs/decisions.md.

import proto "x.proto" { Names } (compiler/src/sema/ProtoImport.cpp, hook Sema::declareImportProto in FfiSema.cpp). There is no protoc dependency: Lexer / Parser read proto3 into a small AST (File, Message, EnumDef, Service), Importer resolves type names with the protobuf scoping rules, selects what the program asked for ({ A, B } plus everything those use) and renders OxL source (OXL_DUMP_FFI=1 prints it). The source is parsed and declared in the importing module exactly like the output of import c. The generated declarations carry the information the rest of the compiler needs to speak the foreign wire format with the existing machinery:

• data fields have @tag(n) = the proto field number and default values (proto3 peers omit zero values; decoding a missing field without a default is DecodeError.Missing);
• an rpc's parameters are the fields of the request message (@tag(n) name: T), so the synthetic request message of ServiceSema.cpp is byte-identical to the proto message;
• a response message that is not the one-field envelope { result = 1 } is the value type with @proto_flat: finalizeServices copies the fields of the value's data into the synthetic response message, and since the layouts are equal the code of CGService.cpp that reads / writes "field 0" needs no change;
• @proto_package("pkg") on the service makes the rpc path /pkg.Service/rpc; @proto_name("pkg.Msg") on an @error type is the name used in grpc-status-details-bin (wireNameOf);
• @serve on the import yields an implementable service instead of an extern service.

oxl export --proto is the inverse and writes the options import proto reads ((oxl.error), (oxl.raises), (oxl.stream), idempotency_level) in oxl/options.proto. Not imported: client / bidi streaming of a foreign .proto (the runtime's stream-in protocol sends a first message with the plain parameters; warning import-skip). Tests: tests/unit/sema/proto_import_test.cpp, ctest proto_import_interop (foreign C++ server), proto_import_roundtrip (export, import, call the live OxL service, all four rpc kinds).

oxl gen --py|--ts (compiler/src/sema/GenClient.cpp, tools/oxl/cmd_gen.cpp). After Compilation::check() the Builder turns the pub(api) services of the main package into a model: TD (type descriptors: scalar kinds with width / sign / zigzag, list, map, opt, union with member numbers in the lockTypeName order, ref to a generated type, refined, time, ...), TypeD (data, fieldless enum, enum with fields, refined type with its where expression) and ServiceD / RpcD (parameters with their wire numbers from ser::schemaOf(requestType), value, error types with their qualified names). PyPrinter and TsPrinter print the model; the wire encoding lives in compiler/gen/oxl_runtime.py / .ts (embedded in the binary by compiler/gen/GenRuntime.cpp.in), which encodes / decodes by the descriptors that the generated module registers (register_data, register_variants, ...). Refiner translates refinement predicates (ranges, len(..), matches(re"…"), comparisons, &&, ||, !) to Python / JavaScript. Client stream and bidi calls send the plain parameters in a first message and each element in its own, like the OxL client (D-3600). Tests: tests/unit/sema/gen_client_test.cpp, ctest gen_clients (tests/gen/): offline checks against protoc, then the generated clients call tests/services/pkg.

К7 relations and queries (std/orm/orm.oxl, compiler/src/sema/CheckTable.cpp, compiler/src/codegen/CGOrm.cpp, runtime/src/orm/orm.cpp). ref<T> is Ref<T> with @repr(i64), so it is serialized as the key and orm.cpp classifies OXL_SK_REPR over a scalar as a column of that kind (CK::Repr, to_repr / from_repr of the node convert). many<T> has no column (CGOrm leaves it out of cols; readRow gives such fields their default from the node). The table descriptor gets rels (OxlOrmRel: field, kind, offset of the list inside the field, target descriptor, back-reference column). checkTableQuery lowers load(a, b) to .load("a").load("b") on the Query<T> builder; Query.load records the names and oxl_orm_q_run runs one extra statement per name after the rows are read (loadRel: a semi-join on the query's own WHERE / ORDER BY / LIMIT, parameters shared), then distributes cloned rows into the lists by integer key. group by over a table sends where / order by to SQL and hands the fetched List<T> to the iterator query of §5.9 (listQueries_ adds .collect() so the result is a List). SQL dialects: the statements are the dialect-neutral text of D-3901, so the same tests run under --orm-backend postgres.

Хранилище таблиц и std.kv (stage 4, wave 3 K, [С §12.8, §11.2], D-4380..D-4459)#

Customer decision (STAGE4_PLAN 10): no own query engine. Tables of Db.open("builtin://…") are SQLite on every replica of a shard; Raft replicates row changesets (SQLite session extension), never SQL text. std.kv and system data live in the LSM (S1). Code: runtime/src/shard/ (liboxlrt), ORM routing in runtime/src/orm/orm.cpp, module std/kv/kv.oxl, tests tests/unit/runtime/shard_*.

 ORM call (insert / update / find / from u in db.T …)           std.kv (Kv.get / put / txn / watch)
        |  orm.cpp: routing by @shard key, fan-out + merge              |  kv_api.cpp: key -> shard ({tag}), OCC txn
        v                                                               v
  BuiltinStore (N shards)                                         KvDb (one LSM, N KvShard)
  SqlShard  ── stage_commit ─▶ ReplicatedLog ◀── KvShard commands
        |                       (LocalLog | RaftShardLog over RaftGroup)
        v apply(index, cmd) on EVERY replica, leader included
  SqlReplica: BEGIN; DDL…; changeset_apply; _oxl_meta.applied = index; COMMIT   |   KvShard: one WriteBatch (data + applied index)

A write on a SQL shard (leader)#

1. begin_txn: wait for the shard's gate (one staging transaction per shard), check leadership, read barrier and caught_up() (every entry of the log is applied), BEGIN IMMEDIATE on the leader's own database, attach a session to all tables. The handle lock is held: no apply runs meanwhile.
2. The ORM executes its statements (inserts, updates, queries — they see their own writes). DDL (CREATE/ALTER/DROP) is also noted in a list.
3. commit_txn = stage_commit (changeset from the session, ROLLBACK, command = DDL list + changeset) + propose + wait for the apply. The leader's database changes only when the command is applied, like every follower's: one code path, no divergence between "executed" and "replicated" state. last_insert_rowid is captured before the rollback.
4. Failure: leader changed (term differs, NotLeader, Superseded) → nothing applied, Busy, the client retries; Timeout/Stopped → outcome unknown, rely on idempotency records.

Apply (SqlReplica::apply, deterministic, atomic): skip if index <= applied; BEGIN IMMEDIATE; DDL statements; (after ALTER … ADD COLUMN the rows are rewritten once, D-4385); changeset check (known tables, same column count) and sqlite3changeset_apply_v2(NOSAVEPOINT) with a conflict handler that aborts; _oxl_meta.applied; COMMIT. Any failure is fatal (the replica stops; entries are never skipped). Snapshot = consistent copy of the database (sqlite3_backup + sqlite3_serialize), install = deserialize + backup over the live database. state_hash() hashes schema + rows (not pages): tests compare replicas with it.

System tables on every replica: _oxl_meta, _oxl_idem (idempotency records of [С §11.2]) and _oxl_outbox (outbox rows); idem_get/idem_put/outbox_add write them inside the staging transaction, i.e. in the same changeset as the row they belong to.

Shards and routing#

• A builtin://name handle is a group of one Conn per shard (orm.cpp, Conn::parts). 1 shard: <data-dir>/name.db (as before D-4380); N shards (?shards=N, $OXL_BUILTIN_SHARDS): name.shard<i>.db. BuiltinStore::set_shard_factory lets the cluster supply replicated shards.
• shard_of(key) = xxh64(key) % N. Row of a table with @shard(by: .f) → shard of its field f; tables without it → shard 0. find/delete by primary key route when f is the key, else probe all shards. Queries (§5.9, lowered to SQL as before) run on the leader's SQLite: routed when the where clause fixes the shard key, otherwise on every shard with ORDER BY + LIMIT skip+take and an in-memory merge by the order expressions (fetched as extra columns), then skip/take; count sums.
• atomic(db) / db.transaction: a transaction is pinned to the shard of its first statement; touching another shard is CrossShardTx (DbErrorKind.CrossShardTx). Savepoints nest as before. Outside a transaction each statement is a write unit (one changeset).
• Schema (ensure) is created/extended in its own write unit before the statement that needs it (the session cannot describe rows of a table altered in the same transaction).

std.kv#

KvShard (LSM, keys s<id>/d/…, version || value, applied index in the same WriteBatch) applies batches of Put/Delete/Require(version): all preconditions hold or nothing is written (CAS). kv_api.cpp adds the shard choice (hash of the key or of its first {tag}), linearizable reads (read barrier), an optimistic txn (reads validated with Require, writes in one batch, body rerun on conflict), prefix scans (merged over shards) and watch (a bounded in-memory log of applied commands per shard, merged by index, Lagged when overrun). Language side: std/kv/kv.oxl (rt: intrinsics, effect db).

Raft binding#

shard_common.h defines the seam: ShardSm (apply/snapshot/install/sync) and ReplicatedLog (propose/read_barrier/is_leader/term/caught_up, blocking). LocalLog is the default single-replica log. raft_adapter.h: RaftShardSm (an oxl::raft::StateMachine over a ShardSm; empty/ConfChange entries apply as 'N') and RaftShardLog (a RaftGroup behind the blocking seam; post runs closures on the host thread). Cluster wiring (a shard factory creating one Raft group per shard and replica, shard map from the coordinator) is the job of agents F/C.

Tests#

tests/unit/runtime/shard_test.cpp (determinism, DDL, crashes, snapshots, leader change, conflicts, hooks, KV), shard_mem_group.h (N-replica fake log), shard_sim_test.cpp (three real RaftHost nodes with SQLite state machines on oxl::sim: kill/restart, partitions, loss, reordering, snapshots; convergence and "acknowledged writes are visible" after 25 seeds, OXL_SHARD_SEEDS=N for more), conformance §12.8/orm-builtin-*, §12.8/kv-*.

Gaps (D-4410)#

Cluster wiring, 2PC, follower reads, staging blocks apply on the same database, no compiler-supplied shard key for queries (SQL text pattern), builtin:// migrations, KV range delete/TTL/durable watch history, no phantom protection in kv.txn.

Наблюдаемость (stage 4, agent O, [С §15.5], ТЗ Р7, D-5240..D-5259)#

Пользовательское руководство: docs/observability.md. Код: runtime/src/telemetry/, крючки в rpc/rpc.cpp, rpc/net.cpp, http1/http1.cpp, orm/orm.cpp, net/tls.cpp, obs/log.cpp; std/obs/obs.oxl.

 rpc handler / http handler / orm / user span ──► SpanScope / SpanInfo ──► telemetry::submitSpan ──► bounded deque
                                                                                                        │
 counters (MState, Keyed, Registry) ──► collect() ──► renderPrometheus ──► GET /metrics (task)          ▼
                                                  └─► encodeMetrics ──┐        exporter task (daemon): batch ──► grpc::Channel::unary
 log.emit ── trace_id/span_id ──► encodeLogs ─────────────────────────┴──────► /opentelemetry.proto.collector.*.v1.*/Export

Контекст трассы#

• Единственный носитель — rpc::Ctx задачи (rpcCtx, наследуется дочерними задачами): trace (32 hex), span (id текущего спана), parentSpan, sampled. ctx.traceId / ctx.spanId читаются из него.
• makeCtx — это и есть серверный спан rpc: идентификатор спана ставится всегда (один xorshift, единицы наносекунд), запись (SpanInfo.record) — только если трасса выбрана и задан экспортёр. Родитель — спан вызывающего (локальный вызов), либо удалённый родитель из traceparent. Решение о выборке принимается один раз, у корня (sampleTrace(traceId, rate): хэш младших 64 бит идентификатора, поэтому все процессы решают одинаково), дальше наследуется (parent-based).
• SpanScope (rpc.h) — RAII-спан для всего остального (клиентский rpc, HTTP, ORM, пользовательские спаны, будущие шаги саг/пайплайнов): при активном спане подменяет rpcCtx задачи копией с новым span, при выходе возвращает прежний. Без экспортёра или при отброшенной трассе конструктор — одна проверка флага, ничего не выделяется. Крючок для шагов саг / пайплайнов / акторов / цепочек: rpc::SpanScope s("saga/step", telemetry::SpanKind::Internal); s.attr(...) (include rpc/rpc.h).
• Сеть: gRPC — метаданные traceparent (W3C, версия 00, флаг 01 выбрано) плюс прежний x-oxl-trace; inboundOf (rpc/net.cpp) отдаёт значение в OxlRpcInbound.traceparent, applyCtx ставит исходящее из текущего спана клиента. HTTP: клиент (oxl_http_do) добавляет traceparent, сервер (serveRequest в std/http) читает его через oxl_obs_http_begin.

Экспорт#

telemetry::start() (из oxl_rt_start, до cluster::autostart) читает флаги и OTEL_*; если задан адрес — запускает daemon-задачу, которая раз в OTEL_BSP_SCHEDULE_DELAY (или при заполнении пакета) забирает до batchSize спанов и шлёт TraceService/Export унарным вызовом grpc::Channel (тот же клиент h2c/TLS, что и у rpc, но мимо слоя rpc: экспортёр не порождает спанов). Очередь ограничена; при переполнении новый спан отбрасывается и считается (oxl_telemetry_spans_dropped_total). Протобуф — вручную (otlp_encode.cpp, поля opentelemetry-proto v1). telemetry::flush() в конце oxl_rt_start (после дренажа) отправляет остаток синхронно. Нет адреса — g_tracing остаётся false, экспортёр не запускается.

Метрики#

Три источника, один формат Family/Sample (collect()): реестр пользовательских метрик (metricHandle: mutex только при создании, дальше атомарные операции), коллекторы встроенных метрик (rpc: счётчики и LatencyHist в MState метода — без блокировок и поиска по картам на горячем пути; builtin.cpp: HTTP/ORM/TLS/планировщик) и статистика экспортёра. Выводы: Prometheus text 0.0.4 (GET /metrics на --metrics-listen, свой минимальный ответчик на net::Socket: один запрос — одно соединение) и OTLP-метрики (cumulative, по OTEL_METRICS_EXPORTER=otlp).

Стоимость#

Без экспортёра на пути внутрипроцессного rpc добавлено: newSpanId, чтение флага tracing(), два clock_gettime и три relaxed-инкремента счётчиков метода (см. D-5254 о цифрах).

Пробелы (D-5259)#

Спаны шагов саг/пайплайнов/акторов/цепочек и клиентских стримов не создаются (нет самих подсистем на момент написания / крючок есть); @trace(sample:) работает на rpc, а не на произвольной функции; tracestate не распространяется; статус ошибок ORM в спане/метрике не различается.

Координатор и маршрутизация (stage 4, wave 4 C, [С §12.1–12.4], D-4560..D-4659)#

The coordinator is Raft group 0 plus the machinery around it (runtime/src/coord/). It is the one place that decides who runs what and where the data lives; everything else (SWIM, Raft, shard state machines, the router) is driven by or reads from it.

 ClusterNode (SWIM, mTLS channel)            coord::Coordinator  (one thread "oxl-coord", event driven like RaftHost)
   members, roles, labels  ------------->     reconcile every 300 ms:
   stream "oxl-coord/1" (Raft frames,                bootstrap -> controller (g0 leader) -> local reconcile -> drive moves -> hints
   forwarded proposals, shard ops) <---->     RaftHost: group 0 (SysSm) + one group per shard replica (RaftShardSm over SQL/KV)

State (group 0, coord/sys_state.*)#

SysState is a deterministic machine over byte commands (no clocks; time stamps are inside the command): nodes (number #n, address, zone, declared and effective roles, labels, served services, life alive/down/left, down-since), shards (group id, kind sql/kv, database, index/count, replication factor, replicas (node, replica id, learner), leader hint, epoch, the move in flight), a system KV with versions and CAS (K/X/C/G commands) for CA/KMS material and dynamic config, and the counters for replica ids, group ids and node numbers. It is durable in one file (<data-dir>/coord/state/g0/sys.state, written by sync() before the Raft log is compacted, FNV checked). The system KV API for the security code is coordnode::sysKvGet/Put/Cas/Del (coord/node_glue.h); reads go through the log, so they are linearizable.

Bootstrap (D-4561, D-4562)#

• oxl cluster coord bootstrap (control command coord bootstrap, or OXL_BOOTSTRAP=1 / --bootstrap) creates group 0 on that node at once (one voter).
• Without it the smallest node id among the live SWIM members bootstraps when it sees OXL_COORD_EXPECT (3) members or after OXL_COORD_BOOTSTRAP_WAIT_MS; a node never bootstraps when some member advertises the SWIM label oxl.g0=1 (a group already exists).
• The leader of group 0 then grows it: voters = the first 3 node ids (journal nodes preferred, choose_group0_voters, replaced after the node-down grace), every other node becomes a learner (so every node has the map locally, no polling). A new node joins by being seen in SWIM.
• Replica ids come from the replicated counter (never reused: a replaced node gets a new id, the old incarnation is destroyed).

The controller (group-0 leader, coordinator.cpp)#

Every 300 ms, when the leader has applied its whole log: mirror SWIM into the registry (J join / S down / left), assign_roles, size group 0, place new shards (place_replicas: least loaded storage node, spread over zones; waits for min(replication, OXL_COORD_EXPECT) storage nodes or placeWaitMs), plan_moves, plan_leader_transfer. All proposals of one round are serialized (next round when they are applied).

• Roles (planner.cpp): explicit OXL_ROLE/label nodes are never touched and count towards replicas a..b. A node without roles asks for automatic assignment (assign by ... auto: label oxl.auto=1, then oxl_node_configure waits up to 60 s for roles_assigned). Deficits are filled largest first by the eligible node (labels of needs) with the fewest roles, so with fewer nodes than role slots a node packs several roles; leftover nodes join the role with the lowest count/min below max. Sticky: assigned roles are never taken away.
• Placement/rebalancing: replicas live on nodes with a journal role (every node when the declaration has none). Moves: replace a replica of a node that left at once, of a node down for NODE_DOWN_GRACE (60 s; OXL_NODE_DOWN_GRACE_MS, 10 s with OXL_COORD_PROFILE=test), restore the replication factor of shards placed on too few nodes, then balance when max - min >= 2 replicas (hysteresis). At most 2 moves in the cluster at a time. A move whose new node vanished is aborted after the grace.
• Leaders: group leaders report L hints; when one node leads at least 2 more shards than another the controller asks the leader node to transfer_leader to a replica on the light node (cooldown 5 s, never during a move).

A move (never below quorum)#

M(group, add, remove) records a learner replica in the map (the shard has repl+1 replicas from now on). The leader of that shard (each node watches the map for the groups it leads, drive_moves) runs add_voter (learner → catch up, snapshot if needed → promote), then remove(old), then proposes F (done) which drops the old replica from the map. The old node destroys its replica (on_group_removed, data deleted by the state machine's on_destroy; a left-over replica of an earlier incarnation is destroyed when the map names a new one for the node). Every step is idempotent, so a new leader simply continues; an interrupted removal is healed by "members the map does not know are removed". Calls during a move wait: ClusterKv::call retries "not leader / superseded" answers for OXL_REBALANCE_BUFFER_MS (15 s) — this is the REBALANCE_BUFFER; a write whose outcome is unknown (timeout) is NOT retried.

Shard state machines in a process (coord/node_glue.*)#

KV shards share one LSM per node (coord/kv.lsm, prefix s<group>), their sync() is a WAL fsync (the Raft log is compacted only after it); SQL shards are one SQLite file per replica (coord/sql/<db>.s<i>.r<replica>.db). std.kv in a cluster (kv_api.cpp → ClusterKv) asks the coordinator for the database (idempotent D command), then sends P (propose), G (get), S (scan) to the leader of the shard — locally or over the coordinator stream ("shard ops": the leader runs them on its coordinator thread, reads after a ReadIndex barrier). builtin:// tables use BuiltinStore::set_shard_factory: a node with a replica of the shard uses it while it leads the shard; a node without a replica, or with a follower, runs the statements on the shard leader (remote SQL, shard/sql_remote.*, D-4602): the leader keeps a staging transaction per client session (ops B begin, X exec, Y query with the rows returned materialised, C commit, R rollback, forwarded as shard ops to the leader), reads outside a transaction run under a ReadIndex barrier; a session idle for 30 s is rolled back. A transaction cut by a leader change fails with the retryable busy error. Forwarded shard requests are resent every 700 ms until the answer arrives and the receiver deduplicates them (one execution per request id, cached answer), because the node channel may lose a frame. A Raft apply (also the no-op and conf-change entries) preempts a staging transaction that holds the replica instead of waiting for it (the thread that would commit it is the apply thread, D-4606). Shard counts: OXL_SHARDS (the acceptance/Jepsen profile uses 16–32), else the shards n of the journal role, else what the program asked for.

Routing Х2 (rpc/route.cpp)#

oxl_rpc_route keeps the in-process path (no serialization) when this node serves the service. Otherwise kind 0 calls pick a node: live members whose roles serve the service (service roles from the generated table), same zone first, then the lowest EWMA latency + 5 * in-flight + load, round-robin on ties; a per-node circuit breaker opens after 3 consecutive UNAVAILABLE/timeouts for 2 s and then admits one half-open probe. @readonly/@idempotent rpcs are retried once on another node on UNAVAILABLE. S.at(node) (kind 1) goes to that node (UNAVAILABLE when it is down or does not serve the service). The call is a gRPC unary call over an mTLS h2 session on the shared cluster port (ClusterNode::h2ClientTls). Language level (std/core/node.oxl): cluster.find(service), findRole, pick(service, by:, prefer:), on(.RoleChanged | .MemberJoined | .MemberLeft) { }, cluster.hasRole, cluster.ids.next(), cluster.leader(name) { }.

cluster.leader (coord/leader.cpp)#

A lease leader/<name> in the system KV (node|deadline_ms, TTL 3 s, renewed every 400 ms by CAS). The holder runs the block; a node that cannot renew cancels its leader task before the lease expires; the others wait in oxl_cluster_leader_wait. Roles that declare runs leader("name") restrict the candidates.

Tests#

• tests/cluster/test_coord_cluster.py (ctest cluster_coord, 5 real processes): bootstrap, routing edge -> store, kill -9 of a shard leader, dead node replaced, partition, builtin:// tables incl. a transaction used from the replica-less edge node, watch on the clustered std.kv. tests/cluster/smoke_256.py: 256 shards (D-4605).
• tests/unit/coord/coord_planner_test.cpp: state machine, snapshot, declaration text, roles (labels, packing), placement, move caps/hysteresis, leader balance.
• tests/unit/coord/coord_sim_test.cpp (test_coord, seeds OXL_COORD_SEEDS): bootstrap with staggered starts, placement, kill/restart of the group-0 leader and of a shard leader, dead node replaced after the grace, new nodes take their share, graceful leave, partition, auto roles. Invariants every 50 ms: ≤ 2 moves, never fewer replicas/voters than placed, distinct nodes, plus final convergence, acknowledged writes present, group 0 identical everywhere.
• tests/cluster/test_coord_cluster.py (cluster_coord): the same on oxl dev --nodes 5 with grpcurl, kill -9 and partition.

Безопасность (stage 4, W4-S, [С §13.1–13.3], D-4660..D-4739)#

Криптография — OpenSSL (AWS-LC — долг к 1.0). Код без зависимостей от рантайма — runtime/oxlsec/ (OBJECT-библиотека: её объекты лежат и в liboxlrt.a, и в oxl, поэтому oxl cluster init / oxl keys rotate работают без запущенного узла). Всё, что нужно рантайму, — runtime/src/cluster/enroll.*, runtime/src/rpc/auth.*, runtime/src/crypto/field_kms.*.

 oxl cluster init ─► <ca dir>/{cluster.id, ca.pem, ca.key.enc, token.key.enc, tokens/, deny}   master.key (0600, OXL_KMS_KEY_FILE)
                                   │ CaStore (seam: coordinator Raft group later)                  │ wraps CA key, token key, data keys
 custodian node  (OXL_CA_DIR) ◄────┴── ALPN oxl-enroll/1 on the cluster port ◄── joiner (OXL_JOIN_TOKEN): CSR + token ──► cert, SAN oxl://<c>/role/<r>/node/<id>
        │ renew at 2/3 of 24 h (Transport::setTls, connections untouched) · signed deny-list (op=crl) · identity cache <data-dir>/identity
        ▼
 mTLS everywhere ─► checkIdentity (SAN == HELLO node id, deny-list, production: SAN required) ─► h2 peer identity ─► x-oxl-via
 client ─ authorization: Bearer <jwt> ─► authorizeInbound ─► x-oxl-principal ─► ctx.caller / ctx.via / ctx.userId()

Доверенные метаданные. x-oxl-principal (JSON вызывающего), x-oxl-via (вызывающий сервис), x-oxl-user ставит только рантайм. У недоверенного клиента все три вырезаются, authorization не пробрасывается дальше никогда. Доверенный — узел кластера с проверенным клиентским сертификатом и oxl://-SAN: он может передавать их дальше (делегирование), x-oxl-via = Service(имя = id узла, роль = первая роль сертификата). Узел, вызывающий другой узел по gRPC, обязан предъявить сертификат (cluster::makeH2ClientContext) — иначе вызываемый не поверит делегированному принципалу. Сокращение стадии 3 x-oxl-user работает только в профиле dev без настроенной аутентификации.

Конфигурация. Узел-хранитель CA: OXL_CA_DIR (+ OXL_KMS_KEY_FILE); присоединяющийся: OXL_JOIN_TOKEN + семена (OXL_SEEDS) или OXL_ENROLL_ADDR, OXL_DATA_DIR для кэша удостоверения. OXL_CERT_TTL_SEC (тесты), OXL_CRL_REFRESH_SEC (15), OXL_ENROLL_TIMEOUT_SEC (60). Аутентификация: OXL_AUTH_JWKS, OXL_AUTH_JWT_KEY, OXL_AUTH_JWT_ISSUER, OXL_AUTH_JWT_AUDIENCE, OXL_AUTH_ROLES_CLAIM, OXL_AUTH_JWKS_MIN_REFRESH_SEC. Профиль: --profile production / OXL_PROFILE=production — сертификат из файлов без oxl:// SAN отвергается, OXL_DEV_PKI_DIR и --insecure-dev запрещены; --insecure-dev вне production принимается, но по-прежнему mTLS с автоматическим dev-CA (D-4670).

CLI. oxl cluster init|token create|token list|token revoke|node remove|node allow|status, oxl keys rotate, oxl dev --nodes N --builtin-ca.

Отказы. Нет учётных данных — Anonymous (решают политики); недействительные — UNAUTHENTICATED до входа в обработчик; обработчик, возвращающий AccessDenied, — прикладная ошибка (D-4679). Токен входа: единый текст отказа для пира (причина — в журнале CA).

Тесты. test_security (21), ClusterEnroll.* в test_runtime, ctest security_cluster_ca (3 процесса, ротация сертификатов 14 с, oxl keys rotate, удаление узла, oxl dev --builtin-ca), security_jwt_grpcurl (grpcurl, JWKS по URL с ротацией ключей), conformance §13.2, §13.3, fuzz fuzz_jwt.

Не сделано (D-4739): одиночный хранитель CA без failover (перенос CaStore в группу координатора — задача агента C), приватный ключ присоединившегося узла лежит открытым в каталоге данных (0600), нет внешнего kms(url), @encrypted для топиков и акторов — только API, oidc/apikey/session — 1.0, политики (Pol, волна 5), AWS-LC.

Политики (stage 4, wave 5 Pol, [С §13.4], D-5010..D-5069)#

policy-блоки, @public, @allow(role:), @internal компилируются в код (без интерпретатора), проверка идёт в рантайме при каждом входе в обработчик; права на данные (@owner, @visible) применяются в ORM и при кодировании ответа; решения @audit пишутся в подключаемый приёмник.

policy Svc { allow get(id) if caller.isUser(db.find(Order, id)?.userId)  deny delete unless … }
   │ parser: условие правила разобрано заново для каждой цели (PolicyRule::conds) — дерево переезжает в функцию
   ▼ sema/PolicySema.cpp  buildPolicies()  (в конце checkServices, когда известны @public/@internal)
 RpcInfo::policyMode  0 нет правил │ 1 allow (константа) │ 2 deny (константа) │ 3 функция policyFn
 RpcInfo::policyFn    fn policy#Svc.rpc(<параметры rpc>) -> bool { let ctx; let caller; let <dep db>;
                        if <deny> { return false }  …  <allow> || <allow> }      ← обычная функция: проверка типов, эффекты, dep
   ▼ codegen/CGService.cpp  tramp (внутри задачи обработчика: ctx и принципал известны)
   dec = oxl_policy_gate(desc, mode, "pkg.Svc.rpc")  →  0 выполнить │ 1 AccessDenied │ 2 вызвать policyFn
   отказ: результат = член NetError `AccessDenied` → wire: outcome 4 / gRPC PERMISSION_DENIED (UNAUTHENTICATED — только плохие учётные данные)
   ▼ runtime/src/rpc/policy.cpp  (контракт — runtime/include/oxl_policy.h)

Кто как судится (gate). Метаданные ctx — доверенные ключи x-oxl-principal, x-oxl-via, x-oxl-user и x-oxl-origin (ставится authorizeInbound на каждом входящем вызове, клиентский стирается):

Без правил (mode 0) rpc закрыт для внешних принципалов — но только если программа вообще использует авторизацию (Sema::policyEnforced(): есть policy, @public/@internal/@allow, auth { … } кластера или @owner/@visible); иначе (программа этапа ≤ 3) всё открыто, как раньше (D-5012). Для rpc без условий решение — константа дескриптора: вызов в процессе из main = вызов oxl_policy_gate и просмотр пустых метаданных. Потоковые rpc: тот же вход; отказ → oxl_policy_stream_denied, рантайм отвечает PERMISSION_DENIED.

Роли. caller.has(role: .Support): has<R: Show>(role: R) в прелюдии; CheckCall.cpp унифицирует R с перечислением из roles from claim(...) as AppRole (ClusterInfo::authRolesType), так что .Support проверяется как вариант AppRole, а рантайм сравнивает имя варианта со списком ролей токена. caller.service("billing") = isService. Поле User.roles остаётся List<str> (D-5011).

Данные. @owner(.f, bypass: [...]): дескриптор таблицы (OxlOrmTable::owner/owner_bypass) → orm.cpp: find добавляет AND f=?2, запросы (oxl_orm_q_run/count) — AND (f = ?N) до маршрутизации по шардам (если f — ключ шарда, запрос идёт на один шард), update/delete — то же условие; запись чужой строки — ошибка; анонимный внешний вызывающий не видит строк. Владелец вычисляется на задаче вызывающего (blocking выполняется в другом потоке). @visible(role: .R): OxlSerField::visible_role; кодировщики (bin/json) пропускают поле между oxl_ser_mask_begin(inbound) и oxl_ser_mask_end — только при кодировании ответа rpc (wire и элементы потока), хранение и вызовы в процессе не затронуты.

Аудит. @audit на rpc или policy: OxlAuditRecord { unix_ns, trace_id, principal, via, action, target, allowed, reason } → oxl_audit_emit → приёмник (oxl_audit_set_sink). Приёмник по умолчанию: строка JSON в <OXL_DATA_DIR>/audit.jsonl (иначе ./.oxl/) и запись лога info; агент топиков подставит приёмник, публикующий в системный топик oxl.audit (@retain 30d). oxl audit tail [--data-dir D] [-n N] [-f] [--json] читает локальный файл.

Крючки для остальных объектов политики (актор — по msg, топик — publish/subscribe, динамический конфиг — set): oxl_policy_gate_object(kind, target, mode, audit) / oxl_policy_done_object — те же классификация и аудит; sema принимает policy на актор/топик/конфиг (имена проверяются), но кода для них ещё не строит — это делают агенты акторов, топиков и конфигурации (D-5020).

Тесты. conformance §13.4/policy-*, data-rights-* (stage 4), test_runtime Policy.* (6: ворота, граф, аудит, @owner, @visible), ctest policy_grpcurl (JWT-пользователи по gRPC: deny-by-default, @public, @allow, условия с параметрами/ctx/чтением db, deny … unless, @internal, @owner на find/query/update/delete и bypass, @visible, @audit + oxl audit tail).

Топики и outbox (stage 4, wave 5 T, [С §10.5, §11.2, §11.4], D-4900..D-4999)#

Топик — журнал с партициями поверх std.kv (D-4402): никакого отдельного хранилища, один путь записи для одного узла и для кластера.

topic OrderEvents: OrderEvent sharded by .userId        (TopicSema.cpp: тип X__T + static X)
  X.publish / emit / emitIn / subscribe / subscribeKey / count / sweep      std/topic/topic.oxl
        │ rt: oxl_topic_*                                                   runtime/src/topic/topic_api.cpp
        ▼
  TopicStore (topic_store.cpp)  ── Backend ──►  KvDb (локально: KvShard+LocalLog; кластер: ClusterKv → Raft-группы шардов)

Раскладка ключей (хеш-тег {топик#партиция} держит всё про партицию на одном шарде, поэтому одна Raft-запись меняет сообщения, курсоры и аренды вместе):

Операции KvShard: Append (шард назначает следующее смещение, ответ несёт смещения) и DeleteRange (retention). Смещения без дыр и монотонны на партицию; счётчик переживает удаление.

Публикация. partition = xxh64(ключ) % N; без sharded by — одна партиция. Порядок гарантирован на ключ. Retry (нет лидера) повторяется внутри, Unknown (исход неизвестен) возвращается как TopicError.Unknown.

Подписка. Без group — широковещательная: позиция в памяти, from задаёт старт, партиции читаются по кругу. С group — очередь: claim берёт первый элемент без живой аренды CAS-пакетом, ack (при запросе следующего элемента циклом for await) двигает курсор; не подтверждённый элемент возвращается после срока аренды (по умолчанию 30 с). Доставка at-least-once.

Outbox. atomic(db) { …; emit T(x) } = db.transaction(() => { … T.emitIn(db, x) }); emitIn кладёт строку в _oxl_outbox шарда транзакции в том же наборе изменений, что и строки (D-4391); откат транзакции убирает и событие. Ретранслятор (outbox_relay.cpp) на лидере шарда читает закоммиченные строки по порядку, публикует с id o:<бд>:<шард>:<id строки> и удаляет. Повтор после сбоя поглощается записью дедупликации. Идентификаторы строк не повторяются (D-4902).

Retention. Фоновая нить каждого узла (OXL_TOPIC_SWEEP_MS) удаляет диапазоны по возрасту и числу; курсоры групп перескакивают вперёд. Аудит (oxl.audit, 30 дней) — очередь в памяти и фоновая публикация из oxl_audit_emit. Политики — oxl_policy_gate_object (topic.publish/topic.subscribe), рёбра графа вызовов topic:<имя>.publish|subscribe.

Тесты: topic_test.cpp (юнит + симуляция отказов на виртуальных часах), topic_sim_test.cpp (3 реплики Raft на oxl::sim), tests/cluster/test_topic_cluster.py (5 процессов, kill -9 лидера), conformance §10.5/topic-*.

Шардированные акторы и Х6 (stage 4, wave 5 A, [С §12.5–12.6, §17.3], D-4800..D-4899)#

Sharded actors#

actor X sharded by key: T — ключ (поле 0 состояния) плюс var. Код:

Path of Account(7).deposit(5):

1. call site: encode the argument tuple, oxl_sactor_call(ref, msg, bytes) creates a random 16-byte message id;
2. runtime: shard = hash of the tag {type\0key} over the shards of _oxl_actors; if this node leads the shard group, the engine runs the message here, otherwise the message goes as shard operation A to the leader (resent with the same id on any doubt, up to 30 s);
3. engine (per actor, one message at a time): activation = read the state record (OXA1: ring of recent message ids, vars) and on activate; a known id is answered from the journal; else invoke, then ONE batch Require(state, version) + Put(state) + Put(marker) + Delete(old markers); the reply leaves after the commit;
4. a changed owner (failover, move): the next message activates the actor from the journal on the new leader; the old activation fails the version Require (zombie) or is dropped by the sweeper (evictUnowned).

 edge n0                 shard leader n1                    followers n2, n3
 Gate.add ─ shard op A ─▶ engine.deliver ─ Raft batch ─────▶ replicate, commit
                           │ activation (memory)  ◀──────── commit index
 ◀──── reply (after commit) ┘

What the tests check (D-4830): unit test_actor (fake store, 120 random schedules: kills, lost answers, rival owners), conformance §12.5, ctest cluster_actors (kill -9 of the owner while sending: ~5 s to resume, nothing lost or doubled; partition of the new owner).

Limits: a zombie can run a message body once before it loses the version race (side effects), the dedup window is the last 64 messages, policy modes for actor messages are not generated yet (D-4817), @encrypted seals the whole record (D-4816).

Х6 (runtime/src/coord/x6.*, std/core/node.oxl)#

• cluster.now() -> Hlc: (ms << 16 | counter, node), strictly increasing, merged from every message of the coordinator channel (H prefix) and every rpc (x-oxl-hlc); a stamp more than MAX_DRIFT (500 ms) ahead is refused (cluster.observe).
• cluster.ids.next(): snowflake (41 ms / 10 node / 12 counter).
• cluster.lock(key, ttl) { } / lockFenced: lease in the coordinator's system KV; the fencing token is the KV version of the taking CAS; renewal thread; Fence.valid().
• cluster.shards(): the coordinator's shard table (info only).
• cluster.leader is C's (D-4585, unchanged).

Саги и пайплайны (stage 4 wave 6 Sg, [С §10.6, §11.3], D-5100..D-5189)#

Код: compiler/src/sema/SagaSema.cpp (генерация, анализатор), std/saga/saga.oxl (журнал, исполнитель, восстановление), runtime/src/rpc/rpc.cpp (локальная идемпотентность по ключу), runtime/src/sched/task.cpp (фоновые задачи без дедлайна вызывающего).

Лоуеринг. Как топики (TopicSema.cpp), saga и pipeline превращаются в обычные функции генерацией OxL-текста и его разбором в том же модуле; тексты шагов копируются дословно, поэтому проверяются как шаги цепочки (§8.4), а диагностики сгенерированного кода переносятся на исходные символы (remapSagaDiagnostics). Типы результатов выводятся (FnDecl::inferReturn — тот же механизм, что у приватных fn … => e, но для любой формы тела): тип вызова — тип цепочки, т.е. R | E… | CompensationFailed | NetError. Параметры идут в журнал как значение сгенерированного Args__name (std.ser), поэтому обязаны быть Serializable.

saga placeOrder(snap) -> Order { A undo r => U1  ->  B undo p => U2  ->  C }

fn placeOrder__body(snap) => await (() -> std.saga.enter(0, it) -> y0 => (y0 -> A retry 6)
    -> std.saga.guard(0, it, r => U1, x0 => await (std.saga.done(0, false, x0)
        -> std.saga.enter(1, it) -> y1 => (y1 -> B retry 6) -> std.saga.guard(1, it, p => U2, x1 => await (done(1,false,x1)
            -> enter(2, it) -> y2 => (y2 -> C retry 6) -> done(2, true, it))))))
fn placeOrder__run(a: bytes) => placeOrder__body(placeOrder__dec(a).snap)       // то, что исполняет исполнитель
pub fn placeOrder(snap) => std.saga.drive("pkg.placeOrder", encode(&Args__placeOrder{ snap })!, a => placeOrder__run(a))
fn boot__sagas() { std.saga.serve("pkg.placeOrder", std.saga.sagaRoles(), a => placeOrder__run(a)) }     // зовёт codegen при старте узла

Компенсации — это вложенность: guard(k, v, comp, rest) исполняет остаток саги и, если он вернул ошибку, comp(v) шага k; поэтому они идут в обратном порядке сами. Это значение «undo x => …, где x — результат шага» без явного стека замыканий (его негде хранить при переезде на другой узел).

Журнал (std.kv, хранилище oxl.saga; все ключи саги несут тег {sagaId} и лежат в одном шарде, транзакция читает и пишет их вместе): s/{id}/h голова (имя, закодированные аргументы, пользователь), s/{id}/o аренда исполнителя {node, expiry, epoch}, s/{id}/t состояние (running, compensating, stucking, затем done|failed|cancelled|stuck, курсор шагов, результат Boxed<T>), s/{id}/c маркер отмены, s/{id}/uK «компенсация шага K сделана», a/<сага>/{id} — индекс незавершённых, x/{id} — stuck. Запись состояния — транзакция, проверяющая epoch аренды (устаревший исполнитель ничего не меняет).

Кто исполняет. drive (вызывающий, любой узел) создаёт голову и индекс и ждёт конечного состояния опросом журнала (4..80 мс); сам сагу он не исполняет. Исполняет цикл восстановления serve на узлах ролей runs sagas (saga.roles — константа из cluster): он просыпается по kv.watch индекса (и раз в 300 мс), берёт аренду (claim: нет аренды, или она истекла, или SWIM пометил владельца dead/left), запускает execute в отдельной фоновой задаче и продлевает аренду (renew, каждую треть срока, срок 3 с). Захват после падения заново выполняет тело саги с первого шага: шаг каждый раз получает idempotency-key = sagaId#N (компенсации — sagaId#uN), а rpc шагов @idempotent (анализатор требует), поэтому повтор отвечает первым результатом (кластерная запись T2 на проводе, запись узла для локальных вызовов — rpc.cpp). Результаты шагов в журнал не пишутся.

Дедлайн (ответ 9). drive оборачивает ожидание в defer { shield { … } }: если задачу отменили (within, дедлайн rpc), одной транзакцией пишется маркер отмены (если сага ещё не «закоммичена») и вызывающий сразу получает Timeout. Исполнитель проверяет маркер в enter перед каждым шагом; последний done(k, true, …) — точка коммита, проверка маркера и запись commit в одной транзакции с маркером отмены: сага либо закоммичена, либо отменена. После маркера guard-ы компенсируют завершённые шаги в фоне.

Компенсации. До SAGA_UNDO_RETRIES = 5 попыток, пауза 100 мс · 2^n, не более 5 с; потом состояние stuck, сага возвращает CompensationFailed{cause, step}, внешние guard-ы не компенсируют дальше. std.saga.status/resolve/list(.Stuck).

Пайплайны. P__front — шаги до respond включительно (значение — ответ; ошибка — ответ-ошибка), шаги после respond — respondThen(v, tail): хвост уходит в фоновый scope узла (node.background.spawn, без дедлайна вызывающего, D-5107) с !> на on_error(e, step) и DLQ (std.saga.dlq/dlqList). С первого durable-шага у пайплайна есть запись в журнале (те же ключи): перед каждым надёжным шагом пишется значение, входящее в шаг; после падения узла цикл servePipe захватывает запись и вызывает P__resumeJ, которая декодирует значение (тип берётся из никогда не вызываемой цепочки-свидетеля, valueOf) и продолжает с шага J. Пайплайн над stream<T> применяет шаги к каждому элементу (@parallel(n) пока читается как 1). on role проверяется на существование роли; шаг исполняется на текущем узле (переход на другой узел — D-5110).

Граф вызовов. Роль runs sagas обходит сгенерированное тело саги (настоящие рёбра), шаги пайплайна приписываются роли on.

И9: bench / fuzz / contract tests, snapshots, coverage (stage 5 wave 4, [С §18.1–18.2, III.9], D-7050..D-7112)#

• Harness: oxl.test.run calls oxl_test_drive(kind, wrapper, …) (runtime/src/testing/test_drivers.cpp) instead of oxl_catch_panic: kind 1 = bench (calibrated batches, oxl-bench: {json} on stdout), kind 2 = fuzz (corpus replay, new inputs, shrinking, report on stderr); other kinds run once. The runner (compiler/src/driver/TestRunner.cpp + TestTools.cpp) passes OXL_TEST_ID, OXL_FUZZ_*, OXL_BENCH_TIME, OXL_SNAPSHOT_DIR, OXL_CONTRACT_SCHEMA, OXL_COVERAGE_DIR to each test process.
• Fuzz arguments: sema turns test fuzz "n" (x: T) into let x: T = oxlFuzzArg("x"); the intrinsic passes the type's OxlSerNode to oxl_fuzz_gen, which writes canonical JSON from a choice sequence (ChoiceSource) and decodes it — layouts, refinements and defaults stay in the serialization runtime. Shrinking minimises the choice sequence (no per-type shrinkers).
• Contracts: the runner resolves schema(old: …) to an old oxl.schema.lock, runs compareLocks (§9.2) and hands the test a line-format old schema; old.roundtrip(&x) projects the JSON encoding of x onto the old tags and decodes it back.
• Coverage: CGOptions::coverage adds atomic counters (CGCoverage.cpp) per statement line, if/match arm and fail site of the main package's files, registered by a global constructor; runtime/src/testing/coverage.cpp dumps them at exit; the runner merges the dumps and renders the summary / HTML.

Source: docs/ARCHITECTURE.md