Local debugger (ч. III.5, requirement И6)

oxl dap is a Debug Adapter Protocol server on stdin/stdout. It is a thin proxy over lldb-dap (D-3230): all protocol messages pass through unchanged, except the few that carry OxL specifics. Source mapping is the DWARF the compiler emits (docs/ARCHITECTURE.md, profile dev/test: full debug info), so breakpoints, stepping, call stacks, watch expressions and variable inspection are lldb's.

oxl dap [--lldb-dap <path>] [--no-formatters] [--no-tasks]

lldb-dap is looked up as --lldb-dap, OXL_LLDB_DAP, lldb-dap-18, lldb-dap in PATH, /usr/lib/llvm-18/bin/lldb-dap.

Launch#

The launch request of lldb-dap with these differences:

VS Code: the extension in tools/vscode contributes the debugger type oxl (.vscode/launch.json: { "type": "oxl", "request": "launch", "name": "Debug OxL", "program": "${workspaceFolder}" }); the setting oxl.lldbDap overrides the adapter.

Values: tools/lldb/oxl_formatters.py (D-3231)#

Loaded by oxl dap with initCommands (command script import), also usable by hand in lldb. The compiler describes the runtime objects in DWARF (CGDebug.cpp): str/bytes are typedefs of a pointer to OxlStr, List<T> of a pointer to OxlList, Map<K, V>/Set<T> of a pointer to OxlMap; unions and T? have a tag enumeration named after the member types. The formatters:

Everything is read from the memory of the stopped process; nothing is evaluated in it. The layouts are those of runtime/include/oxl_rt.h and runtime/src/map.cpp (a change there needs a change in the script; the DAP test checks them). Element types are found by name (primitives and any type that is in the program's debug info); an unknown element type shows no children.

Tasks (D-3232)#

With OXL_DEBUG_TASKS=1 the runtime enters every spawned task in a registry (runtime/src/sched/debug_registry.h: the global list oxl_debug_tasks; id, parent id, entry function, pointers to the live state/status/cancelled fields). Without the variable a spawn costs one load of a flag. The debugger reads it:

• lldb command oxl tasks [--json]: id, parent, state (runnable, running, parked, notified), status, entry function;
• the custom DAP request oxlTasks (optional frameId): the response body is { "registry": true, "tasks": [...] }.

Version 2 of the registry (D-6950) also records what a parked task waits for and its stack at its last park:

• lldb commands oxl tasks (now with a "waits for" column), oxl task-stack <id> [--json] (the frames of a parked task, with file and line), oxl waits [--json] (the "who waits for whom" graph: task → task / channel / rpc / actor / sleep …, deadlock cycles);
• in oxl dap: the parked tasks are appended to threads (ids from 2 000 000 000, e.g. task 5 consume (parked: recv chan 0x…)), their stackTrace is the recorded stack (no variables: the frames are not live); custom requests oxlTaskStack {taskId} and oxlWaits (edges, cycles, waiters, text).

Runtime breakpoints (III.5, D-6952..D-6954)#

Written as function breakpoints, in oxl dap, oxl debug all-stop and non-stop alike:

In all-stop oxl dap removes the runtime frames above the first OxL source frame from a stackTrace that starts in an oxl_dbg_stop_<kind> hook, so the client shows and selects the user frame (D-7791).

Under lldb (all-stop) they are breakpoints on the runtime's oxl_dbg_stop_<kind> functions with a match condition: the stop is in that function, the user's frame is a few frames down. The non-stop engine shows an event frame (error PayError with the fail's file and line), the task's own frames, and also accepts the exception filters panic / error (condition: type names). Logpoints (logMessage, placeholders {name} {detail} {trace} {caller} {kind} {loc}) print without stopping and are set without a restart. setVariable: lldb's in all-stop; in non-stop the fields #N of a parked rpc request (the handler runs with the changed request). The non-stop console understands tasks, waits, task <id>.

Distributed stop (III.5, D-6957)#

During a non-stop session every node of the cluster tags its outgoing rpc calls (x-oxl-dbg-caller) and records the calling task's stack. A stop in a handler called from another node shows, below the handler's frames, a label frame ── caller on node n0: task 6, rpc Users.get -> n1 and the frames of n0's waiting task (n0: callUsers, n0: checkout …), hop by hop up the chain; the scope Caller on n0 has the node, method, target, task, how long it waits and its own caller. (All-stop sessions show only the local stack.)

oxl test --debug (III.9, D-6965)#

oxl test [paths] --exact <name> --debug        # prints the lldb command and a VS Code launch configuration
oxl test [paths] -f <regex> --debug=dap        # a DAP adapter on stdin/stdout: `launch` without `program` runs the test
oxl test [paths] --exact <name> --debug=lldb   # lldb with the pretty-printers

Exactly one test must be selected; its binary is built with -O0 and full debug info and runs the test with OXL_TEST_RUN=<index>.

Profiler (oxl prof, III.1, D-6960..D-6963)#

oxl prof cpu   --node n1 --cluster n0:7000 [--for 10s] [--hz 99] [--folded cpu.folded] [--json] <credentials as for oxl debug>
oxl prof heap  --node n1 --cluster n0:7000 [--for 10s]
oxl prof tasks --node n1 --cluster n0:7000

• cpu: SIGPROF sampling of the whole process (every worker), a flat / cumulative top of functions; --folded writes folded stacks (flamegraph.pl, speedscope).
• heap: the allocations of the window, sampled every 64 KiB per thread, by stack; the allocator's totals and the RSS.
• tasks: counts by state / status / entry function, what parked tasks wait for, the most waited-for objects, deadlocks.

Fetched through the cluster like a debug session (ALPN oxl-debug/1 on the cluster port), policy debug (no debug enable needed), audited as debug.prof on the entry node. --for is at most 5 minutes; one profile at a time per node.

Test#

tests/dap/run_dap_session.py (ctest dap_session, skipped when lldb-dap is missing): initialize, launch an .oxl file (built by the proxy), breakpoint, stack, variables with the pretty-printers, oxlTasks, continue, disconnect; the failed-build path; runtime breakpoints msg / error / panic, parked tasks as threads with their stacks, oxlTaskStack, oxlWaits, setVariable; and oxl test --debug=dap. Unit tests: tests/unit/runtime/debugger_test.cpp (specs, matching, the lldb rewrite, request edits, waits graph, live registry, the non-stop engine for every kind, distributed stop, profiler), RaftCore.HoldKeepsAStoppedLeaderOnlyWhenAsked. Cluster: tests/rdebug/test_debug_cluster.py (ctest cluster_debug_chain, run alone): distributed stop n0 → n1, logpoint, request edit, oxl prof cpu|heap|tasks through the cluster.

Remote debugger (ч. III.7, requirement И7)#

A running node can be debugged over the network, directly or through the cluster, with any DAP client: oxl debug is a local DAP adapter on stdin/stdout (point the IDE's debug adapter command at it). Decisions D-6650..D-6661.

$ ./shop --debug-listen 0.0.0.0:2345 --role api                  # the node serves a debugger on 2345 (mTLS)
$ oxl debug attach 10.0.3.7:2345 --cert me.crt --key me.key --ca me.ca.crt

$ oxl keys issue --role debug --name alice                         # a principal certificate from the cluster CA
$ oxl debug enable --node api-2 --for 30m --cluster shop.internal:7000 --cert alice-debug.crt --key alice-debug.key --ca alice-debug.ca.crt
$ oxl debug --node api-2 --cluster shop.internal:7000 --cert ... [--all-stop|--non-stop] [--binary ./shop]

How it works#

• Debug agent. LLDB stops every thread of the process it debugs, so the session cannot be carried by the node itself. The node starts a debug agent — its own binary re-executed with OXL_DEBUG_AGENT=1 — that holds the client connection, filters and audits the DAP requests and runs the engine. Without --debug-listen no port is opened and no agent runs; a cluster node only answers (and, until oxl debug enable, refuses) enable / session requests on its existing cluster port.
• Through the cluster (--node): the CLI connects to the cluster TLS port of any node (ALPN oxl-debug/1), that node checks the policy debug and asks the target over the cluster transport (mux stream oxl-debug/1); the target's agent dials back to that node with the target's node certificate and the two connections are spliced. No port is opened on the target; an all-stop session must enter through another node than the one it stops.
• Engines. --all-stop: lldb-dap attached to the node's pid (line / function breakpoints, stacks, variables with the OxL pretty-printers, stepping, evaluation). --non-stop (default in the profile prod): the cooperative runtime engine — function breakpoints on rpc methods (Users.get); only the handler task of a matching call parks, everything else keeps running; the stop shows the rpc frame and the request (method, trace, caller, fields by number); continue releases the call. The node's default comes from --all-stop / --non-stop / OXL_DEBUG_MODE (else non-stop in prod, all-stop elsewhere); the client may pick per session.
• All-stop and the cluster. Before attaching, the node publishes the SWIM label oxl.debugged=<ms>; its peers extend the member's suspicion timeout to that, so a node sitting at a breakpoint is suspect, never dead (its replicas are not replaced). Raft elections are not delayed: a stopped leader is replaced and the node rejoins as a follower (D-6656). With --hold-raft (opt-in per session, all-stop only: oxl debug --node n2 --cluster … --all-stop --hold-raft, oxl debug attach … --hold-raft) the node also publishes oxl.debugged.raft: the followers of the Raft groups it leads do not start elections while it is stopped, so it stays leader; writes to those groups wait meanwhile (D-6966).

Access, audit, checks#

A principal certificate never joins the cluster channel as a node. The inspector's action debug.enable (III.8) enables a node too. oxl debug enable --for takes 1 s .. 24 h (default 30 m); at the end new sessions are refused and running ones are detached.

Every connection, enable/disable, session start/end and every DAP request that changes state (breakpoints, continue/step, evaluate, setVariable, attach, refused launch, disconnect) is written to oxl.audit (via = "debug", action debug.<what>).

On attach the agent checks the build id: --binary <local exe> (or --build-id) must match the running node; for all-stop the executable on disk must still be the running one (lldb reads it). launch is refused, and disconnect/terminate never kill the node.

Image requirements (all-stop)#

lldb-dap (LLVM 18) in the image (OXL_LLDB_DAP to point at it), ptrace permitted: CAP_SYS_PTRACE or kernel.yama.ptrace_scope <= 1 (the node declares its agent with PR_SET_PTRACER) and a seccomp profile that allows ptrace. The agent probes this before every all-stop session and refuses with the reason. Pretty-printers: set OXL_LLDB_FORMATTERS or ship oxl_formatters.py next to the binary. The non-stop engine needs none of this.

Tests#

tests/unit/runtime/rdebug_test.cpp (policy, DAP filter and framing, cooperative engine, SWIM tolerance, principal certificates), ClusterTransport.AlpnHandlerForTheDebugger, and tests/rdebug/test_remote_debug.py (ctest cluster_remote_debug, run alone): a 3-node cluster, keys issue, debug enable of n2 through n0, a non-stop and an all-stop session on n2 through n0 with a breakpoint in an rpc handler, n2 not declared dead while stopped, audit, --debug-listen + oxl debug attach, the end of the enabled period. The all-stop part is skipped with the agent's message when ptrace is not permitted.

Source: docs/debugger.md