OxL Reference
EnglishРусский

12.8 Built-in storage and tables

  • The storage is on every node of the journal role, replication is Raft per shard. Data is in the --data-dir directory. Two local engines under Raft: an LSM tree (key-value) for system data and std.kv, and an embedded SQLite for ORM tables.
  • The LSM stores: actor state, saga and durable pipeline journals, idempotency records, topics, the outbox, buffers of resumable streams, coordinator data, the dynamic config.
  • Direct access is std.kv (a transactional KV with prefix subscriptions). A write (put, delete) whose outcome is unknown (the shard leader may have committed it and failed before replying, or the reply was lost) completes with Timeout ("the write may or may not have been applied"). The runtime MUST NOT repeat such a write by itself: repeating a blind write after confirmed writes of other clients violates linearizability. The client repeats it with its own idempotency (CAS, identifiers). Definite failures ("not leader", "superseded", "not sent") are retried automatically; so are reads, scans and actor messages (which the engine deduplicates) (D-8052; D-7472, Jepsen).
  • ORM tables: a data with @table("name") is a table; @pk — primary key, @auto — generation, @index(fields…), relations ref<T> and many<T>. Db.open(url) supports postgres://, mysql://, sqlite://, builtin://. db.T is a Table<T>: insert, update(T, id) { … }, delete, find(T, id), queries §5.9.
  • Tables in builtin://: every shard replica has its own SQLite database. Raft replicates not the SQL text but deterministic row changes (the change set of a transaction), so random(), time and the execution order on replicas do not influence the result. Queries of §5.9 are translated to SQL and executed by SQLite on the shard leader (reads are linearizable); a query without a sharding key is executed on all shards with the results merged by order by/take. atomic(db) is a transaction inside one shard. A shard snapshot is a consistent copy of its SQLite database. SQL features are limited to the SQLite dialect.
  • @shard(by: .field, count: n) on a table is sharding (in the built-in storage — over the shards of the cluster; for external databases — over a list of URLs). count sets the number of shards of the table; if the database has fewer shards, the number of table shards is limited to the number of database shards and a warning is issued. Without count the number of database shards is used. The number of shards of a table is stored with its data (table _oxl_table_meta of the built-in storage); opening the database with another number (a changed count or shards=) is an error that points to moving the data, not a silent loss of sight of rows (D-8320). Resharding is not implemented.
  • Migrations are oxl db diff and oxl db migrate (part III).

Implementation-defined. builtin:// migrations (D-8065; DIV-022; D-8240, D-8241). This implementation: oxl db migrate|rollback|status --db builtin://name works on the shard files of a stopped node (each migration is applied to every shard in its own transaction, bookkeeping is the table _oxl_migrations per shard; disagreeing shards are refused); an online migration of a running or Raft-replicated storage is not implemented (an optional feature) — there the program executes the DDL itself through db.execute(ddl).

Implementation-defined. Corruption of the LSM log (D-8066; D-8310). A damaged record in the middle of a log (with valid records behind it) is not mistaken for a torn tail: by default opening the storage fails with Corruption naming the log and the offset, and nothing on disk is changed; the mode kPointInTime recovers the records before the damage, discards the rest and keeps the damaged logs as <n>.log.corrupt (the loss is reported on stderr and by oxl_store_wal_corruption_total). A damaged last record (nothing valid behind it) is a torn tail: that write was never acknowledged. A damaged manifest is always refused. The mode cannot yet be chosen from a program (the C++ API only).

Implementation-defined. Durability of writes to disk (D-8575; customer decision). fsync(2) on macOS does not flush the drive's write cache (only fcntl(F_FULLFSYNC) does), so after a power loss an acknowledged write of the storage or Raft log can be gone. By default on macOS the logs and snapshots of the built-in storage and Raft (WAL, Raft log, manifests, snapshots) and std.io writeBytes(sync: true) use plain fsync (as SQLite and PostgreSQL do on macOS; macOS is a development platform, TZ 3.1): a write survives a crash of the process and a reboot of the OS but need not survive a power loss. The environment variable OXL_FULLFSYNC=1 (read once at start; 0 means plain fsync) turns on fcntl(F_FULLFSYNC), falling back to fsync if the file system refuses it; set it for data that must survive a power loss. The switch does not change the semantics of the language. Cost of F_FULLFSYNC: about 4 ms per drive per commit (group commit smooths it for parallel writes; tens of times slower for one sequential writer). On Linux writes are synchronized through fdatasync/fsync with no variable.

query-not-sql, shard-count-clamped, table-schema

See also

Source: docs/spec/oxl-standard.en.md