-
v0.17.7 Stable
released this
2026-11-08 19:53:10 +00:00 | 9 commits to main since this releaseVSKI v0.17.7: Durable replication, tango cluster proxy
Replication is rebuilt around a durable, verifiable changeset log:
every committed write is appended to an on-disk log before its
transaction commits, replicas resume from an exact position, and a
promoted replica serves its former siblings from the same log. Nodes
switch between master and replica in-process, without a restart. On
top of that ships tango, a new cluster proxy: it splits reads across
replicas, journals every write to Redis, and fails over to the
most-current replica. Acknowledged writes survive the failover, and none
is applied twice.failover under load (200 writes/s + 400 reads/s, master stopped) acknowledged writes missing on the new master 0 duplicated writes 0 write outage (503 + Retry-After) 3.4 s old master rejoined as replica 2.0 s Upgrade notes:
- Upgrade replicas before the master. A master on the new protocol
answers older replicas with426 Upgrade Required; they stay readable
but stale until upgraded. Each replica re-downloads once after the
upgrade. workflow_eventsgains aseqcolumn, added automatically on start.- Behavior changes, all listed below: replicas refuse more writes, the
SQL console refuses transaction control, and transient write failures
now return 503 instead of 403. - No SDK changes.
Durable changeset log
- Writes through the write queue are captured on the database's pinned
connection and appended toDATA_DIR/replication-log/<db>.clog
(CRC-checked, fsynced) before their batch commits. A failed commit
removes its entry; crash recovery drops uncommitted tails. - Entries form a hash chain. A replica resumes with its last
position plus the chain hash at that position. A position that is
pruned, ahead of the master, or on a different history (a failover
onto a node that missed writes) gets a full download instead of a
mixed history. - Replicas mirror the master's log entry for entry, so after a promotion
the remaining replicas catch up incrementally from the new master.
A master restart costs replicas nothing. - Entries are kept for
REPLICATION_LOG_RETENTION_HOURS(24); a replica
offline longer re-downloads once. A database no replica has asked for
is not logged, so standalone nodes pay nothing. - Full downloads use SQLite's backup API: snapshots include writes still
in the WAL, and replicas restore in place, so open handles stay
valid. Previously a replica's first full download could leave services
on a closed database. - Writes from the SQL console and from migration scripts now go through
the write queue, so they replicate. Inside them, transaction control
(BEGIN,COMMIT,ROLLBACK, savepoints),ATTACH/DETACHand
pragma assignments are refused: each request already runs in its own
transaction. - A changeset carrying columns or tables a replica lacks is no longer
silently skipped: the replica syncs the schema first. - Workflow events carry a per-run
seqassigned when they commit, so a
promoted replica hands workers the same event order as the old
master.
Roles, failover and replica safety
- Hot promote/demote:
POST /api/admin/promoteand
POST /api/admin/demote(superuser token) switch a node's role
in-process: write path, syncer, changeset journal, pipeline dispatcher,
cron, workflow gateway, webhooks, graph index. - Workflow execution is master-only: workflow gateway, scheduler,
cron dispatch, webhooks and realtime run on the master alone.
Replicas answer workflow writes and queue polls with503. workflows.dbreplicates by default as a passive copy, so a
promoted replica starts with every run, queued job, timer and
definition. Jobs the dead master had handed out are reclaimed within
30–60 s and redelivered.REPLICA_INCLUDE_WORKFLOWS=falsekeeps it
node-local.- Replicas no longer write locally. Every write to a replicated
database is refused below the HTTP layer, so a replica never holds rows
its master never saw. Logins still work on replicas. Registration,
OAuth account linking, admin init and API key creation now return
403there and must go to the master; before, they silently created
diverging rows. - Exactly-once replay: a write carrying
X-Tango-Idempotency-Key
records the key in the same transaction; a repeat answers
200 {"duplicate":true}without writing. Keys replicate with the
data and are pruned afterIDEMPOTENCY_KEY_RETENTION(2h), which must
exceed tango's journal retention. - Older replicas (pre-log protocol) get
426 Upgrade Required, logged
once per replica every few minutes. - Security:
/api/admin/replicas*now requires an admin.
Previously any authenticated user could mint replica tokens. - Replicas report their positions to the master after every sync
(POST /api/replica/sync-complete); every node exposes its own at
GET /api/replica/local-status.
Write path fixes
- Transient failures are retryable: a write hitting a node that is
shutting down, a full write queue, a replica, or a busy database now
returns503withRetry-Afterinstead of403, which clients
treated as final. - Write-queue batches take the write lock up front (
BEGIN IMMEDIATE).
A batch that read before writing could fail with "database is locked"
when another connection committed in between; SQLite reports that
without waiting. - Deleting a database released neither its replication session nor its
connection handle, which could hang shutdown or crash the server.
Sessions are now released before their connection closes. - The Docker images are built with SQLite sessions enabled. Without
them a replica's first snapshot failed, so containers could not
replicate.
tango: the cluster proxy
A new component (
tango/): one endpoint in front of a vski master and
its replicas.- Routing: writes go to the master. Reads go to the least-busy
healthy node, except the workflow and automation plane (workflows,
pipelines, cron, webhooks, plugins, secrets) and the realtime and
workflow websockets, which stay on the master. Sessions (/sso,
logins,/api/auth,/api/admins,/api/keys) and admin scripts
(SQL console, migrations) go to the master and are never journaled. - Write journal: every write is appended to a Redis Stream before
it is forwarded, with an idempotency key the master stores with the
write. On failover tango replays, in order, every entry the new master
may lack, including writes the old master acknowledged but never
replicated. Writes whose client saw a failure are dropped, never
replayed. - Failover: tango fences the old master, hot-promotes the replica
with the most entries applied, then replays. Instrictmode
(default) writes get503+Retry-Afterduring the window; in
absorbmode they are journaled and answered202. - Read freshness: a replica that falls behind the master by more
thanTANGO_MAX_READ_LAG(30s) leaves the read pool until it catches
up. Optional read-your-writes (TANGO_READ_YOUR_WRITES) pins a
client's reads to the master briefly after its write. - Actuators carry out role changes:
docker: Engine API, role files read by containers at start;
replicas sync through tango, so promotions need no reconfigurationkubernetes: role ConfigMap, the label the master Service selects
on, and lease election between tango replicaslocal: child processes, for developmentmanual
- Management API (
:8001): status, manual promotion, replay.
Measured (
tango/bench, three nodes on one host, k6 open model),
p95 added by tango:500/s 1,000/s 2,000/s 4,000/s reads (point, list) +0.2 ms +0.2–0.3 ms +0.1–0.2 ms +0.1 ms record creates +0.1 ms +0.1 ms +5.7 ms +8.0 ms No errors at any rate. On writes, most of the cost is the idempotency
key vski stores with each write; the benchmark breaks the cost down.Tests and docs
- New end-to-end suites cover multi-replica convergence, master restarts,
hot promotion and switchover, incremental catch-up after promotion,
workflow state across failover, and divergence handling. vski's full
end-to-end suite (47 tests) passes through tango, and tango's failover
test proves acknowledged-but-unreplicated writes survive exactly once. - Replication.md documents the changeset log, upgrade order, replica
read-only rules and the new settings; tango ships its own README and
benchmark guide.
Downloads
-
Source code (ZIP)
0 downloads
-
Source code (TAR.GZ)
0 downloads
- Upgrade replicas before the master. A master on the new protocol