• v0.17.7 54ad6ccdac

    v0.17.7 Stable

    x released this 2026-11-08 19:53:10 +00:00 | 9 commits to main since this release

    VSKI v0.17.7: Durable replication, tango cluster proxy

    Replication is rebuilt around a durable, verifiable changeset log:
    every committed write is appended to an on-disk log before its
    transaction commits, replicas resume from an exact position, and a
    promoted replica serves its former siblings from the same log. Nodes
    switch between master and replica in-process, without a restart. On
    top of that ships tango, a new cluster proxy: it splits reads across
    replicas, journals every write to Redis, and fails over to the
    most-current replica. Acknowledged writes survive the failover, and none
    is applied twice.

    failover under load (200 writes/s + 400 reads/s, master stopped)
    acknowledged writes missing on the new master 0
    duplicated writes 0
    write outage (503 + Retry-After) 3.4 s
    old master rejoined as replica 2.0 s

    Upgrade notes:

    • Upgrade replicas before the master. A master on the new protocol
      answers older replicas with 426 Upgrade Required; they stay readable
      but stale until upgraded. Each replica re-downloads once after the
      upgrade.
    • workflow_events gains a seq column, added automatically on start.
    • Behavior changes, all listed below: replicas refuse more writes, the
      SQL console refuses transaction control, and transient write failures
      now return 503 instead of 403.
    • No SDK changes.

    Durable changeset log

    • Writes through the write queue are captured on the database's pinned
      connection and appended to DATA_DIR/replication-log/<db>.clog
      (CRC-checked, fsynced) before their batch commits. A failed commit
      removes its entry; crash recovery drops uncommitted tails.
    • Entries form a hash chain. A replica resumes with its last
      position plus the chain hash at that position. A position that is
      pruned, ahead of the master, or on a different history (a failover
      onto a node that missed writes) gets a full download instead of a
      mixed history.
    • Replicas mirror the master's log entry for entry, so after a promotion
      the remaining replicas catch up incrementally from the new master.
      A master restart costs replicas nothing.
    • Entries are kept for REPLICATION_LOG_RETENTION_HOURS (24); a replica
      offline longer re-downloads once. A database no replica has asked for
      is not logged, so standalone nodes pay nothing.
    • Full downloads use SQLite's backup API: snapshots include writes still
      in the WAL, and replicas restore in place, so open handles stay
      valid. Previously a replica's first full download could leave services
      on a closed database.
    • Writes from the SQL console and from migration scripts now go through
      the write queue, so they replicate. Inside them, transaction control
      (BEGIN, COMMIT, ROLLBACK, savepoints), ATTACH/DETACH and
      pragma assignments are refused: each request already runs in its own
      transaction.
    • A changeset carrying columns or tables a replica lacks is no longer
      silently skipped: the replica syncs the schema first.
    • Workflow events carry a per-run seq assigned when they commit, so a
      promoted replica hands workers the same event order as the old
      master.

    Roles, failover and replica safety

    • Hot promote/demote: POST /api/admin/promote and
      POST /api/admin/demote (superuser token) switch a node's role
      in-process: write path, syncer, changeset journal, pipeline dispatcher,
      cron, workflow gateway, webhooks, graph index.
    • Workflow execution is master-only: workflow gateway, scheduler,
      cron dispatch, webhooks and realtime run on the master alone.
      Replicas answer workflow writes and queue polls with 503.
    • workflows.db replicates by default as a passive copy, so a
      promoted replica starts with every run, queued job, timer and
      definition. Jobs the dead master had handed out are reclaimed within
      30–60 s and redelivered. REPLICA_INCLUDE_WORKFLOWS=false keeps it
      node-local.
    • Replicas no longer write locally. Every write to a replicated
      database is refused below the HTTP layer, so a replica never holds rows
      its master never saw. Logins still work on replicas. Registration,
      OAuth account linking, admin init and API key creation now return
      403 there and must go to the master; before, they silently created
      diverging rows.
    • Exactly-once replay: a write carrying X-Tango-Idempotency-Key
      records the key in the same transaction; a repeat answers
      200 {"duplicate":true} without writing. Keys replicate with the
      data and are pruned after IDEMPOTENCY_KEY_RETENTION (2h), which must
      exceed tango's journal retention.
    • Older replicas (pre-log protocol) get 426 Upgrade Required, logged
      once per replica every few minutes.
    • Security: /api/admin/replicas* now requires an admin.
      Previously any authenticated user could mint replica tokens.
    • Replicas report their positions to the master after every sync
      (POST /api/replica/sync-complete); every node exposes its own at
      GET /api/replica/local-status.

    Write path fixes

    • Transient failures are retryable: a write hitting a node that is
      shutting down, a full write queue, a replica, or a busy database now
      returns 503 with Retry-After instead of 403, which clients
      treated as final.
    • Write-queue batches take the write lock up front (BEGIN IMMEDIATE).
      A batch that read before writing could fail with "database is locked"
      when another connection committed in between; SQLite reports that
      without waiting.
    • Deleting a database released neither its replication session nor its
      connection handle, which could hang shutdown or crash the server.
      Sessions are now released before their connection closes.
    • The Docker images are built with SQLite sessions enabled. Without
      them a replica's first snapshot failed, so containers could not
      replicate.

    tango: the cluster proxy

    A new component (tango/): one endpoint in front of a vski master and
    its replicas.

    • Routing: writes go to the master. Reads go to the least-busy
      healthy node, except the workflow and automation plane (workflows,
      pipelines, cron, webhooks, plugins, secrets) and the realtime and
      workflow websockets, which stay on the master. Sessions (/sso,
      logins, /api/auth, /api/admins, /api/keys) and admin scripts
      (SQL console, migrations) go to the master and are never journaled.
    • Write journal: every write is appended to a Redis Stream before
      it is forwarded, with an idempotency key the master stores with the
      write. On failover tango replays, in order, every entry the new master
      may lack, including writes the old master acknowledged but never
      replicated. Writes whose client saw a failure are dropped, never
      replayed.
    • Failover: tango fences the old master, hot-promotes the replica
      with the most entries applied, then replays. In strict mode
      (default) writes get 503 + Retry-After during the window; in
      absorb mode they are journaled and answered 202.
    • Read freshness: a replica that falls behind the master by more
      than TANGO_MAX_READ_LAG (30s) leaves the read pool until it catches
      up. Optional read-your-writes (TANGO_READ_YOUR_WRITES) pins a
      client's reads to the master briefly after its write.
    • Actuators carry out role changes:
      • docker: Engine API, role files read by containers at start;
        replicas sync through tango, so promotions need no reconfiguration
      • kubernetes: role ConfigMap, the label the master Service selects
        on, and lease election between tango replicas
      • local: child processes, for development
      • manual
    • Management API (:8001): status, manual promotion, replay.

    Measured (tango/bench, three nodes on one host, k6 open model),
    p95 added by tango:

    500/s 1,000/s 2,000/s 4,000/s
    reads (point, list) +0.2 ms +0.2–0.3 ms +0.1–0.2 ms +0.1 ms
    record creates +0.1 ms +0.1 ms +5.7 ms +8.0 ms

    No errors at any rate. On writes, most of the cost is the idempotency
    key vski stores with each write; the benchmark breaks the cost down.

    Tests and docs

    • New end-to-end suites cover multi-replica convergence, master restarts,
      hot promotion and switchover, incremental catch-up after promotion,
      workflow state across failover, and divergence handling. vski's full
      end-to-end suite (47 tests) passes through tango, and tango's failover
      test proves acknowledged-but-unreplicated writes survive exactly once.
    • Replication.md documents the changeset log, upgrade order, replica
      read-only rules and the new settings; tango ships its own README and
      benchmark guide.
    Downloads