Actuator for VSKI Replicas
  • Go 86.7%
  • Shell 6.9%
  • Python 3.6%
  • JavaScript 2.2%
  • Dockerfile 0.3%
  • Other 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Anton Nesterov 0209dc2754 tango: master regains its role under the same address; never promote the fenced node
Found by the in-cluster failover test:
- A tango replica that saw the master answer as a replica (its pod being
  replaced) never marked it master again once it came back under the same
  address, so that replica refused every write after the failover. The
  monitor now re-asserts the role of the single node reporting master.
- An automated failover fenced the old master and then picked it as the
  candidate (its replacement pod passed probes and held the most data),
  hot promote hit a pod that was being deleted, and the restart fallback
  ran. The fenced node is skipped unless named explicitly.
- Startup waits up to two minutes for Redis instead of exiting.
2026-11-08 21:36:04 +01:00
bench tango: faster journaled writes; benchmark write-cost breakdown 2026-11-08 19:41:48 +01:00
cmd/tango tango: master regains its role under the same address; never promote the fenced node 2026-11-08 21:36:04 +01:00
internal tango: master regains its role under the same address; never promote the fenced node 2026-11-08 21:36:04 +01:00
scripts tango: session routing, read freshness, node-failure handling, entrypoint 2026-11-08 17:25:30 +01:00
.gitignore tango: session routing, read freshness, node-failure handling, entrypoint 2026-11-08 17:25:30 +01:00
Dockerfile tango: kubernetes fixes — valid lease times, fencing waits for the pod, split-brain write hold 2026-11-08 21:01:55 +01:00
go.mod tango: vski cluster proxy — routing, write journal, hot failover, exactly-once replay 2026-11-08 05:10:24 +01:00
go.sum tango: vski cluster proxy — routing, write journal, hot failover, exactly-once replay 2026-11-08 05:10:24 +01:00
LICENSE add license 2026-11-08 20:19:53 +01:00
Makefile tango: kubernetes fixes — valid lease times, fencing waits for the pod, split-brain write hold 2026-11-08 21:01:55 +01:00
README.md readme: replicas sync through the management port on Kubernetes too 2026-11-08 21:06:23 +01:00

Tango

The vski cluster proxy: read/write splitting across a vski master and its replicas, a Redis-backed write journal, and failover with hot promotion of the most-current replica.

clients ──► tango :8000 ── writes ──► master (journaled first, replayed on failover)
                │
                └── reads ──► least-connections across healthy replicas + master

What it does

  • Routing: every mutation is pinned to the master. Reads go least-connections over healthy nodes, except the workflow/automation plane: realtime and workflow websockets and every read under /api/workflows, /api/pipelines, /api/cron, /api/webhooks, /api/plugins, /api/plugin-instances and /api/secrets go to the master, where that state executes; a worker never polls its run on a lagging replica. Some endpoints go to the master for every method and are never journaled: sessions (/sso/*, /api/auth/*, /api/admins/*, /api/keys, /api/collections/<name>/auth-with-password: hosted login keeps its codes and OAuth state in the master's memory, and logins carry passwords that must not sit in the journal; during a failover they answer 503 + Retry-After rather than a token-less 202) and admin scripts (/api/sql/execute, /api/migrations/*: no idempotency key, so a replay could apply them twice). The instance control endpoints (/api/replica/*, /api/admin/*) are never proxied. A read whose node died between health probes is retried on another node; an unreachable master answers 503 + Retry-After.
  • Read freshness: a replica that has not reached, in every database, the position the master held TANGO_MAX_READ_LAG ago (default 30s) leaves the read pool until it catches up: a stuck syncer, a replica cut off from the master, or one too old to speak the changeset log protocol.
  • Read-your-writes: with TANGO_READ_YOUR_WRITES set, a client's reads go to the master for that window after its write. The client is its Authorization header, its X-API-Key, or else its address.
  • Write journal: with TANGO_REDIS_ADDR set, every write is appended to a Redis Stream before it is forwarded, carrying an idempotency key that the master records in the same transaction as the write. A write the master applied (2xx) stays replayable until trimmed; one the client saw fail (rejected, or refused with 503 during a failover in strict mode) is deleted, because the client may retry it. On failover, the entries the candidate may lack replay into it in journal order with the caller's original credentials: everything never acknowledged, plus everything applied after the newest master position the candidate is known to hold. That recovers writes the old master acknowledged but had not replicated when it died. Writes absorbed while the master was briefly unreachable replay into it when it comes back without a failover.
  • Exactly-once replay: vski records each write's X-Tango-Idempotency-Key in a _tango_applied table inside the same transaction as the write, so a replay of a write the target already holds answers 200 {"duplicate":true} instead of creating a second record. Record and graph creates are keyed; updates and deletes replay in journal order and land on the same final state. The table replicates with everything else, so a promoted replica knows every write it received; keys are pruned after IDEMPOTENCY_KEY_RETENTION (vski, default 2h), which must exceed TANGO_JOURNAL_RETENTION.
  • Failover (hot): fence the old master, then call the candidate's POST /api/admin/promote: vski flips its role in-process (write path, dispatcher, cron, workflow gateway, changeset journal, the full lifecycle) without restarting, and tango persists the new role via the actuator. The candidate is the healthy replica with the most changeset entries applied across all databases. Failover completes in milliseconds; a restart-based promotion remains the fallback for nodes without the endpoint. strict mode refuses writes during the window (503 + Retry-After); absorb mode journals them and returns 202.
  • Durable changeset log: vski appends every committed write to an on-disk changeset log (DATA_DIR/replication-log/, fsync'd before the transaction commits) and replicas mirror it entry for entry, resuming with since=N plus a hash-chain token. A master restart costs replicas nothing, and after a promotion the remaining replicas catch up from the promoted node's log instead of re-downloading. A replica whose position does not belong to the new master's history (the candidate missed writes the replica already had) is told to re-download rather than keep writes the new master never saw; journal replay then restores them through the new master.
  • Workflows survive failover: workflow execution (gateway, scheduler, cron dispatch, webhooks) is master-only, and the master's workflows.db replicates to replicas as a passive copy (REPLICA_INCLUDE_WORKFLOWS, default on). A promoted replica starts with every run, queued job, timer and definition; jobs the dead master had handed to workers are re-pended by the new master's stuck-job reclaim (~30–60s) and redelivered (at-least-once, as with any visibility timeout).
  • Definition snapshotter: the fallback when workflows.db is kept node-local: pipelines, standalone cron jobs and webhooks are snapshotted (every 5 minutes, into Redis) and the ones a freshly promoted master is missing are restored before journal replay, so replayed record writes find their pipeline triggers registered. Secret values are not snapshottable (the API never returns them).
  • Leader election: with two tango replicas (the chart default), a Kubernetes lease (<statefulset>-tango-leader) decides which one may trigger automated failover; the follower still proxies traffic and can run manual promotions. GET /tango/status reports leader per replica.
  • Lag tracking: polls each node's /api/replica/local-status and the master's /api/replica/status, which reports every replica's per-database position.

Management API (:8001)

Endpoint Purpose
GET /tango/status cluster state, per-node role/health/versions, journal state, lease leadership
POST /tango/promote {"target": "http://vski-1..."}: manual failover (target optional)
POST /tango/replay apply buffered journal entries to the current master
GET /health liveness
/api/replica/* forwarded to the current master: replicas whose MASTER_URL is this port follow promotions (docker and Kubernetes deployments)

The management port belongs on an internal network: it carries the promotion controls and, through the replica forwarding, database downloads.

Actuators

How tango performs physical role changes:

  • kubernetes (set TANGO_K8S_STATEFULSET): hot-promotes the pod via the role endpoint, then patches the vski-roles ConfigMap (data[pod] = master|replica) and the vski.sh/role label the master Service selects on; no pod recreation. Fencing still deletes the old master's pod (with its ConfigMap entry set to replica first), and the restart path remains available when a node rejects the hot switch.

  • docker (TANGO_ACTUATOR=docker): drives containers through the Engine API socket (mount /var/run/docker.sock into tango). A node's container is the host of its URL (http://vski-1:3000 → vski-1), or set TANGO_DOCKER_CONTAINERS=url=name,.... Roles persist as files in TANGO_DOCKER_ROLES_DIR (<container> = master|replica), a volume the vski containers also mount, read-only, and read at start:

    command: ["sh", "-c", 'ROLE=$$(cat "/etc/vski-roles/$$VSKI_NODE" 2>/dev/null); export REPLICA_MODE="$${ROLE:-$$REPLICA_MODE}"; exec /app/vski']
    environment: { VSKI_NODE: vski-1, REPLICA_MODE: replica, MASTER_URL: "http://tango:8001" }
    

    A promotion is hot and only rewrites role files. Fencing writes replica for the old master, then SIGKILLs it; a restart policy may bring it straight back, as a replica that resyncs through tango. Replicas use tango's management port as MASTER_URL, so they follow every promotion. bench/docker-compose.yml is a complete example.

  • local (set TANGO_LOCAL_VSKI_BIN): runs vski nodes as child processes for development and the e2e tests; hot switches are recorded in the role map so restarts preserve the topology.

  • manual: refuses and tells the operator what to do.

TANGO_ACTUATOR picks one explicitly (kubernetes, docker, local, manual); unset, a StatefulSet setting means kubernetes, a local vski binary means local, and otherwise manual.

Configuration (env)

TANGO_JWT_SECRET (required, shared with every vski instance) · TANGO_NODES (required, comma-separated vski URLs; the first is presumed master until the monitor observes the cluster) · TANGO_PROXY_PORT (8000) · TANGO_MANAGEMENT_PORT (8001) · TANGO_REDIS_ADDR (empty disables the journal and snapshotter) · TANGO_REDIS_PASSWORD · TANGO_REDIS_DB (0) · TANGO_JOURNAL_RETENTION (30m) · TANGO_BUFFER_MODE (strict|absorb) · TANGO_READ_YOUR_WRITES (0s) · TANGO_MAX_READ_LAG (30s; 0 disables) · TANGO_HEALTH_INTERVAL (2s) · TANGO_HEALTH_TIMEOUT (2s) · TANGO_FAILURE_THRESHOLD (3) · TANGO_K8S_NAMESPACE / TANGO_K8S_STATEFULSET / TANGO_K8S_ROLES_CONFIGMAP · POD_NAME (lease holder identity; defaults to the hostname) · TANGO_LOCAL_VSKI_BIN / TANGO_LOCAL_DATA_ROOT (local actuator) · TANGO_ACTUATOR · TANGO_DOCKER_SOCKET (DOCKER_HOST if it is a unix socket, else /var/run/docker.sock) · TANGO_DOCKER_ROLES_DIR (/var/lib/tango/roles) · TANGO_DOCKER_CONTAINERS.

On the vski nodes (the chart sets the first two from replication.*): REPLICATION_LOG_RETENTION_HOURS (24; a replica offline longer re-downloads once) · REPLICA_INCLUDE_WORKFLOWS (true) · IDEMPOTENCY_KEY_RETENTION (2h; must exceed TANGO_JOURNAL_RETENTION).

Upgrades: upgrade replicas before the master. A master on the changeset log protocol answers an older replica's sync with 426 Upgrade Required (it cannot resume that replica's position safely); the replica stays readable but stale, and leaves tango's read pool after TANGO_MAX_READ_LAG, until it runs the new version.

Building

go build -o tango ./cmd/tango      # or: docker build -t vski/tango .

Tests

go test ./...                      # unit tests

# journal semantics against a real Redis
go test -tags integration ./internal/journal/

# vski's whole e2e suite through tango (master + replica + Redis);
# TARGET=direct runs the same topology with the suite on the master
./scripts/e2e.sh
./scripts/e2e.sh ./07_auth_test.ts

# failover: kill the master mid-write-storm, hot-promote, replay
go test -tags integration ./internal/control/ -run TestFailoverReplaysBufferedWrites -v

All need docker for a throwaway Redis; the failover test also needs a vski build with the sqlite_session tag (cd ../vski && make build). It boots a master + replica (syncing every 8s), writes through tango, writes once more right before killing the master so those writes are acknowledged but not replicated, buffers writes while the master is down (202; a login gets 503), hot-promotes the replica (same process, no restart), and asserts that every acknowledged and buffered write ends up on the new master exactly once.

Benchmark

bench/run.sh runs a three-node vski cluster, Redis and tango (docker actuator) in docker, builds both from the working trees, and compares k6 steps straight at the master with the same steps through tango; then it stops the master under load and audits every write. See bench/README.md.

Deploying

See https://vski.sh/x/k8s: the Helm chart installs the full stack: vski StatefulSet (per-pod role via the ConfigMap protocol), tango Deployment with lease-election RBAC, the selector-based master Service, Redis, RBAC and secrets.