- Go 86.7%
- Shell 6.9%
- Python 3.6%
- JavaScript 2.2%
- Dockerfile 0.3%
- Other 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Found by the in-cluster failover test: - A tango replica that saw the master answer as a replica (its pod being replaced) never marked it master again once it came back under the same address, so that replica refused every write after the failover. The monitor now re-asserts the role of the single node reporting master. - An automated failover fenced the old master and then picked it as the candidate (its replacement pod passed probes and held the most data), hot promote hit a pod that was being deleted, and the restart fallback ran. The fenced node is skipped unless named explicitly. - Startup waits up to two minutes for Redis instead of exiting. |
||
| bench | ||
| cmd/tango | ||
| internal | ||
| scripts | ||
| .gitignore | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
Tango
The vski cluster proxy: read/write splitting across a vski master and its replicas, a Redis-backed write journal, and failover with hot promotion of the most-current replica.
clients ──► tango :8000 ── writes ──► master (journaled first, replayed on failover)
│
└── reads ──► least-connections across healthy replicas + master
What it does
- Routing: every mutation is pinned to the master. Reads go
least-connections over healthy nodes, except the workflow/automation
plane: realtime and workflow websockets and every read under
/api/workflows,/api/pipelines,/api/cron,/api/webhooks,/api/plugins,/api/plugin-instancesand/api/secretsgo to the master, where that state executes; a worker never polls its run on a lagging replica. Some endpoints go to the master for every method and are never journaled: sessions (/sso/*,/api/auth/*,/api/admins/*,/api/keys,/api/collections/<name>/auth-with-password: hosted login keeps its codes and OAuth state in the master's memory, and logins carry passwords that must not sit in the journal; during a failover they answer 503 + Retry-After rather than a token-less 202) and admin scripts (/api/sql/execute,/api/migrations/*: no idempotency key, so a replay could apply them twice). The instance control endpoints (/api/replica/*,/api/admin/*) are never proxied. A read whose node died between health probes is retried on another node; an unreachable master answers 503 + Retry-After. - Read freshness: a replica that has not reached, in every database,
the position the master held
TANGO_MAX_READ_LAGago (default 30s) leaves the read pool until it catches up: a stuck syncer, a replica cut off from the master, or one too old to speak the changeset log protocol. - Read-your-writes: with
TANGO_READ_YOUR_WRITESset, a client's reads go to the master for that window after its write. The client is itsAuthorizationheader, itsX-API-Key, or else its address. - Write journal: with
TANGO_REDIS_ADDRset, every write is appended to a Redis Stream before it is forwarded, carrying an idempotency key that the master records in the same transaction as the write. A write the master applied (2xx) stays replayable until trimmed; one the client saw fail (rejected, or refused with 503 during a failover instrictmode) is deleted, because the client may retry it. On failover, the entries the candidate may lack replay into it in journal order with the caller's original credentials: everything never acknowledged, plus everything applied after the newest master position the candidate is known to hold. That recovers writes the old master acknowledged but had not replicated when it died. Writes absorbed while the master was briefly unreachable replay into it when it comes back without a failover. - Exactly-once replay: vski records each write's
X-Tango-Idempotency-Keyin a_tango_appliedtable inside the same transaction as the write, so a replay of a write the target already holds answers200 {"duplicate":true}instead of creating a second record. Record and graph creates are keyed; updates and deletes replay in journal order and land on the same final state. The table replicates with everything else, so a promoted replica knows every write it received; keys are pruned afterIDEMPOTENCY_KEY_RETENTION(vski, default 2h), which must exceedTANGO_JOURNAL_RETENTION. - Failover (hot): fence the old master, then call the candidate's
POST /api/admin/promote: vski flips its role in-process (write path, dispatcher, cron, workflow gateway, changeset journal, the full lifecycle) without restarting, and tango persists the new role via the actuator. The candidate is the healthy replica with the most changeset entries applied across all databases. Failover completes in milliseconds; a restart-based promotion remains the fallback for nodes without the endpoint.strictmode refuses writes during the window (503 + Retry-After);absorbmode journals them and returns 202. - Durable changeset log: vski appends every committed write to an
on-disk changeset log (
DATA_DIR/replication-log/, fsync'd before the transaction commits) and replicas mirror it entry for entry, resuming withsince=Nplus a hash-chain token. A master restart costs replicas nothing, and after a promotion the remaining replicas catch up from the promoted node's log instead of re-downloading. A replica whose position does not belong to the new master's history (the candidate missed writes the replica already had) is told to re-download rather than keep writes the new master never saw; journal replay then restores them through the new master. - Workflows survive failover: workflow execution (gateway, scheduler,
cron dispatch, webhooks) is master-only, and the master's
workflows.dbreplicates to replicas as a passive copy (REPLICA_INCLUDE_WORKFLOWS, default on). A promoted replica starts with every run, queued job, timer and definition; jobs the dead master had handed to workers are re-pended by the new master's stuck-job reclaim (~30–60s) and redelivered (at-least-once, as with any visibility timeout). - Definition snapshotter: the fallback when
workflows.dbis kept node-local: pipelines, standalone cron jobs and webhooks are snapshotted (every 5 minutes, into Redis) and the ones a freshly promoted master is missing are restored before journal replay, so replayed record writes find their pipeline triggers registered. Secret values are not snapshottable (the API never returns them). - Leader election: with two tango replicas (the chart default), a
Kubernetes lease (
<statefulset>-tango-leader) decides which one may trigger automated failover; the follower still proxies traffic and can run manual promotions.GET /tango/statusreportsleaderper replica. - Lag tracking: polls each node's
/api/replica/local-statusand the master's/api/replica/status, which reports every replica's per-database position.
Management API (:8001)
| Endpoint | Purpose |
|---|---|
GET /tango/status |
cluster state, per-node role/health/versions, journal state, lease leadership |
POST /tango/promote |
{"target": "http://vski-1..."}: manual failover (target optional) |
POST /tango/replay |
apply buffered journal entries to the current master |
GET /health |
liveness |
/api/replica/* |
forwarded to the current master: replicas whose MASTER_URL is this port follow promotions (docker and Kubernetes deployments) |
The management port belongs on an internal network: it carries the promotion controls and, through the replica forwarding, database downloads.
Actuators
How tango performs physical role changes:
-
kubernetes (set
TANGO_K8S_STATEFULSET): hot-promotes the pod via the role endpoint, then patches thevski-rolesConfigMap (data[pod] = master|replica) and thevski.sh/rolelabel the master Service selects on; no pod recreation. Fencing still deletes the old master's pod (with its ConfigMap entry set toreplicafirst), and the restart path remains available when a node rejects the hot switch. -
docker (
TANGO_ACTUATOR=docker): drives containers through the Engine API socket (mount/var/run/docker.sockinto tango). A node's container is the host of its URL (http://vski-1:3000→vski-1), or setTANGO_DOCKER_CONTAINERS=url=name,.... Roles persist as files inTANGO_DOCKER_ROLES_DIR(<container>=master|replica), a volume the vski containers also mount, read-only, and read at start:command: ["sh", "-c", 'ROLE=$$(cat "/etc/vski-roles/$$VSKI_NODE" 2>/dev/null); export REPLICA_MODE="$${ROLE:-$$REPLICA_MODE}"; exec /app/vski'] environment: { VSKI_NODE: vski-1, REPLICA_MODE: replica, MASTER_URL: "http://tango:8001" }A promotion is hot and only rewrites role files. Fencing writes
replicafor the old master, then SIGKILLs it; a restart policy may bring it straight back, as a replica that resyncs through tango. Replicas use tango's management port asMASTER_URL, so they follow every promotion.bench/docker-compose.ymlis a complete example. -
local (set
TANGO_LOCAL_VSKI_BIN): runs vski nodes as child processes for development and the e2e tests; hot switches are recorded in the role map so restarts preserve the topology. -
manual: refuses and tells the operator what to do.
TANGO_ACTUATOR picks one explicitly (kubernetes, docker, local,
manual); unset, a StatefulSet setting means kubernetes, a local vski binary
means local, and otherwise manual.
Configuration (env)
TANGO_JWT_SECRET (required, shared with every vski instance) ·
TANGO_NODES (required, comma-separated vski URLs; the first is presumed
master until the monitor observes the cluster) · TANGO_PROXY_PORT (8000) ·
TANGO_MANAGEMENT_PORT (8001) · TANGO_REDIS_ADDR (empty disables the
journal and snapshotter) · TANGO_REDIS_PASSWORD · TANGO_REDIS_DB (0) ·
TANGO_JOURNAL_RETENTION (30m) · TANGO_BUFFER_MODE (strict|absorb) ·
TANGO_READ_YOUR_WRITES (0s) · TANGO_MAX_READ_LAG (30s; 0 disables) ·
TANGO_HEALTH_INTERVAL (2s) · TANGO_HEALTH_TIMEOUT (2s) ·
TANGO_FAILURE_THRESHOLD (3) · TANGO_K8S_NAMESPACE /
TANGO_K8S_STATEFULSET / TANGO_K8S_ROLES_CONFIGMAP · POD_NAME (lease
holder identity; defaults to the hostname) · TANGO_LOCAL_VSKI_BIN /
TANGO_LOCAL_DATA_ROOT (local actuator) · TANGO_ACTUATOR ·
TANGO_DOCKER_SOCKET (DOCKER_HOST if it is a unix socket, else
/var/run/docker.sock) · TANGO_DOCKER_ROLES_DIR (/var/lib/tango/roles) ·
TANGO_DOCKER_CONTAINERS.
On the vski nodes (the chart sets the first two from replication.*):
REPLICATION_LOG_RETENTION_HOURS (24; a replica offline longer re-downloads
once) · REPLICA_INCLUDE_WORKFLOWS (true) · IDEMPOTENCY_KEY_RETENTION
(2h; must exceed TANGO_JOURNAL_RETENTION).
Upgrades: upgrade replicas before the master. A master on the changeset log
protocol answers an older replica's sync with 426 Upgrade Required (it
cannot resume that replica's position safely); the replica stays readable
but stale, and leaves tango's read pool after TANGO_MAX_READ_LAG, until it
runs the new version.
Building
go build -o tango ./cmd/tango # or: docker build -t vski/tango .
Tests
go test ./... # unit tests
# journal semantics against a real Redis
go test -tags integration ./internal/journal/
# vski's whole e2e suite through tango (master + replica + Redis);
# TARGET=direct runs the same topology with the suite on the master
./scripts/e2e.sh
./scripts/e2e.sh ./07_auth_test.ts
# failover: kill the master mid-write-storm, hot-promote, replay
go test -tags integration ./internal/control/ -run TestFailoverReplaysBufferedWrites -v
All need docker for a throwaway Redis; the failover test also needs a vski
build with the sqlite_session tag (cd ../vski && make build). It boots a
master + replica (syncing every 8s), writes through tango, writes once more
right before killing the master so those writes are acknowledged but not
replicated, buffers writes while the master is down (202; a login gets 503),
hot-promotes the replica (same process, no restart), and asserts that every
acknowledged and buffered write ends up on the new master exactly once.
Benchmark
bench/run.sh runs a three-node vski cluster, Redis and tango (docker
actuator) in docker, builds both from the working trees, and compares k6
steps straight at the master with the same steps through tango; then it
stops the master under load and audits every write. See bench/README.md.
Deploying
See https://vski.sh/x/k8s: the Helm chart installs the full stack: vski StatefulSet (per-pod role via the ConfigMap protocol), tango Deployment with lease-election RBAC, the selector-based master Service, Redis, RBAC and secrets.