12 Replication
Anton Nesterov edited this page 2026-11-08 21:58:35 +01:00

Replication

VSKI supports read-replica architecture for horizontal scaling. This allows you to distribute read traffic across multiple replica servers while maintaining a single master for writes. In production, run the cluster behind tango: it routes reads and writes, journals writes and fails over automatically; see the Hosting guide for deployment options.

Note

Schemas are append-only. You cannot rename columns in a database trough the interface. Dropping and renaming columns is a manual task.

Note

Tables and columns are always soft-deleted when using VSKI API. Data migration and breaking changes is an admin task that has to be done carefully and with cool head.

Note

Realtime and the automation plane (workflows, pipelines, cron, webhooks, plugins) execute on the master only, and a replica refuses those connections. You do not need to work around this: connect through tango and realtime subscriptions, workflow runs and every other feature work as usual: tango pins that traffic to the master. For custom distributed fan-out you can still combine master triggers with Redis (see Distributed Realtime).

Overview

flowchart TD
    C["Clients"] --> T["tango<br/>(proxy, journal, failover)"]
    T -->|"writes (journaled)"| M["Master<br/>(read/write)"]
    T -->|"reads"| M
    T -->|"reads"| R1["Replica 1<br/>(read-only)"]
    T -->|"reads"| R2["Replica 2<br/>(read-only)"]
    M -->|"changeset sync"| R1
    M -->|"changeset sync"| R2
  • Master: Handles all write operations and serves read requests
  • Replicas: Read-only copies that sync from master and serve read traffic
  • tango (optional but recommended): the single address clients talk to: read/write splitting, health and lag tracking, failover

Replication runs entirely over HTTP from the master: a replica never reads, mounts, or shares the master's data directory; it maintains its own DATA_DIR populated by the sync protocol. The server (master or replica) must be the only process accessing its data directory; see Backup for the access rules.

Deploying with tango

tango is the vski cluster proxy. Point every client (SDKs, the web dashboard, workers) at tango and it takes care of the rest:

  • Read/write splitting: writes always go to the master; reads are spread least-connections across the master and every replica that is healthy and fresh. Realtime websockets and the automation plane (/api/workflows, /api/pipelines, /api/cron, /api/webhooks, /api/plugins, /api/secrets) are pinned to the master, where that state executes.
  • Read freshness: a replica that has not reached the position the master held TANGO_MAX_READ_LAG ago (default 30s) leaves the read pool until it catches up.
  • Read-your-writes: with TANGO_READ_YOUR_WRITES set, a client's reads go to the master for that window after its write.
  • Write journal: every write is appended to a Redis Stream before it is forwarded, with an idempotency key the master records in the same transaction. Writes are replayed exactly once after a failover.
  • Failover: when the master dies, tango fences it and hot-promotes the most-current replica: vski flips the node's role in-process (milliseconds, no restart) and the remaining replicas catch up from the promoted node's changeset log. Workflows survive: a promoted replica starts with every run, queued job, timer and definition.
  • Replica sync through tango: replicas set MASTER_URL to tango's management port, which forwards /api/replica/* to whichever node is master, so a promotion needs no reconfiguration anywhere.
  • Management API (:8001): GET /tango/status for cluster state, POST /tango/promote for manual failover, and the replica sync forwarding above.

Deploy it with the vski-cluster Helm chart (Kubernetes) or docker compose; the Hosting guide covers both, including scaling procedures and S3 file storage. The images (vski.sh/x/vski, vski.sh/x/tango) are published for linux/amd64 and linux/arm64; Kubernetes needs tango v1.0.2 or later.

Quick Start

Replicas need no registration on the master: any server with the shared JWT_SECRET can sync. To try replication by hand:

1. Start Master Server

# Master server (default mode)
JWT_SECRET=your-shared-secret DATA_DIR=./data vski serve --port 3001

2. Start Replica Server

# Replica server - just needs MASTER_URL and same JWT_SECRET
REPLICA_MODE=replica \
MASTER_URL=http://localhost:3001 \
JWT_SECRET=your-shared-secret \
DATA_DIR=./data-replica \
SYNC_INTERVAL=5 \
vski serve --port 3002

The replica will automatically:

  1. Generate a JWT token using the shared secret
  2. Connect to master and authenticate
  3. Download all databases
  4. Start periodic sync

That's it! No need to register replicas on master. Just ensure both servers use the same JWT_SECRET. Reads hit the replica directly; writes go to the master, or let tango do that routing for you.

Configuration

Environment Variables

Variable Description Default Required
REPLICA_MODE Server mode: master or replica master No
MASTER_URL URL of the master server - Yes (if replica)
SYNC_INTERVAL Sync interval in seconds (0 = manual only) 60 No
JWT_SECRET Must be the same on master and replicas - Yes
REPLICATION_LOG_RETENTION_HOURS Hours the changeset log keeps entries for incremental catch-up 24 No
REPLICA_INCLUDE_WORKFLOWS Replicate the workflows database as a passive copy (false to keep it node-local) true No
IDEMPOTENCY_KEY_RETENTION How long the master remembers the keys tango stamps on writes (exactly-once replay after a failover); must exceed tango's TANGO_JOURNAL_RETENTION 2h No

In docker deployments, replicas can point MASTER_URL at tango's management port instead: tango forwards the replication API to whichever node is master, so replicas follow every promotion without reconfiguration.

JWT Secret Requirement

Both master and replicas must use the same JWT_SECRET. This ensures that:

  • Tokens generated on master are valid on replicas
  • Users can authenticate on any server
  • Replicas can authenticate with master

How Authentication Works

  1. Replica generates its own JWT token using JWT_SECRET
  2. Replica sends this token to master in the Authorization: Bearer header
  3. Master validates the token using the same JWT_SECRET
  4. No pre-registration needed - any server with the shared secret can sync

This simplified approach means you can spin up new replicas without any configuration on the master.

Sync Process

Initial Sync

When a replica first connects, or its position cannot be verified, a full sync occurs:

  1. Replica requests database status from master
  2. Master takes a consistent snapshot (SQLite online backup, so commits still in the WAL are included) and reports the changeset-log position it covers
  3. Replica restores the snapshot in place and anchors its own changeset log at that position

Incremental Sync

For ongoing updates, replicas read the master's durable changeset log:

  1. Replica computes schema hash from local sqlite_master
  2. Replica fetches schema hash from master
  3. If hashes match: Replica asks for log entries after its position
  4. If hashes differ: Schema migration occurs first (see below)
  5. Replica applies each entry in order and appends it to its own log

Schema Synchronization

When schema changes are detected (new collections, field changes, indexes), replicas automatically migrate their schema:

  1. Hash-Based Detection: Schema hashes are computed from sqlite_master (tables, indexes, triggers, views)
  2. Schema Fetch: Replica fetches full schema from master via /api/replica/schema-sync
  3. Diff Computation: Replica computes schema diff:
    • New tables to create
    • Tables with new columns (ALTER TABLE)
    • New/dropped indexes, triggers, views
  4. Migration Application:
    • CREATE TABLE for new collections
    • ALTER TABLE ADD COLUMN for new fields
    • CREATE INDEX, CREATE TRIGGER, CREATE VIEW for new objects
    • DROP INDEX, DROP TRIGGER, DROP VIEW for removed objects
    • Tables are never dropped on replica

Changeset Sync

After schema migration (if needed), data changes are synced from the changeset log: an append-only file per database under DATA_DIR/replication-log/:

  1. Every write the master commits is captured by an SQLite session; right before the transaction commits, its changeset is appended to the log (fsync'd) under the next changeset version. Committed data therefore never lacks its log entry, and the log survives master restarts.
  2. A replica polls with its position: the version of its newest log entry plus the master's chain hash at that version (each entry hashes over its predecessor).
  3. The master serves the entries after that position if the chain hash matches its own log. Otherwise (the position was pruned, the replica is ahead, or the histories forked after a failover onto a node that missed writes) it answers "full sync required" instead of mixing histories.
  4. Replicas mirror the master's log entry for entry, so a promoted replica serves its former siblings incrementally from the same log.

Entries older than REPLICATION_LOG_RETENTION_HOURS are pruned (the newest entry is always kept); a replica offline longer than that re-downloads once. A master that has never been asked for a database by a replica does not log it; standalone nodes pay nothing.

Writes that land in one batch replay in commit order on every node: workflow history is ordered by a per-run sequence assigned in the committing transaction, so a promoted replica hands workers the same event order the old master did. A changeset that carries a column or table the replica does not have yet is never applied silently short: the replica re-syncs the schema first.

Upgrading: upgrade replicas before the master. A master on the changeset log protocol answers a replica running an older version with 426 Upgrade Required: it cannot resume that replica's position safely, and the replica stays readable but stale until it is upgraded (tango takes it out of the read pool once it falls behind).

Omitted Databases

The stats database (statistics and logs) is always excluded: every node generates its own.

The workflows database (runs, queued jobs, timers, pipeline, cron and webhook definitions) replicates as a passive copy by default, so a promoted replica starts with complete workflow state. Workflow execution is master-only regardless: replicas refuse workflow writes and queue polls (503), and their workflow gateway, scheduler, cron dispatch and webhooks stay idle. Set REPLICA_INCLUDE_WORKFLOWS=false to keep it node-local.

You can also exclude custom databases using the REPLICA_OMIT_DBS environment variable:

# Exclude specific databases from replication
REPLICA_OMIT_DBS=temp,cache,logs

Sync Interval

Configure how often replicas sync:

# Sync every 5 seconds (near real-time)
SYNC_INTERVAL=5

# Sync every minute (default)
SYNC_INTERVAL=60

# Manual sync only
SYNC_INTERVAL=0

Sync Latency Considerations

Replicas may be slightly behind the master depending on the SYNC_INTERVAL configuration. How much that matters depends on how clients reach the cluster:

  • Read-your-writes: set TANGO_READ_YOUR_WRITES (e.g. 5s) and a client's reads go to the master for that window after its write, with no code changes, the guarantee is per client identity.
  • Stale replicas never serve reads: anything behind TANGO_MAX_READ_LAG (default 30s) is out of the read pool until it catches up.
  • Realtime and automation always hit the master, so subscriptions and workflow-driven apps behave exactly as on a single node.

Direct connections

  • Route to master when you need read-your-writes, realtime subscriptions or any automation-plane call
  • Replicas are fine for read-heavy, eventually-consistent workloads and for serving reads from nearby regions
Use case Sync interval Notes
Realtime features any Always master; through tango this is automatic
Near real-time reads 5-10 seconds Acceptable for most interactive apps
Standard reads 30-60 seconds Default, good for most use cases
Analytics / Reporting 60+ seconds Longer intervals reduce master load

Read-Only Behavior

When running in replica mode:

  • All write operations (POST, PUT, PATCH, DELETE) return 403 Forbidden
  • Logins still work: /api/auth/login, /api/auth/refresh, /api/auth/exchange, /api/admins/auth-with-password, /api/collections/<name>/auth-with-password and the hosted login form mint tokens from replicated credentials without writing anything
  • Registration, OAuth account linking, admin init and API key creation create rows, so they are refused like any other write; send them to the master
  • All read operations (GET) work normally
  • Below the HTTP layer, every write a replica would make to a replicated database is refused (replica is read-only: writes go to the master): replicated data changes only through the sync, so a replica can never hold rows its master and the changeset log never saw

Example

# Read operation - allowed on replica
curl http://replica:3002/api/collections/posts/records

# Write operation - blocked on replica
curl -X POST http://replica:3002/api/collections/posts/records \
  -H "Content-Type: application/json" \
  -d '{"title": "New Post"}'
# Returns: {"error": "replica is read-only", "code": 403}

Headers

Every response is stamped with the node's replication role, so proxies and monitors can route without guessing:

Header Description
X-Replica-Mode Current mode: replica or master
X-Replica-Id Unique identifier for this replica
X-Master-URL URL of the master server (replica only)

A replica's ID comes from REPLICA_ID and its public address from PUBLIC_URL. Client SDKs capture the X-Replica-Id of the node that served a response, which is useful for debugging which node answered.

Routing and Load Balancing

Use tango. It is the supported front door for a cluster: it splits reads and writes, keeps stale replicas out of the read pool, pins realtime and automation traffic to the master, journals writes for exactly-once replay, and fails over automatically. Clients need exactly one URL: tango's.

If you cannot put tango in front, the routing rules it implements are the rules to replicate yourself:

  • Writes (non-GET) go to the master and to nobody else
  • Realtime websockets and the automation plane (/api/workflows, /api/pipelines, /api/cron, /api/webhooks, /api/plugins, /api/secrets) go to the master
  • Reads may go to any node that is healthy and in sync; check X-Replica-Mode and each node's /api/replica/local-status
  • Failover is on you: promote a replica (see below) and repoint the write path at it

Failover

Failover is automatic when tango fronts the cluster:

  1. tango detects the master is down: TANGO_FAILURE_THRESHOLD consecutive failed probes (one every TANGO_HEALTH_INTERVAL), then three monitor rounds without a master; about ten seconds with the defaults
  2. It fences the old master so it cannot accept writes again. On Kubernetes the role map records the pod as a replica and that pod is deleted; tango waits up to 15 seconds for the StatefulSet to replace it, then proceeds even if the pod's node is unreachable. With docker it is a SIGKILL through the Docker socket
  3. It picks the healthy replica with the most changeset entries applied, never the node it just fenced, and calls its POST /api/admin/promote: vski flips the node's role in-process (write path, dispatcher, cron, workflow gateway, changeset journal) without a restart, in milliseconds
  4. Journal entries the new master may lack replay into it in order, with the callers' original credentials, so acknowledged-but-unreplicated writes are not lost, and replays are idempotent (X-Tango-Idempotency-Key)
  5. Remaining replicas catch up from the new master's changeset log; one whose position no longer belongs to the new history re-downloads instead of keeping writes the new master never saw

tango runs strict (writes refused with 503 + Retry-After during the window) or absorb (writes journaled and answered 202); set TANGO_BUFFER_MODE. Manual failover is POST /tango/promote on the management port. On Kubernetes two tango replicas elect a leader through a lease so only one acts: send the promotion to the lease holder (leader_id in /tango/status); the other replica answers 409 with the leader's name.

If two nodes report master at once (an old master still answering while its pod terminates), tango holds writes until exactly one does and starts no automated failover meanwhile; /tango/status shows split_brain: true.

Without tango, you can promote a replica by hand: stop the old master, call POST /api/admin/promote on the chosen replica (or restart it with REPLICA_MODE=master), and point the remaining replicas' MASTER_URL at it.

Client SDK Usage

Through tango, one client does everything (reads and writes, realtime, workflows):

from vski import VskiClient

# tango is the only address your app needs
client = VskiClient("http://tango:8000")
await client.admins.auth_with_password("admin@example.com", "password")

# writes are journaled and go to the master
await client.collection("posts").create({"title": "New Post"})

# reads spread across the master and healthy replicas
posts = await client.collection("posts").get_list(1, 10)

Against individual nodes, keep two clients: a read client on a replica and a write client on the master:

read_client = VskiClient("http://replica:3002")   # GETs
write_client = VskiClient("http://master:3001")   # everything else

Monitoring

Health Check

Check any node:

curl http://replica:3002/health

Response includes headers indicating replica status.

Sync Status

Check replication status on master:

curl -H "Authorization: Bearer <replica-jwt>" http://master:3001/api/replica/status

Response:

{
  "dbs": [
    {
      "name": "default",
      "size": 1048576,
      "sha256": "abc123...",
      "schemaVersion": 5
    }
  ]
}

Cluster Status (tango)

curl http://tango:8001/tango/status

Reports every node's role, health and version, per-database positions, the journal state, lease leadership (leader, leader_id) and split_brain.

Security

Authentication

Replicas authenticate using JWT tokens generated from the shared JWT_SECRET. No additional keys or registration required.

Best Practices

  1. Use HTTPS in production
  2. Keep JWT_SECRET secret and consistent across all servers
  3. Use strong secrets - at least 32 characters of random data
  4. Rotate secrets by updating all servers simultaneously
  5. Keep tango's management port (:8001) on an internal network: it carries the promotion controls and replica downloads (the Helm chart's networkPolicy.enabled limits it to the vski pods)

Limitations

Current limitations (may be addressed in future releases):

  1. Single writer: all writes go through the master. tango adds failover and read scaling, not multi-master writes.
  2. Full database changesets: changesets apply to entire databases, not per-collection.
  3. Master-only planes: realtime, workflows, cron and webhooks execute on the master only: transparent through tango, refused on replicas directly.

Troubleshooting

Replica Not Syncing

  1. Check network connectivity to master
  2. Verify MASTER_URL is correct
  3. Verify JWT_SECRET is identical on master and replica
  4. Check master logs for authentication errors
  5. A 426 Upgrade Required answer means the replica runs an older version than the master; upgrade replicas before the master

Replica Dropped from the Read Pool (tango)

The replica is behind TANGO_MAX_READ_LAG: check its sync interval, its logs, and that it can reach the master (directly or through tango's management port). It rejoins on its own once it catches up.

Authentication Fails on Replica

Ensure JWT_SECRET is identical on both master and replica:

# On master
echo $JWT_SECRET

# On replica (must match!)
echo $JWT_SECRET

Data Not Appearing on Replica

  1. Check sync interval: SYNC_INTERVAL
  2. Wait for next sync cycle
  3. Check replica logs for sync errors

Schema Not Syncing

If new collections appear on master but not on replica:

  1. Check replica logs for "schema mismatch" messages
  2. Verify replica can reach /api/replica/schema-sync endpoint
  3. Check for SQL errors in replica logs during migration

Best Practices

  1. Front the cluster with tango: routing, freshness and failover are solved problems there
  2. Monitor sync lag: GET /tango/status or /api/replica/status
  3. Keep a replica for backups: snapshot a stopped replica so production never pauses; see Backup
  4. Use S3 file storage in clustered deployments: every node reads the same bucket and the local file-sync journal turns off; see File Sync
  5. Upgrade replicas before the master
  6. Secure communication: HTTPS between components, management ports internal