Table of contents
- Replication
- Overview
- Deploying with tango
- Quick Start
- Configuration
- How Authentication Works
- Sync Process
- Sync Latency Considerations
- Read-Only Behavior
- Headers
- Routing and Load Balancing
- Failover
- Client SDK Usage
- Monitoring
- Security
- Limitations
- Troubleshooting
- Replica Not Syncing
- Replica Dropped from the Read Pool (tango)
- Authentication Fails on Replica
- Data Not Appearing on Replica
- Schema Not Syncing
- Best Practices
Replication
VSKI supports read-replica architecture for horizontal scaling. This allows you to distribute read traffic across multiple replica servers while maintaining a single master for writes. In production, run the cluster behind tango: it routes reads and writes, journals writes and fails over automatically; see the Hosting guide for deployment options.
Note
Schemas are append-only. You cannot rename columns in a database trough the interface. Dropping and renaming columns is a manual task.
Note
Tables and columns are always soft-deleted when using VSKI API. Data migration and breaking changes is an admin task that has to be done carefully and with cool head.
Note
Realtime and the automation plane (workflows, pipelines, cron, webhooks, plugins) execute on the master only, and a replica refuses those connections. You do not need to work around this: connect through tango and realtime subscriptions, workflow runs and every other feature work as usual: tango pins that traffic to the master. For custom distributed fan-out you can still combine master triggers with Redis (see Distributed Realtime).
Overview
flowchart TD
C["Clients"] --> T["tango<br/>(proxy, journal, failover)"]
T -->|"writes (journaled)"| M["Master<br/>(read/write)"]
T -->|"reads"| M
T -->|"reads"| R1["Replica 1<br/>(read-only)"]
T -->|"reads"| R2["Replica 2<br/>(read-only)"]
M -->|"changeset sync"| R1
M -->|"changeset sync"| R2
- Master: Handles all write operations and serves read requests
- Replicas: Read-only copies that sync from master and serve read traffic
- tango (optional but recommended): the single address clients talk to: read/write splitting, health and lag tracking, failover
Replication runs entirely over HTTP from the master: a replica never reads,
mounts, or shares the master's data directory; it maintains its own
DATA_DIR populated by the sync protocol. The server (master or replica)
must be the only process accessing its data directory; see
Backup for the access rules.
Deploying with tango
tango is the vski cluster proxy. Point every client (SDKs, the web dashboard, workers) at tango and it takes care of the rest:
- Read/write splitting: writes always go to the master; reads are spread
least-connections across the master and every replica that is healthy and
fresh. Realtime websockets and the automation plane (
/api/workflows,/api/pipelines,/api/cron,/api/webhooks,/api/plugins,/api/secrets) are pinned to the master, where that state executes. - Read freshness: a replica that has not reached the position the master
held
TANGO_MAX_READ_LAGago (default 30s) leaves the read pool until it catches up. - Read-your-writes: with
TANGO_READ_YOUR_WRITESset, a client's reads go to the master for that window after its write. - Write journal: every write is appended to a Redis Stream before it is forwarded, with an idempotency key the master records in the same transaction. Writes are replayed exactly once after a failover.
- Failover: when the master dies, tango fences it and hot-promotes the most-current replica: vski flips the node's role in-process (milliseconds, no restart) and the remaining replicas catch up from the promoted node's changeset log. Workflows survive: a promoted replica starts with every run, queued job, timer and definition.
- Replica sync through tango: replicas set
MASTER_URLto tango's management port, which forwards/api/replica/*to whichever node is master, so a promotion needs no reconfiguration anywhere. - Management API (
:8001):GET /tango/statusfor cluster state,POST /tango/promotefor manual failover, and the replica sync forwarding above.
Deploy it with the vski-cluster Helm chart
(Kubernetes) or docker compose; the Hosting guide
covers both, including scaling procedures and S3 file storage. The images
(vski.sh/x/vski, vski.sh/x/tango) are published for linux/amd64 and
linux/arm64; Kubernetes needs tango v1.0.2 or later.
Quick Start
Replicas need no registration on the master: any server with the shared
JWT_SECRET can sync. To try replication by hand:
1. Start Master Server
# Master server (default mode)
JWT_SECRET=your-shared-secret DATA_DIR=./data vski serve --port 3001
2. Start Replica Server
# Replica server - just needs MASTER_URL and same JWT_SECRET
REPLICA_MODE=replica \
MASTER_URL=http://localhost:3001 \
JWT_SECRET=your-shared-secret \
DATA_DIR=./data-replica \
SYNC_INTERVAL=5 \
vski serve --port 3002
The replica will automatically:
- Generate a JWT token using the shared secret
- Connect to master and authenticate
- Download all databases
- Start periodic sync
That's it! No need to register replicas on master. Just ensure both servers
use the same JWT_SECRET. Reads hit the replica directly; writes go to the
master, or let tango do that routing for you.
Configuration
Environment Variables
| Variable | Description | Default | Required |
|---|---|---|---|
REPLICA_MODE |
Server mode: master or replica |
master |
No |
MASTER_URL |
URL of the master server | - | Yes (if replica) |
SYNC_INTERVAL |
Sync interval in seconds (0 = manual only) | 60 |
No |
JWT_SECRET |
Must be the same on master and replicas | - | Yes |
REPLICATION_LOG_RETENTION_HOURS |
Hours the changeset log keeps entries for incremental catch-up | 24 |
No |
REPLICA_INCLUDE_WORKFLOWS |
Replicate the workflows database as a passive copy (false to keep it node-local) |
true |
No |
IDEMPOTENCY_KEY_RETENTION |
How long the master remembers the keys tango stamps on writes (exactly-once replay after a failover); must exceed tango's TANGO_JOURNAL_RETENTION |
2h |
No |
In docker deployments, replicas can point MASTER_URL at tango's management
port instead: tango forwards the replication API to whichever node is master,
so replicas follow every promotion without reconfiguration.
JWT Secret Requirement
Both master and replicas must use the same JWT_SECRET. This ensures that:
- Tokens generated on master are valid on replicas
- Users can authenticate on any server
- Replicas can authenticate with master
How Authentication Works
- Replica generates its own JWT token using
JWT_SECRET - Replica sends this token to master in the
Authorization: Bearerheader - Master validates the token using the same
JWT_SECRET - No pre-registration needed - any server with the shared secret can sync
This simplified approach means you can spin up new replicas without any configuration on the master.
Sync Process
Initial Sync
When a replica first connects, or its position cannot be verified, a full sync occurs:
- Replica requests database status from master
- Master takes a consistent snapshot (SQLite online backup, so commits still in the WAL are included) and reports the changeset-log position it covers
- Replica restores the snapshot in place and anchors its own changeset log at that position
Incremental Sync
For ongoing updates, replicas read the master's durable changeset log:
- Replica computes schema hash from local
sqlite_master - Replica fetches schema hash from master
- If hashes match: Replica asks for log entries after its position
- If hashes differ: Schema migration occurs first (see below)
- Replica applies each entry in order and appends it to its own log
Schema Synchronization
When schema changes are detected (new collections, field changes, indexes), replicas automatically migrate their schema:
- Hash-Based Detection: Schema hashes are computed from
sqlite_master(tables, indexes, triggers, views) - Schema Fetch: Replica fetches full schema from master via
/api/replica/schema-sync - Diff Computation: Replica computes schema diff:
- New tables to create
- Tables with new columns (ALTER TABLE)
- New/dropped indexes, triggers, views
- Migration Application:
CREATE TABLEfor new collectionsALTER TABLE ADD COLUMNfor new fieldsCREATE INDEX,CREATE TRIGGER,CREATE VIEWfor new objectsDROP INDEX,DROP TRIGGER,DROP VIEWfor removed objects- Tables are never dropped on replica
Changeset Sync
After schema migration (if needed), data changes are synced from the
changeset log: an append-only file per database under
DATA_DIR/replication-log/:
- Every write the master commits is captured by an SQLite session; right before the transaction commits, its changeset is appended to the log (fsync'd) under the next changeset version. Committed data therefore never lacks its log entry, and the log survives master restarts.
- A replica polls with its position: the version of its newest log entry plus the master's chain hash at that version (each entry hashes over its predecessor).
- The master serves the entries after that position if the chain hash matches its own log. Otherwise (the position was pruned, the replica is ahead, or the histories forked after a failover onto a node that missed writes) it answers "full sync required" instead of mixing histories.
- Replicas mirror the master's log entry for entry, so a promoted replica serves its former siblings incrementally from the same log.
Entries older than REPLICATION_LOG_RETENTION_HOURS are pruned (the newest
entry is always kept); a replica offline longer than that re-downloads once.
A master that has never been asked for a database by a replica does not log
it; standalone nodes pay nothing.
Writes that land in one batch replay in commit order on every node: workflow history is ordered by a per-run sequence assigned in the committing transaction, so a promoted replica hands workers the same event order the old master did. A changeset that carries a column or table the replica does not have yet is never applied silently short: the replica re-syncs the schema first.
Upgrading: upgrade replicas before the master. A master on the changeset
log protocol answers a replica running an older version with
426 Upgrade Required: it cannot resume that replica's position safely,
and the replica stays readable but stale until it is upgraded (tango takes it
out of the read pool once it falls behind).
Omitted Databases
The stats database (statistics and logs) is always excluded: every node
generates its own.
The workflows database (runs, queued jobs, timers, pipeline, cron and
webhook definitions) replicates as a passive copy by default, so a promoted
replica starts with complete workflow state. Workflow execution is
master-only regardless: replicas refuse workflow writes and queue polls
(503), and their workflow gateway, scheduler, cron dispatch and webhooks
stay idle. Set REPLICA_INCLUDE_WORKFLOWS=false to keep it node-local.
You can also exclude custom databases using the REPLICA_OMIT_DBS environment
variable:
# Exclude specific databases from replication
REPLICA_OMIT_DBS=temp,cache,logs
Sync Interval
Configure how often replicas sync:
# Sync every 5 seconds (near real-time)
SYNC_INTERVAL=5
# Sync every minute (default)
SYNC_INTERVAL=60
# Manual sync only
SYNC_INTERVAL=0
Sync Latency Considerations
Replicas may be slightly behind the master depending on the SYNC_INTERVAL
configuration. How much that matters depends on how clients reach the cluster:
Through tango (recommended)
- Read-your-writes: set
TANGO_READ_YOUR_WRITES(e.g.5s) and a client's reads go to the master for that window after its write, with no code changes, the guarantee is per client identity. - Stale replicas never serve reads: anything behind
TANGO_MAX_READ_LAG(default 30s) is out of the read pool until it catches up. - Realtime and automation always hit the master, so subscriptions and workflow-driven apps behave exactly as on a single node.
Direct connections
- Route to master when you need read-your-writes, realtime subscriptions or any automation-plane call
- Replicas are fine for read-heavy, eventually-consistent workloads and for serving reads from nearby regions
| Use case | Sync interval | Notes |
|---|---|---|
| Realtime features | any | Always master; through tango this is automatic |
| Near real-time reads | 5-10 seconds | Acceptable for most interactive apps |
| Standard reads | 30-60 seconds | Default, good for most use cases |
| Analytics / Reporting | 60+ seconds | Longer intervals reduce master load |
Read-Only Behavior
When running in replica mode:
- All write operations (POST, PUT, PATCH, DELETE) return
403 Forbidden - Logins still work:
/api/auth/login,/api/auth/refresh,/api/auth/exchange,/api/admins/auth-with-password,/api/collections/<name>/auth-with-passwordand the hosted login form mint tokens from replicated credentials without writing anything - Registration, OAuth account linking, admin init and API key creation create rows, so they are refused like any other write; send them to the master
- All read operations (GET) work normally
- Below the HTTP layer, every write a replica would make to a replicated
database is refused (
replica is read-only: writes go to the master): replicated data changes only through the sync, so a replica can never hold rows its master and the changeset log never saw
Example
# Read operation - allowed on replica
curl http://replica:3002/api/collections/posts/records
# Write operation - blocked on replica
curl -X POST http://replica:3002/api/collections/posts/records \
-H "Content-Type: application/json" \
-d '{"title": "New Post"}'
# Returns: {"error": "replica is read-only", "code": 403}
Headers
Every response is stamped with the node's replication role, so proxies and monitors can route without guessing:
| Header | Description |
|---|---|
X-Replica-Mode |
Current mode: replica or master |
X-Replica-Id |
Unique identifier for this replica |
X-Master-URL |
URL of the master server (replica only) |
A replica's ID comes from REPLICA_ID and its public address from
PUBLIC_URL. Client SDKs capture the X-Replica-Id of the node that served
a response, which is useful for debugging which node answered.
Routing and Load Balancing
Use tango. It is the supported front door for a cluster: it splits reads and writes, keeps stale replicas out of the read pool, pins realtime and automation traffic to the master, journals writes for exactly-once replay, and fails over automatically. Clients need exactly one URL: tango's.
If you cannot put tango in front, the routing rules it implements are the rules to replicate yourself:
- Writes (non-GET) go to the master and to nobody else
- Realtime websockets and the automation plane (
/api/workflows,/api/pipelines,/api/cron,/api/webhooks,/api/plugins,/api/secrets) go to the master - Reads may go to any node that is healthy and in sync; check
X-Replica-Modeand each node's/api/replica/local-status - Failover is on you: promote a replica (see below) and repoint the write path at it
Failover
Failover is automatic when tango fronts the cluster:
- tango detects the master is down:
TANGO_FAILURE_THRESHOLDconsecutive failed probes (one everyTANGO_HEALTH_INTERVAL), then three monitor rounds without a master; about ten seconds with the defaults - It fences the old master so it cannot accept writes again. On Kubernetes the role map records the pod as a replica and that pod is deleted; tango waits up to 15 seconds for the StatefulSet to replace it, then proceeds even if the pod's node is unreachable. With docker it is a SIGKILL through the Docker socket
- It picks the healthy replica with the most changeset entries applied,
never the node it just fenced, and calls its
POST /api/admin/promote: vski flips the node's role in-process (write path, dispatcher, cron, workflow gateway, changeset journal) without a restart, in milliseconds - Journal entries the new master may lack replay into it in order, with the
callers' original credentials, so acknowledged-but-unreplicated writes are
not lost, and replays are idempotent (
X-Tango-Idempotency-Key) - Remaining replicas catch up from the new master's changeset log; one whose position no longer belongs to the new history re-downloads instead of keeping writes the new master never saw
tango runs strict (writes refused with 503 + Retry-After during the
window) or absorb (writes journaled and answered 202); set
TANGO_BUFFER_MODE. Manual failover is POST /tango/promote on the
management port. On Kubernetes two tango replicas elect a leader through a
lease so only one acts: send the promotion to the lease holder (leader_id
in /tango/status); the other replica answers 409 with the leader's name.
If two nodes report master at once (an old master still answering while
its pod terminates), tango holds writes until exactly one does and starts
no automated failover meanwhile; /tango/status shows split_brain: true.
Without tango, you can promote a replica by hand: stop the old master, call
POST /api/admin/promote on the chosen replica (or restart it with
REPLICA_MODE=master), and point the remaining replicas' MASTER_URL at it.
Client SDK Usage
Through tango, one client does everything (reads and writes, realtime, workflows):
from vski import VskiClient
# tango is the only address your app needs
client = VskiClient("http://tango:8000")
await client.admins.auth_with_password("admin@example.com", "password")
# writes are journaled and go to the master
await client.collection("posts").create({"title": "New Post"})
# reads spread across the master and healthy replicas
posts = await client.collection("posts").get_list(1, 10)
Against individual nodes, keep two clients: a read client on a replica and a write client on the master:
read_client = VskiClient("http://replica:3002") # GETs
write_client = VskiClient("http://master:3001") # everything else
Monitoring
Health Check
Check any node:
curl http://replica:3002/health
Response includes headers indicating replica status.
Sync Status
Check replication status on master:
curl -H "Authorization: Bearer <replica-jwt>" http://master:3001/api/replica/status
Response:
{
"dbs": [
{
"name": "default",
"size": 1048576,
"sha256": "abc123...",
"schemaVersion": 5
}
]
}
Cluster Status (tango)
curl http://tango:8001/tango/status
Reports every node's role, health and version, per-database positions, the
journal state, lease leadership (leader, leader_id) and split_brain.
Security
Authentication
Replicas authenticate using JWT tokens generated from the shared JWT_SECRET.
No additional keys or registration required.
Best Practices
- Use HTTPS in production
- Keep JWT_SECRET secret and consistent across all servers
- Use strong secrets - at least 32 characters of random data
- Rotate secrets by updating all servers simultaneously
- Keep tango's management port (
:8001) on an internal network: it carries the promotion controls and replica downloads (the Helm chart'snetworkPolicy.enabledlimits it to the vski pods)
Limitations
Current limitations (may be addressed in future releases):
- Single writer: all writes go through the master. tango adds failover and read scaling, not multi-master writes.
- Full database changesets: changesets apply to entire databases, not per-collection.
- Master-only planes: realtime, workflows, cron and webhooks execute on the master only: transparent through tango, refused on replicas directly.
Troubleshooting
Replica Not Syncing
- Check network connectivity to master
- Verify
MASTER_URLis correct - Verify
JWT_SECRETis identical on master and replica - Check master logs for authentication errors
- A
426 Upgrade Requiredanswer means the replica runs an older version than the master; upgrade replicas before the master
Replica Dropped from the Read Pool (tango)
The replica is behind TANGO_MAX_READ_LAG: check its sync interval, its
logs, and that it can reach the master (directly or through tango's
management port). It rejoins on its own once it catches up.
Authentication Fails on Replica
Ensure JWT_SECRET is identical on both master and replica:
# On master
echo $JWT_SECRET
# On replica (must match!)
echo $JWT_SECRET
Data Not Appearing on Replica
- Check sync interval:
SYNC_INTERVAL - Wait for next sync cycle
- Check replica logs for sync errors
Schema Not Syncing
If new collections appear on master but not on replica:
- Check replica logs for "schema mismatch" messages
- Verify replica can reach
/api/replica/schema-syncendpoint - Check for SQL errors in replica logs during migration
Best Practices
- Front the cluster with tango: routing, freshness and failover are solved problems there
- Monitor sync lag:
GET /tango/statusor/api/replica/status - Keep a replica for backups: snapshot a stopped replica so production never pauses; see Backup
- Use S3 file storage in clustered deployments: every node reads the same bucket and the local file-sync journal turns off; see File Sync
- Upgrade replicas before the master
- Secure communication: HTTPS between components, management ports internal
- Introduction
- Performance
- Getting Started
- Coding Agents
- Command Line
- Infra as Code
- CI/CD Guide
- Project Boilerplate
- SDKs
- Authentication
- Databases
- Collections
- Gated Queries
- Graphs
- Rules and Access Control
- Granular Access Control
- Triggers
- Plugins
- Containers Plugin
- vski mcp
- Realtime
- Full Text Search
- Vector Embeddings
- KWH
- Workflows
- Pipelines
- Dashboard Widgets
- Connectors
- Cron
- Webhooks
- Gates
- Migrations
- Replication
- File Sync
- Backup
- Distributed Realtime
- Tutorial: ReactJS Todo
- Tutorial: Graph-Memory Agent
- Tutorial: Image Gallery
- Tutorial: Embeddings
- Tutorial: Graphs
- Recipe: Calendar Timetree
- Tutorial: Plugins
- Tutorial: MCP Server