-
v0.18.0 Stable
released this
2026-11-15 17:34:34 +00:00 | 3 commits to main since this releaseVSKI v0.18.0: Connectors
Connectors are now a first-class platform feature. A definition is a
declarative provider manifest, a connector is one configured instance of
it, and an account is an authorized credential. One generic engine handles
OAuth grants, API keys, basic passwords, custom headers, and public APIs.
Provider tokens stay on the server: pipelines, plugins, and triggers call
through an authenticated proxy instead of holding credentials.Definitions are data. Nothing is built into the binary, so the provider
catalog is exactly what your IaC declares and the dashboard shows.SDKs:
@vski/sdk2.11.0 andvski-sdk0.20.0.Upgrade notes:
- Four system collections (
_connector_definitions,_connectors,
_connector_accounts,_connector_oauth_states) are created
automatically on start. Existing collections are untouched. - Existing deployments start with an empty definition catalog. Apply
connectorDefinitionsbeforeconnectors;vski applyalready orders
definitions first. - OAuth providers need
<server>/api/connectors/oauth/callbackas their
registered callback. SetPUBLIC_URLwhen the server is behind a proxy
that does not forward the public host. - Accounts are connected interactively and are not IaC data. Their
credentials can reference entries in the encrypted secrets store. - Connector management endpoints require an admin. The request proxy
requires theworkflowsrole; admins also pass.
Definitions and IaC
connectorDefinitionsandconnectorsare global IaC kinds.vski plan,
vski apply, andvski pullround-trip them, with definitions applied
before connectors.vski pullwritesconnector-definitions.yamland
connectors.yaml.A definition declares:
api: base URL, exact allowed hosts, default headers, header fields,
auth scheme, and read-only mode.oauth: static client, authorization and token URLs, PKCE, token auth
style, form or JSON token requests, scopes, extra auth and refresh params,
token refresh, and the connect-time form.key: credential label, connect-time form, and auth scheme.account: profile URL and JSON paths for the remote account id, label,
and avatar. It labels OAuth accounts after the grant completes.name,description,icon,docs, and free-formmetadata.
api.baseUrlandapi.allowedHostsaccept{field}placeholders filled from
connect-time values, so one definition can target per-account sites without
giving up the host allow-list.api.readOnlyrejects every method except
GETandHEAD.Auth schemes cover
bearer,header,query,basic, andnone. A
custom-header connector declares itsapi.headerFields; a basic connector
names the form field used as the username.Definition ids match
[a-z0-9][a-z0-9-]{0,63}. OAuth registration is static:
the client id and secret belong to the connector config. Invalid definitions
fail before anything is written.Accounts and secrets
- OAuth: start a connection with
POST /api/connectors/{id}/connect.
The returned authorize URL carries a single-use state that expires after
10 minutes. The callback exchanges the code, stores the encrypted grant,
fetches the account profile, and returns to the dashboard. Access tokens
refresh automatically when an expiry is known, a refresh token is present,
and the definition enablesrefresh. - API key and basic password: connect with a token, a token secret, or
form values. Basic auth combines the username form field with the token. - Custom headers: each declared header is collected per account and can
come from a secret. - None: public APIs need no account.
tokenSecret,valueSecrets, andheaderSecretsname entries in the
encrypted VSKI secrets store and resolve at request time. Rotating a stored
secret updates every account that references it without rewriting the account
or the connector config. Credential values never appear in account responses,
connector configs, pipeline output, or run events.A connector can hold several accounts. Requests select one by id or name, or
fall back to the first active account.Pipelines, plugins, and triggers
Pipelines gain a
connectorstep. It resolvesaccount,path,url,
headers, andbodyas templates, attaches the account credential
server-side, and returns{status, headers, body}. Non-2xx provider responses
fail the step so the usual retries andonErrorbehavior apply. The timeout
defaults to 30 seconds and is capped at 120.{ "id": "projects", "type": "connector", "connector": "internal", "account": "{{input.account}}", "method": "GET", "path": "/v1/projects?limit=50" }Because
accountis templated, one pipeline definition can run against any
connected account supplied as run input, including accounts for different
sites of the same provider.Plugins and external systems use
POST /api/connectors/{id}/requestwith an
API key that carries theworkflowsrole. The proxy applies the stored
credential, enforces the host allow-list, and returns the provider status,
headers, and body. The proxy call itself answers200; a provider rejection
is reported instatus, and transport or configuration failures return
4xxor5xxwith the reason.A connector is bound to its declared hosts. Only
api.allowedHostsentries
and the resolved base URL host are reachable, matched exactly with no
wildcards. A single account at a site root can serve several APIs on that
host, such as Confluence under/wiki/api/v2/...and Jira under
/rest/api/3/....Connector dashboard and SDKs
- The Connectors page lists configured connectors with auth mode, account
count, status, base URL, and connect, accounts, test, and delete actions. - The add and edit wizard walks provider, auth, endpoints, credential rules,
connect-time form, and review as separate steps. Every provider comes from
the applied definition catalog or the custom definition; no provider list
is hardcoded in the UI. - The account dialog renders the definition form and supports stored-secret
references. The test dialog sends an authenticated provider path and shows
the status and response. - Dialog actions sit in a sticky footer outside the scrolling body.
SDK support:
- TypeScript
@vski/sdk2.11.0 addsclient.connectorswith definitions,
configs, accounts, OAuth start, and the request proxy. - Python
vski-sdk0.20.0 addsclient.connectorsand IaC loading, diffing,
applying, and pulling for both connector kinds. The synchronous client
exposes the same namespace.
Agent-facing dashboard
The dashboard can be handed a connection and opened on a single surface,
which is what an agent does when a human has to act: approve a run, filter
records, run a pipeline.GET /connect?endpoint=...&token=...&path=...saves the deployment URL and
its bearer token and forwards topath. Without a token it goes to login
with the target kept as?next=, so the user lands where they were headed.
The endpoint is normalized to an http(s) origin,nextstays
dashboard-relative, and a stale saved API key is cleared: the dashboard's
admin endpoints reject API keys andX-API-Keyis checked before
Authorization, so a leftover key would shadow the token./w/...routes render the same pages inside a chrome-less shell: no
sidebar, connection badge, breadcrumbs, feature chips, or desktop titlebar,
with dense padding. The page keeps its own actions — record create, edit,
delete, run, save, approve, reject — because widget mode strips navigation,
never capability.- Every surface has a widget route:
/w/overview,/w/collections/:name
(and/stats,/:id,/create),/w/workflows,/w/workflows/:runId,
/w/signals,/w/pipelines(and/:name),/w/connectors,/w/cron,
/w/gates,/w/webhooks,/w/api-keys,/w/secrets,/w/plugins,
/w/migrations,/w/graphs(and/:name,/create),/w/logs,/w/sql,
/w/settings(and/auth). - Widget query params are
filter(list and collection tables),sort
(-fieldfor descending),page,perPage, anddb(/w/collections,
/w/graphs,/w/sql;databaseis accepted as an alias on the SQL
console), plusstatus/collectionfor webhook logs.action=create
opens the create dialog on a list page (or the record form on a collection),
andaction=runopens the run dialog in the pipeline editor./w/sqlalso
takessqlto pre-fill the query andexecute=trueto run it on load and
show the results, so an agent can hand over a query and its outcome in one
link. Links stay inside/w, and the surface name is the document title,
so the browser tab identifies the widget. - The widget gains no leftover dashboard space: the SQL results modal fills
the viewport instead of leaving the sidebar's column, and the pipeline
canvas drops React Flow's minimap (it duplicated the canvas and took room
from the step palette). - The Harness Account panel gains Open dashboard:
Vski.dashboard
(GET /api/vski/connections/:id/dashboard) builds a one-shot/connect
URL signed with the selected slot's bearer token and opens it externally.
An API-key slot yields a URL without a token, because the dashboard cannot
be signed in with one.
Signal approvals
A run parked on a signal is a human decision waiting to happen, so the
dashboard treats it as a first-class surface./signals(and the/w/signalswidget) lists every waiting signal across
the deployment: pipelinesignalsteps and code workflows parked in
waitForSignal. Plugin-managed pipeline steps also park on a signal, but
that is the plugin's wake-up, so the inbox hides it.- Each entry renders approve or reject with an optional reason and records
the signed-in admin asby. Delivery uses a single-use workflow hook:
create it with a random token,POST /api/workflows/hooks/:token(emits
signal_receivedand requeues the run), then delete the hook. The payload
is{ ok, by, reason?, signalId? }. The run page renders the same form
above its timeline. - A wait can carry a description:
descriptionon a pipelinesignal
step, or the option onwaitForSignal/wait_for_signal. It is rendered
with the run context when the step parks and carried in thesignal_waiting
payload, so the inbox and the run say what is being approved. Rendering is
best-effort: an unresolved template path is kept as written rather than
failing the run. The pipeline builder edits it in the signal step inspector. - Harness automation notifications carry the prompt as their body ahead of
the run reference, andGET /repos/:name/signalsreturns it as
description.
Tests and docs
- A new end-to-end suite covers definition storage, encrypted accounts, the
authenticated proxy, a pipeline step with a templated account, stored
secret references, and rejection of hosts outside the allow-list. - The connector engine and pipeline step have unit coverage for auth
scheme application, OAuth refresh, stored secret resolution, templating,
non-2xx failures, and host rejection. - Managed providers were validated end to end: Google over OAuth and
Atlassian over one account token serving both Confluence and Jira. - Executor tests cover the signal description payload and its
unresolved-template fallback; harness tests cover the description reaching
Signal.Pendingand the notification body. - The approval flow was validated against a running deployment: a parked
pipeline run and awaitForSignalrun both appear in the inbox, a response
writessignal_receivedwith{ok, by, reason, signalId}, the run resumes
and completes, and the single-use hook is gone afterwards. Every widget
route was opened against a running deployment — tables, editors, and config
pages — checking for the compressed header, no sidebar or connection badge,
the surface title in the tab, the filter andactionparams, and links that
stay under/w; the normal routes were checked unchanged. - The wiki gains a Connectors page linked from the sidebar, and the IaC and
Pipelines pages document the new kinds and step. The Workflows and Pipelines
pages document signal descriptions.
Downloads
-
Source code (ZIP)
0 downloads
-
Source code (TAR.GZ)
0 downloads
- Four system collections (
-
v0.17.7 Stable
released this
2026-11-08 19:53:10 +00:00 | 9 commits to main since this releaseVSKI v0.17.7: Durable replication, tango cluster proxy
Replication is rebuilt around a durable, verifiable changeset log:
every committed write is appended to an on-disk log before its
transaction commits, replicas resume from an exact position, and a
promoted replica serves its former siblings from the same log. Nodes
switch between master and replica in-process, without a restart. On
top of that ships tango, a new cluster proxy: it splits reads across
replicas, journals every write to Redis, and fails over to the
most-current replica. Acknowledged writes survive the failover, and none
is applied twice.failover under load (200 writes/s + 400 reads/s, master stopped) acknowledged writes missing on the new master 0 duplicated writes 0 write outage (503 + Retry-After) 3.4 s old master rejoined as replica 2.0 s Upgrade notes:
- Upgrade replicas before the master. A master on the new protocol
answers older replicas with426 Upgrade Required; they stay readable
but stale until upgraded. Each replica re-downloads once after the
upgrade. workflow_eventsgains aseqcolumn, added automatically on start.- Behavior changes, all listed below: replicas refuse more writes, the
SQL console refuses transaction control, and transient write failures
now return 503 instead of 403. - No SDK changes.
Durable changeset log
- Writes through the write queue are captured on the database's pinned
connection and appended toDATA_DIR/replication-log/<db>.clog
(CRC-checked, fsynced) before their batch commits. A failed commit
removes its entry; crash recovery drops uncommitted tails. - Entries form a hash chain. A replica resumes with its last
position plus the chain hash at that position. A position that is
pruned, ahead of the master, or on a different history (a failover
onto a node that missed writes) gets a full download instead of a
mixed history. - Replicas mirror the master's log entry for entry, so after a promotion
the remaining replicas catch up incrementally from the new master.
A master restart costs replicas nothing. - Entries are kept for
REPLICATION_LOG_RETENTION_HOURS(24); a replica
offline longer re-downloads once. A database no replica has asked for
is not logged, so standalone nodes pay nothing. - Full downloads use SQLite's backup API: snapshots include writes still
in the WAL, and replicas restore in place, so open handles stay
valid. Previously a replica's first full download could leave services
on a closed database. - Writes from the SQL console and from migration scripts now go through
the write queue, so they replicate. Inside them, transaction control
(BEGIN,COMMIT,ROLLBACK, savepoints),ATTACH/DETACHand
pragma assignments are refused: each request already runs in its own
transaction. - A changeset carrying columns or tables a replica lacks is no longer
silently skipped: the replica syncs the schema first. - Workflow events carry a per-run
seqassigned when they commit, so a
promoted replica hands workers the same event order as the old
master.
Roles, failover and replica safety
- Hot promote/demote:
POST /api/admin/promoteand
POST /api/admin/demote(superuser token) switch a node's role
in-process: write path, syncer, changeset journal, pipeline dispatcher,
cron, workflow gateway, webhooks, graph index. - Workflow execution is master-only: workflow gateway, scheduler,
cron dispatch, webhooks and realtime run on the master alone.
Replicas answer workflow writes and queue polls with503. workflows.dbreplicates by default as a passive copy, so a
promoted replica starts with every run, queued job, timer and
definition. Jobs the dead master had handed out are reclaimed within
30–60 s and redelivered.REPLICA_INCLUDE_WORKFLOWS=falsekeeps it
node-local.- Replicas no longer write locally. Every write to a replicated
database is refused below the HTTP layer, so a replica never holds rows
its master never saw. Logins still work on replicas. Registration,
OAuth account linking, admin init and API key creation now return
403there and must go to the master; before, they silently created
diverging rows. - Exactly-once replay: a write carrying
X-Tango-Idempotency-Key
records the key in the same transaction; a repeat answers
200 {"duplicate":true}without writing. Keys replicate with the
data and are pruned afterIDEMPOTENCY_KEY_RETENTION(2h), which must
exceed tango's journal retention. - Older replicas (pre-log protocol) get
426 Upgrade Required, logged
once per replica every few minutes. - Security:
/api/admin/replicas*now requires an admin.
Previously any authenticated user could mint replica tokens. - Replicas report their positions to the master after every sync
(POST /api/replica/sync-complete); every node exposes its own at
GET /api/replica/local-status.
Write path fixes
- Transient failures are retryable: a write hitting a node that is
shutting down, a full write queue, a replica, or a busy database now
returns503withRetry-Afterinstead of403, which clients
treated as final. - Write-queue batches take the write lock up front (
BEGIN IMMEDIATE).
A batch that read before writing could fail with "database is locked"
when another connection committed in between; SQLite reports that
without waiting. - Deleting a database released neither its replication session nor its
connection handle, which could hang shutdown or crash the server.
Sessions are now released before their connection closes. - The Docker images are built with SQLite sessions enabled. Without
them a replica's first snapshot failed, so containers could not
replicate.
tango: the cluster proxy
A new component (
tango/): one endpoint in front of a vski master and
its replicas.- Routing: writes go to the master. Reads go to the least-busy
healthy node, except the workflow and automation plane (workflows,
pipelines, cron, webhooks, plugins, secrets) and the realtime and
workflow websockets, which stay on the master. Sessions (/sso,
logins,/api/auth,/api/admins,/api/keys) and admin scripts
(SQL console, migrations) go to the master and are never journaled. - Write journal: every write is appended to a Redis Stream before
it is forwarded, with an idempotency key the master stores with the
write. On failover tango replays, in order, every entry the new master
may lack, including writes the old master acknowledged but never
replicated. Writes whose client saw a failure are dropped, never
replayed. - Failover: tango fences the old master, hot-promotes the replica
with the most entries applied, then replays. Instrictmode
(default) writes get503+Retry-Afterduring the window; in
absorbmode they are journaled and answered202. - Read freshness: a replica that falls behind the master by more
thanTANGO_MAX_READ_LAG(30s) leaves the read pool until it catches
up. Optional read-your-writes (TANGO_READ_YOUR_WRITES) pins a
client's reads to the master briefly after its write. - Actuators carry out role changes:
docker: Engine API, role files read by containers at start;
replicas sync through tango, so promotions need no reconfigurationkubernetes: role ConfigMap, the label the master Service selects
on, and lease election between tango replicaslocal: child processes, for developmentmanual
- Management API (
:8001): status, manual promotion, replay.
Measured (
tango/bench, three nodes on one host, k6 open model),
p95 added by tango:500/s 1,000/s 2,000/s 4,000/s reads (point, list) +0.2 ms +0.2–0.3 ms +0.1–0.2 ms +0.1 ms record creates +0.1 ms +0.1 ms +5.7 ms +8.0 ms No errors at any rate. On writes, most of the cost is the idempotency
key vski stores with each write; the benchmark breaks the cost down.Tests and docs
- New end-to-end suites cover multi-replica convergence, master restarts,
hot promotion and switchover, incremental catch-up after promotion,
workflow state across failover, and divergence handling. vski's full
end-to-end suite (47 tests) passes through tango, and tango's failover
test proves acknowledged-but-unreplicated writes survive exactly once. - Replication.md documents the changeset log, upgrade order, replica
read-only rules and the new settings; tango ships its own README and
benchmark guide.
Downloads
-
Source code (ZIP)
0 downloads
-
Source code (TAR.GZ)
0 downloads
- Upgrade replicas before the master. A master on the new protocol
-
v0.17.6 Stable
released this
2026-11-08 00:05:13 +00:00 | 11 commits to main since this releaseVSKI v0.17.6 — Plugin gates, containers hardening
Gates — the platform's named HTTP endpoints — can now target plugin
functions natively (url: plugin://<instance>/<function>): the
caller's JSON body is dispatched as the function's params over the warm
plugin pipe and the result comes back verbatim. This replaces the
credential-injecting HTTP bridge as the way to expose one plugin
function to selected callers, and closes that pattern's hole where any
caller passing the gate rule could invoke any function on the
instance. The containers plugin gets the hardening pass from its
official-plugin review.Safe upgrade: no schema migrations and no configuration changes; gates
withhttp(s)://targets behave exactly as before. No SDK changes —
the call side (client.gate(name, body)) was already generic.Gates:
plugin://targetsA gate whose
urlisplugin://<instance>/<function>dispatches
directly into the running plugin instance:- The request body becomes the function's params — it must be a
JSON object (an empty body is an empty object); anything else is a
400. - The function's return value is the response body, verbatim (
200). - The function identity is fixed by the gate record — callers pick
the payload, never the function. Anyone a gate rule lets through can
do exactly one thing; no plugin admin key is involved at any point. - Dispatch rides the warm stdio pipe (no loopback HTTP hop) under the
standard invoke budget (30 s default, 120 s hard cap). The instance
must be running —502with the reason otherwise, as with invoke
failures and disabled plugins. - The instance is looked up in the caller's database (
x-dbname
header, defaultdefault), and the gate's access rule is checked
exactly as for HTTP targets before anything is dispatched.
The web console's gate form accepts
plugin://URLs (validated, with
an inline hint), and the platform wiki gained a dedicated Gates
page — model, CRUD via the_gatesrecords API, proxy semantics and
their caveats (no timeout, no streaming, buffered bodies), plugin
targets, and security notes. The MCP guide now leads withplugin://
targets and keeps the old admin-key bridge documented as a compatibility
pattern, with its caller-picks-the-function caveat spelled out.Containers plugin: official-plugin hardening
- Adoption no longer resets the timeout. Docker reports container
times as RFC3339Nano (up to nine fractional digits), which the
timestamp parser rejected on older supported Pythons — an adopted
running container silently got a fresh full timeout budget after every
plugin restart. All timestamps now parse through one normalizing
helper (naive values read as UTC). - Undeliverable completion signals give up. A completed job whose
workflow run was deleted server-side used to retry its signal forever
and pin its record against retention. Signals now retry for
signal_retry_hours(new instance config, default 24;0= never
retry), after which the record is markedsignalGaveUpand becomes
reclaimable. An attempt-count cap guards against tight retry loops. - The prune sweep never touches a live job's container. Containers
whose job record is still queued or running are skipped regardless of
age — previously a legitimately long job (amax_timeoutabove the
sweep age) could be force-removed mid-run. network: hostandnetwork: container:<id>are rejected — they
defeat the isolation the plugin promises; user-defined Docker
networks only.- Duplicate dispatches can no longer create two jobs — the
find-or-create path ofcontainer.startis serialized. - Per-job runtime state (locks, cancel events) is released once a
job completes and its signal settles, instead of accumulating for the
process lifetime. - The distribution image pins
vski-sdk0.15.0 (Makefile venv and
requirements floor alike), so the plugin's stats collector actually
registers in the shipped image — with the previous 0.14.0 pin the
feature silently no-oped.
All of the above is covered by new unit tests (timestamp table,
adoption-timeout with nanosecond stamps, give-up both ways, prune
sparing live jobs, state release, concurrent start) and the full unit +
end-to-end suites pass.Downloads
-
Source code (ZIP)
6 downloads
-
Source code (TAR.GZ)
1 download
- The request body becomes the function's params — it must be a
-
v0.17.5 Stable
released this
2026-11-07 22:20:47 +00:00 | 12 commits to main since this releaseVSKI v0.17.5 — Plugin steps, plugin stats, WAL durability
The plugin platform grows up in two directions: plugins can now
define their own pipeline step types (acontainerstep instead of
the start → signal → check workaround) and report live stats — both
process metrics the server collects itself and custom metrics the
plugin publishes. Under the hood, the storage layer fixes a
connection-pool race that could make pooled connections read through
deleted WAL files (empty/stale reads, thendatabase disk image is malformedon writes) — that class of failure is now structurally
impossible from inside the server and blocked from outside it.Safe upgrade: no schema migrations; stats tables are created in
stats.dbon start and plugin catalogs are re-scanned automatically.
Everything is on by default. SDKs:@vski/sdk2.10.0,vski-sdk
0.15.0.Plugin-defined pipeline step types
A plugin manifest can now declare step types alongside its functions:
steps: container: description: Build/run an ephemeral Docker container and report its exit code and output. params: containers/step.schema.json # file path or inline JSON Schema functions: start: container.start # required; must ack fast check: container.check # optional, informational- Registry and validation. Scanning inlines the params schema into
the catalog; step types resolve per database from plugins with an
enabled instance. Type names must match^[a-zA-Z0-9_-]{1,64}$and
collisions across plugins are rejected at validation. Param templates
({{...}}) are checked as usual; fully-resolved params are validated
against the plugin's JSON Schema (draft 2020-12). - Asynchronous execution. The server dispatches the start function
with the step params plus an injectedcontext
(runId,stepId,pipeline,db,signal,timeout,attempt)
under a 30-second dispatch budget — the function acks and does its
work in the background. The step parks on a deterministic per-step
signal; the plugin completes it with
client.workflow.send_signal(runId, signal, {ok, error, output})—
ok: falsefails the step witherror, anything else completes it
withoutput. Completion is durable: signals that arrive before the
park are buffered and replayed,runId/stepIdgive the plugin an
idempotency key for re-dispatches, and the steptimeout(1–86400 s)
is passed through as the overall budget. Retries cover dispatch
failures; a plugin-reported failure is terminal. - Builder. Plugin step types appear in the palette next to the
builtins with their own icon and accent color, and the inspector
renders a form from the params schema (typed inputs, enums as
selects, JSON fallback for complex shapes). - The containers plugin ships a native
containerstep — build/run an
ephemeral container with resource caps, log capture and timeout
enforcement — replacing the three-step workaround in a single node.
Plugin stats: process and custom
Every plugin instance gets a live stats view, from two sources merged
by the 30-second stats collector:- Process stats (server-collected). Read from
/procper plugin
child process: RSS, CPU % (utime+stime delta), open file
descriptors, uptime — stored in_stats_plugin_processes. - Custom metrics (plugin-reported). A new
statsJSON-RPC method,
polled only from runtimes that advertisecapabilities: ["stats"]in
initialize(old runners are simply not polled). In py-sdk,
collect metrics with@app.stats— multiple collectors share a 4 s
budget and merge into a flat numeric map, stored one row per metric
in_stats_plugin_metrics. Both tables are queryable system
collections; retention is 7 days. - API.
GET /api/plugin-instances/:name/stats?window=3600returns
bucketed process and per-metric time series plus the latest report's
meta;GET /api/plugins/statsreturns the latest snapshot of every
instance with per-plugin rollups (instances, RSS/CPU/FD totals). - Console. The instance modal shows stat tiles (RSS, CPU %, FDs,
uptime) with history charts and a custom-metric picker with a
multi-line chart; the plugins page shows a totals strip per plugin.
Storage: the server can no longer lose its WAL files
Two failure paths could delete a live database's
-wal/-shmwhile
pooled connections still held them open: a shared-memory
reference-counting race in the vendored SQLite fork under connection
churn, and any external SQLite client closing the same file
cross-process. Connections left on deleted inodes read silently
stale/empty data (an API list suddenly returns[]) and fail writes
withdatabase disk image is malformeduntil restart — the on-disk
database was never actually damaged, but the server was effectively
poisoned.All three layers are fixed:
- The vendored fork never unlinks
-wal/-shm— files left by a
crash or external tool are recovered on the next open. (The fork's
deliberate close-on-reset behavior that cured the benchmarked fd
leak is untouched; only the unlink calls are gone.) - Each database holds a permanent guard connection, so no
in-process close path can ever believe it is the last connection. - Out-of-SQLite sentinel file locks (shared lock over the database
file's lock region and the shm dead-man byte), refreshed every
second, make any external client's last-connection cleanup see a
live holder and stop deleting.
Regression tests cover the churn path and guard boundedness
(internal/db/fdleak_test.go); the full suite is green.Console polish
- Plugin instance configuration is now schema-driven forms — typed
inputs, selects for enum fields, secrets with show/hide, JSON
fallback — for both create and edit, instead of raw JSON textareas. - The plugin catalog moved into a modal behind the Scan button;
the plugins page leads with stat cards and instances. - The sidebar can be hidden (choice remembered per browser), with
a floating reveal button in the top-left corner. - Pipeline builder: combined Save + actions dropdown (Settings,
Validate) and a truncated pipeline title with the full name on
hover. - Consistent page headers and actions on Pipelines and Workflows
(Refresh, then New), and the Authentication page uses the full
content width.
Documentation
- Backup.md gains "Data directory access rules": the server is the
only process that should touchDATA_DIR;-wal/-shmare normal,
never hand-pruned; replicas sync over HTTP and never read the
master's data directory; file-level backups require a stopped
server or a replica. - Replication.md states the HTTP-sync model explicitly; Plugins.md
and Pipelines.md document step-type declarations and the async
step contract.
Downloads
-
Source code (ZIP)
2 downloads
-
Source code (TAR.GZ)
2 downloads
- Registry and validation. Scanning inlines the params schema into
-
v0.17.0 Stable
released this
2026-11-06 17:29:28 +00:00 | 13 commits to main since this releaseVSKI v0.17.0 — Performance
v0.17.0 is a planned, performance-focused release. The write path is
rebuilt around a group-commit write queue, the storage layer's WAL
lifecycle and defaults are tuned for sustained throughput, and the
per-request costs around authentication, settings and realtime are
reduced. The result: 10,000+ requests per second sustained across
every benchmark scenario — reads, creates, updates and mixed traffic —
with zero errors, and flat tail latency under load.scenario (10k rows, 60 s steps) sustained RPS p95 at 10k errors read-point 10,000+ 0.40 ms 0 write-create (~150 B) 10,000+ 37 ms* 0 write-update 10,000+ 8.6 ms 0 mixed-80-20 10,000+ 0.67 ms 0 * near-saturation batching (see below). Full methodology and the
Postgres/MySQL reference comparison:benchmarks/BENCHMARK.md— at
10k mixed, vski's 0.67 ms p95 also leads the Postgres (4.05 ms) and
MySQL (23.0 ms) reference services on this rig.Safe upgrade: no schema migrations; API keys are re-hashed lazily on
first use; every new behavior is on by default and individually
disableable via environment variables.Write queue: group commit for record writes
SQLite allows one writer at a time, so the server now makes every write
cycle count: a per-database write queue funnels concurrent write
requests through a single writer goroutine that commits them together in
one transaction. N concurrent requests pay one commit cycle instead of
N — and on a busy server that is the difference between the writer
thrashing the write lock and staying quiet enough that readers never
notice it (mixed 80/20 traffic sustains 10,000+ RPS at 0.67 ms p95,
ahead of the Postgres reference service's 4.05 ms on the same rig).- Applies to record CRUD (single and bulk), graph node/edge writes, and
vector embedding upserts. - Correct by construction: a request is acknowledged only after its
batch commits, so a read following an acknowledged write always sees
it. No eventual consistency anywhere. - A failing request (e.g. a UNIQUE violation) is rolled back to its
savepoint and fails alone; concurrent requests in the same batch
still commit. - A lone write under no load skips batching entirely and commits
immediately — zero added latency at low traffic. - Under overload beyond capacity the queue applies backpressure:
requests batch-wait (that is the ~37 ms p95 on pure write-create at
10k) and fail loudly afterWRITE_QUEUE_SUBMIT_WAIT_MS(5 s) instead
of piling up unbounded. Errors stay at zero up to capacity.
Configuration (all optional):
WRITE_QUEUE_ENABLED(defaulttrue),
WRITE_QUEUE_DEPTH(4096),WRITE_QUEUE_MAX_BATCH(128),
WRITE_QUEUE_WINDOW_MS(2),WRITE_QUEUE_SUBMIT_WAIT_MS(5000).Storage layer: the server owns the WAL lifecycle
- Auto-checkpointing is disabled on every connection; a background loop
checkpoints databases whose WAL exceeds
WAL_CHECKPOINT_THRESHOLD_MB(16 MB) — a PASSIVE checkpoint first
(no write lock), then a TRUNCATE bounded by a 50 ms busy budget,
retried every second. Under normal and mixed traffic the WAL stays
in the low megabytes. Under extreme bulk-ingest bursts (10 KB records
at 10k RPS ≈ 100 MB/s) it can grow into the gigabytes until the burst
pauses and checkpoints reclaim the space automatically — size disk
headroom accordingly for sustained bulk ingest, or tune the
threshold. - Small WALs are periodically backfilled so slow
synchronous=NORMAL
trickles still get fsynced. - Pools are bounded (data 120 connections, aux 20) with idle pinned to
the open cap;SYNCHRONOUSdefaults toNORMAL— the recommended
WAL-mode setting, and with group commit also the fast one.
SQLite binding: pooled-connection hardening
Pooled SQLite connections closed mid-run (idle trims, bad-connection
discards) could leak file descriptors and crash under sustained load.
The vendoredgo-sqlite3fork fixes the shared WAL/shm handling, and
the pinned-pool configuration removes the churn path entirely.Faster authentication and settings
- API keys are stored as indexed SHA-256 hashes: key lookup is one
indexed row read. Legacy bcrypt-hashed keys keep working and are
transparently re-hashed the first time they are presented. _system_settingsreads are cached (short TTL): values such as
the auth collection name were previously re-read and JSON-parsed on
every authenticated request.
Realtime and stats
- The realtime gateway's per-topic event store enforces its size cap by
actual serialized size with deterministic trimming; consumer-group
offsets survive store compaction. - API hit statistics moved off the request path entirely: hits buffer in
memory and flush as multi-row inserts (500 rows / 5 s), so stats
collection never competes with user writes.
Collections
- Older collections get their automatic
createdindex backfilled on
start (consistent with newly created ones), keeping sorted and
paginated list queries fast on upgraded deployments.
Benchmarks
The repository ships a dockerized k6 harness (
benchmarks/) comparing
vski against minimal Go APIs over Postgres and MySQL across
read/write/mixed/auth scenarios with a rate ladder and early saturation
stop. The v0.17.0 numbers above are reproducible withmake vskiand
published viamake publish(benchmarks/BENCHMARK.md, raw runs under
benchmarks/results/).The report also includes measured read-side guidance for large
collections: a selective filter on a 100k-row table goes from 1,250 to
10,000 RPS with one index on the filtered column (collection index
management), and skipping the per-request listing count with
skipTotal=truedoubles listing throughput at 100k records (5,000 →
10,000 RPS).Downloads
-
Source code (ZIP)
3 downloads
-
Source code (TAR.GZ)
2 downloads
- Applies to record CRUD (single and bulk), graph node/edge writes, and
-
v0.16.2 Stable
released this
2026-11-05 12:51:02 +00:00 | 13 commits to main since this releaseVSKI v0.16.2
SSO with OAuth providers, storage statistics for the dashboard, and
self-contained API docs. Safe upgrade from v0.16.0 — the one schema
change (an optionalscopecolumn on_auth_providers) is applied
automatically on first start; no configuration changes are required.SSO: hosted login for apps and CLIs
Applications can now hand authentication to the server instead of
building their own login screens. The flow is OAuth-like: the app sends
the user toGET /sso/authorize?redirect_uri=...&state=..., the server
renders a hosted login page, and on success redirects back with a
one-time 60-second code that the app exchanges for a token via
POST /api/auth/exchange— the same{token, user}shape as
/api/auth/login, and the token never appears in a URL or browser
history.- Both superusers and auth-collection users can sign in; superusers are
checked first. A valid session skips the form on the next
authorization (HttpOnlyvski_ssocookie, 24h), andGET /sso/logout
clears it. - Redirect targets are locked down: loopback addresses
(127.0.0.1/localhost/::1, any port) and thevski://scheme are
always allowed; anything else must be listed exactly in the
_system_settingskeysso_allowed_redirects(JSON array). - The hosted pages are embedded in the binary and can be overridden
per-file fromTEMPLATES_DIR(default<DATA_DIR>/templates), with
the template data documented in Authentication.md. - Login form fields are readable in both themes (explicit input colors
with autofill overrides — previously nearly invisible in dark mode).
OAuth providers on the hosted page
The hosted login page shows a "continue with" row for every configured
provider. The full server-side client flow lives at
/sso/oauth/{provider}and/sso/oauth/{provider}/callback:
server-bound state (10-minute, single-use), code exchange, user info
resolution,external_authslinking with automatic registration on
first sign-in, then back to the application with a fresh code.- Providers are configured as
_auth_providersrecords — name, auth and
token URLs, userinfo URL, client id/secret,enabled, and the new
optionalscope(a sensible default is used per known provider). - Fixed two latent bugs in the existing
/api/auth/oauth2-loginflow:
the client secret is now actually sent in the token exchange, and
providers that report the subject assub(with no numericid) no
longer fail user resolution. - The callback URL is derived from the request (with
PUBLIC_URLas an
override) so it works behind proxies without extra configuration.
vski login --browser(py-sdk)The CLI can now authenticate through the hosted page instead of a
password prompt. The default method runs a local loopback listener
(http://127.0.0.1:<port>/callback) and waits;--method deeplink
uses thevski://scheme instead, with--register-deeplinkto
register the handler with the OS (Linux.desktop, Windows registry).
--no-browserprints the URL for headless use.Storage statistics
GET /api/stats/storage(admin only — superusers and admin-role API
keys) returns deployment-level numbers in bytes:deployment_size(the
whole data directory),db_size(the selected database file including
WAL sidecars, selected viax-dbname), anddisk_available/
disk_totalfor the disk holding the data directory.The dashboard's Databases page shows these as three cards under the
SQL editor whenever no query results are displayed, refetching when the
selected database changes. SDKs expose it asclient.stats.storage(db?)
(@vski/sdk2.8.0,vski-sdkwith the stats namespace).SQL console results modal
Query results now open in a modal overlaying the SQL view instead of
pushing the page down: full-area card with the same results header plus
a close button, dismissible via the X, Escape, or a click outside;
column headers stay pinned while scrolling. Dry-run status gets a
64px success/error icon with the message.Self-contained API docs
/openapi.htmlno longer depends on a public CDN: the ReDoc bundle
(v2.5.1) is embedded in the binary and served from/openapi.js, so
the docs render on offline and firewalled installations too.Downloads
-
Source code (ZIP)
4 downloads
-
Source code (TAR.GZ)
3 downloads
- Both superusers and auth-collection users can sign in; superusers are
-
v0.16.0 Stable
released this
2026-11-04 17:16:32 +00:00 | 14 commits to main since this releaseVSKI v0.16.0
Dashboard overhaul, collection stats, and sortable tables. Safe upgrade
from v0.15.6 — no migrations, no configuration changes.The dashboard, redesigned
The admin UI keeps its developer-tool character but stops shouting: the
brutalist look (global monospace, weight-900 caps everywhere, square
corners) is replaced by a calmer "refined technical" system.- Two rebuilt themes, light and dark, sharing one indigo accent and one
sky secondary, with status colors tuned per theme. The toggle in the
top bar works as before; your saved preference survives the upgrade. - Typography is split by role: the system sans stack for UI chrome, and
monospace reserved for data — record ids, SQL, JSON, timestamps. - Consistent radii, borders, and focus rings across buttons, inputs,
selects, modals, and cards; charts get themed tooltips, legends, and
bar colors instead of hardcoded blues. - The sidebar is rebuilt as a flat, icon-labeled nav: Databases,
Workflows, Pipelines and Cron on top, then Gates and Webhooks, then
API Keys, Secrets and Authentication, with Plugins and Migrations last
(used, but less often). Collection and graph trees live below it
unchanged.
Collection stats and feature indicators
Collections had embeddings, FTS indexes, triggers and views configured
on them, but the dashboard never showed any of it. Now it does.- Every collection view gets a Stats button opening a dashboard-like
overview: tiles for records, API hits, errors, embeddings, realtime
consumers and triggers; a 24-hour API activity chart with success and
error bars; a features panel with on/off state and details (embedding
model, dimensions and vector count, FTS fields, trigger definitions,
index SQL); the schema field list; and created/updated timestamps with
the API rule chips. - The collection header shows one small colored chip per enabled
feature, each with a tooltip: embeddings (live from the vector table —
pinned model, dimensions, vector count), full-text search, realtime
subscribers, triggers (webhook and pipeline counts combined), SQL
indexes, and views. - Empty charts everywhere — the home dashboard and the stats view — get
a proper placeholder: a faded ghost bar chart over a dashed baseline
with a hint, instead of bare "not enough data" text on white.
Sortable collection tables
- Clicking a column header cycles ascending → descending → off, with
the direction shown by an arrow in the header. Sorting is server-side
(the API's existingsortparameter), so it composes with pagination
and filters. - On first load the table sorts by
iddescending when the collection
has ids (same order as before), and by the first column otherwise. - The id column is only rendered when the table actually has one —
collections without ids no longer show a phantom empty column.
Fixes under the hood
- Per-collection API stats were broken since the beginning: the stats
middleware matched/collections/...while routes are mounted under
/api/collections/..., so every hit was tagged with the collection
name "api". Hits now record the real collection name, which is what
the collection stats view and the home dashboard's API performance
chart are built on. - Listing a collection whose table has no
idcolumn (tables adopted
from raw SQL or migrations) failed the whole query with "no such
column: id", because the implicit default order isid DESC. The
record listing now detects the table's actual columns and falls back
to ordering by the first column.
Repository restructure
- Release notes moved from
docs/release/to.release/. - Submodules renamed to match their packages:
client→ts-sdk,
vski-sdk→py-sdk; the emulator playground submodule was removed.
Downloads
-
Source code (ZIP)
2 downloads
-
Source code (TAR.GZ)
1 download
- Two rebuilt themes, light and dark, sharing one indigo accent and one
-
v0.15.6 Stable
released this
2026-11-03 21:33:17 +00:00 | 17 commits to main since this releaseVSKI v0.15.6
Plugin process keys gain the
workflowsrole. Safe upgrade from v0.15.5.Plugin keys can signal pipeline runs
- The per-process API key the plugin supervisor mints for every instance
now carriesworkflowsalongsideplugin. Plugin functions already
wrote collections with their role-scoped key; now they can also call the
workflow API, which is what an ack-fast plugin needs to report back:
ctx.client.workflow.send_signal(run_id, name, data)completes a
pipeline run's waitingsignalstep, so a pipeline that kicks a long
background job can still finish (or fail) when the job actually lands.
The docs MCP indexer uses exactly this loop: anindexstep acks in
seconds, asignalstep parks the run while indexing runs in the
plugin, and a final step re-raises the error payload so a failed pass
fails the run. - The role grants the workflow endpoints generally (runs, events, queue),
not only signal delivery. Plugins are admin-installed supervised
processes; the role keeps them off the admin surface. - Keys are minted fresh at every process start, so running instances pick
the role up on their first restart after the upgrade. No migration, no
configuration.
Downloads
-
Source code (ZIP)
2 downloads
-
Source code (TAR.GZ)
2 downloads
- The per-process API key the plugin supervisor mints for every instance
-
v0.15.5 Stable
released this
2026-11-03 14:13:59 +00:00 | 19 commits to main since this releaseVSKI v0.15.5
S3 file storage and file upload fixes. Safe upgrade from v0.15.4.
S3 file storage
- Setting
S3_BUCKETnow actually switches file storage to S3-compatible
object storage (MinIO, AWS S3, or any S3 API). Files are stored as objects
keyed{collection}/{recordId}/{filename}, the same layout local storage
uses, and the/files/...download route streams objects directly from
the bucket. - Configure with
S3_ENDPOINT(host[:port], optionalhttp(s)://scheme;
empty means AWS S3),S3_REGION,S3_BUCKET,S3_ACCESS_KEY,
S3_SECRET_KEY, andS3_FORCE_PATH_STYLE=truefor MinIO. With S3
enabled, the replica file journal stays off: the bucket is the shared
storage, so replicas with the same S3 settings read files from it
directly. - Note: the server now requires Go >= 1.26 to build from source, and the
Docker images use the golang:1.26 builder.
File upload fixes
- Creating a record with a multipart file upload no longer fails validation
on requiredfilefields: the uploaded filenames are merged into the
record before validation. Previously a required file field could never be
satisfied on create. - The file e2e suites (TypeScript and Python) now perform a real multipart
upload, download through/files/{collection}/{recordId}/{filename}, and
verify that deleting a record removes its files. They previously asserted
against a wrong URL prefix and effectively tested nothing.
Downloads
-
Source code (ZIP)
4 downloads
-
Source code (TAR.GZ)
3 downloads
- Setting
-
v0.15.4 Stable
released this
2026-11-03 02:31:04 +00:00 | 19 commits to main since this releaseVSKI v0.15.4
Pipelines and validation fixes. Safe upgrade from v0.15.3; no behavior
changes beyond the two fixes below.Pipelines and validation fixes
- Pipeline
httpsteps accept templated URLs (for example
url: "{{input.source.url}}"): validation now checks the prefix only on
literal URLs, matching what the executor always supported. The same
applies tollmstepbaseUrl. urlfields accept local and port-bearing hosts
(http://127.0.0.1:3000/...,http://localhost:5173/...) and query
strings. Previously the validator required a dot-TLD hostname without a
port, which silently rejected localhost URLs.
Downloads
-
Source code (ZIP)
2 downloads
-
Source code (TAR.GZ)
1 download
- Pipeline