• v0.17.5 2e362fb796

    v0.17.5 Stable

    x released this 2026-11-07 22:20:47 +00:00 | 12 commits to main since this release

    VSKI v0.17.5 — Plugin steps, plugin stats, WAL durability

    The plugin platform grows up in two directions: plugins can now
    define their own pipeline step types (a container step instead of
    the start → signal → check workaround) and report live stats — both
    process metrics the server collects itself and custom metrics the
    plugin publishes. Under the hood, the storage layer fixes a
    connection-pool race that could make pooled connections read through
    deleted WAL files (empty/stale reads, then database disk image is malformed on writes) — that class of failure is now structurally
    impossible from inside the server and blocked from outside it.

    Safe upgrade: no schema migrations; stats tables are created in
    stats.db on start and plugin catalogs are re-scanned automatically.
    Everything is on by default. SDKs: @vski/sdk 2.10.0, vski-sdk
    0.15.0.

    Plugin-defined pipeline step types

    A plugin manifest can now declare step types alongside its functions:

    steps:
      container:
        description: Build/run an ephemeral Docker container and report its exit code and output.
        params: containers/step.schema.json   # file path or inline JSON Schema
        functions:
          start: container.start              # required; must ack fast
          check: container.check              # optional, informational
    
    • Registry and validation. Scanning inlines the params schema into
      the catalog; step types resolve per database from plugins with an
      enabled instance. Type names must match ^[a-zA-Z0-9_-]{1,64}$ and
      collisions across plugins are rejected at validation. Param templates
      ({{...}}) are checked as usual; fully-resolved params are validated
      against the plugin's JSON Schema (draft 2020-12).
    • Asynchronous execution. The server dispatches the start function
      with the step params plus an injected context
      (runId, stepId, pipeline, db, signal, timeout, attempt)
      under a 30-second dispatch budget — the function acks and does its
      work in the background. The step parks on a deterministic per-step
      signal; the plugin completes it with
      client.workflow.send_signal(runId, signal, {ok, error, output}) —
      ok: false fails the step with error, anything else completes it
      with output. Completion is durable: signals that arrive before the
      park are buffered and replayed, runId/stepId give the plugin an
      idempotency key for re-dispatches, and the step timeout (1–86400 s)
      is passed through as the overall budget. Retries cover dispatch
      failures; a plugin-reported failure is terminal.
    • Builder. Plugin step types appear in the palette next to the
      builtins with their own icon and accent color, and the inspector
      renders a form from the params schema (typed inputs, enums as
      selects, JSON fallback for complex shapes).
    • The containers plugin ships a native container step — build/run an
      ephemeral container with resource caps, log capture and timeout
      enforcement — replacing the three-step workaround in a single node.

    Plugin stats: process and custom

    Every plugin instance gets a live stats view, from two sources merged
    by the 30-second stats collector:

    • Process stats (server-collected). Read from /proc per plugin
      child process: RSS, CPU % (utime+stime delta), open file
      descriptors, uptime — stored in _stats_plugin_processes.
    • Custom metrics (plugin-reported). A new stats JSON-RPC method,
      polled only from runtimes that advertise capabilities: ["stats"] in
      initialize (old runners are simply not polled). In py-sdk,
      collect metrics with @app.stats — multiple collectors share a 4 s
      budget and merge into a flat numeric map, stored one row per metric
      in _stats_plugin_metrics. Both tables are queryable system
      collections; retention is 7 days.
    • API. GET /api/plugin-instances/:name/stats?window=3600 returns
      bucketed process and per-metric time series plus the latest report's
      meta; GET /api/plugins/stats returns the latest snapshot of every
      instance with per-plugin rollups (instances, RSS/CPU/FD totals).
    • Console. The instance modal shows stat tiles (RSS, CPU %, FDs,
      uptime) with history charts and a custom-metric picker with a
      multi-line chart; the plugins page shows a totals strip per plugin.

    Storage: the server can no longer lose its WAL files

    Two failure paths could delete a live database's -wal/-shm while
    pooled connections still held them open: a shared-memory
    reference-counting race in the vendored SQLite fork under connection
    churn, and any external SQLite client closing the same file
    cross-process. Connections left on deleted inodes read silently
    stale/empty data (an API list suddenly returns []) and fail writes
    with database disk image is malformed until restart — the on-disk
    database was never actually damaged, but the server was effectively
    poisoned.

    All three layers are fixed:

    • The vendored fork never unlinks -wal/-shm — files left by a
      crash or external tool are recovered on the next open. (The fork's
      deliberate close-on-reset behavior that cured the benchmarked fd
      leak is untouched; only the unlink calls are gone.)
    • Each database holds a permanent guard connection, so no
      in-process close path can ever believe it is the last connection.
    • Out-of-SQLite sentinel file locks (shared lock over the database
      file's lock region and the shm dead-man byte), refreshed every
      second, make any external client's last-connection cleanup see a
      live holder and stop deleting.

    Regression tests cover the churn path and guard boundedness
    (internal/db/fdleak_test.go); the full suite is green.

    Console polish

    • Plugin instance configuration is now schema-driven forms — typed
      inputs, selects for enum fields, secrets with show/hide, JSON
      fallback — for both create and edit, instead of raw JSON textareas.
    • The plugin catalog moved into a modal behind the Scan button;
      the plugins page leads with stat cards and instances.
    • The sidebar can be hidden (choice remembered per browser), with
      a floating reveal button in the top-left corner.
    • Pipeline builder: combined Save + actions dropdown (Settings,
      Validate) and a truncated pipeline title with the full name on
      hover.
    • Consistent page headers and actions on Pipelines and Workflows
      (Refresh, then New), and the Authentication page uses the full
      content width.

    Documentation

    • Backup.md gains "Data directory access rules": the server is the
      only process that should touch DATA_DIR; -wal/-shm are normal,
      never hand-pruned; replicas sync over HTTP and never read the
      master's data directory; file-level backups require a stopped
      server or a replica.
    • Replication.md states the HTTP-sync model explicitly; Plugins.md
      and Pipelines.md document step-type declarations and the async
      step contract.
    Downloads