Container management plugin for VSKI
  • Python 98.5%
  • Makefile 1.2%
  • Dockerfile 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Anton Nesterov eb2a4d6aba feat: take step files from IaC file:// references, uploads and URLs
inputs, the build-context files and the dockerfile accept the same file
sources: data: URLs (what `vski apply` makes of IaC file:// references,
decoded byte for byte), uploads ({from: /files/...} or a file reference)
and http(s) URLs when allow_external_inputs is set. Inline text stays for
a few lines of shell.

- build files and a Dockerfile given as sources are fetched before the
  build, within the max_inputs/max_input_size budget shared with inputs
- unresolved file:// references are rejected when a pipeline is written
  (step schema) and explained by the plugin; an embedded file used as
  command or entrypoint is rejected with a hint
- env values that resolve to null become empty instead of failing the
  step; steps.<id>.output.inputs is always a list
- example: scripts/report.py with the datasets collection and the
  container-dataset-report pipeline
- the iac-* targets use their own CLI venv (vski-sdk >= 0.17) while the
  plugin runtime stays on vski-sdk 0.15
2026-11-09 22:35:23 +01:00
containers feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
test feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
vski/default feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
.dockerignore init 2026-11-08 01:02:21 +01:00
.env.example init 2026-11-08 01:02:21 +01:00
.gitignore feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
docker-compose.dev.yml init 2026-11-08 01:02:21 +01:00
docker-compose.e2e.yml init 2026-11-08 01:02:21 +01:00
docker-compose.yml init 2026-11-08 01:02:21 +01:00
Dockerfile feat: file inputs and artifacts through the files API 2026-11-09 20:53:21 +01:00
Makefile feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
plan-vski-containers-plugin.md init 2026-11-08 01:02:21 +01:00
pytest.ini init 2026-11-08 01:02:21 +01:00
README.md feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00
requirements.txt feat: file inputs and artifacts through the files API 2026-11-09 20:53:21 +01:00
vplugin.yaml feat: take step files from IaC file:// references, uploads and URLs 2026-11-09 22:35:23 +01:00

VSKI Ephemeral Containers Plugin

A supervised VSKI plugin that builds and runs short-lived Docker containers on behalf of pipelines: it adds a container step type, so any Docker image can run as a pipeline step. Think of each container as a cloud function with a Dockerfile: scrapers, converters, batch jobs, one-shot CLIs. Overview and install guide on the docs site: Containers Plugin.

The plugin owns the whole lifecycle:

  • unique names and vski.* labels on every container/image
  • resource caps (memory, CPU, pids) with instance-level maximums
  • per-container timeout enforcement
  • stdout/stderr capture to a bounded tail plus a full log file
  • files in and out: inputs come from uploads, URLs or scripts committed next to your IaC (file://) and are copied in before the container starts; artifacts are collected after it exits and uploaded to the platform files API; pipelines only pass small references
  • guaranteed cleanup (containers are always removed after the exit is recorded)
  • crash recovery: after a plugin or server restart, labeled containers are adopted and parked pipeline runs are resumed

Use cases

Anything that fits in a Docker image becomes a pipeline step, without installing its toolchain into the vski image:

  • Scheduled scrapers and API pollers: a cron-triggered pipeline runs the scraper with networking opted in; its output (or a JSON artifact) feeds the next step.
  • File processing on upload: a collection trigger fires when a file is uploaded, the container receives it as an input (pandoc, LibreOffice, ImageMagick, ffmpeg, an OCR engine, …), and the results come back as downloadable artifacts.
  • Data jobs and reports: pandas or DuckDB transforms, exports and reports that read their inputs and publish CSV, Parquet or HTML files.
  • Builds and tests: make test or a linter in a pinned toolchain image; a nonzero exit fails the run.
  • Code written by agents: an llm step or a coding agent (through vski mcp) writes a script, and a container step runs it in a throwaway, network-isolated container with resource caps instead of inside the server. That is an isolation layer for semi-trusted code, not a sandbox for hostile code; see Security model.

Install

Requirements:

  • a Linux host with Docker: the plugin starts sibling containers through the host daemon's socket (/var/run/docker.sock), so step images must be pullable by that daemon (or pre-pulled);
  • vski ≥ 0.17.5 with plugins enabled, for native type: container steps (older servers can use the legacy pattern);
  • a plugin interpreter (PLUGIN_PYTHON, Python ≥ 3.11) with vski-sdk 0.15 and this plugin's requirements.txt (docker, httpx).

Option 1: the bundled distribution

This repository builds a ready-to-run image: the official standalone vski image plus Python, the plugin and its dependencies, with plugins enabled.

git clone https://vski.sh/x/containers.git && cd containers
cp .env.example .env      # set JWT_SECRET (openssl rand -hex 32)
make up                   # build and start on SERVER_PORT (default 3000)

Open the dashboard and create the first admin. The compose file mounts the state volume and the Docker socket:

volumes:
  - ./data:/app/data
  - /var/run/docker.sock:/var/run/docker.sock

Option 2: add it to your own distribution

vski plugins are source components that ship inside your image (see Plugins). Copy this repository into the plugins directory as containers/ and install its dependencies into the plugin interpreter:

# or start from your existing vski image
FROM vski.sh/x/vski:latest-standalone

RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*
COPY containers/requirements.txt /tmp/containers-requirements.txt
RUN python3 -m pip install --break-system-packages --no-cache-dir \
      --extra-index-url https://vski.sh/api/packages/x/pypi/simple/ \
      -r /tmp/containers-requirements.txt
COPY containers/ /app/plugins/containers/

ENV PLUGINS_ENABLED=true PLUGINS_DIR=/app/plugins PLUGIN_PYTHON=python3

Keep PLUGINS_DIR outside the data volume (a volume mounted at /app/data hides anything the image put there), and mount /var/run/docker.sock when you run the container.

Create an instance

The plugin does nothing until it has an instance in a database (dashboard → Plugins, the plugin API, or IaC). The instance config caps what steps may request (see Instance configuration):

# vski/default/plugins.yaml
version: 1
database: default
plugins:
  containers:
    plugin: containers
    enabled: true
    config:
      max_concurrent: 2
      max_memory: 1g
      allow_network: true    # let steps opt into networking (off by default)

make iac-apply applies vski/ (this instance plus the example collections, pipelines and the scripts they reference) with the vski CLI, which make cli installs into .venv-cli (vski-sdk ≥ 0.17, which resolves file:// references); then any example runs from the dashboard or the CLI:

make cli
VSKI_URL=http://localhost:3000 .venv-cli/bin/vski login --email admin@example.com --password '…'
make iac-apply
VSKI_URL=http://localhost:3000 .venv-cli/bin/vski pipelines run container-echo

Native type: container steps

The plugin declares a container pipeline step type in vplugin.yaml (steps: block, validated against containers/step.schema.json; vski servers ≥ 0.17.5 support plugin-defined step types). One step replaces the old three-step plugin/signal/check pattern:

# vski/pipelines.yaml
pipelines:
  scrape:
    enabled: true
    steps:
      - id: run
        type: container
        timeout: 600              # overall budget; the container lifetime
        params:                   # validated against the plugin's JSON Schema
          image: python:3.12-slim
          inputs: file://scripts/scrape.py   # committed next to this file
          command: python scrape.py
          network: true
          memory: 256m
    output:
      exitCode: "{{steps.run.output.exitCode}}"
      output: "{{steps.run.output.output}}"
# vski/scripts/scrape.py
import urllib.request

print(urllib.request.urlopen("https://example.com").status)

vski apply embeds the script into the pipeline definition and the plugin copies it into the container as scrape.py before it starts (see Where files come from).

The server injects the run identity (context: runId, stepId, signal, timeout) into container.start, parks the step, and the plugin completes it when the container exits with the platform contract {ok, error?, output?} — a nonzero exit, timeout or stop fails the step (with the output tail in the error). steps.run.output is the job summary (exitCode, ok, output, durationMs, peakCpuPct, peakMemMb, …). The step timeout doubles as the container lifetime unless params.timeout overrides it (both clamp to max_timeout).

Dispatch is at-least-once across server restarts: jobs are keyed by runId/stepId, and a re-dispatch (or the plugin's own adoption after a restart) resumes the same container instead of starting a duplicate.

Legacy pattern (older servers)

Servers without plugin-defined step types still work with the canonical three-step pattern (vski/default/pipelines.yaml keeps a container-echo-legacy example):

plugin: containers/container_start
   -> signal: <name>
   -> plugin: containers/container_check

container_start validates the step, persists a durable job record and returns immediately. A background task builds/starts/watches the container (the JSON-RPC pipe is never blocked). When the container exits, the plugin calls client.workflow.send_signal(runId, name, result), which completes the parked signal step. container_check raises on a nonzero exit or timeout, turning a failed container into a failed pipeline run. The scrape example above, in this form:

pipelines:
  scrape:
    enabled: true
    options:
      executionTimeout: 900000       # bound the whole run (ms)
    steps:
      - id: run
        type: plugin
        target: containers/container_start
        timeout: 120                 # dispatch only; must return quickly
        params:
          runId: "{{run.id}}"
          stepId: run
          signal: scrape-done
          image: python:3.12-slim
          inputs: file://scripts/scrape.py
          command: python scrape.py
          network: true
          memory: 256m
          timeout: 10m               # container lifetime
      - id: wait
        type: signal
        name: scrape-done
        dependsOn: [run]
      - id: check
        type: plugin
        target: containers/container_check
        params:
          result: "{{steps.wait.output}}"
        dependsOn: [wait]
    output:
      exitCode: "{{steps.check.output.exitCode}}"
      output: "{{steps.check.output.output}}"

Downstream steps can use {{steps.wait.output.output}} directly, or {{steps.check.output.output}} after the gate; inputs and artifacts work the same way in both forms.

Function names in pipelines: the server validates plugin targets with ^[A-Za-z0-9_-]{1,64}/[A-Za-z0-9_-]{1,64}$, which forbids dots. Every function is registered both as container.start (direct invokes, docs) and container_start (pipeline targets). Use the underscore form in target.

Step reference

Where files come from

Every parameter that takes a file (inputs, the values of files, and dockerfile) accepts the same sources:

Source Written as Notes
A file committed with your IaC file://scripts/report.py Relative to the YAML file, inside the IaC directory. vski apply (vski-sdk ≥ 0.17) embeds it as a data: URL: at most 256 KiB per file and 1 MiB per pipeline
An uploaded file /files/<collection>/<recordId>/<filename>, or a reference {collection, recordId, filename} Record file fields, trigger records, artifacts of earlier steps
A URL https://example.com/data.csv Only with allow_external_inputs: true
Inline text the string itself files and dockerfile only: a few lines of shell
  • Strings. In inputs a string is always a source. In files and dockerfile a string is inline text, unless it is a file:// reference or a data: URL; write any other source as {from: <source>} (a file reference object works as is).
  • Embedded files arrive byte for byte. A data: URL is opaque to the pipeline template language: {{…}} inside a script stays as written, and JSON or Jinja braces cannot clash with templates. Pass values to scripts through env. Inline text is a pipeline string instead: {{…}} expands in it, and stray {{/}} fail validation, which is why anything longer than a few lines belongs in its own file.
  • Size. Embedded files travel inside the pipeline definition and inside every run's snapshot of it, hence the limits. Upload datasets, models and archives through the files API and reference them.
  • Unresolved references. A file:// that reaches the server as is (an older CLI, or a definition written by hand) is rejected when the pipeline is written: the error names the parameter (at '/inputs/0': 'not' failed).
  • Limits. max_inputs and max_input_size cover every file a step fetches: inputs, and build files or a Dockerfile given as sources.

Building an image: dockerfile and files

files is the build context, and the dockerfile decides where its files go with COPY. Here the Dockerfile and the program are committed next to the IaC, the dependency list was uploaded through the files API, and a two-line wrapper is inline:

      - id: convert
        type: container
        timeout: 300
        params:
          dockerfile: file://images/convert/Dockerfile
          files:
            convert.py: file://images/convert/convert.py
            requirements.txt: {from: /files/build_files/k3j2…/requirements.txt}
            run.sh: |                # inline: fine for a few lines of shell
              #!/bin/sh
              exec python /app/convert.py "$@"
          memory: 256m
# vski/images/convert/Dockerfile
FROM python:3.12-slim
COPY requirements.txt convert.py run.sh /app/
RUN pip install --no-cache-dir -r /app/requirements.txt
ENTRYPOINT ["sh", "/app/run.sh"]

image + files (without dockerfile) builds a small overlay image (FROM <image>, files copied into /work, which becomes the workdir). A script that needs no build step is simpler as an input: inputs: file://scripts/report.py copies it into the container, no image build involved.

Secrets and env

Pipeline templates resolve secrets before the plugin is invoked; the value never appears in the definition:

        params:
          env:
            SCRAPER_TOKEN: "{{secrets.scraper_token}}"

The plugin also accepts a secrets mapping (merged into env) for callers that already have resolved values:

          secrets:
            SCRAPER_TOKEN: "{{secrets.scraper_token}}"

A list such as secrets: [SCRAPER_TOKEN] is rejected with guidance: current servers do not expose secret values to plugins, so the value must be templated into env/secrets.

Any template works in env, which is how values reach an embedded script. A template that resolves to null (an empty record field) sets an empty value instead of failing the step.

Step parameters

Native steps pass these as params (validated against containers/step.schema.json); direct container.start invokes and the legacy pattern take the same keys. The identity keys (runId, stepId, signal, pipeline) are injected by the server for native steps.

Parameter Default Description
runId / run_id generated for manual invokes Pipeline run to signal on completion
stepId / step_id step Step identity; runId + stepId are the idempotency key
signal container-<stepId> when runId is set Signal name the downstream signal step waits on
pipeline "" Optional label for dashboards
image — Image to run (exactly one of image/dockerfile)
dockerfile — Dockerfile to build: inline text, file:// (IaC) or {from: <file>} (see Where files come from)
files {} Build context: relative path → inline text or a file (256 files; inline text up to 2 MiB)
inputs [] Files copied in before start (see Files)
artifacts [] Paths/globs uploaded after a successful exit (see Files)
command image default List (exec) or string (wrapped in /bin/sh -c)
entrypoint image default String or list
env {} Environment values (templated by the pipeline; null becomes empty)
secrets {} Mapping merged into env
network false false/"none" keeps the container isolated; true uses the configured default network; a string names a Docker network
memory instance default (256m) Docker memory limit, clamped to max_memory
cpus instance default (1.0) CPU limit, clamped to max_cpus
pidsLimit instance default (128) Process limit, clamped to max_pids_limit
timeout instance default (600s) Container lifetime; accepts 90, 30s, 10m, 1h; clamped to max_timeout
workdir unset Working directory (overlay builds default to /work)
labels {} Extra labels; names starting with vski. are reserved
readOnly false Read-only root filesystem
user image default Container user
buildArgs {} Docker build args
noCache false Disable the Docker build cache for this job
pull instance pull_policy missing / always / never
init instance default (true) Use Docker's init process

Unknown keys are rejected (this is how volumes, devices, privileged mode and capability additions are kept out). Paths in files must stay inside the build context and cannot replace its Dockerfile; inputs and artifacts use Docker's archive API, never mounts. A command or entrypoint string that is an embedded file is rejected with a hint: copy the script in with inputs and run it. All requests are clamped to the instance maximums; warnings are returned in container.start/container.status/container.validate and in the final step output.

Step result

The job summary is a native step's output (steps.<id>.output), the legacy pattern's signal payload, and what container.start and container.status return:

{
  "jobId": "wkf_abc__run__a1b2c3",
  "runId": "wkf_abc",
  "stepId": "run",
  "signal": "scrape-done",
  "state": "completed",
  "containerId": "…",
  "containerName": "vski-wkf_abc-run-a1b2c3",
  "exitCode": 0,
  "ok": true,
  "timedOut": false,
  "stopped": false,
  "missing": false,
  "error": null,
  "output": "<stdout+stderr tail>",
  "logFile": "/app/data/container-logs/wkf_abc/run/….log",
  "created": "…", "startedAt": "…", "finishedAt": "…", "durationMs": 1234,
  "signaled": true,
  "warnings": [],
  "artifacts": [ /* artifact references, see Files */ ],
  "artifactsBytes": 48213,           // when artifacts were collected
  "inputs": [ /* {target, source, size, sha256, …} per file copied in */ ]
}

Exit policy: ok is true only for exit code 0 with no timeout, stop, vanished container or error (failed input or artifact transfers included). A native step fails otherwise, with the output tail in its error; in the legacy pattern container_check raises instead. Either way the run fails unless the step is configured with onError: "continue".

The full log remains on the data volume at logFile even after the container is removed.

Files: inputs and artifacts

Files never travel through the pipeline itself: steps pass small references to files stored by the platform's files API. inputs copies files into the container before it starts; artifacts collects files after a successful exit and uploads them:

steps:
  - id: build
    type: container
    params:
      image: python:3.12-slim
      workdir: /work
      inputs:                                    # fetched, then copied in
        - file://scripts/build.py                # committed with the IaC
        - from:                                  # the uploaded file
            collection: uploads
            recordId: "{{trigger.id}}"
            filename: "{{trigger.record.doc}}"
          dest: in/source.csv
      command: python build.py in/source.csv out/
      artifacts: ["out/*.parquet", "out/report.html"]   # uploaded after exit 0
  - id: publish
    type: container
    dependsOn: [build]
    params:
      image: alpine:3.20
      workdir: /work
      inputs: "{{steps.build.output.artifacts}}"         # every file, at its name
      command: ls -lR out

vski/default/pipelines.yaml has three runnable examples: container-artifacts (files from one step to the next), container-upload-digest (an uploaded file triggers a container) and container-dataset-report (a committed script renders every uploaded CSV as an HTML report).

Artifact references

steps.<id>.output.artifacts lists one reference per uploaded file:

{
  "kind": "file",
  "name": "out/report.html",      // relative to the workdir (other paths: without the leading /)
  "filename": "report.html",      // stored filename (sanitized basename)
  "size": 48213,
  "sha256": "9f2c…",
  "collection": "pipeline_artifacts",
  "recordId": "3e11…",
  "path": "/files/pipeline_artifacts/3e11…/report.html",  // GET <server><path>
  "runId": "wkf_…", "stepId": "build", "jobId": "wkf_…__build__a1b2c3"
}

A reference is everything a consumer needs: another container step takes it as an input, {{steps.<id>.output.artifacts[0].path}} can go into a run's output, and anything that can reach the server downloads the file from path.

Every artifact is an ordinary record in the pipeline_artifacts collection (instance config artifacts_collection) with the fields runId, stepId, jobId, pipeline, name, size, sha256, status (pending → ready) and the bytes in file. The plugin creates the collection on first use — authenticated principals can list and view it, only plugin keys can write it — so artifacts are queryable (filter=runId = 'wkf_…'), subscribable over realtime, and can trigger pipelines of their own. vski/default/collections.yaml declares the same collection for IaC users (vski iac apply --prune drops undeclared collections).

inputs

Each item is one of:

Form Example
a file committed with your IaC file://scripts/report.py: embedded by vski apply, lands at report.py
a file reference {collection: uploads, recordId: "{{trigger.id}}", filename: "{{trigger.record.doc}}"}, or an artifact reference from an earlier step
a files path /files/uploads/<recordId>/<filename>
{from, dest?, optional?} from is any of these, or a list of them (e.g. "{{steps.build.output.artifacts}}")
an http(s) URL only with allow_external_inputs: true

inputs itself may also be a single item or a templated list (inputs: "{{steps.build.output.artifacts}}").

  • Placement. A file lands at dest; when dest ends in / (required for a list) at <dest><name>; without dest at <name>, where name is an artifact's relative name (a directory of artifacts keeps its layout), else the filename (for a file:// reference, its basename). Relative destinations resolve against the workdir (workdir param, else the image's WORKDIR, else /).
  • Copying. Files are fetched to the plugin's staging directory with the instance budget enforced while streaming (max_inputs, max_input_size), checked against the reference's size/sha256 when it carries them, and copied in with Docker's archive API while the container is created but not yet started — no volumes or mounts. They arrive owned by root with mode 0644 (run scripts as sh run.sh, or chmod them first); missing parent directories are created. A missing input fails the step before any container exists.
  • Upload races. A collection create event fires before the uploaded bytes are stored. A not-found input is retried for input_wait seconds (default 30), so pipelines can trigger on create.
  • optional: true skips a file that does not exist (with a warning) instead of failing the step.
  • readOnly containers cannot take inputs (Docker refuses writes to a read-only root filesystem); bake such files into the image instead.

steps.<id>.output.inputs reports what was copied in (target, source, size, sha256).

artifacts

A path or glob, or a list of them, relative to the workdir or absolute:

Pattern Collects
out/report.html that file (it must exist)
out every file below out/, recursively
out/*.json * and ? match within one directory, [...] is a character class
out/**/*.csv ** matches any number of directories

Wildcards do not match names starting with . (name hidden files explicitly). A literal path that does not exist fails the step; a glob that matches nothing only adds a warning. Symbolic links below a collected directory are skipped; a symlinked path you name is followed.

  • When. Only after a successful exit (code 0, no timeout or stop). The job is collecting until every file is uploaded, and the parked run is resumed only then. Failed containers keep their full log (container.logs).
  • How. Files are read from the stopped container with Docker's archive API (docker cp semantics: any image, no shell needed), staged, and uploaded as one record per file. The record id is saved before the bytes are sent, so a plugin restart resumes the upload instead of duplicating it.
  • Limits. max_artifacts files and max_artifact_size bytes per step. A glob may read at most 4× max_artifact_size while scanning its base directory, so collect from a dedicated output directory rather than a workdir full of inputs.
  • Failures. A missing literal path, an exceeded limit or an upload that keeps failing fails the step with artifact collection failed: …; the uploads already made for it are deleted.
  • Retention. The hourly sweep deletes artifact records (and with them their files) older than artifact_retention_hours (default 30 days; 0 keeps them), and uploads abandoned for 24 hours.

Scripts and configs that belong to a pipeline are committed with its IaC and come in as file:// references; data comes from uploads.

Resource stats

The plugin reports resource stats to the platform (dashboard → Plugins → instance → Stats):

  • jobs.running, jobs.queued, jobs.completed_1h, jobs.failed_1h, jobs.capacity_pct — job counters;
  • containers.cpu_pct, containers.mem_mb — sums across running containers;
  • container.<step>-<token>.cpu_pct / .mem_mb — per-container series for the most recent running jobs (up to 10);
  • meta: the running-job list (pipeline, run, step, container name) and the instance limits.

Containers are sampled with docker stats every stats_interval seconds (default 10, config) in a background loop; each job's peak CPU/memory is recorded into its job record (peakCpuPct / peakMemMb) when it finishes. Requires vski-sdk ≥ 0.15 for the @app.stats collector; on older SDKs the plugin runs normally and simply reports nothing.

Instance configuration

Key Default Description
max_concurrent 2 Builds + running containers at once (a semaphore); extra jobs park in pending
default_memory / max_memory 256m / 1g Memory default and hard cap
default_cpus / max_cpus 1.0 / 2.0 CPU default and hard cap
default_pids_limit / max_pids_limit 128 / 512 Process default and hard cap
default_timeout / max_timeout 600 / 3600 Container lifetime default and hard cap (seconds)
allow_network false Must be true for steps to request networking
default_network bridge Network used by network: true
allow_build true Permit image builds (dockerfile, or image + files overlays)
build_timeout 600 Docker build API timeout (seconds)
log_tail_bytes 65536 Size of the output tail in the signal/step result
job_retention_hours 72 Completed, signaled job records kept for status
log_retention_hours 168 Full log files kept
image_retention_hours 72 Built images kept by the prune sweep
prune_after_hours 24 Orphaned labeled containers/build dirs age
pull_policy missing missing / always / never
poll_interval 0.5 Container state poll interval (seconds)
dispatch_wait 2.0 How long container.start waits for the container to reach running before returning queued: true
init true Docker init process for reaping
cap_drop_all true Drop all Linux capabilities
no_new_privileges true Set no-new-privileges
artifacts_collection pipeline_artifacts Collection storing artifacts (created on first use)
max_inputs / max_input_size 32 / 512m Files a step fetches (inputs, plus build files and a Dockerfile given as sources) and their total size; max_inputs: 0 disables fetching
max_artifacts / max_artifact_size 64 / 1g Artifact files per step and their total size (max_artifacts: 0 disables artifacts)
transfer_timeout 300 Seconds for one file download or upload request
input_wait 30 Seconds a not-yet-stored input is retried (collection create races)
artifact_retention_hours 720 Artifact records (and files) older than this are swept; 0 keeps them
allow_external_inputs false Allow http(s) URLs as inputs or build files (fetched by the plugin process, i.e. with the server's network reach)

Functions

All functions are available as container.<name> and container_<name>.

Function Purpose
container.start Validate, persist and dispatch; returns a summary (queued or running)
container.status Job state by jobId, runId/stepId, containerId, or all recent jobs; includes live Docker state
container.stop Kill/remove and complete the job with stopped: true (resumes the parked run)
container.adopt Reattach watchers to labeled containers after a restart; finish jobs that already exited; resume interrupted artifact uploads; retry undelivered signals
container.prune Remove old orphaned containers/images/records/logs/staging files and expired artifacts (olderThan, imageOlderThan, artifactsOlderThan, images, jobs, logs, artifacts)
container.logs Tail (tail) or full (full: true) captured output and log path
container.artifacts Artifact references of jobs (jobId, runId/stepId, runId, or every job that has artifacts)
container.check Pipeline failure gate: returns the result when ok, raises otherwise
container.validate Normalize/validate params without starting anything; secrets are redacted

The plugin also runs two internal crons: every 5 minutes container.adopt (re-attach/retry signals) and hourly container.prune (retention sweep).

Lifecycle and crash recovery

pending -> starting -> running -> [collecting ->] completed

starting covers fetching build files and inputs and preparing the image; collecting is artifact copy-out and upload after a successful exit.

Every job has a JSON record under <DATA_DIR>/containers/jobs/ (mode 0600) containing the normalized params while it is active and the final result after completion. On completion the params (including resolved env values) are scrubbed from the record. The record is what lets the plugin:

  • make container.start idempotent for the same runId + stepId
  • serve container.status after the container is gone
  • finish a job whose container exited while the plugin was down
  • resume artifact collection: copy-out and upload progress (including each record id, saved before its bytes are sent) is persisted, so an interrupted upload continues instead of starting over
  • retry an undelivered workflow signal on the next adoption/sweep

In-flight files live in <DATA_DIR>/containers/staging/<jobId>/ and are removed when the job completes (the sweep removes leftovers).

Container labels (vski.ephemeral=true, vski.instance, vski.jobId, vski.runId, vski.stepId, vski.signal, vski.created, vski.createdEpoch, vski.timeout) are the recovery keys. Containers are never AutoRemove: the plugin removes them only after the exit code and logs are recorded.

If the plugin or the whole vski container restarts, the Docker container keeps running. on_start calls adopt, which reattaches a log follower and watcher. A container that exited during the downtime is read from Docker's logs, its exit is recorded, the run is signaled, and the container is removed (after its artifacts are collected, in the background). A container that was created but never started gets its inputs copied in and is started.

The pod/plugin enforces the timeout itself (kill after timeout). The pipeline's signal step has no timeout of its own, so set the pipeline's options.executionTimeout as the outer safety bound. A dead plugin can therefore never wedge a run forever.

Logs and status

  • output in the signal is the last log_tail_bytes of combined stdout+stderr.
  • The full log is at <DATA_DIR>/container-logs/<runId>/<stepId>/<jobId>.log. Because ./data is mounted from the host, the file is directly accessible (e.g. data/container-logs/...) and survives container removal.
  • Build output is written to the plugin log (dashboard → Plugins → Logs); a build failure includes the last lines in the job error.
  • container.logs returns the log path and a tail without shell access.

Security model

The Docker socket is root-equivalent on the host. Mounting /var/run/docker.sock into the vski container means the plugin (and anything that can invoke it) can control every container on the host. This stack is aimed at personal/lab automation, not a hostile multi-tenant environment.

Guardrails provided by the plugin:

  • workers are network-isolated by default and only opt in per step and per instance (allow_network)
  • memory/CPU/pids are always clamped to instance maximums
  • all capabilities are dropped and no-new-privileges is set
  • volumes, devices, privileged mode, host networking and capability additions are rejected as unknown parameters
  • build contexts can only contain the step's files (no host mounts)
  • inputs and artifacts move bytes through Docker's archive API into and out of the container's own filesystem: inputs only while it is created and not yet running, artifacts only after it stopped, so a running container cannot race the copy; archive member names are never used as host paths (files are staged under index names, links are not recreated)
  • URL sources are off by default (allow_external_inputs), for inputs and build files alike: the plugin process fetches them, so enabling it lets pipeline authors reach anything the server can reach; the plugin's API key is only ever sent to the vski server itself
  • file:// references are resolved by the vski CLI on the machine that applies the IaC, never by the server or the plugin. The CLI only reads files inside the IaC directory (symbolic links included), so a definition cannot pull in ~/.ssh or a CI runner's secrets
  • every resource is labeled and removed after its exit is recorded
  • containers are one-shot by contract; there is no restart policy

Input and artifact files are served by the platform's files route, which does not check credentials: anyone who can reach the server and knows a path can download the file (record ids are random UUIDs, not secrets). This holds for every vski file field today; keep secrets out of artifacts.

Building agent-authored Dockerfiles is a deliberate trust decision, the same trust class as agent-written shell steps. Do not let untrusted users create pipeline definitions against this instance. Rootless dockerd or a sibling builder are the natural hardening options if the trust boundary changes.

Dockerfile guidance (for step authors / agents)

  • Prefer alpine, -slim or distroless bases; pin digests in production.
  • One RUN with --no-cache-dir; delete package lists; no build tools in the final image.
  • COPY only what the step needs. Scripts are files of their own, committed next to the IaC and referenced with file:// (as inputs, or as files when the build needs them); inline text is for a few lines of shell.
  • No long-running servers, no volumes — containers are one-shot.
  • Write diagnostics to stdout/stderr; that is the step output.
  • Write result files to a dedicated directory (e.g. out/) and list it in artifacts; read data from inputs, not from the network.
  • Exit nonzero on failure; container_check turns that into a failed run.

An agent using vski mcp can author a pipeline with type: container steps and apply it through the normal pipeline API; no plugin-specific UI is required. Without a checkout to commit into, it uploads its scripts through the files API and references them with /files/… paths.

Layout

.
├── vplugin.yaml                 manifest: runtime, config schema, functions, step type
├── requirements.txt             docker SDK + vski-sdk + httpx
├── containers/
│   ├── main.py                  plugin entry point (thin handlers)
│   ├── runtime.py               job state machine, watcher, artifacts, adoption, pruning
│   ├── dockerops.py             Docker SDK operations (sync, called in threads)
│   ├── artifacts.py             artifact patterns, naming, tar streams in and out
│   ├── files.py                 files API client: downloads, artifact records, uploads
│   ├── config.py                step/config validation, clamping, parsers
│   ├── jobs.py                  durable job records and log/build/staging paths
│   └── step.schema.json         JSON Schema of the `container` step params
├── Dockerfile                   standalone image + Python + plugin deps
├── docker-compose.yml           the stack + /var/run/docker.sock mount
├── docker-compose.dev.yml       hot plugin source overlay
├── docker-compose.e2e.yml       isolated e2e stack (used by make test-e2e)
├── vski/                        example IaC (instance, collections, pipelines, scripts/)
└── test/
    ├── unit/                    pure/fake-Docker tests, no server
    └── e2e/                     real stack + real sibling containers

Testing

make test        # unit + e2e
make test-unit   # validation/store/runtime tests against a fake Docker daemon and an in-memory files API
make test-e2e    # builds the image, boots an isolated stack, runs real containers

The e2e suite (test/e2e/test_containers.py) covers:

  • health, web UI and plugin catalog discovery
  • instance start and direct invocations
  • image runs, image+files overlay builds and inline Dockerfile builds
  • a full pipeline with container_start -> signal -> container_check
  • nonzero exit and timeout failure propagation
  • env/secret resolution by the core
  • memory/CPU/pids clamping and network isolation (inspected on the host)
  • opt-in networking
  • stop and container removal
  • concurrency queuing
  • adoption after a plugin-process restart and a full server restart
  • label-scoped orphan pruning
  • validation errors and secret redaction

test/e2e/test_files_and_artifacts.py covers files end to end, with sha256 digests computed inside the containers:

  • a record's uploaded file as a container input
  • artifacts uploaded, downloaded back through the files route, and listed
  • a missing artifact path failing the job cleanly
  • a native two-step pipeline handing one step's artifacts to the next
  • a collection create trigger feeding the uploaded file to a container
  • embedded files (what vski apply makes of file://) reaching a build and a container byte for byte, braces and all, while the same script as inline text is rejected by the template check
  • build files fetched from an upload
  • unresolved file:// references rejected when the pipeline is written
  • a null template value in env

make test-e2e uses port 3999, state in ./.e2e-data (cleared at the start of each run and kept afterwards for inspection), and the compose project vski-containers-e2e. Set VSKI_E2E_URL to run a single test against an existing stack.

Limitations / next steps

  • Artifacts are collected only from containers that exit 0: a failing step keeps its full log (container.logs) but not its files. Collecting from failed runs as well (test reports) would be a per-step option.
  • Files are served by a files route that does not check credentials (see Security model); private artifacts need an authenticated files route in the platform.
  • readOnly containers cannot take inputs (Docker refuses archive writes to a read-only root filesystem).
  • Embedded (file://) files are capped at 256 KiB each and 1 MiB per pipeline because they live in the definition and in every run's snapshot of it; larger files go through uploads. vski pull writes them back as data: URLs, since the live definition no longer knows their paths.
  • Signals are at-least-once: a crash between send_signal and recording signaled can deliver a duplicate. Signal names are per-run/per-step, so the duplicate sits unused; consumers should still be idempotent. Artifact references are recorded before the first signal, so a duplicate carries the same references.
  • Windows/macOS hosts are out of scope (Linux-only, matching VSKI).
  • Build cache eviction is age-based per label; a size-aware policy is a later improvement.