1 Containers Plugin
Anton Nesterov edited this page 2026-11-09 22:36:07 +01:00

The containers plugin adds a container step type to pipelines: any Docker image runs as a short-lived, resource-capped job, and files move in and out through the files API.

Think of each container as a cloud function with a Dockerfile. The plugin pulls or builds the image, starts the container with resource limits, enforces its timeout, captures its output, copies files in and out, removes it, and resumes the pipeline run. It is an official plugin; the source and the complete parameter reference live at vski.sh/x/containers.

What it does

sequenceDiagram
    participant Run as pipeline run
    participant Plugin as containers plugin
    participant Files as files API
    participant Docker as host Docker daemon
    Run->>Plugin: start the step (params + run context)
    Plugin->>Files: download inputs
    Plugin->>Docker: pull or build the image, create the container
    Plugin->>Docker: copy inputs in, start, watch (limits, timeout)
    Docker-->>Plugin: exit code and logs
    Plugin->>Docker: copy artifacts out, remove the container
    Plugin->>Files: upload artifacts
    Plugin-->>Run: resume the step with the job summary
  • A native step type. type: container steps are validated against the plugin's JSON Schema when a pipeline is saved, and the dashboard's pipeline builder renders a form for them. The step output is the job summary: exit code, output tail, duration, peak CPU and memory, artifacts.
  • Any image, or your own Dockerfile. Run a published image as is, copy your scripts into it, or build a Dockerfile that is committed next to the pipeline's IaC (or uploaded).
  • Limits and isolation. Memory, CPU and process limits (clamped to the instance's maximums), a timeout, no network unless both the step and the instance allow it, all Linux capabilities dropped, no-new-privileges.
  • Files in and out. inputs copies files into the container before it starts: an uploaded record's file, an earlier step's artifacts, a URL, or a script committed next to your IaC (file://scripts/report.py). artifacts uploads result files after a successful exit and lists them in the step output.
  • Logs. The last 64 KiB of stdout/stderr is the step's output; the full log stays on the data volume.
  • Crash recovery. Containers outlive plugin and server restarts: the plugin re-attaches to running ones, finishes those that exited in the meantime, resumes interrupted uploads and resumes the parked runs.
  • Cleanup and stats. A container is removed as soon as its exit is recorded, an hourly sweep removes leftovers and expired artifacts, and running jobs, CPU and memory show on the dashboard's Plugins page.

Use cases

  • File processing on upload. A collection trigger fires when a file is uploaded; the container receives it (pandoc, LibreOffice, ImageMagick, ffmpeg, an OCR engine, …) and the results come back as downloadable artifacts. See the example below.
  • Scheduled scrapers and API pollers. A cron-triggered pipeline runs the scraper with networking opted in; its output or a JSON artifact feeds the next step.
  • Data jobs and reports. pandas or DuckDB transforms, exports and reports that read their inputs and publish CSV, Parquet or HTML files.
  • Builds and tests. make test or a linter in a pinned toolchain image; a nonzero exit fails the run.
  • Code written by agents. An llm step or a coding agent (through vski mcp) writes a script, and a container step runs it in a throwaway, network-isolated container instead of inside the server. This isolates semi-trusted code; it is not a sandbox for hostile code (see Security).
  • Tools outside Python. CLIs such as gdal, ffmpeg or pandoc run in their own images instead of being installed into the vski image.

Install

Requirements

  • A Linux host with Docker. The plugin starts sibling containers through the host daemon's socket, /var/run/docker.sock, so step images must be pullable by that daemon (or pre-pulled).
  • vski ≥ 0.17.5 with plugins enabled.
  • A plugin interpreter (PLUGIN_PYTHON, Python ≥ 3.11) with vski-sdk 0.15 and the plugin's requirements.txt (docker, httpx).
  • To apply IaC that references files with file://: the vski CLI from vski-sdk ≥ 0.17.

Option 1: the bundled distribution

The plugin repository builds a ready-to-run image: the standalone vski image plus Python, the plugin and its dependencies, with plugins enabled.

git clone https://vski.sh/x/containers.git && cd containers
cp .env.example .env      # set JWT_SECRET (openssl rand -hex 32)
make up                   # build and start on SERVER_PORT (default 3000)

Open the dashboard and create the first admin. docker-compose.yml mounts the data volume and the Docker socket:

volumes:
  - ./data:/app/data
  - /var/run/docker.sock:/var/run/docker.sock

Option 2: add it to your own distribution

Plugins are source components that ship inside your image. Put the plugin repository in your build context as containers/, install its dependencies into the plugin interpreter, and copy it into the plugins directory:

# or start from your existing vski image
FROM vski.sh/x/vski:latest-standalone

RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*
COPY containers/requirements.txt /tmp/containers-requirements.txt
RUN python3 -m pip install --break-system-packages --no-cache-dir \
      --extra-index-url https://vski.sh/api/packages/x/pypi/simple/ \
      -r /tmp/containers-requirements.txt
COPY containers/ /app/plugins/containers/

ENV PLUGINS_ENABLED=true PLUGINS_DIR=/app/plugins PLUGIN_PYTHON=python3

Run the image with the Docker socket mounted (-v /var/run/docker.sock:/var/run/docker.sock). Keep PLUGINS_DIR outside the data volume: a volume mounted at /app/data hides anything the image put there.

Create an instance

A plugin runs once it has an instance in a database. Create one on the dashboard's Plugins page, through the plugin API, or with IaC:

# vski/default/plugins.yaml
version: 1
database: default

plugins:
  containers:
    plugin: containers
    enabled: true
    config:
      max_concurrent: 2
      max_memory: 1g
      allow_network: true    # let steps opt into networking (off by default)

The instance config caps what steps can request (see Configuration). Pipelines in that database can now use type: container.

Check that it works

# vski/pipelines.yaml
version: 1

pipelines:
  hello-container:
    enabled: true
    steps:
      - id: hello
        type: container
        timeout: 60
        params:
          image: alpine:3.20
          command: echo "hello from Alpine $(cat /etc/alpine-release)"
          memory: 64m
    output:
      exitCode: "{{steps.hello.output.exitCode}}"
      output: "{{steps.hello.output.output}}"
vski iac apply                       # the instance and the pipeline
vski pipelines run hello-container   # prints the run id
vski runs show <run id>              # output: exitCode 0, "hello from Alpine 3.20.x"

The first run pulls alpine:3.20; later runs reuse it. The plugin repository's vski/ directory has more examples (make iac-apply applies them to the local stack).

Writing container steps

A step names an image (or a Dockerfile) and a command, plus optional limits, environment and files. Scripts are files of their own, committed next to the IaC that uses them:

# vski/pipelines.yaml
- id: scrape
  type: container
  timeout: 600              # the step budget, and the container's lifetime
  params:
    image: python:3.12-slim
    inputs: file://scripts/scrape.py     # copied in as scrape.py
    command: python scrape.py
    env:
      API_TOKEN: "{{secrets.api_token}}"   # resolved by the server, never stored in the definition
    network: true           # needs allow_network on the instance
    memory: 256m
    cpus: 1
# vski/scripts/scrape.py
import urllib.request

print(urllib.request.urlopen("https://example.com").status)

A string command runs through /bin/sh -c; a list runs as is. The step fails on a nonzero exit, a timeout, a stop or a failed file transfer, with the output tail in its error. {{steps.scrape.output.output}} is the output tail and {{steps.scrape.output.exitCode}} the exit code. Unknown parameters are rejected, which is how volumes, devices, privileged mode and host networking stay out. The README lists every parameter and the full output shape.

Where files come from

inputs, the build context files and dockerfile all take files from the same sources:

Source Written as
A file committed with your IaC file://scripts/report.py, relative to the YAML file
An uploaded file /files/<collection>/<recordId>/<filename>, or a reference {collection, recordId, filename}: a record's file field, a trigger's record, an earlier step's artifact
A URL https://…, when the instance sets allow_external_inputs
Inline text the string itself, in files and dockerfile only
  • file:// references are resolved by vski apply, which embeds each file into the pipeline definition as a data: URL (at most 256 KiB per file). The file reaches the container byte for byte: {{…}} inside a script stays as written, so values reach it through env. A definition created any other way cannot use file://: upload the file and reference its /files/… path instead.
  • Inline text is a pipeline string: templates expand in it, and stray {{ or }} (a JSON literal, a Jinja template) fail validation. Keep it to a few lines of shell.
  • Large files (datasets, models, archives) go through the files API, not into the definition.
  • In files and dockerfile, a string is inline text unless it is a file:// reference; write an upload or a URL as {from: <source>}:
- id: convert
  type: container
  params:
    dockerfile: file://images/convert/Dockerfile
    files:                                   # the build context
      convert.py: file://images/convert/convert.py
      requirements.txt: {from: /files/build_files/k3j2…/requirements.txt}
      run.sh: |                              # a few lines inline are fine
        #!/bin/sh
        exec python /app/convert.py "$@"

Files in and out

Files never travel through the pipeline itself: steps pass small references to files stored by the files API (see Pipelines → Files between steps).

- id: merge
  type: container
  dependsOn: [fetch]
  params:
    image: alpine:3.20
    workdir: /work
    inputs: "{{steps.fetch.output.artifacts}}"   # each file at its name, e.g. out/a.csv
    command: mkdir -p result && cat out/*.csv > result/merged.csv
    artifacts: result/merged.csv                 # uploaded after exit 0
  • inputs takes any of the sources above, or {from, dest, optional}; from may be a whole list. Files are fetched (size and checksum verified when the reference carries them) and copied into the container before it starts. An upload that is not stored yet is retried for 30 seconds, so pipelines can trigger on a record's create event.

  • artifacts takes paths or globs (out/*.json, out/**, a whole directory), relative to the workdir. After a successful exit the matching files are copied out and uploaded, one record per file, to the pipeline_artifacts collection (created on first use), and steps.<id>.output.artifacts lists them:

    { "kind": "file", "name": "result/merged.csv", "filename": "merged.csv",
      "size": 1234, "sha256": "…", "collection": "pipeline_artifacts",
      "recordId": "…", "path": "/files/pipeline_artifacts/…/merged.csv", … }
    

    Download any artifact with GET <server><path>.

Example: reports from uploaded CSV files

Every CSV uploaded to a datasets collection becomes an HTML report. The report script is a file next to the IaC that declares the collection and the pipeline:

vski/
├── default/collections.yaml
├── pipelines.yaml
└── scripts/report.py
# vski/default/collections.yaml
version: 1
database: default

collections:
  datasets:
    type: base
    fields:
      - {name: title, type: text}
      - {name: data, type: file, required: true}
# vski/pipelines.yaml
version: 1

pipelines:
  dataset-report:
    enabled: true
    triggers:
      - {type: collection, collection: datasets, actions: [create]}
    steps:
      - id: render
        type: container
        timeout: 120
        params:
          image: python:3.12-alpine
          workdir: /work
          inputs:
            - file://scripts/report.py           # lands at report.py
            - from:                              # the uploaded CSV
                collection: "{{trigger.collection}}"
                recordId: "{{trigger.id}}"
                filename: "{{trigger.record.data}}"
              dest: in/data.csv
          env:
            TITLE: "{{trigger.record.title}}"    # empty when the record has no title
          command: python report.py
          artifacts: out/report.html
          memory: 128m
    output:
      report: "{{steps.render.output.artifacts[0].path}}"
# vski/scripts/report.py
"""Render in/data.csv as an HTML table in out/report.html."""

import csv
import html
import os
import pathlib

with open("in/data.csv", newline="", encoding="utf-8") as handle:
    rows = list(csv.reader(handle))
if not rows:
    raise SystemExit("in/data.csv is empty")
head, body = rows[0], rows[1:]


def cell(tag: str, value: str) -> str:
    return f"<{tag}>{html.escape(value)}</{tag}>"


table = "".join(
    "<tr>" + "".join(cell(tag, value) for value in row) + "</tr>"
    for tag, row in [("th", head)] + [("td", row) for row in body]
)
title = html.escape(os.environ.get("TITLE") or "report")
pathlib.Path("out").mkdir(exist_ok=True)
pathlib.Path("out/report.html").write_text(
    f"<h1>{title}: {len(body)} rows</h1><table>{table}</table>\n", encoding="utf-8"
)
print(f"rendered {len(body)} rows")

vski apply creates the collection and the pipeline, with the script embedded in it; editing report.py and applying again updates the pipeline. Upload a file, here with the Python SDK:

with open("q3.csv", "rb") as f:
    await client.collection("datasets").create(
        {"title": "Q3"}, files={"data": ("q3.csv", f, "text/csv")}
    )

The upload starts a run of dataset-report. When it completes, its output holds the report's path, e.g. /files/pipeline_artifacts/3e11…/report.html; open it on your server. The report's record in pipeline_artifacts carries the run and step ids, so every report can be traced back to the run that produced it. The plugin repository ships this example as container-dataset-report.

Configuration

The instance config caps what steps can request (requests above a maximum are clamped, with a warning in the step output):

Key Default Meaning
max_concurrent 2 Containers (and builds) running at once; further jobs queue
default_memory / max_memory 256m / 1g Memory limit when a step sets none / the hard cap
default_cpus / max_cpus 1.0 / 2.0 CPU limit default / cap
default_timeout / max_timeout 600 / 3600 Container lifetime in seconds, default / cap
allow_network false Steps may opt into networking (network: true)
allow_build true Steps may build images (dockerfile, or image + files)
artifacts_collection pipeline_artifacts Collection that stores artifacts
max_input_size / max_artifact_size 512m / 1g Total size of the files a step fetches (inputs and build files) / of its artifacts
artifact_retention_hours 720 Age at which artifacts are deleted; 0 keeps them
allow_external_inputs false Allow http(s) URLs as inputs or build files (fetched by the plugin process)

The plugin README documents the rest (pids limits, pull policy, log and job retention, transfer timeouts).

Operations

  • Logs. A step's output is its stdout/stderr tail. The plugin function container.logs returns the full log (it also stays on the data volume under container-logs/), and the plugin's own log is on the dashboard's Plugins page.
  • Functions. container.status, container.stop, container.logs, container.artifacts and container.validate (a dry run that validates step params) can be invoked from the dashboard or with POST /api/plugin-instances/<instance>/invoke.
  • Stats. The Plugins page charts jobs.running, jobs.queued, jobs.collecting, containers.cpu_pct and containers.mem_mb, plus per-container CPU and memory, sampled every 10 seconds.
  • Restarts. Restarting the plugin, the server or the whole container never loses a job: running containers are re-attached, and runs parked on a container step resume when it exits.

Security

  • The Docker socket is root-equivalent on the host. Anyone who can create pipelines (admins and workflows-role principals) can run any image through the host's Docker daemon. Use the plugin for trusted pipelines (personal, lab, internal automation), never for untrusted tenants.
  • Containers run without network by default (the step and the instance must both opt in), with all capabilities dropped, no-new-privileges, and memory, CPU, process and time limits. Volumes, devices, privileged mode and host networking are rejected; files move through Docker's archive API, never through mounts.
  • URL sources are off by default: the plugin process fetches them, so enabling them lets pipeline authors reach anything the server can reach.
  • file:// references are read by the vski CLI on the machine that applies the IaC, and only inside the IaC directory (symbolic links included), so a pipeline file cannot pull in ~/.ssh or a CI runner's secrets.
  • Files, artifacts included, are served by a files route that does not check credentials: anyone who can reach the server and knows a file's path can download it. Keep secrets out of artifacts.

Limitations

  • Linux hosts only.
  • Artifacts are collected only from containers that exit 0; a failed step keeps its full log, not its files.
  • readOnly containers cannot take inputs.
  • Files embedded with file:// are limited to 256 KiB each and 1 MiB per pipeline: they live in the definition and in every run's copy of it.
  • Completion is reported at least once: a crash at the wrong moment can deliver a duplicate completion signal, which arrives after the step has completed and sits unused.

See also