No description
  • Python 86%
  • Makefile 14%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-03 03:22:07 +01:00
plugins init 2026-11-03 03:22:07 +01:00
scripts init 2026-11-03 03:22:07 +01:00
seeds init 2026-11-03 03:22:07 +01:00
test/unit init 2026-11-03 03:22:07 +01:00
vski/default init 2026-11-03 03:22:07 +01:00
.env.example init 2026-11-03 03:22:07 +01:00
.gitignore init 2026-11-03 03:22:07 +01:00
Makefile init 2026-11-03 03:22:07 +01:00
pytest.ini init 2026-11-03 03:22:07 +01:00
README.md init 2026-11-03 03:22:07 +01:00

template-agent

A watcher agent with a graph memory, wired together entirely from platform primitives. The agent ("scout") keeps a watchlist of URLs, and every sweep it checks each one, remembers what it saw, and reports only what is new.

                 every 5 min (cron trigger)
                          |
   sources -----> [ pipeline: scout-scan ] load watchlist, fan out
   watchlist             |
                         v   one child run per source
                 [ pipeline: scout-probe ]
                   fetch (http step) -> observe (plugin step)
                                           |
   graph "memory" <------------------------+   fingerprint seen before?
   source/state nodes,                            |
   observed/next edges                           | new only
                                                 v
                                        findings collection
                                                 |  create trigger
                                                 v
                                      notifier plugin -> logs/findings.log
                                      (stands in for Slack/email/webhooks)

Nothing in the loop is bespoke glue: the schedule is a pipeline trigger, the fan-out is a foreach step, the fetching is a plain http step, the brain is a plugin function, the memory is a graph, the output is a collection, and the side effect is a collection trigger into a second plugin.

Layout

vski/                    declarative IaC (make iac-apply)
  default/collections.yaml   sources + findings (with the trigger)
  default/graphs.yaml        the memory graph and its rules
  default/pipelines.yaml     scout-scan + scout-probe
  default/plugins.yaml       scout + notifier instances
plugins/
  scout/                  the brain: watchlist + observe functions
  notifier/               the hands: findings -> log file
seeds/dev.yaml            sample watchlist
scripts/memory.py        prints the memory graph (make memory)
test/unit/               pure-logic tests (make test)
logs/findings.log        notifier output (gitignored)

Quick start

You need a server the template can point at, running with this template's plugins enabled (a plain server binary; the plugin env vars are the part that matters):

PLUGINS_ENABLED=true \
PLUGINS_DIR=$(pwd)/plugins \
PLUGIN_PYTHON=$(pwd)/.venv/bin/python \
/path/to/vski serve --port 3000

PLUGIN_PYTHON must have the vski SDK installed; the template's .venv (created below) does, so it can serve both the CLI and the plugins.

Then, from this directory:

make venv                    # CLI + SDK (also the PLUGIN_PYTHON above)
cp .env.example .env         # set VSKI_URL + VSKI_API_KEY (admin key)
vski keys create agent-dev --roles admin    # on the server, paste into .env

make iac-apply               # collections, graph, pipelines, plugin instances
make seed                    # the sample watchlist
make scan                    # one sweep now (cron also fires every 5 min)

Inspect what the agent did:

make findings                # what it published
make memory                  # what it remembers, per source
make log                     # what the notifier wrote
make runs                    # pipeline runs and their step outputs

Run make scan again: no new findings. The agent remembers the states it has already reported, and only a genuinely new state (the URL's selector value changed, or its payload hash moved) publishes again.

How one sweep works

Piece What it does
scout-scan trigger (cron) fires the sweep every 5 minutes
scout-scan trigger (collection) adding a source fires a sweep immediately
load (plugin step, scout/sources) reads the active watchlist
probe (foreach step) spawns one scout-probe run per source
fetch (http step) GETs the source URL, 10s timeout, one retry
observe (plugin step, scout/observe) fingerprints the response, reconciles with the graph, publishes if new
findings trigger hands each created finding to plugin://notifier/on_finding
notifier/on_finding appends the finding to logs/findings.log

A dead source does not fail the sweep: fetch continues on error and observe is skipped for it via its condition, so the run completes with the healthy sources intact.

The memory graph

One source node per watched URL. One state node per distinct observation, keyed by the fingerprint (the selector value, or a payload hash). observed edges connect a source to every state it has been seen in; next edges chain a source's states in discovery order, so the chain out of a source is its change history. All ids are deterministic, so repeated scans upsert instead of duplicating.

Memory is private to the agent: every rule on the graph (and on sources/findings) admits only the plugin role, and superusers.

Making it yours

  • Add a source: append to seeds/dev.yaml and make seed, or create a record in sources directly. Both fire an immediate scan.
  • Watch something real: any JSON URL with a stable field works; try a releases feed with selector: tag_name.
  • Actually notify someone: replace the body of on_finding in plugins/notifier with your real request; the log line is the seam.
  • Run one probe by hand: vski pipelines run scout-probe --input '{"source": {"id": "manual", "name": "manual", "url": "https://example.com", "selector": ""}}'.

Testing

make test    # unit tests for the fingerprint/slug logic

Production notes

  • The server needs PLUGINS_ENABLED=true, PLUGINS_DIR pointing at this template's plugins/, and a PLUGIN_PYTHON with the SDK installed.
  • The notifier resolves its log path from instance config (vski/default/plugins.yaml): relative paths land next to the project that owns plugins/; absolute paths (or a mounted volume in a container) for anything durable.
  • The 5-minute schedule lives in vski/default/pipelines.yaml; tighten or loosen it there and re-apply.