- Python 86%
- Makefile 14%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| plugins | ||
| scripts | ||
| seeds | ||
| test/unit | ||
| vski/default | ||
| .env.example | ||
| .gitignore | ||
| Makefile | ||
| pytest.ini | ||
| README.md | ||
template-agent
A watcher agent with a graph memory, wired together entirely from platform primitives. The agent ("scout") keeps a watchlist of URLs, and every sweep it checks each one, remembers what it saw, and reports only what is new.
every 5 min (cron trigger)
|
sources -----> [ pipeline: scout-scan ] load watchlist, fan out
watchlist |
v one child run per source
[ pipeline: scout-probe ]
fetch (http step) -> observe (plugin step)
|
graph "memory" <------------------------+ fingerprint seen before?
source/state nodes, |
observed/next edges | new only
v
findings collection
| create trigger
v
notifier plugin -> logs/findings.log
(stands in for Slack/email/webhooks)
Nothing in the loop is bespoke glue: the schedule is a pipeline trigger,
the fan-out is a foreach step, the fetching is a plain http step, the
brain is a plugin function, the memory is a graph, the output is a
collection, and the side effect is a collection trigger into a second
plugin.
Layout
vski/ declarative IaC (make iac-apply)
default/collections.yaml sources + findings (with the trigger)
default/graphs.yaml the memory graph and its rules
default/pipelines.yaml scout-scan + scout-probe
default/plugins.yaml scout + notifier instances
plugins/
scout/ the brain: watchlist + observe functions
notifier/ the hands: findings -> log file
seeds/dev.yaml sample watchlist
scripts/memory.py prints the memory graph (make memory)
test/unit/ pure-logic tests (make test)
logs/findings.log notifier output (gitignored)
Quick start
You need a server the template can point at, running with this template's plugins enabled (a plain server binary; the plugin env vars are the part that matters):
PLUGINS_ENABLED=true \
PLUGINS_DIR=$(pwd)/plugins \
PLUGIN_PYTHON=$(pwd)/.venv/bin/python \
/path/to/vski serve --port 3000
PLUGIN_PYTHON must have the vski SDK installed; the template's .venv
(created below) does, so it can serve both the CLI and the plugins.
Then, from this directory:
make venv # CLI + SDK (also the PLUGIN_PYTHON above)
cp .env.example .env # set VSKI_URL + VSKI_API_KEY (admin key)
vski keys create agent-dev --roles admin # on the server, paste into .env
make iac-apply # collections, graph, pipelines, plugin instances
make seed # the sample watchlist
make scan # one sweep now (cron also fires every 5 min)
Inspect what the agent did:
make findings # what it published
make memory # what it remembers, per source
make log # what the notifier wrote
make runs # pipeline runs and their step outputs
Run make scan again: no new findings. The agent remembers the states it
has already reported, and only a genuinely new state (the URL's selector
value changed, or its payload hash moved) publishes again.
How one sweep works
| Piece | What it does |
|---|---|
scout-scan trigger (cron) |
fires the sweep every 5 minutes |
scout-scan trigger (collection) |
adding a source fires a sweep immediately |
load (plugin step, scout/sources) |
reads the active watchlist |
probe (foreach step) |
spawns one scout-probe run per source |
fetch (http step) |
GETs the source URL, 10s timeout, one retry |
observe (plugin step, scout/observe) |
fingerprints the response, reconciles with the graph, publishes if new |
| findings trigger | hands each created finding to plugin://notifier/on_finding |
notifier/on_finding |
appends the finding to logs/findings.log |
A dead source does not fail the sweep: fetch continues on error and
observe is skipped for it via its condition, so the run completes with
the healthy sources intact.
The memory graph
One source node per watched URL. One state node per distinct
observation, keyed by the fingerprint (the selector value, or a payload
hash). observed edges connect a source to every state it has been seen
in; next edges chain a source's states in discovery order, so the chain
out of a source is its change history. All ids are deterministic, so
repeated scans upsert instead of duplicating.
Memory is private to the agent: every rule on the graph (and on
sources/findings) admits only the plugin role, and superusers.
Making it yours
- Add a source: append to
seeds/dev.yamlandmake seed, or create a record insourcesdirectly. Both fire an immediate scan. - Watch something real: any JSON URL with a stable field works; try a
releases feed with
selector: tag_name. - Actually notify someone: replace the body of
on_findinginplugins/notifierwith your real request; the log line is the seam. - Run one probe by hand:
vski pipelines run scout-probe --input '{"source": {"id": "manual", "name": "manual", "url": "https://example.com", "selector": ""}}'.
Testing
make test # unit tests for the fingerprint/slug logic
Production notes
- The server needs
PLUGINS_ENABLED=true,PLUGINS_DIRpointing at this template'splugins/, and aPLUGIN_PYTHONwith the SDK installed. - The notifier resolves its log path from instance config
(
vski/default/plugins.yaml): relative paths land next to the project that ownsplugins/; absolute paths (or a mounted volume in a container) for anything durable. - The 5-minute schedule lives in
vski/default/pipelines.yaml; tighten or loosen it there and re-apply.