1 KWH
Anton Nesterov edited this page 2026-11-02 06:57:03 +01:00

Knowledge, Workflow, Human In The Loop

Classic software architecture separates concerns into layers with defined contracts between them: MVC splits an application into model, view and controller, and network stacks separate transport from session from application. Agentic systems deserve the same discipline. The literature proposes several layerings, usually along planning, retrieval, tool use and memory, with definitions that vary from paper to paper. We prefer to separate three abstract layers: (K) Knowledge, (W) Workflow and (H) Human In The Loop. Every agent carries all three, from a one-shot script to a multi-agent fleet. The question is never whether the layers exist, only whether each one was designed or left accidental.

The layers are a decomposition for building and operating agents, not a taxonomy for arguing about them. Each layer owns one kind of failure, is evaluated on its own scorecard, and maps to concrete platform primitives, so the architecture lands as configuration rather than as a diagram.

Layer Owns The question it answers Signature failure
K, Knowledge everything the agent knows: retrieval, analysis, synthesis Is it true, and can a human verify it? confident wrong facts
W, Workflow everything the agent does: procedures, tools, budgets Is it the right action, within limits, on the record? wrong or runaway actions
H, Human In The Loop everything that connects the system to accountable people: gates, escalation, review Can a human see it, understand it, and stop it? rubber-stamped gates, surprised humans

The K layer: Knowledge

The K layer is everything the agent knows: knowledge retrieval, analysis, synthesis, reporting. The agent pulls information and produces a structured output: findings, briefs, bundles. That output is a contract, not prose: named fields, evidence behind every claim, a confidence number, sources a reader can open.

Four properties define a healthy K layer:

  1. Retrieval correctness. The right rows and documents, not the nearest ones.
  2. Scope discipline. The question that was asked, not the neighborhood of it.
  3. Evidence. Every claim traceable to something a human can open.
  4. Output contract. Same fields, same shape, every time. Drift is a defect, not a style change.

Knowledge work looks harmless, which is why it rots first: a report with a wrong number in it is not a writing bug, it is a missing eval. The practice is to evaluate the whole trajectory (what was pulled, in what order), each step, and the final output against a golden dataset, and to turn every production failure into a new test case.

K on the platform

K concern Platform primitive Why it holds
Output contract Collections with typed, required fields the schema is the contract; validation is server-side, so a malformed brief cannot be written at all
Scope discipline Rules list and view rules are SQL expressions applied as query filters; the caller never retrieves what the rule excludes, so scope is enforced at storage, not in the prompt
Keyword retrieval Full-Text Search (FTS5) automatic on text collections; no separate search service
Semantic retrieval Vector Embeddings (sqlite-vec) vectors live beside the records in the same database; hybrid keyword plus vector queries in one path
Multi-hop evidence Graphs typed edges between records; traversal answers relationship questions without prompt-side reasoning
Shaping and joins Views SQL views expose exactly the slice each consumer needs, computed where the data lives
Probe sets collections again golden datasets and failure-derived test cases are just records; evals query them like any other corpus

The W layer: Workflow

The W layer is where the agent stops describing and starts acting. Every actionable item the system can perform is a named procedure with preconditions, parameters, guards, a budget and a journal entry. If the system can restart a service, that is a W procedure. If it can spend money, that is a W procedure with a gate in front of it. The procedures are the substance of the agent; everything else feeds them or audits them.

Four properties define a healthy W layer:

  1. Least-privilege tool allowlists. The agent gets exactly the tools its procedures need, enforced at the server layer, never in the prompt. Every extra tool widens what a prompt injection can reach.
  2. Budgets. Hard ceilings on retries, timeouts, concurrency and wall clock, per step and per pipeline. A loop that cannot stop is an outage with a progress bar.
  3. Scope refusals. Anything outside declared scope is refused by rule, and the refusal is an event somebody sees.
  4. Degradation. An autonomous action that cannot proceed safely degrades to a human decision instead of failing silently or acting outside its scope. Degrading is a designed outcome, not a failure.

W on the platform

W concern Platform primitive Notes
Durable procedures Workflows steps are journaled server-side; a crash or deploy resumes the run where it stopped, instead of replaying side effects
Reactions to data Pipelines collection triggers (create, update, delete) start runs the moment a record lands
Time-driven procedures pipeline cron triggers and the Cron table schedules are part of the pipeline definition; cron jobs can also kick pipelines ad hoc with pipeline://
External entry points public trigger tokens unauthenticated POST /api/pipelines/trigger/:token, rate limited, revocable; how CI and third parties start runs
Tool allowlist Plugins plugin steps can only call declared plugin instances over the in-process pipe; there is no tool surface beyond what is declared
Budgets per-step and per-pipeline options step: retries, timeout, condition, onError; pipeline: concurrency, maxEvents, executionTimeout
Composition step graph dependsOn ordering, foreach fanout, sub-pipelines, and workflow steps with wait: true
Credentials secrets (see Pipelines) encrypted at rest; values arrive from environment variables at apply time and never live in YAML

The H layer: Human In The Loop

The H layer is the interface to the only component that can be held accountable: people. Human review, triage, escalation and ambient triggers live here, and so do the properties that make the other two layers safe to run in production.

Four properties define a healthy H layer:

  1. Gates in front of consequential actions. What the approver sees is the K layer's brief plus the W layer's blast radius, not a button with a label.
  2. Silence is not approval. A timeout escalates. "Nobody answered" is a state transition with an owner, never a default yes.
  3. Outputs shaped for people. Readable reports with confidence and evidence, not JSON dumps.
  4. Refusals explain themselves. "I will not roll back the database, because the evidence points at the deploy, not the schema" is an acceptable output; a bare error is not.

H on the platform

H concern Platform primitive Notes
Approval gates workflow signals ctx.wait_for_signal("manager-approval") suspends the run durably; client.workflow.send_signal(run_id, ...) resumes it with the decision
One-click review endpoints workflow hooks token-gated; executing the hook delivers its payload to the run as the signal, with the hook's metadata merged in
Escalation timers ctx.sleep in the workflow, or a cron sweep expiry is journaled data, so "ten minutes of silence" can escalate to a lead deterministically
Hard gates on data Rules a missing rule makes an operation admin-only; granular rules decide which fields a reviewer may touch
Triage surface Realtime subscriptions and the web UI gate requests, refusals and reports are records; subscribe to them like any other collection
Audit run journal and workflow events who approved what, when, on which evidence; queryable after the fact

One incident through the layers

A deliberately small example: one store, one on-call engineer, one agent on a local model. Checkout errors cross 4.1% against a 1% floor at 03:07. The run below touches every layer, and every step leaves an artifact the next step reads.

# Step Layer On the platform Artifact
1 Ambient trigger: error rate crosses the floor H a cron-triggered pipeline compares the rate against the threshold and writes an alert alert record
2 Evidence run: recent deploys, error logs, runbook, dependency status K full-text and vector queries over the deploy, log and runbook collections evidence records
3 Incident brief: what changed, what broke, candidates, each with evidence and confidence K written to a briefs collection whose schema is the output contract brief record
4 Procedure select: restart the checkout service, budget one restart W a workflow step picks from its allowlist; the non-consequential branch runs journal entry
5 The gate, had rollback been chosen H the run suspends on an approval signal; a hook lets the on-call approve from chat; a ten minute timer escalates to the lead signal payload
6 Execute and journal W the restart runs inside the allowlist, budget spent one of one, every step journaled run events
7 Postmortem bundle: what happened, what it cost K one new eval case written to the probe set eval record
8 Readable report: confidence, evidence, triage notes H a report record pushed to the on-call over realtime report record

Note the refusal inside step 4: a database rollback is declined because the evidence points at the deploy, not the schema. That refusal is W layer work doing H layer work in advance: it explains itself before any human had to ask. No step skipped a layer, and no step needed a diagram to see it.

Debugging by layer

The layers give you the fault domain, which is most of debugging.

Symptom Layer First places to look
Wrong or missing fact K the retrieval query, rule scope, probe set coverage for this case
Wrong action, blown budget, silent failure W the run journal, step conditions, retries and timeouts
Human surprised, rubber-stamped gate, ignored report H what the gate displayed, escalation timers, where reports land
Looks like two layers at once the contract between them W reads fields K stopped writing; W journaled evidence the gate never displayed

Three scorecards

Never one score. Each layer gets its own evaluation, run in CI on every change to any layer.

Layer What to measure Where it lives
K retrieval correctness on the probe set, citation coverage, contract conformance probe sets are collections; the eval is a query
W dry-run trajectories against golden sequences, budget conformance, injection red team the run journal, replayed against a twin database
H triage time (a teammate who has never seen the agent can triage an output in two minutes), approval latency, time-to-degrade timestamps in workflow events and run journals

Every layer's production failures become next week's eval cases. That loop is the difference between an agent that improves and an agent that decays.

Declare the whole agent

All three layers land as Infrastructure as Code and apply with vski plan and vski apply, so the architecture is reviewable as a diff:

File Layer What it declares
vski/<database>/collections.yaml K output contracts, rules, FTS fields, views, indexes
vski/pipelines.yaml W triggers, steps, budgets, conditions, plugin targets
vski/cron.yaml W and H schedules, pipeline kicks, escalation sweeps
vski/secrets.yaml W which environment variables deliver which credentials

The long version of this argument, with sources and the incident in full, is on the blog: The KWH Framework. Next stops in this wiki: Workflows for the durable engine, Pipelines for triggers and steps, and Collections with Rules for the K layer's contracts.