Active Perception — pretrained GRU DRQN (LiteRT.js)

Read the essay: https://blog.vski.ai/posts/active-perception/
Active Perception: A GRU DRQN for Discovery Under a Noisy Sensor

---

A single-page web app: an agent moves on a 20×20 grid it can only sense noisily, along its trajectory. The sensor extends further in the last-move direction than to the sides — the defining affordance of the active-perception regime. The agent collects value-graded gold (some gold is worth more than other gold) and learns tradeoffs against graded stone costs. The policy is a pretrained GRU DRQN loaded as a .tflite and run via LiteRT.js — pure in-browser inference, no training, no deadly triad.

What it does

Each tick:

1. Build a 33-dim observation: for each of the 8 candidate single-cell moves, what does the agent believe about the destination (bGold, bStone, can_move, const)? Plus the previous reward.
2. Concatenate the previous action one-hot (8-d) → 41-d input.
3. Run the GRU: [q, h_out] = model.run(h_in, x) via LiteRT.js (state is explicit; the browser threads h_out → h_in between ticks).
4. Mask unwalkable actions (can_move == 0), pick argmax(q).
5. Step the env. Gold collects value; stone pays a graded cost; bumps pay a small penalty; the potential-based shaping bonus Φ(s) = β·coverage(s) rewards exploration without creating a degenerate optimum.

The agent has no learning loop in the browser. The model is a frozen artifact trained in Python (see train/_train_active_perception.py) at 2.66× random on a 20×20 clustered-world regime (1400 episodes, Double DQN + episode replay + target network + BPTT-truncated gradient).

Validated: 3.35× vs random in the browser

npm test runs the smoke test in headless Chromium via Playwright. It loads gru.tflite, runs 5 trained + 5 random episodes on fresh layouts, and asserts trained ≥ 1.2× random. Current result: 3.35× mean — and that number includes the LoopGate described below (without the gate it was 2.70×).

$ npm test
[smoketest] status=PASS  ratio=3.35
MEAN trained=9.41  random=2.81  ratio=3.35x

The LoopGate — diagnose-then-mask for oscillation basins

The trained policy has one visible failure mode: it sometimes gets stuck in a 4×4 / 4×6 basin, bouncing off a wall, a stone, or a believed stone. Telemetry on 30 baseline episodes showed ~29% of all ticks were spent inside such a basin (4×4 box, 15+ ticks, no gold), median basin length 19 ticks, worst 72.

The fix uses a diagnose-then-mask recipe — same discipline as a bandit-gate uses to disable long leaps in cluster states, but tuned for a different failure class. The LoopGate here masks looping directions when the detector fires. Three pieces:

1. Detector (closed-form online regression, no learned weights). A rolling 10-tick window; two OLS slopes σ_r, σ_c give net displacement per tick (speed); positive-reward sum gives progress (gold_W). The loop condition is speed < 0.25 AND gold_W < 0.10 — low displacement and no gold. The gold_W term is the discriminator: a slow drift through a cluster the agent is sweeping is not a loop.
2. Persistent mask. When the gate fires, every direction appearing ≥ 4 times in the window is masked. Unlike a fixed cooldown, the mask stays on until the agent has demonstrably escaped (4 consecutive ticks of speed ≥ 0.25, or any gold event). This is critical: a fixed cooldown lets the agent escape, expire, and get pulled straight back by the still-misleading shadow belief.
3. Recency tie-break. While the mask is active, the agent picks the action whose destination cell was visited least recently, not argmax-of-rest. Argmax-of-rest was tried first: masking one direction just made the agent pick the opposite one and oscillate in a new pattern (net ~0.99×). The recency tie-break cannot oscillate by construction (a 2-cycle visits two cells; recency sends it to a third).

Validation (matched-pair, 30 layouts × 8 base seeds):

              s0    s100   s500  s1000  s5000  s9999 s12345 s31337  mean
loop_gate  1.188× 1.025× 0.963× 1.067× 1.055× 0.959× 1.054× 1.023× 1.042×

Mean +4.2% gold collection; best +18.8%, worst −4.1%. This is a targeted intervention: it helps a lot when the agent is genuinely stuck, and can hurt a little when the agent is already doing well. The net is positive on average. The probe (npm test) happens to land on layouts where the agent was very stuck, so the smoke-test ratio jumps from 2.70× to 3.35× with the gate on.

The gate has a panel toggle — flip it off at runtime to compare the plain policy against the gated one on the same layout. The HUD shows current status (idle / ESCAPING), per-episode fire count, the masked directions, and the live speed / gold_W detector inputs. The Q-bars in the panel dim the bars for masked actions so you can see exactly what the gate is switching off.

The second mask — basin-cell taboo (variance reducer, off by default)

A natural follow-up question: when the agent escapes a basin, does it ever come back? Telemetry says yes — 42% of escaped basins get re-entered, spending a median of 3 ticks (max 18) back in the basin, with 0% of those return visits collecting any gold. The basin is empty; the agent already swept it.

The fix is a second mask with the same diagnose-then-mask shape: when the gate fires, every cell in a 5×5 box around the basin cells goes on a taboo map at strength 1.0; the strength decays by 5% per tick (half-life ~14 ticks); any candidate action whose destination has strength > 0.5 is masked.

This mask is a variance reducer, not an improvement. Across 10 base seeds:

                     s0    s100   s500  s1000  s5000  s9999 s12345 s31337  s4242 s8675309  mean  worst
gate only         1.188× 1.025× 0.963× 1.067× 1.055× 0.959× 1.054× 1.023× 0.945×  1.024×  1.030× 0.945×
gate + 5×5 taboo  1.122× 0.988× 1.010× 1.106× 1.032× 1.000× 1.015× 1.032× 1.041×  1.010×  1.036× 0.988×

It lifts the floor (worst-case 0.945× → 0.988×, eliminating the regressions on adversarial layouts where the gate alone overshot) at the cost of capping the peak (best-case 1.188× → 1.122× on layouts where the gate alone was already doing great). The standard deviation across seeds drops from 0.073 to 0.050 — a third less variance.

The taboo mask ships off by default (the smoke-test headline is the peak) with a panel toggle. Flip it on for smoother demo behavior on adversarial layouts; flip it off for the headline number.

Prototype and sweep: train/_proto_loop_gate.py. Hyperparameter sweep script: train/_loop_gate_sweep.py. Port to TypeScript: src/loop_gate.ts.

Quick start

npm install
npm run dev      # vite dev server on :5176
npm test         # headless Playwright smoke test (~30s)
npm run build    # production bundle (54 KB JS / 18 KB gz + 57 KB model)

Open the dev URL, hit Start, watch the agent collect. The Run 30 episodes button runs the head-to-head comparator on the current layout and displays the trained-vs-random ratio.

The model

The shipped artifact is public/gru.tflite (56.9 KB, float32). Its I/O contract:

name	shape	role
h_in	[1, 48]	previous GRU hidden state (zeros on step 0)
x	[1, 41]	prev_action one-hot (8) + observation (33)
q	[1, 8]	Q-values for the 8 candidate actions
h_out	[1, 48]	next hidden state

The GRU is an explicit-state cell — h_in is an input, h_out is an output, the browser threads the state manually. This dodges keras.layers.GRU(stateful=True) TFLite conversion bugs (tensorflow/tensorflow#97941) and makes the Python↔JS parity check trivial (max|Δ|=3.6e-7 between the Keras model and the .tflite running in LiteRT.js).

Why this design (vs the vacuum-demo)

The sibling vacuum-demo ships the FQI agent's training loop in the browser — closed-form ridge regression on a replay buffer is cheap enough to do live, and the pedagogical payload there is "watch Q converge round by round."

For the active-perception DRQN that's the wrong tradeoff. The whole point of the validated prototype was the multi-episode BPTT-with-replay training procedure. Replaying it live means either shipping a tiny JS GRU trainer that takes minutes per run, or running TF.js online-TD — precisely the deadly-triad recipe the prototype had to escape.

Instead: pretrain in Python (where the validated recipe lives), export the GRU as a .tflite, load it in the browser via LiteRT.js, run pure inference. The user gets a fast, snappy demo with a visibly directional sensor (the regime's defining property) and a side-by-side "trained vs random" comparator — without waiting for training and without the deadly triad.

The full ML story (the three recurrent cells, the BPTT truncation, the potential-based shaping derivation, the linear-FQI baseline that proved the regime was learnable, the diagnosis of why the deadly triad killed the first attempt) lives in the notebook scratchbooks/06_Active_Perception.ipynb. The web demo is the artifact the trained model ships in.

Regenerating the model

The model is the output of train/_train_active_perception.py. The full pipeline (with all commands and validation gates) is documented in train/README.md. Quick reference, from the repo root with .venv active:

# Phase 1: prove the Keras model == the validated prototype (max|Δ| ~3e-7).
.venv/bin/python scratchbooks/apps/active-perception/train/_train_active_perception.py \
    --parity-check scratchbooks/apps/active-perception/train/_gru_g20_s0.npz

# Phase 2: train a fresh GRU at 20×20 (~2 min on CPU). Writes
# train/_gru_g20_s0.{npz,json}.
.venv/bin/python scratchbooks/apps/active-perception/train/_train_active_perception.py \
    --train-grid 20 --seed 0

# Phase 2c: convert to .tflite + verify TFLite↔Keras parity, writing the
# artifact straight into the demo's public/ dir.
.venv/bin/python scratchbooks/apps/active-perception/train/_train_active_perception.py \
    --export-tflite scratchbooks/apps/active-perception/train/_gru_g20_s0.npz \
    --tflite-out scratchbooks/apps/active-perception/public/gru.tflite

# Refresh the metadata sidecar.
cp scratchbooks/apps/active-perception/train/_gru_g20_s0.json \
   scratchbooks/apps/active-perception/public/gru.meta.json

References

• The validated prototype: train/_proto_active_perception.py
• The trainer + TFLite exporter: train/_train_active_perception.py
• The loop-gate prototype: train/_proto_loop_gate.py
• The notebook that captured the validated story: scratchbooks/06_Active_Perception.ipynb
• The plan: plans/active_perception_web_demo.md
• The sibling web app: scratchbooks/apps/vacuum-demo/
• LiteRT for Web: https://developers.google.com/edge/litert/web
• @litertjs/core on npm: https://www.npmjs.com/package/@litertjs/core
• Active perception in POMDPs: Spaan & de Vries 2008
• DRQN: Hausknecht & Stone 2015
• Potential-based reward shaping: Ng, Harada & Russell 1999
• Double DQN: Hasselt, Guez & Silver 2016