Primes GGUF 0.6B

A state-of-the-art edge decision model. Give it a situation and a set of options; it returns a ranked choice with calibrated confidences — in one embedding pass per option, with no text generation. 639 MB quantized, it runs on a laptop CPU in ~100–250 ms per decision, or ~70 ms on an iGPU.

This is the base model designed for fine-tuning.

Weights

state  = "The warehouse door is locked. Security patrols every 20 minutes.
          A side window is open two floors up. Your lockpick is bent."
query  = "get inside without being seen"          # optional
options= ["pick the lock", "climb to the window", "wait for the patrol", "leave"]

-> climb to the window  (0.51)   wait for the patrol (0.29)   ...

Results (public validation set, 424 decisions)

backend	top-1	MRR	ECE	ms/decision
reference — bf16, canonical head (offline)	0.9458	0.9717	0.0976	–
ollama Q8_0 · CPU	0.9693	0.9837	0.1080	~200
llama.cpp Q8_0 · CPU (docker)	0.9623	0.9801	0.1001	~190 *
llama.cpp Q8_0 · Vulkan iGPU (docker)	0.9646	0.9813	0.1033	~170 * / ~70 short

* measured on a busy shared machine; short = game-state-size contexts (4 options, ~120 tokens). ECE = expected calibration error (10 bins). Decisions are single-choice over 2–8 options; random baseline ≈ 0.28, the 8B teacher ≈ 0.79. Reproduce any row: make validate or python val/validate.py --endpoint …. The serving rows share one head variant (see How it works); the Q8_0 quantized backbone shows no measurable decision drift.

Quickstart A — standard ollama (simplest)

# 1. weights into ./models  (GGUF + head; see "weights" below)
make model

# 2. import into ollama (a local copy; nothing is downloaded from anywhere)
make ollama

# 3. decide
python -c "
from primes import Decide
d = Decide()
print(d.decide('Two job offers: a stable team with legacy code, or a risky startup doing ML.',
               ['stable team', 'risky startup'],
               query='maximize career growth in two years'))"

Works with any ollama that serves /v1/embeddings (tested on 0.33). The GGUF carries qwen3.pooling_type = MEAN (see scripts/patch_pooling.py — without it ollama treats the checkpoint as a chat model and refuses embeddings; the head eats the L2-normalized pooled state, which is exactly what ollama returns).

Quickstart B — server + web playground (docker)

make model && make serve          # CPU        -> http://localhost:8080
make serve-vulkan                 # any GPU    -> ~70 ms short decisions

The web page has three tabs: play — a Pac-Man-style maze the model plays (you can take over with arrow keys), debug — score any decision as JSON, api — the endpoint reference.

In the maze, every move is one call to /score_texts. The context is the situation — goal, then a You see: line per direction (what lies that way and how many moves away, ghost motion included) — and each option text carries its direction's own facts inside the option string, e.g. Option: move_right (pellet in 1 move, Blinky in 4 moves moving away). That is the mechanically sound shape for a scorer: options in one request differ only by their tokens, so the facts must ride with the action. The softmax ranking picks the move (classic pac no-reverse applied to the model's own ranking). The side panel shows every text and score. On an iGPU those decisions run ~70 ms; the model clears ~90% of the board.

Vulkan needs /dev/dri passed through (already in compose.vulkan.yaml); on hosts where the video/render groups have different GIDs, adjust group_add. CPU-only boxes: plain make serve.

No docker at all? make llama-native builds llama-server from source (~3 min), then point the client at it: Decide("http://127.0.0.1:8080") — run it with --embedding --pooling mean -fa on -c 8192 --ubatch-size 8192 -np 1.

Validation

make validate            # against the compose API on :8080
make validate-ollama     # against local ollama
# or directly:
python val/validate.py --endpoint http://127.0.0.1:11434 --model primes-decide

val/val_clean.jsonl holds 424 public decisions (natural-language contexts, 2–8 options, gold labels). The script exits non-zero if top-1 or ECE drift from the published numbers, so it doubles as an install smoke test.

API

POST /score with {"state", "query"?, "options"} → {"results": [{label, confidence} …] (sorted), "choice", "confidence", "query", "latency_ms"}. POST /score_texts (raw): {"texts": […]} → {"logits": […]}. GET /health → upstream + head status. Full field reference: the api tab of the web playground.

Python client (primes.Decide) speaks to any /v1/embeddings server — ollama, llama.cpp, or the compose stack — and normalizes embeddings client-side, so server normalization settings never matter.

Weights

make model stages from a local release store (override MODEL_SRC). The repo ships code + head config; the two weight files are:

file	size	what
models/primes-decide-0.6b-Q8_0-mean.gguf	639 MB	backbone, Q8_0, MEAN pooling stamped
models/head-norm.safetensors	4.2 MB	decision head (normalized-input variant)

How it works / limitations

• Input format: Query: {query}\nOption: {option}\nContext: {context}, whitespace-squeezed; a missing query becomes select best. Contexts are truncated at 704 tokens.
• The shipped head is the embed-server variant: it consumes the pooled state L2-normalized and rescaled by a fixed constant (77.327, the training corpus median norm, in head_config.json). That makes the decision invariant to whatever normalization an embedding server applies — ollama (always unit vectors) and llama.cpp (raw) produce identical outputs. The canonical research artifact also keeps a raw-scale head; both pass the same parity battery (top-1 within 1.5 pt / ECE within 0.03 of the bf16 reference).
• One decision = K forward passes (batched server-side) + softmax. Latency scales with K and context length; there is no generation.
• Calibrated for single-choice decisions over small option sets (2–64). Confidence is a probability of correctness among these options — re-rank with a new option set and the distribution changes accordingly.
• English-centric training data.

Repository layout

primes/            python client + vendored runtime (serialize, head)
models/            weights (gitignored; `make model` stages them)
server/            compose stack: llama.cpp + API sidecar + web playground
val/               public validation set + validate.py
scripts/           patch_pooling.py (GGUF pooling stamp for ollama)
Modelfile          ollama import
Makefile           model / ollama / serve / validate / smoke / llama-native