infra/docs/runbooks/beads-auto-dispatch.md

# Beads Auto-Dispatch Runbook

Users can hand work to the headless `beads-task-runner` agent by assigning a
bead to the sentinel user `agent`. Two CronJobs in the `beads-server`
namespace drive the pipeline:

- **`beads-dispatcher`** — every 2 min: picks up the highest-priority
  `assignee=agent`/`status=open` bead with non-empty acceptance criteria,
  claims it by flipping to `in_progress`, and POSTs it to BeadBoard's
  `/api/agent-dispatch`. BeadBoard forwards to `claude-agent-service` with
  the existing bearer-token flow.
- **`beads-reaper`** — every 10 min: flips any `assignee=agent` +
  `status=in_progress` bead whose `updated_at` is older than 30 min to
  `status=blocked` with an explanatory note. Catches pod crashes mid-run.

The manual BeadBoard Dispatch button continues to work in parallel.

## Flow diagram

```
  user: bd assign <id> agent
         │
         ▼
  Dolt @ dolt.beads-server.svc:3306  ◄──── every 2 min ────┐
         │                                                  │
         ▼                                                  │
  CronJob: beads-dispatcher                                 │
    1. GET beadboard/api/agent-status  (busy?)              │
    2. bd query 'assignee=agent AND status=open'            │
    3. bd update -s in_progress   (claim)                   │
    4. POST beadboard/api/agent-dispatch                    │
    5. bd note "dispatched: job=…"                          │
         │                                                  │
         ▼                                                  │
  claude-agent-service /execute                             │
    beads-task-runner agent runs; notes/closes bead         │
         │                                                  │
         ▼                                                  │
  done  ──► next tick picks up the next bead ───────────────┘


  CronJob: beads-reaper  (every 10 min)
    for bead (assignee=agent, status=in_progress, updated_at > 30 min):
      bd note   "reaper: no progress for Nm — blocking"
      bd update -s blocked
```

## Usage

### Hand a bead to the agent

```
bd create "Title" \
    -d "Full context — files, services, error messages. Any agent with no prior context must be able to execute this." \
    --acceptance "Concrete, verifiable criteria" \
    -p 2
bd assign <new-id> agent
```

**Acceptance criteria is required.** Beads without it are skipped by the
dispatcher and stay in `open` forever. This is intentional — the
`beads-task-runner` agent expects clear done conditions.

### Take a bead back (unassign)

```
bd assign <id> ""
```

If the bead is already `in_progress`, also reset it:

```
bd update <id> -s open
```

### Pause auto-dispatch

```
cd infra/stacks/beads-server
scripts/tg apply -var=beads_dispatcher_enabled=false
```

This sets `spec.suspend: true` on both CronJobs. Existing running jobs
continue; no new ticks fire. Re-enable by re-applying with
`beads_dispatcher_enabled=true` (the default). Manual BeadBoard Dispatch
remains available while paused.

### Read the logs

```
# Recent dispatcher runs
kubectl -n beads-server get jobs --selector=job-name --sort-by=.metadata.creationTimestamp | grep beads-dispatcher | tail
kubectl -n beads-server logs job/<dispatcher-job-name>

# Tail the underlying agent once a bead dispatches
kubectl -n claude-agent logs -l app=claude-agent-service -f

# Inspect reaper decisions
kubectl -n beads-server get jobs | grep beads-reaper | tail
kubectl -n beads-server logs job/<reaper-job-name>
```

### Inspect a specific bead's dispatch history

```
bd show <id> --json | jq '{status, assignee, notes, updated_at}'
```

Both the dispatcher and reaper write dated notes (`auto-dispatcher claimed
at…`, `dispatched: job=…`, `reaper: no progress for…`) so the audit trail
lives on the bead itself.

## Reaper semantics — when a bead becomes `blocked`

The reaper flips a bead to `blocked` if:
- `assignee = agent`, AND
- `status = in_progress`, AND
- `updated_at` is more than **30 minutes** in the past.

Every `bd note` bumps `updated_at`, so a well-behaved `beads-task-runner`
agent never trips the reaper — it notes progress as it works. A `blocked`
bead is a signal that:
- the agent pod crashed mid-run (`kubectl -n claude-agent delete pod` test),
- the job hit its 15-minute budget timeout inside `claude-agent-service`
  without notes (rare — the agent usually notes failure before exiting),
- `claude-agent-service` was restarted during the run (in-memory job state
  is lost; see [known risks](#known-risks)).

Recovery: read the reaper note, reopen manually if appropriate:

```
bd update <id> -s open
bd assign <id> agent     # re-arm for next dispatcher tick
```

## Design choices

- **Sentinel assignee `agent`** — free-form, no Beads schema change. Any bd
  client can set it (`bd assign <id> agent`).
- **One-bead-per-tick dispatch** — the dispatcher submits at most one bead
  per 2-min tick, gating on `claude-agent-service`'s `/health` `busy` flag.
  `busy` now means `active >= capacity` (bounded semaphore, default 10) — the
  service no longer single-flight-locks via `asyncio.Lock`. So up to
  ~`capacity` beads can run concurrently; the 2-min poll cadence (not
  single-slot execution) now bounds ramp-up.
- **Fixed agent (`beads-task-runner`)** — read-only rails, matches BeadBoard's
  manual Dispatch button. Broader-privilege agents stay manual.
- **CronJob (not in-service polling, not n8n)** — matches existing infra
  pattern (OpenClaw task-processor, certbot-renewal, backups), TF-managed,
  easy to pause.
- **ConfigMap-mounted `metadata.json`** — declarative TF rather than reusing
  the image-seeded file. The CronJob's init step copies it into `/tmp/.beads/`
  because `bd` may touch the parent directory and ConfigMap mounts are
  read-only.

## Known risks

- **In-memory job state in `claude-agent-service`** — if the pod restarts
  mid-run, the job record is lost. The reaper catches this after 30 min.
  Persistent job store is deferred.
- **Prompt injection via bead fields** — a malicious bead description could
  try to steer the agent. The `beads-task-runner` rails + token budget +
  timeout are the defense. Identical exposure as the manual Dispatch button.
- **Image tag drift** — `claude_agent_service_image_tag` in
  `stacks/beads-server/main.tf` mirrors `local.image_tag` in
  `stacks/claude-agent-service/main.tf`. Bump both when the image rebuilds,
  or the dispatcher/reaper will run on an older layer. (They only need
  `bd`, `curl`, `jq` — stable across rebuilds — so the drift is low-risk.)
- **`bd` JSON schema changes** — the reaper's `jq` reads `.id` and
  `.updated_at`. If a future `bd` upgrade renames these, the reaper breaks
  silently (no reaping, no alert). `BD_VERSION` is pinned in the image
  Dockerfile.

## Verification after change

```
# Both CronJobs exist with the right schedule / SUSPEND state
kubectl -n beads-server get cronjob

# End-to-end smoke test
bd create "auto-dispatch smoke test" \
    -d "Read /etc/hostname inside the agent sandbox and close." \
    --acceptance "bd note includes 'hostname=' and bead is closed."
bd assign <new-id> agent
# within 2 min:
bd show <new-id> --json | jq '.notes'
# → contains 'auto-dispatcher claimed' + 'dispatched: job=<uuid>'
```