claude-agent-service/docs/2026-06-14-afk-implementation-pipeline-design.md

AFK implementation pipeline — design

Date: 2026-06-14 Status: proposed — pilot pending (see "Pilot" below; no code yet) Scope: A new autonomous path that turns a triaged ready-for-agent issue into tested, deployed code with no human at the keyboard. claude-agent-service becomes the control plane; a dedicated in-cluster T3 Code instance becomes the executor + cockpit. Touches: claude-agent-service (new poller

• dispatch + watcher), a new T3 stack in infra/, a shared SSD-NFS volume, and the per-repo issue trackers.

Provenance: this design is the output of a long grilling session (2026-06-14). It records the decisions and the alternatives that were considered and dropped, so the reasoning survives. The three hardest-to-reverse calls are split into ADRs 0002–0004.

Problem

Today the development flow is grill-with-docs → to-prd → to-issues → triage → implement, and every stage is human-in-the-loop (HITL), including implementation. The owner wants the HITL boundary to stop at design + spec: once an issue is triaged ready-for-agent, an agent should pick it up and implement it AFK (away from keyboard) — write it test-first, push it, and see it through to a healthy deploy — escalating to a human only when it genuinely can't proceed.

Two gaps block this today:

• The only existing issue→agent automation is the infra issue-responder, which fires on user-report/feature-request labels on the infra repo only — not on ready-for-agent, not on the other sub-project repos that the general design flow produces.
• claude-agent-service only ever clones infra, runs one-shot fire-and-forget claude -p jobs (no session, no live stream, no attach), and has no multi-repo checkout. The owner wants to watch and steer in-flight work, which the batch model can't offer.

Goal

• HITL covers design + spec only. Publishing ready-for-agent issues is the release signal (the to-issues quiz is the review gate).
• An autonomous loop picks up unblocked ready-for-agent issues from enrolled repos, implements them test-first, and lands them — pushing straight to master so CI deploys them (see ADR 0002 for the risk posture).
• The owner can see all in-flight workers and converse with any of them from one UI — the T3 cockpit (see ADR 0003).
• Reuse before building: lean on the existing CI/CD chain, the design skills, T3 Code's multi-agent cockpit, and the persistence/worktree machinery — rather than hand-building a session console and a bespoke runtime.

Design

Roles: control plane vs executor + cockpit

Concern	Owner
When to start, which issue, the prompt, the safety envelope	claude-agent-service (control plane) — poller + watcher
Running the agent (Claude Agent SDK), the worktree, the fleet UI	T3 Code (executor + cockpit) — one dedicated in-cluster instance
Build → image → deploy → rollout	existing CI/CD (GHA → ghcr → Woodpecker → Keel)
Issue queue + state	the per-repo GitHub issue trackers

The pivotal constraint that forces this split: T3 can only display sessions it launched itself — it has no command to adopt an externally-started session. So "viewable in T3" ⟺ "launched by T3". To keep claude-agent-service in charge and get the fleet view, the control plane dispatches into T3 rather than running claude itself. See ADR 0003.

End-to-end flow

HUMAN (interactive session)
  /grill-with-docs → /to-prd → /to-issues → /triage
     └ produces ready-for-agent issues (dependency-ordered), labeled by a
       trusted collaborator. Publishing them = the release signal.
══════════════════════ HANDOFF ══════════════════════
CONTROL PLANE  (claude-agent-service, in-cluster)
  poller CronJob (every few min):
    for repo in allowlist:
      skip repo if it already has an agent-in-progress issue   (per-repo lock)
      pick highest-priority ready-for-agent issue where:
        • all "Blocked by" closed   • labeled by a trusted collaborator
      → stamp agent-in-progress
      → POST /api/orchestration/dispatch  (thread.turn.start + bootstrap:
            create thread, prepare worktree, run setup, deliver the prompt)
EXECUTOR + COCKPIT  (dedicated T3 instance, in-cluster)
  runs the issue-implementer agent (our prompt) in the worktree:
    read issue + AGENT-BRIEF + repo CONTEXT.md/ADRs → TDD red-green-refactor
    → commit (paraphrase issue, "Closes #N", AFK trailer) → push master
  watcher (control plane) polls GET /api/orchestration/snapshot + CI:
    ├─ healthy ──────► comment + close issue, drop lock, notify ✅
    ├─ pre-push block ► do NOT push, relabel ready-for-human, escalate
    └─ post-push red ► fix-forward (≤5 attempts / 60 min)
                         ├─ recovers ► healthy
                         └─ exhausts ► FREEZE broken (preserve forensics),
                                       relabel ready-for-human, hard page

Trigger & dispatch predicate

A poller CronJob (mirrors the existing beads-dispatcher pattern; stays in-cluster because neither the service nor T3 has public ingress). It dispatches issue I in repo R iff all hold:

• R is in the allowlist ConfigMap, and the kill switch is off;
• I has label ready-for-agent, applied by a trusted collaborator (the trust gate — on private repos only collaborators can label, so the label is the authorization; external/bot issues never auto-run);
• every issue in I's "Blocked by" is closed;
• R has no issue currently labeled agent-in-progress (the per-repo lock).

On dispatch it stamps agent-in-progress; on any terminal outcome it removes it.

Concurrency & locking

Parallel across repos, serial within a repo. Multiple repos progress at once; at most one agent per repo (two agents in one repo would collide on the working tree). Enforced by the agent-in-progress label as a per-repo lock. Starting value; raise later.

Merge & failure posture — see ADR 0002

• Always push to master (no PR gate). Tests-green is the merge gate; CI + rollback are the safety net, matching the human allow-then-audit model.
• Pre-push failure (can't get green / blocked / would need a disallowed op): do not push; relabel ready-for-human; comment what was tried; page.
• Post-push failure (CI build or rollout red): fix-forward up to 5 attempts or 60 minutes, then if still red freeze in the broken state (preserve forensics — do not auto-revert), relabel ready-for-human, hard page. The owner explicitly chose debuggability over availability here.
• Budget: max_budget_usd = 100 per issue (time/attempt caps usually bite first).

Build/test environment & worktrees — see ADR 0004

The agent must run the target repo's test suite (TDD gate) before pushing. Therefore:

• Local toolchains scoped to the allowlist — the executor image carries only the enrolled repos' runtimes; the toolchain set grows in lockstep with the allowlist.
• Persistent per-repo checkout + git worktree per issue on a shared SSD-NFS volume, so git objects, installed deps, and package-manager caches stay warm across jobs. This supersedes the throwaway git clone --local model from 2026-06-02-parallel-execution-design.md; that rejection was correct for concurrent same-repo jobs, but the serial-within-repo choice here removes the .git contention it guarded against (ADR 0004). It pays off precisely because to-issues clusters many slices in one repo, processed serially — slice N reuses the warm checkout slice 1 paid for.

T3 integration: thin dispatch — see ADR 0003

The control plane holds a capability-scoped orchestration:operate bearer token (minted via t3 auth, stored in Vault, refreshed for the 1-hour expiry) and calls T3's HTTP API:

• POST /api/orchestration/dispatch → thread.turn.start with a bootstrap that creates the thread, prepares the worktree, optionally runs a setup script, and delivers the prompt — one call spawns a worktree-isolated worker.
• GET /api/orchestration/snapshot → the full fleet read-model (per-thread running/idle/error, hasPendingUserInput, hasPendingApprovals, branch, worktreePath). T3 has no outbound webhooks, so the watcher polls this to drive CI-watch, freeze, and label transitions.

The AFK behavior and safety (issue-implementer prompt, guardrails, always-push, fix-forward/freeze, issue integration) live in our thin layer, so T3 is a swappable, version-pinned backend — never Keel-auto-upgraded, reversible to a self-hosted runtime if it goes sideways.

Observability & interaction

The "active sessions layer" and the "attach and converse" surface converge into one screen — the T3 cockpit: a live list of all worker threads grouped by project; click one to stream its transcript and send it a turn. This dissolves the earlier intermediate ideas of a generalized-breakglass console and a raw-tmux hybrid attach — T3 provides converse / approve / resume natively (thread.user-input.respond, thread.approval.respond).

Cross-system, durable signals the control plane still emits:

• Phase-checklist comment on the issue, edited in place as phases complete (worktree → tests-red → green → pushed → CI → deployed). Durable, low-noise, lives on the issue, doubles as audit trail.
• Loki logs labeled {repo, issue} for deep-dive.
• Presence claim per running session (repo:<name>, purpose AFK #N), heartbeated — so AFK work shows up next to human sessions in the layer the prompt hook already injects.
• Doorbell: Slack / ntfy ping on terminal states, deep-linking into the T3 thread. Notify, not control — the dedicated-Slack-control-plane idea is dropped in favour of the T3 cockpit.

Safety envelope

• Trust gate — only collaborator-labeled ready-for-agent issues run.
• Allowlist — a repo is untouchable until enrolled (prereqs: tests + GHA CI
  • CONTEXT.md). Start with 1–2 repos; expand deliberately.
• Kill switch — one ConfigMap flag pauses all pickup (the Keel scale-to-0 reflex, built in from day one).
• Per-repo lock — ≤1 agent per repo.
• Guardrails (reused from issue-responder) — no PVC/PV deletes, no direct Vault edits, no force-push to master, infra changes Terraform-only, never [ci skip].
• Identity & audit — shared service identity; each commit body paraphrases the issue and carries Closes #N + an AFK-agent trailer, so the commit message stays the audit trail.

Parameters (chosen starting values — all tunable)

Knob	Value
Merge gate	always push to master
Post-push failure	fix-forward, then freeze-broken
Fix-forward cap	5 attempts or 60 minutes
Per-issue budget	max_budget_usd = 100
Concurrency	parallel across repos, serial within a repo
Repo scope	opt-in allowlist, start small
Progress detail	phase-checklist on issue + Loki logs
Alert channel	Slack (+ ntfy), as a doorbell into T3
Executor	dedicated in-cluster T3 (thin dispatch), version-pinned

Pilot — validate before wiring the poller

The thin model rests on five unknowns. Stand up the dedicated T3 instance and drive a couple of allowlist-repo issues by hand via the dispatch API to confirm each, before building the poller and committing the architecture:

1. Per-thread custom agent + skip-permissions — can a dispatched thread carry our issue-implementer system prompt and run unattended without stalling on T3's approval gating? (biggest unknown)
2. Dispatch auth — mint orchestration:operate, store in Vault, refresh the 1-hour token.
3. Status/completion — drive CI-watch/freeze/labels purely from polling GET /api/orchestration/snapshot.
4. Worktree reconciliation — T3's native prepareWorktree vs our persistent-checkout-with-warm-caches; pick one or make them cooperate on the volume.
5. The in-cluster T3 pod — headless t3 serve --no-browser, version-pinned and Keel-excluded, internal ingress + Authentik, with tokens / toolchains / SSD volume / claude auth provisioned.

Relationship to prior decisions

• Supersedes the worktree rejection in 2026-06-02-parallel-execution-design.md (contextualized, not contradicted — ADR 0004).
• Drops two intermediate ideas explored and rejected this session: evolving claude-agent-service into its own session/tmux/worktree runtime, and building a bespoke breakglass-generalized console — both replaced by T3.
• Reuses the issue-responder guardrails, the CI/CD chain, the beads-dispatcher CronJob pattern, presence, Loki, and the design skills.

Out of scope / open questions

• Raw-terminal "take-over" of a worker (T3 is a GUI cockpit, not a terminal); if ever needed, that's a separate add-on.
• Multi-tenant T3 (it is single-operator by design — fine, it matches the shared service identity).
• Cross-repo dependency orchestration beyond per-issue "Blocked by".
• T3 Code is pre-1.0 (~v0.0.x) and churny; the version-pin + Keel-exclude + swappable-backend discipline (ADR 0003) is the mitigation.