docs: capture AFK implementation pipeline design + ADRs 0002-0004
Some checks are pending
Some checks are pending
Record the architecture for moving code implementation AFK, decided in a design/grilling session. The owner wants the human-in-the-loop boundary to stop at design + spec: once an issue is triaged ready-for-agent, an agent should implement it test-first, push it, and see it to a healthy deploy on its own, escalating only when it can't proceed. Decisions captured: - claude-agent-service is the control plane (poller + watcher + safety); a dedicated in-cluster T3 Code instance is the executor + cockpit, because T3 can only show sessions it launched itself -> we dispatch into it (ADR 0003). - AFK code pushes straight to master; on a broken deploy it fix-forwards then freezes the broken state for forensics rather than reverting (ADR 0002). - Implementation agents use persistent per-repo checkouts + git worktrees on SSD-NFS for warm caches, reversing the throwaway-clone rule for this path because concurrency is serial-within-repo (ADR 0004). Pilot-gated: five integration unknowns must be validated against a dedicated T3 instance before the poller is wired. No code yet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
9d8afdd884
commit
be81005186
4 changed files with 466 additions and 0 deletions
259
docs/2026-06-14-afk-implementation-pipeline-design.md
Normal file
259
docs/2026-06-14-afk-implementation-pipeline-design.md
Normal file
|
|
@ -0,0 +1,259 @@
|
|||
# AFK implementation pipeline — design
|
||||
|
||||
**Date:** 2026-06-14
|
||||
**Status:** proposed — pilot pending (see "Pilot" below; no code yet)
|
||||
**Scope:** A new autonomous path that turns a triaged `ready-for-agent` issue
|
||||
into tested, deployed code with no human at the keyboard. `claude-agent-service`
|
||||
becomes the **control plane**; a dedicated in-cluster **T3 Code** instance
|
||||
becomes the **executor + cockpit**. Touches: `claude-agent-service` (new poller
|
||||
+ dispatch + watcher), a new T3 stack in `infra/`, a shared SSD-NFS volume, and
|
||||
the per-repo issue trackers.
|
||||
|
||||
> Provenance: this design is the output of a long grilling session
|
||||
> (2026-06-14). It records the decisions *and* the alternatives that were
|
||||
> considered and dropped, so the reasoning survives. The three hardest-to-reverse
|
||||
> calls are split into ADRs 0002–0004.
|
||||
|
||||
## Problem
|
||||
|
||||
Today the development flow is **grill-with-docs → to-prd → to-issues → triage →
|
||||
implement**, and *every* stage is human-in-the-loop (HITL), including
|
||||
implementation. The owner wants the HITL boundary to stop at **design + spec**:
|
||||
once an issue is triaged `ready-for-agent`, an agent should pick it up and
|
||||
implement it **AFK** (away from keyboard) — write it test-first, push it, and
|
||||
see it through to a healthy deploy — escalating to a human only when it genuinely
|
||||
can't proceed.
|
||||
|
||||
Two gaps block this today:
|
||||
|
||||
- The only existing issue→agent automation is the **infra `issue-responder`**,
|
||||
which fires on `user-report`/`feature-request` labels on the `infra` repo
|
||||
only — not on `ready-for-agent`, not on the other sub-project repos that the
|
||||
general design flow produces.
|
||||
- `claude-agent-service` only ever clones `infra`, runs one-shot fire-and-forget
|
||||
`claude -p` jobs (no session, no live stream, no attach), and has no
|
||||
multi-repo checkout. The owner wants to *watch and steer* in-flight work, which
|
||||
the batch model can't offer.
|
||||
|
||||
## Goal
|
||||
|
||||
- HITL covers design + spec only. Publishing `ready-for-agent` issues is the
|
||||
release signal (the `to-issues` quiz is the review gate).
|
||||
- An autonomous loop picks up unblocked `ready-for-agent` issues from
|
||||
**enrolled** repos, implements them test-first, and lands them — pushing
|
||||
straight to `master` so CI deploys them (see ADR 0002 for the risk posture).
|
||||
- The owner can **see all in-flight workers and converse with any of them** from
|
||||
one UI — the T3 cockpit (see ADR 0003).
|
||||
- Reuse before building: lean on the existing CI/CD chain, the design skills, T3
|
||||
Code's multi-agent cockpit, and the persistence/worktree machinery — rather
|
||||
than hand-building a session console and a bespoke runtime.
|
||||
|
||||
## Design
|
||||
|
||||
### Roles: control plane vs executor + cockpit
|
||||
|
||||
| Concern | Owner |
|
||||
|---|---|
|
||||
| When to start, which issue, the prompt, the safety envelope | **claude-agent-service** (control plane) — poller + watcher |
|
||||
| Running the agent (Claude Agent SDK), the worktree, the fleet UI | **T3 Code** (executor + cockpit) — one dedicated in-cluster instance |
|
||||
| Build → image → deploy → rollout | existing CI/CD (GHA → ghcr → Woodpecker → Keel) |
|
||||
| Issue queue + state | the per-repo GitHub issue trackers |
|
||||
|
||||
The pivotal constraint that forces this split: **T3 can only display sessions it
|
||||
launched itself** — it has no command to adopt an externally-started session. So
|
||||
"viewable in T3" ⟺ "launched by T3". To keep `claude-agent-service` in charge
|
||||
*and* get the fleet view, the control plane **dispatches into T3** rather than
|
||||
running `claude` itself. See ADR 0003.
|
||||
|
||||
### End-to-end flow
|
||||
|
||||
```
|
||||
HUMAN (interactive session)
|
||||
/grill-with-docs → /to-prd → /to-issues → /triage
|
||||
└ produces ready-for-agent issues (dependency-ordered), labeled by a
|
||||
trusted collaborator. Publishing them = the release signal.
|
||||
══════════════════════ HANDOFF ══════════════════════
|
||||
CONTROL PLANE (claude-agent-service, in-cluster)
|
||||
poller CronJob (every few min):
|
||||
for repo in allowlist:
|
||||
skip repo if it already has an agent-in-progress issue (per-repo lock)
|
||||
pick highest-priority ready-for-agent issue where:
|
||||
• all "Blocked by" closed • labeled by a trusted collaborator
|
||||
→ stamp agent-in-progress
|
||||
→ POST /api/orchestration/dispatch (thread.turn.start + bootstrap:
|
||||
create thread, prepare worktree, run setup, deliver the prompt)
|
||||
EXECUTOR + COCKPIT (dedicated T3 instance, in-cluster)
|
||||
runs the issue-implementer agent (our prompt) in the worktree:
|
||||
read issue + AGENT-BRIEF + repo CONTEXT.md/ADRs → TDD red-green-refactor
|
||||
→ commit (paraphrase issue, "Closes #N", AFK trailer) → push master
|
||||
watcher (control plane) polls GET /api/orchestration/snapshot + CI:
|
||||
├─ healthy ──────► comment + close issue, drop lock, notify ✅
|
||||
├─ pre-push block ► do NOT push, relabel ready-for-human, escalate
|
||||
└─ post-push red ► fix-forward (≤5 attempts / 60 min)
|
||||
├─ recovers ► healthy
|
||||
└─ exhausts ► FREEZE broken (preserve forensics),
|
||||
relabel ready-for-human, hard page
|
||||
```
|
||||
|
||||
### Trigger & dispatch predicate
|
||||
|
||||
A poller CronJob (mirrors the existing `beads-dispatcher` pattern; stays
|
||||
in-cluster because neither the service nor T3 has public ingress). It dispatches
|
||||
issue *I* in repo *R* iff **all** hold:
|
||||
|
||||
- `R` is in the **allowlist** ConfigMap, and the **kill switch** is off;
|
||||
- `I` has label `ready-for-agent`, applied by a **trusted collaborator** (the
|
||||
trust gate — on private repos only collaborators can label, so the label *is*
|
||||
the authorization; external/bot issues never auto-run);
|
||||
- every issue in `I`'s "Blocked by" is closed;
|
||||
- `R` has no issue currently labeled `agent-in-progress` (the per-repo lock).
|
||||
|
||||
On dispatch it stamps `agent-in-progress`; on any terminal outcome it removes it.
|
||||
|
||||
### Concurrency & locking
|
||||
|
||||
**Parallel across repos, serial within a repo.** Multiple repos progress at
|
||||
once; at most one agent per repo (two agents in one repo would collide on the
|
||||
working tree). Enforced by the `agent-in-progress` label as a per-repo lock.
|
||||
Starting value; raise later.
|
||||
|
||||
### Merge & failure posture — see ADR 0002
|
||||
|
||||
- **Always push to master** (no PR gate). Tests-green is the merge gate; CI +
|
||||
rollback are the safety net, matching the human allow-then-audit model.
|
||||
- **Pre-push** failure (can't get green / blocked / would need a disallowed op):
|
||||
do *not* push; relabel `ready-for-human`; comment what was tried; page.
|
||||
- **Post-push** failure (CI build or rollout red): **fix-forward** up to **5
|
||||
attempts or 60 minutes**, then if still red **freeze in the broken state**
|
||||
(preserve forensics — do not auto-revert), relabel `ready-for-human`, hard
|
||||
page. The owner explicitly chose debuggability over availability here.
|
||||
- **Budget:** `max_budget_usd = 100` per issue (time/attempt caps usually bite
|
||||
first).
|
||||
|
||||
### Build/test environment & worktrees — see ADR 0004
|
||||
|
||||
The agent must run the target repo's test suite (TDD gate) before pushing.
|
||||
Therefore:
|
||||
|
||||
- **Local toolchains scoped to the allowlist** — the executor image carries only
|
||||
the *enrolled* repos' runtimes; the toolchain set grows in lockstep with the
|
||||
allowlist.
|
||||
- **Persistent per-repo checkout + `git worktree` per issue** on a shared
|
||||
**SSD-NFS** volume, so git objects, installed deps, and package-manager caches
|
||||
stay warm across jobs. This **supersedes** the throwaway `git clone --local`
|
||||
model from `2026-06-02-parallel-execution-design.md`; that rejection was
|
||||
correct for *concurrent* same-repo jobs, but the serial-within-repo choice
|
||||
here removes the `.git` contention it guarded against (ADR 0004). It pays off
|
||||
precisely because `to-issues` clusters many slices in one repo, processed
|
||||
serially — slice N reuses the warm checkout slice 1 paid for.
|
||||
|
||||
### T3 integration: thin dispatch — see ADR 0003
|
||||
|
||||
The control plane holds a capability-scoped **`orchestration:operate`** bearer
|
||||
token (minted via `t3 auth`, stored in Vault, refreshed for the 1-hour expiry)
|
||||
and calls T3's HTTP API:
|
||||
|
||||
- `POST /api/orchestration/dispatch` → `thread.turn.start` with a `bootstrap`
|
||||
that creates the thread, prepares the worktree, optionally runs a setup
|
||||
script, and delivers the prompt — one call spawns a worktree-isolated worker.
|
||||
- `GET /api/orchestration/snapshot` → the full fleet read-model (per-thread
|
||||
`running`/`idle`/`error`, `hasPendingUserInput`, `hasPendingApprovals`,
|
||||
`branch`, `worktreePath`). T3 has **no outbound webhooks**, so the watcher
|
||||
**polls** this to drive CI-watch, freeze, and label transitions.
|
||||
|
||||
The AFK *behavior and safety* (issue-implementer prompt, guardrails, always-push,
|
||||
fix-forward/freeze, issue integration) live in **our** thin layer, so T3 is a
|
||||
**swappable, version-pinned backend** — never Keel-auto-upgraded, reversible to a
|
||||
self-hosted runtime if it goes sideways.
|
||||
|
||||
### Observability & interaction
|
||||
|
||||
The "active sessions layer" and the "attach and converse" surface **converge
|
||||
into one screen — the T3 cockpit**: a live list of all worker threads grouped by
|
||||
project; click one to stream its transcript and send it a turn. This dissolves
|
||||
the earlier intermediate ideas of a generalized-breakglass console and a
|
||||
raw-tmux hybrid attach — T3 provides converse / approve / resume natively
|
||||
(`thread.user-input.respond`, `thread.approval.respond`).
|
||||
|
||||
Cross-system, durable signals the control plane still emits:
|
||||
|
||||
- **Phase-checklist comment** on the issue, edited in place as phases complete
|
||||
(worktree → tests-red → green → pushed → CI → deployed). Durable, low-noise,
|
||||
lives on the issue, doubles as audit trail.
|
||||
- **Loki** logs labeled `{repo, issue}` for deep-dive.
|
||||
- **Presence** claim per running session (`repo:<name>`, purpose `AFK #N`),
|
||||
heartbeated — so AFK work shows up next to human sessions in the layer the
|
||||
prompt hook already injects.
|
||||
- **Doorbell**: Slack / ntfy ping on terminal states, deep-linking into the T3
|
||||
thread. Notify, not control — the dedicated-Slack-control-plane idea is
|
||||
dropped in favour of the T3 cockpit.
|
||||
|
||||
### Safety envelope
|
||||
|
||||
- **Trust gate** — only collaborator-labeled `ready-for-agent` issues run.
|
||||
- **Allowlist** — a repo is untouchable until enrolled (prereqs: tests + GHA CI
|
||||
+ `CONTEXT.md`). Start with 1–2 repos; expand deliberately.
|
||||
- **Kill switch** — one ConfigMap flag pauses all pickup (the Keel
|
||||
scale-to-0 reflex, built in from day one).
|
||||
- **Per-repo lock** — ≤1 agent per repo.
|
||||
- **Guardrails** (reused from `issue-responder`) — no PVC/PV deletes, no direct
|
||||
Vault edits, no force-push to master, infra changes Terraform-only, never
|
||||
`[ci skip]`.
|
||||
- **Identity & audit** — shared service identity; each commit body paraphrases
|
||||
the issue and carries `Closes #N` + an AFK-agent trailer, so the commit
|
||||
message stays the audit trail.
|
||||
|
||||
## Parameters (chosen starting values — all tunable)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Merge gate | always push to master |
|
||||
| Post-push failure | fix-forward, then freeze-broken |
|
||||
| Fix-forward cap | 5 attempts **or** 60 minutes |
|
||||
| Per-issue budget | `max_budget_usd = 100` |
|
||||
| Concurrency | parallel across repos, serial within a repo |
|
||||
| Repo scope | opt-in allowlist, start small |
|
||||
| Progress detail | phase-checklist on issue + Loki logs |
|
||||
| Alert channel | Slack (+ ntfy), as a doorbell into T3 |
|
||||
| Executor | dedicated in-cluster T3 (thin dispatch), version-pinned |
|
||||
|
||||
## Pilot — validate before wiring the poller
|
||||
|
||||
The thin model rests on five unknowns. Stand up the dedicated T3 instance and
|
||||
drive a couple of allowlist-repo issues **by hand** via the dispatch API to
|
||||
confirm each, *before* building the poller and committing the architecture:
|
||||
|
||||
1. **Per-thread custom agent + skip-permissions** — can a dispatched thread
|
||||
carry *our* `issue-implementer` system prompt and run unattended without
|
||||
stalling on T3's approval gating? *(biggest unknown)*
|
||||
2. **Dispatch auth** — mint `orchestration:operate`, store in Vault, refresh the
|
||||
1-hour token.
|
||||
3. **Status/completion** — drive CI-watch/freeze/labels purely from polling
|
||||
`GET /api/orchestration/snapshot`.
|
||||
4. **Worktree reconciliation** — T3's native `prepareWorktree` vs our
|
||||
persistent-checkout-with-warm-caches; pick one or make them cooperate on the
|
||||
volume.
|
||||
5. **The in-cluster T3 pod** — headless `t3 serve --no-browser`, version-pinned
|
||||
and **Keel-excluded**, internal ingress + Authentik, with tokens / toolchains
|
||||
/ SSD volume / `claude auth` provisioned.
|
||||
|
||||
## Relationship to prior decisions
|
||||
|
||||
- **Supersedes** the worktree rejection in
|
||||
`2026-06-02-parallel-execution-design.md` (contextualized, not contradicted —
|
||||
ADR 0004).
|
||||
- **Drops** two intermediate ideas explored and rejected this session:
|
||||
evolving `claude-agent-service` into its own session/tmux/worktree runtime,
|
||||
and building a bespoke breakglass-generalized console — both replaced by T3.
|
||||
- **Reuses** the `issue-responder` guardrails, the CI/CD chain, the
|
||||
`beads-dispatcher` CronJob pattern, presence, Loki, and the design skills.
|
||||
|
||||
## Out of scope / open questions
|
||||
|
||||
- Raw-terminal "take-over" of a worker (T3 is a GUI cockpit, not a terminal); if
|
||||
ever needed, that's a separate add-on.
|
||||
- Multi-tenant T3 (it is single-operator by design — fine, it matches the shared
|
||||
service identity).
|
||||
- Cross-repo dependency orchestration beyond per-issue "Blocked by".
|
||||
- T3 Code is pre-1.0 (~v0.0.x) and churny; the version-pin + Keel-exclude +
|
||||
swappable-backend discipline (ADR 0003) is the mitigation.
|
||||
69
docs/adr/0002-afk-autonomous-merge-and-failure-posture.md
Normal file
69
docs/adr/0002-afk-autonomous-merge-and-failure-posture.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
# AFK agents push straight to master; failures fix-forward then freeze, not revert
|
||||
|
||||
The AFK implementation pipeline (see
|
||||
`docs/2026-06-14-afk-implementation-pipeline-design.md`) lets an autonomous
|
||||
agent land code with no human at the keyboard. The owner deliberately chose the
|
||||
most hands-off posture: **AFK-written code pushes straight to `master`** (which
|
||||
then deploys via the existing CI/CD chain) with **no pull-request review gate**,
|
||||
and when a deploy breaks, the agent **fixes forward and then freezes the broken
|
||||
state** rather than auto-reverting. This ADR records that risk posture and why it
|
||||
was chosen over the safer alternatives, because it is surprising and not cheap to
|
||||
walk back once callers and habits depend on it.
|
||||
|
||||
## Status
|
||||
|
||||
accepted (2026-06-14) — posture decided; enforced once the pipeline ships
|
||||
(pilot-gated).
|
||||
|
||||
## Context
|
||||
|
||||
`master` on every enrolled repo deploys continuously (GHA build → ghcr →
|
||||
Woodpecker → Keel). So "where AFK code lands" is really "what reaches a live
|
||||
deploy without a human looking". The owner weighed three merge gates and three
|
||||
post-push failure responses and picked the autonomy-maximizing end of both,
|
||||
accepting the blast radius explicitly.
|
||||
|
||||
## Considered options — merge gate
|
||||
|
||||
- **Always push to master (chosen).** Tests-green is the gate; CI + rollback are
|
||||
the safety net. Matches the existing human allow-then-audit model (non-admins
|
||||
already push straight to master). Most hands-off.
|
||||
- **Adaptive (push if confident, else PR)** — rejected as the *default* though it
|
||||
is what `issue-responder` does; the owner wanted full hands-off, not a
|
||||
confidence-gated PR for otherwise-working code.
|
||||
- **Always open a PR** — rejected: reintroduces a human merge step on every
|
||||
issue, i.e. "AFK implementation, human merge" — not the goal.
|
||||
|
||||
## Considered options — post-push failure (CI/rollout goes red after a green push)
|
||||
|
||||
- **Fix-forward then freeze (chosen).** Iterate with corrective commits up to
|
||||
**5 attempts or 60 minutes**; if still red, **leave the broken state in place**
|
||||
(do not revert), relabel the issue `ready-for-human`, and hard-page. Same
|
||||
forensics-first instinct as the breakglass (ADR 0001): preserve the exact
|
||||
failing state for debugging rather than auto-cleaning it away.
|
||||
- **Auto-revert + escalate** — rejected (was the recommendation): restores green
|
||||
fastest, but destroys the forensic state the owner wants to inspect.
|
||||
- **Alert and freeze immediately (no fix-forward)** — rejected: gives up on
|
||||
transient/env-drift failures a corrective commit would clear.
|
||||
|
||||
Pre-push failure (can't reach green, blocked, or would need a disallowed op) is
|
||||
not a dilemma: the agent does **not** push, relabels `ready-for-human`, comments
|
||||
what it tried, and pages.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An unreviewed logic error can deploy before any human sees it; rollback (not
|
||||
review) is the safety net. Bounded by: tests-as-gate, the start-small
|
||||
allowlist, the per-repo lock, and the kill switch.
|
||||
- A frozen-broken deploy can sit unhealthy until the owner answers the page —
|
||||
availability is traded for debuggability, by explicit choice. Acceptable
|
||||
because enrolled repos are non-critical by the allowlist prerequisite, and the
|
||||
owner is paged hard (Slack + ntfy).
|
||||
- Fix-forward can stack up to 5 commits on a bad change before freezing; the
|
||||
60-minute cap bounds the churn window.
|
||||
- Per-issue spend is capped at `max_budget_usd = 100`.
|
||||
- Guardrails still hold underneath this posture: no PVC/PV deletes, no direct
|
||||
Vault edits, no force-push, infra changes Terraform-only, never `[ci skip]`.
|
||||
- Reversible: tightening to adaptive/PR or to auto-revert is a config + watcher
|
||||
change, not a re-architecture — but callers/habits will have formed around
|
||||
"it just lands", so flag loudly if reversing.
|
||||
70
docs/adr/0003-t3-thin-executor-and-cockpit.md
Normal file
70
docs/adr/0003-t3-thin-executor-and-cockpit.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# AFK workers run inside a dedicated T3 Code instance; claude-agent-service dispatches into it
|
||||
|
||||
The owner wants one UI to see and converse with every in-flight AFK worker, and
|
||||
named **T3 Code** (the self-hosted multi-agent cockpit already running at
|
||||
`t3.viktorbarzin.me`) as that UI. Research into T3's source
|
||||
(`pingdotgg/t3code`, ~v0.0.27) found it is genuinely built for this — a fleet of
|
||||
worker "threads" with a live read-model and a scoped HTTP dispatch API — **but**
|
||||
it can only display sessions **it launched itself**; there is no command to adopt
|
||||
a session another process started. So "viewable in T3" ⟺ "launched by T3". This
|
||||
ADR records the resulting architecture: `claude-agent-service` stays the
|
||||
**control plane** and **dispatches into a dedicated, in-cluster T3 instance**
|
||||
which is the **executor + cockpit**. The agent runs inside T3; we keep the brain.
|
||||
|
||||
## Status
|
||||
|
||||
accepted (2026-06-14) — direction decided; **gated on a pilot** (the five
|
||||
unknowns in the design doc) before the poller is wired and the architecture is
|
||||
committed.
|
||||
|
||||
## Why T3, and why "thin"
|
||||
|
||||
T3 provides, out of the box, what we would otherwise hand-build: a three-panel
|
||||
fleet cockpit (`projects → threads → conversation`), an
|
||||
`OrchestrationReadModel` with per-thread live status, and
|
||||
`POST /api/orchestration/dispatch` whose `thread.turn.start` + `bootstrap` can
|
||||
**create a thread, prepare a git worktree, run a setup script, and deliver a
|
||||
prompt in one call** — exactly the worker-spawn primitive. Converse / approve /
|
||||
resume are native (`thread.user-input.respond`, `thread.approval.respond`). For
|
||||
Claude it embeds `@anthropic-ai/claude-agent-sdk`.
|
||||
|
||||
"Thin" = the AFK *behavior and safety* (the `issue-implementer` prompt,
|
||||
guardrails, always-push, fix-forward/freeze, CI-watch, issue integration) live
|
||||
in **our** layer (the poller + watcher), not in T3. T3 is a **swappable backend**
|
||||
we drive over its API.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **Thin: claude-agent-service dispatches into T3 (chosen).** Control plane calls
|
||||
T3's dispatch API; T3 runs the agent in a worktree and shows it. Get the fleet
|
||||
view, keep the brain, least to build. Cost: execution moves into the T3 pod, so
|
||||
T3's runtime is in the *hot path* (not just the window).
|
||||
- **claude-agent-service runs the agent, T3 only displays it** — rejected because
|
||||
it is impossible: T3 cannot adopt an externally-started session
|
||||
(`thread.session.set` is server-internal; no external-session-id field). This
|
||||
is the constraint that shaped the whole decision.
|
||||
- **Deep: claude-agent-service as a custom T3 provider (ACP-style)** — rejected
|
||||
for now: keeps the runtime ours with a T3 UI, but means building and
|
||||
maintaining a provider against a pre-1.0, internal, no-contributions interface
|
||||
— effectively a fork. Revisit only if "thin" proves too limiting.
|
||||
- **Skip T3; build our own console** (generalized breakglass + tmux) — rejected:
|
||||
most stable and fully in-house, but abandons the owner's explicit "see workers
|
||||
in T3" goal and means owning a session console forever.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A **dedicated in-cluster T3 instance** (a pod, consistent with the earlier
|
||||
in-cluster-over-devvm substrate choice) is the worker host, separate from the
|
||||
per-user devvm T3 instances. It needs the SSD worktree volume, git/Anthropic
|
||||
tokens, toolchains, `claude auth`, and an internal Authentik-gated ingress.
|
||||
- T3's runtime is now in the **execution hot path** — its maturity affects
|
||||
whether work *runs*, not only whether it can be *seen*. Mitigations: **pin the
|
||||
version and exclude it from Keel** (its churn + hard-cutover auth migrations
|
||||
make auto-upgrade a Keel-class hazard), keep the integration thin and the
|
||||
backend swappable, and **pilot** the five unknowns first.
|
||||
- T3 is **single-operator** — fine here: it matches the already-accepted shared
|
||||
service identity for AFK work.
|
||||
- No outbound webhooks from T3 → the watcher **polls**
|
||||
`GET /api/orchestration/snapshot`.
|
||||
- This supersedes the intermediate ideas of evolving `claude-agent-service` into
|
||||
its own session/tmux/worktree runtime and building a bespoke attach console.
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
# Implementation agents use persistent per-repo checkouts + git worktrees, reversing the throwaway-clone rule for this path
|
||||
|
||||
`2026-06-02-parallel-execution-design.md` deliberately **rejected git worktrees**
|
||||
and chose throwaway `git clone --local` per job, "because worktrees share one
|
||||
`.git` → agents that `git commit`/`pull` still contend — not truly independent".
|
||||
The AFK implementation pipeline
|
||||
(`docs/2026-06-14-afk-implementation-pipeline-design.md`) **reverses that for its
|
||||
own path**: each enrolled repo gets a **persistent checkout**, and each issue
|
||||
runs in a **`git worktree`** off it, on a shared **SSD-NFS** volume. This ADR
|
||||
records why the earlier rejection does not apply here — so the two decisions
|
||||
read as complementary, not contradictory.
|
||||
|
||||
## Status
|
||||
|
||||
accepted (2026-06-14) — for the AFK implementation path only; the existing
|
||||
job-runner (recruiter-triage, nextcloud-todos, etc.) keeps throwaway clones.
|
||||
|
||||
## Why the 2026-06-02 rejection doesn't bind this path
|
||||
|
||||
The rejection's premise was **concurrent jobs in the same checkout** contending
|
||||
on `.git/index.lock` and racing `git pull`. The AFK pipeline's concurrency model
|
||||
is **serial within a repo, parallel only across repos** (ADR-adjacent decision in
|
||||
the design doc): at most one agent ever touches a given repo's `.git` at a time,
|
||||
and different repos are different checkouts. The contention the rejection guarded
|
||||
against cannot occur here. With that removed, worktrees become the *better*
|
||||
choice because they unlock cache reuse the throwaway model can't.
|
||||
|
||||
## Considered options
|
||||
|
||||
- **Persistent checkout + worktree per issue, on SSD-NFS (chosen).** Warm git
|
||||
objects, **persisted `node_modules`/venv/build caches**, and shared
|
||||
package-manager caches survive across jobs, so the TDD loop stops reinstalling
|
||||
deps every run. Compounds with `to-issues` clustering many slices in one repo,
|
||||
processed serially — slice N reuses slice 1's warm tree.
|
||||
- **Throwaway `git clone --local` per job (status quo elsewhere)** — rejected for
|
||||
this path: correct for the concurrent job-runner, but re-pays dependency
|
||||
install on every issue, which dominates wall-clock for an
|
||||
implement-test-fix-forward loop.
|
||||
- **`cp -a` of a warm tree** — rejected (same reason as 2026-06-02): copies
|
||||
accumulated caches → disk blowup, and no git isolation.
|
||||
|
||||
## Considered options — storage
|
||||
|
||||
- **SSD-NFS (chosen).** The current `/persistent` PVC is `5Gi` **HDD NFS**
|
||||
(`nfs-truenas` → `/srv/nfs`) and unused; git checkouts + `node_modules` are
|
||||
death-by-small-files on HDD NFS and 5Gi is too small. Provision an SSD-backed
|
||||
NFS class over `/srv/nfs-ssd` (other apps already use that path) at a realistic
|
||||
size (tens of GB).
|
||||
- **HDD NFS / `/persistent` as-is** — rejected: too slow for many small files,
|
||||
too small.
|
||||
- **Local block (proxmox-lvm)** — rejected: faster but HDD and node-pinned (RWO),
|
||||
lost on reschedule; NFS RWX survives and the volume also holds session state.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One **SSD-NFS volume** holds, per enrolled repo: the persistent checkout, the
|
||||
warm dep/package caches, and (under ADR 0003) the worktrees T3 prepares. Cache
|
||||
env (`pip`, `GOMODCACHE`/`GOCACHE`, `PNPM_HOME`/npm, cargo) must be wired to it
|
||||
— today caching is off (`pip --no-cache-dir`, no cache envs set).
|
||||
- Housekeeping the throwaway model didn't need: `git fetch` before each
|
||||
`worktree add`, periodic `git worktree prune` + `git gc`, and cache eviction if
|
||||
the volume fills.
|
||||
- **`infra` stays on its own path** — it is git-crypt, and editing encrypted
|
||||
files from a worktree is disallowed; the persistent-worktree model is for the
|
||||
non-`infra` app repos in the allowlist.
|
||||
- Open reconciliation (pilot): whether T3's native `prepareWorktree` writes into
|
||||
this volume + our persistent checkouts, or we manage the checkout and point T3
|
||||
at it. Resolve before committing the architecture.
|
||||
Loading…
Add table
Add a link
Reference in a new issue