claude-agent-service/docs/2026-06-02-parallel-execution-design.md

# Parallel, independent execution — design

**Date:** 2026-06-02
**Status:** approved, in implementation
**Scope:** `claude-agent-service` — remove the single-flight execution lock so
multiple agent calls run concurrently, each in its own isolated workspace.

## Problem

Today a single global `asyncio.Lock` (`execution_lock`) serializes **every**
agent invocation:

- `POST /execute` returns `409 Agent is busy` when a job is in flight.
- `POST /v1/chat/completions` returns `503 agent is busy` likewise.
- All calls run `claude -p` with `cwd=/workspace/infra` — one shared working
  tree, `git pull --rebase`'d before each call.

The lock exists because two `claude -p` processes in the *same* working tree
would clobber each other's file edits and git state (`.git/index.lock`
contention, racing `git pull --rebase`).

## Goal

Run calls **in parallel**, each **fully independent** of the others, without
the git/file collisions that the lock currently prevents — on a single pod
(`replicas=1`), keeping the in-memory job registry coherent for `/jobs/{id}`
polling.

## Design

### Workspace isolation — per-job local clone

Each job gets its **own git checkout** so file edits and git operations never
touch another job's state:

1. A warm **base clone** lives at `/workspace/base` (created by the existing
   init container; renamed from `/workspace/infra`), git-crypt-unlocked.
2. Per job, under a short-held `git_lock`:
   - Debounced `git fetch origin && git reset --hard origin/master` on the base
     (skipped if fetched within `FETCH_DEBOUNCE_SECONDS`) so bursts share one
     network fetch.
   - `git clone --local /workspace/base /workspace/jobs/<id>` — objects are
     hardlinked (near-free disk, no `.terraform` carried since clone takes
     tracked content only).
   - Re-point `origin` to the GitHub URL and `git-crypt unlock <key>` in the
     job dir.
3. The job runs `claude -p` with `cwd=/workspace/jobs/<id>` **holding no lock**.
4. `finally` → `rm -rf /workspace/jobs/<id>`.

`git_lock` is held only for the fast setup/teardown (~<2 s); execution is fully
parallel. Rejected alternatives: **git worktree** (shares one `.git` → agents
that `git commit`/`pull` still contend — not truly independent) and **`cp -a`**
(copies accumulated `.terraform` provider caches → disk blowup).

Distinct `cwd` per job also isolates Claude CLI per-project state
(`~/.claude/projects/<cwd-hash>/`). The long-lived `CLAUDE_CODE_OAUTH_TOKEN`
avoids credential-file write races in the shared `~/.claude`.

### Concurrency model

- `execution_semaphore = asyncio.Semaphore(MAX_CONCURRENCY)` replaces
  `execution_lock`. Default **`MAX_CONCURRENCY=10`** ("soft-unbounded").
- Requests beyond the limit **queue FIFO** (asyncio fairness) — they are not
  rejected.
- `MAX_QUEUE_DEPTH` safety valve (default **100**): if `active + queued` exceeds
  it, reject (`429` on `/execute`, `503` on chat) to bound memory.
- A `concurrency_slot()` async context manager wraps acquire/release and keeps
  `inflight_active` / `inflight_queued` counters for `/health`.

### Endpoint behavior

| Endpoint | Before | After |
|---|---|---|
| `POST /execute` | `202` or `409` busy | `202` always (unless queue full → `429`); job `status="queued"` until a slot frees, then `running`. **Timeout clock starts on execution, not queue-wait.** |
| `POST /v1/chat/completions` | `200` or `503` busy | **queues** for a slot (caller waits, bounded by the 900 s timeout); still `503` on execution failure/timeout or if queue full |
| `GET /jobs/{id}` | unchanged | unchanged (can now report `queued`) |
| `GET /health` | `{status, busy=lock.locked()}` | `{status, busy=(active>=capacity), active, queued, capacity}` — keeps BeadBoard `/api/agent-status` + beads-dispatcher working |

### Housekeeping

- **Job eviction**: completed/failed/timeout/error jobs older than
  `JOB_TTL_SECONDS` (default 3600) are evicted; the in-memory `jobs` dict
  currently grows unbounded and parallelism increases churn.
- Pod restart still loses in-flight jobs (pre-existing; out of scope — no
  shared store, matching the in-pod decision).

### Infra (`infra/stacks/claude-agent-service/main.tf`)

- Mount the existing `git-crypt-key` configmap into the **main container**
  (today only the init container has it) — needed for per-job unlock.
- Pod memory: request `2Gi`, limit `12Gi` (Burstable, tier-aux); CPU request
  `1`, no CPU limit. Fits node2/3/5 headroom (~22–26 GB free).
- Wire `MAX_CONCURRENCY` env. Rename init-container clone target to
  `/workspace/base`; `WORKSPACE_DIR`→ base path.
- `replicas=1`, `Recreate` unchanged.

## Blast radius (verified)

All callers handle the busy responses gracefully or fail safely, so removing
them is safe:

- **n8n DIUN** (`/execute`) — rate-limited 5/6h, no retry; 409 was rare.
- **payslip-ingest** (`/execute`+poll) — 90× retry; big win from parallelism.
- **recruiter-responder** (`/execute`+poll) — returns `busy`, OpenClaw retries.
- **fire-planner** (`/v1/chat/completions`) — client-side semaphore; can be
  relaxed after this.
- **BeadBoard** (`/execute`) — UI shows busy via `/api/agent-status` (`/health`).
- **beads-dispatcher** CronJob — gates on `/health` busy; 2-min tick.

## Testing (TDD)

Rewrite `test_execute_respects_sequential_lock` and
`test_chat_completions_returns_503_when_agent_busy` (they encode the removed
behavior). New tests: two concurrent `/execute` both run; safety-queue at
`MAX_CONCURRENCY=2`; concurrent chat-completions both run; `/health` capacity
fields; per-job distinct workspace `cwd`; timeout excludes queue-wait; job
eviction; queue-depth rejection. An autouse fixture resets semaphore + counters
+ jobs between tests.

## Docs to update (same change)

`infra/docs/architecture/automated-upgrades.md`,
`infra/docs/runbooks/beads-auto-dispatch.md`, `infra/AGENTS.md`, root
`CLAUDE.md` — all currently describe "sequential / single-slot".