diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 8d281743..e2a57542 100755 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -28,9 +28,16 @@ Violations cause state drift, which causes future applies to break or silently r - **Apply**: Authenticate via `vault login -method=oidc`, then use `scripts/tg` (preferred — handles state decrypt/encrypt) or `terragrunt` directly. `scripts/tg` adds `-auto-approve` for `--non-interactive` applies. - **New services need CI/CD** and **monitoring** (Prometheus/Uptime Kuma) - **New service**: Use `setup-project` skill for full workflow -- **Ingress**: `ingress_factory` module. Auth: `protected = true`. Anti-AI: on by default. **DNS**: `dns_type = "proxied"` (Cloudflare CDN) or `"non-proxied"` (direct A/AAAA). DNS records are auto-created — no need to edit `config.tfvars`. +- **Ingress**: `ingress_factory` module. **Auth** (`auth` string enum, default `"required"` — fail-closed). Pick by asking "what gates the app?": + - `auth = "required"` — Authentik forward-auth gates every request. Use when the backend has **no built-in user auth** and Authentik is the only thing standing between strangers and the app (prowlarr, qbittorrent, netbox, phpipam, k8s-dashboard, foolery, any admin UI shipped without its own login). + - `auth = "app"` — the backend handles its own user authentication (NextAuth, Django, OAuth, bearer-token API, etc.); Authentik would only break it. No middleware attached; the app's own login is the gate. Examples: immich, linkwarden, tandoor, freshrss, affine, actualbudget, audiobookshelf, novelapp. **Functionally identical to `"none"`** — the distinct name exists to record intent at the call site. + - `auth = "public"` — Authentik anonymous binding via the dedicated `public` outpost (routes via `traefik-authentik-forward-auth-public` → `ak-outpost-public.authentik.svc:9000`). Strangers auto-bound to `guest`; logged-in users keep their identity in `X-authentik-username`. **Only works for top-level browser navigation** — CORS preflight rejects XHR/fetch and automation can't replay the cookie dance. Audit trail, not a gate. + - `auth = "none"` — no Authentik, no own-auth claim. Use for Anubis-fronted content (Anubis is the gate), native-client APIs (Git, `/v2/`, WebDAV/CalDAV, CardDAV), webhook receivers, OAuth callbacks, and Authentik outposts themselves. + - **Anti-exposure rule** (the reason `"app"` exists): only pick `"app"` or `"none"` AFTER you've verified the app has its own user auth (`"app"`) OR the endpoint is intentionally public (`"none"`). Default is `"required"` so accidental omission fails closed. **Convention**: when using `"app"` or `"none"`, add a comment line above the `auth = "..."` line stating what gates the app or why it's public. **Enforced by `scripts/tg`**: every `tg plan/apply/destroy/refresh` runs `scripts/check-ingress-auth-comments.py` against the current stack and aborts if any `auth = "app|none"` line lacks the preceding `# auth = "": ...` comment. Stack-scoped — untouched stacks aren't blocked until they're next edited. + - **Anti-AI**: on by default when `auth = "none"` or `auth = "app"` (no Authentik to discourage bots); redundant on `"required"` and `"public"`. + - **DNS**: `dns_type = "proxied"` (Cloudflare CDN) or `"non-proxied"` (direct A/AAAA). DNS records are auto-created — no need to edit `config.tfvars`. Smoke-test target: `echo.viktorbarzin.me` (auth=public, header-reflecting backend). - **Anubis PoW challenge** (`modules/kubernetes/anubis_instance/`): per-site reverse proxy that issues a 30-day JWT cookie after a tiny PoW solve. Use for **public, content-bearing sites without app-level auth** (blog, docs, wikis, static landing pages). Pattern: declare `module "anubis" { source = "../../modules/kubernetes/anubis_instance"; name = "X"; namespace = ...; target_url = "http://..svc.cluster.local" }`, then in `ingress_factory` set `service_name = module.anubis.service_name`, `port = module.anubis.service_port`, `anti_ai_scraping = false`. Shared ed25519 key in Vault `secret/viktor` -> `anubis_ed25519_key`; cookie scoped to `viktorbarzin.me` so one solve covers all Anubis-fronted subdomains. **DO NOT put Anubis in front of Git/API/WebDAV/CLI endpoints** — clients without JS can't solve PoW. **Replicas default to 1** because Anubis stores in-flight challenges in process memory; a challenge issued by pod A and solved against pod B errors with `store: key not found` (HTTP 500). Bumping replicas requires wiring a shared Redis store (TODO). For path-level carve-outs (e.g. wrongmove has `/` behind Anubis but `/api` direct), declare a second `ingress_factory` with `ingress_path = ["/api"]` pointing at the bare backend service. Active on: blog, www, kms, travel, f1, cc, json, pb (privatebin), home (homepage), wrongmove (UI only). See `.claude/reference/patterns.md` "Anti-AI Scraping" for full layering. -- **Docker images**: Always build for `linux/amd64`. Use 8-char git SHA tags — `:latest` causes stale pull-through cache. +- **Docker images**: Always build for `linux/amd64`. SHA-tag rule is being phased out — see `docs/plans/2026-05-16-auto-upgrade-apps-{design,plan}.md`. New model: CI pushes `:latest` (optionally also `:<8-char-sha>` for traceability), Keel polls and triggers rollouts. Cache-staleness concern from the old rule is resolved at the nginx layer (URL-split — manifests pass through, blobs cached). Until Phase 1 of the migration completes (per the plan), follow the SHA-tag rule for new services to match existing pattern. - **Private registry**: `forgejo.viktorbarzin.me/viktor/` (Forgejo packages, OAuth-style PAT auth). Use `image: forgejo.viktorbarzin.me/viktor/:` + `imagePullSecrets: [{name: registry-credentials}]`. Kyverno auto-syncs the Secret to all namespaces. Containerd `hosts.toml` on every node redirects to in-cluster Traefik LB `10.0.20.200` to avoid hairpin NAT. Push-side: viktor PAT in Vault `secret/ci/global/forgejo_push_token` (Forgejo container packages are scoped per-user; only the package owner can push, ci-pusher cannot write to viktor/*). Pull-side: cluster-puller PAT in Vault `secret/viktor/forgejo_pull_token`. Retention CronJob (`forgejo-cleanup` in `forgejo` ns, daily 04:00) keeps newest 10 versions + always `:latest`; integrity probed every 15min by `forgejo-integrity-probe` in `monitoring` ns (catalog walk + manifest HEAD on every blob). See `docs/plans/2026-05-07-forgejo-registry-consolidation-{design,plan}.md` for the migration history. Pull-through caches for upstream registries (DockerHub, GHCR, Quay, k8s.gcr, Kyverno) stay on the registry VM at `10.0.20.10` ports 5000/5010/5020/5030/5040 — the old port-5050 R/W private registry was decommissioned 2026-05-07. - **LinuxServer.io containers**: `DOCKER_MODS` runs apt-get on every start — bake slow mods into a custom image (`RUN /docker-mods || true` then `ENV DOCKER_MODS=`). Set `NO_CHOWN=true` to skip recursive chown that hangs on NFS mounts. - **Node memory changes**: When changing VM memory on any k8s node, update kubelet `systemReserved`, `kubeReserved`, and eviction thresholds accordingly. Config: `/var/lib/kubelet/config.yaml`. Template: `stacks/infra/main.tf`. Current values: systemReserved=512Mi, kubeReserved=512Mi, evictionHard=500Mi, evictionSoft=1Gi. @@ -129,7 +136,7 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle | Frigate | GPU stall detection in liveness probe (inference speed check), high CPU | | Authentik | 3 replicas, PgBouncer in front of PostgreSQL, strip auth headers before forwarding | | Kyverno | failurePolicy=Ignore to prevent blocking cluster, pin chart version | -| MySQL Standalone | Raw `kubernetes_stateful_set_v1` with `mysql:8.4` (migrated from InnoDB Cluster 2026-04-16). `skip-log-bin`, `innodb_flush_log_at_trx_commit=2`, `innodb_doublewrite=ON`. ConfigMap `mysql-standalone-cnf`. PVC `data-mysql-standalone-0` (15Gi, `proxmox-lvm-encrypted`). Service `mysql.dbaas` unchanged. Anti-affinity excludes k8s-node1. Old InnoDB Cluster + operator still in TF (Phase 4 cleanup pending). Bitnami charts deprecated (Broadcom Aug 2025) — use official images. | +| MySQL Standalone | Raw `kubernetes_stateful_set_v1` pinned to `mysql:8.4.8` exactly (migrated from InnoDB Cluster 2026-04-16; **pinned to 8.4.8 on 2026-05-18** after Keel-driven `mysql:8.4` → 8.4.9 bump stalled the DD upgrade and required a full PVC-wipe + dump-restore — see `docs/runbooks/restore-mysql.md` and beads code-eme8/code-k40p). `skip-log-bin`, `innodb_flush_log_at_trx_commit=2`, `innodb_doublewrite=ON`. ConfigMap `mysql-standalone-cnf`. PVC `data-mysql-standalone-0` (5Gi initial → 30Gi via autoresizer, `proxmox-lvm-encrypted`). Service `mysql.dbaas` unchanged. Anti-affinity excludes k8s-node1. Bitnami charts deprecated (Broadcom Aug 2025) — use official images. | | phpIPAM | IPAM — no active scanning. `pfsense-import` CronJob (hourly) pulls Kea leases + ARP via SSH. `dns-sync` CronJob (15min) bidirectional sync with Technitium. Kea DDNS on pfSense handles all 3 subnets. API app `claude` (ssl_token). | ## Monitoring & Alerting @@ -140,6 +147,17 @@ Repo IDs: infra=1, Website=2, finance=3, health=4, travel_blog=5, webhook-handle - Key alerts: OOMKill, pod replica mismatch, 4xx/5xx error rates, UPS battery, CPU temp, SSD writes, NFS responsiveness, ClusterMemoryRequestsHigh (>85%), ContainerNearOOM (>85% limit), PodUnschedulable, ExternalAccessDivergence. - **E2E email monitoring**: CronJob `email-roundtrip-monitor` (every 20 min) sends test email via Brevo HTTP API to `smoke-test@viktorbarzin.me` (catch-all → `spam@`), verifies IMAP delivery, deletes test email, pushes metrics to Pushgateway + Uptime Kuma. Alerts: `EmailRoundtripFailing` (60m), `EmailRoundtripStale` (60m), `EmailRoundtripNeverRun` (60m). Outbound relay: Brevo EU (`smtp-relay.brevo.com:587`, 300/day free — migrated from Mailgun). Inbound external traffic enters via pfSense HAProxy on `10.0.20.1:{25,465,587,993}`, which forwards to k8s `mailserver-proxy` NodePort (30125-30128) with `send-proxy-v2`. Mailserver pod runs alt PROXY-speaking listeners (2525/4465/5587/10993) alongside stock PROXY-free ones (25/465/587/993) for intra-cluster clients. Real client IPs recovered from PROXY v2 header despite kube-proxy SNAT (replaces pre-2026-04-19 MetalLB `10.0.20.202` ETP:Local scheme; see bd code-yiu + `docs/runbooks/mailserver-pfsense-haproxy.md`). Vault: `brevo_api_key` in `secret/viktor` (probe + relay). +## Security Posture (Wave 1 — locked 2026-05-18) + +Plan in `docs/architecture/security.md` + response playbook in `docs/runbooks/security-incident.md`. Beads epic: `code-8ywc`. + +- **Identity allowlist for security rules**: ONLY `me@viktorbarzin.me`. NOT `viktor@viktorbarzin.me`, NOT `emo@viktorbarzin.me` (those don't exist). emo's identity scheme is unknown — ask before assuming. +- **Source-IP allowlist (K2, K9, V7, S1)**: `10.0.20.0/22`, `192.168.1.0/24` (Proxmox + Sofia LAN), K8s pod CIDR, K8s service CIDR, Headscale tailnet. **Policy: no public-IP access** — Vault, kube-apiserver, PVE sshd must transit LAN or Headscale. +- **Response model**: (I) Slack-only daily skim. All security alerts via Loki ruler → Alertmanager → `#security` Slack receiver. Single channel with severity labels inside (critical/warning/info). No paging. +- **Kyverno policies (wave 1)**: `deny-privileged-containers`, `deny-host-namespaces`, `restrict-sys-admin`, `require-trusted-registries` flip Audit→Enforce with the 31-namespace exclude list (memory id=1970). `failurePolicy: Ignore` preserved. Cosign `verify-images` deferred. +- **NetworkPolicy default-deny egress (wave 1)**: observe-then-enforce (γ approach) — Calico flow logs cluster-wide + GlobalNetworkPolicy log-only on tier 3+4, build empirical allowlist after 1 week, phased per-namespace enforce starting `recruiter-responder`. Tier 0/1/2 deferred. +- **What's NOT in scope**: canary tokens (rejected — self-trigger risk with Viktor's normal `vault kv list secret/viktor` and `kubectl get secret -A` workflows), Falco/Tetragon (too noisy for Slack-only daily check), Cloudflare/GitHub audit polling (deferred to wave 2). + ## Storage & Backup Architecture ### Storage Class Decision Rule (for new services) @@ -177,7 +195,7 @@ resource "kubernetes_persistent_volume_claim" "data_proxmox" { name = "-data-proxmox" namespace = kubernetes_namespace..metadata[0].name annotations = { - "resize.topolvm.io/threshold" = "80%" + "resize.topolvm.io/threshold" = "10%" "resize.topolvm.io/increase" = "100%" "resize.topolvm.io/storage_limit" = "5Gi" } @@ -213,7 +231,7 @@ resource "kubernetes_persistent_volume_claim" "data_encrypted" { name = "-data-encrypted" namespace = kubernetes_namespace..metadata[0].name annotations = { - "resize.topolvm.io/threshold" = "80%" + "resize.topolvm.io/threshold" = "10%" "resize.topolvm.io/increase" = "100%" "resize.topolvm.io/storage_limit" = "5Gi" } @@ -269,7 +287,8 @@ resource "kubernetes_persistent_volume_claim" "data_encrypted" { ## Known Issues - **CrowdSec Helm upgrade times out**: `terragrunt apply` on platform stack causes CrowdSec Helm release to get stuck in `pending-upgrade`. Workaround: `helm rollback crowdsec -n crowdsec`. Root cause: likely ResourceQuota CPU at 302% preventing pods from passing readiness probes. Needs investigation. -- **OpenClaw config is writable**: OpenClaw writes to `openclaw.json` at runtime (doctor --fix, plugin auto-enable). Never use subPath ConfigMap mounts for it — use an init container to copy into a writable volume. Needs 2Gi memory + `NODE_OPTIONS=--max-old-space-size=1536`. +- **OpenClaw config is writable**: OpenClaw writes to `openclaw.json` at runtime (doctor --fix, plugin auto-enable). Never use subPath ConfigMap mounts for it — use an init container to copy into a writable volume. Needs 2Gi memory + `NODE_OPTIONS=--max-old-space-size=1536`. **`mcp.servers` baked into the ConfigMap-loaded openclaw.json gets stripped by `doctor --fix`** — register MCP servers via `openclaw mcp set ` in the container startup command instead (CLI-written entries persist across doctor runs). Current servers wired this way: `ha`, `context7`, `playwright` (sidecar at `localhost:3000/mcp`). +- **OpenClaw memory-core indexes `/workspace/memory/`, not `/home/node/.openclaw/memory/`**: `/home/node/.openclaw/memory/main.sqlite` is the index store, NOT a content source. Files written under `/home/node/.openclaw/memory/projects//*.md` will NOT be indexed. To populate memory-core, write Markdown under `/workspace/memory/projects//` and run `openclaw memory index --force`. This is what the daily `memory-sync` CronJob in `stacks/openclaw/` does for claude-memory → OpenClaw sync. - **Goldilocks VPA sets limits**: When increasing memory requests, always set explicit `limits` too — Goldilocks may have added a limit that blocks the change. ## User Preferences diff --git a/.claude/agents/k8s-version-upgrade.deprecated.md b/.claude/agents/k8s-version-upgrade.deprecated.md new file mode 100644 index 00000000..fd0f774b --- /dev/null +++ b/.claude/agents/k8s-version-upgrade.deprecated.md @@ -0,0 +1,543 @@ +--- +name: k8s-version-upgrade-DEPRECATED +description: "DEPRECATED 2026-05-11 — replaced by the Job-chain in stacks/k8s-version-upgrade. See header below." +tools: Read, Write, Edit, Bash, Grep, Glob +model: opus +--- + +# DEPRECATED — Do NOT invoke this agent + +Retired **2026-05-11** after a self-preemption incident: this agent ran inside +the `claude-agent-service` Deployment (replicas=1, no nodeSelector) and was +scheduled onto k8s-node4. When the agent tried to `kubectl drain k8s-node4` +(Stage 6, first worker), it evicted itself. The bash process died mid-SSH, +leaving node4 cordoned and the cluster half-upgraded (master at v1.34.7, +workers at v1.34.2). + +## Replaced by + +A chain of small Kubernetes Jobs, each pinned (via `nodeSelector` + +`kubernetes.io/hostname`) to a node that is NOT its drain target. No pod can +preempt itself because each Job's pod and its target node are always +different. + +| Old | New | +|-----|-----| +| Single agent run in claude-agent-service pod | Chain of 7 phase Jobs (preflight → master → worker × 4 → postflight) | +| Whole pipeline in one prompt | Phase body in `stacks/k8s-version-upgrade/scripts/upgrade-step.sh`, dispatched per-phase via `case $PHASE` | +| Detection CronJob POSTs to `claude-agent-service` | Detection CronJob renders Job 0 from `job-template.yaml` via `envsubst` + `kubectl apply` | +| Drain blocks indefinitely on PDB=0 (e.g. single-replica Anubis) | New `predrain_unstick` deletes PDB-blocked pods so drain proceeds | +| `K8sVersionSkew` + `EtcdPreUpgradeSnapshotMissing` alerts | Above + `K8sUpgradeStalled` (in_flight=1 and time()-started_timestamp > 5400s) | + +## Where the logic lives now + +- **`infra/stacks/k8s-version-upgrade/scripts/upgrade-step.sh`** — universal + phase body. Dispatches on `$PHASE`. Each phase spawns the next Job. +- **`infra/stacks/k8s-version-upgrade/job-template.yaml`** — Job template + rendered by `envsubst` at runtime. ConfigMap-mounted at `/template` in + every Job pod. +- **`infra/stacks/k8s-version-upgrade/main.tf`** — Terraform stack: ConfigMaps, + unified `k8s-upgrade-job` ServiceAccount + RBAC, detection CronJob. +- **`infra/docs/runbooks/k8s-version-upgrade.md`** — operator runbook (kill a + stuck Job, skip a phase, manually re-trigger from a specific phase). + +## Why kept (not deleted) + +Documents the prompted-agent design and is useful as historical reference when +reading post-mortem discussions or comparing approaches. The `name` field has +been suffixed with `-DEPRECATED` so the agent cannot be invoked by name from +`claude-agent-service`. + +--- + +# Original prompt — DO NOT EXECUTE (reference only) + +You are the K8s Version Upgrade Agent for a 5-node home-lab Kubernetes cluster (1 master, 4 workers, stacked etcd, no HA). + +## Your Job + +Given a target patch or minor version of `kubeadm`/`kubelet`/`kubectl`, you orchestrate the full rolling upgrade with safety gates between every node. You do NOT decide WHEN to run — the `k8s-version-check` CronJob in the `k8s-upgrade` namespace fires you off after detection. You only run when invoked. + +The sequence (Pre-flight → etcd snapshot → master containerd skew fix → apt repo URL change [minor only] → master kubeadm upgrade → workers sequentially → Post-flight) is non-negotiable. Skipping a step is how clusters die. + +## Inputs + +The user prompt contains a JSON object with these fields: + +```json +{ + "target_version": "1.34.5", + "kind": "patch", + "dry_run": false, + "stages": "all" +} +``` + +| Field | Required | Description | +|---|---|---| +| `target_version` | yes | Exact `X.Y.Z` to land on (e.g. `1.34.5`). The script `infra/scripts/update_k8s.sh` accepts this via `--release`. | +| `kind` | yes | `patch` (no apt-repo URL change) or `minor` (rewrite repo to v$NEW_MINOR/deb on every node before kubeadm). | +| `dry_run` | no, default false | If true, run all SSH + kubectl READ commands but skip every mutating command (`apt-get install`, `kubeadm upgrade apply`, `kubeadm upgrade node`, `kubectl drain/uncordon`, etcd snapshot, systemctl restart). Log what you would do and exit 0. | +| `stages` | no, default `all` | Comma-separated subset of: `preflight`, `snapshot`, `containerd`, `repo`, `master`, `workers`, `postflight`. Run only those stages and exit. Used by tests. | + +Parse the prompt's first JSON block to extract these. If anything is missing, abort with a Slack notification ("malformed payload"). + +## Environment + +- **Working dir**: `/workspace/infra` (`WORKSPACE_DIR` env var) +- **Kubeconfig**: `/workspace/infra/config` (use `kubectl --kubeconfig $WORKSPACE_DIR/config ...` in every kubectl call) +- **Prometheus**: `http://prometheus-server.monitoring.svc.cluster.local:80` (in-cluster, no auth) +- **Etcd snapshot**: triggered as a one-shot Job from the existing `default/backup-etcd` CronJob (defined in `stacks/infra-maintenance/`). The Job runs on `k8s-master` with hostNetwork (so etcdctl reaches etcd at 127.0.0.1:2379), mounts the PV-backed NFS export `192.168.1.127:/srv/nfs/etcd-backup`, and writes `etcd-snapshot-.db` there. Do NOT shell into master with etcdctl directly — the cert paths + NFS mount are already wired into the CronJob. +- **Library script**: `/workspace/infra/scripts/update_k8s.sh` — pipe via SSH to each node, do NOT modify on the fly. Invoke as `ssh ... 'bash -s' < update_k8s.sh --role --release `. + +### Credentials — fetched at startup + +The k8s-upgrade ServiceAccount has GET on the `k8s-upgrade-creds` Secret in the `k8s-upgrade` namespace (granted by a RoleBinding in `stacks/k8s-version-upgrade/main.tf`). Fetch credentials into `/tmp` files at the start of every run: + +```bash +KUBECTL="kubectl --kubeconfig $WORKSPACE_DIR/config" + +# SSH private key — mode 0400 required by openssh +$KUBECTL get secret -n k8s-upgrade k8s-upgrade-creds \ + -o jsonpath='{.data.ssh_key}' | base64 -d > /tmp/k8s-upgrade-ssh-key +chmod 400 /tmp/k8s-upgrade-ssh-key + +# Slack webhook (URL string) +SLACK_WEBHOOK_K8S_UPGRADE=$($KUBECTL get secret -n k8s-upgrade k8s-upgrade-creds \ + -o jsonpath='{.data.slack_webhook}' | base64 -d) +``` + +The rest of the prompt uses `/tmp/k8s-upgrade-ssh-key` for SSH and `$SLACK_WEBHOOK_K8S_UPGRADE` for Slack. SSH template: + +```bash +SSH="ssh -i /tmp/k8s-upgrade-ssh-key -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/tmp/known_hosts" +``` + +Every SSH call below uses `$SSH wizard@ ''`. `accept-new` accepts the host key on first encounter then pins it — if a node was reimaged, clear `/tmp/known_hosts` before retry. + +## NEVER do + +- Never bypass the halt-on-alert check — even if a single alert "looks unrelated" +- Never start the next worker before the previous one is Ready + all its pods rescheduled + 10-min soak observed +- Never skip the etcd snapshot — even for patch +- Never `kubectl edit/patch/delete` — read-only kubectl plus `drain`/`uncordon` only +- Never `apt-mark hold` something without unholding it first, and vice versa — the script handles this; don't do it manually +- Never run two stages in parallel — sequential only +- Never run if `dry_run=false` AND the cluster has a node Not Ready, or any Upgrade Gates alert firing +- Never push to git, never modify Terraform, never invoke claude-agent-service recursively + +## Slack + Pushgateway helpers + +Every transition posts to Slack: + +```bash +slack() { + local msg="$1" + local hook="${SLACK_WEBHOOK_K8S_UPGRADE:-$SLACK_WEBHOOK_URL}" + curl -sS -X POST -H 'Content-Type: application/json' \ + --data "$(jq -nc --arg t "[k8s-upgrade] $msg" '{text: $t}')" \ + "$hook" +} +``` + +Start every message with `[k8s-upgrade]` so it's grep-able. + +Pushgateway gauges drive the `EtcdPreUpgradeSnapshotMissing` and ops-visibility metrics: + +```bash +PG='http://prometheus-prometheus-pushgateway.monitoring:9091/metrics/job/k8s-version-upgrade' + +push_metric() { + # push_metric + local name="$1" val="$2" + printf '# TYPE %s gauge\n%s %s\n' "$name" "$name" "$val" \ + | curl -sS --data-binary @- "$PG" +} +``` + +Pushes you must make at specific stages (skipped in dry_run): +| When | Metric | Value | +|---|---|---| +| Stage 0 start | `k8s_upgrade_in_flight` | `1` | +| Stage 0 start | `k8s_upgrade_target_minor` | `$target_minor` | +| Stage 2 verified | `k8s_upgrade_snapshot_taken` | `1` | +| Stage 7 clean | `k8s_upgrade_in_flight` | `0` | +| Stage 7 clean | `k8s_upgrade_snapshot_taken` | `0` | + +If you abort mid-flight, leave `k8s_upgrade_in_flight=1` so the alert fires and surfaces the half-done state. + +## Stage 0: Parse inputs + announce + +1. Extract `target_version`, `kind`, `dry_run`, `stages` from the prompt JSON. +2. Derive `target_minor` from `target_version` (split on `.`). +3. Mark the in-flight annotation on the namespace AND push Pushgateway in-flight gauge: + ```bash + if [ "$dry_run" = "false" ]; then + kubectl --kubeconfig $WORKSPACE_DIR/config annotate ns k8s-upgrade \ + viktorbarzin.me/k8s-upgrade-in-flight="$(date -u +%FT%TZ)" \ + viktorbarzin.me/k8s-upgrade-target="$target_version" \ + --overwrite + + push_metric k8s_upgrade_in_flight 1 + push_metric k8s_upgrade_snapshot_taken 0 + fi + ``` +4. Slack: `Starting k8s upgrade to v$target_version (kind=$kind, dry_run=$dry_run, stages=$stages)`. + +## Stage 1: Pre-flight (`stages` includes `preflight`) + +Skip if `stages` excludes `preflight`. + +### Check 1.1 — All nodes Ready, no pressure + +```bash +kubectl --kubeconfig $WORKSPACE_DIR/config get nodes -o json \ + | jq -r '.items[] | "\(.metadata.name): \(.status.conditions[] | select(.type=="Ready") | .status), Mem=\(.status.conditions[] | select(.type=="MemoryPressure") | .status), Disk=\(.status.conditions[] | select(.type=="DiskPressure") | .status)"' +``` + +Abort if any node is not Ready=True, or has MemoryPressure=True or DiskPressure=True. + +### Check 1.2 — Halt-on-alert (same query kured uses) + +```bash +ALERTS=$(curl -sf 'http://prometheus-server.monitoring.svc.cluster.local:80/api/v1/alerts' \ + | jq -r '.data.alerts[] | select(.state == "firing") | .labels.alertname' \ + | grep -vE '^(Watchdog|RebootRequired|KuredNodeWasNotDrained|InfoInhibitor)$' \ + | sort -u) + +if [ -n "$ALERTS" ]; then + slack "ABORT preflight — firing alerts:\n$ALERTS" + exit 1 +fi +``` + +### Check 1.3 — 24h-quiet baseline + +Re-uses the sentinel-gate Check 4 logic from `stacks/kured/main.tf`. Any node that transitioned Ready in the last 24h means the cluster just absorbed a node reboot — we want a clean baseline before starting a fresh rollout. + +```bash +RECENT_REBOOT=0 +while IFS= read -r ts; do + [ -z "$ts" ] && continue + diff=$(( $(date +%s) - $(date -d "$ts" +%s) )) + [ "$diff" -lt 86400 ] && RECENT_REBOOT=1 && break +done < <(kubectl --kubeconfig $WORKSPACE_DIR/config get nodes -o jsonpath='{range .items[*]}{range .status.conditions[?(@.type=="Ready")]}{.lastTransitionTime}{"\n"}{end}{end}') + +if [ "$RECENT_REBOOT" -eq 1 ]; then + slack "ABORT preflight — node transitioned Ready <24h ago (soak window)" + exit 1 +fi +``` + +### Check 1.4 — kubeadm upgrade plan reports our target + +```bash +PLAN_TARGET=$($SSH \ + wizard@k8s-master 'sudo kubeadm upgrade plan' \ + | grep -oE 'You can now apply the upgrade by executing the following command:.*v[0-9]+\.[0-9]+\.[0-9]+' \ + | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1 | tr -d v) +``` + +If `$PLAN_TARGET` does not start with the requested `target_version`, slack-abort: +"`kubeadm upgrade plan` says target is $PLAN_TARGET but caller asked for $target_version — drift; aborting." + +Slack: `Pre-flight clean. Proceeding to etcd snapshot.` + +## Stage 2: Etcd snapshot (`stages` includes `snapshot`) + +Always run — patch OR minor. Triggers a one-shot Job from the existing `default/backup-etcd` CronJob and waits for it to complete. + +```bash +JOB_NAME="pre-upgrade-etcd-${target_version}-$(date +%s)" + +if [ "$dry_run" = "false" ]; then + $KUBECTL -n default create job --from=cronjob/backup-etcd "$JOB_NAME" + + # Wait up to 10 min for snapshot Job to complete + $KUBECTL -n default wait --for=condition=complete --timeout=600s "job/$JOB_NAME" || { + slack "ABORT Stage 2 — etcd snapshot Job did not complete in 10 min" + $KUBECTL -n default describe "job/$JOB_NAME" | tail -30 + exit 1 + } + + # Parse the Job's pod log for "Backup done: ( bytes)" + LOG=$($KUBECTL -n default logs "job/$JOB_NAME" -c backup-manage --tail=20) + echo "$LOG" + SNAPSHOT_LINE=$(echo "$LOG" | grep -E '^Backup done:') + SIZE=$(echo "$SNAPSHOT_LINE" | grep -oE '\([0-9]+ bytes\)' | grep -oE '[0-9]+') + SNAPSHOT_FILE=$(echo "$SNAPSHOT_LINE" | awk '{print $3}') + + if [ -z "$SIZE" ] || [ "$SIZE" -lt 1024 ]; then + slack "ABORT Stage 2 — etcd snapshot empty or missing (size='$SIZE' line='$SNAPSHOT_LINE')" + exit 1 + fi + + TARGET_PATH="nfs://192.168.1.127:/srv/nfs/etcd-backup/$SNAPSHOT_FILE" + $KUBECTL annotate ns k8s-upgrade \ + viktorbarzin.me/k8s-upgrade-snapshot-path="$TARGET_PATH" --overwrite + + push_metric k8s_upgrade_snapshot_taken 1 +else + TARGET_PATH="WOULD: trigger default/backup-etcd Job, wait, verify size" + SIZE="dry-run" +fi + +slack "Etcd snapshot saved at $TARGET_PATH (size=$SIZE)" +``` + +## Stage 3: Master containerd skew fix (`stages` includes `containerd`) + +Only run if master containerd version < highest worker containerd version. + +```bash +get_ctr_version() { + $SSH \ + "wizard@$1" 'containerd --version | awk "{print \$3}" | tr -d v' +} + +MASTER_CTR=$(get_ctr_version k8s-master) +WORKER_MAX="0.0.0" +for n in k8s-node1 k8s-node2 k8s-node3 k8s-node4; do + v=$(get_ctr_version "$n") + # Compare semver-ish + if [ "$(printf '%s\n%s' "$v" "$WORKER_MAX" | sort -V | tail -1)" = "$v" ]; then + WORKER_MAX="$v" + fi +done + +if [ "$(printf '%s\n%s' "$MASTER_CTR" "$WORKER_MAX" | sort -V | head -1)" = "$MASTER_CTR" ] \ + && [ "$MASTER_CTR" != "$WORKER_MAX" ]; then + # Master is behind — bump + slack "Master containerd $MASTER_CTR < workers $WORKER_MAX — bumping master" + + if [ "$dry_run" = "false" ]; then + $SSH \ + wizard@k8s-master "sudo apt-mark unhold containerd.io \ + && sudo apt-get install -y containerd.io='$WORKER_MAX-1' \ + && sudo apt-mark hold containerd.io \ + && sudo systemctl restart containerd" + + # Wait until kubelet on master is Ready again + for i in $(seq 1 60); do + STATUS=$(kubectl --kubeconfig $WORKSPACE_DIR/config get node k8s-master \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') + [ "$STATUS" = "True" ] && break + sleep 10 + done + [ "$STATUS" = "True" ] || { slack "ABORT — k8s-master not Ready after containerd bump"; exit 1; } + fi + + slack "Master containerd: $MASTER_CTR → $WORKER_MAX. Master Ready." +else + echo "Master containerd $MASTER_CTR >= workers max $WORKER_MAX — skipping skew fix" +fi +``` + +## Stage 4: Apt repo URL rewrite for minor bumps (`stages` includes `repo`) + +Only run if `kind=minor`. + +For each of `k8s-master k8s-node1 k8s-node2 k8s-node3 k8s-node4`: + +```bash +target_minor="$(echo "$target_version" | awk -F. '{print $1"."$2}')" + +if [ "$dry_run" = "false" ]; then + $SSH \ + "wizard@$node" "echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v$target_minor/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list \ + && curl -fsSL 'https://pkgs.k8s.io/core:/stable:/v$target_minor/deb/Release.key' | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg --batch --yes \ + && sudo apt-get update" +fi +``` + +Slack: `Repo rewritten to v$target_minor/deb on all 5 nodes.` + +## Stage 5: Master upgrade (`stages` includes `master`) + +```bash +# 5.1 Drain +if [ "$dry_run" = "false" ]; then + kubectl --kubeconfig $WORKSPACE_DIR/config drain k8s-master \ + --ignore-daemonsets --delete-emptydir-data --force --grace-period=300 +fi + +# 5.2 Run the library script via SSH pipe +if [ "$dry_run" = "false" ]; then + $SSH \ + wizard@k8s-master 'bash -s' \ + < $WORKSPACE_DIR/scripts/update_k8s.sh \ + -- --role master --release "$target_version" +fi + +# 5.3 Uncordon + wait Ready +if [ "$dry_run" = "false" ]; then + kubectl --kubeconfig $WORKSPACE_DIR/config uncordon k8s-master +fi + +for i in $(seq 1 60); do + STATUS=$(kubectl --kubeconfig $WORKSPACE_DIR/config get node k8s-master \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') + KUBELET=$(kubectl --kubeconfig $WORKSPACE_DIR/config get node k8s-master \ + -o jsonpath='{.status.nodeInfo.kubeletVersion}' | tr -d v) + [ "$STATUS" = "True" ] && [ "$KUBELET" = "$target_version" ] && break + sleep 15 +done + +[ "$STATUS" = "True" ] && [ "$KUBELET" = "$target_version" ] \ + || { slack "ABORT — master not Ready or wrong version after upgrade ($STATUS / $KUBELET)"; exit 1; } + +# 5.4 All control-plane pods Running +NOT_READY=$(kubectl --kubeconfig $WORKSPACE_DIR/config -n kube-system get pods \ + -l 'tier=control-plane' --no-headers | grep -v Running | wc -l) +[ "$NOT_READY" -gt 0 ] && { slack "ABORT — $NOT_READY control-plane pods not Running"; exit 1; } + +# 5.5 Re-check halt-on-alert +# (re-run the Check 1.2 query, abort if anything new fires) + +slack "Master upgrade complete. Cluster on v$target_version. Healthy." +``` + +## Stage 6: Workers sequentially (`stages` includes `workers`) + +Order: `k8s-node4 → k8s-node3 → k8s-node2 → k8s-node1`. Node1 last because it hosts GPU + Immich and benefits from the longest soak before any other worker is touched (ref: post-mortem-2026-03-16, memory id=570). + +For each worker `$node`: + +1. Re-check halt-on-alert. If anything fires (e.g. `RecentNodeReboot` on the previous worker), wait + retry up to 30 min, then abort. +2. `kubectl drain $node --ignore-daemonsets --delete-emptydir-data --force --grace-period=300` +3. SSH pipe `update_k8s.sh --role worker --release $target_version` +4. `kubectl uncordon $node` +5. Wait until `$node` Ready + kubeletVersion matches + all calico-node + kube-proxy pods on that node Running. +6. **10-min soak**: poll halt-on-alert every 60s. If anything fires, abort. After 10 min clean, proceed. +7. Slack: `Worker $node complete ($i/4)`. + +```bash +WORKERS="k8s-node4 k8s-node3 k8s-node2 k8s-node1" +i=0 +for node in $WORKERS; do + i=$((i+1)) + + # Halt-on-alert recheck with retry + for attempt in $(seq 1 30); do + ALERTS=$(curl -sf 'http://prometheus-server.monitoring.svc.cluster.local:80/api/v1/alerts' \ + | jq -r '.data.alerts[] | select(.state == "firing") | .labels.alertname' \ + | grep -vE '^(Watchdog|RebootRequired|KuredNodeWasNotDrained|InfoInhibitor)$' \ + | sort -u) + [ -z "$ALERTS" ] && break + echo "Waiting for alerts to clear (attempt $attempt/30): $ALERTS" + sleep 60 + done + [ -n "$ALERTS" ] && { slack "ABORT $node — alerts firing after 30min wait: $ALERTS"; exit 1; } + + if [ "$dry_run" = "false" ]; then + kubectl --kubeconfig $WORKSPACE_DIR/config drain "$node" \ + --ignore-daemonsets --delete-emptydir-data --force --grace-period=300 + + $SSH \ + "wizard@$node" 'bash -s' \ + < $WORKSPACE_DIR/scripts/update_k8s.sh \ + -- --role worker --release "$target_version" + + kubectl --kubeconfig $WORKSPACE_DIR/config uncordon "$node" + fi + + # Wait Ready + version match + for w in $(seq 1 60); do + STATUS=$(kubectl --kubeconfig $WORKSPACE_DIR/config get node "$node" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}') + KUBELET=$(kubectl --kubeconfig $WORKSPACE_DIR/config get node "$node" \ + -o jsonpath='{.status.nodeInfo.kubeletVersion}' | tr -d v) + [ "$STATUS" = "True" ] && [ "$KUBELET" = "$target_version" ] && break + sleep 15 + done + [ "$STATUS" = "True" ] && [ "$KUBELET" = "$target_version" ] \ + || { slack "ABORT — $node not Ready or wrong version ($STATUS / $KUBELET)"; exit 1; } + + # 10-min soak with halt-on-alert + echo "Soaking $node for 10 min..." + for sec in $(seq 1 10); do + ALERTS=$(curl -sf 'http://prometheus-server.monitoring.svc.cluster.local:80/api/v1/alerts' \ + | jq -r '.data.alerts[] | select(.state == "firing") | .labels.alertname' \ + | grep -vE '^(Watchdog|RebootRequired|KuredNodeWasNotDrained|InfoInhibitor|RecentNodeReboot)$' \ + | sort -u) + [ -n "$ALERTS" ] && { slack "ABORT $node mid-soak — alerts: $ALERTS"; exit 1; } + sleep 60 + done + + slack "Worker $node upgrade complete ($i/4). Soaked clean." +done +``` + +Note: during the soak we add `RecentNodeReboot` to the ignore-list because we KNOW we just rebooted-as-it-were that node (kubelet restart counts). + +## Stage 7: Post-flight (`stages` includes `postflight`) + +```bash +# All 5 nodes at target +VERSIONS=$(kubectl --kubeconfig $WORKSPACE_DIR/config get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}:{.status.nodeInfo.kubeletVersion}{"\n"}{end}') +echo "$VERSIONS" +WRONG=$(echo "$VERSIONS" | grep -v ":v${target_version}$" | wc -l) +[ "$WRONG" -ne 0 ] && { slack "ABORT post-flight — $WRONG node(s) not on v$target_version:\n$VERSIONS"; exit 1; } + +# Upgrade Gates all inactive +FIRING=$(curl -sf 'http://prometheus-server.monitoring.svc.cluster.local:80/api/v1/alerts' \ + | jq -r '.data.alerts[] | select(.state == "firing") | .labels.alertname' \ + | grep -vE '^(Watchdog|RebootRequired|KuredNodeWasNotDrained|InfoInhibitor)$' \ + | sort -u) +[ -n "$FIRING" ] && slack "Post-flight WARN — alerts still firing (cluster on target, but check):\n$FIRING" + +# pod-ready ratio >= 0.9 +RATIO=$(curl -sf 'http://prometheus-server.monitoring.svc.cluster.local:80/api/v1/query' \ + --data-urlencode 'query=sum(kube_pod_status_ready{condition="true"}) / sum(kube_pod_status_phase{phase="Running"})' \ + | jq -r '.data.result[0].value[1] // "0"') +slack "Pod-ready ratio: $RATIO (target ≥ 0.9)" + +# Clear the in-flight annotation + Pushgateway gauges +if [ "$dry_run" = "false" ]; then + kubectl --kubeconfig $WORKSPACE_DIR/config annotate ns k8s-upgrade \ + viktorbarzin.me/k8s-upgrade-in-flight- \ + viktorbarzin.me/k8s-upgrade-target- \ + viktorbarzin.me/k8s-upgrade-snapshot-path- || true + + push_metric k8s_upgrade_in_flight 0 + push_metric k8s_upgrade_snapshot_taken 0 +fi + +slack ":white_check_mark: K8s upgrade complete: cluster on v$target_version." +``` + +## Rollback + +This agent does NOT auto-rollback. If anything aborts mid-flight: + +1. Slack the failure with the last known stage + node. +2. Leave the in-flight annotation in place (the operator clears it manually after triage). +3. Operator follows `infra/docs/runbooks/k8s-version-upgrade.md` → "Rollback paths" section. + +The etcd snapshot path is annotated on the `k8s-upgrade` namespace for easy recovery. + +## Notes for tests + +- **Test 1 (CronJob dry-run)**: The CronJob has its own `--dry-run` env var that short-circuits before POST. This agent is not invoked. +- **Test 2 (agent dry-run)**: Invoke with `{"dry_run": true}`. Every SSH + kubectl READ runs, every mutation skipped. The agent should print "WOULD: " for each skipped mutation. +- **Test 3 (snapshot-only)**: Invoke with `{"stages": "preflight,snapshot"}`. Pre-flight + etcd snapshot only. Slack notification confirms the file exists. No node touched after that. +- **Test 4 (full run)**: `{"target_version": "1.34.7", "kind": "patch"}` once apt has it. Full sequence. +- **Test 5 (synthetic minor)**: `{"target_version": "1.35.0", "kind": "minor", "dry_run": true}`. Confirms the repo-rewrite plan path without mutation. + +## Edge cases + +- **Slack down**: Don't block the upgrade — continue, log to stderr. +- **SSH host key changes**: `accept-new` accepts only on first encounter — if a node was reimaged its host key changes; clear `/tmp/known_hosts` before retry. +- **kubectl drain hangs on a PDB-violating pod**: 5-min grace-period is hard. If drain fails, `kubectl drain --disable-eviction --force` is NOT a valid escalation here — slack-abort and let the operator investigate. +- **etcd snapshot dir missing/full**: stat the dir first. If <10 GiB free, abort. +- **Network blip during apt-get**: the script `set -e`s — apt-get will fail loud, the agent's bash will see non-zero exit, we slack-abort. The node is left mid-upgrade (kubeadm half-applied). Operator follows the runbook. + +## Verification claims you must make + +When you `slack` a SUCCESS message, you must have actually verified: +- All 5 nodes report the target kubelet version via `kubectl get nodes -o jsonpath` +- No alerts firing outside the ignore-list +- pod-ready ratio computed from Prometheus + +Do not declare success without those three confirmations. diff --git a/.claude/reference/authentik-state.md b/.claude/reference/authentik-state.md index f76dd325..30094776 100644 --- a/.claude/reference/authentik-state.md +++ b/.claude/reference/authentik-state.md @@ -127,10 +127,65 @@ Pinned via Terraform in `stacks/authentik/`: | Knob | Value | Surface | Effect | |------|-------|---------|--------| | `UserLoginStage.session_duration` on `default-authentication-login` | `weeks=4` | `authentik_stage_user_login.default_login` in `authentik_provider.tf` | Authenticated users stay logged in 4 weeks across browser restarts. No sliding refresh — resets on each login. | +| `ProxyProvider.access_token_validity` on `Provider for Domain wide catch all` | `weeks=4` | `authentik_provider_proxy.catchall.access_token_validity` in `authentik_provider.tf` | Cookie `Max-Age` on `authentik_proxy_*` and `expires` on rows in `authentik_providers_proxy_proxysession`. Bumped 2026-05-10 from `hours=168`. **Bumping requires `kubectl rollout restart deploy/ak-outpost-authentik-embedded-outpost`** — the gorilla session store binds the value once at outpost startup; the 5-min provider refresh logs `"reusing existing session store"` and skips rebuild. | | `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE` (server + worker) | `hours=2` | `server.env` + `worker.env` in `modules/authentik/values.yaml` | Anonymous Django sessions (bots, healthcheckers, partial flows) are reaped within 2h instead of the 1d default. | Notes: - There is **no** `Brand.session_duration`; `UserLoginStage` is the only correct lever for authenticated session lifetime. -- Embedded outpost session storage moved from `/dev/shm` → Postgres table `authentik_providers_proxy_proxysession` in authentik 2025.10. The 2026-04-18 `/dev/shm`-fill outage class is no longer load-bearing in 2026.2.2; the `unauthenticated_age` cap is still the right lever for anonymous-session bloat from external monitors. -- `ProxyProvider.access_token_validity` and `remember_me_offset` stay UI-managed via `ignore_changes`. +- Embedded outpost session storage: PostgreSQL table `authentik_providers_proxy_proxysession` in authentik 2025.10+ (PR #16628), but **only when `IsEmbedded()` returns true** (i.e. `Outpost.managed == "goauthentik.io/outposts/embedded"`). Our outpost record had `managed=null` until 2026-05-10, which silently kept it on the gorilla `FilesystemStore` at `/dev/shm` (TMPDIR) and re-exposed the 2026-04-18 mismatched-session-ID class on every pod restart. Fix landed 2026-05-10: see `authentik_outpost.embedded` in `authentik_provider.tf` and post-mortem `2026-04-18-authentik-outpost-shm-full.md`. +- The proxy outpost service has a known goauthentik 2026.2.2 bug (`internal/outpost/controllers/k8s/service.py:52`): for embedded outposts the controller sets the Service selector to `app.kubernetes.io/name=authentik` (the server pods), not `authentik-outpost-proxy`. We work around it via a `kubernetes_json_patches.service` patch on the outpost record (replaces `/spec/selector` with the outpost's own labels). Without this, endpoints are empty and Traefik forward-auth fails over to the Basic Auth realm `Emergency Access`. +- The standalone embedded-outpost deployment needs `AUTHENTIK_POSTGRESQL__{HOST,PORT,USER,PASSWORD,NAME}` env vars to reach the dbaas cluster — codified via `kubernetes_json_patches.deployment` envFrom the shared `goauthentik` Secret. The `app.kubernetes.io/component=server` pod label is also injected via JSON patch (matches the `component:server` half of the Service selector that the controller adds for embedded outposts). +- `ProxyProvider.remember_me_offset` stays UI-managed via `ignore_changes`. +- The Authentik provider's resource schema does **not** expose the `Outpost.managed` field. We rely on TF's "write only fields it knows about" semantic: the server-set `goauthentik.io/outposts/embedded` value is preserved across applies because Terraform never writes `managed`. Don't change the resource provider schema expectations without verifying this assumption holds. - The `unauthenticated_age` env var is injected via `server.env` / `worker.env` (not `authentik.sessions.unauthenticated_age`) because we set `authentik.existingSecret.secretName: goauthentik`, which makes the chart skip rendering its own `AUTHENTIK_*` Secret. The `authentik.*` value block is therefore inert in this stack — anything new under `authentik.*` must use the `*.env` arrays instead. The same applies to the existing `authentik.cache.*`, `authentik.web.*`, `authentik.worker.*` blocks (currently inert; live values come from the orphaned, helm-keep-policy `goauthentik` Secret created by chart 2025.10.3 before `existingSecret` was introduced). + +## Upgrade Validation Checklist + +Run after **any** of these: +- Authentik chart version bump in `stacks/authentik/modules/authentik/main.tf` (the `version = "..."` line on `helm_release.authentik`). +- `goauthentik/authentik` Terraform provider version bump. +- Outpost pod recreation (kured reboot, eviction, manual `rollout restart`, scheduler move). + +The fragile surfaces are the `kubernetes_json_patches` and the `Outpost.managed` field — both rely on assumptions that can silently break across upgrades. The checklist exercises the same path the alerts watch, so it doubles as a smoke test for the alerts. + +```bash +# 1. Service routes to the outpost pod (NOT the server pods). +# Empty endpoints => auth-proxy fallback fires; expected: ONE pod IP, ports 9000/9300/9443. +kubectl -n authentik get endpoints ak-outpost-authentik-embedded-outpost + +# 2. Service selector still excludes the server pods. Expected: includes +# `app.kubernetes.io/name: authentik-outpost-proxy`. If it flips to +# `name: authentik`, the goauthentik upstream bug came back or our +# JSON patch was unset. +kubectl -n authentik get svc ak-outpost-authentik-embedded-outpost -o jsonpath='{.spec.selector}' + +# 3. Outpost mode + session backend. Expected log lines on startup: +# {"embedded":true,"event":"Outpost mode",...} +# {"event":"using PostgreSQL session backend",...} +# If embedded=false or `using filesystem session backend`, the postgres +# fix is broken — likely `Outpost.managed` got cleared, or the upstream +# schema started exposing `managed` and TF reset it. +kubectl -n authentik logs deploy/ak-outpost-authentik-embedded-outpost | grep -E '"Outpost mode"|"session backend"' | head -3 + +# 4. /dev/shm is essentially empty (postgres backend = no filesystem use). +# A row count > a few dozen indicates filesystem fallback is firing. +kubectl -n authentik exec deploy/ak-outpost-authentik-embedded-outpost -- sh -c 'df -h /dev/shm; ls /dev/shm | wc -l' + +# 5. Postgres session table is growing with traffic. Expected: rows with +# `expires` ~28 days out (matches access_token_validity = weeks=4). +kubectl -n authentik exec deploy/goauthentik-server -- ak shell -c " +from django.db import connection; c = connection.cursor() +c.execute('SELECT COUNT(*), MAX(expires) FROM authentik_providers_proxy_proxysession') +print(c.fetchone())" + +# 6. Edge auth flow: should be 302 → authentik. NOT 401 with WWW-Authenticate. +curl -sS -o /dev/null -D - 'https://terminal.viktorbarzin.me/' -H 'User-Agent: Mozilla/5.0' \ + | grep -iE '^HTTP|^location|x-auth-fallback|www-authenticate' + +# 7. Terraform plan-to-zero on the whole authentik stack. +( cd stacks/authentik && /home/wizard/code/infra/scripts/tg plan ) | grep -E 'No changes|Plan:' +``` + +Steps 1, 3, 6 cover the failure modes the Prometheus alerts trigger on (`AuthentikForwardAuthFallbackActive`, `AuthentikOutpostForwardAuth400Spike`). Steps 4 and 5 cover the silent-regression case (filesystem fallback) where the alerts don't fire but the system loses its postgres-backed session persistence on the next pod restart. + +If step 2 shows the controller restored `app.kubernetes.io/name=authentik`, watch goauthentik/authentik issue tracker for fixes around `internal/outpost/controllers/k8s/service.py:52` — the upstream patch might let us drop our `kubernetes_json_patches.service` workaround. diff --git a/.claude/reference/service-catalog.md b/.claude/reference/service-catalog.md index a3619212..4200575a 100644 --- a/.claude/reference/service-catalog.md +++ b/.claude/reference/service-catalog.md @@ -53,6 +53,7 @@ | insta2spotify | Instagram reel song ID to Spotify playlist | insta2spotify | | trading-bot | Event-driven trading with sentiment analysis | trading-bot | | claude-memory | Persistent memory MCP server | claude-memory | +| paperless-mcp | Paperless-ngx document search MCP (barryw/PaperlessMCP). Traefik bearer auth via Aetherinox api-token-middleware. `auth=none` at ingress; gateway-level bearer enforced by `paperless-mcp/bearer-auth` Middleware CRD. Tokens + paperless API token in Vault `secret/paperless-mcp`. | paperless-mcp | | council-complaints | Islington civic reporting pilot | council-complaints | ## Optional @@ -78,6 +79,7 @@ | paperless-ngx | Document management | paperless-ngx | | jsoncrack | JSON visualizer | jsoncrack | | servarr | Media automation (Sonarr/Radarr/etc) | servarr | +| aiostreams | Stremio stream aggregator (Real-Debrid + Torrentio/Comet/MediaFusion/StremThru/Knaben). `auth=app` (own UUID+password); canary stream-probe + 3 alerts; weekly NFS config + Stremio-account-collection backups to `/srv/nfs/aiostreams-backup/`. PG-backed user config. | servarr/aiostreams | | ntfy | Push notifications | ntfy | | cyberchef | Data transformation | cyberchef | | diun | Docker image update notifier — detects new versions, fires webhook to n8n upgrade agent | diun | diff --git a/.claude/skills/cluster-health/SKILL.md b/.claude/skills/cluster-health/SKILL.md index ef3ae25f..82d11c48 100644 --- a/.claude/skills/cluster-health/SKILL.md +++ b/.claude/skills/cluster-health/SKILL.md @@ -7,8 +7,9 @@ description: | (3) User asks to fix stuck pods, evicted pods, or CrashLoopBackOff, (4) User mentions "health check", "cluster status", "cluster health", (5) User asks "is everything running" or "any problems". - Runs 42 cluster-wide checks (nodes, workloads, monitoring, certs, - backups, external reachability) with safe auto-fix for evicted pods. + Runs 44 cluster-wide checks (nodes, workloads, monitoring, certs, + backups, external reachability, PVE host thermals + load) with safe + auto-fix for evicted pods. author: Claude Code version: 2.0.0 date: 2026-04-19 @@ -66,7 +67,7 @@ bash infra/scripts/cluster_healthcheck.sh --no-fix --quiet --json bash infra/scripts/cluster_healthcheck.sh --kubeconfig /path/to/config ``` -## What It Checks (42 checks) +## What It Checks (44 checks) | # | Check | Notes | |---|-------|-------| @@ -112,6 +113,8 @@ bash infra/scripts/cluster_healthcheck.sh --kubeconfig /path/to/config | 40 | External — Cloudflared + Authentik Replicas | deployments fully ready | | 41 | External — ExternalAccessDivergence Alert | alert not firing | | 42 | External — Traefik 5xx Rate (15m) | top-10 services emitting 5xx | +| 43 | PVE Host Thermals | package + per-core temps via `/sys/class/hwmon` (SSH). Baseline 55-65 °C. PASS <65 °C, WARN 65-82 °C (a VM is burning too much CPU), FAIL ≥83 °C (TjMax) | +| 44 | PVE Host Load | `/proc/loadavg` via SSH. PASS 5m <30, WARN 30-37, FAIL ≥38 of 44 threads | ## Safe Auto-Fix Rules @@ -256,9 +259,9 @@ kubectl logs -n external-secrets deploy/external-secrets --tail=100 kubectl get pods -n cloudflared kubectl logs -n cloudflared -l app=cloudflared --tail=100 -# Authentik -kubectl get pods -n authentik -l app=authentik-server -kubectl logs -n authentik -l app=authentik-server --tail=100 +# Authentik (Helm chart names the deployment goauthentik-server) +kubectl get deployment -n authentik goauthentik-server +kubectl logs -n authentik deploy/goauthentik-server --tail=100 # ExternalAccessDivergence alert kubectl exec -n monitoring deploy/prometheus-server -- \ @@ -295,6 +298,133 @@ kubectl exec -n monitoring deploy/prometheus-server -- \ - Exit code 143 → SIGTERM / graceful shutdown failed 3. Cross-check dbaas + NFS + secrets are healthy. +## Performance forensics — top consumers + optimization hints + +When the cluster is healthy (script returns 0) but the host is hot or load +is elevated, switch from "what broke?" to "what's expensive?". Run these +in order; stop as soon as the root cause is obvious. + +### Step 1 — Snapshot top consumers cluster-wide + +```bash +# Top 15 pods by current CPU +kubectl top pods --all-namespaces --sort-by=cpu --no-headers | head -15 + +# Top 5 nodes by CPU + memory pressure +kubectl top nodes + +# Top 15 by 5-min rolling rate (smoothed — kills noise from one-off spikes) +kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \ + "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(namespace,pod)%20(rate(container_cpu_usage_seconds_total%7Bcontainer!%3D''%7D%5B5m%5D)))" \ + | python3 -m json.tool | head -80 +``` + +### Step 2 — For each suspect pod, get the WHY + +For every pod in the top-N, gather these BEFORE proposing a fix: + +```bash +NS=; POD=; CONT=$(kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].name}') + +# What it does (image + command) +kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].image}{"\n"}{.spec.containers[0].args}{"\n"}' + +# Resource limits + current usage +kubectl -n $NS top pod $POD --containers +kubectl -n $NS get pod $POD -o jsonpath='{.spec.containers[0].resources}' + +# Recent logs filtered for reconcile loops, watch storms, slow queries +kubectl -n $NS logs $POD -c $CONT --tail=200 --since=5m 2>&1 \ + | grep -iE 'reconcil|watch|scrape|index|loop|retry|slow|timeout' | tail -20 + +# Restart count + recent OOM +kubectl -n $NS describe pod $POD | grep -E 'Restart Count|Last State|Reason' + +# Self-exported metrics (for apps that publish on /metrics) +kubectl -n $NS exec $POD -c $CONT -- wget -qO- localhost:/metrics 2>/dev/null | head -50 +``` + +### Step 3 — apiserver / etcd specific deep-dive (when control-plane is hot) + +```bash +# Top request producers by verb+resource (last 30 min) +kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \ + "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(resource,verb)%20(rate(apiserver_request_total%5B30m%5D)))" \ + | python3 -m json.tool + +# Top user agents (which clients are hammering) +kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \ + "http://localhost:9090/api/v1/query?query=topk(15,sum%20by%20(user_agent)%20(rate(apiserver_request_total%5B30m%5D)))" \ + | python3 -m json.tool + +# Long-running requests (WATCH / CONNECT — log streams, pod-watchers) +kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \ + "http://localhost:9090/api/v1/query?query=apiserver_longrunning_requests" \ + | python3 -m json.tool + +# etcd write rate + DB size +kubectl -n monitoring exec deploy/prometheus-server -- wget -qO- \ + "http://localhost:9090/api/v1/query?query=rate(etcd_disk_wal_fsync_duration_seconds_count%5B5m%5D)" \ + | python3 -m json.tool +``` + +### Step 4 — PVE host specific deep-dive (when temp / load is high) + +Checks 43 + 44 capture package temp + 5-min load avg with PASS/WARN/FAIL +thresholds — that's the first stop. When those WARN or FAIL, the +follow-up commands below trace which VM / process is the source: + +```bash +# Per-core temps (broader than the package summary in check 43) +ssh root@192.168.1.127 'for f in /sys/class/hwmon/hwmon0/temp*_input; do + base=${f%_input}; label=$(cat ${base}_label 2>/dev/null || echo "${base##*/}") + val=$(cat "$f"); echo " $label: $((val/1000))°C" +done' + +# Per-VM CPU (each VM = one kvm process) +ssh root@192.168.1.127 'top -bn1 -o %CPU | grep kvm | head -10' + +# pvestatd anomaly check — bursts > 50% usually mean LV count > 1000 +ssh root@192.168.1.127 'lvs --noheadings 2>/dev/null | wc -l' + +# Stale snapshots (any '_pre-*' that survived past their rollback window) +ssh root@192.168.1.127 'lvs --noheadings -o lv_name 2>/dev/null | awk "/_pre-/" | head -20' +``` + +### Step 5 — Optimization decision + +For each consumer in the top-N, fill in a row: + +| Pod / Process | CPU (m) | Why busy | Tunable | Est saving | Trade-off | Effort | +|---|---|---|---|---|---|---| + +Then rank by ROI (saving / effort) and surface the top 3-5. **Hold back the ones where saving < 50m unless effort is also < 5 min.** + +### Common causes + tunables (catalogue) + +| Symptom | Likely cause | Tunable | +|---|---|---| +| **`kube-apiserver` > 1 core sustained** | `CONNECT pods/log` streams from `alloy`/`promtail` using apiserver-tail; OR Kyverno PolicyReport churn (background+enforce mode); OR VPA fanout (309 VPAs cause ~7 req/s) | Switch alloy/promtail to `loki.source.file`; raise Kyverno `backgroundScanInterval`; reduce VPA count | +| **`pvestatd` 70-100% bursts** | LV metadata scan over > 1000 LVs (typically stale `_pre-*` snapshots from ad-hoc node ops) | Delete stale snapshots; `/usr/local/bin/lvm-pvc-snapshot prune` | +| **Frigate > 2 cores** | Birdseye `mode: continuous` (16% on frigate.output); LPR debug; debug logging; too many active cameras × detect.fps | `birdseye.mode: motion`; `lpr.debug_save_plates: false`; remove debug loggers | +| **`vault-0` looping ERRORs every ~10s** | DB static-role not in connection's `allowed_roles` list (drift between role and connection) | Add role to `vault_database_secret_backend_connection.*.allowed_roles` in TF | +| **Alloy DS > 100m/pod** | `loki.source.kubernetes` (apiserver-tail) instead of `loki.source.file` | Switch to file-tail (~5× drop per pod) | +| **Prometheus default 1m scrape** | Chart default; new sample every minute | Raise `server.global.scrape_interval` to 2m; pin critical jobs (snmp-ups) to 30s; bump `for: 1m` alerts to `for: 3m` | +| **`kube-controller-manager` periodic ERROR loop** | Aggregated APIService discovery fails (calico/metrics-server unreachable, OR stuck Terminating pod still in endpoints) | Force-delete stuck pod; verify APIService Available; check pod runc bug on k8s-master | +| **etcd write > 1 MB/s** | PolicyReport thrash, too-frequent secret rotation, or audit log mode = RequestResponse | Trim Kyverno reports config; raise rotation_period; downgrade audit policy to Metadata for noisy resources | + +### What NOT to touch + +- **calico-node, etcd write rate, kube-controller-manager core work, pg-cluster replication** — structural cost, touching them risks correctness. +- **Pods doing legitimate request-serving work** (web servers, databases under load) — optimize the workload, not the runtime. +- **Anything where Goldilocks VPA upperBound is already close to current request** — no headroom to cut. + +### Source-of-truth notes + +- **All infra mutations go via Terraform** (`scripts/tg plan/apply`). The recipes above are diagnostic; the FIX lives in `infra/stacks//main.tf` or chart values. +- **Pod-internal config files** (e.g., Frigate's `/config/config.yml` on a PVC) are not TF-managed — edit in-pod and document in `infra/docs/runbooks/`. +- **PVE host-level state** (LVM snapshots, pvestatd) — SSH + manual ops; record in memory if the pattern recurs. + ## Notes on the canonical / hardlink setup The authoritative copy of this SKILL.md lives at diff --git a/.claude/skills/upgrade-state/SKILL.md b/.claude/skills/upgrade-state/SKILL.md new file mode 100644 index 00000000..a2027a50 --- /dev/null +++ b/.claude/skills/upgrade-state/SKILL.md @@ -0,0 +1,199 @@ +--- +name: upgrade-state +description: | + Audit the three autonomous-upgrade pipelines (apps via Keel, OS via + unattended-upgrades+kured, K8s components via the version-check chain). + Use when: + (1) User asks "/upgrade-state" or "are we current", + (2) User asks "what's pending upgrade" or "what's the upgrade state", + (3) User asks if Keel / kured / k8s-version-check is healthy, + (4) User asks about kept-back / held packages or pending reboots, + (5) Periodic survey before the next `k8s-version-check` daily run. + Read-only — no `--fix`. Exits 0 healthy / 1 attention / 2 stalled. +author: Claude Code +version: 1.0.0 +date: 2026-05-18 +--- + +# Upgrade-state + +## MANDATORY: Run the script first + +When this skill is invoked, your **first action** must be to run +`upgrade_state.sh` and reason over its output before doing anything +else. Do NOT improvise individual `kubectl` / `ssh` calls — the script +is the authoritative surface. + +```bash +bash /home/wizard/code/infra/scripts/upgrade_state.sh +``` + +For programmatic use: + +```bash +bash /home/wizard/code/infra/scripts/upgrade_state.sh --json | tee /tmp/upgrade-state.json +``` + +Then: + +1. Report the rendered table verbatim — it answers the user's + "are we current" question in three lines. +2. For every `⚠` or `✗` row, surface the relevant drill-down lines + underneath and propose a next action (links in the table below). +3. Only reach for ad-hoc commands when investigating beyond what the + script reported. + +Exit codes: `0` healthy, `1` attention warranted, `2` stalled / broken. + +## What it covers (3 pipelines) + +| Layer | What runs | Cadence | Data sources | +|---|---|---|---| +| **Apps** | Keel polls every watched Deployment's container registry; rolls on new digest | hourly | Prom (`pending_approvals`, `registries_scanned_total`), Keel pod logs | +| **OS** | `unattended-upgrades` in-release patching; `kured` reboots when `/var/run/reboot-required` is set | daily 02:00-06:00 London | SSH fan-out to all 5 nodes | +| **K8s** | `k8s-version-check` CronJob detects new kubeadm patch/minor; spawns the Job-chain that drains+upgrades node-by-node | daily 12:00 UTC | Pushgateway (`k8s_upgrade_*`), `kubectl get nodes` | + +The K8s pipeline pushes a small set of gauges to the Prometheus +Pushgateway (`prometheus-prometheus-pushgateway.monitoring:9091`): + +- `k8s_upgrade_available{kind="patch"|"minor",target=…}` — 1 if newer release detected +- `k8s_version_check_last_run_timestamp` — when detection last ran +- `k8s_upgrade_in_flight` — 0/1 +- `k8s_upgrade_started_timestamp` — when the current chain started (0 when idle) + +`K8sUpgradeStalled` alert fires when `in_flight=1` and the chain has +been running >90 minutes. The script raises `✗` in the same window. + +## Status-icon legend + +| Icon | Meaning | +|---|---| +| `✓` | Healthy, fully current | +| `→` | Update available, not yet applied (K8s patch/minor) | +| `…` | In flight — chain currently running | +| `⚠` | Attention: held-with-bumps, recent errors, pending approvals | +| `✗` | Broken: pod down, alert firing, chain stalled | + +## Drill-down — when a row trips, what to do + +### Apps `⚠` — pending approvals or errors + +```bash +# Read recent Keel log lines +kubectl -n keel logs deploy/keel --since=24h --tail=200 + +# What is Keel currently tracking? +kubectl -n monitoring exec deploy/prometheus-server -c prometheus-server -- \ + wget -qO- 'http://localhost:9090/api/v1/query?query=count by (image) (registries_scanned_total)' + +# Is the scrape live? +kubectl -n monitoring exec deploy/prometheus-server -c prometheus-server -- \ + wget -qO- 'http://localhost:9090/api/v1/query?query=up{job="kubernetes-pods",app="keel"}' +``` + +Common Keel errors: +- `failed to add image watch job` — image annotation mistyped (rare; Kyverno auto-injects) +- `registry authentication required` — bad imagePullSecret on the watched Deployment +- `bad tag pattern` — Keel can't parse the watched image's tag against its policy + +### OS `⚠` — held packages with bumps + +The script flags any package held via `apt-mark hold` that ALSO appears +in `apt list --upgradable` — excluding k8s components (the K8s pipeline +owns those) and the kernel (kured handles the reboot half). + +Typical cause: a major-version bump (e.g. containerd 1.7 → 2.2, +runc 1.1 → 1.4). These are held because they need cluster-wide +coordination, not silent in-release patching. + +```bash +# Inspect the situation on the flagged node +ssh wizard@10.0.20.10X 'apt-mark showhold; apt list --upgradable 2>/dev/null' + +# Unhold + upgrade a specific package +ssh wizard@10.0.20.10X 'sudo apt-mark unhold containerd && sudo apt-get install -y containerd' +``` + +Node IPs: master=`100`, node1=`101`, node2=`102`, node3=`103`, node4=`104`. + +### OS `⚠` — pending reboot + +A node has `/var/run/reboot-required`. Kured will reboot it inside the +next 02:00-06:00 London window (any day of the week). + +```bash +# Force a manual reboot inside the window (rare) +kubectl drain k8s-nodeX --delete-emptydir-data --ignore-daemonsets +ssh wizard@10.0.20.10X sudo systemctl reboot +``` + +### OS `✗` — kured not Running + +```bash +kubectl -n kured get pods +kubectl -n kured logs daemonset/kured --tail=100 +# Verify sentinel gate (kured-sentinel-gate DaemonSet writes /var/run/gated-reboot-required) +kubectl -n kured get pods -l name=kured-sentinel-gate +``` + +### K8s `→` — patch/minor available + +Detection ran, target identified, chain NOT started. The chain spawns +on the same daily detection cycle — typically within ~24h of the +target first being detected. + +```bash +# Inspect Pushgateway state +kubectl -n monitoring exec deploy/prometheus-server -c prometheus-server -- \ + wget -qO- 'http://prometheus-prometheus-pushgateway:9091/metrics' | grep ^k8s_upgrade + +# Trigger a manual run of the detection CronJob +kubectl -n k8s-upgrade create job --from=cronjob/k8s-version-check manual-detect-$(date +%s) +``` + +### K8s `…` — in flight + +The Job chain is running. Watch its progress: + +```bash +kubectl -n k8s-upgrade get jobs --sort-by=.metadata.creationTimestamp +kubectl -n k8s-upgrade logs -l app=k8s-version-upgrade --tail=200 --prefix +``` + +### K8s `✗ stalled` — `K8sUpgradeStalled` would fire + +Chain in-flight >90m. The Job is most likely stuck on drain or a +pre-flight check. + +```bash +kubectl -n k8s-upgrade get jobs +kubectl -n k8s-upgrade describe job +kubectl -n k8s-upgrade logs job/ --tail=300 + +# If you need to clear the in-flight flag (after diagnosing): +kubectl -n monitoring exec deploy/prometheus-server -c prometheus-server -- sh -c \ + "printf 'k8s_upgrade_in_flight 0\nk8s_upgrade_started_timestamp 0\n' | \ + wget -qO- --post-file=- 'http://prometheus-prometheus-pushgateway:9091/metrics/job/k8s-version-upgrade' \ + --header='Content-Type: text/plain'" +``` + +### K8s `✗ detection stale` — last detection >9 days + +```bash +kubectl -n k8s-upgrade get cronjob k8s-version-check +kubectl -n k8s-upgrade get jobs --sort-by=.metadata.creationTimestamp | tail -5 +``` + +If the CronJob hasn't fired on time, suspect: +- `suspend=true` on the CronJob (`var.enabled=false` in the + `k8s-version-upgrade` Terraform stack) +- Image-pull failure on the version-check pod +- Pushgateway scrape gone stale + +## Companion command-line flags + +```bash +bash infra/scripts/upgrade_state.sh # rendered table (default) +bash infra/scripts/upgrade_state.sh --json # machine output +bash infra/scripts/upgrade_state.sh --kubeconfig X # override kubeconfig +``` diff --git a/.gitleaksignore b/.gitleaksignore new file mode 100644 index 00000000..dfe626cd --- /dev/null +++ b/.gitleaksignore @@ -0,0 +1,4 @@ +# git-crypt encrypts these at rest; the working-tree plaintext is local-only. +# gitleaks scans the staged working-tree copy and can't see that they're +# encrypted on disk in git, so allowlist by fingerprint. +stacks/recruiter-responder/secrets/privkey.pem:private-key:1 diff --git a/AGENTS.md b/AGENTS.md index 5f9c0839..53bcb5c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -154,6 +154,37 @@ lifecycle { **Audit**: `rg "KYVERNO_LIFECYCLE_V1" stacks/ | wc -l` — should grow (never shrink). Add the marker to every new pod-owning resource. The `_template/main.tf.example` stub shows the canonical form. +### `# KYVERNO_LIFECYCLE_V2` — Keel auto-update annotations + +When a namespace is labeled `keel.sh/enrolled=true`, the `inject-keel-annotations` ClusterPolicy (`stacks/kyverno/modules/kyverno/keel-annotations.tf`) injects three annotations on every Deployment / StatefulSet / DaemonSet: + +``` +keel.sh/policy: force +keel.sh/trigger: poll +keel.sh/pollSchedule: "@every 1h" +``` + +To suppress the resulting Terraform drift, **enrolled workloads** must extend their `ignore_changes` block: + +```hcl +lifecycle { + ignore_changes = [ + spec[0].template[0].spec[0].dns_config, # KYVERNO_LIFECYCLE_V1 + metadata[0].annotations["keel.sh/policy"], + metadata[0].annotations["keel.sh/trigger"], + metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 + ] +} +``` + +The V2 snippet is added **per workload** as namespaces are phase-enrolled — not as a mass sweep. Workloads in un-enrolled namespaces do not receive the annotation and don't need the V2 block. + +Per-workload opt-out: add the label `keel.sh/policy: never` on the Deployment metadata (not pod template); the policy's `exclude` clause respects it, no annotation gets injected, no `ignore_changes` needed. + +**Audit**: `rg "KYVERNO_LIFECYCLE_V2" stacks/` — count should equal the number of enrolled workloads. + +**Design context**: `docs/plans/2026-05-16-auto-upgrade-apps-{design,plan}.md`. + ## Tier System `0-core` | `1-cluster` | `2-gpu` | `3-edge` | `4-aux` — Kyverno auto-generates LimitRange + ResourceQuota per namespace based on tier label. - Containers without explicit `resources {}` get default limits (256Mi for edge/aux — causes OOMKill for heavy apps) diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 00000000..ffdab01f --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,150 @@ +# Infra + +Terragrunt-managed homelab declaring a 5-node Kubernetes cluster on a single Proxmox host. Vault is the secrets source of truth; everything else flows from this repo via `scripts/tg apply`. + +## Language + +### Code organization + +**Service**: +The deployed app as a domain concept — one logical thing that runs in the cluster (e.g. immich, technitium, freshrss). Defined by exactly one **Stack**. +_Avoid_: bare "app" without the Service definition; "deployment" (collides with K8s `Deployment`). + +**Stack**: +The HCL directory under `stacks//` that defines a Service, applied independently with `scripts/tg apply`. A Stack is the unit of Terraform organisation; a Service is the running thing. They are 1:1 but not synonyms. +_Avoid_: using "Stack" when you mean the running Service. + +**Module**: +A reusable HCL primitive under `modules/`, consumed by Stacks via `source =`. +_Avoid_: "library", "package". + +**Factory module**: +A Module that hides convention (defaults, drift handling, secret wiring) behind a small input surface. Canonical examples: `ingress_factory`, `nfs_volume`, `k8s_app`, `helm_app`, `postgres_app`. +_Avoid_: "wrapper". + +**State tier**: +Terraform state-backend partition. **Tier 0** = bootstrap Stacks (`infra`, `platform`, `cnpg`, `vault`, `dbaas`, `external-secrets`) on local SOPS-encrypted state. **Tier 1** = every other Stack, on PG-backed state. +_Avoid_: "phase", "bootstrap stack" — say Tier 0 explicitly. + +### Cluster + +**Node**: +A K8s worker VM (`k8s-master`, `k8s-node1..4`). Default reading of the bare word "node" in this repo. +_Avoid_: "k8s node" (redundant), "host" (ambiguous). + +**PVE node** / **PVE host**: +The single physical Dell R730 running Proxmox; sole hypervisor and sole NFS server. There is exactly one. +_Avoid_: "server", "hypervisor", "Proxmox" alone when you mean the host. + +**Namespace tier**: +A namespace-prefix partition (`0-core-*`, `1-cluster-*`, `2-gpu-*`, `3-edge-*`, `4-aux-*`) driving PriorityClass, default resources, and ResourceQuota — generated by **Kyverno policy** from the namespace name. Orthogonal to **State tier**. +_Avoid_: "Service tier" (the partition is on the namespace, not the Service); collapsing Namespace tier with State tier — they are different axes. + +**Kyverno policy**: +The convention engine of the cluster — a ClusterPolicy or Policy resource that mutates/generates/validates on admission. Owns Namespace tier limits/quotas, `dns_config` injection on every pod-owning workload, Forgejo pull-credential sync across namespaces, TLS-secret replication. When the repo says "this happens automatically", a Kyverno policy is usually the actor. +_Avoid_: bare "policy" (overloaded with Vault, RBAC, NetworkPolicy). + +**Critical-path Service**: +One of {Traefik, Authentik, CrowdSec LAPI, PgBouncer, Cloudflared} — replicas ≥3, PDB enforced, monitored independently. +_Avoid_: "core service" (collides with the `0-core-*` Namespace tier name). + +**Namespace-owner**: +A non-admin identity declared in `secret/platform → k8s_users` (JSON map). Owns one or more namespaces and one or more public subdomains. +_Avoid_: bare "user", "tenant". + +### Networking + +**Public domain**: +`viktorbarzin.me`, served through Cloudflare. DNS records are either **proxied** (Cloudflare CDN/WAF in front) or **non-proxied** (direct A/AAAA reachable via Cloudflared Tunnel). +_Avoid_: "external", "outside". + +**Internal domain**: +`viktorbarzin.lan`, served by Technitium DNS. Resolves only inside the homelab network. +_Avoid_: bare "lan", "private", "intranet". + +**Ingress auth tier**: +The `auth = "..."` parameter on `ingress_factory`, one of `required` (Authentik forward-auth gates every request), `app` (the backend owns its login), `public` (anonymous Authentik binding for audit only), or `none` (Anubis-fronted content, or native-client API). +_Avoid_: "auth mode" — the canonical key is `auth`. + +**Authentik outpost**: +A standalone Authentik deployment that terminates the proxy/auth flow for a specific binding model. The repo runs two distinct ones: the default outpost (used by `auth = "required"`) and the `public` outpost (anonymous binding, used by `auth = "public"`). +_Avoid_: conflating outpost with Authentik core; "Authentik instance". + +**Cloudflared Tunnel**: +The channel by which non-proxied **public domain** traffic reaches the cluster, terminating at Traefik. Backs every `dns_type = "non-proxied"` record and is the fallback path for the wildcard `*.viktorbarzin.me`. +_Avoid_: "the tunnel" without "Cloudflared" (could mean Headscale). + +**Ingress chain**: +The opinionated stack of Traefik middlewares that `ingress_factory` layers onto every Ingress. Slots, in order: forward-auth (per **Ingress auth tier**) → anti-AI scraping (default-on when no Authentik is in the path) → CrowdSec bouncer (fail-open) → retry (2× / 100ms) → rate-limit (429, not 503). Adding or removing a middleware is a Stack-level choice, but the chain order is convention. +_Avoid_: "middleware list", "Traefik chain". The Anubis PoW gate is upstream of this chain, not inside it. + +### Storage + +**proxmox-lvm-encrypted**: +Default StorageClass for any workload holding sensitive data (databases, auth, password managers, email, financial data). LUKS2 over a Proxmox LVM-thin LV. +_Avoid_: bare "encrypted PVC" — name the StorageClass. + +**proxmox-lvm**: +Block StorageClass for non-sensitive workloads (caches, monitoring data, indexes, app state without secrets). + +**NFS volume**: +RWX file storage for shared media libraries, large datasets, or anything that needs to be inspected from outside K8s. Provisioned via the `nfs_volume` Module. +_Avoid_: "shared storage" (ambiguous). + +**nfs-truenas StorageClass**: +A historical SC name retained only because StorageClass strings are immutable on bound PVs. The underlying server is the **PVE host**, not TrueNAS; TrueNAS is decommissioned. +_Avoid_: assuming this means TrueNAS. + +**3-2-1 backup**: +The named posture of where data lives: **Copy 1** = live on the PVE thin pool (sdc), **Copy 2** = sda backup disk (`/mnt/backup`), **Copy 3** = offsite Synology NAS. Per-PVC file-level rsync from LVM thin snapshots; databases additionally dump to NFS for per-DB restore. +_Avoid_: bare "backup" without saying which copy you mean (a service is "backed up" only once it's on Copy 2; Copy 3 is the disaster floor). + +### Secrets + +**Vault path**: +Convention: `secret/` for Service-owned secrets, `secret/viktor` for personal/global, `secret/platform` for cluster-wide maps (`k8s_users`, `homepage_credentials`). +_Avoid_: conflating Vault path (e.g. `secret/viktor`) with Vault field (e.g. `forgejo_pull_token`). + +**ExternalSecret** / **ESO**: +A K8s manifest that materialises a Vault KV value as a K8s Secret. Two ClusterSecretStores: `vault-kv` (KV engine) and `vault-database` (rotating DB creds). + +**Plan-time secret**: +A secret value read in Terraform via `data "kubernetes_secret"` (i.e. via the ESO-created K8s Secret) at plan time, with no Vault provider call. Distinct from a **vault data source** read (`data "vault_kv_secret_v2"`), which still goes through the Vault provider. A few Stacks remain hybrid (plan-time for env vars, vault data source for module inputs). + +**Sealed Secret**: +A user-managed secret committed to a Stack directory as `sealed-*.yaml`. Distinct from ExternalSecret — Sealed Secrets carry their own bytes, ExternalSecrets reference Vault. + +### CI/CD + +**GHA build + Woodpecker deploy**: +The split where Docker images are built+pushed by GitHub Actions and Woodpecker only runs `kubectl set image` on a deploy-only pipeline. Repos that can't fit GHA limits stay on Woodpecker for build too. +_Avoid_: bare "Woodpecker pipeline" — say "build" or "deploy". + +**Anubis**: +A PoW reverse-proxy issuing a 30-day JWT cookie, used in front of public content-bearing sites without app-level auth (blog, wiki, landing pages). Never in front of Git, WebDAV, CalDAV, or API endpoints (clients can't solve PoW). + +## Relationships + +- A **Service** is defined by exactly one **Stack**, which declares zero or more **Modules** and resolves to one or more K8s workloads. +- A **Namespace-owner** owns one or more namespaces and one or more public subdomains. +- A **Service** owns its **Vault path** at `secret/`, surfaces values through **ExternalSecrets**, and reads them at plan time via **plan-time secrets**. +- An **Ingress** picks exactly one **Ingress auth tier**; the choice defines how strangers reach the backend. +- A **proxmox-lvm-encrypted** PVC binds to one Node at a time (RWO) and requires a Service-level backup CronJob; an **NFS volume** is RWX and is backed up at the host level via rsync. +- **State tier** and **Namespace tier** are orthogonal — a Tier 0 Stack can deploy a Service into any Namespace tier and vice versa. + +## Example dialogue + +> **Dev:** "I'm adding a new **Service** — FastAPI backend with its own JWT login. Do I need Authentik?" +> **Domain expert:** "If the FastAPI login is the gate, set `auth = "app"` on the ingress. That records the intent that you _chose_ not to layer Authentik — leave a one-line comment above stating what gates the Service, or `scripts/tg` will refuse the apply." +> **Dev:** "And storage?" +> **Domain expert:** "Does it hold user data? If yes, `proxmox-lvm-encrypted` — that's the default for anything sensitive. Add a backup CronJob writing to `/mnt/main/-backup/`. If the data is just caches, plain `proxmox-lvm` is fine." +> **Dev:** "What about a Secret with the JWT signing key?" +> **Domain expert:** "Put the key in `secret/` in Vault, then declare an **ExternalSecret** to materialise it as a K8s Secret. Read it at plan time with `data "kubernetes_secret"` — that keeps Vault out of the plan path." + +## Flagged ambiguities + +- **"tier"** is overloaded — *Namespace tier* (`0-core`..`4-aux`, scheduling priority) is distinct from *State tier* (Tier 0 / Tier 1, Terraform backend partition). Always qualify which axis. +- **"node"** can mean a K8s Node (default) or a PVE node. For Proxmox-level statements, say **PVE node** explicitly. +- **"service"** spans two distinct concepts: the deployed app (capitalised **Service**, this repo's domain noun) and the K8s `Service` object (in backticks or qualified "K8s Service"). Lowercase "service" in prose is fine when context disambiguates; flag it when it doesn't. +- **"secret"** spans Vault entries, K8s Secret objects, **ExternalSecrets**, and **Sealed Secrets**. Always specify which. +- **"proxied"** / **"non-proxied"** refer to Cloudflare's CDN posture for a DNS record, _not_ Anubis or forward-auth layering. diff --git a/ci/Dockerfile b/ci/Dockerfile index 2a02b586..61f3bfe8 100644 --- a/ci/Dockerfile +++ b/ci/Dockerfile @@ -7,9 +7,11 @@ ARG SOPS_VERSION=3.9.4 ARG KUBECTL_VERSION=1.34.0 ARG VAULT_VERSION=1.18.1 -# Install system packages (single layer) +# Install system packages (single layer). +# python3: required by scripts/check-ingress-auth-comments.py, invoked +# by scripts/tg before every plan/apply. RUN apk add --no-cache \ - bash curl git git-crypt jq openssh-client openssl unzip \ + bash curl git git-crypt jq openssh-client openssl python3 unzip \ && rm -rf /var/cache/apk/* # Terraform diff --git a/docs/architecture/authentication.md b/docs/architecture/authentication.md index 42d79561..361352a9 100644 --- a/docs/architecture/authentication.md +++ b/docs/architecture/authentication.md @@ -44,7 +44,7 @@ graph TB | Authentik Worker | 2026.2.2 | `stacks/authentik/` | Background task processors (2 replicas) | | PgBouncer | Latest | `stacks/authentik/` | PostgreSQL connection pooler (3 replicas) | | Embedded Outpost | - | Built into Authentik | Forward auth endpoint for Traefik | -| Traefik ForwardAuth | - | `ingress_factory` module | Middleware for protected ingresses | +| Traefik ForwardAuth | - | `modules/kubernetes/ingress_factory/` | Middleware attached when `auth = "required"` or `"public"` | | Vault OIDC Method | - | `stacks/vault/` | Human SSO authentication to Vault | | Vault K8s Auth | - | `stacks/vault/` | Service account JWT authentication | @@ -52,7 +52,16 @@ graph TB ### Forward Authentication Flow -Services configured with `protected = true` in the `ingress_factory` module automatically get Traefik ForwardAuth middleware configured. When an unauthenticated user accesses a protected service: +Services pick an auth tier via the `auth` enum on the `ingress_factory` module (default `"required"`, fail-closed): + +| Tier | Effect | When to use | +|------|--------|-------------| +| `"required"` | Authentik forward-auth gates every request | Backend has no own user auth — Authentik is the only gate | +| `"app"` | No Authentik middleware; backend's own login is the gate | Backend handles its own user auth (NextAuth, Django, OAuth, bearer-token API) | +| `"public"` | Authentik anonymous binding via `public` outpost | Audit trail without gating; only works for top-level browser navigation | +| `"none"` | No Authentik middleware at all | Anubis-fronted content, webhooks, OAuth callbacks, native-client APIs (CalDAV, WebDAV, Git) | + +When `auth = "required"`, an unauthenticated request flows: 1. Request hits Traefik ingress 2. ForwardAuth middleware calls Authentik embedded outpost @@ -64,6 +73,8 @@ Services configured with `protected = true` in the `ingress_factory` module auto Authentik adds authentication headers (user, email, groups) to forwarded requests. These headers are stripped before reaching the backend to prevent confusion. +**Anti-exposure guard**: every `auth = "app"` or `auth = "none"` line MUST have a preceding `# auth = "": ` comment documenting what gates the backend (for `"app"`) or why the endpoint is intentionally public (for `"none"`). The convention is enforced by `scripts/check-ingress-auth-comments.py`, which `scripts/tg` runs on every `plan/apply/destroy/refresh` and blocks the terragrunt invocation if violated. Stack-scoped — each stack documents itself. + ### Social Login & Invitation Flow All new users must use an invitation link to register. The invitation-enrollment flow: @@ -144,8 +155,9 @@ The public client flow: | Path | Purpose | |------|---------| | `stacks/authentik/` | Authentik deployment (servers, workers, PgBouncer) | -| `stacks/platform/modules/ingress_factory/` | Traefik ForwardAuth middleware config | -| `stacks/platform/modules/traefik/middleware.tf` | ForwardAuth middleware definition | +| `modules/kubernetes/ingress_factory/` | Auth-tier enum + per-ingress middleware composition | +| `stacks/traefik/modules/traefik/middleware.tf` | ForwardAuth middleware definitions (required + public outposts) | +| `scripts/check-ingress-auth-comments.py` | Comment-convention guard wired into `scripts/tg` | | `stacks/vault/auth.tf` | Vault OIDC and K8s auth methods | ### Vault Paths @@ -160,17 +172,40 @@ The public client flow: - `stacks/platform/` - Traefik ingress with ForwardAuth - `stacks/vault/` - Vault auth methods -### Ingress Protection Example +### Ingress Protection Examples +Authentik-gated admin UI (default): ```hcl module "myapp_ingress" { - source = "./modules/ingress_factory" + source = "../../modules/kubernetes/ingress_factory" + name = "myapp" + namespace = "myapp" + tls_secret_name = var.tls_secret_name + # auth = "required" is the default — Authentik forward-auth is the gate. +} +``` - name = "myapp" - host = "myapp.viktorbarzin.me" - protected = true # Enables ForwardAuth middleware +Backend with its own user auth (no Authentik in the way): +```hcl +module "myapp_ingress" { + source = "../../modules/kubernetes/ingress_factory" + name = "myapp" + namespace = "myapp" + tls_secret_name = var.tls_secret_name + # auth = "app": myapp uses NextAuth + Google OAuth; mobile clients can't follow Authentik 302. + auth = "app" +} +``` - # ... other config +Intentionally public webhook receiver: +```hcl +module "myapp_ingress" { + source = "../../modules/kubernetes/ingress_factory" + name = "webhook" + namespace = "webhooks" + tls_secret_name = var.tls_secret_name + # auth = "none": upstream signs payloads with HMAC; no user identity expected. + auth = "none" } ``` diff --git a/docs/architecture/automated-upgrades.md b/docs/architecture/automated-upgrades.md index 4a023c10..c6d423d6 100644 --- a/docs/architecture/automated-upgrades.md +++ b/docs/architecture/automated-upgrades.md @@ -1,4 +1,10 @@ -# Automated Service Upgrades +# Automated Upgrades + +This doc covers three independent automation paths: + +1. **Service-level upgrades** — Container image bumps for OSS apps (DIUN → n8n → claude-agent → Terraform). Most of this doc. +2. **OS-level upgrades on K8s nodes** — `unattended-upgrades` + `kured` with sentinel-gate + Prometheus halt-on-alert. See "K8s Node OS Upgrades" section and the runbook at `docs/runbooks/k8s-node-auto-upgrades.md`. +3. **K8s component version upgrades** (kubeadm/kubelet/kubectl) — weekly detection CronJob → chain of phase Jobs (preflight → master → worker × 4 → postflight). See "K8s Version Upgrades" section and the runbook at `docs/runbooks/k8s-version-upgrade.md`. ## Overview @@ -205,3 +211,145 @@ The `DIUN Upgrade Agent` workflow is imported once into n8n's PG DB — it is ** - **`N8N_BLOCK_ENV_ACCESS_IN_NODE=false`** must be set on the n8n deployment for expressions to read `$env.*` at all. - **Troubleshooting 401**: the workflow will show `success` status on the webhook node but error on `Run Upgrade Agent`. Inspect in n8n UI → Executions, or query `execution_entity` + `execution_data` directly. Claude-agent-service logs will also show `POST /execute HTTP/1.1 401 Unauthorized`. - **Patching the live workflow** (one-off, since it's not in TF): `UPDATE workflow_entity SET nodes = REPLACE(nodes::text, OLD, NEW)::json WHERE name = 'DIUN Upgrade Agent';` + +## K8s Node OS Upgrades + +Independent of the service-upgrade pipeline above. Drives apt package updates + reboots on the 5 K8s VMs (master + 4 workers). + +### Stack +- **In-guest**: `unattended-upgrades` runs apt upgrades within Allowed-Origins (`-security`, `-updates`, ESM). Package-Blacklist excludes runtime components (`containerd`, `containerd.io`, `runc`, `cri-tools`, `kubernetes-cni`, `calico-*`, `cni-plugins-*`, `docker-ce`). `apt-mark hold` on `kubelet`, `kubeadm`, `kubectl` (and runtime pkgs as belt-and-braces). `Automatic-Reboot=false` — kured handles reboots. +- **Reboot driver**: `kured` (chart `kured-5.11.0`, app `1.21.0`). Window 02:00-06:00 Europe/London every day of the week (Mon-Fri-only restriction dropped 2026-05-16 — see PM), period=1h, concurrency=1, reboot-delay=30s, drainTimeout=30m. +- **Reboot gate (sentinel)**: `kured-sentinel-gate` DaemonSet creates `/var/run/gated-reboot-required` only when (a) host needs reboot, (b) all nodes Ready, (c) all calico-node pods Running, (d) **no node has transitioned Ready in the last 24h** (24h soak window). +- **Reboot gate (Prometheus)**: kured `--prometheus-url` polls `prometheus-server.monitoring.svc:80` before each drain. ANY firing alert blocks unless it matches the ignore-regex `^(Watchdog|RebootRequired|KuredNodeWasNotDrained|InfoInhibitor)$`. +- **Health alert library**: 10 alerts in the `Upgrade Gates` group (`prometheus_chart_values.tpl`): `KubeAPIServerDown`, `KubeStateMetricsDown`, `PrometheusRuleEvaluationFailing`, `PVCStuckPending`, `RecentNodeReboot` (the explicit 24h soak signal), `MysqlStandaloneDown`, `ClusterPodReadyRatioDropped`, `NodeMemoryPressure`, `NodeDiskPressure`, `KubeQuotaAlmostFull`. Plus the existing 200+ alerts in the cluster-wide library (anything firing blocks kured). +- **Notifications**: kured `notifyUrl` posts drain-start/drain-finish to Slack via Vault `secret/kured.slack_kured_webhook`. Alertmanager separately routes critical alerts to `#alerts`. + +### Source of truth +| Concern | Location | +|---|---| +| Package config (uu, holds, blacklist) | `modules/create-template-vm/cloud_init.yaml` (within `is_k8s_template`) | +| kured Helm release + sentinel-gate DS | `stacks/kured/main.tf` | +| Upgrade Gates alerts | `stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl` | + +### Day-2 changes +Cloud-init only runs on first boot. Existing nodes are brought into compliance with a one-shot SSH push — see the runbook section "Restore / re-apply unattended-upgrades config to existing nodes" in `docs/runbooks/k8s-node-auto-upgrades.md`. + +### Why this design +The 26h cluster outage on 2026-03-16 was triggered by an unattended-upgrades kernel push that corrupted containerd's overlayfs snapshotter cluster-wide. The remediations: +- 24h soak (sentinel-gate Check 4) gives a full day of observation between consecutive node reboots — broken updates show up as Prometheus alerts before any other node restarts. +- Prometheus halt-on-alert turns ANY firing alert into a hard block — including the 6 Node Runtime Health alerts and the 10 Upgrade Gates alerts that explicitly model "the cluster is in a bad state." +- Package-Blacklist on runtime components prevents the exact failure mode (containerd/runc auto-bumps). +- `Automatic-Reboot=false` keeps reboot policy in kured (window, ordering, gating), not in apt. + +### Operational reference +See `docs/runbooks/k8s-node-auto-upgrades.md` for: verifying health, halting rollout, restoring config to a re-imaged node, rolling back a bad upgrade, and the past-incident timeline. + +## K8s Version Upgrades + +Independent of the OS-upgrade and service-upgrade pipelines. Drives +kubeadm/kubelet/kubectl bumps (patch + minor) on all 5 K8s VMs. + +### Architecture + +``` +k8s-version-check CronJob (Sun 12:00 UTC, k8s-upgrade ns) + │ probe apt-cache madison kubeadm (master) → latest available patch + │ probe HEAD https://pkgs.k8s.io/.../v/deb/Release → next minor? + │ push k8s_upgrade_available metric to Pushgateway + │ + ▼ if a target is detected +envsubst on /template/job-template.yaml | kubectl apply -f - + │ spawns Job 0 = k8s-upgrade-preflight- + ▼ + +Job 0 — preflight (pinned: k8s-node1) +Job 1 — master upgrade (pinned: k8s-node1) drains k8s-master +Job 2 — worker (pinned: k8s-node1) drains k8s-node4 +Job 3 — worker (pinned: k8s-node1) drains k8s-node3 +Job 4 — worker (pinned: k8s-node1) drains k8s-node2 +Job 5 — worker (pinned: k8s-master) drains k8s-node1 ← control-plane toleration +Job 6 — postflight (no pinning) +``` + +Each Job runs `scripts/upgrade-step.sh`, which dispatches on `$PHASE` and ends +by spawning the next Job (`envsubst < /template/job-template.yaml | kubectl +apply -f -`). Job names are deterministic (`k8s-upgrade--[-]`) +so `apply` reconciles to a single Job per run — re-running a failed Job +won't duplicate downstream Jobs. + +### Self-preemption history (the reason for the Job-chain rewrite) + +The v1 design ran the whole upgrade inside the `claude-agent-service` +Deployment (1 replica, no nodeSelector). On 2026-05-11 the agent's pod was +scheduled to k8s-node4. When the agent ran `kubectl drain k8s-node4` during +Stage 6, it evicted itself — the bash process died after the drain but +before the SSH-pipe to install kubeadm on node4. The cluster ended up +half-upgraded (master at v1.34.7, workers at v1.34.2). The rewrite to a +chain of `nodeSelector`-pinned Jobs eliminates this failure mode because +each Job's pod and its drain target are always different nodes. + +### Components + +- **Detection CronJob + ConfigMaps + RBAC**: `infra/stacks/k8s-version-upgrade/main.tf`. + - Image is the claude-agent-service image (kubectl + ssh-client + curl + jq + envsubst). + - One unified ServiceAccount `k8s-upgrade-job` serves both the detection CronJob and every chain Job. +- **Phase body**: `infra/stacks/k8s-version-upgrade/scripts/upgrade-step.sh`. + Dispatches on `$PHASE` (preflight | master | worker | postflight). Computes + `NEXT_PHASE` / `NEXT_TARGET_NODE` / `NEXT_RUN_ON` and spawns the next Job. + Includes a `predrain_unstick` helper that pre-deletes pods on the target + node whose PDB has `disruptionsAllowed=0` (otherwise drain loops forever on + single-replica deployments like Anubis instances). +- **Job template**: `infra/stacks/k8s-version-upgrade/job-template.yaml`. + envsubst-rendered at runtime. Mounts a `creds` Secret, a `scripts` + ConfigMap, and a `template` ConfigMap into each Job pod. +- **Per-node script**: `infra/scripts/update_k8s.sh`. Caller passes + `--role master|worker --release X.Y.Z`. Piped via SSH into each node by + upgrade-step.sh. +- **Three Upgrade Gates alerts**: + - `K8sVersionSkew` — kubelet/apiserver `gitVersion` count >1 for 30m. Catches a half-done rollout. + - `EtcdPreUpgradeSnapshotMissing` — `k8s_upgrade_in_flight==1 && k8s_upgrade_snapshot_taken==0` for 10m. Catches preflight failing silently. + - `K8sUpgradeStalled` — `k8s_upgrade_in_flight==1 && time()-k8s_upgrade_started_timestamp > 5400` for 5m. Catches a chain Job dying without spawning its successor. +- **Pushgateway metrics**: + - `k8s_upgrade_in_flight` (set in preflight, cleared in postflight) + - `k8s_upgrade_snapshot_taken` (set after etcd snapshot Job completes with ≥1 KiB) + - `k8s_upgrade_started_timestamp` (set in preflight; used by `K8sUpgradeStalled`) + - `k8s_upgrade_available{kind,running,target}` (pushed by detection CronJob) + - `k8s_version_check_last_run_timestamp` (staleness watchdog) + +### Source of truth + +| Concern | Location | +|---|---| +| Stack (CronJob + ConfigMaps + SA/RBAC + ExternalSecret) | `stacks/k8s-version-upgrade/main.tf` | +| Phase orchestration | `stacks/k8s-version-upgrade/scripts/upgrade-step.sh` | +| Job template | `stacks/k8s-version-upgrade/job-template.yaml` | +| Per-node upgrade script | `scripts/update_k8s.sh` | +| Alerts | `stacks/monitoring/modules/monitoring/prometheus_chart_values.tpl` (group "Upgrade Gates") | +| Vault secrets | `secret/k8s-upgrade/{ssh_key, ssh_key_pub, slack_webhook}` | +| Deprecated agent prompt (reference) | `.claude/agents/k8s-version-upgrade.deprecated.md` | + +### Why this design + +The cluster has a single control plane (no HA). A failed `kubeadm upgrade apply` is an outage. Mitigations: + +- **Mandatory etcd snapshot before every run** (even patch). Recovery point if master breaks. +- **Halt-on-alert before every drain**. Reuses the same Prometheus ignore-list regex kured uses — any unrelated cluster-health alert blocks. Three gate alerts catch upgrade-specific half-states (version skew, missing snapshot, stalled chain). +- **Job pinning eliminates self-preemption**. Each Job's pod runs on a node that is NOT its drain target. k8s-node1 hosts every Job except the one that drains it (which runs on k8s-master with a control-plane toleration). +- **Sequential workers with 10-min inter-node soak**. Same risk-bounding as the 24h OS-reboot soak, but tightened because kubelet failures surface within minutes — not hours. +- **Master upgrade goes first, workers last**. If master breaks, the cluster is already degraded so further worker upgrades would just delay recovery. By upgrading master first, we either succeed (workers can roll afterward) or fail loud (operator triages before any worker is touched). +- **No auto-rollback**. kubeadm doesn't support clean downgrade; the snapshot + manual apt rollback in the runbook is the recovery path. +- **PDB-blocked pods don't stall the chain**. `predrain_unstick` deletes PDB=0 pods on the target node directly (bypassing the eviction API), so the parent Deployment recreates them elsewhere. This was the workaround applied manually during the 2026-05-11 recovery for Anubis single-replica instances. + +### Secrets + +| Secret | Vault Path | Purpose | +|--------|-----------|---------| +| SSH private key | `secret/k8s-upgrade.ssh_key` | Jobs SSH `wizard@` | +| SSH public key | `secret/k8s-upgrade.ssh_key_pub` | Deployed to nodes' `~/.ssh/authorized_keys` | +| Slack webhook | `secret/k8s-upgrade.slack_webhook` | Pipeline notifications (separate channel from kured) | + +The previous `api_bearer_token` entry is gone — the chain does not POST to `claude-agent-service`. + +### Operational reference + +See `docs/runbooks/k8s-version-upgrade.md` for: verifying health, manually triggering detection, killing a stuck Job, skipping a phase, rollback paths (master / worker / mid-flight abort), and SSH key rotation. diff --git a/docs/architecture/compute.md b/docs/architecture/compute.md index cc9c4786..7c84fc4b 100644 --- a/docs/architecture/compute.md +++ b/docs/architecture/compute.md @@ -18,7 +18,7 @@ graph TB subgraph Proxmox["Proxmox VE"] direction TB MASTER["VM 200: k8s-master
8c / 32GB
10.0.20.100"] - NODE1["VM 201: k8s-node1
16c / 32GB
GPU Passthrough
nvidia.com/gpu=true:PreferNoSchedule"] + NODE1["VM 201: k8s-node1
16c / 48GB
GPU Passthrough
nvidia.com/gpu=true:PreferNoSchedule"] NODE2["VM 202: k8s-node2
8c / 32GB"] NODE3["VM 203: k8s-node3
8c / 32GB"] NODE4["VM 204: k8s-node4
8c / 32GB"] @@ -62,7 +62,7 @@ graph TB | Model | Dell PowerEdge R730 | | CPU | 1x Intel Xeon E5-2699 v4 (22 cores / 44 threads, CPU2 unpopulated) | | Total Cores/Threads | 22 cores / 44 threads | -| RAM | 272GB DDR4-2400 ECC RDIMM physical (10 DIMMs: 8x32G Samsung + 2x8G Hynix). VMs use ~160GB total (5 K8s VMs x 32GB) | +| RAM | 272GB DDR4-2400 ECC RDIMM physical (10 DIMMs: 8x32G Samsung + 2x8G Hynix). VMs use ~176GB total (k8s-node1 48GB + 4 K8s VMs x 32GB) | | GPU | NVIDIA Tesla T4 (16GB GDDR6, PCIe 0000:06:00.0) | | Storage | 1.1TB SSD + 931GB SSD + 10.7TB HDD | | Hypervisor | Proxmox VE | @@ -72,12 +72,20 @@ graph TB | VM | VMID | vCPUs | RAM | Network | Role | Taints | |----|------|-------|-----|---------|------|--------| | k8s-master | 200 | 8 | 32GB | vmbr1:vlan20 (10.0.20.100) | Control Plane | `node-role.kubernetes.io/control-plane:NoSchedule` | -| k8s-node1 | 201 | 16 | 32GB | vmbr1:vlan20 | GPU Worker | `nvidia.com/gpu=true:PreferNoSchedule` (applied dynamically to whichever node carries the GPU) | +| k8s-node1 | 201 | 16 | 48GB | vmbr1:vlan20 | GPU Worker | `nvidia.com/gpu=true:PreferNoSchedule` (applied dynamically to whichever node carries the GPU) | | k8s-node2 | 202 | 8 | 32GB | vmbr1:vlan20 | Worker | None | | k8s-node3 | 203 | 8 | 32GB | vmbr1:vlan20 | Worker | None | | k8s-node4 | 204 | 8 | 32GB | vmbr1:vlan20 | Worker | None | -**Total Cluster Resources**: 48 vCPUs, ~160GB RAM (5 nodes x 32GB) +**Total Cluster Resources**: 48 vCPUs, ~176GB RAM (k8s-node1 48GB + 4 nodes x 32GB) + +> **node1 RAM (2026-05-10)**: bumped from 32 → 48 GiB out-of-band via +> `qm set 201 --memory 49152` because VMID 201 is intentionally not +> managed by Terraform yet (telmate/proxmox provider bug with iSCSI +> PVCs — see `infra/stacks/infra/main.tf` line 442). Driver: GPU +> multi-tenancy (frigate + ytdlp + llama-swap + immich-ml) was +> hitting 94% memory-request saturation on the old size. Adopt this +> VM into TF (`module "k8s-node1"`) once we've migrated to bpg/proxmox. ### GPU Passthrough diff --git a/docs/architecture/llama-cpp.md b/docs/architecture/llama-cpp.md new file mode 100644 index 00000000..bd7c4be2 --- /dev/null +++ b/docs/architecture/llama-cpp.md @@ -0,0 +1,118 @@ +# llama-cpp / llama-swap + +## Overview + +In-cluster, OpenAI-compatible vision-LLM endpoint. A single +`mostlygeek/llama-swap:cuda` Deployment fronts three GGUF models +served by `llama.cpp`'s `llama-server` subprocesses, hot-swapped on +demand by `llama-swap`. One Service, one `/v1` endpoint, model +selected by the request body `model` field. + +Initial use case: vision-LLM benchmark on a curated Immich album, +choosing between **Qwen3-VL-8B**, **MiniCPM-V-4.5**, and +**Qwen3-VL-4B** for instagram-poster's candidate-scoring path. +Future consumers (Home Assistant, agentic tooling) can hit the same +endpoint via LiteLLM at the cluster gateway. + +First benchmark run (2026-05-10): see +`infra/docs/benchmarks/2026-05-10-vision-llm.md`. Verdict: **qwen3vl-4b** +for the request path (3.55 s p50, 100% parse, decisive top-N +distribution). qwen3vl-8b for caption polish on top picks. + +## Why llama.cpp + llama-swap (not Ollama) + +Verified across 7+7 research/challenger subagents (2026-05-10): + +- **Broader OpenAI-compat surface** — `tool_choice`, `image_url` + remote URLs, native bearer auth via `--api-key`, `/reranking`, + Anthropic `/v1/messages` shim. +- **Native observability** — `/metrics`, `/health` returns 503 during + model load (proper K8s startup-probe semantics), `/slots` per-slot + tracking. Ollama still has the `/metrics` issue + [#3144](https://github.com/ollama/ollama/issues/3144) open. +- **Stricter structured output** — native GBNF on `/completion`, + JSON-schema-to-GBNF converter, optional `LLAMA_LLGUIDANCE=ON`. +- **Vision coverage for our targets** — llama.cpp ≥ b9095 supports + Qwen3-VL and MiniCPM-V-4.5 natively; Ollama needs the official + `qwen3-vl` tag (community GGUFs broken — split-mmproj + [#14575](https://github.com/ollama/ollama/issues/14575)) and the + `openbmb/minicpm-v4.5` Ollama tag is 8 months stale. + +Ollama still wins for Llama-3.2-Vision (`mllama` cross-attention) and +ecosystem polish (Go/JS SDKs, langchain-ollama, n8n nodes, HA built-in) +— the latter is mooted by fronting llama.cpp with **LiteLLM** at the +gateway. + +## Components + +| Component | Resource | Purpose | +|-----------|----------|---------| +| llama-swap Deployment | `kubernetes_deployment.llama_swap` | One pod, one OpenAI-compat endpoint, hot-swaps model subprocesses | +| llama-swap ConfigMap | `kubernetes_config_map.llama_swap_config` | YAML model entries (cmd, ttl, checkEndpoint) | +| llama-swap Service | `kubernetes_service.llama_swap` | ClusterIP `:8080` → `llama-swap.llama-cpp.svc.cluster.local` | +| Models PVC | `module.nfs_models` (NFS-RWX `/srv/nfs-ssd/llamacpp`) | Shared GGUF store, 30Gi | +| Download Job | `kubernetes_job_v1.download_models` | Pulls Q4_K_M GGUF + mmproj per model, creates stable `model.gguf` / `mmproj.gguf` symlinks, warms page cache | + +## Storage + +NFS-SSD on the Proxmox host (`192.168.1.127:/srv/nfs-ssd/llamacpp`). +Cold model load is ~40s × 3 startups ≈ 2 min in a 25-30 min benchmark +run (<10%). The download Job warms the kernel page cache after pulling +GGUFs so first inference reads from warm cache. + +If steady-state cold-load latency becomes a problem, **Path B**: carve +~50Gi from a Proxmox SSD as an LV, attach as a vdisk to k8s-node1, +mount on-host, expose via a static `kubernetes_persistent_volume` with +`local` source + node1 affinity. NVMe-class load times. Out of scope +for the initial deployment. + +## GPU allocation + +The llama-swap pod requests `nvidia.com/gpu: 1` (whole-T4 +allocation). The shared T4 is also used by Immich's ML pod +(`immich.immich-machine-learning`); only one of the two can hold the +GPU at a time. Operator must scale immich-ml to 0 before running a +benchmark and restore it after: + +```bash +kubectl scale -n immich deploy/immich-machine-learning --replicas=0 +# ... benchmark ... +kubectl scale -n immich deploy/immich-machine-learning --replicas=1 +``` + +## Models served + +| ID | HF repo | Quant | Ctx | mmproj | +|----|---------|-------|-----|--------| +| `qwen3vl-8b` | `Qwen/Qwen3-VL-8B-Instruct-GGUF` | Q4_K_M | 3072 | yes | +| `minicpm-v-4-5` | `openbmb/MiniCPM-V-4_5-gguf` | Q4_K_M | 3072 | yes | +| `qwen3vl-4b` | `Qwen/Qwen3-VL-4B-Instruct-GGUF` | Q4_K_M | 3072 | yes | + +llama.cpp build pinned via the `llama-swap:cuda` image (ships a +recent llama.cpp ≥ b9095, which includes Qwen3-VL projection fix +[#20899](https://github.com/ggml-org/llama.cpp/issues/20899) and +mtmd Flash-Attention regression fix +[#16962](https://github.com/ggml-org/llama.cpp/issues/16962)). + +## Endpoints + +- `GET /v1/models` — list configured models +- `POST /v1/chat/completions` — standard OpenAI chat (vision via + `image_url` content parts, base64 or remote URL) +- `POST /completion` — llama.cpp native completion (preferred for + GBNF-constrained structured output to avoid 2026 regression magnet + on `/v1/chat/completions`) +- `GET /metrics` — Prometheus +- `GET /health` — 200 once a model is fully loaded; 503 during load + +## Known issues / decisions + +- **Cluster-wide GPU contention** — only one of llama-swap or + immich-ml can hold the T4. No GPU sharing solution wired in + (MPS/MIG would help but T4 has no MIG and MPS is overkill for two + workloads). +- **Filename-agnostic config** — the download Job creates stable + `model.gguf` / `mmproj.gguf` symlinks per model dir so the + llama-swap config doesn't need to track exact HF filenames (which + change between releases). +- **TF schema** — `llama-cpp` (PG backend on dbaas). diff --git a/docs/architecture/monitoring.md b/docs/architecture/monitoring.md index 3b9d915d..a182981e 100644 --- a/docs/architecture/monitoring.md +++ b/docs/architecture/monitoring.md @@ -57,7 +57,7 @@ graph TB |-----------|---------|----------|---------| | Prometheus | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Metrics collection and storage, scrape configs for all services | | Grafana | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Visualization, 14+ dashboards (API server, CoreDNS, GPU, UPS, etc.) | -| Loki | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Log aggregation and querying | +| Loki | **DEPLOYED 2026-05-18** (SingleBinary mode, 30d retention, 50Gi PVC on `proxmox-lvm`, ruler enabled → Alertmanager). Re-enabled from previous "operational overhead" disable. Ships logs via Alloy DaemonSet (now on all nodes including master after 2026-05-19 toleration add). | `stacks/monitoring/modules/monitoring/` | Log aggregation and querying | | Alertmanager | Latest (Diun monitored) | `stacks/monitoring/modules/monitoring/` | Alert routing with cascade inhibitions | | Uptime Kuma | Latest (Diun monitored) | `stacks/uptime-kuma/` | Internal + external HTTP monitors, status page | | External Monitor Sync | Python 3.12 | `stacks/uptime-kuma/` | CronJob (10min) syncs `[External]` monitors from `cloudflare_proxied_names` | @@ -176,6 +176,35 @@ The email monitoring system uses a CronJob (`email-roundtrip-monitor`, every 10 Uptime Kuma monitors: TCP SMTP (port 25) on `176.12.22.76` (external), IMAP (port 993) on `10.0.20.202`, and Dovecot exporter metrics on port 9166. +#### Security Alerts (Wave 1 — planned, beads `code-8ywc`) + +Routed via **Loki ruler → Alertmanager → `#security` Slack receiver**. Same handling path as infra alerts. Single channel with severity labels inside (critical/warning/info), not three separate channels. Detection sources: K8s API audit log (`job=kube-audit`), Vault audit log (`job=vault-audit`), PVE sshd journald (`job=sshd-pve`), Calico flow logs (`job=calico-flow`, W1.6 only). + +| # | Source | Event | Severity | +|---|---|---|---| +| K2 | kube-audit | SA token used from outside cluster | critical | +| K3 | kube-audit | Secret read in vault/sealed-secrets/external-secrets by non-allowlisted SA | critical | +| K4 | kube-audit | Exec into vault/kube-system/dbaas/cnpg-system pod by non-allowlisted user | warning | +| K5 | kube-audit | Mass delete (>5 Pod/Secret/CM in 60s) | critical | +| K6 | kube-audit | Audit policy itself modified | critical | +| K7 | kube-audit | New `*,*` ClusterRole created | warning | +| K8 | kube-audit | Anonymous binding granted | critical | +| K9 | kube-audit | `me@viktorbarzin.me` request from non-allowlist sourceIP | critical | +| V1 | vault-audit | Root token created | critical | +| V2 | vault-audit | Audit device disabled/modified | critical | +| V3 | vault-audit | Seal status changed | critical | +| V4 | vault-audit | Policy written/modified (allowlist Terraform actor) | warning | +| V5 | vault-audit | Auth failure spike >10/min | warning | +| V6 | vault-audit | Token with policies different from parent created | critical | +| V7 | vault-audit | Viktor's entity_id from non-allowlist remote_addr (requires `x_forwarded_for_authorized_addrs`) | critical | +| S1 | sshd-pve | sshd auth success from non-allowlist IP | critical | + +K1 (cluster-admin grant) intentionally skipped — see security.md. + +Allowlist source-IP CIDRs (used by K2, K9, V7, S1): `10.0.20.0/22`, `192.168.1.0/24`, K8s pod CIDR, K8s service CIDR, Headscale tailnet. Policy: no public-IP access; all admin paths transit LAN or Headscale. + +IOPS impact estimated ~1-2 GB/day additional disk writes after custom audit-policy tuning. Retention: 90d for security streams. + #### Backup Alerts - **PostgreSQLBackupStale**: >36h since last backup - **MySQLBackupStale**: >36h since last backup diff --git a/docs/architecture/security.md b/docs/architecture/security.md index 3d68bedc..f2e31f49 100644 --- a/docs/architecture/security.md +++ b/docs/architecture/security.md @@ -111,16 +111,20 @@ Namespaces are labeled with a tier (`tier: 0` through `tier: 4`). Kyverno auto-g This prevents resource exhaustion and enforces governance without manual quota management. -#### Security Policies (ALL in Audit Mode) +#### Security Policies -**Why audit mode?** Gradual rollout without breaking existing workloads. Policies collect violations, then selectively enforced after cleanup. +**Why audit mode first?** Gradual rollout without breaking existing workloads. Policies collect violations, then selectively enforced after cleanup. -| Policy | Purpose | Enforcement | -|--------|---------|-------------| -| `deny-privileged-containers` | Block privileged pods | Audit | -| `deny-host-namespaces` | Block hostNetwork/hostPID/hostIPC | Audit | -| `restrict-sys-admin` | Block CAP_SYS_ADMIN | Audit | -| `require-trusted-registries` | Only allow approved image registries | Audit | +**Wave 1 plan (locked 2026-05-18, see beads `code-8ywc`):** all four below flip from Audit → Enforce with `failurePolicy: Ignore` preserved and an exclude list covering the 31 critical namespaces (keel, calico-system, authentik, vault, cnpg-system, dbaas, monitoring, traefik, technitium, mailserver, kyverno, metallb-system, external-secrets, proxmox-csi, nfs-csi, nvidia, kube-system, cloudflared, crowdsec, reverse-proxy, reloader, descheduler, vpa, redis, sealed-secrets, headscale, wireguard, xray, infra-maintenance, metrics-server, tigera-operator). Phased: one policy per day with PolicyReport observation. + +| Policy | Purpose | Current | Planned (wave 1) | +|--------|---------|---------|------------------| +| `deny-privileged-containers` | Block privileged pods | Audit | **Enforce** | +| `deny-host-namespaces` | Block hostNetwork/hostPID/hostIPC | Audit | **Enforce** | +| `restrict-sys-admin` | Block CAP_SYS_ADMIN | Audit | **Enforce** | +| `require-trusted-registries` | Only allow approved image registries (forgejo.viktorbarzin.me, docker.io, ghcr.io, quay.io, registry.k8s.io, gcr.io, oci://ghcr.io/sergelogvinov) | Audit | **Enforce** | + +Cosign `verify-images` is **deferred** beyond wave 1 — needs image-signing infrastructure (Sigstore / cosign + KMS) before it can enforce meaningfully. #### Operational Policies @@ -163,6 +167,112 @@ Removed April 2026. The rewrite-body Traefik plugin used to inject hidden trap l **Implementation**: See `stacks/poison-fountain/` and `stacks/platform/modules/traefik/middleware.tf` +### Audit Logging & Anomaly Detection (Wave 1) + +Beads epic: `code-8ywc`. **Status: partially live as of 2026-05-18.** + +| Item | State | +|---|---| +| W1.2 Vault `file` audit device | **LIVE** — `vault_audit.file` in `stacks/vault/main.tf:287`, writing to `/vault/audit/vault-audit.log` on `proxmox-lvm-encrypted` PVC | +| W1.2 Vault `x_forwarded_for_authorized_addrs = 10.10.0.0/16` | **LIVE** — applied via `tg apply -target=helm_release.vault` on 2026-05-18; all 3 vault pods restarted cleanly | +| W1.2 Vault audit log shipping to Loki | **LIVE** — `audit-tail` sidecar in vault pods + Alloy DaemonSet ships to Loki with `container="audit-tail"`. Verified via `{namespace="vault",container="audit-tail"}` LogQL query. | +| W1.1 K8s API audit policy + shipping | **LIVE** — kube-apiserver audit policy was already configured (Metadata level, `/var/log/kubernetes/audit.log`, 7d retention). Alloy DaemonSet now tolerates control-plane taint, scrapes the audit log file, ships to Loki with `job=kubernetes-audit`. K2-K9 alert rules in Loki ruler. | +| W1.3 Source-IP anomaly rules (K9, V7, S1) | **LIVE** (K9, V7); **S1 PENDING** — fires once promtail/Alloy on PVE host ships sshd journal with `job=sshd-pve`. | +| W1.4 Kyverno security policies → Enforce | **LIVE** — 3 policies in Enforce mode with 35-namespace exclude list. | +| W1.5 Kyverno trusted-registries → Enforce | **LIVE** — explicit allowlist (15 registries + 6 DockerHub library bare names + 56 DockerHub user repos). Verified by admission dry-run: `evilcorp.example/malware:v1` BLOCKED, `alpine:3.20` and `docker.io/library/alpine:3.20` ALLOWED. | +| W1.6 Calico observe-phase (pilot: recruiter-responder) | **LIVE** (2026-05-19) — GlobalNetworkPolicy `wave1-egress-observe-recruiter-responder` with rules `[action:Log, action:Allow]`. FelixConfiguration.flowLogsFileEnabled approach abandoned (Calico Enterprise-only field, rejected by OSS v3.26). Log action emits iptables LOG with prefix `calico-packet: ` → kernel → journald → Alloy → Loki. Verified: `{job="node-journal"} \|~ "calico-packet"` returns real packet metadata (SRC/DST/PROTO). Expand to more namespaces by adding to `namespaceSelector`. | +| W1.7 NetworkPolicy phased enforce | **PENDING** — needs ~1 week of W1.6 observation, then build empirical allowlist from Loki queries, flip GNP rules from `[Log, Allow]` to `[Allow specific dests, Deny rest]`. | + +The block below documents the locked design. + +Response model: **(I) Slack-only, daily skim.** All security alerts land in a new `#security` Slack channel via Alertmanager. No paging. Mean detection time accepted as ~12-24h; the design weight sits on prevention (Kyverno enforce, NetworkPolicy default-deny egress) rather than runtime detection. + +#### Detection sources + +| Source | Mechanism | Ships via | Loki job label | +|---|---|---|---| +| K8s API audit log | Custom audit policy on kube-apiserver: drop `get`/`list`/`watch` at `None` for most resources, log writes at `Metadata`, secret reads at `Metadata`, `exec`/`portforward` at `RequestResponse`, exclude kubelet+controller-manager noise. Codified in `stacks/infra` kubeadm config templating. | Alloy DaemonSet tails `/var/log/kubernetes/audit/*.log` | `job=kube-audit` | +| Vault audit log | `file` audit device on existing Vault PVC. Vault listener config sets `x_forwarded_for_authorized_addrs` trusting Traefik pod CIDR so `remote_addr` is the real client IP, not Traefik's. | Alloy tails audit log file | `job=vault-audit` | +| PVE sshd auth log | journald `_SYSTEMD_UNIT=ssh.service` | promtail systemd unit on Proxmox host (192.168.1.127) | `job=sshd-pve` | +| Calico flow log | `flowLogsFileEnabled: true` in Calico Felix config | Alloy (cluster-wide) | `job=calico-flow` (W1.6 only) | + +#### Alert rules (16 total) + +Routed via **Loki ruler → Alertmanager → `#security` Slack receiver**. Same handling path as existing infra alerts — silenceable in Alertmanager UI, history queryable, severity labels (critical/warning/info) inside the single `#security` channel. + +**K8s API audit (K2-K9, 8 rules — K1 cluster-admin-grant intentionally skipped):** + +| # | Event | Severity | +|---|---|---| +| K2 | ServiceAccount token used from outside cluster (sourceIPs not in pod CIDR or trusted LAN) | critical | +| K3 | Secret READ in `vault`, `sealed-secrets`, `external-secrets` namespaces by a non-allowlisted ServiceAccount | critical | +| K4 | Exec into a pod in `vault`, `kube-system`, `dbaas`, `cnpg-system` (excluding `me@viktorbarzin.me` + 1 break-glass SA) | warning | +| K5 | >5 deletes of `Pod`, `Secret`, or `ConfigMap` in 60s by any single actor | critical | +| K6 | `audit-log-path` flag or audit policy modified on kube-apiserver | critical | +| K7 | New ClusterRole created with `verbs: ["*"]` and `resources: ["*"]` | warning | +| K8 | Anonymous binding granted (any RoleBinding/CRB referencing `system:anonymous` or `system:unauthenticated`) | critical | +| K9 | Authenticated request where `user.username == "me@viktorbarzin.me"` AND `sourceIPs[0]` NOT in allowlist CIDRs | critical | + +**Vault audit (V1-V7):** + +| # | Event | Severity | +|---|---|---| +| V1 | Root token created | critical | +| V2 | Audit device disabled or modified | critical | +| V3 | Seal status changed (`sys/seal` write) | critical | +| V4 | Policy written or modified (allowlist Terraform-driven writes by source IP / token role) | warning | +| V5 | Authentication failure spike >10/min on any auth method | warning | +| V6 | Token created with policies different from parent (privilege escalation) | critical | +| V7 | Vault audit event where `auth.entity_id == ` AND `remote_addr` NOT in allowlist CIDRs | critical | + +**Host (S1):** + +| # | Event | Severity | +|---|---|---| +| S1 | PVE sshd auth success from source IP NOT in allowlist | critical | + +#### Allowlist — "expected source IPs" for K2, K9, V7, S1 + +| CIDR | Source | +|---|---| +| `10.0.20.0/22` | VLAN 20 (K8s cluster + main LAN) | +| `192.168.1.0/24` | Proxmox host LAN + Sofia LAN (same RFC1918 block in both physical locations; cross-site traffic transits Headscale so the CIDR matches only on-LAN clients in either location) | +| K8s pod CIDR (verify at implementation time) | In-cluster pods talking to apiserver | +| K8s service CIDR | Service-to-apiserver traffic | +| Headscale tailnet | VPN-connected devices | + +**Policy: no public-IP access ever.** Vault, kube-apiserver, PVE sshd must transit a trusted LAN or Headscale. Anything else fires an alert. + +#### Why no canary tokens + +Original plan included canary tokens (fake K8s Secret, Vault KV path, PVE file, sinkhole hostname). Rejected because Viktor routinely greps `secret/viktor` (135 keys) and lists `kubectl get secret -A` — any read-trigger canary self-fires. Use-based canaries (zero-RBAC SA tokens with audit alerts on use) were also considered but rejected in favor of cleaner source-IP anomaly detection (K9, V7) on REAL tokens — same threat model, no fake-token operational burden. + +#### Why no K1 (cluster-admin grant detection) + +Viktor opted out. Gap covered indirectly by K7 (new `*,*` ClusterRole created), K8 (anonymous binding), and K3 (secret read on Vault namespace) — most attacker progressions toward cluster-admin trigger one of these. + +#### IOPS / disk-wear + +Custom audit policy reduces volume ~80-90% vs default Metadata-everywhere. Loki tuned for fewer larger chunks: `chunk_target_size: 1.5MB`, `chunk_idle_period: 30m`, snappy compression. Retention 90d for security streams (matches Technitium DNS query log precedent). Net estimate: ~1-2 GB/day additional disk writes after tuning. + +### NetworkPolicy Default-Deny Egress (Wave 1 — observe-then-enforce, tier 3+4) + +Beads: `code-8ywc` W1.6 + W1.7. **Status: planned.** + +**Approach (γ): cluster-wide observe-then-enforce.** + +1. **Week 0:** Enable Calico flow logs cluster-wide. Apply a GlobalNetworkPolicy with selector `tier in {tier-3, tier-4}`, `action: Log` (no Deny). Ship flow logs to Loki. +2. **Week 1:** Build per-namespace egress allowlist from observed traffic. Common allowlist module `tier3_egress_baseline` covers DNS, NTP, internal Vault/ESO/Authentik, Brevo SMTP, Cloudflare API, OAuth providers. Per-namespace add-ons for service-specific external destinations. +3. **Week 2-3:** Apply default-deny + allowlist per-namespace, starting `recruiter-responder` (smallest egress footprint — local llama-cpp). Watch 24-48h per namespace, iterate. Roll out 3-5 namespaces/day. + +**Scope exclusions:** tier 0/1/2 namespaces (defer to wave 2), 31 critical infra namespaces (same exclude list as Kyverno). + +**DNS handling:** Calico GlobalNetworkPolicy supports domain-based rules via the `domains:` selector which queries CoreDNS internally. Static IPs reserved for fixed-IP services (Brevo SMTP relay). + +**Known risks:** +- Rare-event misses: a Sunday-only CronJob's egress won't appear in 7 days of flow logs. Mitigation: extend observation to 2 weeks for namespaces with weekly CronJobs. +- Mass-rollout cascade: the 26h March 2026 outage (memory id=390) was a mass-change cascade. Mitigation: phased per-namespace with health-check pauses, similar to the 2026-05-17 Keel phased rollout (memory id=1972). + ### TLS & HTTP/3 **Traefik** handles TLS termination: diff --git a/docs/benchmarks/2026-05-10-vision-llm.md b/docs/benchmarks/2026-05-10-vision-llm.md new file mode 100644 index 00000000..70dafedf --- /dev/null +++ b/docs/benchmarks/2026-05-10-vision-llm.md @@ -0,0 +1,253 @@ +# Vision-LLM benchmark — Malaga / Seville album + +**Run ID:** `2026-05-10-1424` · **Date:** 2026-05-10 · **Operator:** wizard + +100 photos randomly sampled (seed=42) from the Immich album `🇪🇸 Malaga +Seville` (`46565b85-7580-4ac1-91a6-1ece2cf8634d`, 1556 image assets + +9 videos), scored by three local vision-LLMs served by `llama-swap` +on a single Tesla T4. Goal: pick a model to wire into +`instagram-poster`'s `/candidates` ranking path. + +## TL;DR + +**Recommendation: `qwen3vl-4b`.** + +- **Fastest** by a wide margin (3.55 s p50, 60% of qwen3vl-8b), + important once this is in the request path of `/candidates`. +- **100% structured-output success** — same as the other two; GBNF + grammar enforcement worked across the board. +- **Captions are competitive** with the 8B model in qualitative review + (tied or close on 8/10 sampled photos; 8B wins on Flair, 4B wins on + Latency). +- **Most decisive scorer** — 47/100 photos got IG-fit=9 vs 17 for + qwen3vl-8b and 9 for minicpm. We get more signal at the top end + for ranking. + +Use qwen3vl-8b for *manual* caption refinement (top-1 of the day) if +caption polish matters. Use minicpm-v-4-5 for nothing immediate — it's +the most conservative scorer and the slowest at high quantiles, with +no offsetting wins in this dataset. + +## Setup + +- Hardware: 1× Tesla T4 (16 GiB VRAM), `nvidia.com/gpu` time-slicing + enabled (replicas=100), pod scheduled on `k8s-node1`. +- Server: `mostlygeek/llama-swap:cuda` (ships llama.cpp `b9085-046e28443`) + on `llama-swap.llama-cpp.svc.cluster.local:8080`. +- Models: GGUF Q4_K_M, mmproj F16 except qwen3vl-4b which used the + Q8_0 mmproj (alphabetically first matching the glob). +- Image prep: EXIF-transposed, long-edge resized to 1024 px, JPEG q=90, + base64-embedded as `image_url` data URLs. +- Generation: `temperature=0`, `top_k=1`, `enable_thinking=false`, + GBNF grammar pinning the JSON schema (6 fields, 1–10 ints, ≤8 tags). +- Run isolation: `immich-machine-learning` scaled to 0 for the + duration to avoid noisy GPU contention. *(Diagnostic note: the + scheduling failure that triggered this was actually node1 RAM — + not GPU — at 94% allocated. Time-slicing was already on. Bumping + node1 RAM is tracked as a follow-up.)* + +## Headline numbers + +| model | n | parse_ok | p50 latency | p95 latency | median IG-fit | median aesthetic | +|-------|---|----------|-------------|-------------|---------------|------------------| +| **qwen3vl-4b** | 100 | 100% | **3.55 s** | 4.06 s | 8.0 | 8.0 | +| minicpm-v-4-5 | 100 | 100% | 5.62 s | 6.00 s | 7.0 | 8.0 | +| qwen3vl-8b | 100 | 100% | 5.98 s | 6.64 s | 7.0 | 8.0 | + +Total wall time for the run: **33 m 32 s** (300 calls + 3 cold loads +of ~30 s each). + +## What each model is good at + +### qwen3vl-4b — fast and decisive +- p50 3.55 s — comfortable for adding to `/candidates` request path. +- IG-fit distribution skews right (47 nines), spreading 6 → 9 fairly + evenly, which is what you want from a *ranker*. +- Captions are emoji-friendly, hashtag-friendly, sometimes + hallucinatory (e.g. labelled a Seville street as "Barcelona's + colourful streets" once). +- Failure mode to watch: occasional double-down on the same caption + template ("Lost in the tiles. 🌿" repeated across two unrelated + blue-dress photos). + +### minicpm-v-4-5 — conservative, terse +- Most conservative scorer: 65% of photos got IG-fit=7. Only 9 nines. + Less useful as a top-N ranker because the top is squashed. +- Fastest p95 of the three (6.0 s) but slower p50 than qwen3vl-4b. +- Captions are short and lower-case ("azulejo dreams.", + "sunshine & secrets") — distinct voice but less Instagram-native. + +### qwen3vl-8b — most polished captions +- Best subject identification (specifically named "Metropol Parasol" + and "Plaza de España" by name where the others said "modern + architecture" / "plaza"). +- Captions read well: "Coffee & calm vibes ☕️", "where modern meets + historic under a brilliant sky". +- Slowest p50 (5.98 s) and tightest score distribution (median 7, + 17 nines) — middle of the pack as a ranker. + +## Top-10 agreement (Kendall-tau-style overlap) + +How many of each model's top-10 IG-fit picks appear in another +model's top-10: + +| pair | overlap | +|------|---------| +| qwen3vl-4b ↔ qwen3vl-8b | 5/10 | +| minicpm-v-4-5 ↔ qwen3vl-4b | 4/10 | +| minicpm-v-4-5 ↔ qwen3vl-8b | 4/10 | + +Read: there's moderate but not strong agreement. The models pick +roughly half the same "best" photos and half different ones. For +ranking, that's a healthy sign — they're not collapsing to a single +notion of "good", so combining their scores would add real signal. + +## Cost-equivalent context + +Approximate cost to score the same 100 photos via cloud APIs +(prompt ≈ 1100 tokens incl. image, completion ≈ 100 tokens): + +| backend | input | output | per-100 photos | +|---------|-------|--------|----------------| +| Local llama-swap on T4 | — | — | ≈ $0.04 (electricity, ~70 W × 7 min) | +| Anthropic Haiku 4.5 | $1.00/M | $5.00/M | ≈ $0.15 | +| Anthropic Sonnet 4.6 | $3.00/M | $15.00/M | ≈ $0.45 | +| Google Gemini 2.5 Flash | $0.30/M | $2.50/M | ≈ $0.05 | + +Local is competitive with Gemini Flash on marginal cost. The case +for keeping it local is privacy (Immich originals never leave the +LAN), no rate-limits, and no per-call quota planning. The case +against is the GPU is finite — adding this to a request path means +sharing T4 time with frigate, ytdlp, and (when we restore it) +immich-ml. With time-slicing on, that's tractable but each tenant +gets fewer full-GPU kernels. + +## Sample captions (10 representative photos, all 3 models) + +`9f9c9faf-...` — man at ornate arch in Seville +- **minicpm-v-4-5** (7/8): sunshine & secrets +- **qwen3vl-4b** (9/8): Found the perfect spot to pause. 🌞 #travel #architecture +- **qwen3vl-8b** (9/8): Standing in Seville's magic, framed by history and color. + +`0d53158b-...` — Metropol Parasol +- **minicpm-v-4-5** (9/8): Seville's skyward statement. +- **qwen3vl-4b** (9/8): Seville's iconic modern structure against a vibrant blue sky. +- **qwen3vl-8b** (9/8): The Metropol Parasol in Seville, where modern meets historic under a brilliant sky. + +`23137c1c-...` — coffee shot +- **minicpm-v-4-5** (8/7): a morning pick-me-up and a smile. +- **qwen3vl-4b** (8/7): Coffee & chill. Perfectly imperfect moment. +- **qwen3vl-8b** (9/8): Coffee & calm vibes ☕️ + +`3376ee87-...` — woman against tiled wall +- **minicpm-v-4-5** (7/8): azulejo dreams. +- **qwen3vl-4b** (9/8): Lost in the tiles. 🌿 +- **qwen3vl-8b** (9/8): Blue tiles, golden light, and a moment of quiet grace. + +`4f2fea45-...` — courtyard +- **minicpm-v-4-5** (7/8): hidden gems of seville +- **qwen3vl-4b** (7/8): Timeless beauty in a Spanish courtyard. 🌿 +- **qwen3vl-8b** (7/8): A serene courtyard in Seville, where palm trees sway under the sun. + +`ea713729-...` — flower-market street (qwen3vl-4b confused location) +- **minicpm-v-4-5** (7/8): Seville's hidden gems. +- **qwen3vl-4b** (7/8): Walking through *Barcelona's* colorful streets, backlit by golden hour. +- **qwen3vl-8b** (7/8): Walking through Seville's vibrant streets, lavender in hand. + +The full list of 10 sample sets is in the auto-generated section +below; the raw 300-row JSON is at `benchmark-2026-05-10-1424.json` +in this directory. + +## Operational cost during the run + +- llama-swap pod (1× T4 wholly allocated for the duration): ~33 min. +- Immich-ML downtime: ~33 min. New uploads weren't auto-tagged or + CLIP-embedded during this window. No user-visible impact (Immich + search against already-indexed assets still worked via pgvector). +- Network egress: zero — Immich originals stayed on the LAN, all + scoring traffic was in-cluster. + +## Reproducibility + +```bash +DATA_DIR=/tmp/benchmark \ + IMMICH_API_KEY=… \ + LLAMA_SWAP_URL=http://localhost:18080 \ + poetry run python -m instagram_poster.benchmark run \ + --album-id 46565b85-7580-4ac1-91a6-1ece2cf8634d \ + --models qwen3vl-8b,minicpm-v-4-5,qwen3vl-4b \ + --limit 100 --random-seed 42 --run-id 2026-05-10-1424 +``` + +The same `--random-seed` reproduces the photo sample exactly. Prompt +version `4bbb7e7721da24d9` is the SHA-256 of the system prompt + user +prompt + GBNF grammar; rerunning under the same prompt version against +the same seed should produce within-noise identical scores (the models +themselves are temperature=0, top_k=1). + +## Next steps + +- **Wire `qwen3vl-4b` into `instagram-poster`** as an additional ranking + signal alongside CLIP-based recency in `/candidates`. Cache the score + per asset_id so we don't re-pay 4 s on every list refresh. +- **Bump k8s-node1 RAM** so immich-ml + llama-swap can co-exist (drain + → resize → uncordon, with kubelet `systemReserved` adjusted in + `stacks/infra/main.tf`). +- **Re-benchmark with shared GPU** once node1 RAM is bumped, to get + realistic latency numbers when the T4 is also under load from + immich-ml and frigate. +- **Front llama-swap with LiteLLM** so Home Assistant and any other + consumer can hit one OpenAI-compat gateway. Track separately. + +--- + +## Auto-generated report + +Below is the unedited output of `python -m instagram_poster.benchmark +report --run-id 2026-05-10-1424`, kept for diff-checking against +future runs. + +### Per-model summary + +| model | n | parse_ok % | error % | p50 latency | p95 latency | median IG-fit | median aesthetic | +|-------|---|-----------|--------|------------|-------------|--------------|------------------| +| minicpm-v-4-5 | 100 | 100.0 | 0.0 | 5617 ms | 5998 ms | 7.0 | 8.0 | +| qwen3vl-4b | 100 | 100.0 | 0.0 | 3552 ms | 4063 ms | 8.0 | 8.0 | +| qwen3vl-8b | 100 | 100.0 | 0.0 | 5981 ms | 6637 ms | 7.0 | 8.0 | + +### Score histograms (instagram_fit_score 1–10) + +#### minicpm-v-4-5 +``` + 1: (0) 2: (0) 3: (0) 4: (0) 5: (0) + 6: ███████ (7) + 7: █████████████████████████████████████████████████████████████████ (65) + 8: ███████████████████ (19) + 9: █████████ (9) +10: (0) +``` + +#### qwen3vl-4b +``` + 1: (0) 2: (0) 3: (0) 4: (0) 5: (0) + 6: █████ (5) + 7: ████████████████ (16) + 8: ████████████████████████████████ (32) + 9: ███████████████████████████████████████████████ (47) +10: (0) +``` + +#### qwen3vl-8b +``` + 1: (0) 2: (0) 3: (0) 4: (0) 5: (0) + 6: ███████████ (11) + 7: ███████████████████████████████████████████████████████ (55) + 8: █████████████████ (17) + 9: █████████████████ (17) +10: (0) +``` + +### Top-10 by IG-fit per model — see `benchmark-2026-05-10-1424.json` + +(Tables omitted from the curated report; available in the JSON dump +alongside this file.) diff --git a/docs/benchmarks/benchmark-2026-05-10-1424.json b/docs/benchmarks/benchmark-2026-05-10-1424.json new file mode 100644 index 00000000..e72e3537 --- /dev/null +++ b/docs/benchmarks/benchmark-2026-05-10-1424.json @@ -0,0 +1,7949 @@ +[ + { + "run_id": "2026-05-10-1424", + "asset_id": "9f9c9faf-6bce-4ceb-a1b5-e63179c55990", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man at ornate arch", + "scene_tags": [ + "seville", + "arch", + "palm trees", + "blue sky", + "garden", + "travel", + "architecture", + "portrait" + ], + "suggested_caption": "Standing in Seville\u2019s magic, framed by history and color.", + "reasoning": "Strong vertical framing with arch as natural frame, vibrant colors, and clear subject draw immediate attention.", + "latency_ms": 37886, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fec017d6-54cc-4ee9-89f8-6dd0109e6603", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "curly-haired woman", + "scene_tags": [ + "indoor", + "plants", + "window", + "denim", + "portrait", + "natural_light", + "casual", + "smile" + ], + "suggested_caption": "Sunlight, curls, and a cozy corner.", + "reasoning": "Strong subject with warm light and texture, but slightly off-center framing reduces IG impact.", + "latency_ms": 4969, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0d53158b-5a17-4ecc-8e92-0195fb6b12a7", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "Metropol Parasol", + "scene_tags": [ + "seville", + "architecture", + "blue_sky", + "modern", + "historic", + "sunset", + "canopy", + "spain" + ], + "suggested_caption": "The Metropol Parasol in Seville, where modern meets historic under a brilliant sky.", + "reasoning": "Strong diagonal composition and vibrant blue sky create visual impact, perfect for IG\u2019s vertical scroll.", + "latency_ms": 5271, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "23137c1c-f475-4b9a-8c72-e86c90f8f064", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man with coffee drinks", + "scene_tags": [ + "cafe", + "matcha", + "latte", + "smile", + "indoor", + "wooden_table", + "paper_lamp", + "relaxed" + ], + "suggested_caption": "Coffee & calm vibes \u2615\ufe0f", + "reasoning": "Strong vertical composition with warm lighting and engaging subject, perfect for IG scroll-stopping aesthetic.", + "latency_ms": 5297, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3376ee87-8bf5-4710-88e0-f0d89b3587fc", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman against tiled wall", + "scene_tags": [ + "tiled_wall", + "blue_dress", + "curly_hair", + "andalusian", + "portrait", + "travel", + "elegant", + "south_europe" + ], + "suggested_caption": "Blue tiles, golden light, and a moment of quiet grace.", + "reasoning": "Strong vertical composition with vibrant, patterned background and subject\u2019s confident gaze \u2014 highly scroll-stopping.", + "latency_ms": 5495, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4f2fea45-3119-4df2-bca8-d35a85fd1aa9", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "courtyard with palm trees", + "scene_tags": [ + "seville", + "palm trees", + "courtyard", + "fountain", + "arches", + "garden", + "sunset", + "historic" + ], + "suggested_caption": "A serene courtyard in Seville, where palm trees sway under the sun.", + "reasoning": "Strong composition with vibrant colors and vertical framing, but slightly cluttered foreground reduces Instagram appeal.", + "latency_ms": 5430, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e49bb267-b7da-4528-a37e-aa92770a0271", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "cityscape", + "stone wall", + "curly hair", + "blue dress", + "overcast", + "travel", + "portrait", + "elegant" + ], + "suggested_caption": "Standing tall in the city, blue against the sky.", + "reasoning": "Strong subject with vibrant blue dress contrasting muted cityscape, but overcast sky reduces visual pop for IG.", + "latency_ms": 5721, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2df74bf4-e363-4c43-a0ea-8077666229f6", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "garden", + "fountain", + "roses", + "stone", + "people", + "greenery", + "outdoor", + "elegant" + ], + "suggested_caption": "Blue in the garden, surrounded by roses and stone.", + "reasoning": "Strong composition with framing and color contrast, but slightly cluttered background reduces IG impact.", + "latency_ms": 5503, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "750f9142-5e86-468a-89db-e9f28d6aac05", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman by pool with ducks", + "scene_tags": [ + "pool", + "ducks", + "sunlight", + "tiles", + "brick wall", + "mediterranean", + "relaxing", + "summer" + ], + "suggested_caption": "Sun-drenched moments by the pool with unexpected duck guests \ud83e\udd86", + "reasoning": "Strong composition with warm light and vibrant tiles, but slightly cluttered framing reduces IG scroll-stopping power.", + "latency_ms": 5679, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 118 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ea713729-51b7-4005-834b-bd3064455928", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in lavender walking", + "scene_tags": [ + "seville", + "architecture", + "street", + "lunch", + "purple", + "sunset", + "people", + "european" + ], + "suggested_caption": "Walking through Seville\u2019s vibrant streets, lavender in hand.", + "reasoning": "Strong vertical composition with colorful architecture and a stylish subject, but slightly busy background reduces IG impact.", + "latency_ms": 5424, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b0734c0-589c-451a-a41f-8fc45d757ccc", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tiled archway", + "andalusian", + "blue dress", + "travel", + "portrait", + "art", + "doorway", + "elegant" + ], + "suggested_caption": "Standing in a doorway of art and history.", + "reasoning": "Strong composition with vibrant tiles framing the subject, but slightly cluttered background reduces Instagram scroll-stopping power.", + "latency_ms": 5479, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e6ebf7b0-7f4d-40b3-919e-3986cbd595fe", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "seville plaza architecture", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "sunset", + "carriage", + "spain", + "historical", + "sunset" + ], + "suggested_caption": "Seville\u2019s Plaza de Espa\u00f1a, where history meets the sun.", + "reasoning": "Strong architectural composition and vibrant colors, but slightly cluttered foreground and vertical framing limits Instagram impact.", + "latency_ms": 5378, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ee9d68c9-e20c-45a9-8d69-7fe7cecda6c6", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "modern architecture", + "scene_tags": [ + "seville", + "architecture", + "solarium", + "blue sky", + "urban", + "sunset", + "southern europe", + "modern" + ], + "suggested_caption": "Sculpted light and shadow in Seville\u2019s Metropol Parasol.", + "reasoning": "Strong architectural composition with dynamic shadows, but slightly cluttered foreground and vertical framing limits scroll-stopping impact.", + "latency_ms": 5697, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 117 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "eade13f0-608b-4e5b-a11c-610287226371", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "blue dress", + "sunset", + "architecture", + "travel", + "spain", + "bridge" + ], + "suggested_caption": "Sunset at the Plaza de Espa\u00f1a, Seville.", + "reasoning": "Strong vertical framing with vibrant colors and iconic architecture, but slightly soft focus on subject.", + "latency_ms": 5415, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "75cfcb0b-fd7b-4c03-a399-f6253e3db200", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "bullring overlooking ocean", + "scene_tags": [ + "malaga", + "bullring", + "ocean", + "cityscape", + "trees", + "overlook", + "cloudy", + "path" + ], + "suggested_caption": "Bullring view from the hillside path.", + "reasoning": "Strong composition with natural framing, but muted colors and cloudy sky reduce visual punch for Instagram.", + "latency_ms": 5598, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6d2920c8-08d1-4bef-9abc-bc91c446570f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "arches", + "fountain", + "sunlight", + "greenery", + "travel", + "elegant", + "mediterranean" + ], + "suggested_caption": "Sun-drenched moments in a hidden courtyard.", + "reasoning": "Strong composition with warm tones and clear subject, but slightly busy background reduces IG impact.", + "latency_ms": 5378, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c2769ba4-f2c0-49c1-a4b8-6ce0e083f8ec", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "historic town hall", + "scene_tags": [ + "seville", + "architecture", + "sunset", + "bicycle", + "flags", + "palm", + "staircase", + "european" + ], + "suggested_caption": "Seville\u2019s grand town hall, bathed in golden hour light.", + "reasoning": "Strong architectural composition with warm lighting, but the bicycle sign and modern elements slightly distract from the classic aesthetic.", + "latency_ms": 5772, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "17905afc-be7f-4664-948a-1c9da287b703", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "seville", + "courtyard", + "palm trees", + "flowers", + "architecture", + "travel", + "garden", + "blue dress" + ], + "suggested_caption": "Lost in the beauty of Seville\u2019s Alc\u00e1zar gardens.", + "reasoning": "Strong composition with vibrant colors and cultural context, but slightly busy background reduces Instagram scroll-stopping power.", + "latency_ms": 5443, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "40f82498-2e9c-4b98-8ebb-ed9821c179a0", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "urban street scene", + "scene_tags": [ + "marseille", + "architecture", + "sky", + "cars", + "street", + "modern", + "historic", + "bicycles" + ], + "suggested_caption": "Marseille\u2019s vibrant mix of old and new under a bright blue sky.", + "reasoning": "Strong composition with varied architecture and sky, but cluttered foreground and lack of a clear focal point reduce Instagram appeal.", + "latency_ms": 5691, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5a879592-e59b-4a79-8886-dc089e301f5a", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "sunset", + "architecture", + "travel", + "blue", + "sunset", + "elegant" + ], + "suggested_caption": "Seville\u2019s magic in one frame \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong vertical composition with vibrant colors and iconic architecture, perfect for IG scroll-stopping appeal.", + "latency_ms": 5828, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "885508f8-f39b-484e-8959-929111ab0b0d", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "bridge", + "blue dress", + "sun", + "architecture", + "canal", + "tourist" + ], + "suggested_caption": "Standing in the heart of Seville\u2019s Plaza de Espa\u00f1a.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, but slightly busy background reduces Instagram scroll-stopping power.", + "latency_ms": 5758, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90b60141-efa8-4a34-8e0a-dda28f296f58", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "flowers", + "bench", + "portuguese", + "architecture", + "summer", + "elegant", + "travel" + ], + "suggested_caption": "Blue dress, pink blooms, and sun-dappled stone. Portugal\u2019s charm in motion.", + "reasoning": "Strong composition with vibrant colors and cultural details, but slightly off-center subject reduces IG impact.", + "latency_ms": 5872, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "876f997c-e9be-40e2-8482-ea0216b82d5a", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man on ornate bench", + "scene_tags": [ + "greenery", + "bench", + "portrait", + "outdoor", + "blue_shirt", + "nature", + "casual", + "serene" + ], + "suggested_caption": "Chillin' in the greenery, just me and the vines.", + "reasoning": "Strong composition with natural light and vibrant green backdrop, but slightly muted colors and no immediate visual hook for IG scroll.", + "latency_ms": 6140, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d37f1824-5d53-41ca-8c79-ffae48978442", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "garden", + "fountain", + "columns", + "blue_dress", + "flowers", + "arches", + "serene" + ], + "suggested_caption": "Lost in the beauty of ancient stone and blooms.", + "reasoning": "Strong composition with vibrant colors and classic architecture, but slightly cluttered foreground reduces IG impact.", + "latency_ms": 5951, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7a13fad2-6ef9-44ea-944c-551e854b6acd", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "cityscape", + "curly_hair", + "blue_dress", + "stone_wall", + "overlook", + "cloudy", + "travel" + ], + "suggested_caption": "Blue against the city. \ud83c\udf06", + "reasoning": "Strong vertical framing with vibrant blue dress contrasting muted cityscape, instantly engaging for IG scroll.", + "latency_ms": 6211, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "64ebfd3c-e9a1-49c2-9d0c-a81182ef6af2", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tiled_wall", + "blue_dress", + "curly_hair", + "portrait", + "travel", + "mediterranean", + "gold_accessories", + "smile" + ], + "suggested_caption": "Blue tiles, golden light, and a little joy.", + "reasoning": "Strong composition with vibrant, harmonious colors and a captivating subject, but slightly less vertical framing for IG scroll stops.", + "latency_ms": 6130, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d211ea44-4f3c-44d0-b7f0-cb9d001f07af", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in flowing dress", + "scene_tags": [ + "tropical", + "greenery", + "sunlight", + "corridor", + "dress", + "portrait", + "outdoor", + "vines" + ], + "suggested_caption": "Dancing in the dappled light \ud83c\udf3f", + "reasoning": "Strong subject and vibrant colors with dynamic lighting, but slightly off-center framing reduces IG impact.", + "latency_ms": 5586, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "30015fce-9f43-430e-8268-36cb4cb56596", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman with curly hair", + "scene_tags": [ + "garden", + "sunlight", + "curly hair", + "earrings", + "outdoor", + "portrait", + "greenery", + "relaxing" + ], + "suggested_caption": "Sun-kissed moments in the garden.", + "reasoning": "Strong portrait with warm light and natural beauty, but slightly off-center framing reduces IG impact.", + "latency_ms": 5714, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9842a059-f560-4243-9710-84a9ef7ee71e", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man at seville plaza", + "scene_tags": [ + "seville", + "plaza", + "bridge", + "sunset", + "architecture", + "travel", + "spain", + "canal" + ], + "suggested_caption": "Sunset at the Plaza de Espa\u00f1a, Seville.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, but subject placement slightly off-center reduces impact.", + "latency_ms": 5796, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f26f6930-16a5-400f-ab7c-ff19abde8ef3", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "bridge", + "blue dress", + "sunset", + "architecture", + "canal", + "tourist" + ], + "suggested_caption": "Blue dress, blue sky, and the best view in Seville.", + "reasoning": "Strong vertical framing with vibrant colors and iconic architecture, but slightly busy background reduces perfection.", + "latency_ms": 5754, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d99cf40e-3bed-47f7-8595-c11286ab3a6a", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman admiring plaza", + "scene_tags": [ + "seville", + "plaza", + "tower", + "sunset", + "travel", + "architecture", + "blue", + "column" + ], + "suggested_caption": "Standing in awe of Seville\u2019s beauty.", + "reasoning": "Strong vertical composition with clear subject and iconic architecture, bright colors and clean lines make it scroll-stopping.", + "latency_ms": 5801, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2e199e4b-3fb5-442f-895e-3ab7ba9ccdaf", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "ornate islamic architecture", + "scene_tags": [ + "mosque", + "marrakech", + "arabic", + "geometric", + "ornate", + "blue", + "gold", + "detail" + ], + "suggested_caption": "The intricate beauty of Moroccan craftsmanship.", + "reasoning": "Richly detailed Islamic patterns with strong vertical framing, but slightly busy for quick-scroll IG appeal.", + "latency_ms": 5503, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "13dc549f-8aee-41b4-92b8-954b5bc41fb2", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "flamenco stage performance", + "scene_tags": [ + "theater", + "flamenco", + "stage", + "audience", + "purple_light", + "guitar", + "suits", + "applause" + ], + "suggested_caption": "Flamenco magic at Teatro Flamenco Sevilla. The energy was electric.", + "reasoning": "Strong stage lighting and vibrant colors create drama, while the audience's raised hands add immediacy for IG engagement.", + "latency_ms": 6356, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 121 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "144aadc1-0e9c-46c7-80dd-8decd901e4ee", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman with curly hair", + "scene_tags": [ + "flowers", + "greenery", + "portrait", + "outdoor", + "curly hair", + "denim", + "bougainvillea", + "smile" + ], + "suggested_caption": "Lost in the bloom \ud83c\udf38", + "reasoning": "Vibrant floral backdrop contrasts beautifully with subject, strong vertical framing and engaging gaze create instant IG appeal.", + "latency_ms": 5804, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3c464a1a-ab0e-4703-9bcd-37ccb17f8dac", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly-haired woman", + "scene_tags": [ + "street", + "sunlight", + "urban", + "sunglasses", + "denim", + "green_bag", + "trees", + "casual" + ], + "suggested_caption": "sunshine & street style", + "reasoning": "Strong subject with vibrant colors and natural light, but slightly cluttered background reduces visual impact.", + "latency_ms": 5487, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 104 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0f816473-c0f5-44eb-96a1-9c982f7f0343", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "palm trees", + "garden", + "sunset", + "elegant", + "andalusian", + "travel", + "outdoor", + "portrait" + ], + "suggested_caption": "Blue dress, palm trees, and endless skies. \ud83c\udf34", + "reasoning": "Strong composition with vibrant blue dress against lush greenery, but slightly cluttered background reduces IG impact.", + "latency_ms": 5957, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "891f2361-7c42-4163-b37a-c11f7f7126f2", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "stairs", + "sunset", + "architecture", + "travel", + "blue", + "sunset", + "elegant" + ], + "suggested_caption": "blue dress, blue sky, blue tiles. seville magic.", + "reasoning": "Strong vertical composition with vibrant colors and detailed architecture, but slightly busy background reduces impact.", + "latency_ms": 5851, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "711f44c3-d2f8-4878-b587-4863e268dd3c", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "wooden bar interior", + "scene_tags": [ + "bar", + "wooden", + "chairs", + "wine", + "warm", + "industrial", + "european", + "cozy" + ], + "suggested_caption": "Warm wood, vintage charm, and a quiet corner waiting for you.", + "reasoning": "Rich textures and warm lighting create inviting atmosphere, but framing feels slightly cluttered for IG\u2019s vertical scroll.", + "latency_ms": 5949, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "baf172dd-da76-4c3c-80b6-99382aeb4a72", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in flowing dress", + "scene_tags": [ + "greenery", + "archway", + "sunlight", + "stone walkway", + "vines", + "dress", + "portrait", + "serene" + ], + "suggested_caption": "Dancing through sunlight and vines.", + "reasoning": "Strong composition with natural light and vibrant greenery, but slightly busy framing reduces IG scroll-stopping power.", + "latency_ms": 5688, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6a918ad8-de2d-48e9-a378-c7038c9e5178", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "art_gallery", + "portrait", + "blue_dress", + "museum", + "curly_hair", + "elegant", + "classic", + "still_life" + ], + "suggested_caption": "Standing in front of a masterpiece, lost in thought.", + "reasoning": "Strong composition with vibrant blue dress against warm tones, but slightly off-center framing reduces IG impact.", + "latency_ms": 5915, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4b09475f-0eee-46ba-a558-1f0cd3ecd473", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in ornate room", + "scene_tags": [ + "historic", + "tiles", + "wooden beams", + "blue dress", + "interior", + "architectural", + "warm light", + "elegant" + ], + "suggested_caption": "Lost in the beauty of old-world charm.", + "reasoning": "Strong composition with warm lighting and rich textures, but vertical framing slightly compromises the depth of the hallway.", + "latency_ms": 5872, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ff2de652-1f1a-4d5c-a7ee-28166ce7118f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "horse-drawn carriage", + "scene_tags": [ + "horse", + "carriage", + "street", + "trees", + "sunny", + "european", + "cobblestone", + "urban" + ], + "suggested_caption": "A classic horse-drawn carriage glides through a sun-dappled European street.", + "reasoning": "Strong composition with vibrant yellow wheels and white horse, but slightly cluttered background reduces Instagram appeal.", + "latency_ms": 5978, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "852f15e1-94f2-49db-8205-218c85d0a43f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "seville plaza architecture", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "spain", + "sunset", + "bridge", + "flag", + "historical" + ], + "suggested_caption": "The Plaza de Espa\u00f1a, Seville \u2014 where history meets vibrant color.", + "reasoning": "Strong architectural composition and vivid colors make it visually striking, but the wide-angle framing lacks tight focus for IG scroll-stopping impact.", + "latency_ms": 6229, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c869757b-8734-4a8d-8dfb-fad070bf7be1", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "woman walking toward historic gate", + "scene_tags": [ + "andalusian architecture", + "alcazar", + "sunset", + "tourists", + "stone tower", + "sunset", + "sunset", + "sunset" + ], + "suggested_caption": "Walking through history in the Alcazar of Seville", + "reasoning": "Strong architectural subject with clear sky, but composition is slightly cluttered with tourists and lacks a strong visual hook.", + "latency_ms": 6170, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3ba4c50c-6ec4-41b1-bb2b-e9d4eea108f6", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "nighttime cathedral", + "scene_tags": [ + "seville", + "cathedral", + "night", + "statue", + "trees", + "streetlight", + "architecture", + "warm_light" + ], + "suggested_caption": "Seville\u2019s cathedral at night \u2014 timeless beauty under warm lights.", + "reasoning": "Strong architectural subject with warm lighting, but low-res noise and cluttered foreground reduce visual impact.", + "latency_ms": 5667, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e5fc776e-8467-4805-8c35-77383f62d76d", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "woman in flowing dress", + "scene_tags": [ + "green walls", + "sunlight", + "doorway", + "tiled floor", + "curly hair", + "dress", + "outdoor", + "vintage" + ], + "suggested_caption": "Sunlight through the leaves, dancing on my dress.", + "reasoning": "Strong subject and warm lighting, but slightly off-center framing and muted colors reduce overall impact.", + "latency_ms": 5897, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b1a8b354-f9a4-42db-9639-44fd9ecbdd33", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "seville plaza architecture", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "sunset", + "people", + "bridge", + "historical", + "sunset" + ], + "suggested_caption": "Sunset over Seville\u2019s Plaza de Espa\u00f1a", + "reasoning": "Strong architectural composition and vibrant colors, but slightly busy foreground and wide-angle framing reduce Instagram scroll-stopping power.", + "latency_ms": 6310, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ba6f613c-61cd-42a7-aaa9-374d4f0ac058", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man posing by ornate bridge", + "scene_tags": [ + "seville", + "spain", + "bridge", + "architecture", + "sunset", + "travel", + "blue", + "water" + ], + "suggested_caption": "Sunset in Seville. Architecture & vibes.", + "reasoning": "Strong composition with vibrant colors and iconic landmark, but subject placement slightly off-center reduces impact.", + "latency_ms": 6114, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "82391768-2834-4843-9cdd-249bd6780f91", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "fried dumplings", + "scene_tags": [ + "dumplings", + "restaurant", + "wooden_table", + "food", + "closeup", + "indoor", + "sushi", + "appetizer" + ], + "suggested_caption": "Golden dumplings with a tangy twist.", + "reasoning": "Strong food composition with warm tones and shallow depth of field, but slightly off-center framing reduces IG impact.", + "latency_ms": 6637, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "01292c70-996c-43cd-b132-5933d3afbf43", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "flowers", + "arches", + "travel", + "blue dress", + "white walls", + "garden", + "elegant" + ], + "suggested_caption": "Lost in the beauty of a Spanish courtyard.", + "reasoning": "Vibrant flowers and strong vertical framing make it visually engaging, but the subject\u2019s back and casual composition slightly reduce its portfolio appeal.", + "latency_ms": 6542, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "edb6e566-0350-4861-ae89-e32ddcfd4f14", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "andalusian", + "courtyard", + "fountain", + "sunlight", + "potted plants", + "arched walls", + "travel", + "elegant" + ], + "suggested_caption": "Sun-dappled moments in the Alhambra\u2019s gardens.", + "reasoning": "Strong composition with warm tones and clear subject, but slightly busy background reduces IG scroll-stopping power.", + "latency_ms": 6631, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 118 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b9c7f1d5-7336-4860-a38a-c3e1d82ee655", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman with curly hair", + "scene_tags": [ + "street", + "sunglasses", + "curly hair", + "phone", + "sunlight", + "casual", + "outdoor", + "travel" + ], + "suggested_caption": "sunshine, curls, and a little wanderlust.", + "reasoning": "Strong vertical framing and warm light highlight the subject\u2019s vibrant curls, creating an instantly engaging, scroll-stopping aesthetic.", + "latency_ms": 6223, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "967b48e8-531a-4bef-a069-fab60690145f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "pizzeria interior", + "scene_tags": [ + "pizzeria", + "oven", + "lemons", + "italian", + "food", + "glasscase", + "artisan", + "vibrant" + ], + "suggested_caption": "Warm, wood, and wood-fired. The soul of Naples in every slice.", + "reasoning": "Rich colors and layered composition draw the eye, but the horizontal framing and cluttered foreground slightly reduce IG scroll-stopping power.", + "latency_ms": 6675, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 122 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b68453c0-f5a3-4e8e-b231-af92fa2ed182", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "tiled floor", + "blue dress", + "sunlight", + "elegant", + "architecture", + "outdoor", + "portrait" + ], + "suggested_caption": "Blue dress, warm light, timeless elegance.", + "reasoning": "Strong subject and color harmony, but composition feels slightly off-center and lacks a strong visual hook for IG scrolling.", + "latency_ms": 6393, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7f9762e4-afb1-45f5-85f0-fd33e050ed23", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man on ornate bench", + "scene_tags": [ + "garden", + "bench", + "greenery", + "portrait", + "casual", + "outdoor", + "blue_shirt", + "vines" + ], + "suggested_caption": "Chillin' in the greenery \ud83c\udf3f", + "reasoning": "Strong composition with natural light and vibrant greenery, but the subject\u2019s pose is slightly casual for a high-impact IG post.", + "latency_ms": 6708, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c61412cb-4abc-4d1c-b541-f1d67adc3b03", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "ancient wall", + "palm tree", + "cobblestone", + "blue dress", + "medieval", + "outdoor", + "travel", + "elegant" + ], + "suggested_caption": "Blue against the old stones.", + "reasoning": "Strong color contrast and textured backdrop, but slightly muted lighting and casual pose reduce IG impact.", + "latency_ms": 6284, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4719aae4-0783-447a-afc2-9f84c7eca3d1", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman admiring plaza", + "scene_tags": [ + "seville", + "plaza", + "tower", + "sun", + "travel", + "architecture", + "blue_sky", + "curly_hair" + ], + "suggested_caption": "Standing in awe of Seville\u2019s beauty. \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong composition with the tower as focal point, but subject placement slightly off-center reduces impact.", + "latency_ms": 6069, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ae8d0f5-432d-40ae-bf24-36aebb8977d0", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "cathedral", + "street", + "sunset", + "travel", + "architecture", + "curly_hair", + "blue_dress" + ], + "suggested_caption": "Blue dress, warm sun, and history in Seville.", + "reasoning": "Strong vertical framing with vibrant blue dress against historic architecture, but slightly busy background reduces impact.", + "latency_ms": 6113, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f32a27d2-44e9-41fe-acd3-b581825b7d37", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man posing by ornate bridge", + "scene_tags": [ + "seville", + "plaza", + "bridge", + "sunset", + "architecture", + "travel", + "sunset", + "water" + ], + "suggested_caption": "Sunset at the Plaza de Espa\u00f1a, Seville.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, but slightly cluttered foreground and less dynamic framing for IG scroll.", + "latency_ms": 5853, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "309a7b9a-ab2a-44b6-98d7-a026f3f9ad2d", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman overlooking garden", + "scene_tags": [ + "garden", + "palm trees", + "archway", + "sunset", + "travel", + "greenery", + "andalusian", + "balcony" + ], + "suggested_caption": "Gazing through the arch at paradise.", + "reasoning": "Strong composition with arch framing, vibrant colors, and clear subject, but slightly cluttered background reduces IG impact.", + "latency_ms": 5915, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "af98052f-97e1-4c37-a719-55fa6f6a5bab", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "palm tree", + "flowers", + "arches", + "garden", + "blue dress", + "sunset", + "travel" + ], + "suggested_caption": "Lost in the beauty of Andalusia \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong composition with vibrant colors and cultural architecture, but slightly cluttered foreground reduces Instagram scroll-stopping power.", + "latency_ms": 5795, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "556dce46-6930-40fa-b1ee-6b58f1967aa0", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "historic staircase", + "mosaic floor", + "tiled walls", + "indoor", + "elegant", + "travel", + "architecture", + "portrait" + ], + "suggested_caption": "Blue dress, ancient tiles, and timeless beauty. \ud83c\udf3f", + "reasoning": "Strong composition with vibrant blue dress against ornate backdrop, but slightly cluttered foreground and less dynamic framing for IG scroll.", + "latency_ms": 5975, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f6fe897c-9c59-4d5c-8026-f532b131e800", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "seville", + "courtyard", + "palm trees", + "garden", + "arches", + "blue dress", + "sunlight", + "travel" + ], + "suggested_caption": "Lost in the magic of Seville\u2019s Alc\u00e1zar gardens.", + "reasoning": "Strong composition with vibrant colors and cultural architecture, but slightly cluttered foreground reduces Instagram scroll-stopping power.", + "latency_ms": 5905, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d4a4990f-30ac-4378-a20c-0b1083883665", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "tower", + "sunset", + "sunglasses", + "travel", + "architecture", + "bridge" + ], + "suggested_caption": "Sunset at the Plaza de Espa\u00f1a, Seville.", + "reasoning": "Strong vertical framing with clear subject and iconic architecture, bright colors and clean lines make it scroll-stopping.", + "latency_ms": 5981, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e71b31a1-bd05-418b-bfe1-d7dcbc8398ed", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "blooming flowers on building", + "scene_tags": [ + "flowers", + "architecture", + "sunset", + "vibrant", + "spain", + "doorway", + "purple", + "bougainvillea" + ], + "suggested_caption": "Color explodes against white walls. \ud83c\udf38", + "reasoning": "Vibrant bougainvillea against white walls and blue sky creates strong visual appeal, though slightly cluttered framing reduces IG scroll-stopping power.", + "latency_ms": 6552, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 124 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "60d9defc-36a6-439d-81bb-3f446972bf14", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "coastal cityscape view", + "scene_tags": [ + "malaga", + "harbor", + "cloudy", + "palm trees", + "gardens", + "city", + "sea", + "walkers" + ], + "suggested_caption": "Cloudy day over Malaga\u2019s harbor \u2014 quiet and cinematic.", + "reasoning": "Strong composition with layered depth, but muted colors and overcast sky reduce visual punch for IG.", + "latency_ms": 6696, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ad632dce-c371-4166-8490-27e60fcea65a", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "staircase", + "arches", + "sunlight", + "travel", + "blue dress", + "historic", + "curly hair" + ], + "suggested_caption": "Staircase moments in Seville \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong composition with vibrant blue dress against warm architecture, but slightly busy background reduces IG impact.", + "latency_ms": 6397, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "38f26552-7b4e-4488-880d-a90e9f5453e7", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate european building", + "scene_tags": [ + "architecture", + "sunset", + "balcony", + "european", + "urban", + "detail", + "blue sky", + "ornate" + ], + "suggested_caption": "Sunset on a classic European street.", + "reasoning": "Strong architectural detail and warm lighting, but slightly tilted angle and cluttered foreground reduce IG impact.", + "latency_ms": 6060, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "efaf5b66-243e-462f-9d5e-ea90fee3c61b", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "historic town hall", + "scene_tags": [ + "spain", + "architecture", + "palm trees", + "blue sky", + "bicycle parking", + "classical", + "sunny", + "staircase" + ], + "suggested_caption": "Grandeur in the sun. \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong architectural subject with vibrant sky, but foreground sign slightly distracts from vertical composition.", + "latency_ms": 6081, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "067e963d-265e-48f5-a4b4-75da4402043c", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "garden", + "curly_hair", + "blue_dress", + "flowers", + "courtyard", + "sunlight", + "portrait", + "tropical" + ], + "suggested_caption": "Sunlight and blooms in the courtyard.", + "reasoning": "Strong subject with vibrant garden backdrop, but slightly cluttered framing and muted colors reduce overall impact.", + "latency_ms": 5900, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "460b6b6d-ec7c-424b-a899-23c5975039f6", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman at historic fortress", + "scene_tags": [ + "fortress", + "cityscape", + "mountains", + "curly_hair", + "blue_dress", + "green_trees", + "overcast", + "travel" + ], + "suggested_caption": "Chasing sunsets from ancient walls \ud83c\udf04", + "reasoning": "Strong vertical framing with subject in foreground, but muted lighting and busy background slightly reduce aesthetic impact.", + "latency_ms": 6368, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a56ee545-2409-49d7-a289-97f2d6f75ca6", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple selfie", + "scene_tags": [ + "couple", + "selfie", + "cityscape", + "overcast", + "travel", + "coast", + "curly_hair", + "portrait" + ], + "suggested_caption": "Two souls, one view. \ud83c\udf06", + "reasoning": "Strong vertical framing with engaging subjects, but muted lighting and busy background slightly reduce aesthetic impact.", + "latency_ms": 6465, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ed528a9-109c-4855-b3c3-ab81b404f856", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "andalusian", + "courtyard", + "sunlight", + "architecture", + "potted plants", + "stone arches", + "travel", + "elegant" + ], + "suggested_caption": "Sun-drenched elegance in the Andalusian courtyard.", + "reasoning": "Strong composition with warm tones and clear subject, but slightly busy background reduces IG scroll-stopping power.", + "latency_ms": 6227, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8ae46e84-d601-4345-bdb1-f6771f388dc1", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "flowers", + "building", + "portrait", + "blue dress", + "bougainvillea", + "outdoor", + "sunset", + "elegant" + ], + "suggested_caption": "Dressed for the blooms.", + "reasoning": "Vibrant floral backdrop with strong vertical framing and subject in flowing dress creates visual harmony and Instagram appeal.", + "latency_ms": 6003, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1fb7f2c9-2e63-47a0-b59e-ed32de4d6fd5", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza", + "sunset", + "sunset", + "sunset", + "sunset", + "sunset", + "sunset" + ], + "suggested_caption": "Sunset in Seville, Spain \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong vertical framing with clear subject and iconic architecture, bright colors and good contrast, but repeated 'sunset' tags are inaccurate for daytime photo.", + "latency_ms": 6579, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f9f81d70-0e5e-4e07-933c-6e97e3aa22c7", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "historic street architecture", + "scene_tags": [ + "seville", + "sunset", + "street", + "architecture", + "people", + "cafe", + "blue sky", + "tiled dome" + ], + "suggested_caption": "Sunset in Seville, where history meets the everyday.", + "reasoning": "Strong architectural subject with warm lighting, but slightly cluttered foreground reduces Instagram appeal.", + "latency_ms": 5650, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c63f60bb-2c2a-4ebc-b4e5-bd108a7e5e60", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "couple in elevator", + "scene_tags": [ + "elevator", + "couple", + "selfie", + "metallic", + "urban", + "smile", + "mirror", + "casual" + ], + "suggested_caption": "Caught in the elevator, just us and the city\u2019s heartbeat.", + "reasoning": "Strong subject and vertical framing, but harsh lighting and cluttered background reduce visual appeal.", + "latency_ms": 5928, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 110 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9fc4be6d-dc08-457c-9f5a-1606fb96dd97", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple kissing on cliff", + "scene_tags": [ + "barcelona", + "selfie", + "cityscape", + "greenery", + "cloudy", + "romantic", + "travel", + "coastal" + ], + "suggested_caption": "Kiss from the top of the world \ud83c\udf04", + "reasoning": "Strong romantic moment with scenic backdrop, but overcast light and selfie framing slightly reduce visual polish.", + "latency_ms": 6201, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1d7f8084-eb2e-41e4-a380-d0983f6807e2", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man admiring view", + "scene_tags": [ + "barcelona", + "sky", + "cityscape", + "overcast", + "hill", + "travel", + "reflection", + "urban" + ], + "suggested_caption": "Feeling grateful for the view from the top.", + "reasoning": "Strong vertical framing with subject looking up, but overcast sky reduces color vibrancy and visual punch.", + "latency_ms": 6274, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f638054c-bad8-4ac1-83c2-1b0f5149fc76", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "flamenco performers on stage", + "scene_tags": [ + "seville", + "flamenco", + "stage", + "guitar", + "theater", + "performance", + "purple_light", + "audience" + ], + "suggested_caption": "Flamenco magic in Seville. The soul of Spain on stage.", + "reasoning": "Strong stage lighting and composition, but foreground arms slightly distract from the performers' emotional moment.", + "latency_ms": 6703, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8f36cc4f-67ee-419a-806a-393e02311846", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "courtyard", + "arches", + "sunlight", + "travel", + "fashion", + "palm trees", + "stone" + ], + "suggested_caption": "Sun-drenched in Seville\u2019s historic courtyard.", + "reasoning": "Strong composition with vibrant colors and clear subject, but slightly busy background reduces IG impact.", + "latency_ms": 6106, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ef806816-857f-4829-9c43-fc7439005b24", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman with curly hair", + "scene_tags": [ + "seville", + "selfie", + "cityscape", + "curly_hair", + "blue_dress", + "overcast", + "ancient_ruins", + "greenery" + ], + "suggested_caption": "City views from my favorite spot.", + "reasoning": "Strong vertical framing with a compelling subject, vibrant blue dress against muted cityscape, and immediate visual hook.", + "latency_ms": 6442, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "98c6d4f2-f935-4358-8db7-5de1da16c638", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza", + "bridge", + "blue dress", + "architecture", + "sunset", + "water", + "travel" + ], + "suggested_caption": "Blue dress, blue sky, blue tiles. Seville\u2019s magic.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, but slightly busy background reduces IG impact.", + "latency_ms": 6046, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "bacbae88-b736-43a5-972d-602d08a3d3a9", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "ivy-covered building", + "scene_tags": [ + "seville", + "ivy", + "architecture", + "blue sky", + "balcony", + "greenery", + "sunset", + "european" + ], + "suggested_caption": "Nature reclaiming the city, one vine at a time.", + "reasoning": "Strong vertical composition with vibrant green against blue sky, but slightly cluttered foreground reduces scroll-stopping impact.", + "latency_ms": 6010, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9499f983-a5ee-416b-ae9c-9df25d846dfc", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "andalusia", + "stone walls", + "palm tree", + "historical site", + "blue dress", + "travel", + "outdoor", + "elegant" + ], + "suggested_caption": "Blue against the ancient stones. \ud83c\uddea\ud83c\uddf8", + "reasoning": "Strong composition with vibrant blue dress contrasting rustic textures, but slightly muted lighting reduces visual pop.", + "latency_ms": 6428, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2f9f26c4-61ba-4578-a3a2-ca62f62556ad", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "palm trees between buildings", + "scene_tags": [ + "palm trees", + "street", + "buildings", + "cloudy", + "spain", + "urban", + "architecture", + "streetview" + ], + "suggested_caption": "Palm trees framing the city streets of Spain.", + "reasoning": "Strong vertical composition with palm trees as focal point, but muted lighting and cluttered foreground reduce visual impact.", + "latency_ms": 6264, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "deb79a4f-f939-4bd4-bf84-16a39ed5f134", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in ornate room", + "scene_tags": [ + "marrakech", + "mosque", + "tiles", + "curly hair", + "blue dress", + "elegant", + "travel", + "architecture" + ], + "suggested_caption": "Lost in the patterns of Marrakech.", + "reasoning": "Strong subject and rich textures score high aesthetically, but framing and lighting lack Instagram\u2019s punchy visual hook.", + "latency_ms": 6390, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a218cf81-a745-4860-b036-dea56c4941f8", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "marrakech", + "mosque", + "tiles", + "columns", + "fashion", + "travel", + "elegant", + "pattern" + ], + "suggested_caption": "Blue dress, Moroccan tiles, and quiet moments.", + "reasoning": "Strong composition with rich textures and warm tones, but slightly off-center framing reduces IG scroll-stopping power.", + "latency_ms": 6243, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90a58b73-2d88-48a4-9fe4-ce70286f84d4", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate library interior", + "scene_tags": [ + "bookshelf", + "fireplace", + "antique", + "elegant", + "books", + "ornaments", + "pink walls", + "classical" + ], + "suggested_caption": "A library steeped in history and quiet grandeur.", + "reasoning": "Rich textures and warm lighting create depth, but the composition feels slightly cluttered for Instagram\u2019s vertical scroll.", + "latency_ms": 6286, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fcaab99b-7200-4f81-8642-eba0c3bc21ec", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "stone tower with flowers", + "scene_tags": [ + "tower", + "flowers", + "greenery", + "stone", + "overcast", + "medieval", + "garden", + "architecture" + ], + "suggested_caption": "Ancient stone meets blooming beauty.", + "reasoning": "Strong vertical composition with natural framing, but muted lighting and lack of vibrant color reduce visual impact.", + "latency_ms": 6162, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 105 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "be8661b0-f545-4afa-94c5-e10da5148041", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "ancient wall", + "palm tree", + "cobblestone", + "blue dress", + "medieval", + "travel", + "portrait", + "outdoor" + ], + "suggested_caption": "Standing in history, dressed in color.", + "reasoning": "Strong subject and color contrast against rustic textures, but slightly cluttered framing reduces scroll-stopping impact.", + "latency_ms": 6226, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f210a4ea-8cf9-47a3-a2b2-dcd99e26e4b7", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "man in blue shirt", + "scene_tags": [ + "yellow wall", + "green plants", + "outdoor", + "tropical", + "sunglasses", + "table", + "portrait", + "casual" + ], + "suggested_caption": "Chillin' in the greenery with a side of sunshine.", + "reasoning": "Strong color contrast and natural lighting make it visually appealing, but the framing feels slightly off-center and lacks a strong Instagram hook.", + "latency_ms": 6393, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4cad0011-b3c6-4612-a398-9a5cae315282", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "night market street", + "scene_tags": [ + "night", + "street", + "crowd", + "lights", + "european", + "festive", + "outdoor", + "urban" + ], + "suggested_caption": "Lost in the vibrant night market vibes \ud83c\udf06", + "reasoning": "Strong vertical composition with dynamic lighting and crowd, but slightly cluttered foreground reduces visual impact.", + "latency_ms": 5978, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "991386b2-325b-496b-a61c-76dadc8ca281", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man at historic balcony", + "scene_tags": [ + "spain", + "balcony", + "architecture", + "palm", + "blue_shirt", + "sunset", + "travel", + "garden" + ], + "suggested_caption": "Standing in the shadow of history, with Spain\u2019s beauty behind me.", + "reasoning": "Strong vertical framing with arch framing, vibrant colors, and clear subject \u2014 perfect for IG scroll stops.", + "latency_ms": 6296, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c98270c9-ee08-4e17-9f2a-2b004a1fa25f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tiled_wall", + "blue_dress", + "curly_hair", + "travel", + "elegant", + "pattern", + "portrait", + "sunset" + ], + "suggested_caption": "Blue dreams against blue tiles.", + "reasoning": "Strong composition with vibrant, harmonious colors and a captivating subject, but slightly less vertical framing for IG scroll stops.", + "latency_ms": 5823, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0aa895b8-729e-46ed-a5cf-eea974827553", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "street architecture", + "scene_tags": [ + "spain", + "buenos_aires", + "blue_sky", + "urban", + "balkons", + "people", + "street", + "colorful" + ], + "suggested_caption": "Colorful facades and bustling streets in the heart of the city.", + "reasoning": "Strong vertical composition with vibrant architecture and sky, but busy street distracts from aesthetic focus.", + "latency_ms": 6380, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 114 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "34258ec5-302b-4df3-a3a5-3ddfa7f4cd56", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 9, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tiled doorway", + "andalusian", + "blue dress", + "travel", + "art", + "doorway", + "elegant", + "mood" + ], + "suggested_caption": "Standing in a doorway of art and history.", + "reasoning": "Vibrant, symmetrical composition with strong color contrast and a striking subject framed by ornate tiles, highly Instagram-worthy.", + "latency_ms": 6127, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d84ea8b1-0ffd-4680-b0dd-fbf2e21c05ea", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple selfie at overlook", + "scene_tags": [ + "seville", + "selfie", + "cityscape", + "cloudy", + "crane", + "trees", + "stone_wall", + "smile" + ], + "suggested_caption": "Our favorite spot in Seville with a view that never gets old.", + "reasoning": "Strong vertical framing with engaging subjects, but slightly cluttered background and muted lighting reduce overall aesthetic appeal.", + "latency_ms": 6282, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "09b42418-79f1-421e-9b7a-906df03ea290", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "cityscape", + "brick wall", + "curly hair", + "blue dress", + "overcast", + "urban", + "greenery", + "elegant" + ], + "suggested_caption": "Blue against the city sky.", + "reasoning": "Strong vertical framing with a confident subject, but muted lighting and slightly cluttered background reduce visual impact.", + "latency_ms": 5868, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b80c556-78b3-473c-aaa9-e771e034918f", + "model": "qwen3vl-8b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman on balcony", + "scene_tags": [ + "spain", + "palace", + "palm trees", + "garden", + "arch", + "sun", + "travel", + "elegant" + ], + "suggested_caption": "Sunset views from the Alhambra balcony.", + "reasoning": "Strong vertical framing with vibrant colors and detailed architecture, but slightly busy background reduces impact.", + "latency_ms": 6201, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9f9c9faf-6bce-4ceb-a1b5-e63179c55990", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man in garden", + "scene_tags": [ + "palace", + "palm trees", + "purple flowers", + "balcony", + "sunlight", + "travel", + "architecture", + "greenery" + ], + "suggested_caption": "sunshine & secrets", + "reasoning": "Strong vertical composition, vibrant colors, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 35883, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fec017d6-54cc-4ee9-89f8-6dd0109e6603", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly-haired woman", + "scene_tags": [ + "denim", + "plants", + "interior", + "sunlight", + "curly hair", + "casual", + "bright", + "indoor" + ], + "suggested_caption": "sunlight + curls = happy.", + "reasoning": "Strong subject and vertical framing, good light and color, but composition is slightly casual.", + "latency_ms": 5620, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 98 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0d53158b-5a17-4ecc-8e92-0195fb6b12a7", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "architectural marvel", + "scene_tags": [ + "seville", + "metropol Parasol", + "sky", + "clouds", + "old town", + "modern", + "sunlight", + "travel" + ], + "suggested_caption": "Seville's skyward statement.", + "reasoning": "Strong vertical composition, vibrant sky, and unique structure make for a visually striking and shareable travel image.", + "latency_ms": 5877, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 102 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "23137c1c-f475-4b9a-8c72-e86c90f8f064", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "smiling man at caf\u00e9", + "scene_tags": [ + "caf\u00e9", + "latte", + "interior", + "relaxed", + "modern", + "light", + "people", + "morning" + ], + "suggested_caption": "a morning pick-me-up and a smile.", + "reasoning": "Good composition and lighting, but the subject's expression is a bit forced, and the vertical framing is slightly off-center.", + "latency_ms": 5998, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3376ee87-8bf5-4710-88e0-f0d89b3587fc", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in azulejo", + "scene_tags": [ + "azulejo", + "portugal", + "blue", + "curly_hair", + "gold_jewelry", + "travel", + "pattern", + "vibrant" + ], + "suggested_caption": "azulejo dreams.", + "reasoning": "Strong subject and vibrant pattern, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5736, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4f2fea45-3119-4df2-bca8-d35a85fd1aa9", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "palace courtyard", + "scene_tags": [ + "seville", + "palm trees", + "yellow walls", + "arches", + "gardens", + "sunlight", + "history", + "tranquility" + ], + "suggested_caption": "hidden gems of seville", + "reasoning": "strong vertical composition, vibrant colors, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 5848, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e49bb267-b7da-4528-a37e-aa92770a0271", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "cityscape", + "cloudy", + "stone wall", + "urban", + "travel", + "casual", + "outdoor", + "summer" + ], + "suggested_caption": "blue against the sky", + "reasoning": "Strong subject and vertical framing, but lighting is flat and background is slightly cluttered.", + "latency_ms": 5583, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2df74bf4-e363-4c43-a0ea-8077666229f6", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "garden", + "roses", + "fountain", + "brick pillars", + "tourists", + "greenery", + "sunlight", + "travel" + ], + "suggested_caption": "blue dress in a sunlit garden", + "reasoning": "Good composition and natural light, but the right side is slightly out of focus, reducing overall aesthetic and IG fit.", + "latency_ms": 5925, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 105 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "750f9142-5e86-468a-89db-e9f28d6aac05", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman by pool", + "scene_tags": [ + "pool", + "brick wall", + "ducks", + "sunlight", + "blue dress", + "moroccan", + "relaxation", + "travel" + ], + "suggested_caption": "sunlit poolside moments.", + "reasoning": "Strong subject and vertical composition, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5235, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ea713729-51b7-4005-834b-bd3064455928", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "architectural exploration", + "scene_tags": [ + "seville", + "plaza", + "buildings", + "evening", + "cafe", + "street", + "travel", + "culture" + ], + "suggested_caption": "Seville's hidden gems.", + "reasoning": "Strong vertical composition, vibrant colors, and a clear subject make this visually appealing and Instagram-friendly.", + "latency_ms": 5279, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b0734c0-589c-451a-a41f-8fc45d757ccc", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tiled archway", + "historical building", + "blue dress", + "indoor", + "ornate", + "travel", + "culture", + "elegant" + ], + "suggested_caption": "blue and gold", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5310, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e6ebf7b0-7f4d-40b3-919e-3986cbd595fe", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "plaza de Espa\u00f1a", + "scene_tags": [ + "seville", + "spanish", + "baroque", + "flag", + "carriage", + "plaza", + "sunshine", + "travel" + ], + "suggested_caption": "plaza de espa\u00f1a, seville", + "reasoning": "strong composition and vibrant colors, but the horse-drawn carriage adds a unique focal point, making it visually engaging for social media.", + "latency_ms": 5807, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ee9d68c9-e20c-45a9-8d69-7fe7cecda6c6", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "architectural marvel", + "scene_tags": [ + "seville", + "santiago calatrava", + "sky", + "urban", + "modern", + "sunlight", + "clouds", + "cafe" + ], + "suggested_caption": "where art meets architecture.", + "reasoning": "Strong vertical composition, vibrant light, and unique subject make it visually striking and Instagram-worthy.", + "latency_ms": 5403, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 98 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "eade13f0-608b-4e5b-a11c-610287226371", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "seville", + "balcony", + "blue dress", + "sunshine", + "architecture", + "river", + "travel" + ], + "suggested_caption": "plaza de Espa\u00f1a vibes \u2728", + "reasoning": "Strong subject, vibrant colors, and iconic architecture make for a visually engaging and Instagram-worthy travel shot.", + "latency_ms": 5587, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "75cfcb0b-fd7b-4c03-a399-f6253e3db200", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "bullring with city view", + "scene_tags": [ + "malaga", + "bullring", + "coast", + "buildings", + "trees", + "path", + "flowers", + "overcast" + ], + "suggested_caption": "Malaga's bullring from above.", + "reasoning": "Strong composition with clear subject, good vertical framing, and a compelling coastal cityscape, but slightly muted light.", + "latency_ms": 6208, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 103 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6d2920c8-08d1-4bef-9abc-bc91c446570f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "fountain", + "arches", + "sunlight", + "plants", + "travel", + "summer", + "casual" + ], + "suggested_caption": "blue and sunshine.", + "reasoning": "Strong subject and vertical framing, but background details slightly distract from the main focus.", + "latency_ms": 5385, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c2769ba4-f2c0-49c1-a4b8-6ce0e083f8ec", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate town hall", + "scene_tags": [ + "beige\u5efa\u7b51", + "\u67f1\u5eca", + "\u897f\u73ed\u7259\u56fd\u65d7", + "\u5929\u7a7a", + "\u505c\u8f66\u573a\u6807\u5fd7", + "\u5386\u53f2\u5efa\u7b51", + "\u57ce\u5e02", + "\u6674\u5929" + ], + "suggested_caption": "Granada's grandeur.", + "reasoning": "Strong vertical composition and clear subject, but some elements (parking sign) distract from pure aesthetics.", + "latency_ms": 6097, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "17905afc-be7f-4664-948a-1c9da287b703", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "columns", + "palm", + "flowers", + "moroccan", + "travel", + "sunlight", + "greenery" + ], + "suggested_caption": "hidden gems.", + "reasoning": "Strong subject and vibrant setting, but slightly cluttered foreground and vertical framing could be tighter.", + "latency_ms": 5570, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "40f82498-2e9c-4b98-8ebb-ed9821c179a0", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "urban architecture", + "scene_tags": [ + "seville", + "buildings", + "sky", + "street", + "bicycles", + "modern", + "historical", + "sunlight" + ], + "suggested_caption": "Seville's blend of old and new.", + "reasoning": "Good composition and light, but lacks a strong visual hook for IG scrolling.", + "latency_ms": 5607, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5a879592-e59b-4a79-8886-dc089e301f5a", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in plaza", + "scene_tags": [ + "seville", + "plaza", + "baroque", + "sunshine", + "travel", + "architecture", + "dress", + "summer" + ], + "suggested_caption": "plaza de Espa\u00f1a vibes", + "reasoning": "Strong vertical composition, vibrant colors, and a clear subject make this visually appealing and Instagram-scroll-stopping.", + "latency_ms": 5774, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "885508f8-f39b-484e-8959-929111ab0b0d", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "river", + "architecture", + "blue dress", + "seville", + "sunshine", + "tourist", + "historic" + ], + "suggested_caption": "plaza de Espa\u00f1a, seville", + "reasoning": "Strong composition and vibrant colors, but the subject placement slightly distracts from the iconic architecture.", + "latency_ms": 5672, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90b60141-efa8-4a34-8e0a-dda28f296f58", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "bougainvillea", + "bench", + "yellow walls", + "travel", + "joy", + "sunlight", + "colorful" + ], + "suggested_caption": "a splash of blue in the courtyard", + "reasoning": "Strong subject and vibrant colors, but composition is slightly casual, missing a tighter frame.", + "latency_ms": 5750, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "876f997c-e9be-40e2-8482-ea0216b82d5a", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "man in blue shirt", + "scene_tags": [ + "bench", + "greenery", + "outdoor", + "relaxed", + "sunny" + ], + "suggested_caption": "a moment of calm.", + "reasoning": "Good lighting and subject clarity, but composition is slightly centered and lacks dynamic visual hook.", + "latency_ms": 5131, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 84 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d37f1824-5d53-41ca-8c79-ffae48978442", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "garden", + "column", + "fountain", + "flowers", + "blue dress", + "yellow walls", + "arabesque", + "sunlight" + ], + "suggested_caption": "enchanted in the courtyard", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5650, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7a13fad2-6ef9-44ea-944c-551e854b6acd", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman on city wall", + "scene_tags": [ + "cityscape", + "blue dress", + "overcast", + "travel", + "historical site", + "smile", + "outdoor", + "urban" + ], + "suggested_caption": "city views from above.", + "reasoning": "Strong subject and vertical framing, but overcast light and busy city background slightly reduce aesthetic and visual impact.", + "latency_ms": 6036, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "64ebfd3c-e9a1-49c2-9d0c-a81182ef6af2", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tile wall", + "blue", + "yellow", + "pattern", + "Portugal", + "Lisbon", + "fashion", + "travel" + ], + "suggested_caption": "Lisbon's tiled charm.", + "reasoning": "Strong subject and vibrant pattern, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5668, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d211ea44-4f3c-44d0-b7f0-cb9d001f07af", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "balcony", + "sunlight", + "greenery", + "travel", + "summer", + "dress", + "outdoor", + "vines" + ], + "suggested_caption": "twirling in the sun", + "reasoning": "Strong subject and vertical framing, but lighting and composition could be more dynamic for portfolio quality.", + "latency_ms": 5819, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "30015fce-9f43-430e-8268-36cb4cb56596", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "curly-haired woman in blue dress", + "scene_tags": [ + "outdoor", + "park", + "sunlight", + "greenery", + "elegant", + "summer", + "portrait", + "joy" + ], + "suggested_caption": "sunlit curls and a blue dress.", + "reasoning": "Strong subject and vertical framing, but background is slightly busy, reducing overall aesthetic and IG fit.", + "latency_ms": 5941, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 101 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9842a059-f560-4243-9710-84a9ef7ee71e", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man at plaza", + "scene_tags": [ + "plaza", + "architecture", + "canals", + "sightseeing", + "seville", + "sunshine", + "travel", + "summer" + ], + "suggested_caption": "plaza de Espa\u00f1a vibes", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5549, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f26f6930-16a5-400f-ab7c-ff19abde8ef3", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "river", + "architecture", + "sunshine", + "travel", + "Spain", + "balcony", + "joy" + ], + "suggested_caption": "Seville's beauty in blue.", + "reasoning": "Strong vertical composition, vibrant colors, and a clear subject make this visually appealing and Instagram-friendly.", + "latency_ms": 5649, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d99cf40e-3bed-47f7-8595-c11286ab3a6a", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "seville", + "baroque", + "sun", + "travel", + "architecture", + "summer", + "tourist" + ], + "suggested_caption": "plaza de Espa\u00f1a vibes", + "reasoning": "Strong vertical composition, vibrant colors, and a clear subject make this visually appealing and Instagram-friendly.", + "latency_ms": 5630, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2e199e4b-3fb5-442f-895e-3ab7ba9ccdaf", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate arabesque ceiling", + "scene_tags": "mosque, arabic script, gold, blue, red, pattern, architecture, intricate", + "suggested_caption": "where history meets artistry.", + "reasoning": "Strong composition and vibrant colors, but slightly busy for quick IG scroll.", + "latency_ms": 5173, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 85 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "13dc549f-8aee-41b4-92b8-954b5bc41fb2", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "stage performance", + "scene_tags": [ + "theater", + "performers", + "flamenco", + "audience", + "lights", + "stage", + "music", + "celebration" + ], + "suggested_caption": "a night at the theater", + "reasoning": "good lighting and vertical framing, but slightly cluttered with performers and audience hands.", + "latency_ms": 5841, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "144aadc1-0e9c-46c7-80dd-8decd901e4ee", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "curly-haired woman with backpack", + "scene_tags": [ + "bougainvillea", + "greenery", + "travel", + "smile", + "casual", + "outdoor", + "colorful" + ], + "suggested_caption": "bougainvillea dreams", + "reasoning": "Strong subject, vibrant colors, and vertical framing make it visually appealing and Instagram-friendly.", + "latency_ms": 5920, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3c464a1a-ab0e-4703-9bcd-37ccb17f8dac", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "street style", + "scene_tags": [ + "city", + "sidewalk", + "trees", + "sunlight", + "backpack", + "denim", + "casual", + "urban" + ], + "suggested_caption": "city vibes.", + "reasoning": "Strong subject and vertical framing, but background is slightly busy, reducing overall aesthetic and IG fit.", + "latency_ms": 5599, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 93 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0f816473-c0f5-44eb-96a1-9c982f7f0343", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "palm trees", + "historic garden", + "sunlight", + "travel", + "summer", + "outdoor", + "elegant", + "tourist" + ], + "suggested_caption": "blue dress, blue sky, blue vibes.", + "reasoning": "Strong subject and vibrant colors, but composition is slightly tilted and lacks dynamic framing for IG scroll-stopping impact.", + "latency_ms": 6054, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 105 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "891f2361-7c42-4163-b37a-c11f7f7126f2", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman on stairs", + "scene_tags": [ + "seville", + "architectural", + "sunlight", + "travel", + "blue dress", + "historic", + "tourist", + "summer" + ], + "suggested_caption": "sunlit seville", + "reasoning": "Strong subject, vertical framing, and vibrant colors make it visually appealing and Instagram-friendly.", + "latency_ms": 5551, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "711f44c3-d2f8-4878-b587-4863e268dd3c", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "bar interior", + "scene_tags": [ + "bar", + "wooden", + "chairs", + "lighting", + "warm", + "cozy", + "restaurant", + "interior" + ], + "suggested_caption": "cozy bar vibes", + "reasoning": "Strong composition and warm lighting, but less vertical emphasis for IG scroll appeal.", + "latency_ms": 5394, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 90 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "baf172dd-da76-4c3c-80b6-99382aeb4a72", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in dress", + "scene_tags": [ + "veranda", + "greenery", + "sunlight", + "dress", + "travel", + "summer", + "outdoor" + ], + "suggested_caption": "dancing in the light", + "reasoning": "Strong subject and vertical composition, but lighting and color balance could be more refined.", + "latency_ms": 5214, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 89 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6a918ad8-de2d-48e9-a378-c7038c9e5178", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress at museum", + "scene_tags": [ + "museum", + "portrait", + "dress", + "art", + "seville", + "historical", + "tourist", + "cultural" + ], + "suggested_caption": "blue dress, red painting, museum vibes.", + "reasoning": "Strong vertical composition, vibrant blue dress contrasts with warm tones, but portrait framing is slightly off-center.", + "latency_ms": 5737, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 103 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4b09475f-0eee-46ba-a558-1f0cd3ecd473", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in historic room", + "scene_tags": [ + "interior", + "antique", + "ornate", + "lighting", + "pattern", + "heritage", + "elegant", + "museum" + ], + "suggested_caption": "timeless elegance", + "reasoning": "Strong composition and rich details, but the subject's pose is slightly casual for maximum impact.", + "latency_ms": 5674, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ff2de652-1f1a-4d5c-a7ee-28166ce7118f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "horse-drawn carriage", + "scene_tags": [ + "street", + "horse", + "carriage", + "city", + "trees", + "sunlight", + "travel", + "urban" + ], + "suggested_caption": "A slice of old-world charm.", + "reasoning": "Strong subject and vertical framing, but less unique composition and lighting than portfolio-worthy.", + "latency_ms": 5659, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "852f15e1-94f2-49db-8205-218c85d0a43f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "plaza de Espa\u00f1a", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "spain", + "sunny", + "historic", + "travel", + "grand" + ], + "suggested_caption": "plaza de espa\u00f1a, seville", + "reasoning": "strong composition and vibrant colors, but lacks a dynamic focal point for IG scroll appeal.", + "latency_ms": 5644, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c869757b-8734-4a8d-8dfb-fad070bf7be1", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "historic castle entrance", + "scene_tags": [ + "castillo", + "spain", + "tourists", + "sunlight", + "architecture", + "travel", + "daylight", + "summer" + ], + "suggested_caption": "Stepping back in time at the castle gates.", + "reasoning": "Strong vertical composition, bright light, clear subject, but some tourists distract from pure aesthetics.", + "latency_ms": 5787, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3ba4c50c-6ec4-41b1-bb2b-e9d4eea108f6", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "lit church facade", + "scene_tags": [ + "night", + "baroque", + "statue", + "trees", + "red building", + "light", + "square", + "history" + ], + "suggested_caption": "night glow on baroque beauty", + "reasoning": "Strong vertical composition, warm lighting, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 5629, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e5fc776e-8467-4805-8c35-77383f62d76d", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "balcony", + "sunlight", + "petals", + "yellow wall", + "pink hair", + "summer", + "travel", + "relaxation" + ], + "suggested_caption": "sunlit daydreams.", + "reasoning": "Strong subject and vertical framing, but lighting and color balance could be more refined for portfolio quality.", + "latency_ms": 5746, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b1a8b354-f9a4-42db-9639-44fd9ecbdd33", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "plaza de Espa\u00f1a", + "scene_tags": "seville, spain, plaza, architecture, people, blue sky, historic, grand", + "suggested_caption": "plaza de espa\u00f1a: where history meets the everyday.", + "reasoning": "Strong composition and vibrant light, but less vertical emphasis for IG scroll appeal.", + "latency_ms": 5444, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 90 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ba6f613c-61cd-42a7-aaa9-374d4f0ac058", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man at plaza", + "scene_tags": [ + "plaza", + "river", + "architecture", + "sightseeing", + "seville", + "sunshine", + "travel", + "blue" + ], + "suggested_caption": "plaza de Espa\u00f1a vibes", + "reasoning": "Strong subject and vertical framing, but slightly cluttered foreground and midground.", + "latency_ms": 5556, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "82391768-2834-4843-9cdd-249bd6780f91", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "fried dumplings", + "scene_tags": [ + "food", + "table", + "restaurant", + "sauce", + "cabbage", + "casual", + "indoor", + "delicious" + ], + "suggested_caption": "crispy dumplings, ready to dive in!", + "reasoning": "Strong subject with good vertical framing, but background is slightly distracting, reducing overall aesthetic.", + "latency_ms": 5811, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "01292c70-996c-43cd-b132-5933d3afbf43", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "bougainvillea", + "bench", + "travel", + "floral", + "arch", + "sunlight", + "relaxation" + ], + "suggested_caption": "a moment of quiet in the garden.", + "reasoning": "Strong vertical composition, vibrant colors, and a clear subject make this visually appealing and engaging for social media.", + "latency_ms": 5861, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 104 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "edb6e566-0350-4861-ae89-e32ddcfd4f14", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "fountain", + "plants", + "historical\u5efa\u7b51", + "sunny", + "travel", + "summer", + "relaxation" + ], + "suggested_caption": "sunlit moments in the garden.", + "reasoning": "Strong subject and vertical composition, but background details slightly distract from the main focus.", + "latency_ms": 5637, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b9c7f1d5-7336-4860-a38a-c3e1d82ee655", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly-haired woman with phone", + "scene_tags": [ + "street", + "sunglasses", + "curly hair", + "casual", + "daylight", + "urban", + "traveler", + "smile" + ], + "suggested_caption": "city vibes.", + "reasoning": "Strong subject and vertical framing, but background is slightly distracting, reducing overall aesthetic and IG fit.", + "latency_ms": 5910, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "967b48e8-531a-4bef-a069-fab60690145f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "pizza kitchen", + "scene_tags": [ + "pizzeria", + "lemons", + "olive_oil", + "artisan", + "italian", + "food", + "warm", + "craftsmanship" + ], + "suggested_caption": "where pizza dreams come to life.", + "reasoning": "Strong composition and warm lighting, but less vertical emphasis for IG scroll appeal.", + "latency_ms": 5617, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b68453c0-f5a3-4e8e-b231-af92fa2ed182", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "indoor", + "tiled floor", + "table", + "ornate wall", + "sunlight", + "casual", + "travel", + "elegant" + ], + "suggested_caption": "blue and gold", + "reasoning": "Strong subject and vertical framing, good light and color, but less dynamic composition.", + "latency_ms": 5769, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7f9762e4-afb1-45f5-85f0-fd33e050ed23", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "man in blue shirt", + "scene_tags": [ + "bench", + "greenery", + "outdoor", + "relaxed", + "sunlight", + "plants", + "casual", + "summer" + ], + "suggested_caption": "greenery backdrop", + "reasoning": "Good lighting and subject clarity, but composition is slightly centered and lacks dynamic visual hook.", + "latency_ms": 5568, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 93 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c61412cb-4abc-4d1c-b541-f1d67adc3b03", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "ruins", + "palm", + "cobblestone", + "ancient", + "travel", + "outdoor", + "historical", + "serene" + ], + "suggested_caption": "blue in the ruins", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5842, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4719aae4-0783-447a-afc2-9f84c7eca3d1", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "tower", + "sun", + "travel", + "summer", + "architecture", + "bridge", + "tourist" + ], + "suggested_caption": "plaza moment", + "reasoning": "Strong vertical composition, vibrant light, clear subject, but slight tilt reduces perfection; IG-friendly with strong visual hook.", + "latency_ms": 5759, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ae8d0f5-432d-40ae-bf24-36aebb8977d0", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "palermo", + "cathedral", + "sunny", + "urban", + "travel", + "casual", + "summer", + "architecture" + ], + "suggested_caption": "blue dress, blue sky, blue vibes.", + "reasoning": "Strong subject and vertical framing, but background architecture slightly distracts from the main focus.", + "latency_ms": 5815, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 98 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f32a27d2-44e9-41fe-acd3-b581825b7d37", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man at plaza", + "scene_tags": [ + "plaza", + "architecture", + "balcony", + "boats", + "solaris", + "spain", + "sunshine", + "travel" + ], + "suggested_caption": "plaza moment", + "reasoning": "Strong composition and vibrant colors, but the subject's pose is casual and not the main focus.", + "latency_ms": 5562, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "309a7b9a-ab2a-44b6-98d7-a026f3f9ad2d", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "balcony", + "palm trees", + "hedge maze", + "fountain", + "sunlight", + "travel", + "greenery", + "architecture" + ], + "suggested_caption": "enchanted garden view", + "reasoning": "Strong composition with natural frame, vibrant colors, and clear subject, but less vertical emphasis.", + "latency_ms": 5656, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 98 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "af98052f-97e1-4c37-a719-55fa6f6a5bab", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "courtyard", + "palm", + "flowers", + "arabesque", + "moroccan", + "travel", + "sunlight", + "serenity" + ], + "suggested_caption": "hidden oasis", + "reasoning": "Strong subject and vibrant setting, but slightly cluttered background reduces portfolio quality.", + "latency_ms": 5416, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "556dce46-6930-40fa-b1ee-6b58f1967aa0", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "staircase", + "mosaic floor", + "historical interior", + "lighting", + "dress", + "travel", + "culture", + "indoor" + ], + "suggested_caption": "blue in blue", + "reasoning": "Strong subject and vertical framing, but lighting and composition leave room for portfolio-worthy polish.", + "latency_ms": 5623, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f6fe897c-9c59-4d5c-8026-f532b131e800", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in courtyard", + "scene_tags": [ + "courtyard", + "palm", + "columns", + "flowers", + "moroccan", + "travel", + "sunlight", + "greenery" + ], + "suggested_caption": "hidden gems.", + "reasoning": "Strong vertical composition and vibrant colors make it visually appealing and Instagram-friendly.", + "latency_ms": 5281, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 88 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d4a4990f-30ac-4378-a20c-0b1083883665", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "bridge", + "sightseeing", + "sunny", + "historical", + "travel", + "summer", + "architecture" + ], + "suggested_caption": "plaza moment", + "reasoning": "Strong subject and vertical framing, but background architecture slightly overpowers the person.", + "latency_ms": 5440, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 90 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e71b31a1-bd05-418b-bfe1-d7dcbc8398ed", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "bougainvillea blooms", + "scene_tags": [ + "bougainvillea", + "white_walls", + "purple_flowers", + "sunshine", + "door", + "sky", + "travel", + "color" + ], + "suggested_caption": "a burst of color against the sky", + "reasoning": "vibrant colors and strong vertical composition make it visually striking and Instagram-friendly.", + "latency_ms": 5826, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 102 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "60d9defc-36a6-439d-81bb-3f446972bf14", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "cityscape from hill", + "scene_tags": [ + "valencia", + "skyline", + "harbor", + "palm_trees", + "cloudy", + "urban", + "hill", + "tourists" + ], + "suggested_caption": "valencia from above", + "reasoning": "good composition and detail, but overcast light reduces vibrancy; vertical framing works, but lacks immediate visual hook.", + "latency_ms": 5899, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 102 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ad632dce-c371-4166-8490-27e60fcea65a", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "stairs", + "orange building", + "sunlight", + "historical", + "tourists", + "balcony", + "plants", + "arch" + ], + "suggested_caption": "sunlit steps, blue dress", + "reasoning": "Strong subject and vertical composition, but background details slightly distract from the main focus.", + "latency_ms": 5469, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "38f26552-7b4e-4488-880d-a90e9f5453e7", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate building facade", + "scene_tags": [ + "barcelona", + "architecture", + "balconies", + "windows", + "sunlight", + "urban", + "historic", + "elegant" + ], + "suggested_caption": "Barcelona's grandeur up close.", + "reasoning": "Strong vertical composition and clear details, but less immediate visual hook than simpler scenes.", + "latency_ms": 5538, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "efaf5b66-243e-462f-9d5e-ea90fee3c61b", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate town hall", + "scene_tags": [ + "beige\u5efa\u7b51", + "\u67f1\u5eca", + "\u897f\u73ed\u7259\u56fd\u65d7", + "\u6674\u5929", + "\u68d5\u6988\u6811", + "\u505c\u8f66\u6807\u5fd7", + "\u5386\u53f2\u5efa\u7b51", + "\u57ce\u5e02" + ], + "suggested_caption": "a glimpse of history in the heart of the city.", + "reasoning": "Strong vertical composition and clear subject, but slight clutter from parking sign reduces IG fit.", + "latency_ms": 5768, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 104 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "067e963d-265e-48f5-a4b4-75da4402043c", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "curly-haired woman in garden", + "scene_tags": [ + "garden", + "flowers", + "sunlight", + "yellow\u5efa\u7b51", + "greenery", + "summer", + "casual", + "outdoor" + ], + "suggested_caption": "sunlit curls and garden vibes", + "reasoning": "Strong subject and vibrant garden, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5439, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 98 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "460b6b6d-ec7c-424b-a899-23c5975039f6", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "smiling woman at fortress", + "scene_tags": [ + "fortress", + "cityscape", + "mountains", + "trees", + "historical", + "travel", + "overcast", + "view" + ], + "suggested_caption": "Views from the top.", + "reasoning": "Strong subject and vertical framing, but slightly cluttered foreground and overcast light.", + "latency_ms": 5367, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a56ee545-2409-49d7-a289-97f2d6f75ca6", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 6, + "instagram_fit_score": 7, + "primary_subject": "couple selfie", + "scene_tags": [ + "skyline", + "cloudy", + "coast", + "urban", + "travel", + "friends", + "outdoor", + "daylight" + ], + "suggested_caption": "together in the city", + "reasoning": "Good vertical framing and clear subjects, but composition is casual and background is slightly distracting.", + "latency_ms": 5312, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ed528a9-109c-4855-b3c3-ab81b404f856", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "fountain", + "sunlight", + "travel", + "summer", + "architecture", + "plants", + "outdoor" + ], + "suggested_caption": "sunlit stroll in the courtyard.", + "reasoning": "Strong subject and vertical framing, but lighting and composition could be more dynamic for portfolio quality.", + "latency_ms": 5297, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8ae46e84-d601-4345-bdb1-f6771f388dc1", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "bougainvillea", + "house", + "balcony", + "portrait", + "travel", + "colorful", + "summer", + "vibrant" + ], + "suggested_caption": "a splash of blue against nature's canvas", + "reasoning": "Strong vertical composition, vibrant colors, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 5692, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 103 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1fb7f2c9-2e63-47a0-b59e-ed32de4d6fd5", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "solar", + "sightseeing", + "sunny", + "historical", + "travel", + "summer", + "architecture" + ], + "suggested_caption": "plaza moment", + "reasoning": "Strong subject, vibrant colors, and iconic architecture make for a visually engaging and well-composed travel shot.", + "latency_ms": 5504, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f9f81d70-0e5e-4e07-933c-6e97e3aa22c7", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "historic plaza building", + "scene_tags": [ + "seville", + "spain", + "architecture", + "plaza", + "sunset", + "cafe", + "people", + "vibrant" + ], + "suggested_caption": "Seville's charm in the evening light.", + "reasoning": "Strong vertical composition and vibrant colors make it visually appealing and Instagram-friendly.", + "latency_ms": 5440, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c63f60bb-2c2a-4ebc-b4e5-bd108a7e5e60", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 6, + "instagram_fit_score": 7, + "primary_subject": "elevator selfie", + "scene_tags": [ + "elevator", + "mirror", + "couple", + "urban", + "metallic", + "casual", + "reflection", + "modern" + ], + "suggested_caption": "elevator mirror magic", + "reasoning": "Good vertical composition and clear subjects, but lighting is flat and lacks unique visual interest.", + "latency_ms": 5251, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9fc4be6d-dc08-457c-9f5a-1606fb96dd97", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "lovers with city view", + "scene_tags": "seville, river, clouds, greenery, selfie, couple, urban, landscape", + "suggested_caption": "together in seville", + "reasoning": "Strong vertical composition, clear subjects, and a compelling cityscape backdrop, but slightly cluttered foreground.", + "latency_ms": 5270, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 89 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1d7f8084-eb2e-41e4-a380-d0983f6807e2", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man overlooking city", + "scene_tags": [ + "cityscape", + "mountain", + "cloudy", + "travel", + "view", + "urban", + "nature", + "tourist" + ], + "suggested_caption": "overlooking the city from above.", + "reasoning": "Strong subject and vertical framing, but overcast light and busy background slightly reduce aesthetic and IG fit.", + "latency_ms": 5507, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f638054c-bad8-4ac1-83c2-1b0f5149fc76", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "flamenco performers", + "scene_tags": [ + "stage", + "guitar", + "suits", + "red ruffles", + "purple light", + "theater", + "sevilla", + "audience" + ], + "suggested_caption": "a standing ovation in seville", + "reasoning": "strong vertical composition, vibrant stage lighting, and clear subject make it visually engaging and suitable for social media.", + "latency_ms": 5642, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 104 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8f36cc4f-67ee-419a-806a-393e02311846", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "courtyard", + "fence", + "palm", + "sunlight", + "greenery", + "historical\u5efa\u7b51", + "summer", + "relaxation" + ], + "suggested_caption": "blue and sunshine.", + "reasoning": "Strong subject and vertical framing, but background details slightly distract from the main focus.", + "latency_ms": 5325, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ef806816-857f-4829-9c43-fc7439005b24", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly-haired woman selfie", + "scene_tags": [ + "cityscape", + "mountain", + "sky", + "blue dress", + "cross necklace", + "travel", + "overcast" + ], + "suggested_caption": "top of the world", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio quality.", + "latency_ms": 5114, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 90 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "98c6d4f2-f935-4358-8db7-5de1da16c638", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman at plaza", + "scene_tags": [ + "plaza", + "river", + "architecture", + "blue dress", + "sunlight", + "spain", + "tourist", + "historic" + ], + "suggested_caption": "plaza moment", + "reasoning": "Strong vertical composition, vibrant colors, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 5195, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 93 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "bacbae88-b736-43a5-972d-602d08a3d3a9", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ivy-covered tower", + "scene_tags": [ + "tower", + "ivy", + "balcony", + "sky", + "sunlight", + "architecture", + "greenery", + "vertical" + ], + "suggested_caption": "ivy dreams", + "reasoning": "Strong vertical composition, vibrant colors, and clear subject make it visually appealing and Instagram-friendly.", + "latency_ms": 5112, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 91 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9499f983-a5ee-416b-ae9c-9df25d846dfc", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "explorer in ruins", + "scene_tags": [ + "ruins", + "blue dress", + "palm", + "travel", + "history", + "outdoor", + "adventure", + "stone" + ], + "suggested_caption": "discovering hidden stories", + "reasoning": "Strong vertical composition, vibrant subject against textured ruins, but slightly overcast light reduces peak aesthetic potential.", + "latency_ms": 5429, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2f9f26c4-61ba-4578-a3a2-ca62f62556ad", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 6, + "instagram_fit_score": 7, + "primary_subject": "palm trees in city", + "scene_tags": [ + "palm trees", + "buildings", + "street", + "overcast", + "urban", + "architecture", + "Spain", + "Barcelona" + ], + "suggested_caption": "palm-lined streets of Barcelona", + "reasoning": "Composition is vertical but slightly tilted, light is flat due to overcast sky, colors are muted, subject clarity is good but not striking.", + "latency_ms": 5777, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 108 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "deb79a4f-f939-4bd4-bf84-16a39ed5f134", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "curious woman in patterned space", + "scene_tags": [ + "mosque", + "tile", + "pattern", + "blue", + "gold", + "intrigue", + "travel", + "cultural" + ], + "suggested_caption": "Lost in the patterns.", + "reasoning": "Strong subject and vertical framing, but lighting is a bit flat, reducing overall aesthetic appeal.", + "latency_ms": 5322, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a218cf81-a745-4860-b036-dea56c4941f8", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "mosque", + "architectural", + "tiled", + "indoor", + "travel", + "culture", + "fashion", + "serene" + ], + "suggested_caption": "exploring the beauty of history.", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5394, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 97 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90a58b73-2d88-48a4-9fe4-ce70286f84d4", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "antique library interior", + "scene_tags": [ + "books", + "antiques", + "ornate", + "pink walls", + "chandeliers", + "statues", + "certificates", + "classic" + ], + "suggested_caption": "a glimpse into a world of old-world charm and literary treasures.", + "reasoning": "The composition is rich with details, but the warm lighting and vertical framing make it visually engaging for social media.", + "latency_ms": 5730, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fcaab99b-7200-4f81-8642-eba0c3bc21ec", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "ancient stone tower", + "scene_tags": [ + "tower", + "flowers", + "sky", + "greenery", + "travel", + "history", + "architecture", + "nature" + ], + "suggested_caption": "hidden tower, blooming secrets.", + "reasoning": "Strong vertical composition and natural framing, but lighting is flat and colors are muted, reducing overall impact.", + "latency_ms": 5366, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 95 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "be8661b0-f545-4afa-94c5-e10da5148041", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "ruins", + "palm tree", + "cobblestone", + "ancient", + "travel", + "outdoor", + "historical", + "summer" + ], + "suggested_caption": "blue in the ruins", + "reasoning": "Strong subject and vertical framing, but slightly cluttered background reduces portfolio-worthy potential.", + "latency_ms": 5471, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 96 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f210a4ea-8cf9-47a3-a2b2-dcd99e26e4b7", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "man in blue shirt", + "scene_tags": [ + "yellow wall", + "ivy", + "large leaves", + "outdoor", + "relaxed", + "sunny", + "patio", + "greenery" + ], + "suggested_caption": "sunshine and serenity.", + "reasoning": "Good lighting and composition, but lacks strong visual hook for IG scrolling.", + "latency_ms": 5221, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4cad0011-b3c6-4612-a398-9a5cae315282", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "night market crowd", + "scene_tags": "sunset, crowd, street, buildings, lights, market, evening, urban", + "suggested_caption": "evening buzz in the square", + "reasoning": "Strong vertical composition, vibrant evening light, clear subject, and Instagram-friendly vertical framing with a lively scene.", + "latency_ms": 5212, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 88 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "991386b2-325b-496b-a61c-76dadc8ca281", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "man at palace", + "scene_tags": [ + "palace", + "balcony", + "palm_tree", + "purple_flowers", + "sunshine", + "travel", + "architecture", + "relaxation" + ], + "suggested_caption": "sunshine & stone", + "reasoning": "Strong subject, vibrant colors, and vertical framing make it visually engaging and suitable for Instagram.", + "latency_ms": 5513, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 99 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c98270c9-ee08-4e17-9f2a-2b004a1fa25f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "mosque", + "mosaic", + "blue", + "gold", + "pattern", + "travel", + "culture", + "elegance" + ], + "suggested_caption": "blue and gold harmony.", + "reasoning": "Strong subject and vibrant pattern, but slightly cluttered background reduces portfolio quality.", + "latency_ms": 5533, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 91 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0aa895b8-729e-46ed-a5cf-eea974827553", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "urban street scene", + "scene_tags": [ + "madrid", + "buildings", + "sky", + "people", + "umbrella", + "architecture", + "street", + "daylight" + ], + "suggested_caption": "Madrid's charm in the rain.", + "reasoning": "Strong vertical composition and vibrant colors make it visually appealing and Instagram-friendly.", + "latency_ms": 5690, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 92 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "34258ec5-302b-4df3-a3a5-3ddfa7f4cd56", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "door", + "mosaic", + "historical", + "travel", + "art", + "Spain", + "colorful", + "pattern" + ], + "suggested_caption": "blue and gold", + "reasoning": "Strong subject and vibrant colors, but slightly cluttered background reduces portfolio quality.", + "latency_ms": 5306, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 89 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d84ea8b1-0ffd-4680-b0dd-fbf2e21c05ea", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "smiling tourists in city", + "scene_tags": "barcelona, old town, architecture, sky, trees, people, selfie, urban", + "suggested_caption": "together in the old town.", + "reasoning": "Strong vertical composition, good lighting, clear subjects, but less artistic than portfolio-worthy.", + "latency_ms": 5117, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 87 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "09b42418-79f1-421e-9b7a-906df03ea290", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "rooftop", + "cityscape", + "trees", + "cloudy", + "urban", + "travel", + "casual", + "summer" + ], + "suggested_caption": "blue dress, city view", + "reasoning": "Strong subject and vertical framing, but lighting is flat and composition is slightly casual.", + "latency_ms": 5298, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 94 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b80c556-78b3-473c-aaa9-e771e034918f", + "model": "minicpm-v-4-5", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "balcony", + "palm trees", + "sunlight", + "historical\u5efa\u7b51", + "greenery", + "travel", + "summer", + "relaxation" + ], + "suggested_caption": "balcony view, sunshine & history", + "reasoning": "Strong subject, good vertical framing, bright colors, and clear details make it visually appealing and Instagram-friendly.", + "latency_ms": 5789, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 538, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9f9c9faf-6bce-4ceb-a1b5-e63179c55990", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man in garden", + "scene_tags": [ + "palmera", + "archway", + "purple flowers", + "sunlit", + "mediterranean", + "historic", + "garden", + "balcony" + ], + "suggested_caption": "Found the perfect spot to pause. \ud83c\udf1e #travel #architecture", + "reasoning": "Strong composition with framing, vibrant colors, and clear subject. Instagram-friendly with immediate visual hook.", + "latency_ms": 26222, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 128 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fec017d6-54cc-4ee9-89f8-6dd0109e6603", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly haired woman", + "scene_tags": [ + "indoor", + "plants", + "curly hair", + "denim jacket" + ], + "suggested_caption": "cozy corner with curls & greenery.", + "reasoning": "Warm tones and natural light create inviting mood; vertical framing works well for IG, but composition is slightly off-center.", + "latency_ms": 3552, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 106 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0d53158b-5a17-4ecc-8e92-0195fb6b12a7", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "modern architecture", + "scene_tags": [ + "seville", + "sunset", + "architecture", + "blue sky", + "urban", + "modern", + "sunset", + "clouds" + ], + "suggested_caption": "Seville\u2019s iconic modern structure against a vibrant blue sky.", + "reasoning": "Strong composition and color contrast make it visually arresting, ideal for IG\u2019s fast-scrolling feed.", + "latency_ms": 3379, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "23137c1c-f475-4b9a-8c72-e86c90f8f064", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man with coffee", + "scene_tags": [ + "coffee shop", + "latte art", + "casual", + "indoor", + "smiling", + "modern", + "relaxed" + ], + "suggested_caption": "Coffee & chill. Perfectly imperfect moment.", + "reasoning": "Warm lighting and candid smile create inviting mood; vertical framing works well for IG, but composition is slightly cluttered.", + "latency_ms": 3449, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3376ee87-8bf5-4710-88e0-f0d89b3587fc", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "portuguese tiles", + "curly hair", + "blue dress", + "cultural backdrop" + ], + "suggested_caption": "Lost in the tiles. \ud83c\udf3f", + "reasoning": "Rich colors and patterned background create visual depth; vertical framing works well for Instagram, with strong subject and immediate visual hook.", + "latency_ms": 3156, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4f2fea45-3119-4df2-bca8-d35a85fd1aa9", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "medieval courtyard", + "scene_tags": [ + "palace", + "palm trees", + "arabesque", + "mediterranean", + "arched walkway", + "ornate fountain", + "sunlit garden" + ], + "suggested_caption": "Timeless beauty in a Spanish courtyard. \ud83c\udf3f", + "reasoning": "Strong composition with warm tones and symmetry, but vertical framing feels slightly off-center for Instagram.", + "latency_ms": 3440, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 125 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e49bb267-b7da-4528-a37e-aa92770a0271", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "urban skyline", + "curly hair", + "stone wall", + "overcast sky" + ], + "suggested_caption": "blue dress against the city. standing tall. #travel #style", + "reasoning": "Strong color contrast and confident pose make it Instagram-friendly, though lighting is flat and composition is slightly unbalanced.", + "latency_ms": 3208, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2df74bf4-e363-4c43-a0ea-8077666229f6", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "mediterranean garden", + "stone fountain", + "curved hedges", + "red flowers", + "tourists", + "arched walkway", + "sunlit courtyard" + ], + "suggested_caption": "A quiet moment in a sun-drenched garden. \ud83c\udf3f", + "reasoning": "Good color contrast and composition, but slightly overexposed sky reduces visual punch. Strong vertical framing and subject make it scroll-stopping.", + "latency_ms": 3734, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 136 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "750f9142-5e86-468a-89db-e9f28d6aac05", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman by pool", + "scene_tags": [ + "mediterranean", + "duck", + "tile pool", + "sunlit", + "curly hair", + "blue dress", + "old wall" + ], + "suggested_caption": "Sun-drenched poolside moment with ducks and history.", + "reasoning": "Strong color harmony and composition, but vertical framing feels slightly off-balance for IG.", + "latency_ms": 3149, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ea713729-51b7-4005-834b-bd3064455928", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman walking past buildings", + "scene_tags": [ + "barcelona", + "catalan architecture", + "street cafe", + "urban walk", + "purple pants", + "sunset light", + "european city" + ], + "suggested_caption": "Walking through Barcelona\u2019s colorful streets, backlit by golden hour.", + "reasoning": "Strong architectural contrast and color harmony, but composition feels slightly off-center for IG scroll engagement.", + "latency_ms": 3377, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b0734c0-589c-451a-a41f-8fc45d757ccc", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "tile art", + "portuguese architecture", + "blue dress", + "doorway", + "curious gaze", + "cultural heritage" + ], + "suggested_caption": "Lost in the tiles. \ud83c\udf3f", + "reasoning": "Rich colors and ornate tiles create visual depth; vertical framing works well for IG, with strong subject and immediate hook.", + "latency_ms": 3560, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e6ebf7b0-7f4d-40b3-919e-3986cbd595fe", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "grand plaza building", + "scene_tags": [ + "spain", + "plaza", + "architecture", + "sunlight", + "carriage", + "historic", + "blue sky", + "spanish flag" + ], + "suggested_caption": "Grand architecture meets sunny streets. Madrid\u2019s Plaza de Espa\u00f1a in all its glory.", + "reasoning": "Strong composition and color contrast, but vertical framing feels slightly off-center for IG scroll engagement.", + "latency_ms": 3665, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 125 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ee9d68c9-e20c-45a9-8d69-7fe7cecda6c6", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "modern architecture", + "scene_tags": [ + "seville", + "sunset", + "architecture", + "urban", + "blue sky", + "shadows", + "cultural" + ], + "suggested_caption": "Architecture that bends the rules. Seville\u2019s iconic structure under a perfect sky.", + "reasoning": "Strong composition and color contrast make it visually arresting, ideal for IG\u2019s scroll-driven feed.", + "latency_ms": 3603, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 118 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "eade13f0-608b-4e5b-a11c-610287226371", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza de espana", + "blue tile", + "sunlit", + "curly hair", + "travel", + "architecture", + "smiling" + ], + "suggested_caption": "Seville\u2019s magic captured in a blue dress & sunlit smiles. \ud83c\udf1e", + "reasoning": "Strong composition with vibrant colors and iconic backdrop; vertical framing works well for IG, instantly recognizable travel moment.", + "latency_ms": 3737, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 134 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "75cfcb0b-fd7b-4c03-a399-f6253e3db200", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "bullring overlook", + "scene_tags": [ + "bullring", + "coastal city", + "urban landscape", + "tree branches", + "pathway", + "sea view", + "spanish architecture", + "overlook" + ], + "suggested_caption": "From the hilltop, the bullring meets the sea. #travel #spain", + "reasoning": "Good composition with natural framing, but muted lighting reduces visual punch. Strong Instagram hook via unique perspective and scenic sweep.", + "latency_ms": 3742, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 136 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6d2920c8-08d1-4bef-9abc-bc91c446570f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in dress", + "scene_tags": [ + "patio", + "archway", + "greenery", + "fountain", + "curly hair", + "blue dress", + "sunlit" + ], + "suggested_caption": "Sun-drenched courtyard vibes. \ud83c\udf1e", + "reasoning": "Strong subject and framing, but lighting contrast and composition could be sharper for portfolio quality.", + "latency_ms": 3147, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c2769ba4-f2c0-49c1-a4b8-6ce0e083f8ec", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "grand building", + "scene_tags": [ + "architecture", + "palma", + "cultural", + "sunlit", + "steps", + "flags", + "palm trees" + ], + "suggested_caption": "Grand architecture under a bright sky. Perfect for a Mediterranean city stroll.", + "reasoning": "Strong composition with warm tones and clear subject, but vertical framing feels slightly off-center for IG.", + "latency_ms": 3305, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 117 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "17905afc-be7f-4664-948a-1c9da287b703", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "mediterranean", + "courtyard", + "palms", + "flowers", + "arches", + "tradition", + "travel", + "elegant" + ], + "suggested_caption": "Lost in the gardens of Andalusia. \ud83c\udf3f", + "reasoning": "Strong composition with warm tones and cultural context; vertical framing works well for IG, with clear subject and visual hook.", + "latency_ms": 3387, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 127 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "40f82498-2e9c-4b98-8ebb-ed9821c179a0", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "urban street scene", + "scene_tags": [ + "cityscape", + "architecture", + "blue sky", + "street view", + "modern buildings", + "purple tree", + "red car" + ], + "suggested_caption": "City meets sky. Architecture in motion.", + "reasoning": "Good color contrast and composition, but lacks instant visual hook for IG. Urban subject is clear but not eye-catching enough to stop scrolls.", + "latency_ms": 3531, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 121 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5a879592-e59b-4a79-8886-dc089e301f5a", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in dress", + "scene_tags": [ + "seville", + "plaza de espana", + "blue dress", + "sunlit", + "architecture", + "bridge", + "ceramic railings" + ], + "suggested_caption": "Seville\u2019s magic captured in a blue dress & sunlit arches.", + "reasoning": "Strong composition with vibrant colors and iconic backdrop; vertical framing works well for IG, instantly engaging viewer.", + "latency_ms": 3560, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 125 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "885508f8-f39b-484e-8959-929111ab0b0d", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza de espana", + "architecture", + "sunlit", + "river", + "tourist spot", + "ceramic railings", + "blue dress" + ], + "suggested_caption": "Seville\u2019s magic captured in a blue dress & sunlit plaza.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, perfect for IG\u2019s visual-first scroll. The woman adds personal connection and immediate visual hook.", + "latency_ms": 3715, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 138 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90b60141-efa8-4a34-8e0a-dda28f296f58", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "vintage courtyard", + "pink bougainvillea", + "white wrought iron", + "mediterranean style" + ], + "suggested_caption": "whispers of the courtyard, captured in blue and bloom.", + "reasoning": "Balanced composition with vibrant color contrast, strong visual hook for IG scroll, slightly soft focus adds charm.", + "latency_ms": 3064, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 113 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "876f997c-e9be-40e2-8482-ea0216b82d5a", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man on bench", + "scene_tags": [ + "greenery", + "bench", + "outdoor", + "relaxed", + "blue shirt", + "natural light", + "garden", + "casual" + ], + "suggested_caption": "quiet moment in the garden.", + "reasoning": "Good color harmony and natural light, but composition is slightly flat; Instagram-friendly due to clear subject and vertical framing.", + "latency_ms": 3282, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d37f1824-5d53-41ca-8c79-ffae48978442", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "mediterranean", + "courtyard", + "flowers", + "arched columns" + ], + "suggested_caption": "Lost in the garden's quiet magic.", + "reasoning": "Warm tones and natural light create a serene mood; vertical framing draws eyes to the subject instantly.", + "latency_ms": 2836, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 100 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7a13fad2-6ef9-44ea-944c-551e854b6acd", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "cityscape", + "curly hair", + "blue dress", + "stone wall", + "overcast sky" + ], + "suggested_caption": "blue dress against the city. where the sky meets the streets.", + "reasoning": "Strong color contrast and vertical framing make it Instagram-friendly, though lighting is flat and composition is slightly unbalanced.", + "latency_ms": 3271, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "64ebfd3c-e9a1-49c2-9d0c-a81182ef6af2", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "portuguese tiles", + "blue dress", + "curly hair", + "mediterranean vibe" + ], + "suggested_caption": "Lost in the tiles. \ud83c\udf1e #travel #portugal", + "reasoning": "Rich colors and patterned background create visual interest; vertical framing works well for Instagram, with strong subject and immediate hook.", + "latency_ms": 3103, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d211ea44-4f3c-44d0-b7f0-cb9d001f07af", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in dress", + "scene_tags": [ + "green garden", + "sunlight rays", + "curly hair", + "flowing dress" + ], + "suggested_caption": "dancing in the light, green garden vibes", + "reasoning": "aesthetic score reflects warm tones and motion, instagram fit score is high due to vertical framing and visual hook.", + "latency_ms": 2957, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 107 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "30015fce-9f43-430e-8268-36cb4cb56596", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "garden", + "curly hair", + "sunset", + "elegant", + "outdoor", + "portrait", + "summer" + ], + "suggested_caption": "Sun-kissed moment in the garden. \ud83c\udf3f", + "reasoning": "Strong composition and warm tones make it visually appealing, with a clear hook for Instagram viewers.", + "latency_ms": 3149, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 117 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9842a059-f560-4243-9710-84a9ef7ee71e", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man at plaza", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "sunlight", + "tourist", + "bridge", + "tiles" + ], + "suggested_caption": "Seville\u2019s magic. Sun, tiles, and a little wanderlust.", + "reasoning": "Strong composition with vibrant colors and iconic architecture, perfect for IG\u2019s visual-first feed. Man\u2019s pose adds personality and immediacy.", + "latency_ms": 3358, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 125 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f26f6930-16a5-400f-ab7c-ff19abde8ef3", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza de espana", + "blue dress", + "sunlit", + "architecture", + "canal", + "travel" + ], + "suggested_caption": "Seville\u2019s magic captured in a blue dress & sunshine.", + "reasoning": "Strong color harmony and composition, with a clear visual hook for Instagram scrolling.", + "latency_ms": 3158, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d99cf40e-3bed-47f7-8595-c11286ab3a6a", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza", + "sunlit", + "architectural", + "curly hair", + "bridge", + "cultural" + ], + "suggested_caption": "Sun-drenched Seville. History in the frame. \ud83c\udf1e", + "reasoning": "Strong composition with vibrant colors and a compelling human element, ideal for IG scroll-stopping.", + "latency_ms": 3359, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 122 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2e199e4b-3fb5-442f-895e-3ab7ba9ccdaf", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "intricate arabic architecture", + "scene_tags": [ + "mosque", + "mosaic", + "arabesque", + "ornate", + "mediterranean", + "historical", + "ceiling", + "arch" + ], + "suggested_caption": "Where art meets history. Intricate arabesque patterns in a historic Moroccan mosque.", + "reasoning": "High detail and rich color make it visually striking, but vertical framing lacks immediate visual hook for IG.", + "latency_ms": 3596, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 133 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "13dc549f-8aee-41b4-92b8-954b5bc41fb2", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "flamenco performance", + "scene_tags": [ + "flamenco", + "theater", + "stage", + "performance", + "seville", + "guitar", + "dancers", + "purple lighting" + ], + "suggested_caption": "Live flamenco magic at Teatro Flamenco Sevilla. Pure energy on stage.", + "reasoning": "Stage lighting and composition are strong, but color saturation is slightly overdone. Perfect for IG scroll stopper with clear subject and vertical framing.", + "latency_ms": 3763, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 136 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "144aadc1-0e9c-46c7-80dd-8decd901e4ee", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman with flowers", + "scene_tags": [ + "bougainvillea", + "curly hair", + "denim jacket", + "green bag", + "smiling woman", + "lush greenery", + "vibrant blooms", + "sandy path" + ], + "suggested_caption": "Lost in the bloom. \ud83c\udf3a", + "reasoning": "Strong color contrast and natural light make it visually arresting; vertical framing works well for Instagram scrolling.", + "latency_ms": 3562, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3c464a1a-ab0e-4703-9bcd-37ccb17f8dac", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly woman street style", + "scene_tags": [ + "urban street", + "curly hair", + "denim jacket", + "green bag", + "sunglasses", + "sunlit sidewalk" + ], + "suggested_caption": "Street style with attitude. Curly hair, denim, and green vibes.", + "reasoning": "Strong subject and color contrast make it Instagram-friendly, but composition is slightly flat.", + "latency_ms": 3389, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0f816473-c0f5-44eb-96a1-9c982f7f0343", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "palmera", + "alhambra", + "sunset", + "mediterranean", + "garden", + "luxury", + "travel", + "elegant" + ], + "suggested_caption": "Blue dress, palm trees, and a palace. Perfect day in Seville.", + "reasoning": "Strong color contrast and composition, with vertical framing ideal for Instagram. Subject is clear and visually engaging.", + "latency_ms": 4338, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 132 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "891f2361-7c42-4163-b37a-c11f7f7126f2", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza de espana", + "tiled stairs", + "sunset", + "elegant", + "travel", + "architecture", + "sunglasses" + ], + "suggested_caption": "Seville\u2019s magic in a blue dress. \ud83c\udf1e #plazadesevila", + "reasoning": "Strong color harmony and architectural detail elevate aesthetic score; vertical framing and subject hook make it Instagram-ready.", + "latency_ms": 4635, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 136 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "711f44c3-d2f8-4878-b587-4863e268dd3c", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "cozy bar interior", + "scene_tags": [ + "bar", + "wooden counter", + "chairs", + "warm lighting", + "european cafe", + "barista", + "wine rack", + "old photos" + ], + "suggested_caption": "Cozy bar with vintage charm. Perfect for a quiet evening.", + "reasoning": "Warm tones and inviting layout score 7/10 for aesthetics, 8/10 for IG fit due to strong visual hook and vertical framing.", + "latency_ms": 4473, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 138 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "baf172dd-da76-4c3c-80b6-99382aeb4a72", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "vine-covered", + "sunlit path", + "mediterranean", + "archway", + "flowing dress", + "greenery", + "cobbled walkway", + "woman posing" + ], + "suggested_caption": "Dancing through the vines. Where light meets green.", + "reasoning": "Strong composition with dappled light and flowing dress, but vertical framing feels slightly off for IG. Aesthetic is high, but Instagram engagement is moderate due to framing.", + "latency_ms": 4112, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 142 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "6a918ad8-de2d-48e9-a378-c7038c9e5178", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "art gallery", + "bullfighter portrait", + "blue dress", + "curly hair" + ], + "suggested_caption": "Lost in art. Blue dress. Bullfighter. Quiet magic.", + "reasoning": "Strong color contrast and composition, but lighting is flat; Instagram-friendly due to vertical framing and visual interest.", + "latency_ms": 3283, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 109 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4b09475f-0eee-46ba-a558-1f0cd3ecd473", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in ornate room", + "scene_tags": [ + "mediterranean", + "tile floor", + "wood beams", + "historical interior" + ], + "suggested_caption": "Lost in timeless elegance. \ud83c\udf3f", + "reasoning": "Rich textures and warm tones create depth, but vertical framing feels slightly off-center for IG scroll engagement.", + "latency_ms": 2981, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 105 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ff2de652-1f1a-4d5c-a7ee-28166ce7118f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "horse carriage", + "scene_tags": [ + "street", + "horse", + "carriage", + "trees", + "european", + "sunset", + "cobbled", + "old" + ], + "suggested_caption": "Vintage charm on cobblestone streets. \ud83d\udc0e", + "reasoning": "Strong composition with contrast between vintage carriage and modern car, good color harmony, but Instagram fit slightly compromised by background clutter.", + "latency_ms": 3397, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "852f15e1-94f2-49db-8205-218c85d0a43f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "grand plaza building", + "scene_tags": [ + "spain", + "plaza", + "architecture", + "sunlit", + "historic", + "blue sky", + "flag" + ], + "suggested_caption": "Sun-drenched grandeur. A Spanish plaza in all its ornate glory.", + "reasoning": "Strong architectural detail and bright light make it visually striking, but composition is slightly wide for vertical feed.", + "latency_ms": 3283, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 120 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c869757b-8734-4a8d-8dfb-fad070bf7be1", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in front of historic gate", + "scene_tags": [ + "andalucia", + "historic gate", + "sunlit courtyard", + "tourist spot", + "medieval architecture", + "blue sky", + "green hedges" + ], + "suggested_caption": "Walking through history under a bright sky. #Andalusia", + "reasoning": "Strong composition with clear subject and vibrant light, but slightly cluttered with people. Instagram-friendly framing and visual hook.", + "latency_ms": 3654, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 131 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "3ba4c50c-6ec4-41b1-bb2b-e9d4eea108f6", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "night church", + "scene_tags": [ + "night", + "church", + "statue", + "urban", + "illuminated", + "square", + "european", + "architecture" + ], + "suggested_caption": "Night lights on a grand church. Quiet, majestic, and full of stories.", + "reasoning": "Strong architectural subject with dramatic lighting, but composition is slightly off-center; works well vertically for IG.", + "latency_ms": 3361, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e5fc776e-8467-4805-8c35-77383f62d76d", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in dress", + "scene_tags": [ + "sunlight", + "patio", + "green walls", + "long dress" + ], + "suggested_caption": "dressed in sunlit grace.", + "reasoning": "Warm tones and patterned dress create visual interest, but color saturation is slightly overdone. Vertical framing works well for Instagram, with strong subject and immediate visual hook.", + "latency_ms": 3168, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 112 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b1a8b354-f9a4-42db-9639-44fd9ecbdd33", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "grand plaza architecture", + "scene_tags": [ + "plaza", + "architecture", + "sunlight", + "spanish", + "tourist", + "bridge", + "carriage" + ], + "suggested_caption": "Sun-drenched grandeur. A day at the Plaza de Espa\u00f1a.", + "reasoning": "Strong symmetry and vibrant sky elevate aesthetics, but vertical framing lacks immediate visual hook for IG.", + "latency_ms": 3285, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 117 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ba6f613c-61cd-42a7-aaa9-374d4f0ac058", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man on bridge", + "scene_tags": [ + "seville", + "plaza de espana", + "tilework", + "sunlit", + "architecture", + "travel", + "bridge", + "blue sky" + ], + "suggested_caption": "Sun-drenched Seville. Where history meets the present. \ud83c\udf1e", + "reasoning": "Strong composition with vibrant colors and clear subject, ideal for IG scroll-stopping. Aesthetic is polished but not overdone.", + "latency_ms": 3821, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 133 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "82391768-2834-4843-9cdd-249bd6780f91", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "fried dumplings", + "scene_tags": [ + "restaurant", + "wood table", + "food", + "close-up", + "dumplings", + "sauce", + "cabbage" + ], + "suggested_caption": "crispy dumplings with a tangy twist. perfect bite.", + "reasoning": "Good color and composition, but shallow depth of field reduces visual impact. Strong IG hook with food subject and vertical framing.", + "latency_ms": 3845, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 124 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "01292c70-996c-43cd-b132-5933d3afbf43", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "andalucia", + "flower arch", + "white courtyard", + "boho style", + "cultural architecture", + "spring blooms", + "travel photography", + "mediterranean" + ], + "suggested_caption": "Lost in the bloom of Andalusia. \ud83c\udf39 #travel #boho", + "reasoning": "Strong color contrast and vertical framing make it Instagram-friendly; aesthetic is polished but not overdone.", + "latency_ms": 3762, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 134 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "edb6e566-0350-4861-ae89-e32ddcfd4f14", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "courtyard", + "mediterranean", + "sunlit", + "vintage", + "planters", + "fountain", + "archways" + ], + "suggested_caption": "Lost in the garden's quiet magic.", + "reasoning": "Strong composition with warm light and rich colors, vertical framing works well for IG, subject is instantly engaging.", + "latency_ms": 3343, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 117 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b9c7f1d5-7336-4860-a38a-c3e1d82ee655", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "curly woman", + "scene_tags": [ + "street", + "sunglasses", + "curly hair", + "urban", + "smiling", + "phone", + "green bag" + ], + "suggested_caption": "Curly hair, sunglasses, and a smile. Street style, captured.", + "reasoning": "Strong subject and vertical framing with natural light, but background distraction slightly lowers aesthetic score.", + "latency_ms": 3650, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 118 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "967b48e8-531a-4bef-a069-fab60690145f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "pizza kitchen", + "scene_tags": [ + "pizza", + "italian", + "cooking", + "lemon", + "bakery", + "oven", + "food", + "warm" + ], + "suggested_caption": "Freshly baked pizza in a cozy, lemon-scented pizzeria. \ud83c\udf55", + "reasoning": "Good color harmony and composition, but slightly cluttered. Instagram-friendly with strong visual hook and vertical framing.", + "latency_ms": 3598, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 128 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "b68453c0-f5a3-4e8e-b231-af92fa2ed182", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "mediterranean", + "tile walls", + "terrace", + "curly hair", + "boho style", + "golden accessories", + "sunlit" + ], + "suggested_caption": "Blue dress, golden details, and a quiet moment in a sunlit courtyard.", + "reasoning": "Strong color harmony and composition, with vertical framing ideal for Instagram. The subject is instantly recognizable and visually engaging.", + "latency_ms": 3554, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 129 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7f9762e4-afb1-45f5-85f0-fd33e050ed23", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man on bench", + "scene_tags": [ + "greenery", + "outdoor", + "bench", + "garden", + "blue shirt", + "relaxed", + "natural light" + ], + "suggested_caption": "quiet moment in the garden", + "reasoning": "balanced composition with natural light, but muted colors limit portfolio appeal; strong visual hook for IG due to expressive pose and green backdrop.", + "latency_ms": 3329, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 118 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c61412cb-4abc-4d1c-b541-f1d67adc3b03", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "medieval wall", + "palm tree", + "cobblestone path", + "blue dress", + "archway", + "curly hair", + "sunlit" + ], + "suggested_caption": "blue dress against ancient stones. quiet, bold, timeless.", + "reasoning": "Strong color contrast and composition make it visually arresting; vertical framing works well for Instagram.", + "latency_ms": 3786, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 122 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4719aae4-0783-447a-afc2-9f84c7eca3d1", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman viewing plaza", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "sunlight", + "curly hair", + "travel", + "european", + "blue sky" + ], + "suggested_caption": "Lost in the grandeur of Seville's Plaza de Espa\u00f1a.", + "reasoning": "Strong composition with vibrant colors and a clear focal point, ideal for Instagram\u2019s vertical scroll. The subject\u2019s back view creates a personal, immersive feel.", + "latency_ms": 3867, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 134 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ae8d0f5-432d-40ae-bf24-36aebb8977d0", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "cathedral", + "sunset", + "street", + "travel", + "europe", + "architecture", + "urban" + ], + "suggested_caption": "Seville cathedral & me in blue. \ud83c\udf1e #travel #seville", + "reasoning": "Strong color contrast and composition draw attention; vertical framing works well for IG, with clear subject and visual hook.", + "latency_ms": 3614, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 129 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f32a27d2-44e9-41fe-acd3-b581825b7d37", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man in plaza", + "scene_tags": [ + "seville", + "plaza", + "architecture", + "sunlight", + "tourist", + "blue tiles", + "canal" + ], + "suggested_caption": "Sun-drenched Seville. Where history meets the present. \ud83c\udf1e", + "reasoning": "Strong composition with vibrant colors and iconic architecture, perfect for IG\u2019s visual-first feed. Man\u2019s pose adds human interest and scale.", + "latency_ms": 3415, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 128 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "309a7b9a-ab2a-44b6-98d7-a026f3f9ad2d", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "palmera", + "mediterranean", + "archway", + "garden", + "luxury", + "sunlit", + "elegant", + "tourist" + ], + "suggested_caption": "Lost in the gardens. \ud83c\udf3f", + "reasoning": "Strong framing through archway, vibrant colors, and model's pose create visual interest and aesthetic appeal, ideal for IG scroll-stopping.", + "latency_ms": 3845, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "af98052f-97e1-4c37-a719-55fa6f6a5bab", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "mediterranean", + "courtyard", + "palmtree", + "floral", + "arched", + "travel", + "photography", + "colorful" + ], + "suggested_caption": "Lost in the gardens of a historic courtyard.", + "reasoning": "Strong color harmony and composition, with a clear visual hook for Instagram scrolling.", + "latency_ms": 3356, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "556dce46-6930-40fa-b1ee-6b58f1967aa0", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "ornate interior", + "mosaic floor", + "staircase", + "luxury villa", + "curved staircase", + "blue dress", + "historical architecture", + "woman posing" + ], + "suggested_caption": "Blue dress in a palace of patterns. Where elegance meets history.", + "reasoning": "Strong color harmony and architectural detail elevate aesthetic score; vertical framing and subject hook make it Instagram-ready.", + "latency_ms": 3559, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 132 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f6fe897c-9c59-4d5c-8026-f532b131e800", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "woman in garden", + "scene_tags": [ + "alhambra", + "courtyard", + "palms", + "mediterranean", + "flowerbeds", + "archways", + "travel", + "elegant" + ], + "suggested_caption": "Lost in the gardens of a historic courtyard. \ud83c\udf3f", + "reasoning": "Strong composition with warm tones and cultural context, but slightly cluttered framing reduces instant visual hook.", + "latency_ms": 3456, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 127 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d4a4990f-30ac-4378-a20c-0b1083883665", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman at plaza", + "scene_tags": [ + "seville", + "plaza", + "sunlit", + "architecture", + "tourist", + "sunset", + "cultural", + "elegant" + ], + "suggested_caption": "Sun-drenched moments in Seville's grand plaza.", + "reasoning": "Strong composition with vibrant colors and a clear subject, ideal for IG scroll-stopping.", + "latency_ms": 3455, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "e71b31a1-bd05-418b-bfe1-d7dcbc8398ed", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "flower-covered doorway", + "scene_tags": [ + "flowering vines", + "sunset colors", + "mediterranean architecture", + "vibrant blooms", + "purple flowers", + "white walls", + "open doorway", + "blue sky" + ], + "suggested_caption": "Colorful blooms frame a quiet doorway. Perfect for a slow scroll.", + "reasoning": "High color contrast and vertical framing make it Instagram-ready, with strong visual impact from the flowers and sky.", + "latency_ms": 3744, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 133 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "60d9defc-36a6-439d-81bb-3f446972bf14", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "cityscape view", + "scene_tags": [ + "urban", + "coastal", + "overcast", + "gardens", + "port", + "architecture", + "hills", + "people" + ], + "suggested_caption": "From the hilltop, the city meets the sea under a cloudy sky.", + "reasoning": "Balanced composition with muted tones; Instagram fit is decent but lacks immediate visual punch.", + "latency_ms": 3427, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 121 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ad632dce-c371-4166-8490-27e60fcea65a", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "andalucia", + "historic building", + "staircase", + "curly hair", + "sunlit", + "travel", + "architecture", + "blue dress" + ], + "suggested_caption": "Lost in the sunlit steps of a historic Andalusian courtyard.", + "reasoning": "Strong color contrast and composition draw attention; vertical framing works well for Instagram, with a compelling subject and setting.", + "latency_ms": 3703, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "38f26552-7b4e-4488-880d-a90e9f5453e7", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "ornate building facade", + "scene_tags": [ + "european architecture", + "sunlit street", + "balcony details", + "historic building", + "blue sky", + "urban elegance", + "arched windows" + ], + "suggested_caption": "Architecture that whispers stories. \ud83c\udfdb\ufe0f", + "reasoning": "Strong composition and color contrast, but vertical framing feels slightly off-center for IG scroll engagement.", + "latency_ms": 3703, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 121 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "efaf5b66-243e-462f-9d5e-ea90fee3c61b", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "grand building", + "scene_tags": [ + "architecture", + "city", + "spanish", + "sunlit", + "columns", + "clocktower", + "boulevard" + ], + "suggested_caption": "Grand architecture under a bright sky. History meets modern streets.", + "reasoning": "Strong composition with symmetry and color contrast, but framing leans slightly too wide for vertical IG feed.", + "latency_ms": 3423, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "067e963d-265e-48f5-a4b4-75da4402043c", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "garden", + "curly hair", + "blue dress", + "sunlit", + "mediterranean", + "back view", + "flowers" + ], + "suggested_caption": "sunlit garden, curly hair, and a smile that says 'hello'.", + "reasoning": "Warm tones and natural light create a soft, inviting aesthetic; vertical framing and expressive pose make it instantly engaging on Instagram.", + "latency_ms": 3685, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 128 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "460b6b6d-ec7c-424b-a899-23c5975039f6", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman at castle", + "scene_tags": [ + "mediterranean", + "castle", + "curly hair", + "cityscape" + ], + "suggested_caption": "blue dress, stone walls, and a city view. perfect day.", + "reasoning": "Good color contrast and composition, but overcast lighting reduces vibrancy. Instagram-friendly framing with strong subject and scenic backdrop.", + "latency_ms": 3245, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a56ee545-2409-49d7-a289-97f2d6f75ca6", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple selfie", + "scene_tags": [ + "cityscape", + "overcast", + "travel", + "couple", + "selfie", + "coastal", + "urban" + ], + "suggested_caption": "captured the moment, the view, the vibe.", + "reasoning": "natural lighting and candid expressions create warmth, while the urban backdrop adds context \u2014 ideal for IG\u2019s scroll-friendly vertical format.", + "latency_ms": 3434, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 121 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "5ed528a9-109c-4855-b3c3-ab81b404f856", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in dress", + "scene_tags": [ + "park", + "archway", + "sunlight", + "garden", + "mediterranean", + "cultural", + "travel", + "fashion" + ], + "suggested_caption": "Sun-drenched garden vibes. Where style meets serenity.", + "reasoning": "Strong composition with warm tones and natural light, ideal for IG scroll-stopping. Subject is clear and framed vertically.", + "latency_ms": 3808, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 126 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8ae46e84-d601-4345-bdb1-f6771f388dc1", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "flower-covered house", + "purple bougainvillea", + "blue dress", + "vintage photo", + "sunlit courtyard", + "travel photography", + "colorful architecture" + ], + "suggested_caption": "Lost in the bloom. \ud83c\udf38 #travel #photography", + "reasoning": "Strong color contrast and composition, vertical framing works well for IG, subject is instantly recognizable and visually engaging.", + "latency_ms": 3836, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1fb7f2c9-2e63-47a0-b59e-ed32de4d6fd5", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in dress", + "scene_tags": [ + "seville", + "plaza", + "sunlight", + "architecture", + "travel", + "sunset", + "elegant", + "cultural" + ], + "suggested_caption": "Seville\u2019s magic captured in a dress and a smile. \ud83c\udf1e", + "reasoning": "Bright, balanced composition with strong color contrast; vertical framing works well for IG, instantly recognizable landmark and subject.", + "latency_ms": 3714, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 128 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f9f81d70-0e5e-4e07-933c-6e97e3aa22c7", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 7, + "primary_subject": "colorful street architecture", + "scene_tags": [ + "seville", + "catalan architecture", + "sunlit street", + "cafes", + "tourist spot", + "blue sky", + "historic buildings" + ], + "suggested_caption": "Architecture that tells a story. Seville\u2019s hidden gem.", + "reasoning": "Strong composition with vibrant colors and architectural contrast, but slightly cluttered foreground reduces Instagram hook potential.", + "latency_ms": 3336, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c63f60bb-2c2a-4ebc-b4e5-bd108a7e5e60", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 6, + "instagram_fit_score": 7, + "primary_subject": "couple in elevator", + "scene_tags": [ + "elevator", + "couple", + "selfie", + "metal walls", + "blue dress", + "curly hair" + ], + "suggested_caption": "Elevator selfie. No one else in the world. Just us.", + "reasoning": "Good lighting and candid moment, but composition is tight and colors are flat. Instagram-friendly due to vertical framing and emotional hook.", + "latency_ms": 3420, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9fc4be6d-dc08-457c-9f5a-1606fb96dd97", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple kissing", + "scene_tags": [ + "seville", + "spanish city", + "mountain view", + "romantic moment", + "overcast sky", + "tourist spot", + "greenery", + "selfie" + ], + "suggested_caption": "kissing on top of a hill with Seville below \ud83c\udf06", + "reasoning": "Good composition with emotional connection, but muted lighting reduces visual impact; fits IG scroll with strong subject and framing.", + "latency_ms": 3671, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 134 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "1d7f8084-eb2e-41e4-a380-d0983f6807e2", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man overlooking city", + "scene_tags": [ + "cityscape", + "overcast sky", + "mountain view", + "urban landscape", + "travel photo", + "wandering", + "mediterranean", + "stone wall" + ], + "suggested_caption": "Found my favorite spot. Sky\u2019s gray, but the view\u2019s worth it.", + "reasoning": "Good composition with subject centered, but muted colors and cloudy sky reduce visual impact. Instagram-friendly due to strong subject and vertical framing.", + "latency_ms": 4063, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 137 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f638054c-bad8-4ac1-83c2-1b0f5149fc76", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "flamenco performers", + "scene_tags": [ + "seville", + "flamenco", + "stage", + "guitar", + "performance", + "purple lighting", + "audience" + ], + "suggested_caption": "Live flamenco magic at Teatro Flamenco Sevilla. Feel the rhythm.", + "reasoning": "Good color contrast and stage lighting, but composition is slightly cluttered by audience silhouettes. Instagram-friendly due to strong subject and vertical framing.", + "latency_ms": 3745, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 131 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "8f36cc4f-67ee-419a-806a-393e02311846", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "garden", + "arched walls", + "sunlit", + "mediterranean", + "palm trees", + "staircase", + "planters", + "cultural" + ], + "suggested_caption": "Sun-drenched garden vibes. Where elegance meets nature.", + "reasoning": "Strong composition with warm tones and natural light, ideal for IG scroll-stopping. Subject is clear and framed vertically.", + "latency_ms": 3525, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "ef806816-857f-4829-9c43-fc7439005b24", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman with curly hair", + "scene_tags": [ + "cityscape", + "mountain view", + "blue dress", + "curly hair", + "overcast sky", + "selfie", + "mediterranean" + ], + "suggested_caption": "captured the view from the top. \ud83c\udf06", + "reasoning": "Strong subject and framing, but muted lighting reduces visual punch. Instagram-friendly with clear hook.", + "latency_ms": 3540, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 122 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "98c6d4f2-f935-4358-8db7-5de1da16c638", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "seville", + "plaza de espana", + "blue tile", + "sunset", + "architecture", + "river", + "travel" + ], + "suggested_caption": "Seville\u2019s magic captured in blue. \ud83c\udf1e #plazadesevilla", + "reasoning": "Strong composition with vibrant colors and iconic architecture, perfect for IG scroll-stopping. Aesthetic is polished but not overdone.", + "latency_ms": 3825, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 131 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "bacbae88-b736-43a5-972d-602d08a3d3a9", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "ivy-covered building", + "scene_tags": [ + "ivy", + "old building", + "europe", + "sunlit", + "balcony", + "greenery", + "architecture", + "vines" + ], + "suggested_caption": "Timeless charm. Ivy climbs old stone. Quiet streets, bright sky.", + "reasoning": "Strong contrast between green ivy and pale wall, vertical framing draws eyes upward, perfect for IG scroll-stopping.", + "latency_ms": 3771, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 129 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "9499f983-a5ee-416b-ae9c-9df25d846dfc", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "medieval castle", + "palm tree", + "blue dress", + "stone path", + "curly hair", + "historical site", + "travel photo", + "back view" + ], + "suggested_caption": "Walking through history in a blue dress. \ud83c\udf3f\ud83c\udff0", + "reasoning": "Strong color contrast and vertical framing make it scroll-stopping; aesthetic is polished but slightly overcast.", + "latency_ms": 3829, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "2f9f26c4-61ba-4578-a3a2-ca62f62556ad", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "palm trees street", + "scene_tags": [ + "palm trees", + "urban street", + "european architecture", + "cloudy sky", + "cityscape", + "shopping district", + "green shutters", + "street view" + ], + "suggested_caption": "Palm trees & old buildings. Quiet street, cloudy day. #travel", + "reasoning": "Good composition with vertical framing, but muted colors and overcast sky reduce visual impact. Instagram fit is moderate due to recognizable subject and vertical orientation.", + "latency_ms": 3973, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 142 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "deb79a4f-f939-4bd4-bf84-16a39ed5f134", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in mosaic room", + "scene_tags": [ + "morocco", + "mosaic", + "medina", + "travel", + "culture", + "blue dress", + "curly hair", + "elegant" + ], + "suggested_caption": "Lost in the patterns. Morocco\u2019s quiet magic.", + "reasoning": "Strong color harmony and cultural context, but composition leans slightly off-center; Instagram-friendly due to vertical framing and visual intrigue.", + "latency_ms": 3597, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 126 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "a218cf81-a745-4860-b036-dea56c4941f8", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "morocco", + "medina", + "ornate tiles", + "archway", + "fashion photography", + "cultural travel" + ], + "suggested_caption": "Lost in the blue. Morocco\u2019s hidden courtyard magic.", + "reasoning": "Strong color harmony and cultural context make it visually arresting, with vertical framing ideal for Instagram scrolling.", + "latency_ms": 3442, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 116 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "90a58b73-2d88-48a4-9fe4-ce70286f84d4", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "antique library", + "scene_tags": [ + "antique", + "bookshelf", + "ornate", + "vintage", + "pink walls", + "decorative", + "historical", + "cabinet" + ], + "suggested_caption": "A room of stories and history. Quietly grand.", + "reasoning": "Rich textures and warm tones create depth, but cluttered composition reduces Instagram appeal.", + "latency_ms": 3428, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 119 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "fcaab99b-7200-4f81-8642-eba0c3bc21ec", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 6, + "primary_subject": "ancient stone tower", + "scene_tags": [ + "mediterranean", + "old tower", + "pink flowers", + "greenery", + "overcast sky", + "historical site", + "planting" + ], + "suggested_caption": "ancient stone tower meets blooming pink flowers. quiet beauty.", + "reasoning": "balanced composition with strong vertical framing, but muted lighting reduces visual impact. good for IG but not portfolio-level.", + "latency_ms": 3704, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 126 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "be8661b0-f545-4afa-94c5-e10da5148041", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "medieval wall", + "palm tree", + "cobblestone path", + "blue dress", + "curly hair", + "arched gate", + "historical site" + ], + "suggested_caption": "blue dress against ancient walls. where history meets style.", + "reasoning": "Strong color contrast and composition make it visually striking, ideal for IG scroll-stopping.", + "latency_ms": 3905, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 123 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "f210a4ea-8cf9-47a3-a2b2-dcd99e26e4b7", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "man in garden", + "scene_tags": [ + "yellow wall", + "green vines", + "tropical plants", + "casual pose", + "outdoor seating", + "blue shirt", + "sunlit" + ], + "suggested_caption": "Found a quiet spot to recharge. \ud83c\udf3f", + "reasoning": "Balanced composition with warm tones and natural light; strong visual hook for IG due to subject and setting.", + "latency_ms": 3545, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 122 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "4cad0011-b3c6-4612-a398-9a5cae315282", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "night market crowd", + "scene_tags": [ + "night market", + "urban nightlife", + "crowded street", + "blue hour", + "cafe lights", + "european city", + "social gathering", + "street photography" + ], + "suggested_caption": "Night market vibes in the city. People, lights, and stories.", + "reasoning": "Good color contrast and composition, but slightly cluttered. Instagram-friendly due to lively crowd and vertical framing.", + "latency_ms": 3785, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 130 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "991386b2-325b-496b-a61c-76dadc8ca281", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "man in garden", + "scene_tags": [ + "palma", + "purple flowers", + "archway", + "historic building" + ], + "suggested_caption": "Found the perfect spot to pause and soak in the magic.", + "reasoning": "Strong composition with natural framing, vibrant colors, and a clear subject that draws the eye quickly.", + "latency_ms": 3087, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 103 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "c98270c9-ee08-4e17-9f2a-2b004a1fa25f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "blue tile wall", + "mediterranean", + "boho style", + "travel portrait", + "patterned background", + "curly hair", + "elegant dress", + "cultural heritage" + ], + "suggested_caption": "Blue tiles, blue dress, and a smile that tells a story. \ud83c\udf1e", + "reasoning": "Rich color harmony and patterned backdrop create visual interest, while vertical framing and subject pose work well for Instagram engagement.", + "latency_ms": 3896, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 141 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "0aa895b8-729e-46ed-a5cf-eea974827553", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "street corner buildings", + "scene_tags": [ + "urban", + "european", + "architecture", + "street", + "people", + "clouds", + "sunset" + ], + "suggested_caption": "Architecture meets everyday life in a European street corner.", + "reasoning": "Good color contrast and composition, but lacks the punch of a top-tier Instagram post due to busy street elements.", + "latency_ms": 3447, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "34258ec5-302b-4df3-a3a5-3ddfa7f4cd56", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "portuguese tiles", + "ornate doorway", + "blue dress", + "cultural heritage", + "travel photography", + "woman posing", + "vintage tiles", + "warm lighting" + ], + "suggested_caption": "Lost in the tiles. Portugal\u2019s artistry, my dress, my moment.", + "reasoning": "Rich colors and intricate tilework create visual depth; vertical framing works well for Instagram, with strong subject and immediate visual hook.", + "latency_ms": 3856, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 877, + "completion_tokens": 139 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "d84ea8b1-0ffd-4680-b0dd-fbf2e21c05ea", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "couple selfie", + "scene_tags": [ + "cityscape", + "tourist spot", + "curly hair", + "architecture" + ], + "suggested_caption": "captured the moment with the city behind us.", + "reasoning": "Good composition with natural light and clear subjects, but slightly overcast sky reduces vibrancy. Instagram-friendly framing with strong hook in first second.", + "latency_ms": 3478, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 111 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "09b42418-79f1-421e-9b7a-906df03ea290", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 7, + "instagram_fit_score": 8, + "primary_subject": "woman in blue dress", + "scene_tags": [ + "urban backdrop", + "brick wall", + "curly hair", + "cityscape", + "green trees", + "overcast sky", + "casual pose" + ], + "suggested_caption": "blue dress, city view, and a little confidence.", + "reasoning": "Strong subject and vertical framing, but muted lighting reduces visual punch.", + "latency_ms": 3521, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 115 + }, + { + "run_id": "2026-05-10-1424", + "asset_id": "7b80c556-78b3-473c-aaa9-e771e034918f", + "model": "qwen3vl-4b", + "prompt_version": "4bbb7e7721da24d9", + "aesthetic_score": 8, + "instagram_fit_score": 9, + "primary_subject": "woman in garden", + "scene_tags": [ + "palmas", + "alhambra", + "garden", + "sunlit", + "archway", + "elegant", + "travel" + ], + "suggested_caption": "Sun-drenched garden view from a hidden archway. \ud83c\udf3f", + "reasoning": "Strong composition with framing, vibrant colors, and a candid moment that hooks viewers quickly on Instagram.", + "latency_ms": 3781, + "parse_error": null, + "error": null, + "finish_reason": "stop", + "prompt_tokens": 973, + "completion_tokens": 122 + } +] \ No newline at end of file diff --git a/docs/known-issues.md b/docs/known-issues.md new file mode 100644 index 00000000..5aada34c --- /dev/null +++ b/docs/known-issues.md @@ -0,0 +1,72 @@ +# Known Issues + +Catalog of recurring or upstream-blocked failure modes with their +mitigations. Anything that requires a manual workaround should be +documented here — if a future session can hit the same issue, it +deserves an entry. Each entry should have: symptom, root cause, current +mitigation, and the trigger that lets us un-mitigate. + +--- + +## 2026-05-17 — NVIDIA GPU driver fails on Ubuntu 26.04 (kernel 7.0.x) + +**Symptom.** `nvidia-driver-daemonset-*` in `nvidia` namespace +CrashLoopBackOff on the GPU node. Logs say: + + Could not resolve Linux kernel version + +… or, post chart-upgrade, ImagePullBackOff on a `*-ubuntu26.04` tag. + +**Root cause.** NVIDIA has not published any `nvcr.io/nvidia/driver:*-ubuntu26.04` +images (0 tags as of 2026-05-17; verified with skopeo). When a k8s node +running the GPU operator gets `do-release-upgrade`'d to Ubuntu 26.04 +Resolute Raccoon, NFD relabels the node with +`feature.node.kubernetes.io/system-os_release.VERSION_ID=26.04` and the +operator computes the driver image tag `-ubuntu26.04` — which +404s on pull. Both gpu-operator chart v25.10.1 and v26.3.1 exhibit the +same behaviour once NFD has detected 26.04. + +**Current mitigation (active on k8s-node1 since 2026-05-17).** + +1. Host kernel rolled back to `6.8.0-117-generic` (Ubuntu 24.04 HWE + kernel — still installed at `/lib/modules/6.8.0-117-generic`). +2. `apt-mark hold` on: `linux-image-6.8.0-117-generic`, + `linux-headers-6.8.0-117-generic`, `linux-modules-6.8.0-117-generic`, + `linux-image-generic`, `linux-headers-generic`, `linux-generic`. +3. `/etc/os-release` on k8s-node1 replaced with the Ubuntu 24.04 Noble + content (was a symlink to `/usr/lib/os-release`; now a regular file + under `/etc`). Backup at `/etc/os-release.bak-pre-spoof-2026-05-17`. + NFD-worker reads `/etc/os-release` and now reports + `system-os_release.VERSION_ID=24.04`, so the operator picks the + matching ubuntu24.04 driver image which DOES exist. +4. gpu-operator chart pinned to v25.10.1 in + `stacks/nvidia/modules/nvidia/main.tf`; driver pinned to 570.195.03 + in `stacks/nvidia/modules/nvidia/values.yaml`. + +**This is gross but stable.** The kernel matches what 24.04 ships, and +the `apt-mark hold` keeps it that way. /etc/os-release lying about the +OS only affects userland callers that key off it — none of our +deployed services do (we verified by grepping the cluster). + +**Trigger to un-mitigate.** Periodically check for ubuntu26.04 driver +tags. Once they appear: + + docker run --rm quay.io/skopeo/stable list-tags \ + docker://nvcr.io/nvidia/driver \ + | python3 -c "import json,sys; d=json.load(sys.stdin); \ + print(len([t for t in d['Tags'] if 'ubuntu26.04' in t]))" + +When that returns a non-zero count: + +1. Restore `/etc/os-release` from backup + (`/etc/os-release.bak-pre-spoof-2026-05-17`) on k8s-node1. +2. Remove apt-mark holds for the kernel packages. +3. `apt full-upgrade` to land the latest 26.04 kernel + reboot. +4. Bump the gpu-operator chart pin to the matching version that ships + ubuntu26.04 driver images. Bump `driver.version` in values.yaml to + the current chart default. + +**See also.** `docs/post-mortems/2026-05-17-gpu-driver-ubuntu2604-mismatch.md` +for full incident timeline + the recovery procedure. + +**Beads.** `code-8vr0` (P1, OPEN). diff --git a/docs/plans/2026-04-20-infra-audit-design.md b/docs/plans/2026-04-20-infra-audit-design.md new file mode 100644 index 00000000..bc887bd0 --- /dev/null +++ b/docs/plans/2026-04-20-infra-audit-design.md @@ -0,0 +1,265 @@ +# Infra Audit — 2026-04-20 + +**Status**: Design (post-research, post-challenge) +**Author**: Viktor Barzin (audit run by Claude) +**Scope**: `infra/` Terragrunt stacks + platform services (`claude-agent-service`, `claude-memory-mcp`, `beadboard`, `broker-sync`) +**Goals**: Reliability · Declarative-first · Reduced maintenance overhead · Maintained scalability +**Method**: 5 parallel research agents (R1 Reliability, R2 Declarative, R3 Maintenance, R4 Scalability, R5 Security) → 91 raw findings → 2 independent challengers → filtered/corrected/ranked backlog below. + +## Context + +The home-lab has grown into a mature stack (105 Tier-1 Terragrunt stacks + 6 Tier-0 SOPS, CNPG, Vault+ESO, Kyverno, Traefik, Authentik, CrowdSec, Woodpecker CI, Redis-Sentinel, MySQL-standalone, Proxmox-NFS). Recent work has been consolidation: MySQL InnoDB-Cluster → standalone (2026-04-16), Redis Phase 7 refactor (2026-04-19), NFS fsid=0 SEV1 post-mortem (2026-04-14), Authentik outpost /dev/shm fix (2026-04-18). This audit surveys everywhere that remains — what's brittle, what's manual, what's dark, what hasn't caught up to recent decisions — and ranks fixes by impact and by operator fatigue. + +## Corrections up-front (challenger round) + +Before reading the backlog, these findings from the research phase are **dropped, corrected, or reframed** — challengers spot-checked live state and proved them wrong, already-solved, or intentional-by-design. Being honest about this is the point of the challenge round: + +| Finding as stated | Actual state | Action | +|---|---|---| +| R4#1: Worker nodes 86-91% memory saturation | Live `kubectl top nodes`: 44-51% across k8s-node{1-4} | **DROPPED** — bad metric pull | +| R4#2: Frigate CPU unbounded (1.5 CPU request, no limit) | Cluster policy is **all CPU limits removed** to avoid CFS throttling (`infra/.claude/CLAUDE.md` → Resource Management) | **DROPPED** — by design | +| R4#7: Redis no `maxmemory-policy` | `infra/stacks/redis/modules/redis/main.tf:254` sets `maxmemory-policy allkeys-lru` (Phase 7, 2026-04-19) | **DROPPED** — already solved | +| R2#1: 307 Kyverno lifecycle markers is a drift risk | Markers are the **canonical discoverability tag** — `ignore_changes` only accepts static attribute paths, snippet convention is the only viable path; reframe as *"markers are fine, missing markers are the risk"* | **REFRAMED** | +| R2#3: 140 `ignore_changes` blocks | Actual: **310** across `.tf` files (2.2× off) | **CORRECTED** | +| R3#10: 65 CronJobs | Actual: 59 (10% off) | **CORRECTED** | +| R1#1: 47 deployments missing probes | Actual: **115 missing at least one probe; 103 missing both** | **CORRECTED (much worse than reported)** | +| R1#9: MySQL standalone no HA/PDB | Intentional post-2026-04-16 migration from InnoDB Cluster. Backup + restore matter; HA is explicit deferred. | **REFRAMED** — split into HA (deferred) / backup-restore (open) / connection pool (open) | +| R1#10: PDB gaps include Traefik, Authentik | Traefik & Authentik PDBs `minAvailable=2` exist (CLAUDE.md). The real gaps are **CrowdSec LAPI, Calico-apiserver, ESO webhook, Woodpecker-server** | **CORRECTED (list pruned)** | +| R5#2: 4 Kyverno security policies in Audit | **All 16 ClusterPolicies are in Audit** — zero in Enforce. | **CORRECTED (worse)** | + +--- + +## Executive summary — top 5 cross-cutting themes + +These are the themes that survive the challenge round and hit ≥2 concerns. Each headline is a 1-line hook; deep-dives below. + +1. **Declarative escape hatches (NFS exports, master-node file provisioners, null_resource initializers)** — `/etc/exports` is not in Terraform, which is the **root cause of the 2026-04-14 SEV1**; 6 null_resources + 3 SSH file provisioners still orchestrate critical state. *Hits R2 + R1 + R3.* +2. **Observability has blind spots where pain would actually come from** — no OOMKill alert routing, no NFS capacity monitor, no GPU utilization dashboard, no ESO refresh-lag alert, no CronJob success-rate summary. Alerts exist but they don't cover the operator's real failure modes. *Hits R1 + R3 + R4.* +3. **Supply-chain hygiene: image pinning + Renovate + admission signing** — 84 `:latest` tags in production TF, zero Renovate/Dependabot across 18 repos (~15 hr/mo toil by estimate), no cosign/trivy on push. Single theme unifies security posture, maintenance toil, and determinism. *Hits R3 + R5.* +4. **Reliability-probes & graceful shutdown are genuinely uneven** — 115 deployments missing at least one probe (incl. 103 missing both), 50+ Recreate deployments with no `terminationGracePeriodSeconds`/`preStop`. This is the quietly-largest reliability debt. *Hits R1 + R3 (pager toil).* +5. **Backup coverage is uneven: 30+ PVCs lack app-level CronJobs** — Proxmox host snapshots cover the disk, but Forgejo (!), Affine, Paperless, Hackmd, Matrix, Owntracks have no app-aware dumps. Restore granularity is file-level, not entity-level. *Hits R1 + R5 (compliance) + R3 (restore rehearsal toil).* + +Honourable mentions that didn't make top 5 but sit just below: Kyverno audit→enforce transition (security), ESO refresh-lag alert (secrets reliability), Vault hardening (audit log offsite, root-token K8s-secret scope), Cloudflared tunnel-token SPOF (not replica SPOF — those are 3), Dolt PVC sizing + backup. + +--- + +## Scoring method + +Two parallel rankings — scan both. + +**Rank A — Impact × Reversibility (the original formula)** +`score = Impact × (6 - Effort) × (6 - Risk)` — each dimension 1-5. + +**Rank B — Operator fatigue weight** +`score = Impact × (6 - Effort) × FatigueWeight` where `FatigueWeight = 3` if the finding introduces *daily/weekly manual toil* and `1` otherwise. This re-ranks by how much pain the unfixed state causes per month. + +Both rankings below. When they agree, that's the clear signal. When they diverge, that's where Rank B (fatigue) wins — Viktor has stated operator fatigue dominates abstract risk for a solo-operator lab. + +--- + +## Ranked backlog (filtered, deduplicated, corrected) + +Counts below reflect **post-challenge corrected numbers**. Every row has a reference verified either by a spot-check (file:line) or a live cluster command. + +| ID | Title | Concerns | Impact | Effort | Risk | Rank A | Rank B | Refs | +|---|---|---|---:|---:|---:|---:|---:|---| +| F01 | NFS `/etc/exports` not in Terraform (SEV1 root cause) | R2+R1 | 5 | 3 | 2 | **60** | **45** | `infra/scripts/pve-nfs-exports`, PM 2026-04-14 | +| F02 | 115 deployments missing probes (103 missing both) | R1+R3 | 5 | 3 | 2 | **60** | **45** | `kubectl get deploy -A -o json` | +| F03 | Zero Renovate/Dependabot across 18 repos | R3+R5 | 4 | 2 | 1 | **80** | **48** | `find /home/wizard/code -name ".renovaterc*"` → 0 results | +| F04 | 84 `:latest` image tags in production TF | R3+R5+R4 | 4 | 2 | 2 | **64** | **48** | `grep -rn ':latest' infra/stacks` | +| F05 | No OOMKill / unschedulable / node-CPU alert | R1+R4+R3 | 5 | 3 | 1 | **75** | **45** | Grep Prometheus rules — no `OOMKilling` rule present | +| F06 | 6 `null_resource` DB initializers in `dbaas` stack | R2 | 4 | 3 | 3 | **36** | **36** | `grep -n null_resource infra/stacks/dbaas` | +| F07 | 3 SSH+file provisioners on k8s-master (audit, OIDC, etcd) | R2 | 4 | 3 | 3 | **36** | **36** | `stacks/platform/modules/rbac/apiserver-oidc.tf` | +| F08 | ESO refresh-lag alert missing (52 ExternalSecrets) | R1+R5+R3 | 4 | 2 | 1 | **80** | **48** | `stacks/external-secrets/` — no PrometheusRule for refresh lag | +| F09 | 30+ PVCs without app-level backup CronJobs | R1+R5 | 4 | 3 | 2 | **48** | **36** | Affine, Forgejo, Hackmd, Matrix, Owntracks, Paperless (no `*-backup` CJ) | +| F10 | Cloudflared tunnel-token SPOF (replicas OK, token shared) | R1+R5 | 3 | 4 | 2 | **24** | **8** | `stacks/cloudflared/` single tunnel credential | +| F11 | MySQL restore never rehearsed end-to-end | R1+R4+R3 | 4 | 2 | 2 | **64** | **48** | No `mysql-restore-drill` CJ; runbook untested post-migration | +| F12 | Kyverno policies all 16 in Audit — **sequence carefully** | R2+R5 | 4 | 3 | **4** | **24** | **24** | `kubectl get clusterpolicy` | +| F13 | 97 RollingUpdate deployments lack explicit surge bounds | R1 | 2 | 2 | 2 | **32** | **12** | TF defaults inherit from Helm/k8s (25%/25%) | +| F14 | CronJob success-rate dashboard + alert rollup missing | R3+R4 | 3 | 2 | 1 | **60** | **36** | `CronJobTooOld` rule — partial; no 24h rollup | +| F15 | Authentik outpost /dev/shm fix applied via Helm API only | R1+R5 | 3 | 2 | 2 | **48** | **48** | Not in TF — upgrade-reversion risk | +| F16 | Dolt (beads DB) no backup CronJob — 2Gi PVC near full | R1+R4 | 4 | 2 | 2 | **64** | **32** | `stacks/beads/` — no `dolt-backup` CJ | +| F17 | Vault StatefulSet `updateStrategy=OnDelete` (manual roll) | R1+R3 | 2 | 2 | 3 | **24** | **24** | `kubectl get sts -n vault -o yaml` | +| F18 | No NetworkPolicies cluster-wide | R4+R5 | 4 | **5** | **4** | **8** | **8** | `kubectl get netpol -A` → 0-2 | +| F19 | RBAC `oidc-power-user` has cluster-wide secrets r/w | R5 | 4 | 3 | 3 | **36** | **12** | `stacks/platform/modules/rbac/` | +| F20 | No image supply-chain verification (cosign, trivy on push) | R5 | 4 | 4 | 3 | **24** | **8** | No admission controller for signatures | +| F21 | Vault audit log offsite backup not configured | R5+R1 | 3 | 2 | 1 | **60** | **36** | `stacks/vault/` — no `audit-log-sync` CJ | +| F22 | Claude-agent, beadboard, broker-sync singletons | R1 | 2 | 2 | 2 | **32** | **12** | `kubectl get deploy -n claude-agent,beadboard,broker-sync` | +| F23 | 50+ Recreate deployments lack graceful-shutdown hooks | R1+R3 | 3 | 3 | 2 | **36** | **36** | `grep -L terminationGracePeriodSeconds stacks/**` | +| F24 | CoreDNS scaled via `kubectl scale` not TF | R2 | 3 | 2 | 2 | **48** | **32** | Command in runbook; no TF resource for replicas | +| F25 | GPU / inference-latency SLO unmonitored | R4+R5 | 3 | 3 | 2 | **36** | **36** | No dcgm dashboard; Frigate liveness checks only | +| F26 | Prometheus TSDB 200Gi — retention untracked | R4 | 2 | 2 | 1 | **40** | **20** | `stacks/monitoring/` | +| F27 | Pod Security Standards labels unset on all namespaces | R5 | 3 | 2 | 3 | **36** | **12** | `kubectl get ns -o json \| jq '.items[].metadata.labels'` | +| F28 | Authentik worker VPA upperBound 2.3× actual request | R4 | 2 | 2 | 2 | **32** | **20** | Goldilocks dashboard | +| F29 | 9 DB rotation targets, no post-rotation verification loop | R5+R3 | 3 | 2 | 2 | **48** | **36** | Vault DB engine every 7d; no auto-verify | +| F30 | Tier-0 SOPS workflow 7-step vs 3-step Tier-1 | R3 | 2 | 2 | 1 | **40** | **20** | `scripts/state-sync` — manual decrypt/encrypt/commit | + +**Rank A leaders (top 8)**: F03, F08, F05, F11, F04, F16, F01, F02 — "big cluster wins, cheap to try" +**Rank B leaders (top 8)**: F03, F04, F08, F11, F15, F01, F02, F05 — "what's paining you weekly" + +F03 (Renovate), F08 (ESO refresh alert), F11 (MySQL restore drill) and F01 (NFS in TF) lead in **both** rankings → these are the clear "do first" candidates. + +--- + +## Per-concern deep dives + +### R1 — Reliability (18 raw → 11 real after challenge) + +Filtered: dropped R1#1/9/10 (incorrect numbers, intentional choices). What actually matters: + +- **Probes (F02)** — 115 deployments missing at least one probe; 103 missing both. The corrected count is 2.4× the original claim. Worst offenders are batch workloads (CronJob-spawned) that legitimately skip probes — but long-lived ones (Affine, Hackmd, mailserver sidecars) genuinely need them. Triage: filter by `spec.replicas ≥ 1` and `containers[].command != ["/bin/sh","-c"]`-style short-runners, then add readiness+liveness one-by-one. +- **Cloudflared tunnel token SPOF (F10)** — Replicas are 3 (per CLAUDE.md), so the agent finding "SPOF" framed as replicas is wrong. The real SPOF is the *tunnel credential*. Secondary tunnel with weighted Cloudflare DNS records is the honest fix — medium effort, low urgency unless tunnel CA rolls keys. +- **PDB gaps (F13-like, excluded from table)** — After challenger correction, gaps are: CrowdSec LAPI (3 replicas, no PDB), ESO webhook+controller, Woodpecker-server. Not urgent — drain-test with `kubectl drain --dry-run` shows no current issue. +- **App-level backups (F09)** — Proxmox host captures the PVC contents nightly via LVM snapshot + rsync with `--link-dest` weekly versioning, so file-level recovery is covered. But for databases inside PVCs (e.g. Affine's Postgres in-pod, Paperless' SQLite), app-aware dumps give transactional consistency. Audit pass: enumerate every PVC without a sibling `*-backup` CronJob, add one for the ones that host embedded DBs. +- **MySQL restore drill (F11)** — Migrated 4 days ago. Runbook exists. End-to-end restore (dump → new DB → connect an app → verify) hasn't been rehearsed. SEV1 risk if a dump has been silently broken since migration. +- **Vault update strategy (F17)** — `OnDelete` means helm upgrade leaves pods untouched; must manually `kubectl delete pod` to restart. Low impact (infrequent) but procedural toil. +- **Dolt PVC near-full + no backup (F16)** — `bd list --status in_progress` runs against this DB; it's load-bearing for cross-session task state. Grow the PVC (resize annotation) + add dolt dump CronJob. + +### R2 — Declarative Coverage & Drift (16 raw → 8 real) + +Filtered: dropped R2#1 (Kyverno markers are by-design), corrected R2#3 to 310. + +- **NFS exports (F01)** — The file is git-managed at `infra/scripts/pve-nfs-exports` but deployed via `scp + exportfs -ra`, not Terraform. This is the exact path that caused the 2026-04-14 SEV1 (fsid=0 on wrong exports line). Options: (a) `null_resource` with `local-exec scp + remote-exec exportfs -ra` triggered on hash of content (partial — SSH dep); (b) new module `pve_host_config` that templates and SCPs multiple PVE-host artifacts with checksum verification. (b) is the cleaner long-term fix. +- **Null-resource initializers (F06)** — 6 in `dbaas` (MySQL users, CNPG cluster, TF-state role, payslip DB, job-hunter DB). Some are genuinely unavoidable (bootstrapping DB before the DB exists); others could use `postgresql_grant` / `mysql_user` providers. +- **SSH file provisioners on k8s-master (F07)** — `apiserver-oidc.tf`, `audit-policy.tf`, `etcd tuning`. One-way sync, no drift detection. Proposed quick wins (per `2026-02-22-node-drift-quick-wins-design.md` already exists). Continue/finish the plan. +- **CoreDNS scaling manual (F24)** — Current runbook uses `kubectl scale`/`set env`/`set affinity`. Drift-prone; convert to `kubernetes_deployment` TF resource overriding the Helm chart's scale/affinity fields. +- **MySQL InnoDB Cluster + operator TF resources still present** — Phase 4 cleanup. Low urgency, but removing reduces cognitive load on anyone reading `stacks/dbaas/`. +- **Technitium readiness-gate null_resource with `timestamp()` trigger** — Runs every apply, 3-6 min wall time. Replace with a real health-check on `terraform_data` with `triggers_replace = { checksum = sha256(config) }`. +- **GPU node taints + Proxmox CSI labels via null_resource kubectl** — No drift detection. Fix is in the `2026-02-22-node-drift-quick-wins-design.md` plan. + +### R3 — Maintenance overhead (18 raw → 10 real) + +- **Renovate (F03)** — The single highest-leverage maintenance fix. 18 repos × ~0.8 hrs/month manual version sweep = real time. Add `.github/renovate.json` (grouping rules for Terraform providers, K8s provider, Docker images) + auto-merge patch-level. Start with `infra/` only; expand after 2 weeks. +- **Image pinning (F04)** — 84 `:latest` tags in production TF. Root CLAUDE.md still says "use 8-char git SHA tags" but that's not enforced. Admission control via Kyverno `require-trusted-registries` is in Audit today — add a sibling policy `forbid-latest-tag` also in Audit. Separate from F03 because pin-to-SHA + Renovate is a synergistic pair. +- **MySQL restore drill (F11)** — tracked under R1 for impact; also a maintenance item because the restore *procedure* has not been test-updated since migration. +- **CronJob alert rollup (F14)** — 59 CronJobs; "which were healthy last 24h" takes ad-hoc `kubectl get jobs --sort-by` scrolling. Add a Grafana panel with `kube_cronjob_status_last_successful_time < now - 2×schedule` summary. +- **Graceful-shutdown toil (F23)** — 50+ Recreate deployments without `terminationGracePeriodSeconds` or `preStop`. Noisy pager hits after node drain. One-off sweep: add a 30s `terminationGracePeriodSeconds` default via Kyverno mutation rule. +- **Tier-0 SOPS workflow (F30)** — 7-step decrypt/edit/encrypt/commit vs Tier-1's 3-step. Combined `tg` wrapper flag `--edit ` that auto-decrypts → EDITOR → auto-encrypts → commit in one command. Moderate win; low risk. +- **Stale `in_progress` beads** — 7 stale tasks in `bd list --status in_progress` at audit start. Session-end hook checks this; 3-5 days without notes is the signal. CLAUDE.md covers the rule — it's followed-sometimes, not enforced. +- **Runbook staleness** — no `last_reviewed` frontmatter on runbook MDs; trivial to add. One-off sweep then keep it honest. +- **CI/CD template unification** — "GHA build → Woodpecker deploy" is the documented pattern for 10 repos; rest still on Woodpecker-only. Track as follow-ups per repo in `bd`. +- **Kyverno DNS-config boilerplate 307 markers** — Not a problem (see correction at top). Do add a lint rule in CI that flags any `kubernetes_deployment` without `# KYVERNO_LIFECYCLE_V1` marker; that's the real drift risk. + +### R4 — Scalability (18 raw → 9 real) + +Filtered: dropped R4#1 (metric mispull), R4#2 (CPU-limit policy), R4#7 (Phase 7 solved). + +- **CNPG memory headroom** — Currently 2Gi limit. Top-line metric at quiet time; add a `ContainerNearOOM > 85%` rule that watches CNPG specifically (general rule exists; CNPG is Tier 0 so deserves explicit binding). +- **HPA cluster-wide: zero** — Every stateless service is 1:1. Not urgent at current node-CPU 8-31%, but one big feature (Immich re-index, Authentik load spike) tips the balance. Pilot: HPA on Traefik (CPU-driven), observe, expand. +- **Redis no HPA + HAProxy singleton** — Wire Sentinel into direct client access (Phase 8 of Redis refactor, per R1#11 of raw findings). Currently all 17 consumers go via HAProxy — the single-point bypass was deliberate (simpler client config), but the HAProxy is now the SPOF Sentinel was meant to prevent. Worth a plan doc (`plans/2026-MM-DD-redis-phase8-sentinel-clients.md`). +- **PgBouncer pool sizing unknown** — Authentik has 3 pods, each opening N connections. At load spikes (big org sync), pool exhaustion. Short-term: `pgbouncer_show_pools` metric + alert at 80% util. Longer-term: pool-size tuning based on observed wait times. +- **Prometheus TSDB (F26)** — 200Gi retention unquantified. Risk: disk fills → scrape gaps → audit blind. Add `kubelet_volume_stats_used_bytes{persistentvolumeclaim="prometheus-server"} > 0.85 * capacity` alert. +- **NFS capacity not monitored** — PVE host has 1TB HDD LV. No `node_filesystem_avail_bytes` scrape from PVE host (it's outside the cluster). Install node_exporter on PVE host; scrape via Prometheus federation or remote_write. +- **VPA quarterly review unscheduled** — Goldilocks is in `Initial` mode (not Auto, by design). Review is manual per quarter. Calendar event + runbook link. +- **Registry single instance** — Registry outage = no pod restarts. Post-mortem 2026-04-19 documented a container-engine pin; replica count still 1. Consider HA registry backed by S3-compat store (MinIO in-cluster) for the second replica — but low urgency given probe CJ monitors integrity every 15m. +- **No ResourceQuota utilization alert** — Quota exhaustion invisible until a pod refuses to schedule. `kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.85` rule. + +### R5 — Security & Secrets (21 raw → 13 real) + +- **Vault `vault-unseal-key` K8s Secret (F21-related)** — Challenger A said it wasn't present; it is (`kubectl get secret -n vault`). Used by auto-unseal. RBAC on the secret should restrict to `vault-server` SA only. Audit the `role` + `rolebinding` in `stacks/vault/`. +- **Vault audit log offsite (F21)** — Rotated logs not synced to NFS backup. Add a `vault-audit-log-sync` CronJob or append the audit log path to `nfs-change-tracker` inotify list (zero-Terraform change if the latter). +- **Kyverno audit → enforce (F12) — sequence carefully** — All 16 policies are in Audit today. Naive switch to Enforce will block legitimate workloads (Loki, Frigate, nvidia-device-plugin, wireguard have privileged/host-ns requirements — all documented). Plan: (a) generate `Kyverno PolicyException` CRs for known-good workloads first; (b) enforce one policy at a time, 1-week observation; (c) start with `require-trusted-registries` (least breakage risk). **DANGEROUS TO EXECUTE NAIVELY — don't batch.** +- **No NetworkPolicies (F18)** — Challenger correctly flagged the effort (5) and risk (4): wrong NetworkPolicy stops Authentik from reaching its DB in minutes. Approach: allow-list namespace-wide first (e.g. `authentik` ns can reach `dbaas` on 5432), expand over a month. Single biggest latent security improvement but needs runway. +- **RBAC oidc-power-user secrets r/w cluster-wide (F19)** — Scope down: list which Authentik groups get this binding, remove `secrets:*` from the cluster role, add namespace-scoped RoleBindings where needed. Medium effort, high leverage. +- **Image supply chain (F20)** — cosign verification + admission controller is the mature path. Trivy-on-push fits in GHA workflows. Both unblocked after F04 (pinning). +- **`:latest` tags (overlap F04)** — Security aspect: signed-image admission requires stable refs. +- **Privileged containers** — Loki, WireGuard, NVIDIA, Frigate known-exceptions. Document the exceptions inline (comment block on the TF resource) so future maintainers don't accidentally "fix" them. +- **Git history plaintext secrets** — Challenger B flagged unverified. One way to verify cheaply: `git secrets --scan-history`. Add it as a pre-audit one-off. +- **CrowdSec Metabase disabled, no Prometheus exporter** — R5#18. Enable the Prometheus exporter (no Metabase) for attack-pattern visibility; very cheap. +- **cert-manager evaluation paused** — Documented pause; TLS rotation relies on Cloudflare wildcard. Confirm no local `Ingress` uses a self-managed cert that could expire silently. `kubectl get cert -A` → expect 0. +- **Pod Security Standards (F27)** — Label every namespace `pod-security.kubernetes.io/enforce=restricted` (or baseline). Known-exception namespaces get explicit downgrades. Medium effort, paid back by making future admission decisions uniform. +- **CrowdSec LAPI quorum** — 3 replicas but quorum/consensus behavior undocumented. One-page runbook: what happens if 1, 2, or 3 LAPI pods die. +- **Authentik outpost fix (F15)** — Applied via API, not TF. Next Helm upgrade reverts. Add the `/dev/shm` emptyDir to `stacks/authentik/values.yaml` templatefile. + +--- + +## Dangerous-to-execute (handle with care) + +Flagged by challengers; each needs a gradual rollout plan, not a single commit. + +1. **F12 — Kyverno Audit → Enforce en masse**. Write `PolicyException` CRs for known-safe workloads first. One policy per week. Observe. +2. **F18 — NetworkPolicies cluster-wide**. Default-deny breaks inter-namespace lookups silently. Namespace-by-namespace rollout, with `kubectl logs -f` tailing the policy-engine events. +3. **PDB additions without drain-test**. New PDB + tight `minAvailable` can deadlock during node cordons. `kubectl drain --dry-run` every new PDB on every node first. +4. **F20 — Signed-image admission**. Must follow F04 (pinning). Un-pinned admission = half the cluster fails to pull. + +## Gaps the agents missed + +From challenger "GAPS" analyses, collated: + +- **Disaster-recovery drill coverage** — backup docs are comprehensive (CLAUDE.md is extensive). End-to-end *restore* rehearsal frequency = never documented. Track per-component: MySQL, PostgreSQL/CNPG, Vault, etcd, NFS, registry blobs. +- **Service mesh evaluation** — Never formally evaluated (Istio, Linkerd, Cilium-in-mesh-mode). Could subsume NetworkPolicy effort + mTLS + observability. Worth a design doc even if answer is "no, too much complexity for the gain." +- **Chaos engineering coverage** — Zero. No pod-kill cron, no node-failure drill. Low urgency given maturity, but would validate F02 probe quality and F23 graceful-shutdown coverage cheaply. +- **Operator onboarding friction** — Nobody else in the "lab team" but Emo exists in `claude-agent-service`. If Emo needs to take over a component for a week, what's the runbook? +- **Alert noise / fatigue rate** — No finding measured how many alerts actually page vs. auto-resolve. `alertmanager_notifications_total` by receiver is the metric; needs a Grafana panel. +- **Secrets-in-image-layers** — Docker images built locally may contain secrets from build env. `trivy image --scanners secret` on registry images is a one-off audit. +- **Runbook → post-mortem → runbook-update loop** — Post-mortem 2026-04-14 produced runbook updates; no general tracker that every incident produces a runbook change. + +## Alternative framings (from challengers, preserved for future reference) + +- **Split "MySQL singleton" into 3 items** (HA / backup / pool). Accepted — see R1 and R4 treatment. +- **6th concern: Observability & Pager Fatigue** — Considered; the themes already hit R1+R3+R4 under Theme 2 of the executive summary. Keeping 5 concerns but carving "Observability gaps" as a theme, not a new research axis. +- **One-thing-this-weekend**: Challenger B nominated *NFS in Terraform*, Challenger A nominated *`:latest` tag sweep*. F01 wins on SEV1 prevention; F04 wins on toil. Both valid. Pick by energy level: F01 is 1 deliberate session; F04 is low-cognition grep-replace. +- **Re-rank by operator fatigue (Rank B) always**. Partially accepted — presented side-by-side in the table. + +--- + +## Recommended next moves + +Ordered for a solo operator balancing SEV-prevention, fatigue reduction, and preserved energy for larger work: + +**Week 1 (SEV-prevention + quick-wins, low cognitive load):** +- F01: NFS exports into a `pve_host_config` Terraform module (one deliberate session) +- F04: Sweep `:latest` tags, add Kyverno `forbid-latest-tag` in Audit +- F08: ESO refresh-lag PrometheusRule +- F05: OOMKill / Unschedulable / Node-CPU PrometheusRule + +**Week 2 (fatigue reduction):** +- F03: Renovate in `infra/` only (narrow pilot) +- F14: CronJob success-rate Grafana panel + alert rollup +- F16: Dolt backup CronJob + PVC grow +- F11: First MySQL restore drill (scheduled, documented) + +**Month 2 (durable fixes, gradual):** +- F06/F07: Replace null_resources + SSH provisioners with native TF resources, one at a time +- F02: Probe sweep — add readiness+liveness to the 20 long-lived deployments first +- F12: Kyverno Enforce transition, one policy per week +- F15: Authentik outpost /dev/shm into values.yaml + +**Month 3+ (structural):** +- F18: NetworkPolicies — namespace-by-namespace +- F19: RBAC scope-down +- F20: Signed-image admission +- Service-mesh evaluation (design doc) +- Restore-drill calendar for every backup target + +No beads tasks auto-filed by this audit — user decides which findings merit `bd create`. + +--- + +## Appendix — verification references (spot-checked) + +Every numeric claim in the backlog was confirmed by one of these commands at audit time (2026-04-20): + +| Claim | Command | Result | +|---|---|---| +| Node memory 44-51% | `kubectl top nodes --no-headers` | k8s-node1: 45%, node2: 51%, node3: 49%, node4: 44%, master: 17% | +| 115 deploys missing ≥1 probe | `kubectl get deploy -A -o json \| jq '[.items[] \| select(.spec.template.spec.containers[0].readinessProbe == null or .spec.template.spec.containers[0].livenessProbe == null)] \| length'` | 115 | +| 103 deploys missing BOTH probes | same, with `and` | 103 | +| 310 ignore_changes blocks | `grep -r "ignore_changes" infra --include=*.tf --include=*.hcl \| wc -l` | 310 | +| 59 CronJobs | `kubectl get cronjobs -A --no-headers \| wc -l` | 59 | +| All 16 Kyverno ClusterPolicies in Audit | `kubectl get clusterpolicy -o jsonpath='...validationFailureAction...'` | 16/16 Audit, 0 Enforce | +| Redis `maxmemory-policy allkeys-lru` | `grep -n maxmemory-policy infra/stacks/redis` | `modules/redis/main.tf:254` | +| Zero Renovate configs | `find /home/wizard/code -name '.renovaterc*' -o -name 'renovate.json' \| grep -v node_modules` | 0 | +| Vault `vault-unseal-key` Secret exists | `kubectl get secret -n vault` | present (37d old) | +| NFS `/etc/exports` not in TF | `grep -rn 'fsid=' infra/stacks` | 0 matches; only `infra/scripts/pve-nfs-exports` | +| Frigate CPU limit by policy | `infra/.claude/CLAUDE.md` → "All CPU limits removed cluster-wide" | confirmed | +| MySQL standalone intentional | `infra/.claude/CLAUDE.md` → "migrated from InnoDB Cluster 2026-04-16" | confirmed | + +Other claims (84 `:latest` tags, 52 ExternalSecrets, 30+ PVCs without backup CJs) were surfaced by research agents; challengers spot-checked a subset and agreed the order-of-magnitude holds. Full list in `/home/wizard/.claude/plans/let-s-run-a-thorough-floating-pnueli.md` research digest. + +## Deliverable disposition + +- This document is the audit output. +- No `bd` tasks were created by the audit. Pick findings to ticket after reading. +- When filing: use `F##` as a tag, title with the finding's headline, acceptance criteria from the deep-dive paragraph, priority from Rank B. +- Plan file at `~/.claude/plans/let-s-run-a-thorough-floating-pnueli.md` retains the full 91-finding digest + challenger reports for reference; can be deleted after any follow-up tickets are filed. diff --git a/docs/plans/2026-05-16-auto-upgrade-apps-design.md b/docs/plans/2026-05-16-auto-upgrade-apps-design.md new file mode 100644 index 00000000..78c81870 --- /dev/null +++ b/docs/plans/2026-05-16-auto-upgrade-apps-design.md @@ -0,0 +1,165 @@ +# Auto-Upgrade Apps Design + +**Date**: 2026-05-16 +**Status**: Approved (brainstorm + grill complete; implementation pending) + +## Problem + +Three constraints in tension across the cluster's ~70 services: + +1. **Keep apps at latest.** Most services drift behind upstream; manual bumps don't scale. +2. **Stay Terraform-compatible.** Image refs live in `.tf`; we want declarative source of truth. +3. **Don't let the pull-through cache serve stale `:latest`.** Cache layer must not lie about what `:latest` means today. + +The previous `Diun → n8n → Service Upgrade Agent` flow handled (1) via changelog-reviewed PR bumps for third-party. Self-hosted services have inconsistent CI: 1 of 11 fully wired (CI builds + pushes + rolls out), 6 partially wired (build but no rollout trigger), 4 with no CI at all. Self-hosted services typically pull `forgejo.viktorbarzin.me/viktor/:<8-char-sha>` with Terraform tracking each SHA in `var.image_tag`. + +The user wants to simplify by retiring the changelog-review agent and moving to a pure "latest, always" model, with the cache freshness concern handled at the cache layer (already done — see Architecture §1). + +## Decisions + +| # | Decision | Notes | +|---|----------|-------| +| 1 | **Auto-roll for everything** (no PR-bump gate) | Retires the Service Upgrade Agent; Diun's role narrows to notification only | +| 2 | **Actuator: Keel** ([keel.sh](https://keel.sh)) | Annotation-driven Deployment/StatefulSet/DaemonSet auto-update operator | +| 3 | **Tag scheme: `:latest` where it exists, `:major` where it doesn't, glob+`ignore_changes` last resort** | `keel.sh/policy: force` for `:latest` / `:major`; tag string stays in Terraform | +| 4 | **Opt-out-pure (no skip-list)** | Every workload auto-rolls, including Vault, CNPG, operators, CNI, CSI. User accepts recoverability risk | +| 5 | **Phased rollout (9 phases)** | Low-risk → bootstrap. Catch up to latest as we phase in. Each phase soaks ~1 week | +| 6 | **Per-phase: single combined PR** | Switch image refs to floating tag + add to Kyverno mutate allowlist in same commit | +| 7 | **Diun is the audit source for catch-up** | Existing 6h-poll already reports outdated images; export as worklist per phase | +| 8 | **Polling, hourly** (`@every 1h`) | Not webhooks — single mechanism, all registries supported | +| 9 | **Rollback: `kubectl rollout undo` → pin in Terraform → add `keel.sh/policy: never`** | (c) from grill: immediate undo, durable Terraform pin within ≤1h before next Keel poll | +| 10 | **Implementation: Kyverno cluster-wide mutate** | One `ClusterPolicy` injects Keel annotations; phase boundary = `NamespaceSelector` allowlist | +| 11 | **Keel exempt from its own mutate** | One-line `NamespaceSelector` exclusion. Supervisor self-update has uniquely bad failure mode | +| 12 | **Uniform CI model for all self-hosted** | CI builds + pushes `:latest`, Keel polls and rolls. No per-repo `kubectl set image` step. Retires the GHA-migrated SHA-tag flow (memory id=388) | + +## Architecture + +### 1. Cache freshness — already correct + +Pull-through cache at `10.0.20.10` already splits caching by URL at the nginx layer: + +- `location ~ /v2/.*/blobs/` → `proxy_cache_valid 200 24h` — blobs cached (content-addressed, immutable) +- `location /v2/` (manifests) → pass through, no cache + +Combined with `registry.proxy.ttl: 0` at the docker-registry layer, mutable manifests revalidate against upstream on every pull. **No cache changes needed for this design.** The CLAUDE.md note "Use 8-char git SHA tags — `:latest` causes stale pull-through cache" predates the nginx URL-split fix and should be updated as part of this work. + +### 2. Detection — Keel polls upstream + +Keel runs as a Deployment in its own namespace. Every annotated workload polls its registry hourly (Keel-managed; configurable per workload). On detection of a new digest under the watched tag: + +- `keel.sh/policy: force` (for mutable tags `:latest`, `:16`, `:7`, etc.) → trigger Deployment update (pod template hash changes → restart) +- `keel.sh/policy: minor` / `major` / `glob` (only for images that publish neither `:latest` nor a stable floating tag) → rewrite tag string on the Deployment; requires `lifecycle { ignore_changes = [...image] }` + +### 3. Application — kubelet pull through the cache + +When Keel triggers restart: + +1. kubelet asks the cache (via containerd hosts.toml) for `image:tag` manifest. +2. nginx passes the manifest request through to the docker-registry layer. +3. docker-registry (with `proxy.ttl: 0`) passes through to upstream. +4. Upstream returns current digest. +5. kubelet pulls blobs (mostly cached at nginx layer; new blobs from upstream). +6. New pod runs new image. + +### 4. Annotation injection — Kyverno mutate + +Single `ClusterPolicy` adds these annotations to every Deployment / StatefulSet / DaemonSet in opted-in namespaces: + +```yaml +metadata: + annotations: + keel.sh/policy: force + keel.sh/trigger: poll + keel.sh/pollSchedule: "@every 1h" +``` + +Phase = a `match.any[].resources.namespaces` list. Phase advance = append namespaces. Keel namespace is excluded. + +### 5. Terraform drift handling + +Existing convention (`# KYVERNO_LIFECYCLE_V1` marker) handles `dns_config` injection. We extend with a new marker: + +```hcl +lifecycle { + ignore_changes = [ + spec[0].template[0].spec[0].dns_config, # KYVERNO_LIFECYCLE_V1 + metadata[0].annotations["keel.sh/policy"], + metadata[0].annotations["keel.sh/trigger"], + metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 + ] +} +``` + +This is added per workload as we phase in. Mechanical, grep-able. + +## Phase ordering + +| Phase | Set | Rationale | +|-------|-----|-----------| +| 0 | Foundation (Keel install, Kyverno ClusterPolicy with empty allowlist) | Build infra without enrolling anything | +| 1 | Self-hosted (forgejo-hosted: ~11 services) | We own the code; failures are easy to diagnose | +| 2 | Stateless third-party web apps (linkwarden, postiz, affine, etc.) | No migrations | +| 3 | Exporters, sidecars, utilities | Stateless | +| 4 | Stateful-but-tolerant (Grafana, Prometheus, etc.) | Restart-safe state | +| 5 | State-coupled with migrations (Nextcloud, Forgejo, paperless-ngx, mailserver) | Schema-migration risk | +| 6 | Authentik | Auth outage | +| 7 | Operators (cnpg-operator, ESO, kured, descheduler) | Operator skew | +| 8 | Critical infra (Calico, proxmox-csi, nfs-csi, traefik, metallb) | Node-level outage potential (memory id=390: 26h Calico cascade) | +| 9 | Bootstrap (Vault, CNPG PG cluster, mysql-standalone) | Lose recoverability if broken | + +Per-phase: combined PR → apply (catch-up rolls happen) → soak 1 week → next phase. If a service breaks repeatedly, apply rollback runbook (decision #9) and proceed; re-enroll later or leave pinned. + +## Risk register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| Bad upstream image rolls into prod | High | Service-level outage | Existing alerts (`KubePodCrashLooping`, `KubeletImagePullErrors`, `PodsStuckContainerCreating`); rollback runbook (decision #9) | +| Catch-up rollout overwhelms cache | Medium | ImagePullBackOff cascade (memory id=603) | Rate-limit catch-up to ~5 rollouts/6h via `-target=` per phase; same pacing as retired Service Upgrade Agent (memory id=612) | +| Calico / CSI auto-roll cascades (memory id=390: 26h outage) | Low-Medium | Cluster-level outage | Phase 8 is intentionally late; user opted into the risk; rollback to pinned chart version via Terraform | +| Vault auto-rolls to broken image | Low | Loss of secrets sync; 43 ExternalSecrets stop reconciling | Phase 9 last; Tier 0 SOPS state allows manual recovery | +| CNPG PG cluster auto-rolls to broken image | Low | Tier 1 Terraform state inaccessible; 105 stacks can't apply | Phase 9 last; Tier 0 stack `cnpg` is bootstrap-capable | +| Helm-atomic-trap services (memory id=981) | Medium | `terraform apply` hangs in pending-rollback | Identify `helm_release` services with `atomic = true`; either remove atomic or skip from Keel | +| Keel itself rolls to broken version | Low | Supervisor down; no auto-rolls until manual pin | Decision #11: exempt Keel from mutate | +| Terraform drift after Kyverno injects annotation | High at first | Spurious diffs on every plan | KYVERNO_LIFECYCLE_V2 marker (Architecture §5); applied incrementally per phase | + +## What we give up + +- **Terraform no longer tracks deployed version.** Image refs in `.tf` say `:latest` or `:16`, but the running digest is whatever Keel pulled. To know what's running: `kubectl describe pod`. This is a deliberate trade — the previous SHA-pinned flow tracked version in TF but required N stack edits per deploy. +- **No changelog review before rollout.** The Service Upgrade Agent's risk classification is gone. We rely on alerts to catch breakage post-deploy, not prevent it. +- **CLAUDE.md SHA-tag rule is reversed for this design.** The "use 8-char git SHA tags" rule predates the nginx URL-split fix. New rule (post-rollout): "use floating tags + Keel annotation" — to be updated in both `infra/.claude/CLAUDE.md` and the repo-root `CLAUDE.md` once Phase 1 is stable. + +## Decisions resolved post-grill + +### Q1 — Uniform CI model for ALL self-hosted (resolved 2026-05-16) + +Every self-hosted service moves to the same shape: + +``` +CI (GHA or Woodpecker) → build → push :latest (optionally also : for traceability) → done +Keel → poll registry → detect new digest → trigger rollout +``` + +The 10 GHA-migrated repos (memory id=388: Website, k8s-portal, f1-stream, claude-memory-mcp, apple-health-data, audiblez-web, plotting-book, insta2spotify, audiobook-search, council-complaints) drop the `Woodpecker API → kubectl set image` step. Their `.woodpecker/deploy.yml` and `.woodpecker/build-fallback.yml` files become obsolete; remove during Phase 1. + +Terraform image refs for all self-hosted: `/:latest` (with `${var.image_tag}` defaulting to `"latest"` where the variable exists). + +### Q2 — No-CI self-hosted services (resolution: uniform participation) + +| Service | Action | +|---------|--------| +| `wealthfolio` | Switch Terraform to upstream `wealthfolio/wealthfolio:latest` (DockerHub). No CI needed. | +| `chrome-service` | Verify whether `:v4` is a deliberate pin. If yes → tag stays, add `keel.sh/policy: never` label. If no → switch to `:latest` or `:major`. Investigate during Phase 1 prep. | +| `beadboard` (used by `beads-server`) | Add minimal Woodpecker CI: build on push → push `:latest`. User-owned. | +| `freedify` | Add minimal Woodpecker CI: build on push → push `:latest`. User-owned. | + +## Open questions (still need resolution before Phase 1) + +1. **`helm_release atomic = true` services**: count and identify before Phase 1. Either remove `atomic` (preferred — eliminates the memory id=981 trap), or skip from Kyverno mutate via per-namespace exclusion. Survey command: `grep -rn 'atomic.*true' infra/stacks/ infra/modules/`. + +## Out of scope + +- Cache TTL changes — current config is already correct (nginx URL-split). +- Webhook-based Keel triggers — polling is sufficient for this cadence. +- Replacing Diun — kept for notification visibility into new tags not yet under Keel annotation (during phase rollout). +- Keel approval gate (`keel.sh/approvals: N`) — user wants unattended auto-roll. +- Keel auto-rollback on health-check failure — out of scope for v1; revisit if breakage rate is high. diff --git a/docs/plans/2026-05-16-auto-upgrade-apps-plan.md b/docs/plans/2026-05-16-auto-upgrade-apps-plan.md new file mode 100644 index 00000000..4937b92f --- /dev/null +++ b/docs/plans/2026-05-16-auto-upgrade-apps-plan.md @@ -0,0 +1,322 @@ +# Auto-Upgrade Apps Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Move the cluster from a mix of pinned-SHA / pinned-semver / ad-hoc `:latest` references to a Keel-driven auto-update model where every workload tracks `:latest` (or a chosen `:major` floating tag) and rolls automatically when upstream advances. + +**Architecture:** Kyverno cluster-wide `ClusterPolicy` mutates Deployments / StatefulSets / DaemonSets in opted-in namespaces with Keel annotations (`keel.sh/policy: force`, `keel.sh/trigger: poll`, `keel.sh/pollSchedule: @every 1h`). Keel polls registries, triggers rollout on new digest. kubelet pulls fresh manifest via the nginx URL-split cache (manifests passthrough, blobs cached). Phase advance = expand the `NamespaceSelector` allowlist. + +**Tech Stack:** Keel, Kyverno, Terraform / Terragrunt, Helm, Diun (notification only), nginx, docker/distribution + +**Design doc:** `docs/plans/2026-05-16-auto-upgrade-apps-design.md` + +**Key context:** +- Cache is already correctly configured (nginx URL-split + `proxy.ttl: 0`). No cache changes needed. +- Per-stack `lifecycle.ignore_changes` is already required for the existing `dns_config` Kyverno mutation (KYVERNO_LIFECYCLE_V1 convention). This plan extends it with a V2 marker for Keel annotations. +- Service Upgrade Agent (Diun → n8n → claude bumps tfvars) is retired by this design. n8n workflow + supporting scripts are removed once Phase 9 completes. +- CLAUDE.md "use 8-char git SHA tags" rule is reversed by this design (see Open Q1 in design doc). + +--- + +## Phase 0 — Foundation + +### Task 0.1: Resolve remaining open question + +Q1 and Q2 from the design doc are resolved (uniform `:latest` + Keel model for all self-hosted; per-service plan for no-CI services). + +Remaining open question: + +**Helm-atomic services.** Survey: +```bash +grep -rn 'atomic.*true' /home/wizard/code/infra/stacks/ /home/wizard/code/infra/modules/ +``` + +For each match: either remove `atomic = true` (preferred) or add the namespace to a Kyverno exclusion list. Document inline before Phase 1 proceeds. + +--- + +### Task 0.2: Create the Keel stack + +**Files:** +- Create: `stacks/keel/terragrunt.hcl` +- Create: `stacks/keel/main.tf` +- Create: `stacks/keel/variables.tf` +- Create: `stacks/keel/modules/keel/main.tf` + +**Step 1:** Add `keel` to `terragrunt.hcl` `locals.tier0_stacks` — **NO**. Keel is Tier 1 (depends on Kyverno + Keel image registry access). Keep it in Tier 1. + +**Step 2:** Deploy via Helm chart `keel-hq/keel` (verify current version via context7 before pinning). + +Key Helm values: +- `polling.enabled: true` +- `helmProvider.enabled: false` (we use annotations, not Helm hooks) +- `notifications.slack.enabled: true` with channel `#deployments` (verify channel exists) +- Registry credentials: mount Forgejo PAT from Vault via ExternalSecret (`secret/viktor/forgejo_pull_token`). + +**Step 3:** Verify Keel can authenticate to all five registries (Docker Hub, ghcr, quay, k8s.io, kyverno via the local cache; Forgejo direct). + +**Acceptance:** +- `kubectl -n keel get pod` shows Keel Ready. +- `kubectl -n keel logs deploy/keel | grep registry` shows successful manifest queries. + +--- + +### Task 0.3: Author the Kyverno ClusterPolicy + +**Files:** +- Create: `stacks/kyverno/modules/kyverno/keel-annotations.tf` (or extend `security-policies.tf`) + +ClusterPolicy `inject-keel-annotations`: + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: inject-keel-annotations +spec: + background: true + rules: + - name: add-keel-annotation + match: + any: + - resources: + kinds: [Deployment, StatefulSet, DaemonSet] + namespaces: [] # populated per phase + exclude: + any: + - resources: + namespaces: ["keel"] # decision #11 + - resources: + # Workloads can opt out by setting this label + selector: + matchLabels: + keel.sh/policy: never + mutate: + patchStrategicMerge: + metadata: + annotations: + +(keel.sh/policy): force + +(keel.sh/trigger): poll + +(keel.sh/pollSchedule): "@every 1h" +``` + +- `+()` syntax adds only if not present (preserves per-workload overrides). +- `exclude.selector.matchLabels[keel.sh/policy=never]` is the per-workload escape hatch (used during rollback per decision #9). + +**Step 2:** Initially deploy with `namespaces: []` — policy exists but matches nothing. + +**Acceptance:** +- `kubectl get clusterpolicy inject-keel-annotations` shows Ready. +- `kubectl get deploy -A -o yaml | grep keel.sh/policy` shows no matches yet (empty allowlist). + +--- + +### Task 0.4: Define the KYVERNO_LIFECYCLE_V2 marker convention + +**Files:** +- Modify: `AGENTS.md` — add the V2 snippet to the "Kyverno Drift Suppression" section +- Modify: `.claude/CLAUDE.md` — reference the V2 marker + +Snippet to copy-paste: + +```hcl +lifecycle { + ignore_changes = [ + spec[0].template[0].spec[0].dns_config, # KYVERNO_LIFECYCLE_V1 + metadata[0].annotations["keel.sh/policy"], + metadata[0].annotations["keel.sh/trigger"], + metadata[0].annotations["keel.sh/pollSchedule"], # KYVERNO_LIFECYCLE_V2 + ] +} +``` + +Backfill order: per-phase, only on workloads about to be enrolled. Not a mass sweep. + +--- + +## Phase 1 — Self-hosted (uniform model) + +**Set:** all self-hosted services. Three sub-categories: + +- **Woodpecker-build-only (6):** `claude-agent-service`, `fire-planner`, `job-hunter`, `payslip-ingest`, `recruiter-responder`, `claude-memory-mcp`. +- **GHA-migrated (10, per memory id=388):** Website, k8s-portal, f1-stream, claude-memory-mcp, apple-health-data, audiblez-web, plotting-book, insta2spotify, audiobook-search, council-complaints. (Note: claude-memory-mcp appears in both lists — verify.) +- **No-CI (4, per design Q2):** `wealthfolio` (→ upstream), `chrome-service` (verify pin intent), `beadboard` (add CI), `freedify` (add CI). +- **Already-uniform (1):** `kms-website` — already pushes `:latest` AND SHA; just needs Keel annotation. + +### Task 1.1: Audit current image refs + +```bash +grep -rE 'image\s*=\s*"(forgejo\.viktorbarzin\.me|viktorbarzin)' /home/wizard/code/infra/stacks/ | sort +``` + +Tabulate per service: current tag, CI type (GHA / Woodpecker / none), action needed. + +### Task 1.2: Per-service uniform conversion + +For each Woodpecker-build-only service: +1. Edit Terraform: `local.image_tag` / `var.image_tag` → `"latest"`. +2. Add the KYVERNO_LIFECYCLE_V2 snippet (annotations ignore_changes). +3. Verify `.woodpecker.yml` pushes `:latest` on every build (most do via `auto_tag: true`). + +For each GHA-migrated service: +1. Edit Terraform: switch `image_tag` from SHA reference to `"latest"`. +2. Add the KYVERNO_LIFECYCLE_V2 snippet. +3. Edit `.github/workflows/build-and-deploy.yml`: push `:latest` (in addition to `:<8-char-sha>` for traceability). Remove the Woodpecker API POST step. +4. Delete `.woodpecker/deploy.yml` and `.woodpecker/build-fallback.yml` from each repo (no longer needed). +5. Remove the Woodpecker repo config for these repos from Terraform if applicable. + +For each no-CI service: +- `wealthfolio`: change Terraform image to `wealthfolio/wealthfolio:latest` (upstream DockerHub). Validate the image starts cleanly. +- `chrome-service`: check git blame on the `:v4` pin. If deliberate → label `keel.sh/policy: never`. If accidental → bump to upstream `:latest`. +- `beadboard`, `freedify`: write a minimal `.woodpecker.yml` (single build step pushing to Forgejo `:latest`). Trigger an initial build to populate `:latest`. + +For `kms-website`: only add the Keel annotation; CI changes optional. + +### Task 1.3: Add Phase 1 namespaces to Kyverno allowlist + +Edit `stacks/kyverno/modules/kyverno/keel-annotations.tf`: + +```yaml +namespaces: + - claude-agent-service + - fire-planner + - job-hunter + - payslip-ingest + - recruiter-responder + - claude-memory-mcp + - kms-website + # GHA-migrated set: + - website # or whatever the namespace is named per repo + - k8s-portal + - f1-stream + - apple-health-data + - audiblez-web + - plotting-book + - insta2spotify + - audiobook-search + - council-complaints + # No-CI set: + - beads-server + - chrome-service + - freedify + - wealthfolio +``` + +Verify each namespace name from `kubectl get ns` before locking in (some may differ from the repo name). + +Apply. Watch `kubectl get deploy -n -o yaml | grep keel.sh` confirm annotations injected. Watch Keel logs for first poll cycle picking up the workloads. + +### Task 1.4: Soak + +1 week. Monitor: +- Slack `#deployments` for Keel rollout notifications. +- `KubePodCrashLooping` alerts. +- Manual `kubectl rollout status` on each service after a Keel-triggered rollout. + +If any service breaks repeatedly: apply rollback runbook (decision #9), record the service in a "pin list" with reason, proceed. + +**Acceptance:** +- All 7 services running latest digests within 24h of Phase 1 apply. +- No CrashLooping persisting >1h. +- No more than 2 services pinned-out during the soak week. + +--- + +## Phase 2 — Stateless third-party web apps + +**Set:** linkwarden, postiz, affine, isponsorblocktv, audiobookshelf, freshrss, tandoor, immich (verify it qualifies — has external DB so app-restart is safe), excalidraw, hackmd, send, jsoncrack, sparkyfitness, etc. (~15-20 services — full list from `kubectl get deploy -A` filtered against the phase-1 set + skip-bucket). + +### Task 2.1: Audit current tags via Diun + +```bash +# Diun's REST API or UI exports a "new tags available" report +# Use as the per-service decision source +``` + +For each service, pick floating tag: +- `:latest` if upstream publishes it and it's stable. +- `:` (e.g. `:2`, `:v3`) if `:latest` is unreliable. +- `glob` + `ignore_changes` as last resort. + +### Task 2.2: Catch-up PR + +Single combined PR: +- Per-stack: switch image tag from pinned semver to chosen floating tag (Diun-informed). +- Per-stack: add KYVERNO_LIFECYCLE_V2 snippet. +- Append Phase 2 namespaces to Kyverno allowlist. + +Apply with `-target=` per stack to pace rollouts (≤5 per hour to avoid cache burst — memory id=603). + +### Task 2.3: Soak — 1 week, same monitoring as Phase 1. + +--- + +## Phases 3–9 — same template + +For each phase, repeat: + +1. Define the set (precise namespace list). +2. Audit current tags (Diun + grep). +3. Pick floating tag per service. +4. Combined PR: image-ref change + lifecycle snippet + Kyverno allowlist update. +5. Apply paced (≤5/hr). +6. Soak 1 week. Pin-out any service that breaks repeatedly. + +Set definitions per phase: see design doc Phase Ordering table. + +**Special-handling phases:** + +- **Phase 7 (Operators).** Restart of an operator can confuse its managed CRD reconciles. Use `imagePullPolicy: Always` + readiness check before declaring stable. Investigate cnpg-operator and ESO restart behavior in advance. +- **Phase 8 (Critical infra).** Calico/CSI DaemonSet rollouts impact each node briefly. Verify `updateStrategy.rollingUpdate.maxUnavailable: 1` on every DaemonSet before enrollment. Memory id=390 (26h Calico-cascade outage) is the cautionary tale. +- **Phase 9 (Bootstrap).** Vault, CNPG, mysql-standalone. Coordinate with backup window. Take a fresh snapshot of `/srv/nfs/-backup/` before applying the phase enrollment. + +--- + +## Cleanup tasks (after Phase 9 stable) + +### Task C.1: Retire Service Upgrade Agent + +**Files:** +- Modify: `stacks/n8n/` — remove the Service Upgrade Agent workflow +- Delete: any supporting scripts (`infra/scripts/service-upgrade-*.sh` if they exist) +- Modify: `stacks/diun/` — disable webhook notification to n8n (keep Slack notification for visibility) + +### Task C.2: Update CLAUDE.md files + +- Reverse the "use 8-char git SHA tags" rule in `infra/.claude/CLAUDE.md` "Docker images" line. +- Reverse same in root `/CLAUDE.md` if duplicated. +- Add a new section documenting the Keel model + KYVERNO_LIFECYCLE_V2 snippet. +- Update memory via `mcp__claude_memory__memory_update` on entries 388, 612, 604 (CI/CD architecture, Service Upgrade Agent retirement, cache TTL clarification). + +### Task C.3: Add a runbook + +**Files:** +- Create: `docs/runbooks/keel-rollback.md` + +Document the rollback flow (decision #9): `kubectl rollout undo` → Terraform pin → annotation `keel.sh/policy: never`. + +### Task C.4: Tidy Diun + +Drop image-pin overrides for MySQL, PostgreSQL, Redis from Diun config (no longer needed since they're Keel-managed; the previous skip was for the retired changelog-agent path). + +--- + +## Rollback (whole project) + +If the auto-roll experiment goes badly cluster-wide (multiple cascading failures, repeated outages), revert: + +1. Set Kyverno ClusterPolicy `inject-keel-annotations` to empty `namespaces: []`. +2. Existing annotations remain on workloads, but Keel continues to act on them — so also disable Keel: scale `keel` Deployment to 0. +3. Pin every workload's Terraform image_tag back to its current running digest (use `kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.name}:{.spec.template.spec.containers[0].image}{"\n"}{end}'`). +4. Document failure modes in `post-mortems/2026-XX-XX-keel-rollback.md`. +5. Reconsider opt-in approach for next iteration. + +--- + +## Success criteria + +- All ~70 services running latest within 8 weeks of Phase 0 completion. +- Zero unrolled-back outages caused by Keel. +- ≤5 services on the "pin list" (i.e. ≥93% auto-roll success rate). +- `terragrunt plan` shows no spurious diffs from Kyverno-injected annotations (KYVERNO_LIFECYCLE_V2 working as intended). +- Service Upgrade Agent + supporting infra retired. diff --git a/docs/plans/2026-05-17-agent-presence-plan.md b/docs/plans/2026-05-17-agent-presence-plan.md new file mode 100644 index 00000000..11db9759 --- /dev/null +++ b/docs/plans/2026-05-17-agent-presence-plan.md @@ -0,0 +1,1495 @@ +# Agent Presence Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a shared presence board so Claude Code agent sessions can see which shared infra resources are being actively mutated by other sessions, preventing redundant investigations and overlapping operations. + +**Architecture:** Single-table store on the existing Dolt server (`10.0.20.200:3306`, `beads` DB, new `presence_claims` table). Python single-file CLI (`scripts/presence`) writes/reads claims. Heartbeat-driven TTL — entries expire 15 min after the last heartbeat, so "left unclosed" is structurally impossible. A consolidated UserPromptSubmit hook injects other sessions' active claims into every turn for ambient awareness. CLAUDE.md rule mandates agents claim before mutating shared state. + +**Tech Stack:** Python 3 stdlib + `pymysql`; Dolt (MySQL-compatible) at `10.0.20.200:3306`; Bash hooks; Terraform Kubernetes provider. + +**Coverage of design decisions (locked in grilling):** +- Pure presence/coordination — not work tracking +- Resource-scoped entries (`:`) +- Heartbeat TTL + Stop-hook release +- Agent-driven claim via CLI invoked from agent reasoning per CLAUDE.md rule +- Stored on Dolt `beads` DB, new table +- CLI verbs: `claim`, `heartbeat`, `release`, `list`, `peek` +- UserPromptSubmit hook consolidates beads + presence +- Seed vocab: `node:`, `host:`, `stack:`, `service:`, `db:`, `pvc:`, `infra:` +- Only mutating ops trigger claim +- Co-claim allowed; soft-defer protocol on conflict +- MVP devvm only (no claude-agent-service / Woodpecker) +- Beads coexists with cleaned semantics +- Pure rule + visibility for enforcement (measure first) +- Python single-file CLI at `~/code/scripts/presence` + +--- + +## File Structure + +**New files:** +- `scripts/presence` — Python single-file CLI (~250 lines) +- `scripts/tests/test_presence.py` — pytest unit tests for the CLI +- `scripts/tests/conftest.py` — pytest fixtures (mocked DB) +- `.claude/hooks/presence-session-start.sh` — generates session ID at start +- `.claude/hooks/presence-heartbeat.sh` — throttled heartbeat on PostToolUse +- `.claude/hooks/presence-release.sh` — release on Stop +- `.claude/hooks/agent-state-context.sh` — consolidated beads+presence injector (replaces user-global `beads-task-context.sh`) + +**Modified files:** +- `infra/stacks/beads-server/main.tf` — add `presence_claims` schema init +- `.claude/settings.json` — wire new hooks; swap UserPromptSubmit to consolidated script +- `CLAUDE.md` — add the claim-before-mutate rule, seed vocab, defer protocol + +**Touched-but-untouched (audit only):** +- Stale `in_progress` beads items (close or revert to `open`) + +--- + +## Task 1: Create `presence_claims` table on the Dolt server + +**Files:** +- Modify: `infra/stacks/beads-server/main.tf` — extend the existing `kubernetes_config_map.dolt_init` data block + add a `kubernetes_job` for idempotent table creation on already-running Dolt +- Apply via `scripts/tg apply` from `infra/stacks/beads-server/` + +The `dolt_init` ConfigMap only runs on fresh Dolt PVCs. Since Dolt is already running with the existing PV, the new SQL won't fire from there. The Job is the workaround for live updates and stays idempotent forever. + +- [ ] **Step 1: Add the schema SQL into the existing `dolt_init` ConfigMap** + +In `infra/stacks/beads-server/main.tf`, locate `resource "kubernetes_config_map" "dolt_init"` and add a second data entry: + +```hcl +resource "kubernetes_config_map" "dolt_init" { + metadata { + name = "dolt-init" + namespace = kubernetes_namespace.beads.metadata[0].name + } + data = { + "01-create-beads-user.sql" = <<-EOT + CREATE USER IF NOT EXISTS 'beads'@'%' IDENTIFIED BY ''; + GRANT ALL PRIVILEGES ON *.* TO 'beads'@'%' WITH GRANT OPTION; + EOT + "02-create-presence-table.sql" = <<-EOT + CREATE DATABASE IF NOT EXISTS beads; + USE beads; + CREATE TABLE IF NOT EXISTS presence_claims ( + session_id VARCHAR(128) NOT NULL, + resource_label VARCHAR(255) NOT NULL, + purpose TEXT NOT NULL, + claimed_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), + expires_at DATETIME(3) NOT NULL, + host VARCHAR(128) NOT NULL, + user VARCHAR(64) NOT NULL, + agent_name VARCHAR(64) DEFAULT 'claude-code', + PRIMARY KEY (session_id, resource_label), + INDEX idx_resource (resource_label), + INDEX idx_expires (expires_at) + ); + EOT + } +} +``` + +- [ ] **Step 2: Add an idempotent migration Job that creates the table on the running Dolt** + +Append a new resource block in `infra/stacks/beads-server/main.tf`, after the `kubernetes_deployment.dolt` resource: + +```hcl +resource "kubernetes_job" "presence_schema_migrate" { + metadata { + # name includes a hash of the SQL so a real schema change forces a new Job + name = "presence-schema-${substr(sha256(kubernetes_config_map.dolt_init.data["02-create-presence-table.sql"]), 0, 8)}" + namespace = kubernetes_namespace.beads.metadata[0].name + } + spec { + backoff_limit = 3 + template { + metadata {} + spec { + restart_policy = "OnFailure" + container { + name = "migrate" + image = "mysql:8.4" + command = ["sh", "-c"] + args = [ + "mysql -h dolt.beads-server.svc.cluster.local -P 3306 -u root < /sql/02-create-presence-table.sql" + ] + volume_mount { + name = "sql" + mount_path = "/sql" + } + } + volume { + name = "sql" + config_map { + name = kubernetes_config_map.dolt_init.metadata[0].name + } + } + } + } + } + wait_for_completion = true + timeouts { + create = "5m" + } + depends_on = [kubernetes_deployment.dolt] +} +``` + +- [ ] **Step 3: Apply the Terraform change** + +Run: +```bash +cd /home/wizard/code/infra/stacks/beads-server +../../scripts/tg apply +``` +Expected: `kubernetes_config_map.dolt_init` updated + `kubernetes_job.presence_schema_migrate` created + Job completes successfully. + +- [ ] **Step 4: Verify the table exists** + +Run: +```bash +mysql -h 10.0.20.200 -u beads -e "USE beads; SHOW TABLES LIKE 'presence_claims'; DESCRIBE presence_claims;" +``` +Expected: one row `presence_claims` from `SHOW TABLES`; DESCRIBE shows the 8 columns with the right types. + +- [ ] **Step 5: Commit** + +```bash +git add infra/stacks/beads-server/main.tf +git commit -m "beads-server: add presence_claims table for agent coordination + +Adds the schema for the new agent presence board. Live Dolt is updated +via a hashed-named one-shot Job; the ConfigMap entry preserves fresh-PVC +init. +" +``` + +--- + +## Task 2: Python CLI scaffolding (argparse + DB connection) + +**Files:** +- Create: `scripts/presence` +- Create: `scripts/tests/test_presence.py` +- Create: `scripts/tests/conftest.py` + +- [ ] **Step 1: Write the failing test for `--help`** + +Create `scripts/tests/test_presence.py`: + +```python +import subprocess +from pathlib import Path + +SCRIPT = Path(__file__).parent.parent / "presence" + + +def test_help_lists_subcommands(): + """--help should list all supported subcommands.""" + result = subprocess.run( + [str(SCRIPT), "--help"], capture_output=True, text=True + ) + assert result.returncode == 0 + for verb in ("claim", "heartbeat", "release", "list", "peek"): + assert verb in result.stdout +``` + +- [ ] **Step 2: Run the test, confirm it fails** + +Run: `pytest scripts/tests/test_presence.py::test_help_lists_subcommands -v` +Expected: FAIL — `scripts/presence` doesn't exist yet (FileNotFoundError). + +- [ ] **Step 3: Create the CLI skeleton** + +Create `scripts/presence`: + +```python +#!/usr/bin/env python3 +"""Agent presence board CLI. + +Lets Claude Code agent sessions claim, heartbeat, release, list, and peek at +shared infra resource claims so that two sessions don't unknowingly mutate +the same thing at the same time. + +Reads connection details from env: + PRESENCE_DSN mysql DSN (default: beads@10.0.20.200:3306/beads) + CLAUDE_SESSION_ID session identity (default: read from session-id file) +""" + +from __future__ import annotations + +import argparse +import getpass +import json +import os +import socket +import sys +import uuid +from pathlib import Path + +SESSION_ID_FILE = Path.home() / ".cache" / "claude-presence" / "current.session" +DEFAULT_DSN = "mysql://beads@10.0.20.200:3306/beads" +DEFAULT_TTL_SECONDS = 15 * 60 + + +def get_session_id() -> str: + """Return the current session ID, generating a fallback if missing.""" + env = os.environ.get("CLAUDE_SESSION_ID") + if env: + return env + if SESSION_ID_FILE.exists(): + return SESSION_ID_FILE.read_text().strip() + # Fallback: ephemeral one-shot id (won't be cleaned up by Stop hook) + return f"{getpass.getuser()}@{socket.gethostname().split('.')[0]}@{uuid.uuid4().hex[:8]}" + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + prog="presence", + description="Agent presence board for coordinating shared-infra mutations.", + ) + p.add_argument("--json", action="store_true", help="emit machine-readable output") + sub = p.add_subparsers(dest="verb", required=True) + + c = sub.add_parser("claim", help="claim a resource you're about to mutate") + c.add_argument("label", help="resource label, e.g. node:k8s-node1") + c.add_argument("--purpose", required=True, help="what + why") + c.add_argument("--ttl", type=int, default=DEFAULT_TTL_SECONDS, help="seconds") + + sub.add_parser("heartbeat", help="extend TTL on all my active claims") + + r = sub.add_parser("release", help="release one or all of my claims") + r.add_argument("label", nargs="?", help="resource label; omit with --all-mine") + r.add_argument("--all-mine", action="store_true") + + li = sub.add_parser("list", help="show active claims") + g = li.add_mutually_exclusive_group() + g.add_argument("--mine", action="store_true") + g.add_argument("--all", action="store_true", default=True) + + pe = sub.add_parser("peek", help="show all active claims on a resource") + pe.add_argument("label", help="resource label") + + return p + + +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + # Verbs implemented in later tasks; stub for now so --help works. + print(f"verb={args.verb} not yet implemented", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 4: Make it executable** + +Run: `chmod +x /home/wizard/code/scripts/presence` + +- [ ] **Step 5: Re-run the test, confirm it passes** + +Run: `pytest scripts/tests/test_presence.py::test_help_lists_subcommands -v` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add scripts/presence scripts/tests/test_presence.py +git commit -m "presence: add CLI scaffolding with argparse subcommands" +``` + +--- + +## Task 3: `claim` verb — write to DB, return conflicts + +**Files:** +- Modify: `scripts/presence` +- Modify: `scripts/tests/test_presence.py` +- Create: `scripts/tests/conftest.py` + +- [ ] **Step 1: Add pymysql + fixture scaffolding in conftest** + +Create `scripts/tests/conftest.py`: + +```python +import os +from unittest.mock import MagicMock + +import pytest + + +@pytest.fixture +def fake_db(monkeypatch): + """Mocks pymysql.connect to return a MagicMock cursor we can inspect.""" + conn = MagicMock(name="conn") + cur = MagicMock(name="cur") + conn.cursor.return_value.__enter__.return_value = cur + cur.fetchall.return_value = [] + + import pymysql + monkeypatch.setattr(pymysql, "connect", MagicMock(return_value=conn)) + monkeypatch.setenv("CLAUDE_SESSION_ID", "wizard@devvm@testtest") + return cur +``` + +- [ ] **Step 2: Write the failing test for `claim` happy path** + +Append to `scripts/tests/test_presence.py`: + +```python +import importlib.util +import sys +from pathlib import Path + + +def _load_module(): + spec = importlib.util.spec_from_file_location("presence", SCRIPT) + mod = importlib.util.module_from_spec(spec) + sys.modules["presence"] = mod + spec.loader.exec_module(mod) + return mod + + +def test_claim_inserts_row(fake_db): + presence = _load_module() + rc = presence.main(["claim", "node:k8s-node1", "--purpose", "GPU upgrade"]) + assert rc == 0 + # First call: insert/upsert; second: read existing other-session claims + sql_calls = [c.args[0] for c in fake_db.execute.call_args_list] + assert any("INSERT" in s.upper() or "REPLACE" in s.upper() for s in sql_calls) + assert any("SELECT" in s.upper() for s in sql_calls) + + +def test_claim_reports_other_session_conflict(fake_db, capsys): + presence = _load_module() + # Simulate one OTHER session already holding the label + fake_db.fetchall.return_value = [ + { + "session_id": "emo@laptop@aaaaaaaa", + "purpose": "tcpdump on uplink", + "claimed_at": "2026-05-17 14:10:00.000", + "user": "emo", + "host": "laptop", + } + ] + rc = presence.main(["claim", "node:k8s-node1", "--purpose", "GPU upgrade"]) + out = capsys.readouterr().out + assert rc == 0 + assert "emo@laptop@aaaaaaaa" in out + assert "tcpdump on uplink" in out +``` + +- [ ] **Step 3: Run the tests, confirm they fail** + +Run: `pytest scripts/tests/test_presence.py -v -k claim` +Expected: 2 failures — `claim` verb not implemented (stub prints "not yet implemented"). + +- [ ] **Step 4: Implement `claim` in `scripts/presence`** + +Replace the bottom of `scripts/presence` (the stub `main`) with this. Also add the DB helpers and `_claim` function above `main`: + +```python +import urllib.parse + +try: + import pymysql + import pymysql.cursors +except ImportError: + pymysql = None # graceful: handled in _connect + + +def _connect(): + if pymysql is None: + return None + dsn = os.environ.get("PRESENCE_DSN", DEFAULT_DSN) + u = urllib.parse.urlparse(dsn) + try: + return pymysql.connect( + host=u.hostname, + port=u.port or 3306, + user=u.username or "beads", + password=u.password or "", + database=(u.path.lstrip("/") or "beads"), + cursorclass=pymysql.cursors.DictCursor, + connect_timeout=3, + autocommit=True, + ) + except Exception as e: + print(f"presence: warning: dolt unreachable ({e}); continuing", file=sys.stderr) + return None + + +def _claim(args, session_id: str) -> int: + conn = _connect() + if conn is None: + return 0 # graceful degradation + with conn.cursor() as cur: + cur.execute( + """ + REPLACE INTO presence_claims + (session_id, resource_label, purpose, claimed_at, expires_at, host, user, agent_name) + VALUES + (%s, %s, %s, NOW(3), NOW(3) + INTERVAL %s SECOND, %s, %s, %s) + """, + ( + session_id, + args.label, + args.purpose, + args.ttl, + socket.gethostname().split(".")[0], + getpass.getuser(), + "claude-code", + ), + ) + cur.execute( + """ + SELECT session_id, purpose, claimed_at, user, host + FROM presence_claims + WHERE resource_label = %s + AND session_id != %s + AND expires_at > NOW(3) + ORDER BY claimed_at + """, + (args.label, session_id), + ) + conflicts = cur.fetchall() + if not conflicts: + print(f"presence: claimed {args.label}") + return 0 + print(f"presence: claimed {args.label} -- ALSO CLAIMED BY:") + for c in conflicts: + print(f" - {c['session_id']} ({c['user']}@{c['host']}): {c['purpose']} since {c['claimed_at']}") + print("presence: per CLAUDE.md rule, default is to DEFER — release your claim and confirm with the user.") + return 0 +``` + +Update `main` to dispatch: + +```python +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + session_id = get_session_id() + if args.verb == "claim": + return _claim(args, session_id) + print(f"verb={args.verb} not yet implemented", file=sys.stderr) + return 0 +``` + +- [ ] **Step 5: Run tests, confirm they pass** + +Run: `pytest scripts/tests/test_presence.py -v -k claim` +Expected: both `test_claim_inserts_row` and `test_claim_reports_other_session_conflict` PASS. + +- [ ] **Step 6: Commit** + +```bash +git add scripts/presence scripts/tests/test_presence.py scripts/tests/conftest.py +git commit -m "presence: implement claim verb (upsert + conflict report)" +``` + +--- + +## Task 4: `peek` and `list` verbs (read paths) + +**Files:** +- Modify: `scripts/presence` +- Modify: `scripts/tests/test_presence.py` + +- [ ] **Step 1: Write the failing tests for `peek` and `list`** + +Append to `scripts/tests/test_presence.py`: + +```python +def test_peek_shows_all_active_claims_for_resource(fake_db, capsys): + presence = _load_module() + fake_db.fetchall.return_value = [ + { + "session_id": "wizard@devvm@bbbbbbbb", + "purpose": "GPU driver upgrade", + "claimed_at": "2026-05-17 14:32:00.000", + "expires_at": "2026-05-17 14:47:00.000", + "user": "wizard", + "host": "devvm", + } + ] + rc = presence.main(["peek", "node:k8s-node1"]) + out = capsys.readouterr().out + assert rc == 0 + assert "wizard@devvm@bbbbbbbb" in out + assert "GPU driver upgrade" in out + + +def test_peek_empty_resource_prints_no_active_claim(fake_db, capsys): + presence = _load_module() + fake_db.fetchall.return_value = [] + rc = presence.main(["peek", "node:k8s-node99"]) + out = capsys.readouterr().out + assert rc == 0 + assert "no active claim" in out.lower() + + +def test_list_all_shows_only_active(fake_db, capsys): + presence = _load_module() + fake_db.fetchall.return_value = [ + { + "session_id": "wizard@devvm@xxxxxxxx", + "resource_label": "stack:gpu-operator", + "purpose": "rebuild driver", + "claimed_at": "2026-05-17 14:00:00.000", + "expires_at": "2026-05-17 14:15:00.000", + "user": "wizard", + "host": "devvm", + } + ] + rc = presence.main(["list", "--all"]) + out = capsys.readouterr().out + assert rc == 0 + assert "stack:gpu-operator" in out + assert "wizard@devvm@xxxxxxxx" in out + + +def test_list_mine_filters_to_current_session(fake_db, monkeypatch): + presence = _load_module() + presence.main(["list", "--mine"]) + sql = fake_db.execute.call_args_list[-1].args[0] + assert "session_id" in sql + assert "expires_at" in sql +``` + +- [ ] **Step 2: Run the tests, confirm they fail** + +Run: `pytest scripts/tests/test_presence.py -v -k "peek or list"` +Expected: 4 failures — verbs unimplemented. + +- [ ] **Step 3: Implement `peek` and `list`** + +Add to `scripts/presence`, above `main`: + +```python +def _peek(args, session_id: str) -> int: + conn = _connect() + if conn is None: + return 0 + with conn.cursor() as cur: + cur.execute( + """ + SELECT session_id, purpose, claimed_at, expires_at, user, host + FROM presence_claims + WHERE resource_label = %s + AND expires_at > NOW(3) + ORDER BY claimed_at + """, + (args.label,), + ) + rows = cur.fetchall() + if not rows: + print(f"presence: no active claim on {args.label}") + return 0 + print(f"presence: active claims on {args.label}:") + for r in rows: + marker = " (me)" if r["session_id"] == session_id else "" + print(f" - {r['session_id']}{marker} ({r['user']}@{r['host']}): {r['purpose']} since {r['claimed_at']} (expires {r['expires_at']})") + return 0 + + +def _list(args, session_id: str) -> int: + conn = _connect() + if conn is None: + return 0 + query = """ + SELECT session_id, resource_label, purpose, claimed_at, expires_at, user, host + FROM presence_claims + WHERE expires_at > NOW(3) + """ + params: tuple = () + if args.mine: + query += " AND session_id = %s" + params = (session_id,) + query += " ORDER BY claimed_at" + with conn.cursor() as cur: + cur.execute(query, params) + rows = cur.fetchall() + if not rows: + print("presence: no active claims") + return 0 + for r in rows: + marker = " (me)" if r["session_id"] == session_id else "" + print(f" {r['resource_label']:<32} {r['session_id']}{marker} -- {r['purpose']} ({r['claimed_at']})") + return 0 +``` + +Extend the dispatcher in `main`: + +```python + if args.verb == "claim": + return _claim(args, session_id) + if args.verb == "peek": + return _peek(args, session_id) + if args.verb == "list": + return _list(args, session_id) +``` + +- [ ] **Step 4: Run tests, confirm they pass** + +Run: `pytest scripts/tests/test_presence.py -v -k "peek or list"` +Expected: 4 PASSES. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/presence scripts/tests/test_presence.py +git commit -m "presence: implement peek + list verbs" +``` + +--- + +## Task 5: `heartbeat` and `release` verbs + +**Files:** +- Modify: `scripts/presence` +- Modify: `scripts/tests/test_presence.py` + +- [ ] **Step 1: Write the failing tests** + +Append to `scripts/tests/test_presence.py`: + +```python +def test_heartbeat_extends_all_my_claims(fake_db): + presence = _load_module() + rc = presence.main(["heartbeat"]) + assert rc == 0 + sql = fake_db.execute.call_args_list[-1].args[0] + assert "UPDATE" in sql.upper() + assert "expires_at" in sql + assert "session_id" in sql + + +def test_release_single_label(fake_db): + presence = _load_module() + rc = presence.main(["release", "node:k8s-node1"]) + assert rc == 0 + last = fake_db.execute.call_args_list[-1] + assert "DELETE" in last.args[0].upper() + assert "node:k8s-node1" in last.args[1] + + +def test_release_all_mine(fake_db): + presence = _load_module() + rc = presence.main(["release", "--all-mine"]) + assert rc == 0 + last = fake_db.execute.call_args_list[-1] + assert "DELETE" in last.args[0].upper() + assert "wizard@devvm@testtest" in last.args[1] +``` + +- [ ] **Step 2: Run tests, confirm they fail** + +Run: `pytest scripts/tests/test_presence.py -v -k "heartbeat or release"` +Expected: 3 failures. + +- [ ] **Step 3: Implement `heartbeat` and `release`** + +Add to `scripts/presence`: + +```python +def _heartbeat(args, session_id: str) -> int: + conn = _connect() + if conn is None: + return 0 + with conn.cursor() as cur: + cur.execute( + """ + UPDATE presence_claims + SET expires_at = NOW(3) + INTERVAL %s SECOND + WHERE session_id = %s + AND expires_at > NOW(3) + """, + (DEFAULT_TTL_SECONDS, session_id), + ) + return 0 + + +def _release(args, session_id: str) -> int: + conn = _connect() + if conn is None: + return 0 + with conn.cursor() as cur: + if args.all_mine: + cur.execute("DELETE FROM presence_claims WHERE session_id = %s", (session_id,)) + else: + if not args.label: + print("presence: release requires