trading/docs/plans/2026-05-21-meet-kevin-revival-design.md

# Meet Kevin Revival — Design

## Overview

Revive the trading bot from its current fully-disabled state and add a new pipeline that watches the Meet Kevin (Kevin Paffrath) YouTube channel, extracts captions from every new upload, runs the transcript through Claude Sonnet 4.6 to infer per-ticker recommendations and overall market outlook, and surfaces the results in a ticker-centric UI inside the existing dashboard. Bot integration (feeding the recommendations into `signal_generator`) is explicitly v2; auto-trading stays off in v1.

## Goals

- Subscribe to the Meet Kevin channel and process each new upload end-to-end without human intervention.
- Present sentiment and recommendations keyed on the **stock ticker** (the user's stated primary need), with the video feed as a secondary surface.
- Revive the bot in K8s with the lowest reasonable footprint — keep the scaffolding warm, leave the resource-heavy scrapers and auto-trading off.

## Scope

### In scope (v1)

- New `services/meet_kevin_watcher/` Python service inside `viktor/trading`.
- 5 new Postgres tables via a single Alembic migration.
- `/api/meet-kevin/*` routes added to the existing `api_gateway`.
- 5 new dashboard pages under `/meet-kevin/*`.
- Terraform revival of `infra/stacks/trading-bot/` with 3 services held at `replicas=0` (news-fetcher, sentiment-analyzer, trade-executor) and the new watcher container added to the workers Deployment.
- 1 new pyproject extras group (`meet_kevin`) pinning a current `yt-dlp`.

### Out of scope (v2 or later)

- Signal injection into `signal_generator` (will be wired via a new `kevin:signals` Redis stream + a "kevin" strategy weight).
- Multi-channel support in the UI (schema is multi-channel-ready, surface is single-channel).
- Audio archival or local Whisper fallback (captions-only).
- Slack notification on high-conviction signals.
- Auto-reanalysis when the LLM prompt version changes (manual "Reprocess" only).

## Architecture

```
┌────────────────────────── viktor/trading (revived K8s stack) ──────────────────────────────┐
│                                                                                            │
│  NEW   services/meet_kevin_watcher/   single async service, sequential stages              │
│         ├── rss_poller.py        every 3h, GET YouTube /feeds/videos.xml, dedupe           │
│         ├── caption_extractor.py yt-dlp --write-auto-sub --skip-download                   │
│         ├── llm_analyzer.py      Claude Sonnet 4.6, structured JSON output                 │
│         └── main.py              orchestrates 3 stages, writes Postgres                    │
│                                                                                            │
│  EXTEND                                                                                    │
│         services/api_gateway/    +1 router meet_kevin.py + 1 schemas module                │
│         dashboard/               +1 page group /meet-kevin/* + components                  │
│         shared/models/           +kevin_channel, kevin_video, kevin_transcript,            │
│                                   kevin_analysis, kevin_stock_mention                      │
│         shared/schemas/          +meet_kevin.py                                            │
│         alembic/                 +1 migration with 5 tables                                │
│                                                                                            │
│  DISABLED (container removed from `trading-bot-workers` Pod):                              │
│                           news_fetcher, sentiment_analyzer, trade_executor                 │
│  ACTIVE                   dashboard, api_gateway,                            (frontend Pod)│
│                           signal_generator, learning_engine, market_data,    (workers Pod) │
│                           meet_kevin_watcher (NEW)                           (workers Pod) │
│                                                                                            │
└────────────────────────────────────────────────────────────────────────────────────────────┘
```

### Key architectural choices

- **No Redis streams in v1.** With ~3 videos/day the coordination cost of streams would dwarf the value. The watcher walks each video through all three stages and writes directly to Postgres. A `kevin:signals` stream gets added in v2 when wiring into `signal_generator`.
- **Single long-running async service** (matching the `news_fetcher` pattern). Reuses the same `shared.config.BaseConfig`, `shared.db`, `shared.telemetry` setup as the other services.
- **K8s revival is a single Terraform edit.** No new namespace, no new PVC, no new ingress — transcripts live in Postgres, the embedded YouTube iframe handles video display, no file storage needed.

## Channel

- **Channel**: Meet Kevin (Kevin Paffrath)
- **YouTube channel ID**: `UCUvvj5lwue7PspotMDjk5UA`
- **Feed**: `https://www.youtube.com/feeds/videos.xml?channel_id=UCUvvj5lwue7PspotMDjk5UA`
- **Verified 2026-05-21**: feed parses (Atom), returns 15 most recent uploads, ~2–3 videos/day.
- Multi-channel support is schema-ready; only this one channel is wired in v1.

## Data model

New Postgres tables (single Alembic migration). All `id` columns are `bigserial`; all timestamp columns are `timestamptz NOT NULL DEFAULT now()`.

```
kevin_channels
  id, youtube_channel_id (unique), title,
  poll_enabled (bool, default true), poll_interval_seconds (int, default 10800),
  daily_cost_cap_usd (numeric(8,2), default 5.00),
  last_polled_at (nullable), created_at, updated_at

kevin_videos
  id, channel_id (FK kevin_channels.id), youtube_video_id (unique, indexed),
  title, description, published_at (indexed), duration_seconds (nullable),
  thumbnail_url, status (enum {discovered, captioned, analyzed, failed, skipped}),
  failure_reason (nullable), retry_count (int, default 0),
  processed_at (nullable), created_at, updated_at

kevin_transcripts
  id, video_id (FK kevin_videos.id, unique), source (enum {captions_manual, captions_auto, none}),
  language (varchar(8)), raw_text (text), segments_json (jsonb), word_count, created_at

kevin_analyses                            -- one row per LLM run; supports re-runs
  id, video_id (FK kevin_videos.id, indexed), model, prompt_version,
  market_outlook_direction (enum {bullish, neutral, bearish, mixed}),
  market_outlook_reasoning (text),
  macro_themes_json (jsonb),  -- array of strings
  key_risks_json (jsonb),     -- array of strings
  summary (text), raw_response_json (jsonb),
  prompt_tokens, completion_tokens, cost_usd (numeric(10,4)),
  created_at

kevin_stock_mentions                       -- denormalized per-ticker
  id, video_id (FK kevin_videos.id, indexed),
  analysis_id (FK kevin_analyses.id),
  symbol (varchar(16), indexed), action (enum {buy, sell, hold, watch, avoid}),
  conviction (numeric(4,3)),     -- 0.000 to 1.000
  time_horizon (enum {intraday, days, weeks, months, long_term, unspecified}),
  rationale_quote (text),
  video_timestamp_seconds (int, nullable),  -- deep-link target
  created_at
```

Indexes: `(symbol, created_at desc)` on `kevin_stock_mentions` (ticker timeline queries), `(published_at desc)` on `kevin_videos` (video feed), `(status)` partial index on `kevin_videos WHERE status IN ('discovered', 'captioned')` (pipeline scan).

## Pipeline

Three stages, run sequentially per video by a single async loop.

### Stage 1 — RSS poll (every 3h)

1. For each `kevin_channels.poll_enabled = true` row, GET `/feeds/videos.xml?channel_id=<id>`.
2. Parse the Atom feed with `feedparser`.
3. For each entry, `INSERT … ON CONFLICT (youtube_video_id) DO NOTHING` into `kevin_videos` with `status = 'discovered'`.
4. Update `kevin_channels.last_polled_at`.
5. Failure: log warning, increment metric, retry on next 3h cycle. No DLQ — RSS replays the same 15 entries on every poll.

**First-run backfill**: the feed returns the 15 most recent uploads at once, so the first poll inserts up to 15 `discovered` rows. The pipeline processes them sequentially (~60s each, ~$1.50 in Claude calls total). One-off burst, contained by the daily cost cap.

### Stage 2 — Caption extraction

For each video with `status = 'discovered'`, in `published_at` ascending order:

1. `yt-dlp --write-auto-sub --write-sub --sub-lang 'en.*' --skip-download --convert-subs srt -o '<workdir>/<vid>.%(ext)s' 'https://www.youtube.com/watch?v=<vid>'`
2. If the SRT file exists, parse into segments `[{start, end, text}]` and a flat `raw_text`.
3. Insert `kevin_transcripts` row, advance `kevin_videos.status` to `captioned`.
4. Cleanup the workdir.

**Failure modes**:
- No captions available → `status = 'failed'`, `failure_reason = 'no_captions'`. Surfaces in the dashboard with a "Reprocess" button (no-op for now; meaningful when we add Whisper fallback in v2).
- yt-dlp bot-detection / network → increment `retry_count`, exp-backoff (1m, 5m, 15m, 1h, 6h), keep `status = 'discovered'`. After 5 retries → `failed`.

**yt-dlp version**: the system's `apt`-installed yt-dlp (`2024.04.09`) trips YouTube's bot-detection. The Dockerfile pip-installs a current PyPI release (pinned `>=2026.04`).

### Stage 3 — LLM analysis

For each video with `status = 'captioned'`:

1. Check daily cost: `SELECT SUM(cost_usd) FROM kevin_analyses WHERE created_at >= date_trunc('day', now() at time zone 'UTC')`. If `>= daily_cost_cap_usd`, log + skip (video remains `captioned`, resumes tomorrow).
2. Build the Claude Sonnet 4.6 request:
   - System prompt (cached via Anthropic prompt caching — 2–3K tokens describing schema, examples, output format)
   - User message: `published_at`, `title`, `description`, plus the full transcript with segment timestamps preserved
   - Tool definition that constrains response to the structured JSON schema below
3. Call the API. On HTTP error, exp-backoff retry up to 3x, then `failed`.
4. Validate the tool-input JSON via Pydantic. On parse error, `failed` with `raw_response_json` captured for debugging.
5. Insert one `kevin_analyses` row + N `kevin_stock_mentions` rows in a single transaction. Advance `kevin_videos.status` to `analyzed`.

### LLM output schema (Pydantic, also the tool-input schema)

```python
class MeetKevinTickerMention(BaseModel):
    symbol: str                       # uppercased, validated [A-Z]{1,6}
    action: Literal["buy","sell","hold","watch","avoid"]
    conviction: float                 # 0.0–1.0
    time_horizon: Literal["intraday","days","weeks","months","long_term","unspecified"]
    rationale_quote: str              # short verbatim or paraphrased quote
    video_timestamp_seconds: int | None  # deep-link target

class MeetKevinAnalysis(BaseModel):
    market_outlook_direction: Literal["bullish","neutral","bearish","mixed"]
    market_outlook_reasoning: str
    macro_themes: list[str]
    key_risks: list[str]
    summary: str                      # ~200 words
    tickers: list[MeetKevinTickerMention]
```

### Cost & rate

- ~3 videos/day × ~7–15K input tokens × Sonnet 4.6 → ~$0.07–0.10 per video
- Prompt cache: the 2–3K-token system prompt is reused across videos in a 5-min window → ~95% cache hit on the system portion
- Default daily cap **$5/day**, configurable per channel via `daily_cost_cap_usd`
- Expected monthly spend: $3–10

## API surface

All routes added to `services/api_gateway/routes/meet_kevin.py`. Behind the existing JWT/passkey auth.

```
GET    /api/meet-kevin/health
GET    /api/meet-kevin/channels
PATCH  /api/meet-kevin/channels/{id}
GET    /api/meet-kevin/videos                       ?status=&from=&to=&q=&page=&limit=
GET    /api/meet-kevin/videos/{id}
GET    /api/meet-kevin/videos/{id}/transcript
POST   /api/meet-kevin/videos/{id}/reprocess        ?stage={captions|analysis}
GET    /api/meet-kevin/stocks
GET    /api/meet-kevin/stocks/{symbol}
GET    /api/meet-kevin/stocks/{symbol}/timeline     ?bucket=day|week
GET    /api/meet-kevin/dashboard                    home aggregate
```

Response schemas live in `shared/schemas/meet_kevin.py` so both backend and frontend (via codegen or hand-written TS interfaces) consume the same shape.

## Dashboard pages

New page group under `dashboard/src/pages/meet-kevin/` with React Router routes. Sidebar gains a "Meet Kevin" section.

| Route | Purpose | Key components |
|---|---|---|
| `/meet-kevin` | Home — latest video card, top-conviction tickers this week, outlook trendline (14d), pipeline status pill | `LatestVideoHero`, `TopConvictionTable`, `OutlookTrendChart` |
| `/meet-kevin/videos` | Feed with filters (status, date range, ticker search) | `VideoFeed`, `VideoCard`, `FilterBar` |
| `/meet-kevin/videos/:id` | Detail — embedded YouTube iframe + tabs (Analysis / Transcript / Raw) | `YouTubeEmbed`, `AnalysisPanel`, `TranscriptPanel`, `TickerCard` |
| `/meet-kevin/stocks` | **Primary surface** — sortable table: symbol, mention count, last seen, latest action, avg conviction, 14d sparkline | `StocksTable`, `Sparkline` |
| `/meet-kevin/stocks/:symbol` | Drill-down — sentiment trendline + chronological mention list + (optional) market-data price overlay | `TickerHeader`, `SentimentTrendChart`, `MentionList` |

**Stack reuse** (all already in `dashboard/package.json`): TanStack Query, Tailwind 4, recharts, lightweight-charts (for the optional price overlay), React Router 7. No new top-level dependencies.

**Empty states**: home shows "Waiting for first poll" with a manual-trigger button. Stocks list shows "No analyses yet" until Claude returns first response.

**Frontend stack note**: `~/.claude/CLAUDE.md` defaults new web apps to Svelte; this is extending the existing React 19 dashboard, so React stays. Documented for traceability.

## K8s revival changes

Single edit to `~/code/infra/stacks/trading-bot/main.tf` (file is currently inside a `/* … */` block-comment, disabled 2026-04-06):

1. **Uncomment the entire `/* … */` block.**
2. In the `trading-bot-workers` Deployment **remove the 3 disabled containers** from the Pod spec: `news-fetcher`, `sentiment-analyzer`, `trade-executor`. (Note: all 6 workers share one Pod, so "disable" means delete the container block — there is no per-container `replicas` knob.)
3. **Add a new `meet-kevin-watcher` container** to the same Pod spec:
   ```hcl
   container {
     name              = "meet-kevin-watcher"
     image             = "viktorbarzin/trading-bot-service:latest"
     image_pull_policy = "Always"
     command           = ["python", "-m", "services.meet_kevin_watcher.main"]
     dynamic "env" {
       for_each = local.common_env
       content { name = env.key, value = env.value }
     }
     env { name = "TRADING_OTEL_METRICS_PORT", value = "9097" }
     env_from { secret_ref { name = "trading-bot-secrets" } }
     env_from { secret_ref { name = "trading-bot-db-creds" } }
     resources {
       requests = { cpu = "10m",  memory = "128Mi" }
       limits   = {                memory = "256Mi" }
     }
   }
   ```
4. **Final container order** in the `trading-bot-workers` Pod (4 containers):
   - `[0]` signal-generator
   - `[1]` learning-engine
   - `[2]` market-data
   - `[3]` meet-kevin-watcher (NEW)

   Update `lifecycle.ignore_changes` accordingly: 4 image-index lines (`spec[0].template[0].spec[0].container[0..3].image`) plus the existing `dns_config` line.
5. Extend the `external_secret` ExternalSecret template with two new keys:
   ```
   TRADING_ANTHROPIC_API_KEY  = "{{ .anthropic_api_key }}"
   TRADING_MEET_KEVIN_CHANNEL_ID = "{{ .meet_kevin_channel_id }}"
   ```
   and add the matching `data` entries.
6. Extend `locals.common_env`:
   ```
   TRADING_MEET_KEVIN_POLL_INTERVAL_SECONDS = "10800"   # 3h
   TRADING_MEET_KEVIN_DAILY_COST_CAP_USD    = "5"
   TRADING_MEET_KEVIN_LLM_MODEL             = "claude-sonnet-4-6"
   TRADING_MEET_KEVIN_PROMPT_VERSION        = "v1"
   ```

No Helm changes, no ingress changes, no new namespace, no new PVC, no new Service.

### Vault secrets

Two new keys at `secret/trading-bot` (Vault KV v2):

```
anthropic_api_key       = sk-ant-...           # Claude API key (pay-per-use)
meet_kevin_channel_id   = UCUvvj5lwue7PspotMDjk5UA
```

Populate via:
```
vault kv patch secret/trading-bot \
    anthropic_api_key=sk-ant-... \
    meet_kevin_channel_id=UCUvvj5lwue7PspotMDjk5UA
```

The existing ExternalSecret refresh interval (15m) picks them up.

## Container & dependency changes

`pyproject.toml` — new optional dependency group:

```toml
meet_kevin = [
  "yt-dlp>=2026.04",     # avoid the apt-shipped 2024.04.09 (bot-detected by YouTube)
  "feedparser>=6.0",
  "anthropic>=0.40",
]
```

`docker/Dockerfile.service` — add `EXTRAS=meet_kevin` to the install matrix so the watcher container has its deps.

## Observability

Reuses the existing OpenTelemetry → Prometheus pattern. New metrics (port 9097):

```
meet_kevin_videos_discovered_total{channel}
meet_kevin_captions_extracted_total{result="ok|no_captions|error"}
meet_kevin_llm_calls_total{result="ok|api_error|parse_error|cost_capped"}
meet_kevin_llm_cost_usd_total                    # counter
meet_kevin_llm_cost_usd_today                    # gauge, resets at UTC midnight
meet_kevin_pipeline_lag_seconds{stage}           # published_at → stage transition
```

Existing pod-annotation scrape config auto-discovers the new port.

## Failure modes (consolidated)

| Stage | Failure | Behavior | User-visible signal |
|---|---|---|---|
| RSS | network / 5xx | log + retry next 3h | `/health` flags `last_poll_age > 6h` |
| RSS | XML parse error | log + skip cycle | same |
| Caption | no captions | `failed` / `no_captions` | "Reprocess" button (no-op v1) |
| Caption | yt-dlp / network | exp-backoff to 5x → `failed` | per-row "Reprocess" |
| LLM | API error | exp-backoff 3x → `failed` (raw response captured) | "Reprocess"; Raw JSON tab |
| LLM | malformed JSON | `failed`, raw response retained | Raw JSON tab |
| LLM | daily cost cap hit | stop processing, leave `captioned` | pipeline-status badge: "Cost cap reached, resumes 00:00 UTC" |

## Testing strategy

Match the existing test-quality bar (1 behavior per test, no internal mocking of own code).

```
tests/services/meet_kevin_watcher/
  test_rss_parser.py          fixture: committed Meet Kevin RSS XML; parse + dedupe paths
  test_caption_extractor.py   mock yt-dlp subprocess; SRT parse, no_captions path
  test_llm_analyzer.py        mock anthropic.Anthropic.messages.create; prompt structure,
                              response validation, cost calc, daily cap logic
  test_pipeline.py            status transitions, retry, failure paths
                              (pytest-asyncio + in-memory SQLite where viable)

tests/integration/test_meet_kevin_e2e.py     [@pytest.mark.integration]
  full Postgres + Redis; mock the external HTTP layer only (RSS, yt-dlp, Anthropic);
  assert discovered → captioned → analyzed flow + ticker mentions row count

tests/api_gateway/routes/test_meet_kevin.py
  FastAPI TestClient per endpoint, auth bypass via existing dev-mode flag

dashboard/src/pages/meet-kevin/__tests__/    (manual QA only — repo has no FE test suite)
```

Target: ~30 unit tests + 1 integration test + 8 API tests.

## Future work (v2 candidates)

1. **Signal injection into `signal_generator`**: new `kevin:signals` Redis stream, new "kevin" strategy weight in the ensemble, opt-in approval queue in the UI.
2. **Whisper API fallback** when captions are missing — currently `no_captions` is terminal.
3. **Multi-channel support** in the UI (schema already supports it).
4. **Slack notification** on new high-conviction mention (`conviction > 0.8`).
5. **Auto re-analysis on prompt version bump** — currently manual via "Reprocess" button.
6. **Audio archival** to NFS for offline reprocessing or training data.