diff --git a/.planning/research/ARCHITECTURE.md b/.planning/research/ARCHITECTURE.md new file mode 100644 index 00000000..59a26900 --- /dev/null +++ b/.planning/research/ARCHITECTURE.md @@ -0,0 +1,434 @@ +# Architecture Research + +**Domain:** Live stream aggregation and proxy service (F1 streaming) +**Researched:** 2026-02-23 +**Confidence:** MEDIUM — HLS spec and proxy mechanics are HIGH confidence from RFC 8216 and Apple docs; extractor patterns are MEDIUM confidence from yt-dlp/streamlink analysis; system composition for this specific use-case is inferred from domain knowledge. + +--- + +## Standard Architecture + +### System Overview + +``` +┌───────────────────────────────────────────────────────────────────┐ +│ CLIENT LAYER │ +│ ┌──────────────────────────────────────────────────────────┐ │ +│ │ Svelte Frontend (schedule view, stream picker, player) │ │ +│ └────────────────────────────┬─────────────────────────────┘ │ +└───────────────────────────────│───────────────────────────────────┘ + │ HTTP/REST +┌───────────────────────────────▼───────────────────────────────────┐ +│ API LAYER │ +│ ┌──────────────────────────────────────────────────────────┐ │ +│ │ Backend API (schedule, streams, health state) │ │ +│ └────────┬──────────────────┬──────────────────────────────┘ │ +└───────────│──────────────────│────────────────────────────────────┘ + │ │ + ▼ ▼ +┌───────────────────┐ ┌──────────────────────────────────────────┐ +│ SCHEDULE │ │ EXTRACTION LAYER │ +│ SUBSYSTEM │ │ ┌───────────┐ ┌───────────┐ │ +│ │ │ │ Extractor │ │ Extractor │ ... │ +│ Jolpica/OpenF1 │ │ │ Site A │ │ Site B │ │ +│ API client │ │ └─────┬─────┘ └─────┬─────┘ │ +│ │ │ │ │ │ +│ Cron: refresh │ │ ┌─────▼───────────────▼──────────────┐ │ +│ schedule │ │ │ Extractor Registry / Dispatcher │ │ +└───────────────────┘ │ └─────────────────────┬──────────────┘ │ + │ │ │ + │ ┌─────────────────────▼──────────────┐ │ + │ │ Stream Health Checker │ │ + │ │ (HEAD/partial GET on .m3u8 URLs) │ │ + │ └─────────────────────────────────────┘ │ + └──────────────────────────────────────────┘ + │ + ▼ valid stream URLs + ┌──────────────────────────────────────────┐ + │ PROXY LAYER │ + │ │ + │ Master Playlist Rewriter │ + │ ┌────────────────────────────────────┐ │ + │ │ GET /proxy?url= │ │ + │ │ → fetch upstream m3u8 │ │ + │ │ → rewrite all URIs to proxy paths │ │ + │ │ → return modified playlist │ │ + │ └────────────────────────────────────┘ │ + │ │ + │ Segment Relay │ + │ ┌────────────────────────────────────┐ │ + │ │ GET /relay?url= │ │ + │ │ → upstream fetch with headers │ │ + │ │ → pipe response to client │ │ + │ └────────────────────────────────────┘ │ + └──────────────────────────────────────────┘ + │ + ▼ piped bytes + ┌──────────────────────────────────────────┐ + │ STORAGE / CACHE │ + │ ┌─────────────────┐ ┌───────────────┐ │ + │ │ In-memory cache │ │ NFS mount │ │ + │ │ (stream links, │ │ (schedule │ │ + │ │ health status) │ │ snapshots, │ │ + │ └─────────────────┘ │ config) │ │ + │ └───────────────┘ │ + └──────────────────────────────────────────┘ +``` + +--- + +### Component Responsibilities + +| Component | Responsibility | Typical Implementation | +|-----------|----------------|------------------------| +| **Svelte Frontend** | Schedule display, stream picker UI, embedded HLS player | SvelteKit app; hls.js or Video.js for player | +| **Backend API** | Serves schedule, current stream list, health status to frontend | Python (FastAPI) or Node.js; REST endpoints | +| **Schedule Subsystem** | Polls Jolpica/OpenF1 API, normalises session data, stores locally | Async background task with cron interval | +| **Extractor Registry** | Maps site hostnames to extractor implementations; dispatches extraction | Plain dict/map of site-key → extractor class | +| **Per-Site Extractor** | Performs HTTP requests with session cookies/CSRF, parses HTML/JS, follows redirect chains, returns raw stream URL | Python class per site; uses `httpx`/`requests` + `BeautifulSoup`/`regex` | +| **Stream Health Checker** | Verifies extracted URLs are live (partial GET on m3u8, checks HTTP 200 + content-type) | Background poller; marks streams up/down in cache | +| **Proxy / Playlist Rewriter** | Fetches upstream m3u8, rewrites all embedded URIs to go through `/relay`, returns modified playlist | Stateless HTTP handler; no buffering of media data | +| **Segment Relay** | Fetches upstream `.ts`/`.fmp4` segments and pipes bytes to client; forwards necessary headers | Streaming HTTP proxy (not buffered); forwards Range, Content-Type | +| **In-Memory Cache** | Stores current stream states and health, avoids redundant extraction on every client request | Python dict with TTL, or Redis (existing cluster Redis) | +| **NFS Storage** | Persists schedule snapshots, extractor configuration, optional diagnostics | NFS at `10.0.10.15` via existing pattern | + +--- + +## Recommended Project Structure + +``` +f1-streams/ +├── backend/ +│ ├── api/ +│ │ ├── routes/ +│ │ │ ├── schedule.py # GET /schedule +│ │ │ ├── streams.py # GET /streams, POST /streams/refresh +│ │ │ └── proxy.py # GET /proxy, GET /relay +│ │ └── main.py # FastAPI app, lifespan hooks +│ ├── extractors/ +│ │ ├── base.py # Extractor ABC: extract() -> list[StreamInfo] +│ │ ├── registry.py # Map site-key -> extractor class +│ │ ├── site_a.py # Site-A specific extractor +│ │ └── site_b.py # Site-B specific extractor +│ ├── schedule/ +│ │ ├── client.py # Jolpica/OpenF1 API client +│ │ ├── models.py # Session, Race pydantic models +│ │ └── poller.py # Background cron task +│ ├── health/ +│ │ └── checker.py # Stream liveness verification +│ ├── proxy/ +│ │ ├── playlist.py # m3u8 fetch + URI rewriting +│ │ └── relay.py # Segment pipe-through handler +│ ├── cache.py # In-memory store with TTL +│ └── config.py # Site list, polling intervals, NFS paths +├── frontend/ +│ ├── src/ +│ │ ├── routes/ +│ │ │ ├── +page.svelte # Schedule home +│ │ │ └── watch/ +│ │ │ └── +page.svelte # Stream picker + player +│ │ ├── lib/ +│ │ │ ├── api.ts # Backend API client +│ │ │ ├── player.ts # hls.js wrapper +│ │ │ └── schedule.ts # Session time formatting +│ │ └── app.html +│ ├── static/ +│ └── package.json +├── stacks/ +│ └── f1-streams/ +│ ├── main.tf +│ └── terragrunt.hcl +└── Dockerfile # Multi-stage: backend + frontend +``` + +### Structure Rationale + +- **backend/extractors/**: One file per site; base class enforces interface. Adding a new site = add one file + register it. No change to core. +- **backend/proxy/**: Isolated from extraction. Proxy only knows about URLs — it does not care how they were found. +- **backend/schedule/**: Completely independent subsystem. Can fail without breaking stream delivery. +- **backend/health/**: Decoupled checker; stores results in cache, consulted by API on `/streams` requests. +- **frontend/**: Standard SvelteKit layout. Minimal — schedule + player, nothing else. +- **stacks/f1-streams/**: Single Terragrunt stack following existing pattern in repo. + +--- + +## Architectural Patterns + +### Pattern 1: Extractor Plugin Interface + +**What:** Each site extractor implements a fixed interface (`extract(session_hint) -> list[StreamURL]`). The registry maps site keys to extractor classes. The dispatcher iterates the registry, calls each extractor, aggregates results. + +**When to use:** Always — the number of sites will grow and their anti-scraping measures change independently. Isolation prevents one broken extractor from affecting others. + +**Trade-offs:** Slightly more boilerplate per site; but each extractor is testable in isolation and replaceable without touching shared code. + +**Example:** +```python +class BaseExtractor(ABC): + site_key: str # e.g. "siteA" + + @abstractmethod + async def extract(self, hint: SessionHint | None = None) -> list[StreamURL]: + """Return list of live stream URLs found on this site.""" + ... + +class SiteAExtractor(BaseExtractor): + site_key = "siteA" + + async def extract(self, hint=None) -> list[StreamURL]: + # 1. GET page, parse CSRF token from HTML + # 2. POST with token to get obfuscated JSON + # 3. Decode JS-obfuscated URL + # 4. Follow redirects to final .m3u8 + ... +``` + +### Pattern 2: Playlist Rewriting Proxy + +**What:** The proxy layer fetches the upstream m3u8 and rewrites every URL inside it (both master → variant pointers, and variant → segment pointers) to point back through `/relay?url=`. The client never contacts upstream directly. + +**When to use:** Always when proxying HLS — the player will follow URLs in the playlist; if those URLs point to the origin CDN, the proxy is bypassed for segment delivery. + +**Trade-offs:** Adds ~1 hop latency per segment request. For a private service with 1-5 users, this is negligible. Benefit: hides origin, enables header injection (e.g., `Referer`), unified player experience. + +**Example:** +```python +def rewrite_playlist(m3u8_text: str, base_url: str, proxy_base: str) -> str: + """Rewrite all URIs in an m3u8 to go through the proxy relay endpoint.""" + lines = [] + for line in m3u8_text.splitlines(): + if line and not line.startswith("#"): + # resolve relative URL, then encode through proxy + absolute = urllib.parse.urljoin(base_url, line) + proxied = f"{proxy_base}/relay?url={b64encode(absolute)}" + lines.append(proxied) + else: + lines.append(line) + return "\n".join(lines) +``` + +### Pattern 3: Background Polling with In-Memory Cache + +**What:** Extraction and health checking run as background tasks on a schedule (e.g., every 2 minutes). Results are stored in a shared in-memory dict with timestamps. The API layer reads from cache and returns immediately — no per-request extraction. + +**When to use:** Always — on-demand extraction per client request would be slow (2-10s per site) and would hammer the source sites. + +**Trade-offs:** Cache staleness window (default 2 min). Acceptable for live sports: streams stay stable once live. + +**Example:** +```python +# cache.py +_stream_cache: dict[str, CachedResult] = {} + +async def get_streams() -> list[StreamURL]: + if cache_is_fresh(): + return _stream_cache["streams"].data + # else trigger background refresh + ... +``` + +--- + +## Data Flow + +### Stream Discovery Flow (background) + +``` +[Cron trigger: every 2 min] + ↓ +[Extractor Registry] + ↓ (fan-out, concurrent) +[SiteA Extractor] [SiteB Extractor] [SiteN Extractor] + ↓ +[Raw stream URLs: list of .m3u8 candidates] + ↓ +[Health Checker: partial GET each URL] + ↓ (filter: only HTTP 200 + video/mpegURL content-type) +[Validated stream URLs] + ↓ +[Cache: store with timestamp + site metadata] +``` + +### Client Playback Flow (per request) + +``` +[User opens /watch in browser] + ↓ +[Frontend GET /api/streams] + ↓ +[Backend reads cache → returns stream list (site, quality, label)] + ↓ +[User picks a stream] + ↓ +[Player requests: GET /proxy?url=] + ↓ +[Backend: fetch upstream m3u8, rewrite URIs → return modified m3u8] + ↓ +[Player follows variant playlist: GET /proxy?url=] + ↓ +[Backend: rewrite segment URIs] + ↓ +[Player fetches segments: GET /relay?url=] + ↓ +[Backend: upstream fetch, pipe bytes → client] + ↓ +[Video plays in browser] +``` + +### Schedule Flow + +``` +[Cron: daily or on-demand] + ↓ +[Schedule Client: GET Jolpica API /ergast/f1/current.json] + ↓ +[Parse: races, session types, UTC timestamps] + ↓ +[Normalise: map to internal Session model] + ↓ +[Store: NFS JSON file + in-memory cache] + ↓ +[Frontend GET /api/schedule → displays session list] +``` + +### Key Data Flows + +1. **Extraction → Cache → API → Frontend**: All stream data originates from extractors, flows through the cache as the single source of truth, and is served read-only to the frontend. No frontend-triggered extraction. +2. **Client → Proxy → Upstream CDN**: The proxy is a pure pass-through relay. It does not store segments. Bytes from upstream go directly to client socket. +3. **Schedule API → NFS**: Schedule data is written to NFS on refresh so the pod can serve it immediately on restart without waiting for the next API poll. + +--- + +## Component Boundaries + +| Component | Owns | Does Not Own | +|-----------|------|--------------| +| Extractor (per site) | How to get stream URL from that site | Health checking, caching, proxying | +| Health Checker | Liveness state of each URL | How the URL was found | +| Proxy / Relay | Rewriting m3u8 URIs, piping bytes | Authentication with upstream (that's extractor's job) | +| Schedule Subsystem | F1 session calendar data | Stream availability for a given session | +| Backend API | Serving current state to frontend | Fetching or refreshing state | +| Frontend | User interaction, player | Any backend logic | + +--- + +## Suggested Build Order (Phase Dependencies) + +The dependencies flow strictly upward — each layer depends only on the layer below it being stable: + +``` +Phase 1: Schedule Subsystem + ↓ (F1 data available) +Phase 2: Extractor Framework + First Site Extractor + ↓ (raw URLs available) +Phase 3: Health Checker + ↓ (validated URLs available) +Phase 4: Proxy / Relay Layer + ↓ (streams playable through service) +Phase 5: Frontend (schedule + player) + ↓ (end-to-end usable) +Phase 6: Additional Site Extractors + ↓ (stream coverage widened) +Phase 7: K8s Deployment (Terraform/Terragrunt stack) +``` + +**Rationale:** +- Schedule first: gives a testable data source with zero anti-scraping complexity. +- Extractor framework before specific sites: the base class and registry must exist before any site can plug in. +- Health checker before proxy: no point proxying dead streams; the checker filters the list fed to the proxy. +- Proxy before frontend: the frontend player needs a working `/proxy` endpoint to function. +- Frontend last of core: all backend components are independently testable via curl/httpie before a UI exists. +- Additional extractors after core is working: adding more sites is low-risk incremental work once the pattern is proven. +- Deployment last: deploy once the service works end-to-end locally; avoids debugging infra and app simultaneously. + +--- + +## Anti-Patterns + +### Anti-Pattern 1: On-Demand Extraction Per Client Request + +**What people do:** Trigger extraction when the user clicks "show streams" in the browser. + +**Why it's wrong:** Extraction takes 2-10 seconds per site (HTTP round trips, JS parsing, redirect following). With multiple sites, this is 10-30 seconds of wall time. Source sites may rate-limit aggressive bursts. Multiple concurrent users would multiply the load. + +**Do this instead:** Run extraction on a background schedule. Cache results. The API returns immediately from cache. The user sees streams in <100ms. + +### Anti-Pattern 2: Single Extractor Handles All Sites + +**What people do:** One big function with `if site == "A": ... elif site == "B": ...` branches. + +**Why it's wrong:** Sites change their obfuscation methods independently. A change to Site A's extraction logic can accidentally break Site B. Testing is impossible in isolation. Adding Site C requires modifying a shared file. + +**Do this instead:** One class per site, implementing a common interface. Changes to Site A's extractor never touch Site B's code. + +### Anti-Pattern 3: Buffering Segments in Memory Before Sending + +**What people do:** Download the entire `.ts` segment to memory, then serve it to the client. + +**Why it's wrong:** HLS segments can be 2-10 MB each. With multiple concurrent viewers, memory pressure grows quickly. Introduces unnecessary latency (client waits for full download before first byte). + +**Do this instead:** Pipe bytes from the upstream response directly to the client socket as they arrive (chunked transfer). The client starts receiving immediately, memory stays flat. + +### Anti-Pattern 4: Hardcoding Site URLs and Tokens in Extractor Logic + +**What people do:** Hardcode `BASE_URL = "https://site-a.example.com"` and referer/cookie values inside the extractor file. + +**Why it's wrong:** Sites change domains and anti-scraping parameters frequently. When a site moves, you have to find and edit code rather than config. + +**Do this instead:** Extractor reads its config (base URL, required headers, any known static tokens) from a config object injected at construction. The registry passes config to extractors at instantiation. + +--- + +## Integration Points + +### External Services + +| Service | Integration Pattern | Notes | +|---------|---------------------|-------| +| Jolpica F1 API (`api.jolpi.ca/ergast/f1/`) | REST GET, poll daily | No API key required; backwards-compatible Ergast endpoints; schedule data available | +| OpenF1 API (`api.openf1.org/`) | REST GET, poll as needed | No API key; 3 req/s rate limit; 2023+ data only; useful for session status (live/upcoming) | +| Upstream streaming sites (Site A, B, N) | HTTP GET/POST with session cookies, CSRF tokens | Per-site; no shared pattern; treated as black boxes by the framework | +| Upstream CDN (HLS segments) | HTTP GET with Range support | Proxy relays bytes; must forward `Referer` and sometimes `Origin` headers or CDN rejects | + +### Internal Boundaries + +| Boundary | Communication | Notes | +|----------|---------------|-------| +| Extractor → Cache | Direct function call (write) | Extractors do not call the cache directly — the dispatcher aggregates results then writes once | +| API → Cache | Direct read | Synchronous, O(1) | +| API → Proxy | Not direct — frontend calls `/proxy` endpoint, which is part of the same backend process | Can be split into separate service later if needed | +| Proxy → Upstream CDN | Outbound HTTP | Must preserve session headers; upstream CDN may check Referer/Origin | +| Schedule Poller → NFS | File write (JSON) | On pod restart, reads NFS before first API poll | + +--- + +## Scaling Considerations + +This is a single-user or small-group private service. Scaling is not a primary concern, but here are the natural pressure points: + +| Scale | Architecture Adjustments | +|-------|--------------------------| +| 1-5 concurrent viewers | Single backend pod, in-memory cache, direct pipe relay — fully sufficient | +| 10-20 concurrent viewers | Same architecture; segment relay becomes the bandwidth bottleneck (each viewer streams independently) — add HLS caching proxy (nginx) in front of relay | +| 50+ concurrent viewers | Segment relay load increases linearly; consider a CDN or caching layer for segments; extraction/health remain unchanged | + +### Scaling Priorities + +1. **First bottleneck:** Outbound bandwidth on segment relay. Each viewer pulls full bitrate independently through the service. At private-use scale this is negligible (1-5 viewers). +2. **Second bottleneck:** In-memory cache invalidation if multiple pods deploy (stateless pods don't share cache). Solved by using existing cluster Redis instead of in-process dict — but unnecessary until horizontal scaling. + +--- + +## Sources + +- HLS specification: RFC 8216 (IETF) — playlist structure, master/media playlist relationship, segment mechanics (HIGH confidence) +- HLS proxy pattern: Apple Developer Documentation (conceptual), corroborated by yt-dlp extractor framework analysis (MEDIUM confidence) +- yt-dlp plugin architecture: github.com/yt-dlp/yt-dlp README + docs (MEDIUM confidence) +- OpenF1 API: openf1.org official page — endpoints, rate limits, data coverage (HIGH confidence) +- Jolpica F1 API: github.com/jolpica/jolpica-f1 — Ergast compatibility, availability (MEDIUM confidence) +- System composition for this domain: inference from domain patterns, corroborated by extractor tool analysis (MEDIUM confidence) + +--- + +*Architecture research for: Live stream aggregation and proxy service (F1)* +*Researched: 2026-02-23* diff --git a/.planning/research/FEATURES.md b/.planning/research/FEATURES.md new file mode 100644 index 00000000..3e2147c0 --- /dev/null +++ b/.planning/research/FEATURES.md @@ -0,0 +1,215 @@ +# Feature Research + +**Domain:** Live Stream Aggregation / Sports Stream Proxy Service +**Researched:** 2026-02-23 +**Confidence:** MEDIUM + +--- + +## Feature Landscape + +### Table Stakes (Users Expect These) + +Features users assume exist. Missing these = product feels incomplete. + +| Feature | Why Expected | Complexity | Notes | +|---------|--------------|------------|-------| +| Race schedule view | Users need to know when sessions are live without external lookup | LOW | Pull from OpenF1 API (`/sessions` endpoint). Session types: FP1, FP2, FP3, Quali, Sprint, Sprint Quali, Race. Confidence: HIGH (OpenF1 API confirmed). | +| Live session indicator | Users need to distinguish live vs upcoming vs finished sessions at a glance | LOW | Visual status badge (LIVE / UPCOMING / FINISHED) based on session start time + duration. No polling needed at schedule level. | +| Stream picker | Multiple stream sources per session — user picks which one to watch | LOW | List available extracted stream links with source label. Core UX of the whole product. | +| Embedded video player | Users won't navigate to external players for each stream | MEDIUM | HLS.js in Svelte for in-page playback. Must handle m3u8 sources natively. Confidence: HIGH (HLS.js is the standard client-side HLS library). | +| Stream health indicator | Users don't want to click a dead stream and stare at a spinner | MEDIUM | Backend health-check each extracted URL before displaying. Simple HEAD or short-lived GET on the m3u8 playlist. Mark dead streams visually. | +| CORS-transparent stream proxy | Browsers block cross-origin HLS requests; streams can't play directly from scraped origins | HIGH | Proxy all m3u8 manifests + .ts/.m4s segments through your own backend. Rewrite manifest URLs to point to your proxy. This is architecturally mandatory, not optional. Confidence: HIGH (HLS-Proxy documentation confirms this). | +| All F1 session types covered | Users specifically want FP, Quali, Sprint, Race, and pre/post content — not just race day | MEDIUM | Scraper scheduler must run for every session type on the F1 calendar. OpenF1 `/sessions` endpoint returns `session_type` field. | +| Session countdown timer | For upcoming sessions, users want to know time-until-start without mental math | LOW | Client-side countdown from schedule data already fetched. Zero backend cost. | +| Stream auto-refresh / re-extraction | Stream links expire (tokens, redirect chains rotate) — stale links silently fail | HIGH | Periodic re-extraction (e.g., every 5-10 min during a live session). Depends on extractor infrastructure. | +| Multiple quality options (if available) | Users on slow connections need lower bitrate; users on fast connections want max quality | MEDIUM | Expose quality variants from multi-variant HLS playlists if source provides them. Let user pick or default to auto (hls.js handles ABR natively). | + +--- + +### Differentiators (Competitive Advantage) + +Features that set the product apart. Not required, but valuable. + +| Feature | Value Proposition | Complexity | Notes | +|---------|-------------------|------------|-------| +| Automatic stream extraction at session start | Zero manual effort — streams appear when the session goes live | HIGH | Cron/scheduler tied to F1 calendar. Triggers extractors N minutes before session start. Eliminates "is there a stream yet?" manual checking. | +| Per-site extractor isolation | Bypassing CSRF/JS obfuscation cleanly per site without shared code that breaks globally | HIGH | Each extractor is a self-contained module. One site's changes don't break others. Confidence: MEDIUM (pattern from streamlink plugin system). | +| Session timeline: pre/post shows + press conferences | Competitors (scrapers, IPTV playlists) cover race only; full weekend coverage is rare | MEDIUM | Requires scheduling extractors for non-race events. OpenF1 does not cover pre/post shows — need site-specific session detection. | +| Stream source labeling | Shows which site/feed each stream came from — users learn which sources are reliable | LOW | Store source metadata with each extracted URL. Display in picker. | +| Fallback stream ordering | Automatically surfaces known-good streams first when multiple sources exist | MEDIUM | Health-check result + historical success rate drives ordering. Depends on: stream health checking + a minimal persistence layer to store success history. | +| Proxy-cached segment prefetch | Reduces buffering by prefetching upcoming .ts segments into local cache | HIGH | Node-HLS-Proxy pattern: maintain per-stream segment cache up to N segments ahead. High implementation cost for marginal UX gain at private scale. | +| Session notes / source reputation | Lightweight annotations (e.g., "this source often drops at lap 40") | LOW | Simple static config or admin-editable markdown. No database needed at MVP. | +| Race weekend overview page | One page showing all sessions for a Grand Prix weekend — not just next session | LOW | Group sessions by event/round from schedule API. Pure frontend feature once schedule data is available. | + +--- + +### Anti-Features (Commonly Requested, Often Problematic) + +Features to explicitly NOT build. + +| Feature | Why Requested | Why Problematic | Alternative | +|---------|---------------|-----------------|-------------| +| DVR / stream recording | Users want to rewatch if they miss something | Massive storage cost, legal exposure, complexity (recording live HLS streams, serving VOD). Out of scope by design. | Live viewing only. Accept the constraint. | +| Chat / comments | Social viewing experience | Scope creep. You're building a stream aggregator, not a community platform. Auth, moderation, and DB schema all follow. | None — explicitly out of scope. | +| User accounts / watchlists | "Remember my preferred stream source" | Requires auth layer, session storage, DB. Contradicts the "no auth, private URL" design decision. | Persist last-used quality/source in browser localStorage. Zero backend cost. | +| Stream transcoding / re-encoding | Normalize quality across sources | Enormous CPU cost, latency, and complexity. An FFmpeg transcoding pipeline per stream is overkill for a private service. | Pass-through proxy only. Let hls.js handle ABR on the client. | +| Headless browser extraction | Universal extractor that handles any site's JS obfuscation | Puppeteer/Playwright adds 200-400 MB RAM per session, slow cold starts, flaky in containers, and complex cluster scheduling. Per-site custom extractors are faster and more reliable. | Custom per-site extractors (Go/Python HTTP + regex/DOM parser). | +| Mobile app | Access on phone | Web app with responsive Svelte layout is sufficient. Native app is weeks of work for a private tool. | Responsive web design. PWA if needed. | +| Discovery / search for new stream sites | Auto-find new sources | Scraping discovery is an unsolved problem and a rabbit hole. You have a fixed list of sites. | User-provided site list. Extractor per site. | +| Telemetry overlay / timing data | F1 fans love live timing alongside streams | Different product category (timing dashboard vs stream aggregator). OpenF1 has timing data but integrating it is a separate project. | Link to existing timing tools (e.g., openf1.org). | +| DRM stream support | Some quality sources use Widevine/FairPlay | DRM circumvention is legally distinct from re-streaming. Avoid. | Non-DRM HLS sources only. | + +--- + +## Feature Dependencies + +``` +Race Schedule View + └──requires──> F1 Schedule API Integration (OpenF1 or Ergast) + └──enables──> Session Countdown Timer + └──enables──> Automatic Extraction Trigger + +Stream Picker + └──requires──> CORS-Transparent Stream Proxy (browser cannot directly fetch cross-origin m3u8) + └──requires──> Stream Health Indicator (to filter dead streams before display) + └──requires──> Stream Health Checker (backend periodic HEAD/GET) + +Embedded Video Player + └──requires──> CORS-Transparent Stream Proxy (proxied URLs served from same origin) + └──requires──> Stream Picker (to know which URL to play) + +Stream Auto-Refresh + └──requires──> Per-Site Extractor (to re-run extraction) + └──requires──> Session-live detection (know when to run vs stop) + +Fallback Stream Ordering + └──requires──> Stream Health Indicator + └──enhances──> Stream Picker (surfaces best streams first) + +Multiple Quality Options + └──requires──> CORS-Transparent Stream Proxy (proxy must rewrite variant playlist URLs too) + └──enhances──> Embedded Video Player (user control or ABR) + +Proxy-Cached Segment Prefetch + └──requires──> CORS-Transparent Stream Proxy (must be same proxy layer) + └──conflicts──> Minimal resource footprint (high memory cost) + +Session Timeline (pre/post/press conf) + └──requires──> F1 Schedule API Integration (for race events) + └──requires──> Per-Site Session Detection (API doesn't include pre/post show timing) +``` + +### Dependency Notes + +- **Stream Picker requires CORS proxy:** Browsers enforce same-origin policy. A scraped m3u8 URL from `site.com` cannot be fetched by a Svelte app on `f1.viktorbarzin.me`. Every user-facing stream URL must route through the proxy backend. This is a hard architectural dependency, not an option. +- **Stream health checker enables stream picker quality:** Without health checking, the picker shows dead links. Health checking must run before streams are displayed and periodically during live sessions. +- **Automatic extraction trigger depends on schedule:** The scheduler must know when sessions start. Schedule API integration is therefore the first thing to build — everything else gates on it. +- **Multiple quality options conflict with simple proxy:** If the source provides a multi-variant HLS playlist, the proxy must rewrite ALL variant URLs (not just the master manifest). Adds complexity to the proxy rewriting layer. +- **Fallback ordering conflicts with stateless proxy:** Tracking success history requires at least a lightweight persistence layer (e.g., Redis or SQLite). If staying fully stateless, fall back to health-check-only ordering. + +--- + +## MVP Definition + +### Launch With (v1) + +Minimum viable product — what's needed to validate the concept. + +- [ ] **F1 Schedule view** — Show upcoming/live sessions for the current season. Single page, no navigation needed. +- [ ] **CORS-transparent HLS proxy** — Proxy m3u8 manifests + segment URLs through the backend. Without this, nothing plays in the browser. +- [ ] **Per-site stream extractor(s)** — At least one working extractor for at least one reliable source site. Proves the extraction pipeline end-to-end. +- [ ] **Stream health checker** — Validate extracted URLs before showing. Dead streams must not surface to users. +- [ ] **Stream picker** — List available working streams for the current session. User clicks, player loads. +- [ ] **Embedded HLS player** — HLS.js in Svelte. Plays proxied m3u8 URL in-page. +- [ ] **Session countdown** — Time-until-start for upcoming sessions. Pure frontend, zero cost. +- [ ] **Live session indicator** — Visual LIVE/UPCOMING/FINISHED badge. Core navigational signal. + +### Add After Validation (v1.x) + +Features to add once core pipeline is working and streams actually play reliably. + +- [ ] **Stream auto-refresh** — Re-run extractors every 5-10 min during live sessions. Trigger: user reports dead stream or health check fails on previously-valid URL. +- [ ] **Fallback stream ordering** — Sort by health-check recency and past reliability. Trigger: multiple sources available per session. +- [ ] **Source labeling in picker** — Show site name with each stream link. Low effort, high trust signal for users. +- [ ] **Race weekend overview** — All sessions grouped per Grand Prix. Trigger: users navigating between sessions in a weekend. +- [ ] **Additional extractors** — Expand site coverage once first extractor is stable. Each adds incremental reliability. + +### Future Consideration (v2+) + +Features to defer until product-market fit is established. + +- [ ] **Pre/post show + press conference coverage** — Complex site-specific session detection. Defer until core race coverage is solid. +- [ ] **Multiple quality options** — Source sites may or may not provide multi-variant playlists. Complexity of rewriting variant URLs in proxy is non-trivial. Validate first if sources actually offer quality tiers. +- [ ] **Proxy segment prefetch/cache** — High memory cost. Only valuable if buffering is a real user complaint at private scale. +- [ ] **Session reputation annotations** — Nice UX polish. Not needed at launch. + +--- + +## Feature Prioritization Matrix + +| Feature | User Value | Implementation Cost | Priority | +|---------|------------|---------------------|----------| +| F1 Schedule view | HIGH | LOW | P1 | +| CORS-transparent HLS proxy | HIGH | HIGH | P1 (architectural blocker) | +| Per-site stream extractor | HIGH | HIGH | P1 (core value) | +| Embedded HLS player | HIGH | LOW | P1 | +| Stream health checker | HIGH | MEDIUM | P1 | +| Stream picker | HIGH | LOW | P1 | +| Session countdown timer | MEDIUM | LOW | P1 | +| Live session indicator | HIGH | LOW | P1 | +| Stream auto-refresh | HIGH | MEDIUM | P2 | +| Source labeling | MEDIUM | LOW | P2 | +| Fallback stream ordering | MEDIUM | MEDIUM | P2 | +| Race weekend overview page | MEDIUM | LOW | P2 | +| Additional extractors | HIGH | MEDIUM | P2 | +| Multiple quality options | MEDIUM | HIGH | P3 | +| Pre/post show coverage | MEDIUM | HIGH | P3 | +| Proxy segment prefetch | LOW | HIGH | P3 | +| Session reputation annotations | LOW | LOW | P3 | + +**Priority key:** +- P1: Must have for launch +- P2: Should have, add when possible +- P3: Nice to have, future consideration + +--- + +## Competitor Feature Analysis + +Reference products surveyed: RaceControl (unofficial F1TV client), f1viewer (TUI F1TV client), streamlink (stream extraction CLI), HLS-Proxy (node HLS proxy), Threadfin (M3U proxy), ErsatzTV (self-hosted IPTV). + +| Feature | RaceControl (F1TV client) | Streamlink (CLI extractor) | HLS-Proxy (node) | Our Approach | +|---------|--------------------------|---------------------------|-----------------|--------------| +| Session schedule | F1TV API (official, auth required) | None (site-specific) | None | OpenF1/Ergast (free, unauthenticated) | +| Stream extraction | Official F1TV API | Plugin-per-site Python | N/A | Custom per-site extractors (Go/Python HTTP) | +| Stream quality selection | Multi-variant picker + Chromecast | CLI flag `--default-stream` | Pass-through | HLS.js ABR + manual picker | +| Multi-stream view | Yes (layout builder, experimental sync) | Multiple instances | N/A | Single stream (MVP), multi optional later | +| Health checking | None visible | None | None | Active periodic health checks (our differentiator) | +| Stream proxy | No (plays direct from F1TV CDN) | No (piped to local player) | Yes (manifest + segment rewrite) | Yes (mandatory for browser CORS) | +| CORS handling | N/A (desktop app) | N/A (local) | Yes (adds permissive CORS headers) | Yes (same-origin proxy) | +| Auto-extraction at session start | Via F1TV live schedule | None | None | Yes (scheduler + extractor trigger) | +| Embedded browser player | No (external VLC/mpv) | No (external player) | N/A | Yes (HLS.js in Svelte) | +| No auth required | No (F1TV subscription) | Varies by source | None | Yes (private URL, no auth layer) | + +**Key insight:** Existing tools either require official F1TV credentials (RaceControl, f1viewer) or extract streams to local players (streamlink). None combine automated extraction from unofficial sources + browser-native proxied playback + schedule integration in a single web service. That combination is the product's core novelty. + +--- + +## Sources + +- OpenF1 API documentation: https://openf1.org/ — MEDIUM confidence (marketing page, limited technical detail on session endpoints) +- HLS-Proxy (warren-bank/HLS-Proxy) README — HIGH confidence for proxy architecture requirements (CORS, manifest rewriting, segment caching) +- HLS.js README (video-dev/hls.js) — HIGH confidence for client-side HLS capabilities (ABR modes, quality switching, error recovery) +- Streamlink documentation: https://streamlink.github.io/ — HIGH confidence for extraction patterns and plugin architecture +- yt-dlp README — HIGH confidence for extractor-per-site pattern and format selection +- RaceControl (robvdpol/RaceControl) README — MEDIUM confidence for F1 streaming UX expectations +- f1viewer (SoMuchForSubtlety/f1viewer) README — MEDIUM confidence for F1 session coverage expectations +- Threadfin README — MEDIUM confidence for IPTV/HLS proxy feature patterns +- Telly README — LOW confidence (Plex-specific, limited relevance) +- Eyevinn/hls-proxy README — HIGH confidence for HLS manifest manipulation patterns + +--- + +*Feature research for: F1 Live Stream Aggregation Service* +*Researched: 2026-02-23* diff --git a/.planning/research/PITFALLS.md b/.planning/research/PITFALLS.md new file mode 100644 index 00000000..7cef6041 --- /dev/null +++ b/.planning/research/PITFALLS.md @@ -0,0 +1,291 @@ +# Pitfalls Research + +**Domain:** Live stream aggregation and proxy service (F1-focused) +**Researched:** 2026-02-23 +**Confidence:** MEDIUM — findings synthesized from yt-dlp/streamlink source analysis (HIGH), nginx proxy documentation (HIGH), HLS RFC/spec analysis (HIGH), OpenF1 API docs (HIGH), and web searches that returned sparse results (LOW where noted) + +--- + +## Critical Pitfalls + +### Pitfall 1: Treating JavaScript-Rendered Tokens as Static + +**What goes wrong:** +Stream URLs on sports streaming sites are not present in raw HTML. They are computed client-side by obfuscated JavaScript — the page HTML contains an encrypted or encoded config blob, and the actual HLS URL is assembled by executing that JS. A scraper that fetches the raw HTML page and runs a regex over it finds nothing. + +**Why it happens:** +Developers assume "the URL must be somewhere in the page source." They inspect the page in DevTools, see the URL in the network tab, and try to replicate what they observe — but miss that the URL was produced by JS execution, not served in the initial response. + +**How to avoid:** +- For each target site: trace the actual network request that fetches the m3u8 in browser DevTools (Network tab, filter by `.m3u8`). Identify the API endpoint that the JS calls to get the signed URL. Replicate *that* API call (often a JSON endpoint), not the page fetch. +- If the token is computed entirely client-side (e.g., via CryptoJS with a hardcoded key), implement the same algorithm in your extractor. Do not run headless browser — reverse-engineer the JS algorithm. +- Document which sites require JS execution vs. which expose a clean API endpoint. Sites often have a backend API that the JavaScript calls; scraping that API is faster and more stable than re-implementing the JS. + +**Warning signs:** +- Extractor returns empty results on a page you can watch in the browser +- Network tab shows the m3u8 URL appearing only after JavaScript fires an XHR/fetch call +- Page HTML contains a large base64 blob or heavily obfuscated JS variable (e.g., `var _0x1a2b = [...]`) + +**Phase to address:** +Extractor design phase (before writing a single extractor). Establish upfront: for each target site, determine if raw HTTP fetch is sufficient or if API reverse-engineering is required. + +--- + +### Pitfall 2: m3u8 Segment URLs Break When Proxied Through a Different Domain + +**What goes wrong:** +When you fetch an m3u8 playlist and serve it through your proxy, the segment URLs inside the playlist may be absolute URLs pointing to the original CDN. The browser (or HLS.js) follows those segment URLs directly, bypassing your proxy entirely. This means you cannot control access, cannot inject headers, and CORS blocks the segments if the CDN doesn't allow cross-origin requests from your frontend domain. + +**Why it happens:** +HLS playlists can contain either absolute URLs (`https://cdn.example.com/seg001.ts`) or relative paths (`seg001.ts`). Most streaming CDNs use absolute URLs with signed tokens. Proxying only the m3u8 is insufficient — every segment URL must also be rewritten to route through your relay. + +**How to avoid:** +- When serving the m3u8 through your proxy, **rewrite all segment URLs** to point to your relay endpoint before sending the playlist to the client. Example: replace `https://cdn.site.com/segment001.ts?token=xyz` with `https://your-relay.domain/proxy/segment?url=`. +- Your relay endpoint then fetches the original segment and streams it to the client. +- Handle multi-level playlists: master playlists (variant streams) reference child playlists which reference segments — rewrite at each level. + +**Warning signs:** +- Client-side CORS errors in browser console referencing the original CDN domain +- Network tab shows segment fetches bypassing your proxy after the m3u8 loads +- Some quality variants play but others don't (partial rewriting) + +**Phase to address:** +Stream relay/proxy phase. Must be designed before the first end-to-end stream test. + +--- + +### Pitfall 3: CDN-Signed Token URLs Expire Mid-Stream + +**What goes wrong:** +Many CDNs sign stream URLs with a short-lived token (often 5–30 minutes). The m3u8 playlist URL itself may be signed, and so may each segment URL. A user who starts watching near the token expiry time will get a working stream for the first few segments, then receive 403 Forbidden errors as the token expires mid-playback. + +**Why it happens:** +Developers test stream extraction, confirm the URL works, and ship it — without accounting for token TTL. The token was valid at extraction time but expires before or during playback. Live streams compound this: the m3u8 playlist updates every few seconds and each update may contain newly signed segment URLs. If your relay cached an old playlist, segments within it are expired. + +**How to avoid:** +- Never cache m3u8 playlist files. Always fetch the live playlist from upstream on each client request. Cache only TS/m4s segments (which have longer or no expiry). +- When extracting the initial stream URL, record the extraction timestamp and the token TTL (if discoverable from response headers like `Cache-Control: max-age=N`). Re-extract before expiry. +- Implement a background refresh: when serving a stream, periodically re-run the extractor to get a fresh URL and pivot the relay to the new upstream without interrupting the client. +- Test expiry by extracting a URL and waiting 30 minutes before playing — a failing test here reveals token TTL issues. + +**Warning signs:** +- Stream plays for exactly N minutes then fails with 403 on segments +- Extracting a URL works in isolation but fails when embedded in the player after a delay +- Sites with Cloudflare or custom CDN always add `?token=` or `?sig=` parameters to segment URLs + +**Phase to address:** +Stream relay phase. The relay architecture must include a URL refresh loop from day one. + +--- + +### Pitfall 4: Per-Site Extractor Maintenance Burden Is Dramatically Underestimated + +**What goes wrong:** +Each target site is a custom engineering problem. Sites change their HTML structure, JavaScript obfuscation, API endpoints, or anti-bot measures without notice. A working extractor can break silently overnight. With 5 target sites, you effectively have 5 separate maintenance tracks. Expecting "set and forget" behavior is the most common planning mistake in this domain. + +**Why it happens:** +Developers build an extractor, it works, and they move on. Sites then deploy a CDN update, change their frontend framework, or rotate their obfuscation keys. The failure is silent — no exception is raised, the extractor just returns no URL, and the user sees an empty player. + +**How to avoid:** +- Build a health check system that runs each extractor on a schedule (every 15 minutes during race weekends, every hour otherwise), logs success/failure, and triggers alerts on failure. +- Design extractors with failure visibility: log exactly which step failed (page fetch, URL parse, API call, etc.) so debugging is fast. +- Keep extractor logic isolated and testable: each extractor is a module that takes no inputs and returns a stream URL or raises an exception. Run integration tests against live sites on a schedule. +- Plan 1–2 hours of maintenance per extractor per month as baseline, more during site redesigns. + +**Warning signs:** +- No automated testing of extractors against live sites +- Extractor code tightly coupled to specific HTML element IDs or class names (breaks on any frontend change) +- No alerting when an extractor returns no URL + +**Phase to address:** +Extractor design phase. The monitoring/health-check system must be built alongside the first extractor, not added later. + +--- + +### Pitfall 5: Missing or Incorrect CORS Headers on the Relay Breaks Browser Playback + +**What goes wrong:** +HLS.js in the browser makes cross-origin requests to fetch m3u8 playlists and segment files. If your relay doesn't serve the correct CORS headers, every segment request fails with a CORS error. Even a single missing header (e.g., on `.ts` segment responses but not `.m3u8` responses) breaks the stream. + +**Why it happens:** +Developers test the relay with `curl` or server-to-server calls, where CORS is irrelevant. The relay works in isolation but fails when the browser's HLS.js player makes the requests. + +**How to avoid:** +- Set CORS headers on **all** relay endpoints: `Access-Control-Allow-Origin: *` (or your specific frontend domain), `Access-Control-Allow-Methods: GET, HEAD, OPTIONS`, `Access-Control-Allow-Headers: Range`. +- The `Range` header is critical: HLS.js often sends range requests for segments. If `Range` is not in `Allow-Headers`, preflight OPTIONS requests fail. +- Do not use wildcard `*` if you also send `Access-Control-Allow-Credentials: true` — that combination is invalid and browsers reject it. +- Test from the actual browser environment (or use a CORS testing tool) before calling any relay endpoint "done." + +**Warning signs:** +- Browser console shows `No 'Access-Control-Allow-Origin' header` errors +- Streams work when loaded directly in a `