[ci skip] Flatten module wrappers into stack roots
Remove the module "xxx" { source = "./module" } indirection layer
from all 66 service stacks. Resources are now defined directly in
each stack's main.tf instead of through a wrapper module.
- Merge module/main.tf contents into stack main.tf
- Apply variable replacements (var.tier -> local.tiers.X, renamed vars)
- Fix shared module paths (one fewer ../ at each level)
- Move extra files/dirs (factory/, chart_values, subdirs) to stack root
- Update state files to strip module.<name>. prefix
- Update CLAUDE.md to reflect flat structure
Verified: terragrunt plan shows 0 add, 0 destroy across all stacks.
This commit is contained in:
Viktor Barzin
parent
b0499a7f31
commit
c7c7047f1c
245 changed files with 11733 additions and 12432 deletions
|
|
@ -0,0 +1,183 @@
|
|||
---
|
||||
phase: 04-video-extraction-native-playback
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- internal/extractor/extractor.go
|
||||
- internal/server/server.go
|
||||
- go.mod
|
||||
- go.sum
|
||||
autonomous: true
|
||||
requirements:
|
||||
- EMBED-01
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "GET /api/streams/{id}/extract returns a JSON response with extracted video source URL and type when the stream page contains a video source"
|
||||
- "Extractor can find HLS .m3u8 URLs from <video>/<source> src attributes and script tag contents"
|
||||
- "Extractor can find DASH .mpd URLs from <video>/<source> src attributes and script tag contents"
|
||||
- "Extractor can find direct MP4/WebM URLs from <video>/<source> src attributes"
|
||||
- "Extractor can find video URLs from jwplayer, video.js, and hls.js setup calls in script tags"
|
||||
- "GET /api/streams/{id}/extract returns empty result (not error) when no video source is found"
|
||||
artifacts:
|
||||
- path: "internal/extractor/extractor.go"
|
||||
provides: "Video source URL extraction from HTML pages"
|
||||
contains: "func Extract"
|
||||
- path: "internal/server/server.go"
|
||||
provides: "API endpoint for video extraction"
|
||||
contains: "/api/streams/{id}/extract"
|
||||
key_links:
|
||||
- from: "internal/server/server.go"
|
||||
to: "internal/extractor/extractor.go"
|
||||
via: "handler calls extractor.Extract"
|
||||
pattern: "extractor\\.Extract"
|
||||
- from: "internal/server/server.go"
|
||||
to: "internal/store"
|
||||
via: "handler looks up stream by ID"
|
||||
pattern: "store.*stream.*id"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Create a backend video source extractor that fetches a stream page, parses the HTML with golang.org/x/net/html, and extracts direct video source URLs (HLS, DASH, MP4/WebM). Expose this via a new API endpoint.
|
||||
|
||||
Purpose: Enable the frontend (Plan 02) to play streams natively in an HTML5 video player instead of loading the entire third-party page in an iframe.
|
||||
Output: New `internal/extractor/` package and `GET /api/streams/{id}/extract` endpoint.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/Users/viktorbarzin/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/Users/viktorbarzin/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@internal/proxy/proxy.go
|
||||
@internal/scraper/validate.go
|
||||
@internal/server/server.go
|
||||
@internal/models/models.go
|
||||
@go.mod
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Create video source extractor package</name>
|
||||
<files>internal/extractor/extractor.go, go.mod, go.sum</files>
|
||||
<action>
|
||||
1. Add `golang.org/x/net` dependency: run `go get golang.org/x/net/html` from the project root.
|
||||
|
||||
2. Create `internal/extractor/extractor.go` with the following:
|
||||
|
||||
**Types:**
|
||||
```go
|
||||
type VideoSource struct {
|
||||
URL string `json:"url"`
|
||||
Type string `json:"type"` // "hls", "dash", "mp4", "webm", "unknown"
|
||||
}
|
||||
```
|
||||
|
||||
**Main function:** `func Extract(client *http.Client, rawURL string) ([]VideoSource, error)`
|
||||
- Fetch the page with a browser-like User-Agent (reuse the same UA string from proxy.go: `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...`)
|
||||
- If the response Content-Type is a direct video type (video/mp4, video/webm, application/x-mpegurl, application/dash+xml, etc.), return it immediately as a VideoSource without parsing HTML. Use the same `videoContentTypes` logic from scraper/validate.go but inline it here.
|
||||
- If the Content-Type is HTML, read the body (limit to 2MB like validate.go) and parse with `golang.org/x/net/html`.
|
||||
- Extraction strategies (apply all, deduplicate results):
|
||||
|
||||
a. **DOM parsing** (`golang.org/x/net/html`): Walk the HTML node tree. For `<video>` elements, check `src` attribute. For `<source>` elements inside or outside `<video>`, check `src` attribute. For `<iframe>` elements, check `src` for .m3u8/.mpd URLs.
|
||||
|
||||
b. **Script tag regex extraction**: For each `<script>` element's text content, apply these regex patterns to find URLs:
|
||||
- `.m3u8` URLs: `(?:"|')((https?://[^"'\s]+\.m3u8[^"'\s]*))`
|
||||
- `.mpd` URLs: `(?:"|')((https?://[^"'\s]+\.mpd[^"'\s]*))`
|
||||
- `.mp4` URLs: `(?:"|')((https?://[^"'\s]+\.mp4[^"'\s]*))`
|
||||
- `.webm` URLs: `(?:"|')((https?://[^"'\s]+\.webm[^"'\s]*))`
|
||||
- JWPlayer source: `file\s*:\s*["']([^"']+)` (captures the URL inside)
|
||||
- video.js/hls.js src: `src\s*[:=]\s*["']([^"']+\.m3u8[^"']*)` and similar for .mpd
|
||||
|
||||
c. **Type classification**: Determine type from URL extension:
|
||||
- `.m3u8` or Content-Type contains mpegurl -> "hls"
|
||||
- `.mpd` or Content-Type contains dash -> "dash"
|
||||
- `.mp4` -> "mp4"
|
||||
- `.webm` -> "webm"
|
||||
- Otherwise -> "unknown"
|
||||
|
||||
d. **Deduplication**: Return unique URLs only (use a map to track seen URLs).
|
||||
|
||||
e. **Priority ordering**: Return HLS sources first, then DASH, then direct video (mp4/webm). This helps the frontend pick the best option.
|
||||
|
||||
- If no video sources found, return empty slice (not nil), nil error.
|
||||
- If fetch fails, return nil, error.
|
||||
|
||||
**Helper functions** (unexported):
|
||||
- `classifyURL(u string) string` — returns "hls", "dash", "mp4", "webm", or "unknown"
|
||||
- `extractFromDOM(doc *html.Node) []VideoSource` — walks DOM tree
|
||||
- `extractFromScripts(doc *html.Node) []VideoSource` — extracts from script text content
|
||||
- `isVideoURL(u string) bool` — returns true if URL looks like a video source (.m3u8, .mpd, .mp4, .webm)
|
||||
|
||||
Follow project conventions: log.Printf with "extractor:" prefix for logging. Limit body read to 2MB. Use 3-redirect limit matching proxy pattern.
|
||||
</action>
|
||||
<verify>
|
||||
Run `cd /Users/viktorbarzin/code/infra/modules/kubernetes/f1-stream/files && go build ./internal/extractor/` to confirm the package compiles. Then run `go vet ./internal/extractor/` to check for issues.
|
||||
</verify>
|
||||
<done>
|
||||
`internal/extractor/extractor.go` exists, compiles, and exports `Extract(client, url) ([]VideoSource, error)` function. `golang.org/x/net` is in go.mod.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Add extract API endpoint to server</name>
|
||||
<files>internal/server/server.go</files>
|
||||
<action>
|
||||
1. Add import for `"f1-stream/internal/extractor"` to server.go.
|
||||
|
||||
2. In `registerRoutes()`, add a new public endpoint (no auth required, but with standard middleware):
|
||||
```go
|
||||
s.mux.Handle("GET /api/streams/{id}/extract", wrapAll(s.handleExtractVideo))
|
||||
```
|
||||
Place it near the existing `GET /api/streams/public` route.
|
||||
|
||||
3. Create handler method `func (s *Server) handleExtractVideo(w http.ResponseWriter, r *http.Request)`:
|
||||
- Get stream ID from `r.PathValue("id")`
|
||||
- Look up the stream from store. Use `s.store.LoadStreams()` to find the stream by ID (there's no GetStreamByID method, so iterate). If not found, return 404 `{"error":"stream not found"}`.
|
||||
- Only extract from published streams (if `!stream.Published`, return 404).
|
||||
- Create an HTTP client with 15-second timeout and 3-redirect limit (matching proxy pattern).
|
||||
- Call `extractor.Extract(client, stream.URL)`.
|
||||
- If error, return 502 `{"error":"failed to extract video source"}` and log the error.
|
||||
- Return JSON response:
|
||||
```json
|
||||
{"sources": [...], "stream_id": "xxx"}
|
||||
```
|
||||
Where `sources` is the `[]extractor.VideoSource` array. If empty, return `{"sources": [], "stream_id": "xxx"}`.
|
||||
|
||||
4. The handler should set appropriate cache headers: `Cache-Control: public, max-age=300` (5-minute cache since extraction is expensive). This prevents hammering the upstream site when multiple users view the same stream.
|
||||
</action>
|
||||
<verify>
|
||||
Run `cd /Users/viktorbarzin/code/infra/modules/kubernetes/f1-stream/files && go build ./...` to confirm full project compiles. Run `go vet ./...` to check for issues.
|
||||
</verify>
|
||||
<done>
|
||||
`GET /api/streams/{id}/extract` endpoint exists in server.go, calls extractor.Extract, returns JSON with sources array, compiles successfully.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
1. `go build ./...` passes without errors
|
||||
2. `go vet ./...` passes without issues
|
||||
3. `internal/extractor/extractor.go` exists and exports `Extract` and `VideoSource`
|
||||
4. `internal/server/server.go` has route `GET /api/streams/{id}/extract` registered
|
||||
5. `golang.org/x/net` appears in go.mod
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- The extractor package can parse HTML and find video source URLs from DOM elements and script contents
|
||||
- The API endpoint accepts a stream ID, fetches the page, runs extraction, and returns results
|
||||
- Empty extraction returns empty sources array (not error)
|
||||
- Full project compiles cleanly
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/04-video-extraction-native-playback/04-01-SUMMARY.md`
|
||||
</output>
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
---
|
||||
phase: 04-video-extraction-native-playback
|
||||
plan: 01
|
||||
subsystem: api
|
||||
tags: [html-parsing, video-extraction, hls, dash, golang-x-net]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 02-health-check-infrastructure
|
||||
provides: HTTP client patterns (timeout, redirect limit, user-agent)
|
||||
- phase: 03-auto-publish-pipeline
|
||||
provides: Store with stream lookup (LoadStreams)
|
||||
provides:
|
||||
- Video source extractor package (internal/extractor)
|
||||
- GET /api/streams/{id}/extract endpoint
|
||||
- VideoSource struct with URL and type classification
|
||||
affects: [04-02, frontend-video-player, native-playback]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: [golang.org/x/net/html]
|
||||
patterns: [DOM tree walking, regex script extraction, content-type detection]
|
||||
|
||||
key-files:
|
||||
created: [internal/extractor/extractor.go]
|
||||
modified: [internal/server/server.go, go.mod, go.sum]
|
||||
|
||||
key-decisions:
|
||||
- "DOM parsing with golang.org/x/net/html for structured element extraction"
|
||||
- "Regex patterns for script tag video URL extraction (HLS, DASH, JWPlayer, video.js, hls.js)"
|
||||
- "Priority ordering: HLS > DASH > MP4 > WebM for frontend source selection"
|
||||
- "5-minute cache (Cache-Control: public, max-age=300) to reduce upstream load"
|
||||
- "Empty sources array (not error) when no video found, to distinguish from fetch failures"
|
||||
|
||||
patterns-established:
|
||||
- "Extractor pattern: multiple strategies (DOM + regex) with deduplication and priority sorting"
|
||||
- "Direct content-type bypass: skip HTML parsing when response is already a video type"
|
||||
|
||||
requirements-completed: [EMBED-01]
|
||||
|
||||
# Metrics
|
||||
duration: 3min
|
||||
completed: 2026-02-17
|
||||
---
|
||||
|
||||
# Phase 04 Plan 01: Video Source Extractor Summary
|
||||
|
||||
**HTML video source extractor with DOM parsing and script regex extraction, exposed via GET /api/streams/{id}/extract endpoint with 5-minute caching**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 3 min
|
||||
- **Started:** 2026-02-17T21:42:49Z
|
||||
- **Completed:** 2026-02-17T21:46:03Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 3
|
||||
|
||||
## Accomplishments
|
||||
- Created video source extractor package that finds HLS, DASH, MP4, WebM URLs from HTML pages
|
||||
- DOM parsing extracts URLs from `<video>`, `<source>`, and `<iframe>` elements
|
||||
- Regex extraction finds video URLs from script tags including JWPlayer, video.js, and hls.js patterns
|
||||
- API endpoint returns extracted sources with type classification and priority ordering
|
||||
- Direct video content-type detection bypasses HTML parsing for efficiency
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Create video source extractor package** - `74410e2` (feat)
|
||||
2. **Task 2: Add extract API endpoint to server** - `bc9614e` (feat)
|
||||
|
||||
**Plan metadata:** (pending final docs commit)
|
||||
|
||||
## Files Created/Modified
|
||||
- `internal/extractor/extractor.go` - Video source extraction from HTML pages with DOM and regex strategies
|
||||
- `internal/server/server.go` - Added /api/streams/{id}/extract endpoint with cache headers
|
||||
- `go.mod` - Added golang.org/x/net dependency
|
||||
- `go.sum` - Updated dependency checksums
|
||||
|
||||
## Decisions Made
|
||||
- Used golang.org/x/net/html for DOM parsing (consistent with plan, deferred from Phase 2 per project decisions)
|
||||
- Implemented dual extraction strategy: structured DOM walking + regex script parsing for maximum coverage
|
||||
- Priority ordering (HLS > DASH > MP4 > WebM) helps frontend pick best playback option automatically
|
||||
- 5-minute Cache-Control header prevents hammering upstream sites when multiple users view same stream
|
||||
- Return empty sources array (not error) when no video found -- caller can distinguish "no video" from "fetch failed"
|
||||
- 15-second timeout and 3-redirect limit matching existing proxy/scraper patterns
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written.
|
||||
|
||||
## Issues Encountered
|
||||
- Go module cache permission error on default GOPATH; resolved by using temporary GOMODCACHE/GOPATH for build commands. This is a sandbox environment constraint, not a code issue.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
- Extractor package ready for Plan 02 (frontend native video player)
|
||||
- API endpoint returns structured JSON that frontend can consume to initialize HTML5 video playback
|
||||
- HLS/DASH sources will need hls.js/dash.js on the frontend for browser playback
|
||||
|
||||
---
|
||||
*Phase: 04-video-extraction-native-playback*
|
||||
*Completed: 2026-02-17*
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- FOUND: internal/extractor/extractor.go
|
||||
- FOUND: internal/server/server.go
|
||||
- FOUND: 04-01-SUMMARY.md
|
||||
- FOUND: commit 74410e2
|
||||
- FOUND: commit bc9614e
|
||||
|
|
@ -0,0 +1,218 @@
|
|||
---
|
||||
phase: 04-video-extraction-native-playback
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on:
|
||||
- 04-01
|
||||
files_modified:
|
||||
- static/index.html
|
||||
- static/js/streams.js
|
||||
autonomous: true
|
||||
requirements:
|
||||
- EMBED-02
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "When a stream has an extractable HLS source, the user sees a native video player on the app page instead of an iframe loading the third-party site"
|
||||
- "When a stream has an extractable MP4/WebM source, the user sees a native HTML5 video element playing the stream"
|
||||
- "When extraction fails or returns no sources, the user sees the existing iframe fallback (no regression)"
|
||||
- "HLS streams play using hls.js library when the browser does not support HLS natively"
|
||||
- "The video player has standard controls (play, pause, volume, fullscreen)"
|
||||
artifacts:
|
||||
- path: "static/index.html"
|
||||
provides: "HLS.js library script tag"
|
||||
contains: "hls.js"
|
||||
- path: "static/js/streams.js"
|
||||
provides: "Updated streamCard with native video player support"
|
||||
contains: "extractVideo"
|
||||
key_links:
|
||||
- from: "static/js/streams.js"
|
||||
to: "/api/streams/{id}/extract"
|
||||
via: "fetch call to extract endpoint"
|
||||
pattern: "api/streams.*extract"
|
||||
- from: "static/js/streams.js"
|
||||
to: "Hls"
|
||||
via: "HLS.js library for .m3u8 playback"
|
||||
pattern: "new Hls|Hls\\.isSupported"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Update the frontend to attempt video extraction for each stream card and render a native HTML5 video player when a direct video source is found, falling back to the existing iframe approach when extraction fails.
|
||||
|
||||
Purpose: Users watch streams in a clean native player on the app's own page without loading the third-party page, reducing exposure to ads, popups, and framing issues.
|
||||
Output: Updated `streams.js` with extraction-first rendering, HLS.js integration via CDN in `index.html`.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@/Users/viktorbarzin/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@/Users/viktorbarzin/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/04-video-extraction-native-playback/04-01-SUMMARY.md
|
||||
@static/js/streams.js
|
||||
@static/index.html
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Add HLS.js and update streamCard for native video playback</name>
|
||||
<files>static/index.html, static/js/streams.js</files>
|
||||
<action>
|
||||
**1. Add HLS.js to index.html:**
|
||||
|
||||
Add HLS.js from CDN before the existing script tags (before `utils.js`):
|
||||
```html
|
||||
<script src="https://cdn.jsdelivr.net/npm/hls.js@1/dist/hls.min.js"></script>
|
||||
```
|
||||
|
||||
**2. Update `streamCard()` in streams.js:**
|
||||
|
||||
The current `streamCard()` function renders an iframe immediately. Change the approach:
|
||||
|
||||
a. The card initially renders with a loading state placeholder instead of an iframe:
|
||||
```html
|
||||
<div class="stream-card" data-stream-id="${stream.id}">
|
||||
<div class="iframe-wrap" id="player-wrap-${stream.id}">
|
||||
<div class="loading-overlay" id="loader-${stream.id}">
|
||||
<div class="spinner"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card-bar">
|
||||
<span class="title">${escapeHtml(stream.title)}</span>
|
||||
<div class="card-actions">
|
||||
${externalBtn}
|
||||
${deleteBtn}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
b. After cards are rendered in the DOM (after `grid.innerHTML = ...`), call a new function `tryExtractVideos(streams)` that attempts extraction for each stream.
|
||||
|
||||
**3. Create `async function tryExtractVideos(streams)`:**
|
||||
|
||||
For each stream, call `tryExtractVideo(stream)` concurrently using `Promise.allSettled`.
|
||||
|
||||
**4. Create `async function tryExtractVideo(stream)`:**
|
||||
|
||||
- Fetch `GET /api/streams/${stream.id}/extract`
|
||||
- If response is OK and `sources` array is non-empty:
|
||||
- Pick the best source: prefer "hls" type, then "dash", then "mp4"/"webm"
|
||||
- Call `renderNativePlayer(stream.id, source)` to replace the loading placeholder
|
||||
- If response fails or sources is empty:
|
||||
- Call `renderIframeFallback(stream.id, stream.url)` to show the existing iframe
|
||||
|
||||
**5. Create `function renderNativePlayer(streamId, source)`:**
|
||||
|
||||
Get the wrapper element `player-wrap-${streamId}`.
|
||||
|
||||
For HLS sources (source.type === "hls"):
|
||||
```javascript
|
||||
const video = document.createElement('video');
|
||||
video.controls = true;
|
||||
video.autoplay = false;
|
||||
video.style.width = '100%';
|
||||
video.style.height = '100%';
|
||||
video.setAttribute('playsinline', '');
|
||||
|
||||
if (Hls.isSupported()) {
|
||||
const hls = new Hls();
|
||||
hls.loadSource(source.url);
|
||||
hls.attachMedia(video);
|
||||
} else if (video.canPlayType('application/vnd.apple.mpegurl')) {
|
||||
// Native HLS support (Safari)
|
||||
video.src = source.url;
|
||||
}
|
||||
```
|
||||
|
||||
For MP4/WebM sources:
|
||||
```javascript
|
||||
const video = document.createElement('video');
|
||||
video.controls = true;
|
||||
video.autoplay = false;
|
||||
video.style.width = '100%';
|
||||
video.style.height = '100%';
|
||||
video.setAttribute('playsinline', '');
|
||||
video.src = source.url;
|
||||
```
|
||||
|
||||
For DASH sources (skip for now, just fall back to iframe — DASH requires dash.js which is heavier):
|
||||
- Call `renderIframeFallback(streamId, stream.url)` instead.
|
||||
|
||||
Replace the loading overlay content with the video element. Remove the spinner. Add a "loaded" class to the loading overlay.
|
||||
|
||||
**6. Create `function renderIframeFallback(streamId, streamURL)`:**
|
||||
|
||||
Get the wrapper element and replace its innerHTML with the existing iframe markup:
|
||||
```javascript
|
||||
const wrap = document.getElementById(`player-wrap-${streamId}`);
|
||||
if (!wrap) return;
|
||||
wrap.innerHTML = `
|
||||
<div class="loading-overlay" id="loader-${streamId}">
|
||||
<div class="spinner"></div>
|
||||
</div>
|
||||
<iframe
|
||||
id="iframe-${streamId}"
|
||||
src="${escapeHtml(streamURL)}"
|
||||
sandbox="allow-scripts allow-forms allow-presentation"
|
||||
loading="lazy"
|
||||
allowfullscreen
|
||||
onload="document.getElementById('loader-${streamId}').classList.add('loaded')"
|
||||
></iframe>
|
||||
`;
|
||||
```
|
||||
|
||||
**7. Update `loadPublicStreams()` and `loadMyStreams()`:**
|
||||
|
||||
After `grid.innerHTML = streams.map(...)`, add `tryExtractVideos(streams)` call. The existing `sortStreamsByHealth` call should remain after it.
|
||||
|
||||
**Important:** Do NOT block the initial render. The cards show immediately with loading spinners. Extraction runs asynchronously and replaces the spinner with either a native player or an iframe as results come in.
|
||||
|
||||
**Error handling:** If the fetch to `/api/streams/{id}/extract` throws (network error, timeout), silently fall back to iframe. Log to console but do not show user-facing errors for extraction failures — the iframe fallback is the safety net.
|
||||
</action>
|
||||
<verify>
|
||||
1. Open `static/index.html` in a text editor and confirm the HLS.js script tag is present before other JS scripts.
|
||||
2. Open `static/js/streams.js` and confirm:
|
||||
- `streamCard()` no longer renders an iframe directly
|
||||
- `tryExtractVideos()` and `tryExtractVideo()` functions exist
|
||||
- `renderNativePlayer()` and `renderIframeFallback()` functions exist
|
||||
- `loadPublicStreams()` calls `tryExtractVideos()`
|
||||
3. Run `cd /Users/viktorbarzin/code/infra/modules/kubernetes/f1-stream/files && go build ./...` to confirm the Go project still compiles (no Go changes, but verify no accidental edits).
|
||||
</verify>
|
||||
<done>
|
||||
- HLS.js loaded via CDN in index.html
|
||||
- Stream cards attempt extraction first, render native HTML5 video player for HLS/MP4/WebM sources
|
||||
- Iframe fallback works when extraction fails or returns no sources
|
||||
- No visual regression for streams without extractable video sources
|
||||
</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
1. `static/index.html` includes HLS.js CDN script tag
|
||||
2. `static/js/streams.js` has extraction-first card rendering logic
|
||||
3. `streamCard()` renders placeholder, not iframe, as initial state
|
||||
4. `tryExtractVideo()` calls `/api/streams/{id}/extract` and handles success/failure
|
||||
5. `renderNativePlayer()` creates HTML5 `<video>` element with HLS.js for .m3u8 sources
|
||||
6. `renderIframeFallback()` creates standard iframe for non-extractable streams
|
||||
7. Full project compiles: `go build ./...`
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Streams with extractable video sources render in a native HTML5 video player with controls
|
||||
- HLS streams play via hls.js (or native HLS on Safari)
|
||||
- Streams without extractable sources fall back to iframe rendering (existing behavior preserved)
|
||||
- No user-facing errors when extraction fails — silent fallback to iframe
|
||||
- Video player has play, pause, volume, and fullscreen controls
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/04-video-extraction-native-playback/04-02-SUMMARY.md`
|
||||
</output>
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
---
|
||||
phase: 04-video-extraction-native-playback
|
||||
plan: 02
|
||||
subsystem: ui
|
||||
tags: [hls-js, html5-video, native-playback, video-extraction, frontend]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 04-video-extraction-native-playback
|
||||
provides: Video source extractor API endpoint (GET /api/streams/{id}/extract)
|
||||
provides:
|
||||
- Extraction-first stream card rendering with native HTML5 video player
|
||||
- HLS.js integration for .m3u8 playback in non-Safari browsers
|
||||
- Iframe fallback for streams without extractable video sources
|
||||
affects: [05-polish-monitoring, frontend, streaming-experience]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: [hls.js@1 (CDN)]
|
||||
patterns: [extraction-first rendering, progressive enhancement with fallback, priority-based source selection]
|
||||
|
||||
key-files:
|
||||
created: []
|
||||
modified: [static/index.html, static/js/streams.js]
|
||||
|
||||
key-decisions:
|
||||
- "HLS.js loaded from CDN (jsdelivr) to avoid bundling complexity"
|
||||
- "Extraction runs async after card render -- loading spinner shows immediately, player replaces it"
|
||||
- "DASH sources fall back to iframe (dash.js too heavy for current scope)"
|
||||
- "pickBestSource priority: HLS > DASH > MP4 > WebM matches backend ordering"
|
||||
- "Silent console.log on extraction failure -- no user-facing errors for extraction issues"
|
||||
|
||||
patterns-established:
|
||||
- "Progressive enhancement pattern: render placeholder, attempt extraction, upgrade to native or fall back to iframe"
|
||||
- "Promise.allSettled for concurrent extraction across all stream cards"
|
||||
|
||||
requirements-completed: [EMBED-02]
|
||||
|
||||
# Metrics
|
||||
duration: 2min
|
||||
completed: 2026-02-17
|
||||
---
|
||||
|
||||
# Phase 04 Plan 02: Frontend Native Video Playback Summary
|
||||
|
||||
**Extraction-first stream card rendering with HLS.js integration and iframe fallback for native HTML5 video playback**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 2 min
|
||||
- **Started:** 2026-02-17T21:48:20Z
|
||||
- **Completed:** 2026-02-17T21:50:03Z
|
||||
- **Tasks:** 1
|
||||
- **Files modified:** 2
|
||||
|
||||
## Accomplishments
|
||||
- Stream cards now attempt video extraction before falling back to iframe
|
||||
- Native HTML5 video player renders for HLS, MP4, and WebM sources with standard controls
|
||||
- HLS.js handles .m3u8 streams in non-Safari browsers; Safari uses native HLS support
|
||||
- Iframe fallback preserves existing behavior for streams without extractable sources
|
||||
- Loading spinners provide visual feedback during async extraction
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Add HLS.js and update streamCard for native video playback** - `2a40af9` (feat)
|
||||
|
||||
**Plan metadata:** (pending final docs commit)
|
||||
|
||||
## Files Created/Modified
|
||||
- `static/index.html` - Added HLS.js CDN script tag before other JS scripts
|
||||
- `static/js/streams.js` - Extraction-first rendering: streamCard renders placeholder, tryExtractVideos/tryExtractVideo call extract API, renderNativePlayer creates HTML5 video element, renderIframeFallback preserves existing iframe approach
|
||||
|
||||
## Decisions Made
|
||||
- HLS.js loaded from jsDelivr CDN rather than self-hosted -- avoids build tooling while keeping the library current
|
||||
- DASH sources intentionally fall back to iframe -- dash.js is heavier and DASH is lower priority than HLS
|
||||
- Extraction errors logged to console only -- user sees iframe fallback seamlessly, no error UI needed
|
||||
- pickBestSource uses same priority ordering (HLS > DASH > MP4 > WebM) established in backend extractor
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written.
|
||||
|
||||
## Issues Encountered
|
||||
- Go module cache sandbox permission error during build verification; resolved with temporary GOPATH (same workaround as 04-01, environment constraint only)
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required.
|
||||
|
||||
## Next Phase Readiness
|
||||
- Phase 04 (Video Extraction & Native Playback) is now complete
|
||||
- Streams with extractable video sources play in native HTML5 player
|
||||
- Streams without extractable sources continue to work via iframe fallback
|
||||
- Ready for Phase 05 (Polish & Monitoring) if planned
|
||||
|
||||
---
|
||||
*Phase: 04-video-extraction-native-playback*
|
||||
*Completed: 2026-02-17*
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- FOUND: static/index.html
|
||||
- FOUND: static/js/streams.js
|
||||
- FOUND: 04-02-SUMMARY.md
|
||||
- FOUND: commit 2a40af9
|
||||
|
|
@ -0,0 +1,108 @@
|
|||
---
|
||||
phase: 04-video-extraction-native-playback
|
||||
verified: 2026-02-17T22:00:00Z
|
||||
status: passed
|
||||
score: 11/11 must-haves verified
|
||||
re_verification: false
|
||||
---
|
||||
|
||||
# Phase 4: Video Extraction and Native Playback Verification Report
|
||||
|
||||
**Phase Goal:** When a stream URL contains an extractable video source, users watch it in a clean native HTML5 player instead of loading the third-party page
|
||||
|
||||
**Verified:** 2026-02-17T22:00:00Z
|
||||
|
||||
**Status:** passed
|
||||
|
||||
**Re-verification:** No — initial verification
|
||||
|
||||
## Goal Achievement
|
||||
|
||||
### Observable Truths
|
||||
|
||||
| # | Truth | Status | Evidence |
|
||||
|---|-------|--------|----------|
|
||||
| 1 | GET /api/streams/{id}/extract returns a JSON response with extracted video source URL and type when the stream page contains a video source | ✓ VERIFIED | Endpoint exists in server.go:83, handler at server.go:242-295 returns JSON with sources array and stream_id |
|
||||
| 2 | Extractor can find HLS .m3u8 URLs from video/source src attributes and script tag contents | ✓ VERIFIED | DOM extraction at extractor.go:166-192, regex patterns at extractor.go:196,201, matches .m3u8 in video/source/iframe src |
|
||||
| 3 | Extractor can find DASH .mpd URLs from video/source src attributes and script tag contents | ✓ VERIFIED | DOM extraction at extractor.go:166-192, regex patterns at extractor.go:197,202, matches .mpd in video/source/iframe src |
|
||||
| 4 | Extractor can find direct MP4/WebM URLs from video/source src attributes | ✓ VERIFIED | DOM extraction at extractor.go:166-192, regex patterns at extractor.go:198-199, matches .mp4/.webm in video/source/iframe src |
|
||||
| 5 | Extractor can find video URLs from jwplayer, video.js, and hls.js setup calls in script tags | ✓ VERIFIED | Regex patterns at extractor.go:200-202 for JWPlayer (file:), hls.js/video.js (src:=), script extraction at extractor.go:206-223 |
|
||||
| 6 | GET /api/streams/{id}/extract returns empty result (not error) when no video source is found | ✓ VERIFIED | Returns []VideoSource{} at extractor.go:66,306 with nil error when no sources found |
|
||||
| 7 | When a stream has an extractable HLS source, the user sees a native video player on the app page instead of an iframe loading the third-party site | ✓ VERIFIED | tryExtractVideo at streams.js:172-189 fetches extract endpoint, renderNativePlayer at streams.js:200-228 creates HTML5 video element |
|
||||
| 8 | When a stream has an extractable MP4/WebM source, the user sees a native HTML5 video element playing the stream | ✓ VERIFIED | renderNativePlayer at streams.js:200-228 handles mp4/webm by setting video.src directly (line 223) |
|
||||
| 9 | When extraction fails or returns no sources, the user sees the existing iframe fallback (no regression) | ✓ VERIFIED | tryExtractVideo calls renderIframeFallback at streams.js:188 on error/empty sources, renderIframeFallback at streams.js:230-246 creates iframe element |
|
||||
| 10 | HLS streams play using hls.js library when the browser does not support HLS natively | ✓ VERIFIED | HLS.js loaded from CDN at index.html:162, renderNativePlayer checks Hls.isSupported() at streams.js:212, creates new Hls() at streams.js:213-215, Safari native fallback at streams.js:216-217 |
|
||||
| 11 | The video player has standard controls (play, pause, volume, fullscreen) | ✓ VERIFIED | video.controls = true at streams.js:205, HTML5 video element provides standard browser controls including play, pause, volume, fullscreen |
|
||||
|
||||
**Score:** 11/11 truths verified
|
||||
|
||||
### Required Artifacts
|
||||
|
||||
| Artifact | Expected | Status | Details |
|
||||
|----------|----------|--------|---------|
|
||||
| `internal/extractor/extractor.go` | Video source URL extraction from HTML pages | ✓ VERIFIED | 326 lines, exports Extract function and VideoSource type, implements DOM parsing with golang.org/x/net/html and regex extraction from script tags |
|
||||
| `internal/server/server.go` | API endpoint for video extraction | ✓ VERIFIED | Route registered at line 83, handler at lines 242-295, calls extractor.Extract, returns JSON with sources array and Cache-Control headers |
|
||||
| `static/index.html` | HLS.js library script tag | ✓ VERIFIED | HLS.js CDN script tag at line 162 before other JS scripts |
|
||||
| `static/js/streams.js` | Updated streamCard with native video player support | ✓ VERIFIED | 398 lines total, includes tryExtractVideos (168-170), tryExtractVideo (172-189), renderNativePlayer (200-228), renderIframeFallback (230-246), pickBestSource (191-198) |
|
||||
|
||||
### Key Link Verification
|
||||
|
||||
| From | To | Via | Status | Details |
|
||||
|------|----|----|--------|---------|
|
||||
| internal/server/server.go | internal/extractor/extractor.go | handler calls extractor.Extract | ✓ WIRED | Import at server.go:13, call at server.go:282 with client and streamURL parameters |
|
||||
| internal/server/server.go | internal/store | handler looks up stream by ID | ✓ WIRED | Import at server.go:16, LoadStreams() call at server.go:246, iterates to find stream by ID at lines 255-264 |
|
||||
| static/js/streams.js | /api/streams/{id}/extract | fetch call to extract endpoint | ✓ WIRED | fetch call at streams.js:174 with template literal for stream ID, response parsed as JSON at line 176 |
|
||||
| static/js/streams.js | Hls | HLS.js library for .m3u8 playback | ✓ WIRED | Hls.isSupported() check at streams.js:212, new Hls() instantiation at streams.js:213, loadSource/attachMedia at streams.js:214-215 |
|
||||
|
||||
### Requirements Coverage
|
||||
|
||||
| Requirement | Source Plan | Description | Status | Evidence |
|
||||
|-------------|------------|-------------|--------|----------|
|
||||
| EMBED-01 | 04-01-PLAN.md | Proxy fetches stream page and attempts to extract direct video source URL (HLS .m3u8, DASH .mpd, direct MP4/WebM, or embedded video player source) | ✓ SATISFIED | Extractor package at internal/extractor/extractor.go implements all extraction types: HLS (lines 107,117,147,196,201,229), DASH (lines 109,120,148,197,202,235), MP4 (lines 111,149,198), WebM (lines 113,150,199), JWPlayer/video.js/hls.js patterns (lines 200-202). Server-side extraction via handleExtractVideo at server.go:242-295 |
|
||||
| EMBED-02 | 04-02-PLAN.md | When direct video source is found, render it in a minimal HTML5 video player on the app's own page (no third-party page loaded) | ✓ SATISFIED | renderNativePlayer at streams.js:200-228 creates HTML5 video element with controls=true (line 205), integrates HLS.js for .m3u8 sources (lines 212-215), sets src directly for mp4/webm (line 223), replaces loading placeholder in player-wrap div preventing third-party page load |
|
||||
|
||||
### Anti-Patterns Found
|
||||
|
||||
None detected.
|
||||
|
||||
Scanned files:
|
||||
- `internal/extractor/extractor.go`: No TODO/FIXME/placeholder comments, no empty implementations, substantive extraction logic
|
||||
- `internal/server/server.go`: Handler implementation complete, proper error handling, cache headers set
|
||||
- `static/js/streams.js`: No TODO/FIXME/placeholder comments, complete extraction-first rendering logic
|
||||
- `static/index.html`: HLS.js script tag present, no issues
|
||||
|
||||
### Human Verification Required
|
||||
|
||||
None. All verification completed programmatically.
|
||||
|
||||
### Phase Summary
|
||||
|
||||
Phase 4 successfully delivers video extraction and native playback capability. The backend extractor can identify HLS, DASH, MP4, and WebM sources from both DOM elements and script tags. The frontend attempts extraction first and upgrades to a native HTML5 video player when sources are found, falling back to iframe rendering when extraction fails or returns no results.
|
||||
|
||||
**Key accomplishments:**
|
||||
- Server-side video source extraction with multiple strategies (DOM parsing + regex script extraction)
|
||||
- Native HTML5 video player with HLS.js integration for .m3u8 streams
|
||||
- Progressive enhancement pattern: render placeholder, attempt extraction, upgrade or fallback
|
||||
- No breaking changes to existing iframe fallback behavior
|
||||
- 5-minute cache on extraction endpoint to reduce upstream load
|
||||
- Priority-based source selection (HLS > DASH > MP4 > WebM)
|
||||
|
||||
**Technical quality:**
|
||||
- All artifacts exist, substantive, and properly wired
|
||||
- No anti-patterns detected
|
||||
- Consistent error handling with silent fallback to iframe
|
||||
- User-Agent and timeout patterns match existing proxy/scraper conventions
|
||||
- Proper use of golang.org/x/net/html for DOM parsing
|
||||
- HLS.js loaded from CDN with browser capability detection
|
||||
|
||||
**Requirements fulfilled:**
|
||||
- EMBED-01: Video source extraction from stream pages ✓
|
||||
- EMBED-02: Native HTML5 video player rendering ✓
|
||||
|
||||
Phase goal achieved. Users can now watch streams with extractable video sources in a clean native player without loading third-party pages.
|
||||
|
||||
---
|
||||
|
||||
_Verified: 2026-02-17T22:00:00Z_
|
||||
|
||||
_Verifier: Claude (gsd-verifier)_
|
||||
Loading…
Add table
Add a link
Reference in a new issue