Commit graph

1763 commits

Author SHA1 Message Date
Viktor Barzin
2316ac752a
fix: replace removed woodpeckerci/plugin-slack with curl-based webhook 2026-02-28 18:25:23 +00:00
Viktor Barzin
a5dff85f02
trigger CI: test BuildKit caching 2026-02-28 18:23:50 +00:00
Viktor Barzin
91fcfa02a6
[ci skip] add network visualization implementation plan 2026-02-28 18:19:36 +00:00
Viktor Barzin
df0348d4f9
[ci skip] add network traffic visualization design doc 2026-02-28 18:14:42 +00:00
Viktor Barzin
0ebb259ade
trigger CI: test BuildKit caching for build-cli pipeline 2026-02-28 18:04:23 +00:00
Viktor Barzin
d232cf72c2
[ci skip] add private registry to Terraform cloud-init provisioning 2026-02-28 17:57:24 +00:00
Viktor Barzin
038974e0c7
[ci skip] add registry-private service to Docker Compose stack 2026-02-28 17:57:04 +00:00
Viktor Barzin
d38d2493ca
[ci skip] add nginx upstream and server block for private registry on port 5050 2026-02-28 17:57:03 +00:00
Viktor Barzin
b9ad1a4e17
[ci skip] add BuildKit layer caching and dual-push to f1-stream pipeline 2026-02-28 17:56:56 +00:00
Viktor Barzin
7f8d9a6064
[ci skip] add BuildKit layer caching and dual-push to build-cli pipeline 2026-02-28 17:56:52 +00:00
Viktor Barzin
d47737fc4e
[ci skip] add private R/W registry config for CI build caching 2026-02-28 17:56:50 +00:00
Viktor Barzin
4606ae3a3c
[ci skip] update CI caching plan: add Terraform provisioning for private registry 2026-02-28 17:51:55 +00:00
Viktor Barzin
34315d88f0
[ci skip] add CI build caching implementation plan 2026-02-28 17:46:44 +00:00
Viktor Barzin
d2a994eb80
[ci skip] add CI build caching design doc 2026-02-28 17:43:42 +00:00
Viktor Barzin
3f558bd4da
[ci skip] install CloudNativePG operator as platform module
- CNPG v0.27.1 operator in cnpg-system namespace
- CRDs installed: clusters, backups, poolers, databases, etc.
- local-path StorageClass already exists (from cloud-init template)
- Prerequisite for PostgreSQL migration off NFS
2026-02-28 17:22:53 +00:00
Viktor Barzin
5318761336
[ci skip] fix OOM crashes: add resource limits for osrm-bicycle, aiostreams, listenarr, authentik
- osrm-bicycle: 1Gi limit (loads 403MB routing graph)
- aiostreams: 768Mi limit (loads 44K anime entries)
- listenarr: 1Gi limit (.NET + Playwright/Chromium)
- authentik server: 1Gi limit, worker: 1Gi limit (Django + gunicorn)
- servarr: pass nfs_server variable to all submodules
2026-02-28 17:03:33 +00:00
Viktor Barzin
de4dffbab7
[ci skip] nextcloud: increase resource limits to prevent OOM crash loop
Default LimitRange (256Mi) was too low — pod was using 227Mi/256Mi and
getting OOM killed under sync client load, causing 500s and blank web UI.
2026-02-28 16:26:19 +00:00
Viktor Barzin
6e1586342c
[ci skip] expand k8s worker nodes to 256G, update inventory and extend script
- k8s-node2: 128G → 256G (160GB free)
- k8s-node3: 128G → 256G (135GB free)
- k8s-node4: 128G → 256G (127GB free)
- k8s-node1: already 256G (51GB free)
- extend_vm_storage.sh: increase drain timeout to 300s, add --force flag
- Remove Vaultwarden from SQLite migration plan (too risky)
2026-02-28 16:00:16 +00:00
Viktor Barzin
6cc1da4bd6
[ci skip] revise storage reliability design based on research agent findings
Key changes from v1:
- Drop 3-instance replication → 2-instance CNPG, single Redis/MySQL
- Remove Headscale from PG migration (project discourages it)
- Remove MeshCentral from PG migration (NeDB, not SQLite)
- Replace Redis Sentinel with single redis:7 on local disk (modules unused)
- Add RAM overcommit warning and mitigation
- Add explicit single-host limitation acknowledgment
- Add per-component rollback plans
- Fix backup strategy (CNPG can't archive WAL to NFS natively)
- Reorder migration: low-risk services first, authentik last
- Add research gate before each service migration
2026-02-28 14:38:01 +00:00
Viktor Barzin
6381bcee40
[ci skip] add storage reliability design: DB replication + SQLite consolidation 2026-02-28 14:24:42 +00:00
Viktor Barzin
2bbf52ac9b
[ci skip] audiblez-web: switch from digest to tag for CI-driven deploys
Woodpecker CI pipeline now pushes tagged images and patches the
deployment with the build number tag. Using :latest as the Terraform
baseline so CI can override with specific build tags.
2026-02-28 14:17:19 +00:00
Viktor Barzin
6aa29e9f77
[ci skip] technitium: add primary-secondary DNS HA with AXFR zone replication
Secondary instance on a separate node replicates all zones from primary via
zone transfer. LoadBalancer routes DNS queries to both pods. PDB ensures at
least 1 DNS pod survives voluntary disruptions. Setup job automates zone
transfer enablement and secondary zone creation via Technitium REST API.
2026-02-28 14:14:20 +00:00
Viktor Barzin
098c0eb752
[ci skip] update claude knowledge: never restart NFS, NFS export dir prereq 2026-02-28 12:20:36 +00:00
Viktor Barzin
16343475b1
[ci skip] VPA: reduce LimitRange defaults, add overcommit check, protect tier-0
- Reduce Kyverno LimitRange default limits ~4x across all tiers to fix
  800-900% memory overcommitment on worker nodes
- Add cluster health check #25: per-node resource overcommitment
  showing requests and limits vs allocatable capacity
- Add Kyverno policy for Goldilocks VPA mode by tier: tier-0 namespaces
  get VPA Off mode (recommend only, no evictions) to prevent downtime
  on critical infra (traefik, cloudflared, authentik, technitium, etc.)
- Non-tier-0 namespaces get VPA Auto mode for active right-sizing
2026-02-26 23:15:43 +00:00
Viktor Barzin
a1b669ae97
[ci skip] Deploy VPA + Goldilocks for dynamic resource right-sizing
Add Vertical Pod Autoscaler (recommender, updater, admission-controller)
and Goldilocks dashboard to monitor resource recommendations across all
namespaces. Dashboard at goldilocks.viktorbarzin.me behind Authentik.
2026-02-25 21:54:01 +00:00
Viktor Barzin
a0799d525d
[ci skip] poison-fountain: fix single point of failure causing transient service outages
- Scale to 2 replicas with RollingUpdate (maxUnavailable=0)
- Add topology spread constraint to place pods on different nodes
- Switch from single-threaded to ThreadingMixIn HTTP server so tarpit
  slow-drip requests no longer block /auth and /healthz endpoints
2026-02-25 21:05:14 +00:00
Viktor Barzin
9caec43351
[ci skip] rybbit: increase clickhouse memory limit to fix OOMKilled crash loop 2026-02-25 20:53:08 +00:00
Viktor Barzin
ef158ed431
[ci skip] kyverno: scale to 2 replicas, eliminate API calls from policies
- Scale admission controller to 2 replicas with topology spread across nodes
- Rewrite inject-priority-class-from-tier: use namespaceSelector instead of
  API call per pod admission (eliminates Kyverno→API server round-trip)
- Rewrite sync-tier-label-from-namespace: same namespaceSelector approach
- Extract governance_tiers local to DRY up tier definitions
2026-02-24 23:09:56 +00:00
Viktor Barzin
900f72d60d
[ci skip] Fix Woodpecker GitHub forge: add explicit GITHUB_URL to prevent Forgejo URL bleed
When both WOODPECKER_GITHUB and WOODPECKER_FORGEJO are enabled without
an explicit WOODPECKER_GITHUB_URL, the GitHub forge inherits the Forgejo
URL causing all GitHub API calls to hit forgejo.viktorbarzin.me with
GitHub OAuth credentials, resulting in 401 Unauthorized on repo add and
cron jobs. Also adds Forgejo forge variables to Terraform.
2026-02-24 23:02:33 +00:00
Viktor Barzin
882df4cc5c
[ci skip] kyverno: fix crash loop — failurePolicy Ignore, increase memory, pin chart
Admission controller was restarting every ~5min due to API server timeouts
causing leader election loss. failurePolicy:Fail meant the webhook blocked
all pod creation cluster-wide when Kyverno was unavailable.
2026-02-24 23:00:45 +00:00
Viktor Barzin
c06cca288a
[ci skip] fix cluster health: GPU tolerations, actualbudget nfs_server, AuthentikDown alert
- Add missing nvidia.com/gpu toleration to ollama and yt-highlights deployments
- Add node_selector gpu=true to ollama deployment
- Pass nfs_server variable through to actualbudget factory modules
- Fix AuthentikDown alert to match actual deployment name (goauthentik-server)
2026-02-24 22:55:58 +00:00
Viktor Barzin
18a873a630
[ci skip] wrongmove dashboard: add per-path latency table, fix layout, sort top offenders
Add "Per-Path Latency Breakdown" table with p50/p95/p99 and request rate
per endpoint. Fix bar gauge position to sit next to timeseries. Add sort
transformation to "Top Offenders (Avg Duration)" panel.
2026-02-24 22:31:41 +00:00
Viktor Barzin
2327c73154
[ci skip] f1-stream: update project state - all 8 phases complete 2026-02-24 00:28:54 +00:00
Viktor Barzin
650d785dcd
[ci skip] f1-stream: use v5.0.0 tag to bypass stale pull-through cache 2026-02-24 00:28:12 +00:00
Viktor Barzin
6d88a5df1e
f1-stream: fix frontend routing with catch-all handler for SvelteKit SPA 2026-02-24 00:18:28 +00:00
Viktor Barzin
0dc3fec3f9
f1-stream: fix SvelteKit routing - add trailingSlash for static adapter 2026-02-24 00:11:44 +00:00
Viktor Barzin
fd6a8bb6f2
f1-stream: add pydantic dependency and trigger CI build 2026-02-24 00:00:02 +00:00
Viktor Barzin
a050db616e
[ci skip] f1-stream: add CDN token refresh, SvelteKit frontend, multi-stream layout (Phases 6-8)
- Phase 6: CDN token lifecycle with 3-strategy URL matching and periodic refresh
- Phase 7: SvelteKit 2/Svelte 5 frontend with schedule calendar and hls.js player
- Phase 8: Multi-stream layout supporting up to 4 simultaneous HLS streams
- Update Dockerfile to multi-stage build (Node.js frontend + Python backend)
- Switch deployment to :latest tag with Always pull policy for CI-driven deploys
- Update Woodpecker CI to use explicit latest tag
2026-02-23 23:59:35 +00:00
Viktor Barzin
9bf0523ea9
[ci skip] f1-stream: add stream health checker and HLS proxy (Phases 4-5)
Phase 4 - Stream Health and Fallback:
- StreamHealthChecker with partial GET validation of m3u8 content
- Bitrate extraction from BANDWIDTH tags
- Response time measurement for quality ranking
- Fallback ordering: live first, fastest response time first
- GET /streams now only returns health-verified streams

Phase 5 - HLS Proxy Core:
- GET /proxy?url= - m3u8 playlist fetch with full URI rewriting
- GET /relay?url= - chunked segment relay (never buffers full segment)
- m3u8 rewriter handles master, variant, and segment URIs
- Base64url encoding for URL parameters
- CORS middleware for browser playback
- Range header forwarding for seeking support
2026-02-23 23:41:16 +00:00
Viktor Barzin
00d9458dec
[ci skip] trim CLAUDE.md: remove discoverable info, deduplicate 2026-02-23 23:10:13 +00:00
Viktor Barzin
b29f5ddb06
[ci skip] f1-stream: add extractor framework with demo streams (Phase 3)
- BaseExtractor ABC with health_check method
- ExtractorRegistry with concurrent fan-out extraction
- ExtractionService with in-memory cache and background polling
- DemoExtractor with 3 public HLS test streams
- Adaptive polling: 5min during live sessions, 30min otherwise
- GET /streams, GET /extractors, POST /extract endpoints
2026-02-23 23:02:56 +00:00
Viktor Barzin
4fd3e2d770
[ci skip] f1-stream: add gitignore for __pycache__, remove committed .pyc 2026-02-23 22:55:38 +00:00
Viktor Barzin
d7d347de27
[ci skip] f1-stream: add F1 schedule subsystem (Phase 2)
- Fetch 2026 F1 race calendar from jolpica API with all sessions
  (FP1-3, Qualifying, Sprint, Race) and UTC timestamps
- Persist schedule to NFS as JSON, load on startup if fresh
- APScheduler daily refresh at 03:00 UTC
- GET /schedule endpoint with live/upcoming/past session status
- POST /schedule/refresh for manual refresh trigger
2026-02-23 22:55:13 +00:00
Viktor Barzin
f423a4d60c
[ci skip] f1-stream: replace Go service with Python/FastAPI skeleton
Replaces the existing Go-based f1-stream service with a new Python/FastAPI
backend as the foundation for the rebuilt F1 streaming aggregation service.

- New FastAPI backend with health and root endpoints
- Python 3.13 slim Dockerfile (replaces Go multi-stage build)
- Updated Terraform deployment (port 8000, reduced resources)
- Buildx-based redeploy.sh with --platform linux/amd64
- Added Woodpecker CI pipeline for automated builds
- Removed all old Go source, node_modules, static assets
2026-02-23 22:47:06 +00:00
Viktor Barzin
c665544f41
[ci skip] fix plotting-book: add SESSION_SECRET env var
Session secret stored in encrypted terraform.tfvars, referenced
via variable to avoid committing secrets in plain text.
2026-02-23 22:44:04 +00:00
Viktor Barzin
85f88bf167
[ci skip] platform: add ndots=2 dns_config to all deployment pod specs
Prevents Terraform from reverting the Kyverno inject-ndots mutation
on every apply. 23 pod specs across 19 platform module files.
2026-02-23 22:43:05 +00:00
Viktor Barzin
a2a83d30aa
[ci skip] monitoring: increase resource quota limits
Bump limits.cpu 80→120 and limits.memory 160Gi→240Gi to provide
headroom. Previous values were at 87% and 92% utilization.
2026-02-23 22:42:30 +00:00
Viktor Barzin
e982a8ad81
[ci skip] fix redis OOMKilled: increase memory limits to 2Gi
Redis was CrashLoopBackOff due to OOMKilled - 512Mi limit was
insufficient for 650MB RDB dataset plus redis-stack modules.
2026-02-23 22:37:56 +00:00
Viktor Barzin
2789c0fa5c
[ci skip] add trading-bot Terraform stack 2026-02-23 22:29:59 +00:00
Viktor Barzin
a2a3f8027c
docs(01-infrastructure-and-deployment): create phase plan 2026-02-23 22:28:54 +00:00