stem95su: scheduled Drive->site sync CronJob (every 10m)
CronJob stem95su-gdrive-sync (*/10) mounts the content PVC RW and rclone-syncs the read-only Drive folder "claude" (stem claude/files) onto it (rclone/rclone:1.74.3, scope=drive.readonly, empty-source guard + --max-delete 25). ESO ExternalSecret stem95su-rclone <- Vault secret/stem95su. Requires the GCP OAuth app published to Production or the refresh token expires ~weekly. Lands the gdrive-sync stack on master (it had landed on a feature branch by accident on the shared devvm checkout). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin
parent
05b50d2b96
commit
6d224861c4
1168 changed files with 120 additions and 358547 deletions
|
|
@ -1,522 +0,0 @@
|
|||
---
|
||||
name: setup-project
|
||||
description: |
|
||||
Deploy a new self-hosted service to the Kubernetes cluster from a GitHub repository.
|
||||
Use when: (1) User provides a GitHub URL or project name and wants to deploy it,
|
||||
(2) User says "deploy [service]" or "set up [service]",
|
||||
(3) User wants to add a new service to the cluster.
|
||||
Automated workflow: Docker image → Terraform module → Deploy.
|
||||
Handles database setup, ingress, DNS configuration.
|
||||
author: Claude Code
|
||||
version: 1.0.0
|
||||
date: 2025-01-01
|
||||
---
|
||||
|
||||
# Setup Project Skill
|
||||
|
||||
**Purpose**: Deploy a new self-hosted service to the Kubernetes cluster from a GitHub repository.
|
||||
|
||||
**When to use**: User provides a GitHub URL or project name and wants to deploy it to the cluster.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Research Phase
|
||||
|
||||
**Input**: GitHub repository URL or project name
|
||||
|
||||
**Actions**:
|
||||
- Visit the GitHub repository
|
||||
- Check the README for:
|
||||
- Official Docker image (Docker Hub, ghcr.io, etc.)
|
||||
- docker-compose.yml file
|
||||
- Self-hosting documentation
|
||||
- Required dependencies (PostgreSQL, MySQL, Redis, etc.)
|
||||
- Environment variables needed
|
||||
- Default ports
|
||||
- Storage requirements
|
||||
|
||||
**Find Docker Image Priority**:
|
||||
1. Check official documentation for recommended image
|
||||
2. Look in docker-compose.yml for `image:` directive
|
||||
3. Check GitHub Container Registry: `ghcr.io/<org>/<repo>`
|
||||
4. Check Docker Hub: `<org>/<repo>`
|
||||
5. Check releases page for container images
|
||||
6. Last resort: Build from Dockerfile (avoid if possible)
|
||||
|
||||
**Classify Dockerfile State** (drives whether we contribute a PR back upstream later):
|
||||
|
||||
| State | When | Action on deploy success |
|
||||
|---|---|---|
|
||||
| `image-used` | An official/community image worked (priority 1-5). | No upstream PR. Default case. |
|
||||
| `used-as-is` | Upstream ships a Dockerfile; it built and ran fine. | No upstream PR. |
|
||||
| `fixed-broken-upstream` | Upstream Dockerfile exists but fails to build / run; we patched it. | Open a `fix-dockerfile` PR after stability gate. |
|
||||
| `written-from-scratch` | Upstream has no Dockerfile at all; we authored one. | Open an `add-dockerfile` PR after stability gate. |
|
||||
|
||||
Record the chosen state and supporting metadata in `modules/kubernetes/<service>/.contribution-state.json`. When we author or fix a Dockerfile, also write `modules/kubernetes/<service>/files/Dockerfile`, `.dockerignore`, and `BUILD.md` (from `templates/Dockerfile.README.md`) — these travel with the upstream PR.
|
||||
|
||||
```json
|
||||
{
|
||||
"upstream_repo": "owner/name",
|
||||
"dockerfile_state": "written-from-scratch",
|
||||
"dockerfile_path_in_infra": "modules/kubernetes/<service>/files/Dockerfile",
|
||||
"deploy_target_url": "https://<service>.viktorbarzin.me",
|
||||
"image_tag": "registry.viktorbarzin.me/<service>:<sha>",
|
||||
"image_size": "<MB>",
|
||||
"base_image": "<e.g. python:3.12-slim>",
|
||||
"dockerfile_shape": "multi-stage, non-root, linux/amd64",
|
||||
"deploy_verified_at": null,
|
||||
"contribution_pr_url": null
|
||||
}
|
||||
```
|
||||
|
||||
**Dockerfile quality bar** (when writing one ourselves — enforced before PR):
|
||||
- Multi-stage build where it makes sense (Node, Go, Rust, Python with compiled deps).
|
||||
- Explicit non-root `USER`.
|
||||
- `HEALTHCHECK` when the app exposes a known endpoint.
|
||||
- Minimal base image (alpine / distroless preferred; `-slim` otherwise).
|
||||
- No secrets baked in; runtime config via `ENV`.
|
||||
- `.dockerignore` that excludes `.git`, `node_modules`, test artifacts.
|
||||
|
||||
**Extract Configuration**:
|
||||
- Container port (default port the app listens on)
|
||||
- Environment variables (DATABASE_URL, REDIS_HOST, SMTP, etc.)
|
||||
- Volume mounts (what data needs persistence)
|
||||
- Dependencies (database type, cache, etc.)
|
||||
|
||||
### 2. Database Setup (if needed)
|
||||
|
||||
**If project requires PostgreSQL**:
|
||||
- User provides database credentials or use pattern: `<service>` user with secure password
|
||||
- Database will be created in shared `postgresql.dbaas.svc.cluster.local`
|
||||
- Connection string format: `postgresql://<user>:<password>@postgresql.dbaas.svc.cluster.local:5432/<dbname>`
|
||||
|
||||
**If project requires MySQL**:
|
||||
- User provides database credentials
|
||||
- Database in shared `mysql.dbaas.svc.cluster.local`
|
||||
- Connection string format: `mysql://<user>:<password>@mysql.dbaas.svc.cluster.local:3306/<dbname>`
|
||||
|
||||
**If project requires Redis**:
|
||||
- Use shared Redis: `redis.redis.svc.cluster.local:6379`
|
||||
- No password required
|
||||
|
||||
**IMPORTANT**: Never create databases yourself - always ask user for credentials to use.
|
||||
|
||||
### 3. NFS Storage Setup (if service needs persistent data)
|
||||
|
||||
**IMPORTANT**: NFS directories must exist and be exported on the NFS server BEFORE deploying the service. If the directory doesn't exist, the pod will fail to mount the volume and get stuck in `ContainerCreating`.
|
||||
|
||||
**Steps**:
|
||||
|
||||
1. **Create the directory on the NFS server**:
|
||||
```bash
|
||||
ssh root@10.0.10.15 'mkdir -p /mnt/main/<service> && chmod 777 /mnt/main/<service>'
|
||||
```
|
||||
|
||||
2. **Export the directory via TrueNAS**:
|
||||
- The NFS export must be configured in TrueNAS so Kubernetes nodes can mount it
|
||||
- Create the export via TrueNAS WebUI or API, allowing access from the Kubernetes network (10.0.20.0/24)
|
||||
- Verify the export is accessible:
|
||||
```bash
|
||||
# From a k8s node or the dev VM
|
||||
showmount -e 10.0.10.15 | grep <service>
|
||||
```
|
||||
|
||||
3. **Verify the mount works before proceeding**:
|
||||
```bash
|
||||
# Quick test from a k8s node
|
||||
ssh root@10.0.20.100 'mount -t nfs 10.0.10.15:/mnt/main/<service> /tmp/test-mount && ls /tmp/test-mount && umount /tmp/test-mount'
|
||||
```
|
||||
|
||||
**Only proceed to Terraform module creation after confirming the NFS export is accessible.**
|
||||
|
||||
### 4. Terraform Module Creation
|
||||
|
||||
**Create module directory**:
|
||||
```bash
|
||||
mkdir -p modules/kubernetes/<service-name>/
|
||||
```
|
||||
|
||||
**Create `modules/kubernetes/<service-name>/main.tf`**:
|
||||
|
||||
```hcl
|
||||
variable "tls_secret_name" {}
|
||||
variable "tier" { type = string }
|
||||
variable "postgresql_password" {} # Only if needed
|
||||
# Add other variables as needed (smtp_password, api_keys, etc.)
|
||||
|
||||
resource "kubernetes_namespace" "<service>" {
|
||||
metadata {
|
||||
name = "<service>"
|
||||
}
|
||||
}
|
||||
|
||||
module "tls_secret" {
|
||||
source = "../setup_tls_secret"
|
||||
namespace = kubernetes_namespace.<service>.metadata[0].name
|
||||
tls_secret_name = var.tls_secret_name
|
||||
}
|
||||
|
||||
# If database migrations needed, add init_container
|
||||
resource "kubernetes_deployment" "<service>" {
|
||||
metadata {
|
||||
name = "<service>"
|
||||
namespace = kubernetes_namespace.<service>.metadata[0].name
|
||||
labels = {
|
||||
app = "<service>"
|
||||
tier = var.tier
|
||||
}
|
||||
}
|
||||
spec {
|
||||
replicas = 1
|
||||
selector {
|
||||
match_labels = {
|
||||
app = "<service>"
|
||||
}
|
||||
}
|
||||
template {
|
||||
metadata {
|
||||
labels = {
|
||||
app = "<service>"
|
||||
}
|
||||
}
|
||||
spec {
|
||||
# Init container for migrations (if needed)
|
||||
# init_container { ... }
|
||||
|
||||
container {
|
||||
name = "<service>"
|
||||
image = "<docker-image>:<tag>"
|
||||
|
||||
port {
|
||||
container_port = <port>
|
||||
}
|
||||
|
||||
# Environment variables
|
||||
env {
|
||||
name = "DATABASE_URL"
|
||||
value = "postgresql://<service>:${var.postgresql_password}@postgresql.dbaas.svc.cluster.local:5432/<service>"
|
||||
}
|
||||
# Add other env vars as needed
|
||||
|
||||
# Volume mounts for persistent data
|
||||
volume_mount {
|
||||
name = "data"
|
||||
mount_path = "<mount-path>"
|
||||
sub_path = "<optional-subpath>"
|
||||
}
|
||||
|
||||
resources {
|
||||
requests = {
|
||||
memory = "256Mi"
|
||||
cpu = "100m"
|
||||
}
|
||||
limits = {
|
||||
memory = "2Gi"
|
||||
cpu = "1"
|
||||
}
|
||||
}
|
||||
|
||||
# Health checks (if endpoints exist)
|
||||
liveness_probe {
|
||||
http_get {
|
||||
path = "/health" # or /healthz, /, etc.
|
||||
port = <port>
|
||||
}
|
||||
initial_delay_seconds = 60
|
||||
period_seconds = 30
|
||||
}
|
||||
}
|
||||
|
||||
# NFS volume for persistence
|
||||
volume {
|
||||
name = "data"
|
||||
nfs {
|
||||
server = "10.0.10.15"
|
||||
path = "/mnt/main/<service>"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "kubernetes_service" "<service>" {
|
||||
metadata {
|
||||
name = "<service>"
|
||||
namespace = kubernetes_namespace.<service>.metadata[0].name
|
||||
labels = {
|
||||
app = "<service>"
|
||||
}
|
||||
}
|
||||
|
||||
spec {
|
||||
selector = {
|
||||
app = "<service>"
|
||||
}
|
||||
port {
|
||||
name = "http"
|
||||
port = 80
|
||||
target_port = <container-port>
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module "ingress" {
|
||||
source = "../ingress_factory"
|
||||
namespace = kubernetes_namespace.<service>.metadata[0].name
|
||||
name = "<service>"
|
||||
tls_secret_name = var.tls_secret_name
|
||||
# Add extra_annotations if needed (proxy-body-size, timeouts, etc.)
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Update Main Terraform Files
|
||||
|
||||
**Add to `modules/kubernetes/main.tf`**:
|
||||
|
||||
1. Add variable declarations at top:
|
||||
```hcl
|
||||
variable "<service>_postgresql_password" { type = string }
|
||||
```
|
||||
|
||||
2. Add to appropriate DEFCON level (ask user which level, default to 5):
|
||||
```hcl
|
||||
5 : [
|
||||
...,
|
||||
"<service>"
|
||||
]
|
||||
```
|
||||
|
||||
3. Add module block at bottom:
|
||||
```hcl
|
||||
module "<service>" {
|
||||
source = "./<service>"
|
||||
for_each = contains(local.active_modules, "<service>") ? { <service> = true } : {}
|
||||
tls_secret_name = var.tls_secret_name
|
||||
postgresql_password = var.<service>_postgresql_password
|
||||
tier = local.tiers.aux # or appropriate tier
|
||||
|
||||
depends_on = [null_resource.core_services]
|
||||
}
|
||||
```
|
||||
|
||||
**Add to `main.tf`**:
|
||||
|
||||
1. Add variable:
|
||||
```hcl
|
||||
variable "<service>_postgresql_password" { type = string }
|
||||
```
|
||||
|
||||
2. Pass to kubernetes_cluster module:
|
||||
```hcl
|
||||
module "kubernetes_cluster" {
|
||||
...
|
||||
<service>_postgresql_password = var.<service>_postgresql_password
|
||||
}
|
||||
```
|
||||
|
||||
**Update `terraform.tfvars`**:
|
||||
|
||||
1. Add password/credentials:
|
||||
```hcl
|
||||
<service>_postgresql_password = "<secure-password>"
|
||||
```
|
||||
|
||||
2. Add to Cloudflare DNS (ask user if proxied or non-proxied):
|
||||
```hcl
|
||||
cloudflare_non_proxied_names = [
|
||||
...,
|
||||
"<service>"
|
||||
]
|
||||
```
|
||||
|
||||
### 6. Email/SMTP Configuration (if needed)
|
||||
|
||||
If service needs to send emails:
|
||||
```hcl
|
||||
env {
|
||||
name = "MAILER_HOST"
|
||||
value = "mailserver.viktorbarzin.me" # Public hostname for TLS
|
||||
}
|
||||
env {
|
||||
name = "MAILER_PORT"
|
||||
value = "587"
|
||||
}
|
||||
env {
|
||||
name = "MAILER_USER"
|
||||
value = "info@viktorbarzin.me"
|
||||
}
|
||||
env {
|
||||
name = "MAILER_PASSWORD"
|
||||
value = var.mailserver_accounts["info@viktorbarzin.me"] # Pass from module
|
||||
}
|
||||
```
|
||||
|
||||
Add to module call:
|
||||
```hcl
|
||||
smtp_password = var.mailserver_accounts["info@viktorbarzin.me"]
|
||||
```
|
||||
|
||||
### 7. Apply Terraform
|
||||
|
||||
```bash
|
||||
terraform init
|
||||
terraform apply -target=module.kubernetes_cluster.module.<service> -var="kube_config_path=$(pwd)/config" -auto-approve
|
||||
```
|
||||
|
||||
**IMPORTANT: Also apply the cloudflared module to create the Cloudflare DNS record:**
|
||||
```bash
|
||||
terraform apply -target=module.kubernetes_cluster.module.cloudflared -var="kube_config_path=$(pwd)/config" -auto-approve
|
||||
```
|
||||
Without this step, the DNS record won't be created even though it's defined in `terraform.tfvars`.
|
||||
|
||||
### 8. Verification
|
||||
|
||||
```bash
|
||||
kubectl get pods -n <service>
|
||||
kubectl logs -n <service> -l app=<service> --tail=50
|
||||
```
|
||||
|
||||
Test URL: `https://<service>.viktorbarzin.me`
|
||||
|
||||
### 8b. Stability Gate (required when `dockerfile_state ∈ {written-from-scratch, fixed-broken-upstream}`)
|
||||
|
||||
Before committing — and before any upstream PR in §10 — run a 10-minute stability check to catch pods that crash-loop a few minutes after Ready.
|
||||
|
||||
```bash
|
||||
.claude/skills/setup-project/scripts/stability-gate.sh <service> <service> https://<service>.viktorbarzin.me
|
||||
```
|
||||
|
||||
Polls pod readiness + `curl` 200 every 30s × 20 iterations. Requires 18/20 successes (tolerates 2 blips).
|
||||
|
||||
- **Pass** → update the state file: `jq '.deploy_verified_at = (now | todate)' .contribution-state.json | sponge .contribution-state.json` → proceed to §9 and §10.
|
||||
- **Fail** → stop. Investigate via `kubectl logs`, `kubectl describe`. Do NOT commit. Do NOT fire §10. Re-run the gate after fixes.
|
||||
|
||||
For `image-used` / `used-as-is` states, the gate is optional (app is already running a known-good image).
|
||||
|
||||
### 9. Commit Changes
|
||||
|
||||
```bash
|
||||
git add modules/kubernetes/<service>/ main.tf modules/kubernetes/main.tf terraform.tfvars
|
||||
git commit -m "Add <service> deployment
|
||||
|
||||
- Deploy <service> as <description>
|
||||
- Uses <dependencies>
|
||||
- Ingress at <service>.viktorbarzin.me
|
||||
|
||||
[ci skip]"
|
||||
```
|
||||
|
||||
### 10. Contribute Dockerfile Upstream (only when `dockerfile_state ∈ {written-from-scratch, fixed-broken-upstream}`)
|
||||
|
||||
Goal: give the community the working Dockerfile we just validated in production.
|
||||
|
||||
**Preconditions** (script enforces):
|
||||
- `.contribution-state.json` present with a trigger state and `deploy_verified_at` set.
|
||||
- `files/Dockerfile`, `files/.dockerignore`, `files/BUILD.md` exist next to the module.
|
||||
- `GITHUB_TOKEN` in env — or `vault kv get -field=github_pat secret/viktor` is reachable.
|
||||
|
||||
**Run**:
|
||||
```bash
|
||||
.claude/skills/setup-project/scripts/contribute-dockerfile.sh modules/kubernetes/<service>
|
||||
```
|
||||
|
||||
**What the script does** (all via GitHub REST — `gh` CLI is sandbox-blocked):
|
||||
1. Reads `.contribution-state.json`; skips unless state is `written-from-scratch` or `fixed-broken-upstream` and no `contribution_pr_url` is already recorded.
|
||||
2. Upstream sanity checks: repo exists, public, not archived; default branch discoverable; for `written-from-scratch`, verifies a `Dockerfile` didn't land upstream while we were deploying; bails cleanly if an open PR from our fork already exists.
|
||||
3. `POST /repos/<owner>/<name>/forks` — idempotent; waits up to 30s for the fork to be ready at `ViktorBarzin/<name>`.
|
||||
4. `POST /repos/ViktorBarzin/<name>/merge-upstream` — keeps fork current with upstream default branch.
|
||||
5. Creates branch `add-dockerfile` (or `fix-dockerfile`), timestamp-suffixed if that branch already exists with unrelated commits.
|
||||
6. Commits `Dockerfile`, `.dockerignore`, `BUILD.md` via Contents API. Each commit message carries `Signed-off-by:` for DCO-enforcing repos.
|
||||
7. Opens PR against upstream with body rendered from `templates/PR_BODY.md`.
|
||||
8. Writes `contribution_pr_url` back into `.contribution-state.json` and echoes the URL.
|
||||
|
||||
**Failure handling**:
|
||||
- Upstream archived / private / deleted → logged as SKIP, deploy success stands.
|
||||
- Fork/branch/PR already exists → treated as idempotent success; existing URL recorded.
|
||||
- GitHub 5xx → 3× exponential backoff, then hard fail with a clear message — safe to re-run the script.
|
||||
|
||||
**After the PR opens**: the URL is in `.contribution-state.json`. Share it with the user. No automated follow-up on merge/reject — that's a manual check for now.
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Init Container for Migrations
|
||||
```hcl
|
||||
init_container {
|
||||
name = "migration"
|
||||
image = "<same-image>"
|
||||
command = ["sh", "-c", "<migration-command>"]
|
||||
|
||||
# Same env vars and volumes as main container
|
||||
}
|
||||
```
|
||||
|
||||
### Dynamic Environment Variables
|
||||
```hcl
|
||||
locals {
|
||||
common_env = [
|
||||
{ name = "VAR1", value = "value1" },
|
||||
{ name = "VAR2", value = "value2" },
|
||||
]
|
||||
}
|
||||
|
||||
dynamic "env" {
|
||||
for_each = local.common_env
|
||||
content {
|
||||
name = env.value.name
|
||||
value = env.value.value
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### External URL Configuration
|
||||
Many apps need their public URL configured:
|
||||
```hcl
|
||||
env {
|
||||
name = "APP_URL" # or PUBLIC_URL, EXTERNAL_URL, etc.
|
||||
value = "https://<service>.viktorbarzin.me"
|
||||
}
|
||||
env {
|
||||
name = "HTTPS" # or ENABLE_HTTPS, etc.
|
||||
value = "true"
|
||||
}
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Find official Docker image or docker-compose
|
||||
- [ ] Identify dependencies (DB, Redis, etc.)
|
||||
- [ ] Ask user for database credentials (never create yourself)
|
||||
- [ ] Create NFS directory and export on TrueNAS (if persistent storage needed)
|
||||
- [ ] Verify NFS mount is accessible from k8s nodes
|
||||
- [ ] Create `modules/kubernetes/<service>/main.tf`
|
||||
- [ ] Classify `dockerfile_state` and write `.contribution-state.json`
|
||||
- [ ] If writing/fixing Dockerfile: satisfy the quality bar (multi-stage, non-root, `.dockerignore`, `BUILD.md`)
|
||||
- [ ] Update `modules/kubernetes/main.tf` (variables, DEFCON level, module block)
|
||||
- [ ] Update `main.tf` (variable, pass to module)
|
||||
- [ ] Update `terraform.tfvars` (password, Cloudflare DNS)
|
||||
- [ ] Run `terraform init` and `terraform apply`
|
||||
- [ ] Verify pods are running
|
||||
- [ ] Test the URL
|
||||
- [ ] Run stability-gate.sh — needed for contribution, optional otherwise
|
||||
- [ ] Commit changes with `[ci skip]`
|
||||
- [ ] Run contribute-dockerfile.sh if state triggers an upstream PR
|
||||
|
||||
## Questions to Ask User
|
||||
|
||||
1. What DEFCON level should this service be in? (Default: 5)
|
||||
2. Should Cloudflare proxy this domain? (Default: no, add to non_proxied_names)
|
||||
3. Does this need email/SMTP? (Configure if yes)
|
||||
4. What database credentials should I use? (Never create yourself)
|
||||
5. What tier? (core/cluster/gpu/edge/aux - default: aux)
|
||||
|
||||
## Notes
|
||||
|
||||
- **Always create NFS directories and exports BEFORE deploying** - pods will get stuck in `ContainerCreating` if the NFS path doesn't exist or isn't exported
|
||||
- **Always use official documentation** as the source of truth
|
||||
- **Prefer stable/latest tags** over specific versions for self-hosted
|
||||
- **Use shared infrastructure**: PostgreSQL at `postgresql.dbaas.svc.cluster.local`, Redis at `redis.redis.svc.cluster.local`
|
||||
- **NFS storage**: Always at `10.0.10.15:/mnt/main/<service>`
|
||||
- **Email**: Use `mailserver.viktorbarzin.me` (public hostname) not internal service name
|
||||
- **Resource limits**: Start conservative, can increase if needed
|
||||
- **Health checks**: Only add if the app has health endpoints
|
||||
|
|
@ -1,270 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# Contribute a working Dockerfile back to an upstream GitHub repo.
|
||||
#
|
||||
# Reads state from <service-module-dir>/.contribution-state.json and:
|
||||
# 1. Validates triggers (dockerfile_state ∈ {written-from-scratch, fixed-broken-upstream})
|
||||
# 2. Confirms upstream is public, not archived, no concurrent Dockerfile landed
|
||||
# 3. Forks upstream to ViktorBarzin (idempotent)
|
||||
# 4. Syncs fork with upstream default branch
|
||||
# 5. Creates branch (add-dockerfile or fix-dockerfile), appends -<ts> on collision
|
||||
# 6. Commits Dockerfile + .dockerignore + BUILD.md via Contents API
|
||||
# 7. Opens PR against upstream with body rendered from PR_BODY.md
|
||||
# 8. Writes contribution_pr_url back into state file
|
||||
#
|
||||
# Usage:
|
||||
# contribute-dockerfile.sh <service-module-dir>
|
||||
#
|
||||
# Example:
|
||||
# contribute-dockerfile.sh /home/wizard/code/infra/modules/kubernetes/myapp
|
||||
#
|
||||
# Requires: jq, curl, vault CLI (logged in).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TEMPLATES_DIR="$(cd "$SCRIPT_DIR/../templates" && pwd)"
|
||||
|
||||
FORK_OWNER="ViktorBarzin"
|
||||
|
||||
log() { echo "contribute-dockerfile: $*"; }
|
||||
die() { echo "contribute-dockerfile: ERROR: $*" >&2; exit 1; }
|
||||
skip() { echo "contribute-dockerfile: SKIP: $*"; exit 0; }
|
||||
|
||||
if [ "$#" -ne 1 ]; then
|
||||
die "usage: $0 <service-module-dir>"
|
||||
fi
|
||||
|
||||
MODULE_DIR="$1"
|
||||
STATE_FILE="$MODULE_DIR/.contribution-state.json"
|
||||
|
||||
[ -f "$STATE_FILE" ] || die "state file not found: $STATE_FILE"
|
||||
|
||||
# --- Read + validate state ---
|
||||
dockerfile_state=$(jq -r '.dockerfile_state // ""' "$STATE_FILE")
|
||||
upstream_repo=$(jq -r '.upstream_repo // ""' "$STATE_FILE")
|
||||
dockerfile_path=$(jq -r '.dockerfile_path_in_infra // ""' "$STATE_FILE")
|
||||
deploy_verified_at=$(jq -r '.deploy_verified_at // ""' "$STATE_FILE")
|
||||
existing_pr_url=$(jq -r '.contribution_pr_url // ""' "$STATE_FILE")
|
||||
|
||||
if [ -n "$existing_pr_url" ] && [ "$existing_pr_url" != "null" ]; then
|
||||
skip "PR already exists: $existing_pr_url"
|
||||
fi
|
||||
|
||||
case "$dockerfile_state" in
|
||||
written-from-scratch) BRANCH_NAME="add-dockerfile"; reason_type="none" ;;
|
||||
fixed-broken-upstream) BRANCH_NAME="fix-dockerfile"; reason_type="broken" ;;
|
||||
*) skip "dockerfile_state='$dockerfile_state' — nothing to contribute" ;;
|
||||
esac
|
||||
|
||||
[ -z "$deploy_verified_at" ] || [ "$deploy_verified_at" = "null" ] && die "deploy not verified yet (deploy_verified_at empty); run stability-gate first"
|
||||
|
||||
[ -z "$upstream_repo" ] && die "upstream_repo empty in state file"
|
||||
[[ "$upstream_repo" == */* ]] || die "upstream_repo must be owner/name, got: $upstream_repo"
|
||||
|
||||
UP_OWNER="${upstream_repo%/*}"
|
||||
UP_NAME="${upstream_repo#*/}"
|
||||
|
||||
abs_dockerfile="$MODULE_DIR/$(basename "$dockerfile_path")"
|
||||
if [ ! -f "$MODULE_DIR/files/Dockerfile" ]; then
|
||||
die "Dockerfile not found at $MODULE_DIR/files/Dockerfile"
|
||||
fi
|
||||
DOCKERFILE_SRC="$MODULE_DIR/files/Dockerfile"
|
||||
DOCKERIGNORE_SRC="$MODULE_DIR/files/.dockerignore"
|
||||
BUILDMD_SRC="$MODULE_DIR/files/BUILD.md"
|
||||
for f in "$DOCKERIGNORE_SRC" "$BUILDMD_SRC"; do
|
||||
[ -f "$f" ] || die "required file missing: $f"
|
||||
done
|
||||
|
||||
# --- GitHub auth ---
|
||||
GITHUB_TOKEN="${GITHUB_TOKEN:-$(vault kv get -field=github_pat secret/viktor 2>/dev/null || true)}"
|
||||
[ -n "$GITHUB_TOKEN" ] || die "GITHUB_TOKEN not set and vault lookup failed (vault login -method=oidc first)"
|
||||
|
||||
gh_api() {
|
||||
local method="$1"; local path="$2"; local data="${3:-}"
|
||||
local url="https://api.github.com${path}"
|
||||
local curl_args=(-sS -w "\n%{http_code}" -X "$method"
|
||||
-H "Authorization: token $GITHUB_TOKEN"
|
||||
-H "Accept: application/vnd.github+json"
|
||||
-H "X-GitHub-Api-Version: 2022-11-28")
|
||||
[ -n "$data" ] && curl_args+=(-d "$data")
|
||||
curl "${curl_args[@]}" "$url"
|
||||
}
|
||||
|
||||
gh_api_retry() {
|
||||
local method="$1"; local path="$2"; local data="${3:-}"
|
||||
local attempt=1
|
||||
local max_attempts=3
|
||||
local out http
|
||||
while [ "$attempt" -le "$max_attempts" ]; do
|
||||
out=$(gh_api "$method" "$path" "$data")
|
||||
http=$(printf '%s' "$out" | tail -n1)
|
||||
body=$(printf '%s' "$out" | sed '$d')
|
||||
if [ "$http" -ge 500 ] || [ "$http" = "000" ]; then
|
||||
log "retry $attempt/$max_attempts on $method $path (http=$http)"
|
||||
attempt=$((attempt + 1))
|
||||
sleep $((2 ** attempt))
|
||||
continue
|
||||
fi
|
||||
printf '%s\n%s' "$body" "$http"
|
||||
return 0
|
||||
done
|
||||
die "GitHub API 5xx after $max_attempts attempts on $method $path"
|
||||
}
|
||||
|
||||
# Helpers that parse the combined body+http form.
|
||||
gh_http() { printf '%s' "$1" | tail -n1; }
|
||||
gh_body() { printf '%s' "$1" | sed '$d'; }
|
||||
|
||||
# --- Upstream sanity checks ---
|
||||
log "checking upstream $upstream_repo"
|
||||
resp=$(gh_api_retry GET "/repos/$UP_OWNER/$UP_NAME")
|
||||
http=$(gh_http "$resp"); body=$(gh_body "$resp")
|
||||
if [ "$http" = "404" ]; then skip "upstream repo not found (may be private or deleted): $upstream_repo"; fi
|
||||
[ "$http" = "200" ] || die "GET upstream failed http=$http body=$body"
|
||||
|
||||
archived=$(printf '%s' "$body" | jq -r '.archived')
|
||||
default_branch=$(printf '%s' "$body" | jq -r '.default_branch')
|
||||
[ "$archived" = "true" ] && skip "upstream is archived — not opening PR"
|
||||
[ -n "$default_branch" ] || die "could not determine upstream default branch"
|
||||
log "upstream default branch: $default_branch"
|
||||
|
||||
# If we wrote the Dockerfile from scratch, make sure one didn't land upstream meanwhile.
|
||||
if [ "$dockerfile_state" = "written-from-scratch" ]; then
|
||||
resp=$(gh_api_retry GET "/repos/$UP_OWNER/$UP_NAME/contents/Dockerfile?ref=$default_branch")
|
||||
http=$(gh_http "$resp")
|
||||
if [ "$http" = "200" ]; then
|
||||
skip "a Dockerfile landed upstream since we started — aborting to avoid clobbering"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check for an existing open PR from our fork.
|
||||
resp=$(gh_api_retry GET "/repos/$UP_OWNER/$UP_NAME/pulls?state=open&head=${FORK_OWNER}:${BRANCH_NAME}")
|
||||
http=$(gh_http "$resp"); body=$(gh_body "$resp")
|
||||
if [ "$http" = "200" ]; then
|
||||
existing=$(printf '%s' "$body" | jq -r '.[0].html_url // ""')
|
||||
if [ -n "$existing" ]; then
|
||||
log "existing open PR found: $existing — recording and skipping"
|
||||
jq --arg url "$existing" '.contribution_pr_url = $url' "$STATE_FILE" > "$STATE_FILE.tmp" && mv "$STATE_FILE.tmp" "$STATE_FILE"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- Fork ---
|
||||
log "ensuring fork exists at $FORK_OWNER/$UP_NAME"
|
||||
resp=$(gh_api_retry POST "/repos/$UP_OWNER/$UP_NAME/forks" '{}')
|
||||
http=$(gh_http "$resp")
|
||||
if [ "$http" != "202" ] && [ "$http" != "200" ]; then
|
||||
die "fork call failed http=$http"
|
||||
fi
|
||||
|
||||
# Wait for fork to be ready (GitHub can take up to ~30s).
|
||||
for i in $(seq 1 15); do
|
||||
resp=$(gh_api_retry GET "/repos/$FORK_OWNER/$UP_NAME")
|
||||
if [ "$(gh_http "$resp")" = "200" ]; then break; fi
|
||||
sleep 2
|
||||
done
|
||||
[ "$(gh_http "$resp")" = "200" ] || die "fork $FORK_OWNER/$UP_NAME did not become ready"
|
||||
|
||||
# --- Sync fork with upstream default branch ---
|
||||
log "syncing fork with upstream/$default_branch"
|
||||
resp=$(gh_api_retry POST "/repos/$FORK_OWNER/$UP_NAME/merge-upstream" "$(jq -n --arg b "$default_branch" '{branch:$b}')")
|
||||
http=$(gh_http "$resp")
|
||||
[ "$http" = "200" ] || [ "$http" = "409" ] || log "merge-upstream returned http=$http (continuing)"
|
||||
|
||||
# --- Determine base SHA for new branch ---
|
||||
resp=$(gh_api_retry GET "/repos/$FORK_OWNER/$UP_NAME/git/ref/heads/$default_branch")
|
||||
http=$(gh_http "$resp"); body=$(gh_body "$resp")
|
||||
[ "$http" = "200" ] || die "could not read default branch ref on fork (http=$http)"
|
||||
base_sha=$(printf '%s' "$body" | jq -r '.object.sha')
|
||||
|
||||
# --- Create branch (or append timestamp on collision) ---
|
||||
attempt_branch="$BRANCH_NAME"
|
||||
resp=$(gh_api_retry GET "/repos/$FORK_OWNER/$UP_NAME/git/ref/heads/$attempt_branch")
|
||||
if [ "$(gh_http "$resp")" = "200" ]; then
|
||||
attempt_branch="${BRANCH_NAME}-$(date +%s | tail -c 9)"
|
||||
log "branch existed; using $attempt_branch"
|
||||
fi
|
||||
|
||||
log "creating branch $attempt_branch off $base_sha"
|
||||
payload=$(jq -n --arg r "refs/heads/$attempt_branch" --arg s "$base_sha" '{ref:$r,sha:$s}')
|
||||
resp=$(gh_api_retry POST "/repos/$FORK_OWNER/$UP_NAME/git/refs" "$payload")
|
||||
[ "$(gh_http "$resp")" = "201" ] || die "could not create branch: $(gh_body "$resp")"
|
||||
|
||||
# --- Helper to PUT a file via Contents API ---
|
||||
put_file() {
|
||||
local src="$1"; local dst="$2"; local message="$3"
|
||||
local b64 payload exists_resp http existing_sha=""
|
||||
b64=$(base64 -w0 < "$src")
|
||||
|
||||
exists_resp=$(gh_api_retry GET "/repos/$FORK_OWNER/$UP_NAME/contents/$dst?ref=$attempt_branch")
|
||||
if [ "$(gh_http "$exists_resp")" = "200" ]; then
|
||||
existing_sha=$(gh_body "$exists_resp" | jq -r '.sha')
|
||||
fi
|
||||
|
||||
if [ -n "$existing_sha" ]; then
|
||||
payload=$(jq -n --arg m "$message" --arg c "$b64" --arg b "$attempt_branch" --arg sha "$existing_sha" \
|
||||
'{message:$m, content:$c, branch:$b, sha:$sha}')
|
||||
else
|
||||
payload=$(jq -n --arg m "$message" --arg c "$b64" --arg b "$attempt_branch" \
|
||||
'{message:$m, content:$c, branch:$b}')
|
||||
fi
|
||||
|
||||
resp=$(gh_api_retry PUT "/repos/$FORK_OWNER/$UP_NAME/contents/$dst" "$payload")
|
||||
http=$(gh_http "$resp")
|
||||
[ "$http" = "200" ] || [ "$http" = "201" ] || die "PUT $dst failed http=$http body=$(gh_body "$resp")"
|
||||
}
|
||||
|
||||
commit_msg_prefix="Add Dockerfile"
|
||||
[ "$dockerfile_state" = "fixed-broken-upstream" ] && commit_msg_prefix="Fix Dockerfile"
|
||||
|
||||
log "committing Dockerfile, .dockerignore, BUILD.md"
|
||||
put_file "$DOCKERFILE_SRC" "Dockerfile" "$commit_msg_prefix
|
||||
|
||||
Signed-off-by: Viktor Barzin <viktorbarzin@meta.com>"
|
||||
put_file "$DOCKERIGNORE_SRC" ".dockerignore" "Add .dockerignore
|
||||
|
||||
Signed-off-by: Viktor Barzin <viktorbarzin@meta.com>"
|
||||
put_file "$BUILDMD_SRC" "BUILD.md" "Add BUILD.md
|
||||
|
||||
Signed-off-by: Viktor Barzin <viktorbarzin@meta.com>"
|
||||
|
||||
# --- Render PR body ---
|
||||
reason_paragraph="This project currently has no Dockerfile, making it harder for the self-hosting community to run this. I put together a working one while deploying this app to my home Kubernetes cluster and wanted to upstream it."
|
||||
if [ "$reason_type" = "broken" ]; then
|
||||
reason_paragraph="The existing Dockerfile in this repo does not build cleanly for \`linux/amd64\`. I tracked down the fixes while deploying this app to my home Kubernetes cluster and wanted to upstream them."
|
||||
fi
|
||||
|
||||
IMAGE_SIZE=$(jq -r '.image_size // "unknown"' "$STATE_FILE")
|
||||
BASE_IMAGE=$(jq -r '.base_image // "unknown"' "$STATE_FILE")
|
||||
IMAGE_TAG=$(jq -r '.image_tag // "myapp:latest"' "$STATE_FILE")
|
||||
DOCKERFILE_SHAPE=$(jq -r '.dockerfile_shape // "multi-stage, non-root, linux/amd64"' "$STATE_FILE")
|
||||
|
||||
pr_body=$(cat "$TEMPLATES_DIR/PR_BODY.md")
|
||||
pr_body="${pr_body//\{\{REASON_PARAGRAPH\}\}/$reason_paragraph}"
|
||||
pr_body="${pr_body//\{\{DOCKERFILE_SHAPE\}\}/$DOCKERFILE_SHAPE}"
|
||||
pr_body="${pr_body//\{\{IMAGE_SIZE\}\}/$IMAGE_SIZE}"
|
||||
pr_body="${pr_body//\{\{BASE_IMAGE\}\}/$BASE_IMAGE}"
|
||||
pr_body="${pr_body//\{\{IMAGE_TAG\}\}/$IMAGE_TAG}"
|
||||
|
||||
pr_title="$commit_msg_prefix"
|
||||
|
||||
# --- Open PR ---
|
||||
log "opening PR against $UP_OWNER/$UP_NAME:$default_branch"
|
||||
payload=$(jq -n \
|
||||
--arg t "$pr_title" \
|
||||
--arg h "${FORK_OWNER}:${attempt_branch}" \
|
||||
--arg b "$default_branch" \
|
||||
--arg body "$pr_body" \
|
||||
'{title:$t, head:$h, base:$b, body:$body, maintainer_can_modify:true}')
|
||||
resp=$(gh_api_retry POST "/repos/$UP_OWNER/$UP_NAME/pulls" "$payload")
|
||||
http=$(gh_http "$resp"); body=$(gh_body "$resp")
|
||||
if [ "$http" != "201" ]; then
|
||||
die "PR creation failed http=$http body=$body"
|
||||
fi
|
||||
|
||||
pr_url=$(printf '%s' "$body" | jq -r '.html_url')
|
||||
log "PR opened: $pr_url"
|
||||
|
||||
# --- Record PR URL in state file ---
|
||||
jq --arg url "$pr_url" '.contribution_pr_url = $url' "$STATE_FILE" > "$STATE_FILE.tmp" && mv "$STATE_FILE.tmp" "$STATE_FILE"
|
||||
log "state file updated with PR URL"
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
# 10-minute deploy stability gate for setup-project skill.
|
||||
# Polls pod readiness + HTTP 200 on target URL every 30s for 20 iterations.
|
||||
# Requires 18/20 probes to succeed (tolerates 2 blips for restarts/DNS propagation).
|
||||
#
|
||||
# Usage:
|
||||
# stability-gate.sh <namespace> <app-label> <url>
|
||||
#
|
||||
# Example:
|
||||
# stability-gate.sh myapp myapp https://myapp.viktorbarzin.me
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 - Stable (>=18/20 probes OK)
|
||||
# 1 - Unstable (<18/20 probes OK)
|
||||
# 2 - Usage error
|
||||
|
||||
set -u
|
||||
|
||||
if [ "$#" -ne 3 ]; then
|
||||
echo "Usage: $0 <namespace> <app-label> <url>" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
NS="$1"
|
||||
APP="$2"
|
||||
URL="$3"
|
||||
|
||||
TOTAL_PROBES=20
|
||||
MIN_SUCCESSES=18
|
||||
INTERVAL_SECONDS=30
|
||||
|
||||
ok_count=0
|
||||
fail_count=0
|
||||
|
||||
echo "stability-gate: ns=$NS app=$APP url=$URL"
|
||||
echo "stability-gate: $TOTAL_PROBES probes x ${INTERVAL_SECONDS}s (need $MIN_SUCCESSES/$TOTAL_PROBES)"
|
||||
|
||||
for i in $(seq 1 "$TOTAL_PROBES"); do
|
||||
probe_ok=true
|
||||
|
||||
if ! kubectl wait --for=condition=Ready pod -l "app=$APP" -n "$NS" --timeout=25s >/dev/null 2>&1; then
|
||||
probe_ok=false
|
||||
fi
|
||||
|
||||
status=$(curl -sS -o /dev/null -w "%{http_code}" --max-time 10 "$URL" || echo "000")
|
||||
if [ "$status" != "200" ]; then
|
||||
probe_ok=false
|
||||
fi
|
||||
|
||||
if [ "$probe_ok" = "true" ]; then
|
||||
ok_count=$((ok_count + 1))
|
||||
printf " probe %2d/%d: OK (http=%s)\n" "$i" "$TOTAL_PROBES" "$status"
|
||||
else
|
||||
fail_count=$((fail_count + 1))
|
||||
printf " probe %2d/%d: FAIL (http=%s)\n" "$i" "$TOTAL_PROBES" "$status"
|
||||
fi
|
||||
|
||||
if [ "$i" -lt "$TOTAL_PROBES" ]; then
|
||||
sleep "$INTERVAL_SECONDS"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "stability-gate: results ok=$ok_count fail=$fail_count"
|
||||
|
||||
if [ "$ok_count" -ge "$MIN_SUCCESSES" ]; then
|
||||
echo "stability-gate: PASS"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "stability-gate: FAIL (need $MIN_SUCCESSES, got $ok_count)" >&2
|
||||
exit 1
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
# Build notes
|
||||
|
||||
## Build
|
||||
|
||||
```
|
||||
docker build --platform linux/amd64 -t {{IMAGE_NAME}}:{{TAG}} .
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```
|
||||
docker run --rm -p {{CONTAINER_PORT}}:{{CONTAINER_PORT}} {{IMAGE_NAME}}:{{TAG}}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
{{ENV_VARS_TABLE}}
|
||||
|
||||
## Notes
|
||||
|
||||
- Built for `linux/amd64`; multi-arch not tested.
|
||||
- Image size: `{{IMAGE_SIZE}}`, base: `{{BASE_IMAGE}}`.
|
||||
- Runs as a non-root user.
|
||||
{{EXTRA_NOTES}}
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
## Add a working Dockerfile
|
||||
|
||||
### Why
|
||||
{{REASON_PARAGRAPH}}
|
||||
|
||||
### What this adds
|
||||
- `Dockerfile` — {{DOCKERFILE_SHAPE}}
|
||||
- `.dockerignore`
|
||||
- `BUILD.md` with the build command and notes
|
||||
|
||||
### Tested
|
||||
- Built and pushed to a private registry, deployed to a Kubernetes cluster.
|
||||
- Pod has been Ready and serving HTTP 200 at the ingress for 10+ minutes of continuous probing before this PR was opened.
|
||||
- Image size: {{IMAGE_SIZE}}, base: {{BASE_IMAGE}}
|
||||
- Platform tested: `linux/amd64`
|
||||
|
||||
### Build command
|
||||
```
|
||||
docker build --platform linux/amd64 -t {{IMAGE_TAG}} .
|
||||
```
|
||||
|
||||
Happy to iterate on base image, build args, or multi-arch support if you'd prefer a different shape. Thanks for the project!
|
||||
|
||||
---
|
||||
<sub>Contributed after self-hosting this project. Filed by the repo owner's deployment workflow; feel free to mention me (@ViktorBarzin) with any follow-ups.</sub>
|
||||
Loading…
Add table
Add a link
Reference in a new issue