infra/stacks/owntracks/dawarich-hook.lua

83 lines
2.9 KiB
Lua
Raw Normal View History

[owntracks] Bridge Recorder → Dawarich via Lua hook script ## Context Viktor wanted live forwarding from Owntracks to Dawarich so his map stays in sync without a periodic backfill. The original plan assumed ot-recorder honoured an `OTR_HTTPHOOK` environment variable — but Recorder 1.0.1 (latest on Docker Hub as of Aug 2025) has no such feature: ``` $ kubectl -n owntracks exec deploy/owntracks -- \ strings /usr/bin/ot-recorder | grep -iE 'hook|webhook|http_post' (no matches) ``` Lua hooks, on the other hand, are first-class: `--lua-script` loads a file and calls the `otr_hook(topic, _type, data)` function for every publish. That is the pivot this commit makes. ## This change Mount a Lua script via ConfigMap and tell ot-recorder to load it: ``` Phone POST /pub ---> Traefik ---> Recorder pod | | handle_payload() writes .rec | otr_hook(topic,_type,data) | | | +---> os.execute("curl … &") | | | v | Dawarich /api/v1/owntracks/points | +---> HTTP 200 to phone ``` Per-publish cost: one `curl` subprocess, `--max-time 5`, backgrounded with `&` so it doesn't block the HTTP response to the phone. A Dawarich 5xx drops exactly one point — the `.rec` write still happens, so the one-shot backfill Job can always re-play. `DAWARICH_API_KEY` is injected from K8s Secret `owntracks-secrets` (sourced from Vault `secret/owntracks.dawarich_api_key` via the existing `dataFrom.extract` ExternalSecret). The Lua reads it with `os.getenv()` so the key never lands in Terraform state. ### Key discoveries in the verification loop (why iteration count > 1) 1. The hook function must be named `otr_hook`, not `hook` (recorder's `luasupport.c` calls `lua_getglobal(L, "otr_hook")`). The recorder logs `cannot invoke otr_hook in Lua script` when missing — the plan's `hook()` naming was wrong. 2. Dawarich's `latitude`/`longitude` scalar columns are legacy and always NULL; the authoritative geometry is in the `lonlat` PostGIS column (`ST_AsText(lonlat::geometry)`). Early "it's broken" readings were me querying the wrong columns. 3. Default Recreate-strategy rollouts cause ~30s 502/503 windows on the ingress — tolerable, but every apply is visible as an outage to the phone. Batching edits is important. ## What is NOT in this change - **Not** OTR_HTTPHOOK. Removed with this commit (dead env var). - **Not** the one-shot backfill Job — that comes after the phone buffer has flushed to avoid racing against incoming hook POSTs (follow-up: code-h2r). - **Not** Anca's bridge — a second Recorder instance or a smarter hook is needed to route her posts under her own Dawarich api_key (follow-up: code-72g). - No Ingress or Service change — Commit 1 (`a21d4a44`) already landed those. ## Test Plan ### Automated ``` $ ../../scripts/tg apply --non-interactive Apply complete! Resources: 1 added, 1 changed, 0 destroyed. $ kubectl -n owntracks logs deploy/owntracks --tail=5 + initializing Lua hooks from `/hook/dawarich-hook.lua' + dawarich-bridge: init + HTTP listener started on 0.0.0.0:8083, without browser-apikey ... + dawarich-bridge: tst=1 lat=0 lon=0 ok=true ``` ### Manual Verification ``` $ VIKTOR_PW=$(vault kv get -field=credentials secret/owntracks | jq -r .viktor) $ TST=$(date +%s) $ kubectl -n owntracks run t --rm -i --image=curlimages/curl -- \ curl -s -w 'HTTP %{http_code}\n' -X POST -u "viktor:$VIKTOR_PW" \ -H 'Content-Type: application/json' \ -H 'X-Limit-U: viktor' -H 'X-Limit-D: iphone-15pro' \ -d "{\"_type\":\"location\",\"lat\":51.5074,\"lon\":-0.1278,\"tst\":$TST,\"tid\":\"vb\"}" \ https://owntracks.viktorbarzin.me/pub HTTP 200 $ sleep 3 && kubectl -n dbaas exec pg-cluster-1 -c postgres -- \ psql -U postgres -d dawarich -c \ "SELECT timestamp, ST_AsText(lonlat::geometry) FROM points \ WHERE user_id=1 AND timestamp=$TST" timestamp | st_astext ------------+------------------------- 1776555707 | POINT(-0.1278 51.5074) ``` Real phone traffic (from in-flight buffer flush) lands in Dawarich too: `traefik logs -l app.kubernetes.io/name=traefik | grep 'POST /api/v1/owntracks/points'` shows ingress POSTs from `owntracks` namespace to `dawarich` backend with status 200. ### Reproduce locally 1. `vault login -method=oidc` 2. `kubectl -n owntracks logs deploy/owntracks --tail=20` — expect `dawarich-bridge: init` after the Lua loader line. 3. Do the curl above, poll the DB, expect `POINT(lon lat)`. Closes: code-z9b Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 23:47:22 +00:00
-- ot-recorder Lua hook: forward every location publish to Dawarich.
-- Loaded by ot-recorder via `--lua-script`. The hook() function is invoked
-- synchronously per publish; we fork curl with `&` to keep it fire-and-forget.
-- Dawarich's points table has UNIQUE (lonlat, timestamp, user_id) — duplicates
-- are safely dropped. The .rec file is always written regardless of hook result,
-- so a Dawarich 5xx loses nothing long-term (re-playable via backfill Job).
local function escape_shell_single(s)
return "'" .. tostring(s):gsub("'", "'\\''") .. "'"
end
local function json_escape_string(s)
return (s:gsub("\\", "\\\\")
:gsub('"', '\\"')
:gsub("\n", "\\n")
:gsub("\r", "\\r")
:gsub("\t", "\\t"))
end
-- Minimal JSON serializer — scalars, arrays, maps. Owntracks payloads are
-- all primitive/flat; no bignum or cyclic-ref concerns.
local function to_json(v)
local t = type(v)
if t == "nil" then return "null" end
if t == "number" then return tostring(v) end
if t == "boolean" then return tostring(v) end
if t == "string" then return '"' .. json_escape_string(v) .. '"' end
if t == "table" then
if #v > 0 or next(v) == nil then
local parts = {}
for i, x in ipairs(v) do parts[i] = to_json(x) end
return "[" .. table.concat(parts, ",") .. "]"
end
local parts = {}
for k, x in pairs(v) do
parts[#parts + 1] = '"' .. json_escape_string(tostring(k)) .. '":' .. to_json(x)
end
return "{" .. table.concat(parts, ",") .. "}"
end
return "null"
end
function otr_init()
otr.log("dawarich-bridge: init")
if not os.getenv("DAWARICH_API_KEY") then
otr.log("dawarich-bridge: WARN DAWARICH_API_KEY unset — hook will skip")
end
end
function otr_exit()
otr.log("dawarich-bridge: exit")
end
function otr_hook(topic, _type, data)
if _type ~= "location" then return end
local api_key = os.getenv("DAWARICH_API_KEY")
if not api_key or api_key == "" then
otr.log("dawarich-bridge: DAWARICH_API_KEY missing — dropping point")
return
end
[owntracks] Strip face avatar from hook payload + drop orphan PVC Bundles two small follow-ups to the live bridge + port-fix work: ## Face avatar fix (dawarich-hook.lua) After the Recorder ran in production for a while it began enriching publish payloads with a `face` field — the base64-encoded user avatar uploaded via the Recorder's web UI (~120 KB). Our Lua hook builds a curl command that embeds the JSON payload as `-d '<payload>'`, which hit `E2BIG` / `Argument list too long` (os.execute reason=code=7) on Linux's `execve` argv limit (~128 KB). Every live POST stopped making it to Dawarich, even though the HTTP POST from the phone to Owntracks still returned 200 and the .rec write still happened. Fix: `data.face = nil` before serializing. Dawarich doesn't use it anyway (not persisted into any column — `raw_data` stored without it). Also upgraded the debug log: on failure we now emit `dawarich-bridge: FAIL tst=... reason=... code=... cmd=...` so any future variant of this problem (next big field surfaced upstream, etc.) is one log tail away from a diagnosis. ``` $ kubectl -n owntracks logs deploy/owntracks --tail=5 | grep dawarich-bridge + dawarich-bridge: init + dawarich-bridge: ok tst=1776600238 ``` ## Orphan PVC removal (main.tf) `owntracks-data-proxmox` (1 Gi, proxmox-lvm, unencrypted) was a leftover from the encrypted-migration attempt; the Deployment has been mounting `owntracks-data-encrypted` the whole time. Verified `Used By: <none>` on the live PVC before removal. Removing the resource from Terraform destroys the PVC — harmless, no data loss. ## Test Plan ### Automated ``` $ ../../scripts/tg plan Plan: 0 to add, 1 to change, 1 to destroy. $ ../../scripts/tg apply --non-interactive Apply complete! Resources: 0 added, 1 changed, 1 destroyed. $ kubectl -n owntracks get pvc NAME STATUS VOLUME ... owntracks-data-encrypted Bound ... (owntracks-data-proxmox gone) ``` ### Manual Verification ``` $ VIKTOR_PW=$(vault kv get -field=credentials secret/owntracks | jq -r .viktor) $ TST=$(date +%s) $ kubectl -n owntracks run t --rm -i --image=curlimages/curl -- \ curl -s -w 'HTTP %{http_code}\n' -X POST -u "viktor:$VIKTOR_PW" \ -H 'Content-Type: application/json' \ -H 'X-Limit-U: viktor' -H 'X-Limit-D: iphone-15pro' \ -d "{\"_type\":\"location\",\"lat\":51.5074,\"lon\":-0.1278,\"tst\":$TST,\"tid\":\"vb\"}" \ https://owntracks.viktorbarzin.me/pub HTTP 200 $ sleep 3 && kubectl -n dbaas exec pg-cluster-1 -c postgres -- \ psql -U postgres -d dawarich -tAc \ "SELECT ST_AsText(lonlat::geometry) FROM points WHERE user_id=1 AND timestamp=$TST" POINT(-0.1278 51.5074) ``` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 12:05:18 +00:00
-- Strip the base64 user avatar: ot-recorder appends a ~120KB `face` field
-- to enriched payloads which pushes the curl command past ARG_MAX (code=7
-- "Argument list too long"). Dawarich doesn't need it.
data.face = nil
[owntracks] Bridge Recorder → Dawarich via Lua hook script ## Context Viktor wanted live forwarding from Owntracks to Dawarich so his map stays in sync without a periodic backfill. The original plan assumed ot-recorder honoured an `OTR_HTTPHOOK` environment variable — but Recorder 1.0.1 (latest on Docker Hub as of Aug 2025) has no such feature: ``` $ kubectl -n owntracks exec deploy/owntracks -- \ strings /usr/bin/ot-recorder | grep -iE 'hook|webhook|http_post' (no matches) ``` Lua hooks, on the other hand, are first-class: `--lua-script` loads a file and calls the `otr_hook(topic, _type, data)` function for every publish. That is the pivot this commit makes. ## This change Mount a Lua script via ConfigMap and tell ot-recorder to load it: ``` Phone POST /pub ---> Traefik ---> Recorder pod | | handle_payload() writes .rec | otr_hook(topic,_type,data) | | | +---> os.execute("curl … &") | | | v | Dawarich /api/v1/owntracks/points | +---> HTTP 200 to phone ``` Per-publish cost: one `curl` subprocess, `--max-time 5`, backgrounded with `&` so it doesn't block the HTTP response to the phone. A Dawarich 5xx drops exactly one point — the `.rec` write still happens, so the one-shot backfill Job can always re-play. `DAWARICH_API_KEY` is injected from K8s Secret `owntracks-secrets` (sourced from Vault `secret/owntracks.dawarich_api_key` via the existing `dataFrom.extract` ExternalSecret). The Lua reads it with `os.getenv()` so the key never lands in Terraform state. ### Key discoveries in the verification loop (why iteration count > 1) 1. The hook function must be named `otr_hook`, not `hook` (recorder's `luasupport.c` calls `lua_getglobal(L, "otr_hook")`). The recorder logs `cannot invoke otr_hook in Lua script` when missing — the plan's `hook()` naming was wrong. 2. Dawarich's `latitude`/`longitude` scalar columns are legacy and always NULL; the authoritative geometry is in the `lonlat` PostGIS column (`ST_AsText(lonlat::geometry)`). Early "it's broken" readings were me querying the wrong columns. 3. Default Recreate-strategy rollouts cause ~30s 502/503 windows on the ingress — tolerable, but every apply is visible as an outage to the phone. Batching edits is important. ## What is NOT in this change - **Not** OTR_HTTPHOOK. Removed with this commit (dead env var). - **Not** the one-shot backfill Job — that comes after the phone buffer has flushed to avoid racing against incoming hook POSTs (follow-up: code-h2r). - **Not** Anca's bridge — a second Recorder instance or a smarter hook is needed to route her posts under her own Dawarich api_key (follow-up: code-72g). - No Ingress or Service change — Commit 1 (`a21d4a44`) already landed those. ## Test Plan ### Automated ``` $ ../../scripts/tg apply --non-interactive Apply complete! Resources: 1 added, 1 changed, 0 destroyed. $ kubectl -n owntracks logs deploy/owntracks --tail=5 + initializing Lua hooks from `/hook/dawarich-hook.lua' + dawarich-bridge: init + HTTP listener started on 0.0.0.0:8083, without browser-apikey ... + dawarich-bridge: tst=1 lat=0 lon=0 ok=true ``` ### Manual Verification ``` $ VIKTOR_PW=$(vault kv get -field=credentials secret/owntracks | jq -r .viktor) $ TST=$(date +%s) $ kubectl -n owntracks run t --rm -i --image=curlimages/curl -- \ curl -s -w 'HTTP %{http_code}\n' -X POST -u "viktor:$VIKTOR_PW" \ -H 'Content-Type: application/json' \ -H 'X-Limit-U: viktor' -H 'X-Limit-D: iphone-15pro' \ -d "{\"_type\":\"location\",\"lat\":51.5074,\"lon\":-0.1278,\"tst\":$TST,\"tid\":\"vb\"}" \ https://owntracks.viktorbarzin.me/pub HTTP 200 $ sleep 3 && kubectl -n dbaas exec pg-cluster-1 -c postgres -- \ psql -U postgres -d dawarich -c \ "SELECT timestamp, ST_AsText(lonlat::geometry) FROM points \ WHERE user_id=1 AND timestamp=$TST" timestamp | st_astext ------------+------------------------- 1776555707 | POINT(-0.1278 51.5074) ``` Real phone traffic (from in-flight buffer flush) lands in Dawarich too: `traefik logs -l app.kubernetes.io/name=traefik | grep 'POST /api/v1/owntracks/points'` shows ingress POSTs from `owntracks` namespace to `dawarich` backend with status 200. ### Reproduce locally 1. `vault login -method=oidc` 2. `kubectl -n owntracks logs deploy/owntracks --tail=20` — expect `dawarich-bridge: init` after the Lua loader line. 3. Do the curl above, poll the DB, expect `POINT(lon lat)`. Closes: code-z9b Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 23:47:22 +00:00
local url = "https://dawarich.viktorbarzin.me/api/v1/owntracks/points?api_key=" .. api_key
local payload = to_json(data)
local cmd = table.concat({
"curl -sS -o /dev/null --max-time 5 -X POST",
"-H 'Content-Type: application/json'",
"-d", escape_shell_single(payload),
escape_shell_single(url),
"&",
}, " ")
[owntracks] Strip face avatar from hook payload + drop orphan PVC Bundles two small follow-ups to the live bridge + port-fix work: ## Face avatar fix (dawarich-hook.lua) After the Recorder ran in production for a while it began enriching publish payloads with a `face` field — the base64-encoded user avatar uploaded via the Recorder's web UI (~120 KB). Our Lua hook builds a curl command that embeds the JSON payload as `-d '<payload>'`, which hit `E2BIG` / `Argument list too long` (os.execute reason=code=7) on Linux's `execve` argv limit (~128 KB). Every live POST stopped making it to Dawarich, even though the HTTP POST from the phone to Owntracks still returned 200 and the .rec write still happened. Fix: `data.face = nil` before serializing. Dawarich doesn't use it anyway (not persisted into any column — `raw_data` stored without it). Also upgraded the debug log: on failure we now emit `dawarich-bridge: FAIL tst=... reason=... code=... cmd=...` so any future variant of this problem (next big field surfaced upstream, etc.) is one log tail away from a diagnosis. ``` $ kubectl -n owntracks logs deploy/owntracks --tail=5 | grep dawarich-bridge + dawarich-bridge: init + dawarich-bridge: ok tst=1776600238 ``` ## Orphan PVC removal (main.tf) `owntracks-data-proxmox` (1 Gi, proxmox-lvm, unencrypted) was a leftover from the encrypted-migration attempt; the Deployment has been mounting `owntracks-data-encrypted` the whole time. Verified `Used By: <none>` on the live PVC before removal. Removing the resource from Terraform destroys the PVC — harmless, no data loss. ## Test Plan ### Automated ``` $ ../../scripts/tg plan Plan: 0 to add, 1 to change, 1 to destroy. $ ../../scripts/tg apply --non-interactive Apply complete! Resources: 0 added, 1 changed, 1 destroyed. $ kubectl -n owntracks get pvc NAME STATUS VOLUME ... owntracks-data-encrypted Bound ... (owntracks-data-proxmox gone) ``` ### Manual Verification ``` $ VIKTOR_PW=$(vault kv get -field=credentials secret/owntracks | jq -r .viktor) $ TST=$(date +%s) $ kubectl -n owntracks run t --rm -i --image=curlimages/curl -- \ curl -s -w 'HTTP %{http_code}\n' -X POST -u "viktor:$VIKTOR_PW" \ -H 'Content-Type: application/json' \ -H 'X-Limit-U: viktor' -H 'X-Limit-D: iphone-15pro' \ -d "{\"_type\":\"location\",\"lat\":51.5074,\"lon\":-0.1278,\"tst\":$TST,\"tid\":\"vb\"}" \ https://owntracks.viktorbarzin.me/pub HTTP 200 $ sleep 3 && kubectl -n dbaas exec pg-cluster-1 -c postgres -- \ psql -U postgres -d dawarich -tAc \ "SELECT ST_AsText(lonlat::geometry) FROM points WHERE user_id=1 AND timestamp=$TST" POINT(-0.1278 51.5074) ``` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 12:05:18 +00:00
local ok, reason, code = os.execute(cmd)
if not ok then
otr.log("dawarich-bridge: FAIL tst=" .. tostring(data.tst) ..
" reason=" .. tostring(reason) .. " code=" .. tostring(code) ..
" cmd=" .. cmd)
else
otr.log("dawarich-bridge: ok tst=" .. tostring(data.tst))
end
[owntracks] Bridge Recorder → Dawarich via Lua hook script ## Context Viktor wanted live forwarding from Owntracks to Dawarich so his map stays in sync without a periodic backfill. The original plan assumed ot-recorder honoured an `OTR_HTTPHOOK` environment variable — but Recorder 1.0.1 (latest on Docker Hub as of Aug 2025) has no such feature: ``` $ kubectl -n owntracks exec deploy/owntracks -- \ strings /usr/bin/ot-recorder | grep -iE 'hook|webhook|http_post' (no matches) ``` Lua hooks, on the other hand, are first-class: `--lua-script` loads a file and calls the `otr_hook(topic, _type, data)` function for every publish. That is the pivot this commit makes. ## This change Mount a Lua script via ConfigMap and tell ot-recorder to load it: ``` Phone POST /pub ---> Traefik ---> Recorder pod | | handle_payload() writes .rec | otr_hook(topic,_type,data) | | | +---> os.execute("curl … &") | | | v | Dawarich /api/v1/owntracks/points | +---> HTTP 200 to phone ``` Per-publish cost: one `curl` subprocess, `--max-time 5`, backgrounded with `&` so it doesn't block the HTTP response to the phone. A Dawarich 5xx drops exactly one point — the `.rec` write still happens, so the one-shot backfill Job can always re-play. `DAWARICH_API_KEY` is injected from K8s Secret `owntracks-secrets` (sourced from Vault `secret/owntracks.dawarich_api_key` via the existing `dataFrom.extract` ExternalSecret). The Lua reads it with `os.getenv()` so the key never lands in Terraform state. ### Key discoveries in the verification loop (why iteration count > 1) 1. The hook function must be named `otr_hook`, not `hook` (recorder's `luasupport.c` calls `lua_getglobal(L, "otr_hook")`). The recorder logs `cannot invoke otr_hook in Lua script` when missing — the plan's `hook()` naming was wrong. 2. Dawarich's `latitude`/`longitude` scalar columns are legacy and always NULL; the authoritative geometry is in the `lonlat` PostGIS column (`ST_AsText(lonlat::geometry)`). Early "it's broken" readings were me querying the wrong columns. 3. Default Recreate-strategy rollouts cause ~30s 502/503 windows on the ingress — tolerable, but every apply is visible as an outage to the phone. Batching edits is important. ## What is NOT in this change - **Not** OTR_HTTPHOOK. Removed with this commit (dead env var). - **Not** the one-shot backfill Job — that comes after the phone buffer has flushed to avoid racing against incoming hook POSTs (follow-up: code-h2r). - **Not** Anca's bridge — a second Recorder instance or a smarter hook is needed to route her posts under her own Dawarich api_key (follow-up: code-72g). - No Ingress or Service change — Commit 1 (`a21d4a44`) already landed those. ## Test Plan ### Automated ``` $ ../../scripts/tg apply --non-interactive Apply complete! Resources: 1 added, 1 changed, 0 destroyed. $ kubectl -n owntracks logs deploy/owntracks --tail=5 + initializing Lua hooks from `/hook/dawarich-hook.lua' + dawarich-bridge: init + HTTP listener started on 0.0.0.0:8083, without browser-apikey ... + dawarich-bridge: tst=1 lat=0 lon=0 ok=true ``` ### Manual Verification ``` $ VIKTOR_PW=$(vault kv get -field=credentials secret/owntracks | jq -r .viktor) $ TST=$(date +%s) $ kubectl -n owntracks run t --rm -i --image=curlimages/curl -- \ curl -s -w 'HTTP %{http_code}\n' -X POST -u "viktor:$VIKTOR_PW" \ -H 'Content-Type: application/json' \ -H 'X-Limit-U: viktor' -H 'X-Limit-D: iphone-15pro' \ -d "{\"_type\":\"location\",\"lat\":51.5074,\"lon\":-0.1278,\"tst\":$TST,\"tid\":\"vb\"}" \ https://owntracks.viktorbarzin.me/pub HTTP 200 $ sleep 3 && kubectl -n dbaas exec pg-cluster-1 -c postgres -- \ psql -U postgres -d dawarich -c \ "SELECT timestamp, ST_AsText(lonlat::geometry) FROM points \ WHERE user_id=1 AND timestamp=$TST" timestamp | st_astext ------------+------------------------- 1776555707 | POINT(-0.1278 51.5074) ``` Real phone traffic (from in-flight buffer flush) lands in Dawarich too: `traefik logs -l app.kubernetes.io/name=traefik | grep 'POST /api/v1/owntracks/points'` shows ingress POSTs from `owntracks` namespace to `dawarich` backend with status 200. ### Reproduce locally 1. `vault login -method=oidc` 2. `kubectl -n owntracks logs deploy/owntracks --tail=20` — expect `dawarich-bridge: init` after the Lua loader line. 3. Do the curl above, poll the DB, expect `POINT(lon lat)`. Closes: code-z9b Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 23:47:22 +00:00
end