viktor/infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1106 commits 2 branches 0 tags 2.3 GiB
Commit graph

113 commits

Author SHA1 Message Date
Viktor Barzin
a1d945a0b2 add prometheus alerts for deployment/statefulset/daemonset replica mismatches [ci skip] 
- Add DeploymentReplicasMismatch alert
- Add StatefulSetReplicasMismatch alert
- Add DaemonSetMissingPods alert
- Add .claude/ directory with remote executor and knowledge base
 2026-01-18 11:04:51 +00:00
Viktor Barzin
185a138cc5 dedup ram alert and increase threshold to 95% [ci skip] 2026-01-17 22:42:22 +00:00
Viktor Barzin
61e318398c scale grafana to 3 pods for resilience [ci skip] 2026-01-12 18:27:54 +00:00
Viktor Barzin
f1e9fb9afe add tier to all deployments [ci skip] 2026-01-10 16:28:14 +00:00
Viktor Barzin
1b5cbeb9c8 monitor idrac more frequently [ci skip] 2026-01-07 18:55:59 +00:00
Viktor Barzin
934fa34c79 update cpu temp alert to above 60 [ci skip] 2026-01-04 12:26:46 +00:00
Viktor Barzin
01d4c9c3e1 update definition of high cpu usage to use pve metrics in stead for a longer period [ci skip] 2026-01-03 23:30:28 +00:00
Viktor Barzin
31c403cadb update cpu temp alert to 55C down from 75C [ci skip] 2026-01-03 16:48:54 +00:00
Viktor Barzin
d37c693a94 increase idrac scrape timeout in attempt to reduce 499 [ci skip] 2025-12-29 20:34:40 +00:00
Viktor Barzin
253e77f22d add registry low cache hit rate alert [ci skip] 2025-12-29 10:43:57 +00:00
Viktor Barzin
f1dde96d80 replace hardcoded namespace with module reference [ci skip] 2025-12-29 10:23:42 +00:00
Viktor Barzin
cf9d346cae add more alerts in prometheus and gorup them better [ci skip] 2025-12-28 20:07:33 +00:00
Viktor Barzin
a595c4db56 move out all monitoring resources to separate tf files [ci skip] 2025-12-28 20:07:00 +00:00
Viktor Barzin
26d55c6637 move grafana into separate file and tunr off persistence as we use external db now [ci skip] 2025-12-28 20:05:27 +00:00
Viktor Barzin
f06e050eaa migrate grafana to mysql from sqlite [ci skip] 2025-12-27 20:51:05 +00:00
Viktor Barzin
0b2e6d09d2 move prometheus wal to tmpfs to reduce wear [ci skip] 2025-12-26 20:10:20 +00:00
Viktor Barzin
6c1ae20448 add job to monitor pve host using node exporter and add alert for high ssd writes [ci skip] 2025-12-26 16:23:49 +00:00
Viktor Barzin
d07c625064 add pve exporter playbook + pve exporter in k8s [ci skip] 2025-12-26 16:23:17 +00:00
Viktor Barzin
59e6591e2a update most important grafana dashboards [ci skip] 2025-12-23 18:13:25 +00:00
Viktor Barzin
a225bad3cb add alert for docker registry [ci skip] 2025-12-18 10:45:32 +00:00
Viktor Barzin
397fa0cba7 add local-only ingress for snmp and idrac exporters [ci skip] 2025-12-14 19:08:44 +00:00
Viktor Barzin
b4f45c7e73 add separate idrac monitoring tool and dashboard [ci skip] 2025-12-14 09:50:16 +00:00
Viktor Barzin
34df786fe4 add haos monitoring job in prometheus 2025-11-29 11:46:42 +00:00
Viktor Barzin
1b0d5e60d8 add ${__field.name:wrap} in the idrac dashboard to fix wrapping issue[ci skip] 2025-11-15 05:15:50 +00:00
Viktor Barzin
0b7b092c26 add api key to tiny tuya target in prometheus scrape [ci skip] 2025-11-09 22:03:25 +00:00
Viktor Barzin
5f2cc75a8e add prometheus targets for fuses [ci skip] 2025-10-29 21:59:06 +00:00
Viktor Barzin
76103f52e3 add alert if we use inverter power for 1d straight - probably an issue with switching [ci skip] 2025-10-29 20:09:21 +00:00
Viktor Barzin
18a2695e64 add breakdown in main power source from inverterer in grafana [ci skip] 2025-10-28 22:41:44 +00:00
Viktor Barzin
77f92ae4ef update ups grafana dash to have inverter stats [ci skip] 2025-10-28 22:17:32 +00:00
Viktor Barzin
3161da29a4 add scrape config for tuya bridge and prohibit access to the metrics path via ingress [ci skip] 2025-10-28 21:38:40 +00:00
Viktor Barzin
d5dd81ba30 increaes threshold for high power usage to 180 as we have bigger cpu now [ci skip] 2025-10-08 20:33:51 +00:00
Viktor Barzin
c5bb343ebe disable errors for matrix ingress [ci skip] 2025-08-23 20:38:53 +00:00
Viktor Barzin
87c33629b4 backup all grafana dashboards [ci skip] 2025-08-23 20:30:37 +00:00
Viktor Barzin
1530323477 update registry prometheus url to devvm as pi was too slow [ci skip] 2025-08-23 20:15:05 +00:00
Viktor Barzin
a87b9793ad disable loki and alloy as it is not used [ci skip] 2025-08-23 20:02:37 +00:00
Viktor Barzin
c49e4d0a86 add loki + alloy deployments for logs collection [ci skip] 2025-05-04 11:25:39 +00:00
Viktor Barzin
c5ae6873c9 add registry monitoring to prometheus [ci skip] 2025-03-30 11:15:54 +00:00
Viktor Barzin
6100372a9e adjust batter low alert to fire only when there is no pwoer [ci skip] 2025-03-22 15:47:30 +00:00
Viktor Barzin
8eb18ec651 add power and ups battery over time widgets to grafana [ci skip] 2025-03-22 15:46:17 +00:00
Viktor Barzin
cd89d13ab2 disable alert for pods less than in spec [ci skip] 2025-03-16 18:27:13 +00:00
Viktor Barzin
59a651c2ad add 2 more oids for ups to monitor active and reactive power consumption [ci skip] 2025-03-15 17:54:04 +00:00
Viktor Barzin
dfe47657ee disable perms errors and server errors for grafana and nextcloud ingresses as they were too noisy [ci skip] 2025-03-15 17:53:24 +00:00
Viktor Barzin
1e674bff7e add alert for ups low battery remaining [ci skip] 2025-03-02 20:48:07 +00:00
Viktor Barzin
ab9b5b356a increase low voltage alert to 10 min [ci skip] 2025-03-01 14:28:56 +00:00
Viktor Barzin
6a6a6974e2 increase interval for 500 alerts to 20m [ci skip] 2025-01-10 20:47:25 +00:00
Viktor Barzin
f291d8545b move prometheus alerts to different channel and move high cpu period [ci skip] 2025-01-04 14:27:48 +00:00
Viktor Barzin
4643e22cc8 increase idle power threshold to 130w [ci skip] 2025-01-03 17:49:24 +00:00
Viktor Barzin
46736680a6 add alert status to message [ci skip] 2025-01-02 21:13:09 +00:00
Viktor Barzin
53d8b2d2c6 update prometheus alerts to be correctly grouped and sent to slack and deprecate some old ones [ci skip] 2025-01-02 20:33:55 +00:00
Viktor Barzin
48a0deb283 update prometheus chart values to get slack notiifcations to work and add alerts for 4xx and 5xx on ingress [ci skip] 2025-01-01 11:39:16 +00:00