Commit graph

77 commits

Author SHA1 Message Date
Viktor Barzin
534fcdbfe3
adjust batter low alert to fire only when there is no pwoer [ci skip] 2025-03-22 15:47:30 +00:00
Viktor Barzin
daeb3b6693
add power and ups battery over time widgets to grafana [ci skip] 2025-03-22 15:46:17 +00:00
Viktor Barzin
987fc402b5
disable alert for pods less than in spec [ci skip] 2025-03-16 18:27:13 +00:00
Viktor Barzin
d9e06a9853
add 2 more oids for ups to monitor active and reactive power consumption [ci skip] 2025-03-15 17:54:04 +00:00
Viktor Barzin
72bedfdd6e
disable perms errors and server errors for grafana and nextcloud ingresses as they were too noisy [ci skip] 2025-03-15 17:53:24 +00:00
Viktor Barzin
f7eff3cb74
add alert for ups low battery remaining [ci skip] 2025-03-02 20:48:07 +00:00
Viktor Barzin
095624a337
increase low voltage alert to 10 min [ci skip] 2025-03-01 14:28:56 +00:00
Viktor Barzin
5ef9ba5917
increase interval for 500 alerts to 20m [ci skip] 2025-01-10 20:47:25 +00:00
Viktor Barzin
aeee71751f
move prometheus alerts to different channel and move high cpu period [ci skip] 2025-01-04 14:27:48 +00:00
Viktor Barzin
3473f64670
increase idle power threshold to 130w [ci skip] 2025-01-03 17:49:24 +00:00
Viktor Barzin
4b725b02a6
add alert status to message [ci skip] 2025-01-02 21:13:09 +00:00
Viktor Barzin
c7113fa495
update prometheus alerts to be correctly grouped and sent to slack and deprecate some old ones [ci skip] 2025-01-02 20:33:55 +00:00
Viktor Barzin
9b0d686873
update prometheus chart values to get slack notiifcations to work and add alerts for 4xx and 5xx on ingress [ci skip] 2025-01-01 11:39:16 +00:00
Viktor Barzin
40f4354316
fix monitoring stack [ci skip] 2024-12-31 17:15:06 +00:00
Viktor Barzin
d94f39f531
add all grafana dashboards models [ci skip] 2024-12-24 13:48:21 +00:00
Viktor Barzin
ce90629b54
add low voltage alert to prometheus and update some dashboards [ci skip] 2024-12-23 18:21:01 +00:00
Viktor Barzin
0ef7430b6f
fix typo in idrac voltage to be in volts not watts [ci skip] 2024-12-17 19:35:27 +00:00
Viktor Barzin
e6aa28be1c
update ups grafana [ci skip] 2024-12-17 19:22:08 +00:00
Viktor Barzin
23a882a3d5
update idract refresh rate to 1m[ci skip] 2024-12-17 19:05:57 +00:00
Viktor Barzin
63df62ce1f
add idrac grafana dashboard to repo [ci skip] 2024-12-16 22:36:00 +00:00
Viktor Barzin
ec8f672dfd
add grafana dashboard for ups [ci skip] 2024-12-15 20:58:01 +00:00
Viktor Barzin
fbe305a891 add ups snmp exporter to prometheus [ci skip] 2024-12-15 18:13:33 +00:00
Viktor Barzin
718ab77e68
move grafana and k8s dashboard to use authentik instead of oauth proxy [ci skip] 2024-11-22 00:47:00 +00:00
Viktor Barzin
185a944acd
replace oauth proxy with authentik auth [ci skip] 2024-11-18 22:06:31 +00:00
Viktor Barzin
64f81621c8 add homepage module and some more integrations [ci skip] 2024-10-20 13:05:03 +00:00
Viktor Barzin
b54fbf72fd
add meshcentral and diun[ci skip] 2024-08-18 18:14:22 +00:00
Viktor Barzin
506b4a2f87
reduce prometheus storage retention from 12w -> 8w to save ~30gb [ci skip] 2024-08-07 20:18:13 +00:00
Viktor Barzin
828f3f115a
update old prometheus alert detectors and upgrade immich to 101 [ci skip] 2024-04-12 21:15:31 +00:00
Viktor Barzin
8afbec0d23
remove hack for london openwrt monitoring after having tailscale now [ci skip] 2024-03-30 18:28:11 +00:00
Viktor Barzin
e5061dec27
update openwrt london prometheus target address [ci skip] 2024-03-29 22:20:29 +00:00
Viktor Barzin
215deb5568
add monitoring jobs to p8s for istiod and the service mesh [ci skip] 2024-01-07 17:47:36 +00:00
Viktor Barzin
15bade148c
upgrade prometheus helm chart [ci skip] 2023-12-25 21:40:19 +00:00
Viktor Barzin
e3a8cd16b4
add baseurl to prometheus helm to chart so alertmanager sends correct links with prometheus public url instead of podname [ci skip] 2023-12-25 13:48:19 +00:00
Viktor Barzin
3019f1cca8 add prometheus monitoring to crowdsec [ci skip] 2023-11-25 13:34:16 +00:00
Viktor Barzin
bd4754b339
move grafana to nfs [ci skip] 2023-11-11 00:16:58 +00:00
Viktor Barzin
73d63b7713
add alert if node memory exceeds 90% [ci skip] 2023-11-10 22:48:45 +00:00
Viktor Barzin
0162435a88
use nfs to prometheus [ci skip] 2023-11-10 22:20:25 +00:00
Viktor Barzin
1afb83e426
use .lan domain for idrac metrics scrape [ci skip] 2023-11-01 20:44:17 +00:00
Viktor Barzin
c192d32127
add repo for the dockerfile for the redifsh exporter [ci skip] 2023-10-24 11:46:18 +00:00
Viktor Barzin
3c394e0e82
update redifhs exporter to new implementation [ci skip] 2023-10-24 11:44:19 +00:00
Viktor Barzin
3d7ca3c57d
make dashy publicly accessible [ci skip] 2023-10-23 22:05:56 +00:00
Viktor Barzin
50b57e1373
replace tls client cert auth with oauth and add localai stub [ci skip] 2023-10-22 14:07:18 +00:00
Viktor Barzin
9b5ed514cd
add alert on new client registration and update dns to use pfsense [ci skip] 2023-09-18 08:03:50 +00:00
Viktor Barzin
cd47f924b7
disable email notifications as they are spammy and using sendgrid quota [ci skip] 2023-06-20 14:04:02 +00:00
viktorbarzin
c87376b670
remove 1gb limit for tsdb to confirm it was the root cause for memory issues [ci skip] 2023-04-21 23:04:39 +01:00
viktorbarzin
64491f9028
attempt ot reduce prometheus memory by setting --storage.tsdb.retention.size; laso add metrics-api which is not working atm [ci skip] 2023-04-17 01:28:03 +01:00
viktorbarzin
a75e647b48
add alert for unhandled exceptions [ci skip] 2023-04-05 00:15:10 +01:00
viktorbarzin
9fe57e2ec6
reword finance app webhook exception alerting [ci skip] 2023-04-03 23:35:33 +01:00
viktorbarzin
6e7de7e195
add prometheus pv and pvc [ci skip] 2023-04-03 23:21:48 +01:00
viktorbarzin
6838d319a2
add counter for overall webhook failures 2023-04-03 22:37:59 +01:00