ymrtech/nix-config
1
0
Fork 0
Code Issues 6 Pull requests 2 Projects Releases Packages Activity Actions

feat: Rewrite monitoring dashboards + fix Nix modules #42

Merged
ymrtech merged 17 commits from feat/monitoring-dashboards into main 2026-08-29 00:23:03 -06:00
Conversation 0 Commits 17 Files changed 17 +8014 -6310
ymrtech commented 2026-08-28 21:28:52 -06:00
Owner
Copy link

Summary

Complete overhaul of the monitoring stack: 9 rewritten Grafana dashboards with 90 panels, plus 6 Nix module fixes.

Dashboard Changes

New Dashboard:

  • security-overview.json (15 panels) - Centralized security monitoring with SSH, sudo, audit, and network security metrics

Rewritten Dashboards:

  • command-tracking.json (12 panels) - Audit-based command execution tracking with real auditd data shapes
  • public-services-logs.json (17 panels) - Caddy access logs with request metrics, latency, and error tracking
  • victorialogs-overview.json (8 panels) - VictoriaLogs ingestion, query performance, and error rates
  • victoriametrics-performance.json (10 panels) - VM internals: storage, ingestion, query latency
  • network-outbound.json (4 panels) - Network egress, connection tracking, blocklist hits
  • anomaly-detection-ai.json (10 panels) - Z-score anomaly detection for load, memory, disk, network
  • system-overview.json (6 panels) - Host-level CPU, memory, disk, network metrics
  • log-browser-raw.json (8 panels) - Raw journald log browser with filtering

Total: 90 panels across 9 dashboards

All dashboards use verified LogsQL/PromQL expressions, correct queryType fields (stats/statsRange/logs), and proper datasource UIDs.

Nix Module Fixes

  1. command-tracking.nix: Audit rules moved from /etc/audit/rules.d/ to security.audit.rules
  2. victorialogs.nix: Extended FilterMatches to include audit, sshd, sudo logs
  3. hermes-webhook.nix: Added tmpfiles rule for spool directory ownership
  4. grafana.nix: Removed broken Alerting Rules dashboard provider hack
  5. hosts/command/default.nix: Added missing monitoring module import
  6. hosts/mail/default.nix: Fixed services.monitoring-server prefix

Validation

All configurations evaluate successfully (giga, command, mail).

Deployment Notes

After merge and deploy:

  1. Restart grafana: sudo systemctl restart grafana
  2. Audit rules will load automatically via audit-rules-nixos service
  3. VictoriaLogs will now receive audit + auth logs (previously filtered out)
  4. Hermes webhook spool directory will be created with correct ownership
## Summary Complete overhaul of the monitoring stack: 9 rewritten Grafana dashboards with 90 panels, plus 6 Nix module fixes. ### Dashboard Changes **New Dashboard:** - **security-overview.json** (15 panels) - Centralized security monitoring with SSH, sudo, audit, and network security metrics **Rewritten Dashboards:** - **command-tracking.json** (12 panels) - Audit-based command execution tracking with real auditd data shapes - **public-services-logs.json** (17 panels) - Caddy access logs with request metrics, latency, and error tracking - **victorialogs-overview.json** (8 panels) - VictoriaLogs ingestion, query performance, and error rates - **victoriametrics-performance.json** (10 panels) - VM internals: storage, ingestion, query latency - **network-outbound.json** (4 panels) - Network egress, connection tracking, blocklist hits - **anomaly-detection-ai.json** (10 panels) - Z-score anomaly detection for load, memory, disk, network - **system-overview.json** (6 panels) - Host-level CPU, memory, disk, network metrics - **log-browser-raw.json** (8 panels) - Raw journald log browser with filtering **Total: 90 panels across 9 dashboards** All dashboards use verified LogsQL/PromQL expressions, correct queryType fields (stats/statsRange/logs), and proper datasource UIDs. ### Nix Module Fixes 1. **command-tracking.nix**: Audit rules moved from `/etc/audit/rules.d/` to `security.audit.rules` 2. **victorialogs.nix**: Extended FilterMatches to include audit, sshd, sudo logs 3. **hermes-webhook.nix**: Added tmpfiles rule for spool directory ownership 4. **grafana.nix**: Removed broken Alerting Rules dashboard provider hack 5. **hosts/command/default.nix**: Added missing monitoring module import 6. **hosts/mail/default.nix**: Fixed `services.monitoring-server` prefix ### Validation All configurations evaluate successfully (giga, command, mail). ### Deployment Notes After merge and deploy: 1. Restart grafana: `sudo systemctl restart grafana` 2. Audit rules will load automatically via audit-rules-nixos service 3. VictoriaLogs will now receive audit + auth logs (previously filtered out) 4. Hermes webhook spool directory will be created with correct ownership
- command-tracking.json: full rewrite around real auditd shapes
- public-services-logs.json: use verified copy, fix corrupted file
- victorialogs-overview.json: replace __stream__ filters with *, add queryType
- victoriametrics-performance.json: fix histogram refs, add queryType
- network-outbound.json: add missing queryType fields
- anomaly-detection-ai.json: replace __stream__ filters, add queryType
- system-overview.json: add missing queryType fields
- security-overview.json: NEW centerpiece security dashboard (15 panels)
- All 90 panels across 9 dashboards pass syntax checks
- command-tracking.nix: move audit rules from /etc/audit/rules.d to security.audit.rules
  (NixOS loads rules via audit-rules-nixos service, not rules.d directory)
- victorialogs.nix: add FilterMatches for audit, sshd, sudo logs
  (PRIORITY<=3 was blocking audit events at PRIORITY=4)
- hermes-webhook.nix: add tmpfiles rule for spool directory ownership
  (preStart touch was failing due to permission denied)
- grafana.nix: remove broken Alerting Rules dashboard provider hack
  (use proper provision.alerting.rules.settings instead)
- hosts/command/default.nix: import monitoring module, fix services.monitoring-server prefix
- hosts/mail/default.nix: fix services.monitoring-server prefix
- log-browser-raw.json: fix queryType for timeseries panels (statsRange not stats)
- vpn: Add externalLabels = { instance = "vpn"; } so metrics are properly labeled
- mail: Add services.monitoring-agent to send metrics and logs to central server

This ensures all hosts (except command) send monitoring data to public.ymr
Add externalLabels = { instance = "mail"; } to ensure metrics from mail
host are properly labeled in VictoriaMetrics instead of showing as localhost.
Mail host is on 10.0.0.x network without WireGuard, so use
public's LAN IP (10.0.0.220) instead of WireGuard IP (11.0.0.7)
- Generate WireGuard keys on mail host (11.0.0.6)
- Add mail as peer on VPN host
- Update monitoring URLs to use WireGuard IP (11.0.0.7)
- Create preshared key on VPN for mail peer
Mail's WireGuard was standard wg, but the VPN runs amneziawg with
Jc/Jmin/Jmax/S1/S2/H1-H4 obfuscation. Without these the handshake
never completes. Match giga's working config pattern.
The peer publicKey must be the VPN server interface key
(j3fM602CdZVuGJBACyR8nB7SKmB7T4zZGkA9SHjnjHg=), not public's key.
Giga's working config confirms the correct key.
Break every panel down by host so baselines don't mix across machines:
- Z-score stats (CPU/mem/disk) compute per instance, not pooled avg()
- Add host template variable (All/giga/public/vpn/mail) via {instance=~"$host"}
- VL error panels group by _HOSTNAME
- Legends identify instance/host; no more phantom baselines
- Fix PromQL selector placement (matchers on metric, not expression tail)
The Grafana VL plugin strips quotes around $host interpolation,
producing invalid re(.*). Match log-browser-raw pattern: use
stats by (_HOSTNAME) directly — one series per host, no filter var.
On larger timeframes some records lack _HOSTNAME (kernel/tmpfs
sources) -> plugin renders an extra series labeled with the raw
query text. Add _HOSTNAME:* filter so only per-host series remain.
Misconfigured + wasting resources (firezone-server web+api Elixir
processes, ports 13000/13001). Commented out with clear markers:
- services.firezone.server block (re-enable by uncommenting)
- firezone_web / firezone_api vmagent scrape targets

nginx vhost for vpn.ymrtech.com left in place (keeps ACME cert
renewal alive for re-enable); will 502 until firezone returns.
postgresql on vpn exists solely for firezone's DB (ensureUsers/
ensureDatabases firezone) and was failing to start after the rebuild.
Disable service + drop its vmagent scrape target.
hermes force-pushed feat/monitoring-dashboards from 81132f7167 to 64bfd0bd79 2026-08-29 00:22:15 -06:00 Compare
ymrtech deleted branch feat/monitoring-dashboards 2026-08-29 00:23:03 -06:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backlog

   
bug

   
enhancement

   
in-progress

   
review

   
todo
No labels
backlog
bug
enhancement
in-progress
review
todo
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ymrtech/nix-config!42
No description provided.