I wanted monitoring on my home cluster, so I set up Prometheus, Alertmanager and PagerDuty step by step. Writing the configs myself instead of using a pre-made stack helped me understand what each part actually does. These are my notes.

The setup

Three Proxmox nodes. Each host runsnode_exporter. One VM runs Prometheus and Alertmanager. Another runsblackbox_exporter for HTTP probes.

Prometheus has three scrape jobs:

scrape_configs:-job_name:'nodes'static_configs:-targets:['10.0.0.10:9100','10.0.0.11:9100','10.0.0.12:9100']-job_name:'blackbox-http'metrics_path:/probeparams:module:[http_2xx]static_configs:-targets:-https://cloud.emmanueltekle.nl-https://emmanueltekle.nlrelabel_configs:-source_labels:[__address__]target_label:__param_target-source_labels:[__param_target]target_label:instance-target_label:__address__replacement:127.0.0.1:9115-job_name:'prometheus'static_configs:-targets:['localhost:9090']

Theblackbox-http relabel block took me a while to get right. The__param_target rewrite is what tellsblackbox_exporter which URL to probe on each scrape — without it, you get metrics but the target is empty.

Alertmanager

The mental model that helped me: labels are inputs, routes are filters, receivers are destinations.

route:receiver:'default-pagerduty'group_by:['alertname','instance']group_wait:30sgroup_interval:5mrepeat_interval:4hroutes:-matchers:-severity = "info"receiver:'null'receivers:-name:'default-pagerduty'pagerduty_configs:-service_key:'...'-name:'null'

Info-level alerts go to a null receiver. Everything else pages.

Testing the chain

I needed to verify alerts actually reach PagerDuty without breaking a real service. I added aTestAlert rule with a constant true expression:

-alert:TestAlertexpr:vector(1) > 0for:0mlabels:severity:criticalannotations:summary:"Manual test alert"

Reload Prometheus, wait for the alert to fire, get the SMS. Five minutes later the voice escalation came in because I didn’t acknowledge it. Then I disabled the rule again.

Things that tripped me up

• A wrong__param_target rewrite meant every blackbox metric had an emptyinstance label, so Alertmanager couldn’t deduplicate alerts.
• group_interval: 30s (which I had early on) flooded my phone with 12 pages from one outage.5m is much saner.
• I had the wrong PagerDuty integration key for a few days. Alerts fired in Alertmanager but PagerDuty showed nothing. The Alertmanager logs surfaced it eventually.

Repo

github.com/E-mma9/Monitoring-Homelab — full config and Grafana dashboard JSON.

I wrote two Bash scripts to harden a fresh Ubuntu/Debian server and audit it afterwards. Posting the structure and a few notes here.

What gets configured

harden.sh:

• SSH — disable root login, disable password auth, change to a non-default port
• UFW — default-deny, allow SSH/HTTP/HTTPS
• auditd for security logging
• unattended-upgrades for security patches
• fail2ban for brute-force protection

security-audit.sh:

• Verifies SSH config against the hardened baseline
• Checks UFW status and rules
• Reports failed logins fromjournalctl
• Lists pending updates
• Exits non-zero on drift (useful for CI / cron)

Idempotency

Bash idempotency takes a bit of care. Each change is wrapped in a check so the script can be re-run safely:

ssh_set(){localkey="$1"val="$2"if grep -qE"^${key}[[:space:]]" /etc/ssh/sshd_config;then sed -i"s|^${key}.*|${key}${val}|" /etc/ssh/sshd_configelseecho"${key}${val}" >> /etc/ssh/sshd_configfi}ssh_set"PermitRootLogin""no"ssh_set"PasswordAuthentication""no"ssh_set"Port""2222"

Matching^${key}[[:space:]] avoids picking up commented-out lines. The function handles both update and insert in one place.

The audit script

check(){localdesc="$1"cmd="$2"ifeval"$cmd" >/dev/null 2>&1;thenecho" [OK]$desc"elseecho" [FAIL]$desc"EXIT=1fi}check"SSH root login disabled"\"grep -qE '^PermitRootLogin no' /etc/ssh/sshd_config"check"SSH password auth disabled"\"grep -qE '^PasswordAuthentication no' /etc/ssh/sshd_config"check"UFW active""ufw status | grep -q 'Status: active'"check"auditd running""systemctl is-active --quiet auditd"

[OK]/[FAIL] columnar output is easy to pipe into a Slack notification or store in a log.

When this stops being the right tool

For one server, Bash is fine. For ten, I’d move to Ansible — the inventory and playbook structure pay for themselves. The scripts were a way to get familiar with the actual files and directives. After that, config management makes more sense.

Repo

github.com/E-mma9/Linux-Security-Hardening

A short write-up on my home Proxmox cluster — why three nodes, how it’s wired, and what I’ve learned. I get asked about this a lot, so this is the canonical answer.

Hardware

Three identical second-hand Intel N100 mini-PCs, ~€180 each. Each one has:

• N100 CPU, 16 GB RAM, 512 GB NVMe
• 2.5 GbE — matters for cluster + storage traffic
• Headless, Proxmox VE 8

Total around €600. Power draw is roughly 8 W idle, 15 W loaded per node.

Why three instead of one

A single bigger server would be cheaper to run and quieter. Three nodes give me things a single host can’t:

• HA pairs — Pi-hole and a few other services run as an HA pair across two nodes. I can take a node down for kernel updates without breaking DNS for the house.
• Live migration —qm migrate <vmid> <target> moves a VM between hosts with a few seconds of network blip. Useful operationally, and useful for testing.
• Real failure modes — pulling a power cable on one node behaves differently from rebooting a single host. The behaviour during recovery is the actual reason I have three.

Networks

Four VLANs, segmented on an OPNsense box:

• MGMT (VLAN 10) — Proxmox UI, SSH, cluster heartbeat
• STORAGE (VLAN 20) — Ceph backend, jumbo frames (MTU 9000)
• PROD (VLAN 30) — VM traffic
• TRUST (VLAN 100) — daily-driver hosts

Splitting storage from production traffic means a heavy workload doesn’t starve the management plane.

Storage

I tried Ceph for a while because I wanted to learn it. On three N100s it’s not fast, but it does heal correctly when I pull a disk. After about three months I moved most things back to ZFS replication — simpler to reason about, and at this scale the resilience of Ceph wasn’t worth its operational weight for me.

What it’s not

• Not cheaper than a single server, on hardware or electricity
• Not quieter — three fans humming
• Not simpler

It’s just closer to a production setup, which is the only reason I run it.

What I’d do differently

Start with three nodes again, but skip Ceph and go straight to ZFS + scheduled replication. The time I spent on Ceph was educational but not directly useful for the workloads I actually run at home.