Homelab Monitoring Stack

Prometheus + Blackbox Exporter + Grafana running in Docker Compose, with mTLS on all internal connections and Ansible for node management.

Architecture

                        ┌─────────────────────────────────────┐
                        │           Docker Compose             │
                        │                                      │
  Browser ──────────────┤→ Grafana :3009  ──mTLS──→ Prometheus :9090
                        │                               │       │
                        │                          mTLS │       │ HTTP
                        │                               ↓       ↓
                        │                        Blackbox :9115  │
                        └───────────────────────────────────────┘
                                                        │
                               ┌────────────────────────┘
                               │ HTTPS probes
                               ↓
                    pasantru.es, *.homelab.pasantru.es

  Prometheus ──mTLS──→ node_exporter :9100  (nfs: 192.168.1.25)
  Prometheus ──mTLS──→ node_exporter :9100  (videoserver: 192.168.1.10)

Services

Service	Image	Port (host)
Prometheus	prom/prometheus:v3.9.1	9090
Blackbox Exporter	quay.io/prometheus/blackbox-exporter:latest	—
Grafana	grafana/grafana:latest	3009

Prerequisites

• Docker + Docker Compose
• step CLI (for cert generation)
• Ansible (for node_exporter deployment to remote nodes)

Directory Structure

.
├── docker-compose.yml                        # Service definitions
├── prometheus.yml                            # Scrape configs and targets
├── prometheus-web-config.yml                 # Prometheus TLS server + mTLS client auth config
├── blackbox.yml                              # Blackbox prober modules
├── .gitignore                                # Ignores certs/ and generated datasource YAML
├── scripts/
│   └── gen-certs.sh                          # Generates all PKI certs + Grafana datasource YAML
├── certs/                                    # (gitignored) Generated TLS certificates
│   ├── ca.crt / ca.key                       # Root CA
│   ├── prometheus-server.crt / .key          # Prometheus TLS server cert
│   ├── prometheus-client.crt / .key          # Prometheus mTLS client cert (scrapes nodes)
│   ├── grafana-client.crt / .key             # Grafana mTLS client cert (datasource)
│   ├── nfs-server.crt / .key                 # node_exporter cert for nfs
│   └── videoserver-server.crt / .key         # node_exporter cert for videoserver
├── grafana/
│   └── provisioning/
│       ├── datasources/
│       │   ├── prometheus.yml                # (gitignored) Generated with embedded certs
│       │   └── prometheus.yml.example        # Tracked template showing structure
│       ├── dashboards/
│       │   ├── dashboards.yml                # Dashboard provisioning config
│       │   ├── homelab.json                  # Blackbox probe dashboard
│       │   └── node-overview.json            # Node metrics dashboard (CPU/mem/disk/net)
│       └── alerting/
│           └── smart-alerts.yml              # SMART disk alert rules
└── ansible/
    ├── ansible.cfg
    ├── inventory/
    │   └── hosts.yml                         # nfs + videoserver hosts
    ├── playbooks/
    │   ├── site.yml                          # Full setup: common + node_exporter
    │   └── update.yml                        # Apt update/upgrade only
    └── roles/
        ├── common/                           # apt update, dist-upgrade, autoremove
        └── node_exporter/                    # Install binary + mTLS config

Setup

1. Generate certificates

Install step CLI, then from the repo root:

./scripts/gen-certs.sh

This creates all certs under certs/ (gitignored) and writes grafana/provisioning/datasources/prometheus.yml with the CA and Grafana client cert embedded directly — no volume mounts needed for Grafana.

Re-run the script any time you need to rotate certs. CA is preserved if it already exists.

2. Deploy node_exporter to nodes

ansible-playbook ansible/playbooks/site.yml

This installs node_exporter on nfs (192.168.1.25) and videoserver (192.168.1.10), deploys the host-specific server cert, and starts node_exporter with mTLS enabled.

3. Start the stack

docker compose up -d

Grafana: http://localhost:3009 — default credentials admin / admin (change on first login).

4. Reload Prometheus config (if already running)

curl -X POST http://localhost:9090/-/reload

---

mTLS

All internal traffic is encrypted and mutually authenticated:

Connection	Client cert	Server cert
Grafana → Prometheus	grafana-client	prometheus-server
Prometheus → node_exporter (nfs)	prometheus-client	nfs-server
Prometheus → node_exporter (videoserver)	prometheus-client	videoserver-server

All certs are signed by the local homelab-ca. TTL is 1 year — re-run gen-certs.sh before expiry.

Verify mTLS

# node_exporter responds over mTLS
curl --cacert certs/ca.crt \
     --cert certs/prometheus-client.crt \
     --key  certs/prometheus-client.key \
     https://192.168.1.25:9100/metrics | head

# Prometheus HTTPS endpoint
curl --cacert certs/ca.crt \
     --cert certs/grafana-client.crt \
     --key  certs/grafana-client.key \
     https://localhost:9090/-/healthy

---

Monitored Targets

Blackbox (HTTPS probe + SSL check)

Configured in prometheus.yml under the blackbox job:

• https://pasantru.es
• https://router.homelab.pasantru.es
• https://r620.homelab.pasantru.es
• https://r740.homelab.pasantru.es
• https://r230.homelab.pasantru.es

Add new targets to the blackbox job's static_configs list in prometheus.yml.

Node metrics (node_exporter over mTLS)

Host	IP	Metrics collected
nfs	192.168.1.25	CPU, memory, disk, network
videoserver	192.168.1.10	CPU, memory, disk, network, SMART

---

Dashboards

Dashboard	Description
Homelab	Blackbox probe status and SSL expiry
Node Overview	Per-node CPU, memory, disk I/O, network — filterable by job

Dashboards are auto-provisioned from grafana/provisioning/dashboards/.

---

Common Operations

# Start the stack
docker compose up -d

# Stop the stack
docker compose down

# Restart after config changes
docker compose restart

# Reload Prometheus config without restart
curl -X POST http://localhost:9090/-/reload

# View logs
docker compose logs -f prometheus
docker compose logs -f grafana

# Update packages on nodes (no node_exporter changes)
ansible-playbook ansible/playbooks/update.yml

# Redeploy node_exporter (e.g. after cert rotation)
ansible-playbook ansible/playbooks/site.yml

---

Grafana Credentials

• URL: http://localhost:3009
• Default: admin / admin
• SMTP alerts via Mailgun are pre-configured in docker-compose.yml