gitea-migration/setup/pi-monitoring/USAGE_GUIDE.md

Pi Monitoring Usage Guide

Step-by-step runbook for setting up a brand-new Raspberry Pi as your monitoring and container admin node.

1) Prepare Raspberry Pi OS

Recommended image: Raspberry Pi OS Lite 64-bit (Bookworm).

In Raspberry Pi Imager advanced options:

• set hostname (example: pi-ops)
• enable SSH
• configure SSH key auth
• set username/password
• set timezone/locale

Use wired Ethernet and an SSD for persistent data.

2) Bootstrap host

SSH to the Pi:

ssh <user>@<pi-ip>
cd /path/to/gitea-migration/setup/pi-monitoring
./bootstrap_pi.sh --timezone=America/New_York --yes

What this does:

• updates OS packages
• installs hardening tools (ufw, fail2ban, unattended upgrades)
• installs Docker Engine + Compose plugin
• sets Docker daemon log rotation/live-restore
• opens firewall ports for monitoring stack

If this is the first Docker install, log out and log back in once.

3) Mount SSD

Identify disk/partition:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

Mount it at /srv/ops (example partition /dev/sda1):

./mount_ssd.sh --device=/dev/sda1 --mount-point=/srv/ops --yes

This creates persistent directories:

• /srv/ops/portainer/data
• /srv/ops/grafana/data
• /srv/ops/prometheus/data
• /srv/ops/prometheus/targets
• /srv/ops/uptime-kuma/data
• /srv/ops/backups

4) Configure stack env

cp stack.env.example stack.env

Edit stack.env and set at minimum:

• OPS_ROOT (usually /srv/ops)
• GRAFANA_ADMIN_PASSWORD (strong value)
• any non-default ports if needed

5) Deploy stack

./deploy_stack.sh --yes
./status.sh

Expected endpoints:

• Portainer: https://<pi-ip>:9443
• Grafana: http://<pi-ip>:3000
• Prometheus: http://<pi-ip>:9090
• Uptime Kuma: http://<pi-ip>:3001

6) Add Fedora + Unraid into single Portainer view

Install Portainer Agent on each remote Docker host.

From your admin machine (or from Pi if it can SSH to hosts):

./install_portainer_agent_remote.sh --host=<unraid-ip> --user=<unraid-user> --port=<unraid-ssh-port> --yes
./install_portainer_agent_remote.sh --host=<fedora-ip> --user=<fedora-user> --port=<fedora-ssh-port> --yes

Then in Portainer UI:

1. Environments -> Add environment
2. Choose Docker Standalone
3. Endpoint URL examples:
   • tcp://<unraid-ip>:9001
   • tcp://<fedora-ip>:9001

7) Add Prometheus targets for Fedora/Unraid node metrics

deploy_stack.sh creates /srv/ops/prometheus/targets/external.yml from template.

Edit that file to point to remote node-exporter targets:

- labels:
    job: unraid-node
  targets:
    - 192.168.1.82:9100

- labels:
    job: fedora-node
  targets:
    - 192.168.1.90:9100

Reload Prometheus config:

docker compose --env-file stack.env -f docker-compose.yml exec prometheus \
  wget -qO- --post-data='' http://127.0.0.1:9090/-/reload

8) Day-2 operations

Upgrade stack:

./upgrade_stack.sh --yes

Upgrade and prune old dangling images:

./upgrade_stack.sh --prune --yes

Backup stack:

./backup_stack.sh --retention-days=14 --yes

Restore stack:

./restore_stack.sh --archive=/srv/ops/backups/pi-monitoring-<timestamp>.tar.gz --yes
./deploy_stack.sh --yes

Stop stack:

./teardown_stack.sh --yes

Stop and delete persistent data:

./teardown_stack.sh --remove-data --yes

9) Recommended hardening

• Keep all services on LAN/VPN only; avoid WAN exposure.
• Use unique strong admin passwords for Portainer/Grafana.
• Keep stack.env readable only by admin user (chmod 600 stack.env).
• Back up /srv/ops/backups to another host/NAS.
• Regularly patch OS + container images.

10) Troubleshooting

Docker permission denied:

• re-login after usermod -aG docker <user>

Grafana container restart loop:

• check permissions on ${OPS_ROOT}/grafana/data (UID/GID 472)

Prometheus not scraping remote hosts:

• verify remote exporter reachable: curl http://<host>:9100/metrics
• verify entries in /srv/ops/prometheus/targets/external.yml

Portainer cannot connect to endpoint:

• verify agent running on remote host: docker ps | grep portainer_agent
• check firewall for TCP 9001