Server Stats Monitoring Stack

A Docker Compose setup for monitoring server statistics using Prometheus, Node Exporter, and Grafana.

Overview

This stack provides a complete monitoring solution:

• Node Exporter: Collects hardware and OS metrics from the host system
• Prometheus: Time-series database for storing and querying metrics
• Grafana: Visualization and dashboarding platform (optional)

Security Note: By default, no ports are exposed to the host. Services communicate internally via Docker's network. For external access, use the included reverse proxy configuration or uncomment ports for local development.

Prerequisites

• Docker Engine 20.10 or later
• Docker Compose v2.0 or later

Quick Start

1. Initial Configuration

Copy the example environment file and configure your settings:

cp .env.example .env

Edit .env and set your Grafana credentials:

GRAFANA_SERVER=https://your-domain.com
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=your-secure-password

2. Data Directories

The data directories prometheus/data/ and grafana/data/ are already created and ready to use.

If you encounter permission issues when starting services, set appropriate ownership:

sudo chown -R 65534:65534 prometheus/data  # Prometheus runs as nobody:nogroup
sudo chown -R 472:472 grafana/data         # Grafana runs as user 472

3. Start the Services

Basic setup (Prometheus + Node Exporter only):

docker compose up -d

With Grafana:

docker compose --profile grafana up -d

4. Verify Services

By default, service ports are not exposed to the host. Services communicate internally via Docker's network.

To access services from the host, you have two options:

Option A: Expose ports (for local development/testing)

Uncomment the port mappings in docker-compose.yml:

ports:
    - 9090:9090  # Prometheus
    - 9100:9100  # Node Exporter
    - 3000:3000  # Grafana

Then access services at:

• Prometheus: http://localhost:9090
• Node Exporter: http://localhost:9100/metrics
• Grafana: http://localhost:3000

Option B: Use a reverse proxy (recommended for production)

Use docker-compose.override.yml to configure a reverse proxy (e.g., Caddy, Nginx) that handles external access with HTTPS.

Services & Ports

Service	Port	Exposed by Default	Description
Prometheus	9090	No	Metrics database and query interface
Node Exporter	9100	No	System metrics collector
Grafana	3000	No	Visualization dashboard (optional)

Note: Ports are commented out in docker-compose.yml by default for security. Services communicate internally via Docker network names (e.g., prometheus:9090).

Configure Grafana

1. Access Grafana (see "Verify Services" section above for access methods) and login with the credentials from your .env file.

2. Go to Connections > Data Sources.

3. Click Add data source and select Prometheus.

4. Set the URL to http://prometheus:9090 and click Save & Test.

5. Import a pre-built dashboard:

   • Click Dashboards > Import
   • Enter dashboard ID 1860 for "Node Exporter Full"
   • Select your Prometheus data source
   • Click Import

Configuration Files

• docker-compose.yml - Main service definitions
• prometheus/config/prometheus.yml - Prometheus scrape configuration
• .env - Environment variables (create from .env.example)

Persistent Data Storage

The stack uses local directories to persist data across container restarts:

• ./prometheus/data: Stores Prometheus time-series metrics data
• ./grafana/data: Stores Grafana dashboards, users, and settings

These directories are bind-mounted into the containers, making your data easily accessible from the host filesystem. Your data will be preserved even if you stop or recreate the containers.

Important: These directories are excluded from git via .gitignore to prevent committing sensitive data.

Customization

Exposing Services with Reverse Proxy

The included docker-compose.override.yml configures Grafana to work with an external Caddy reverse proxy. This is the recommended approach for production deployments.

Requirements:

• An external Caddy container with Docker proxy support
• The caddy network created: docker network create caddy
• GRAFANA_SERVER variable set in .env to your domain

The override file automatically:

• Connects Grafana to the Caddy network
• Sets up automatic HTTPS via Caddy labels
• Keeps services secure by not exposing ports directly

See docker-compose.example-caddy.override.yml for reference.

Prometheus Configuration

Edit prometheus/config/prometheus.yml to add additional scrape targets or modify the scrape interval:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Useful Commands

# View logs
docker compose logs -f

# Stop services (data persists in local directories)
docker compose down

# Restart a specific service
docker compose restart prometheus

# Check Prometheus config
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

# Check service status
docker compose ps

# Access services from command line (if ports not exposed)
docker compose exec prometheus wget -qO- http://prometheus:9090/-/healthy
docker compose exec prometheus wget -qO- http://node-exporter:9100/metrics

# Backup data (stop services first)
tar czf prometheus-backup-$(date +%Y%m%d).tar.gz prometheus/data
tar czf grafana-backup-$(date +%Y%m%d).tar.gz grafana/data

# Restore data (stop services first)
tar xzf prometheus-backup-YYYYMMDD.tar.gz
tar xzf grafana-backup-YYYYMMDD.tar.gz

# Reset all data (WARNING: deletes all metrics and dashboards)
docker compose down -v && rm -rf ./prometheus/data/* ./grafana/data/*

Troubleshooting

Cannot access services from host:

• By default, ports are not exposed. See "Verify Services" section for access options.
• If you uncommented ports in docker-compose.yml, restart services: docker compose restart

Grafana can't connect to Prometheus:

• Ensure both services are running: docker compose ps
• Always use the internal Docker network URL in Grafana: http://prometheus:9090 (not localhost)
• Check Prometheus logs: docker compose logs prometheus

Node Exporter not collecting metrics:

• Verify the host volume mount is working: docker compose exec node-exporter ls /host
• Check Node Exporter internally: docker compose exec prometheus wget -qO- http://node-exporter:9100/metrics
• If ports are exposed, test from host: curl http://localhost:9100/metrics

Port conflicts (if exposing ports):

• Check if ports are already in use: netstat -tulpn | grep -E '9090|9100|3000'
• Modify port mappings in docker-compose.yml if needed

Permission denied errors in data directories:

• Fix ownership: sudo chown -R 65534:65534 prometheus/data && sudo chown -R 472:472 grafana/data
• Check directory exists: ls -la prometheus/data grafana/data