Monitoring and Observability

Monitoring vs Observability

Monitoring tells you when something is wrong. CPU is at 95%. Error rate spiked. Disk is full. It answers "is the system healthy?"

Observability tells you why it is wrong. It combines metrics, logs, and traces so you can drill down from "error rate increased" to "this specific database query is timing out for users in this region." You need both.

Prometheus - Metrics Collection

Prometheus scrapes metrics from your applications at regular intervals and stores them as time series data. Your app exposes a /metrics endpoint, and Prometheus pulls from it.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'node-app'
    static_configs:
      - targets: ['app:3000']
    metrics_path: /metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Instrumenting Your App

Add metrics to your application with the Prometheus client library. Track request counts, response times, error rates, and any business metrics that matter.

const client = require('prom-client');

// Collect default metrics (CPU, memory, event loop)
client.collectDefaultMetrics();

// Custom counter - track total requests
const httpRequests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'path', 'status'],
});

// Custom histogram - track response times
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'path'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
});

// Middleware to track every request
app.use((req, res, next) => {
  const end = httpDuration.startTimer({
    method: req.method,
    path: req.route?.path || req.path,
  });
  res.on('finish', () => {
    httpRequests.inc({
      method: req.method,
      path: req.route?.path || req.path,
      status: res.statusCode,
    });
    end();
  });
  next();
});

// Expose metrics endpoint for Prometheus
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

Alerting Rules

Alerts fire when conditions are met for a specified duration. The "for" clause prevents flapping - the condition must persist for that long before the alert fires. Route critical alerts to PagerDuty, warnings to Slack.

groups:
  - name: app-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High 5xx error rate"
          description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes"

      - alert: SlowResponses
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "95th percentile response time above 2 seconds"

Grafana - Visualisation

Grafana connects to Prometheus and lets you build dashboards. Start with four panels: request rate, error rate, response time (p50, p95, p99), and active connections. These are the "four golden signals" of monitoring. Every team should have this dashboard visible at all times.