Theory & Background

Prometheus uses a pull model: it scrapes metrics from targets at configured intervals. Targets expose metrics at /metrics endpoint in Prometheus format (text, line-by-line). Metric types: Counter (monotonically increasing: request count, error count), Gauge (current value: CPU%, memory, queue depth), Histogram (request duration distribution: count observations in buckets), Summary (similar to histogram, calculated client-side). PromQL is the query language: rate(http_requests_total[5m]) = per-second rate over last 5 minutes.

Components & Requirements

10 components required for this project.

#	Component	Purpose	Qty
1	Prometheus	Time-series metrics collection and storage	x1
2	Grafana	Metrics visualization and dashboards	x1
3	Alertmanager	Alert routing and notification	x1
4	Loki	Log aggregation (Prometheus for logs)	x1
5	Tempo	Distributed request tracing	x1
6	Node Exporter	Linux system metrics	x1
7	Blackbox Exporter	Endpoint probing (HTTP/TCP checks)	x1
8	PagerDuty / OpsGenie	On-call notification routing	x1
9	Docker Compose	Stack deployment	x1
10	Custom exporters (Python)	Application-specific metrics	x1

Step-by-Step Implementation

Follow these 6 steps carefully.

1

Prometheus Metrics Architecture

Prometheus uses a pull model: it scrapes metrics from targets at configured intervals. Targets expose metrics at /metrics endpoint in Prometheus format (text, line-by-line). Metric types: Counter (monotonically increasing: request count, error count), Gauge (current value: CPU%, memory, queue depth), Histogram (request duration distribution: count observations in buckets), Summary (similar to histogram, calculated client-side). PromQL is the query language: rate(http_requests_total[5m]) = per-second rate over last 5 minutes.

2

Node Exporter and Service Metrics

Node Exporter exposes 1000+ Linux system metrics: CPU (per-core utilization), memory (total, available, cached), disk (I/O, usage per filesystem), network (bytes in/out, packet errors), file descriptors, systemd service states, hardware temperatures (if lm-sensors installed). Application metrics: instrument with client libraries (prometheus_client for Python, prom-client for Node.js). Custom metrics: expose business KPIs (orders per minute, active users) as gauges/counters.

3

Grafana Dashboard Design

Design dashboards with SRE principles: RED method for services (Rate, Errors, Duration). USE method for resources (Utilization, Saturation, Errors). Key panels: request rate (rate[5m] graph), error rate (above 1% = red alert), 50th/95th/99th percentile latency (histogram_quantile), resource saturation (CPU>80% warning), top 10 resource consumers by label. Use template variables for service/host selection — one dashboard serves all services. Dark mode, 30s refresh.

4

Alerting Rules and Routing

Prometheus alerting rules (prometheus/rules.yml): alert: HighErrorRate, expr: sum(rate(http_requests_total{status=~'5..'} [5m])) / sum(rate(http_requests_total[5m])) > 0.05, for: 2m (must persist 2 minutes before firing), labels: severity=critical, annotations: summary/description. Alertmanager routes: critical → PagerDuty (wake someone up), warning → Slack #alerts, info → email digest. Silencing: mute known maintenance windows. Alert grouping: batch related alerts to prevent notification storms.

5

Log Aggregation with Loki

Loki stores logs indexed only by label (not full text), making it cost-efficient. Promtail agents on each server stream logs to Loki. Label schema: {job='nginx', host='web1', env='prod'}. Grafana Explore: query logs: {job='nginx'} |= 'error' | rate[5m]. Correlate logs with metrics: click on a Grafana graph spike, drill down to relevant log lines at that exact timestamp. LogQL alerting: alert when error pattern appears > N times in 5 minutes.

6

Distributed Tracing with Tempo

Instrument applications with OpenTelemetry SDK: create spans for each operation (HTTP request, database query, external API call). Each request gets a trace ID propagated through all microservices via HTTP headers. Spans include: service name, operation, duration, success/failure, custom attributes. Grafana Tempo stores traces, Grafana visualizes flame charts. Correlate: from a slow request in metrics dashboard → view the specific trace → see which service/query caused latency.

Code & Implementation

Core code for prometheus_rules.yml:

prometheus_rules.yml
  YAML

groups: - name: catb-alerts   interval: 30s   rules:   - alert: HighErrorRate     expr: |       sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)       / sum(rate(http_requests_total[5m])) by (service) > 0.05     for: 2m     labels:       severity: critical     annotations:       summary: "High error rate on {{ $labels.service }}"       description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"       runbook: "https://catb.in/runbooks/high-error-rate"    - alert: HighMemoryUsage     expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.90     for: 5m     labels:       severity: warning     annotations:       summary: "Memory usage above 90% on {{ $labels.instance }}"    - alert: DiskSpaceCritical     expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.10     for: 1m     labels:       severity: critical     annotations:       summary: "Less than 10% disk space on {{ $labels.instance }}"    - alert: ServiceDown     expr: up{job="catb-web"} == 0     for: 30s     labels:       severity: critical     annotations:       summary: "{{ $labels.instance }} is down"

Testing & Troubleshooting

Test Infrastructure Monitoring and Alerting by verifying each subsystem individually before full integration.

!

Troubleshooting Tips

Verify power voltages, check ground connections, use serial monitor for debug.

Real-World Applications

*Production web service SRE monitoring

*Database performance tracking

*Kubernetes cluster health observability

*IoT sensor data monitoring dashboard

*Business KPI real-time tracking

*Security event monitoring SIEM integration

*Cloud cost tracking and anomaly detection

*E-commerce conversion and revenue monitoring

Extensions & Next Steps

Implement AIOps anomaly detection for automatic threshold setting
Add synthetic monitoring with browser automation tests
Build chaos engineering integration to test alert coverage
Implement on-call rotation automation with escalation policies
Add capacity planning dashboards for resource procurement

Interactive Playground

Coming Soon

An interactive simulator will be available here — simulate circuits and run code in-browser without hardware.

Frequently Asked Questions

What is the difference between monitoring, observability, and APM?

Monitoring: watching pre-defined metrics and alerting on thresholds — tells you something is wrong. Observability: capability to understand system internal state from external outputs (metrics, logs, traces — the 'three pillars'). Allows asking arbitrary questions about system behavior without pre-instrumenting for those specific questions. APM (Application Performance Monitoring): monitoring focused on application-level performance metrics — request rates, latencies, error rates, database query performance. APM tools (Datadog, New Relic) provide distributed tracing and code-level insights.

How do I set good alert thresholds without too many false positives?

Alert fatigue from false positives causes engineers to ignore alerts — defeating the purpose. Best practices: use SLOs (Service Level Objectives) as alert thresholds, not arbitrary values. Set error budget burn rate alerts (burn through weekly error budget in 1 hour → page immediately). Use 'for' clause: require condition to persist 2–5 minutes before firing (eliminates transient spikes). Alert on symptoms (user-facing: high latency, error rate) not causes (internal: CPU, memory). Review alert history monthly: muted/acked frequently = wrong threshold or wrong alert.

Introduction