Practical observability: logs, metrics, traces, and SLOs for production

Observability is more than monitoring#

Monitoring usually starts with dashboards and alerts: CPU is high, requests are failing, or the database is slow. Observability helps engineers ask new questions about production without deploying new debug code. During an incident, you need to know which requests failed, which users were affected, which dependency is slow, and which release changed behavior.

The three core signals are logs, metrics, and traces. Logs capture detailed events, metrics show trends over time, and traces connect the steps of a request across services. Missing one signal often turns incident response into guessing.

Structured logs and correlation IDs#

Free-form text logs are hard to query at scale. For production APIs, prefer structured JSON logs with request IDs or correlation IDs that flow from the proxy into application logs, background jobs, and error reports.

{
  "level": "error",
  "request_id": "req_7f2a",
  "route": "POST /api/orders",
  "user_id": "u_123",
  "duration_ms": 842,
  "error_code": "payment_timeout"
}

Do not log passwords, tokens, secrets, card data, or unnecessary personal data. Useful logs provide debugging context without becoming a sensitive data warehouse.

Metrics should reflect user experience#

Infrastructure metrics matter, but important alerts should focus on user-facing symptoms: rising error rate, p95 latency above threshold, failed checkout, queue backlog, or unexpected 5xx responses. For web APIs, start with traffic, errors, latency, and saturation.

• Traffic: request rate, active users, and queue throughput.
• Errors: 4xx by type, 5xx, exception rate, and failed jobs.
• Latency: p50, p95, and p99 by route or dependency.
• Saturation: CPU, memory, database connections, queue depth, and disk.

Tracing shows where time went#

Distributed traces are valuable when a request crosses frontend, API gateway, backend, cache, database, and third-party services. Instead of knowing only that an endpoint took two seconds, a trace shows whether the time came from a database query, payment API, or rendering path.

You do not need to trace every request. Use sampling, increase sampling during incidents, and make sure trace IDs also appear in logs.

SLOs make alerts disciplined#

SLOs define target service behavior from the user's perspective, such as 99.9% of article reads completing under 800ms over 30 days. With an error budget, teams can decide when to prioritize reliability over feature work.

Production checklist#

• Logs include request id, route, status code, and duration.
• Public APIs expose request rate, error rate, and p95/p99 latency metrics.
• Background jobs expose queue depth, retry count, and dead-letter count.
• Database monitoring includes slow queries, connections, and backup status.
• Alerts map to SLOs or clear user impact.

Primary references#

• OpenTelemetry Docs: Signals
• Google SRE Book: Service Level Objectives
• Prometheus Docs: Alerting