Logging Observability

Logging & Observability

Patterns for building observable systems across the three pillars: logs, metrics, and traces.

Three Pillars

Each pillar is strongest when correlated. Embed trace_id in every log line to jump from a log entry to the full distributed trace.

---

Structured Logging

Always emit logs as structured JSON — never free-text strings.

Required Fields

Context Enrichment

Attach context at the middleware level so downstream logs inherit automatically:

app.use((req, res, next) => {
  const ctx = {
    trace_id: req.headers['x-trace-id'] || crypto.randomUUID(),
    request_id: crypto.randomUUID(),
    user_id: req.user?.id,
    method: req.method,
    path: req.path,
  };
  asyncLocalStorage.run(ctx, () => next());
});

Library Recommendations

Choose a logger that outputs structured JSON natively. Avoid loggers requiring post-processing.

---

Log Levels

Rules: Production default = INFO and above. If you log an ERROR, someone should act on it. Every FATAL should trigger an alert.

---

Distributed Tracing

OpenTelemetry Setup

Always prefer OpenTelemetry over vendor-specific SDKs:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'order-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Span Creation

const tracer = trace.getTracer('order-service');

async function processOrder(order: Order) {
  return tracer.startActiveSpan('processOrder', async (span) => {
    try {
      span.setAttribute('order.id', order.id);
      span.setAttribute('order.total_cents', order.totalCents);
      await validateInventory(order);
      await chargePayment(order);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

Context Propagation

• Use W3C Trace Context (traceparent header) — default in OTel
• Propagate across HTTP, gRPC, and message queues
• For async workers: serialise traceparent into the job payload

Trace Sampling

Always sample 100% of error traces regardless of strategy.

---

Metrics Collection

RED Method (Request-Driven)

Monitor these three for every service endpoint:

USE Method (Resource-Driven)

For infrastructure components (CPU, memory, disk, network):

---

Monitoring Stack

Recommendation: Start with OTel Collector → Prometheus + Grafana + Loki + Jaeger. Migrate to SaaS only when operational overhead justifies cost.

---

Alert Design

Severity Levels

Alert Fatigue Prevention

• Alert on symptoms, not causes — "error rate > 5%" not "pod restarted"
• Multi-window, multi-burn-rate — catch both sudden spikes and slow burns
• Require runbook links — every alert must link to diagnosis and remediation
• Review monthly — delete or tune alerts that never fire or always fire
• Group related alerts — use inhibition rules to suppress child alerts
• Set appropriate thresholds — if alert fires daily and is ignored, raise threshold or delete

---

Dashboard Patterns

Overview Dashboard ("War Room")

• Total requests/sec across all services
• Global error rate (%) with trendline
• p50 / p95 / p99 latency
• Active alerts count by severity
• Deployment markers overlaid on graphs

Service Dashboard (Per-Service)

• RED metrics for each endpoint
• Dependency health (upstream/downstream success rates)
• Resource utilisation (CPU, memory, connections)
• Top errors table with count and last seen

---

Observability Checklist

Every service must have:

• Structured JSON logging with consistent schema
• Correlation / trace IDs propagated on all requests
• RED metrics exposed for every external endpoint
• Health check endpoints (/healthz and /readyz)
• Distributed tracing with OpenTelemetry
• Dashboards for RED metrics and resource utilisation
• Alerts for error rate, latency, and saturation with runbook links
• Log level configurable at runtime without redeployment
• PII scrubbing verified and tested
• Retention policies defined for logs, metrics, and traces

Anti-Patterns

NEVER Do

1. NEVER log passwords, tokens, API keys, or secrets — even at DEBUG level
2. NEVER use console.log / print in production — use a structured logger
3. NEVER use user IDs, emails, or request IDs as metric labels — cardinality will explode
4. NEVER create alerts without a runbook link — unactionable alerts erode trust
5. NEVER rely on logs alone — you need metrics and traces for full observability
6. NEVER log request/response bodies by default — opt-in only, with PII redaction
7. NEVER ignore log volume — set budgets and alert when a service exceeds daily quota
8. NEVER skip context propagation in async flows — broken traces are worse than no traces