Observability beyond three dashboards

Every backend team has a monitoring screen with the same three charts: CPU load, HTTP 500 error count, and p99 latency. When a known failure occurs (e.g., a database connection pool exhausts), these dashboards spike predictably and point you to the fix in minutes.

When a novel, high-severity incident strikes at 2:00am, traditional dashboards fail. You see an aggregate latency bump from 120ms to 2.4s, but none of the pre-aggregated metric graphs can tell you which subset of tenants, payloads, or downstream dependencies is triggering the degradation.

The limitation of pre-aggregated time series

Time-series databases (like Prometheus) aggregate data at ingestion time into fixed counters and gauges. To keep memory bounded, they penalize high cardinality. The moment you attempt to tag metrics with user_id, tenant_id, or order_uuid, the index explodes into millions of series, crashing your collector.

Without high cardinality, you are blind to the long tail: you cannot isolate whether the p99 latency spike affects all users by 2% or completely bricks 0.1% of high-value enterprise accounts whose queries hit an unindexed JSON field.

// Emitting a Canonical Wide Log Event per HTTP Request
type RequestLogEvent struct {
    Timestamp      time.Time `json:"timestamp"`
    TraceID        string    `json:"trace_id"`
    SpanID         string    `json:"span_id"`
    TenantID       string    `json:"tenant_id"`
    UserID         string    `json:"user_id"`
    Route          string    `json:"route"`
    StatusCode     int       `json:"status_code"`
    DurationMs     float64   `json:"duration_ms"`
    DBQueryCount   int       `json:"db_query_count"`
    DBDurationMs   float64   `json:"db_duration_ms"`
    CacheHit       bool      `json:"cache_hit"`
    PayloadBytes   int64     `json:"payload_bytes"`
    ErrorReason    string    `json:"error_reason,omitempty"`
}

Observability is not about having more graphs. It is the ability to interrogate the internal state of a system using only its external outputs, without deploying new debug code during an incident.

The canonical wide event model

Instead of scattering hundreds of disconnected metric increments and unstructured log.Printf() statements across execution paths, modern observability relies on canonical wide events:

• One event per unit of work: Initialize a structured context at the ingress boundary (HTTP request, Kafka event, or cron job).
• Context accumulation: Throughout execution, append metadata (database timings, tenant tiers, feature flag states, downstream response codes) to the context.
• Single atomic emission: Emit the completed wide event (containing 50–100 structured fields) as structured JSON on completion.

Querying columnar event stores

Columnar analytics engines (such as ClickHouse or Honeycomb) ingest millions of unindexed wide events per second. During an outage, you can slice and dice by arbitrary combinations: WHERE duration_ms > 1000 GROUP BY tenant_id, db_query_count. Within seconds, the outlier pattern emerges: 100% of slow requests originate from tenant acme_corp running an unpaginated export.

Transitioning legacy logging pipelines to structured wide event engines? Let's talk telemetry architecture.