Your Data Platform Needs a Nervous System, Not Just a Brain

Your Data Platform Needs a Nervous System, Not Just a Brain

Picture a Formula 1 car with a 1,000bhp engine, carbon fibre everything, and no telemetry. No tyre temperatures, no fuel flow data, no engine diagnostics. The driver gets in, hopes for the best, and finds out something is wrong when the car is on fire at turn three. That, broadly, is how most enterprises run their cloud data platforms. We obsess over the engine. We forget the nervous system.

OpenTelemetry just graduated to full project status at the CNCF. That sounds like a story for platform engineers and SREs, but it is actually a strategic signal for anyone running a data modernisation programme. The era of treating observability as a post-go-live nice-to-have is over, and the arrival of AI agents inside the data stack is the reason why.

The Observability Debt Hidden in Every Cloud Migration

Lift-and-shift programmes almost always under-invest in telemetry. The reasons are predictable. Budgets are framed around compute, storage, and licences. Steering committees ask about workload parity and cutover dates, not about distributed tracing. The instinct is to get the workloads landed and then "wire up monitoring later."

Later rarely arrives. What does arrive is a familiar set of symptoms:

• Pipelines that run green in Databricks but silently drop rows somewhere between ingestion and the gold layer

• Cost spikes that nobody can attribute to a specific job, team, or business process

• SLA breaches that take three days to root-cause analyse because logs are scattered across Spark UI, dbt Cloud, Azure Monitor, and a Slack channel

• A platform that technically works but cannot be confidently operated

Observability debt is the most expensive debt to repay after the fact, because retrofitting instrumentation means revisiting every pipeline, every transformation, and every integration point you swore you would never touch again.

That said, real-world migrations rarely have the luxury of doing everything at once. Budget cycles, delivery timelines, and the sheer complexity of moving a platform mean that sequencing decisions are inevitable. Getting workloads landed and stable is a legitimate first step - the risk is not in deferring observability, but in treating that deferral as permanent. The teams that come out ahead are the ones who plan for observability from day one, even if they build it in phases.

What OpenTelemetry's Graduation Actually Means for Platform Teams

OpenTelemetry (OTel) is not a tool. It is a specification and a set of SDKs that standardise how telemetry data, traces, metrics, and logs, are generated and exported. Its CNCF graduation matters for three practical reasons.

Vendor neutrality is now real. You can instrument once and route telemetry to Datadog, Grafana, Azure Monitor, Splunk, or any combination, without rewriting your code. For platform leaders who have been burned by observability tooling lock-in, this is a structural shift.

Pipelines are standardised. The OTel Collector becomes the single ingestion point for telemetry across your estate. That means consistent semantics for what a "job," a "pipeline," or an "agent run" actually is, regardless of whether it originates in dbt, Spark, Airflow, or a LangChain agent.

It is the connective tissue for governed AI. This is the part most architects are not yet planning for. When your platform starts running agentic workflows, agents that decide which tools to call, which queries to run, which data to retrieve, you lose the deterministic execution model that traditional logging assumes. You cannot grep your way to understanding why an agent made a bad decision. You need traces, spans, and contextual metadata that show the reasoning path. OTel gives you the vocabulary to capture that.

Put plainly: choosing your observability vendor matters less now than ensuring your platform speaks OTel natively.

Designing Observability-First on Modern Data Platforms

If you are migrating from a Teradata-style on-prem warehouse onto Azure Databricks, or modernising an existing lakehouse, here are the patterns that actually hold up in production.

Instrument dbt with run metadata as spans

Treat each dbt model execution as a span. Capture model name, materialisation, rows processed, test outcomes, and upstream lineage. Export through the OTel Collector. You now have a traceable dependency graph of every transformation, queryable in the same place as your infrastructure metrics.

Wrap Spark jobs with structured trace context

Spark's native logging is verbose and unstructured. Add an OTel wrapper that emits job-level and stage-level spans with business context: which pipeline, which tenant, which data domain. This is how you turn "a job failed" into "the customer enrichment pipeline failed at the dedup stage for the EMEA region."

Trace AI agent workflows end to end

For any LLM or agent component, instrument the full reasoning chain. Each tool call, each retrieval, each model invocation should be a span with token counts, latency, and decision context attached. Without this, debugging an agent that produced a wrong answer is guesswork.

Push telemetry to Unity Catalog and your observability backend

Mirror telemetry into governed tables. Yes, your observability platform handles real-time alerting, but having queryable telemetry in the lakehouse itself means analysts and FinOps teams can build their own views of platform health, cost, and usage without a separate licence.

The Operational Readiness Checklist

Before you declare a modern data platform production-ready, particularly one that will run agentic workloads, work through this list honestly:

• OTel instrumentation is in the migration blueprint, not the year-two roadmap

• Every pipeline emits structured spans with business context, not just technical logs

• Trace context propagates across orchestration, transformation, and AI agent layers

• Cost telemetry is unified with operational telemetry, so you can correlate spend with workload behaviour

• Agent decisions are traceable, including prompts, retrievals, and tool calls

• SLOs are defined for data freshness, pipeline success rates, and agent response quality, not just uptime

• Runbooks reference traces, not dashboards alone, because dashboards tell you something is wrong, traces tell you why

If you cannot tick most of these, you do not have a production platform. You have a platform you hope works.

Build the Nervous System Now

The platforms that will earn trust over the next three years are not the ones with the cleverest models or the biggest clusters. They are the ones whose operators can explain, in real time, exactly what happened, why, and what it cost. OpenTelemetry is the standard that makes that possible. Treat it as architecture, not afterthought.

At Volta, we embed alongside data engineering teams during migration and modernisation programmes to make sure decisions like this get made at blueprint stage. That does not always mean delivering full observability on day one - budgets and timelines are real constraints - but it does mean having a clear plan for how you get there, so you are not retrofitting at 2am after a production incident. If you are scoping a Databricks migration or rebuilding a legacy warehouse and observability is still sitting in the "we'll get to it" column, that is usually a good moment to have a conversation.