Schema Evolution & Data Contracts

Schema Evolution & Data Contracts

An upstream team renames a column from user_id to userId. It's a one-line change, totally reasonable from their side. No deployment fails. No alerts fire. But three downstream Spark jobs are now silently joining on a field that returns null for every row, and the revenue dashboard has been understating numbers for four days before anyone notices. That's not a hypothetical. That's Tuesday at a mid-size data company.

The underlying problem is that every data pipeline is built on assumptions about shape: this field exists, it's a string, it means this thing. Those assumptions live in your code, your SQL, your Spark schemas, sometimes just in someone's head. When the upstream data changes, those assumptions don't automatically update with it.

Schema evolution is the reality that schemas change constantly as products grow. Data contracts are the mechanism teams use to manage that change without destroying each other's pipelines. The two are inseparable: you can't build reliable contracts without understanding how schemas evolve, and you can't evolve schemas safely without contracts to define what "safe" even means.

At companies like Airbnb, Uber, and Spotify, schema breakage is consistently one of the top sources of data quality incidents. Interviewers at these companies don't just want to hear "add a nullable column." They want to know you understand compatibility modes, enforcement mechanisms, and what happens when a producer team doesn't play by the rules.

How It Works

Every data pipeline is built on an implicit promise: the data arriving at your consumer looks the same as it did yesterday. When that promise breaks, things go wrong quietly.

Here's the basic flow. A producer service, say a checkout microservice, serializes an event into Avro or Parquet and writes it to a Kafka topic or S3 prefix. Before that write happens, it registers the schema definition with a schema registry. The registry stores that definition, assigns it a version ID, and embeds that ID in the message. On the other side, a consumer pipeline reads the message, looks up the schema by ID, and deserializes the bytes back into structured data. Every field, every type, every name is reconstructed from that registered definition.

Think of the schema registry like a shared dictionary. The producer and consumer don't have to speak the same version of the language at the same time, as long as they both have access to the dictionary and agree on which edition to use.

Here's what that flow looks like:

Compatibility Modes

This is where most candidates stumble, so get this locked in before your interview.

When a producer evolves a schema, say by adding a new field called device_type to an Avro event, the registry checks whether that change is safe. "Safe" depends on which compatibility mode you've configured, and there are three you need to know cold.

Backward compatibility means the new schema can read data written with the old schema. This is the most common mode. You add device_type with a default of null, and old messages that don't have that field can still be deserialized correctly because the reader fills in the default. This matters because in an interview, when they ask about rolling upgrades, backward compatibility is why you can deploy a new consumer before the producer has started emitting the new field.

Forward compatibility flips the direction: the old schema can read data written with the new schema. Old consumers encountering a message with device_type just ignore the unknown field. This is what you need when producers upgrade before consumers do.

Full compatibility requires both directions to work simultaneously. It's the strictest mode and the safest for teams that can't coordinate deployments tightly. The trade-off is that it limits what changes you can make, since anything that breaks either direction is rejected.

Common mistake: Candidates mix up the direction. Backward is about the reader being new and the data being old. Forward is the opposite. If you can't remember which is which under pressure, just say "the new reader can handle old data" rather than using the term and getting it backwards.

Where the Registry Lives

The specific tool doesn't change the pattern. Confluent Schema Registry handles this for Kafka-based pipelines, enforcing compatibility at publish time so a bad schema never reaches consumers. For S3 and Athena, AWS Glue Data Catalog plays the same role, storing table definitions that Athena uses to parse Parquet files. Iceberg and Delta Lake go a step further by tracking schema history directly in the table metadata, so you can see exactly when a column was added and query old snapshots with the schema that was valid at that point in time.

The interviewer probably doesn't care which tool you name. They care that you understand the registry is the enforcement point, not just a documentation store.

Data Contracts: Beyond the Schema

A schema tells you the shape of the data. A data contract tells you what the data means, who owns it, and what guarantees come with it.

A contract between a producer team and a consumer team specifies four things: the schema itself, the semantics of each field (what does revenue actually mean? gross? net? before refunds?), SLAs around freshness and completeness (this table is updated by 9am UTC, with no more than 0.1% null IDs), and ownership (who you page when something breaks). Without that last layer, you can have a perfectly valid schema and still have a pipeline producing wrong numbers because the definition of a field quietly changed.

The Enforcement Gap

Here's the uncomfortable reality of most data platforms: none of this is enforced by default. A producer team renames a column, sends a Slack message to a channel that three of the five affected teams don't monitor, and ships the change on Friday. By Monday, dashboards are wrong. Two of the pipelines errored out and got noticed. The third silently coerced nulls into zeros and nobody caught it for two weeks.

That gap between "we have a schema registry" and "schema changes can't break consumers without warning" is exactly what data contracts are designed to close. The registry handles structural compatibility. The contract handles everything else.

Your 30-second explanation: "A producer registers its schema with a registry, which enforces compatibility rules before any change ships. Consumers deserialize data using the registered schema version, so both sides stay in sync even as the schema evolves. But structural compatibility alone isn't enough. A data contract layers on top to define what fields actually mean, what freshness guarantees the producer commits to, and who owns the data. Together, they prevent the silent failures that happen when teams change data without coordinating with the people who depend on it."

Patterns You Need to Know

In an interview, you'll usually need to pick a specific approach. Here are the ones worth knowing.

---

Backward-Compatible Evolution

This is your starting point for almost every schema change. The idea is simple: you only make changes that new readers can handle while still reading old data. Adding a nullable field, adding a column with a default value, adding a new enum value at the end. These are all backward-compatible because a consumer using the new schema can still make sense of records written before the change existed.

In Avro, this works because fields with default values are optional during deserialization. If the field isn't present in an old record, the reader fills it in with the default. Iceberg's ADD COLUMN works the same way at the table level: existing data files don't change, and the new column reads as null for historical partitions. The key discipline is that you never rename, remove, or change the type of an existing field under this pattern. Those are breaking changes, and they belong in a different category.

When to reach for this: any time the interviewer asks how you'd evolve a schema in a live system with active consumers. This should be your default answer before you escalate to anything more complex.

---

Breaking Change Management with Versioned Topics

Sometimes a backward-compatible change isn't possible. A product team renames user_id to userId. A field type needs to change from STRING to TIMESTAMP. A field gets removed entirely. You can't paper over these with a default value. The answer is versioning.

You create a new topic or table (say, events_v2) with the new schema, and you dual-write: the producer sends every event to both events_v1 and events_v2 during a migration window. Consumers migrate to v2 on their own schedule, with a hard deprecation deadline for v1. This keeps the old consumers running without modification while new consumers adopt the updated schema. Once all consumers have migrated and the deadline passes, you stop writing to v1 and eventually delete it. The migration window is the part candidates forget to mention. Without it, you're forcing a coordinated cutover across every downstream team simultaneously, which almost never works cleanly.

When to reach for this: when the interviewer gives you a scenario involving a field rename, type change, or removal, and asks how you'd roll it out without breaking downstream pipelines.

---

Schema-on-Read with Drift Detection

In a data lake environment, you often can't enforce schema at write time. Raw JSON lands in S3 from dozens of upstream services, and those services don't coordinate with you before they ship. Schema-on-read accepts this reality: you store whatever arrives, and you apply the schema interpretation at query time.

The risk is obvious. If a producer quietly changes a field name, your downstream jobs start returning nulls and nobody notices until a dashboard looks wrong. The fix is a validation layer between raw storage and your curated layer. Tools like Great Expectations, Soda, or custom PySpark checks run on ingestion and compare the arriving data against an expected schema. Records or files that fail get routed to a quarantine zone and trigger an alert. Only data that passes validation gets promoted to the curated layer that downstream consumers actually query. This pattern doesn't prevent schema drift, but it catches it fast and stops it from propagating silently.

When to reach for this: data lake architectures, multi-producer environments, or any scenario where the interviewer asks how you'd handle schema validation when you don't control the producer.

---

Data Contracts as Code

A schema registry tells you whether a schema is structurally compatible. It doesn't tell you what a field means, who owns the data, what the freshness SLA is, or what happens when a producer wants to make a breaking change. Data contracts fill that gap.

The pattern is to define the contract in a YAML or JSON file, commit it to Git alongside the producer's code, and enforce it in CI. When a producer opens a PR that changes the schema, the CI pipeline checks whether the change violates any downstream contract. If it does, the PR is blocked until the producer either makes the change backward-compatible or coordinates with affected consumers. Tools like the Datacontract.com CLI implement this directly. Internal platforms at Uber and Airbnb built similar systems in-house. The contract file typically specifies the schema, field-level semantics (what "revenue" actually means), SLA expectations, and the owner who's accountable for breaking changes. Storing it in Git gives you a full audit trail and makes contract reviews part of the normal code review process.

When to reach for this: when the interviewer asks about preventing schema breakage at the organizational level, or when they push on "what if the producer team doesn't follow the rules?"

---

Consumer-Driven Contract Testing

Most contract enforcement is producer-centric: the producer defines the schema, and consumers adapt. Consumer-driven contract testing flips that. Each consumer publishes a formal declaration of what it needs from the producer: specific fields, specific types, specific nullability constraints. The producer then runs all of those consumer expectations as tests before shipping any schema change.

This pattern comes from the API world, where Pact is the standard tool. In data pipelines, the same idea applies. A consumer team says "I need event_type to be a non-null STRING and user_id to be a non-null BIGINT." That expectation lives in a contract broker. Before the producer deploys a schema change, its test suite fetches all registered consumer expectations and validates the new schema against each one. If any consumer's expectations would break, the deployment is blocked. This catches breakage before it ever reaches production, and it shifts the conversation from "why did my pipeline break" to "we knew this would break Consumer B, so we coordinated first."

When to reach for this: when the interviewer asks how you'd give consumers more control over schema stability, or in scenarios involving many downstream consumers with different requirements.

---

Comparing the Patterns

For most interview problems, you'll default to backward-compatible evolution and mention the schema registry as your enforcement mechanism. Reach for versioned topics when the change is genuinely breaking and you need to walk through a migration strategy. If the interviewer pushes on organizational enforcement or "what happens when teams don't coordinate," that's your cue to bring in contracts as code or consumer-driven testing.

What Trips People Up

The Mistake: Getting Compatibility Direction Backwards

The wrong answer sounds like: "Backward compatibility means the old schema can still read new data." That's actually forward compatibility. Candidates mix these up constantly, and the moment they do, the interviewer knows they've memorized the terms without understanding the mechanics.

Backward compatibility means the new schema can read old data. That's the case you care about during a rolling upgrade, when your new consumer is deployed but old messages are still sitting in the Kafka topic. Forward compatibility is the reverse: the old schema can read data written with the new schema. Full compatibility is both directions simultaneously.

The reason this matters in practice: if you configure your schema registry with backward compatibility and then try to add a required field without a default (which breaks backward compatibility, because the new schema can't deserialize old messages that are missing that field), the registry will reject the change. Removing a field is actually the forward compatibility concern: old consumers expecting that field won't be able to handle new messages where it's gone. If you've got the directions mixed up in your head, you'll be confused about which operations the registry blocks and why.

Interview tip: Anchor the definition to a concrete scenario. Say: "Backward compatibility protects consumers during rolling upgrades. If I deploy a new consumer before all old messages are consumed, it needs to deserialize data written with the old schema. That's the backward case. Adding a required field without a default breaks that, because old messages don't have it."

The Mistake: Assuming Nullable Columns Are Always Safe

"I'd just add a nullable column with a default of null. That's non-breaking." You'll hear this constantly. It's mostly true, and mostly is what gets you into trouble.

The failure modes are specific. A Spark job that does SELECT * into a fixed struct type will break if the column count changes. Schema inference in Spark is particularly nasty here: if a job cached the inferred schema from an earlier run, it won't pick up the new nullable column at all, and you'll get silent nulls or deserialization errors depending on the format. Parquet handles this better than JSON; Avro requires the default to be specified in the schema definition itself or the compatibility check fails.

The better answer acknowledges the nuance: adding a nullable field with a default is the safest path, but you still need to verify that downstream consumers aren't doing positional reads, that schema inference is disabled or pinned, and that the Avro schema explicitly declares the default value.

Common mistake: Candidates say "nullable is safe." The interviewer hears "I've never debugged a Spark schema inference issue at 2am."

The Mistake: Treating Schema Contracts as Purely Structural

This one separates candidates who've worked on real data platforms from those who haven't. A field called revenue that changes from gross to net revenue is a schema contract violation. The type is still DECIMAL. The column name is still revenue. The schema registry will not catch it. Your downstream ML model will silently train on the wrong numbers.

Semantic drift is the category of breakage that structural tooling is blind to. When candidates describe data contracts as "just schema versioning with a registry," they're missing the organizational half of the problem. A real data contract specifies what a field means, who owns it, what its valid range is, and what downstream teams are allowed to depend on. The schema is just one clause in that agreement.

When an interviewer at a company like Airbnb or Spotify asks about data contracts, they want to hear you distinguish between structural compatibility (what a registry enforces) and semantic correctness (what documentation, ownership, and data quality checks enforce). Bring up this distinction proactively. It signals you've thought about data quality as a sociotechnical problem, not just a tooling problem.

The Mistake: Forgetting That Historical Data Has the Old Shape

Candidates will walk through a clean schema migration, explain dual-writing, versioned topics, consumer cutover, and then stop. The interviewer asks: "What about your historical data in the data lake?" Silence.

When you evolve a schema, three years of Parquet files in S3 still have the old column names, old types, old structure. If a downstream job needs to backfill a new metric using the device_type field you just added, that field doesn't exist in any partition before the migration date. That's not a hypothetical; it's one of the most common sources of incorrect backfill results.

The answer has three options: reprocess historical partitions with a transformation that synthesizes the new field (expensive but clean), use Iceberg or Delta Lake's built-in schema evolution so the table layer handles the mismatch transparently, or maintain a compatibility shim in your pipeline that maps old field names to new ones based on the partition date. Mention at least one of these. Candidates who skip the backfill question are implicitly saying they only think about schemas in the context of live data, and that's not how data engineering works.

Interview tip: After explaining your migration strategy, explicitly say: "And for historical data, here's how I'd handle the backfill..." Interviewers notice when you close that loop without being prompted.

How to Talk About This in Your Interview

When to Bring It Up

Schema evolution and data contracts aren't always the explicit topic, but they're almost always relevant. Train yourself to hear the signals.

When an interviewer says "upstream teams keep breaking our pipelines" or "we're seeing data quality issues we can't explain," that's your opening. Same with "how do you ensure reliability across teams" or "how do you handle changes to your data sources."

If the question is about pipeline reliability, data quality, or cross-team data sharing, pivot here. Say something like: "One of the biggest sources of silent failures I've seen is schema drift. The way I'd address that is by enforcing contracts between producers and consumers..." Then let the conversation follow.

Even in a question about Kafka or Spark architecture, if you're discussing how data flows between systems, mentioning schema registries and compatibility modes signals that you think about the operational reality, not just the happy path.

Sample Dialogue

Interviewer: "Say a producer team wants to change a field type from STRING to INT. How do you handle that?"

You: "First thing I'd do is treat that as a breaking change, full stop. STRING to INT isn't backward-compatible because any old data with non-numeric strings will fail to deserialize. So before anything else, I'd audit downstream consumers to understand who's reading that field and how."

Interviewer: "Okay, but the producer team says they need to ship this in two weeks."

You: "Two weeks is workable if we're disciplined about it. I'd have the producer dual-write to a versioned topic, say events_v2, with the new schema, while keeping events_v1 alive. Consumers migrate on their own timeline, but we set a hard deprecation deadline for v1, maybe 30 days out. That way the producer isn't blocked and consumers aren't blindsided."

Interviewer: "What if a consumer just... doesn't migrate in time?"

You: "That's where the contract matters. If we have a data contract with SLAs and ownership defined, the deprecation deadline isn't a suggestion, it's a commitment both sides signed up for. In practice, you'd also have CI gates that alert consumer teams when a breaking change is registered against a topic they depend on. The goal is no surprises, not no changes."

Interviewer: "What about the historical data in the lake? The old Parquet files still have STRING."

You: "Right, and this is where candidates often stop short. You've got a few options: reprocess the historical partitions with a migration job, use Iceberg's schema evolution to handle the type coercion transparently, or maintain a compatibility shim in the read layer. Which one you pick depends on data volume and how far back consumers need to query. I'd usually lean on Iceberg if it's already in the stack."

Follow-Up Questions to Expect

"How do you handle semantic drift, like a field that changes meaning without changing type?" Explain that structural compatibility checks won't catch this; it requires field-level documentation in the contract itself, plus data quality tests that validate business logic, not just types.

"What's the difference between schema-on-read and schema-on-write, and when would you choose each?" Schema-on-write (Avro with a registry, Iceberg tables) catches problems at ingestion; schema-on-read (raw JSON in S3) is more flexible but pushes validation downstream, which means you need a strong drift-detection layer to compensate.

"What if the producer team refuses to follow the contract?" This is partly an organizational problem. The technical answer is automated CI gates that block non-compliant schema publishes; the organizational answer is that contracts need executive sponsorship and clear ownership, otherwise they're just suggestions.

"How would you backfill after a schema change?" Walk through reprocessing historical partitions with the new schema, using Iceberg's built-in schema evolution for non-breaking changes, and validating the backfill output against the same data quality checks as the live pipeline.

What Separates Good from Great

• A mid-level answer covers the technical mechanics: versioned topics, nullable fields, schema registries. A senior answer connects those mechanics to organizational failure modes, specifically what happens when there's no contract and the producer team doesn't know who their consumers are.
• Mid-level candidates stop at structural compatibility. Senior candidates bring up semantic drift unprompted, because they've seen a "revenue" field quietly change from gross to net and watched a finance dashboard go wrong for two weeks before anyone noticed.
• The best answers close with business impact. Schema breakage isn't an infrastructure inconvenience; it's wrong numbers in executive dashboards, corrupted ML training data, and financial reports that have to be restated. Framing it that way shows you understand why data contracts exist, not just how they work.

Key takeaway: Schema evolution is inevitable; what separates reliable data platforms from fragile ones is whether you've made the rules of change explicit, enforced them automatically, and given every consumer enough warning to adapt.