ACE Journal

Data Contract Enforcement with Schema Registries

Who this is for: data engineers and platform engineers who own Kafka topics or event schemas and want to move from “we broke consumers last quarter” to “our pipeline blocks breaking changes automatically.” You should be comfortable with Avro or Protobuf schemas and have some exposure to Kafka producers. You do not need to be running Confluent Cloud; the REST API and CLI shown here work against any Confluent Schema Registry-compatible server, including Karapace.

The incident that motivates all of this

It is 9 AM on a Tuesday. The payments team ships a new version of their Kafka producer. The change looks innocent in the pull request: they dropped the currency_code field from the PaymentEvent schema because they normalized it into a separate lookup table. Two Avro consumers on the analytics side deserialized fine in staging because staging used a fresh schema. Twelve downstream consumers in production, however, were running the old Avro schema and used currency_code directly. By 9:15 AM, their jobs had started throwing org.apache.kafka.common.errors.SerializationException. By 9:30 AM, the fraud detection pipeline was producing garbage. The incident took two hours to trace back to the schema change and another three hours to roll back the producer and rehydrate the backfill.

The root cause was not carelessness. The payments team ran their unit tests. The schema change was reviewed by a colleague. What was missing was a machine-checkable enforcement point: something that would have told them, before any code was merged, “this schema change is incompatible with what your consumers currently use, and the registry will reject it.”

That enforcement point is a schema registry operating in BACKWARD compatibility mode, wired into CI.

Mechanics: data contracts, schemas, and registries

“Data contract” is an umbrella term for a machine-verifiable agreement between a data producer and its consumers. At minimum it specifies schema: the names, types, and optionality of every field in the data. More complete contracts also cover semantics (what user_id actually means), SLOs (the freshness guarantee, the null-rate ceiling), and ownership (who is responsible for producer-side compatibility). Schema registries enforce the schema layer; the other layers require complementary tooling.

A schema registry stores versioned schemas identified by a subject: typically the Kafka topic name plus a suffix indicating whether the schema applies to the message key or value (for example, payments.events-value). When a producer registers a new schema version, the registry checks it for compatibility against the existing version before accepting it. If the check fails, the registration is rejected with an HTTP 409 and the producer cannot write data in the new format.

Figure 1 shows the registration and consumption flow.

flowchart LR
    P[producer] -->|register schema| SR[(schema registry)]
    SR -->|409 if incompatible| P
    SR -->|schema_id in message| K[(Kafka topic)]
    P -->|write message| K
    C[consumer] -->|fetch schema by id| SR
    C -->|read message| K

Figure 1. Schema registration and consumption. The producer registers a schema before writing; the registry gates compatibility. Messages carry only a schema ID, not the full schema. Consumers resolve the ID to the schema on first encounter.

The wire format for Avro on Confluent-compatible registries prefixes each message with a magic byte (0x00), a 4-byte big-endian schema ID, and then the Avro binary payload. The consumer uses the schema ID to fetch the writer schema from the registry, and Avro’s schema-evolution rules handle field mismatches between writer and reader schemas. This is why compatibility mode enforcement at registration time is the right leverage point: it keeps the set of writer schemas in circulation within the bounds the readers can handle.

Format choices: Avro, Protobuf, JSON Schema

All three formats are supported by the Confluent Schema Registry and its OSS peers. The compatibility guarantees differ.

Apache Avro is the most common choice for Kafka pipelines. Its schema evolution rules are built into the binary serialization: field defaults enable backward and forward compatibility without coordination. The schema must be registered in the registry; the binary payload is compact and does not carry field names, which makes the registry indispensable.

Protocol Buffers (Protobuf) use field numbers rather than names. Adding a new field with a new number is safe in both directions; reusing a number is catastrophic. The registry enforces Protobuf compatibility rules natively in Confluent Schema Registry v5.5+. Protobuf’s reserved keyword and field-number discipline give producers a second layer of documentation that Avro lacks.

JSON Schema is the weakest choice for streaming. It is human-readable and easy to adopt, but the compatibility semantics are coarser and the payload is verbose. It is appropriate for REST event webhooks and low-volume contract validation; for high-throughput Kafka it carries unnecessary overhead.

For the rest of this piece the examples use Avro, which is where most teams encounter schema registries first.

Compatibility modes

The registry’s compatibility mode is set per-subject (or globally as a default) and determines which schema changes are allowed. Figure 2 maps the four base modes and their transitive variants.

graph TD
    NONE["NONE<br/>no checks"]
    BWD["BACKWARD<br/>new reads old"]
    FWD["FORWARD<br/>old reads new"]
    FULL["FULL<br/>both"]
    BWDT["BACKWARD_TRANSITIVE<br/>new reads ALL old"]
    FWDT["FORWARD_TRANSITIVE<br/>old reads ALL new"]
    FULLT["FULL_TRANSITIVE<br/>all versions compatible"]

    NONE --> BWD
    NONE --> FWD
    BWD --> FULL
    FWD --> FULL
    FULL --> FULLT
    BWD --> BWDT
    FWD --> FWDT
    BWDT --> FULLT
    FWDT --> FULLT

Figure 2. Compatibility mode hierarchy. Non-transitive modes check only against the immediately preceding version; transitive modes check against every registered version. Most production pipelines should use BACKWARD or FULL; NONE is appropriate only for a scratch topic or development subject.

BACKWARD (the default): a consumer using the new schema can read data written with the previous schema. Practically: you can add a field with a default, remove a field with a default. You cannot remove a field that has no default, change a field’s type, or rename a field.

FORWARD: the previous schema can read data written by the new schema. Adding a field without a default is allowed (the old consumer ignores it). Removing a field with a default is allowed.

FULL: both BACKWARD and FORWARD simultaneously. The strictest safe evolution. New fields require defaults; field removals are not allowed without prior deprecation.

Transitive variants (suffix _TRANSITIVE): the check applies to every prior version, not just the most recent. If you registered v1, v2, and v3, and now try to register v4 under BACKWARD_TRANSITIVE, the registry checks v4 against v1, v2, and v3. This is the right choice for pipelines where consumers may lag by multiple versions.

NONE: no compatibility check. Use this only for development or throwaway subjects. A production topic on NONE is a liability.

Worked example: catching a breaking change before it ships

The following walks through a complete enforcement cycle: defining an Avro schema, registering it, proposing a breaking change, and wiring a CI gate to catch the break automatically.

The base schema

The PaymentEvent schema in Avro JSON format:

{
  "type": "record",
  "name": "PaymentEvent",
  "namespace": "com.example.payments",
  "fields": [
    { "name": "payment_id",   "type": "string" },
    { "name": "user_id",      "type": "string" },
    { "name": "amount_cents", "type": "long" },
    { "name": "currency_code","type": "string", "default": "USD" },
    { "name": "status",       "type": "string" }
  ]
}

Register this as version 1 against the subject payments.events-value. The Confluent Schema Registry REST API accepts a POST to /subjects/{subject}/versions with the schema wrapped in a JSON envelope:

curl -s -X POST \
  http://schema-registry:8081/subjects/payments.events-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"PaymentEvent\",\"namespace\":\"com.example.payments\",\"fields\":[{\"name\":\"payment_id\",\"type\":\"string\"},{\"name\":\"user_id\",\"type\":\"string\"},{\"name\":\"amount_cents\",\"type\":\"long\"},{\"name\":\"currency_code\",\"type\":\"string\",\"default\":\"USD\"},{\"name\":\"status\",\"type\":\"string\"}]}"}' \
  | jq .
# Response: {"id": 1}

Set the compatibility mode to BACKWARD for this subject:

curl -s -X PUT \
  http://schema-registry:8081/config/payments.events-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "BACKWARD"}' \
  | jq .
# Response: {"compatibility": "BACKWARD"}

Proposing a breaking change

Now the payments team proposes their Tuesday change: they remove currency_code with no default, because they will normalize it into a lookup table. The new schema omits the field entirely.

# Check compatibility WITHOUT registering (dry-run endpoint)
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/payments.events-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"PaymentEvent\",\"namespace\":\"com.example.payments\",\"fields\":[{\"name\":\"payment_id\",\"type\":\"string\"},{\"name\":\"user_id\",\"type\":\"string\"},{\"name\":\"amount_cents\",\"type\":\"long\"},{\"name\":\"status\",\"type\":\"string\"}]}"}' \
  | jq .

The response from the registry:

{
  "is_compatible": false,
  "messages": [
    "Compatibility check failed. Schema is not backward compatible."
  ]
}

The change is rejected by the registry before any data was written. This is the moment that saves the two-hour incident.

Note the URL path: /compatibility/subjects/{subject}/versions/{version}. This is the test endpoint, distinct from /subjects/{subject}/versions. It performs the compatibility check without writing anything to the registry. This is exactly the call to wire into CI.

The CI gate

Figure 3 shows where the schema compatibility check fits into a standard pull request pipeline.

flowchart LR
    PR[pull request] --> Lint[lint + unit tests]
    Lint --> SchemaCheck[schema compatibility check]
    SchemaCheck -->|compatible| Build[build + publish]
    SchemaCheck -->|incompatible| Fail[pipeline fails]
    Build --> Deploy[deploy to staging]

Figure 3. Schema compatibility gate in CI. The check runs against the registry’s test endpoint after lint but before build, so an incompatible schema never reaches staging or production. The check uses the same REST call as the manual curl above.

A GitHub Actions step for this check:

- name: Check schema compatibility
  env:
    SCHEMA_REGISTRY_URL: $
  run: |
    PAYLOAD=$(jq -Rs '{schema: .}' schemas/payments-event.avsc)
    RESULT=$(curl -s -o /tmp/compat.json -w "%{http_code}" \
      -X POST "${SCHEMA_REGISTRY_URL}/compatibility/subjects/payments.events-value/versions/latest" \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -d "${PAYLOAD}")
    if [ "$RESULT" != "200" ]; then
      echo "Registry returned HTTP $RESULT"; exit 1
    fi
    IS_COMPAT=$(jq -r '.is_compatible' /tmp/compat.json)
    if [ "$IS_COMPAT" != "true" ]; then
      jq '.messages[]' /tmp/compat.json
      exit 1
    fi
    echo "Schema is compatible."

The schema lives in schemas/payments-event.avsc in the producer’s repository. The CI step reads it, POSTs to the test endpoint, and fails the job if the registry says the change is incompatible. This is the full enforcement loop: schema change is proposed, CI checks it against the registry before merge, incompatible changes never reach production.

For teams running the Confluent CLI, the same check is:

confluent schema-registry schema validate \
  --url http://localhost:8081 \
  --schema schemas/payments-event.avsc \
  --type AVRO \
  --subject payments.events-value

The REST path is more portable; the CLI is convenient locally.

The hard part

The worked example above handles the structural case perfectly. The payments team’s currency_code removal is caught before it ships. What the registry does not catch is the semantic case.

Suppose instead the payments team renames currency_code to currency_iso4217 to signal that the field is now always a valid ISO 4217 code, not a free-text legacy value. From the registry’s perspective this is a field removal and a field addition. In BACKWARD mode, if the new field has a default, the check passes: a consumer using the old schema will simply not see currency_iso4217, and it will still read currency_code from messages written by the old producer. The schemas are structurally compatible. But any consumer that explicitly selects currency_iso4217 from messages written by the new producer will now get the value, while consumers reading old messages will not. If the field name itself is what downstream SQL or business logic queries, the rename is a semantic break with no structural signal.

Schema registries cannot detect this class of problem. The tools that can help are human review (a schema change that renames a field should require explicit sign-off from consumer team leads) and contract testing at the consumer level, using tools like Pact, which verifies that a consumer’s actual parsing logic works against the new producer contract in a test harness.

Ownership at the org boundary is the second hard part. A topic schema owned by team A and consumed by teams B, C, and D has no single owner for compatibility policy under a purely federated registry model. Who decides whether a change to PaymentEvent is acceptable? In practice this requires encoding ownership in the contract metadata (a CODEOWNERS-style annotation linking the subject to a review group) and enforcing it through pull request rules: a schema change to payments.events-value must be approved by the consumer team leads, not just by the payments team. Technical tooling cannot substitute for this organizational mechanism; it can only make it easier to route the right approvals.

Schema-on-read systems present a third challenge. Data lakes and Parquet stores often apply schemas at query time rather than at write time. A compatibility mode in a schema registry only protects the Kafka layer. Once data has landed in S3 or GCS in the new Avro encoding, a reader using the old schema may still break if it was not updated. The registry is a gate at write time; landing storage requires its own schema management strategy, typically through table-format tools like Apache Iceberg or Delta Lake that maintain their own schema evolution semantics.

Integration and day-to-day operation

In a working pipeline, the schema CI gate is the active enforcement point. But a few operational habits make the system work at scale.

Evolve by addition, not substitution. The safest schema evolution pattern is: add a new field with a default, write to it in the new producer, wait for all consumers to deploy and read the new field, then deprecate the old field. Never remove a field in the same release that adds its replacement. The multi-step pattern is more operational work but it is the only approach that keeps both BACKWARD and FORWARD compatibility while changing field semantics.

Automate subject creation with default compatibility. Set the global registry default to BACKWARD or FULL, not NONE. New topics that register schemas without an explicit per-subject policy then inherit a safe default. A producer that accidentally registers under NONE in a development cluster cannot repeat the mistake in production if the production cluster’s default is BACKWARD.

Version your schemas alongside your producer code. Schema files (.avsc, .proto) should live in the producer’s source repository under version control, not in a wiki or a shared drive. The CI gate depends on reading the schema from a known path in the repo. Drift between what is in the repo and what is registered in the registry is a sign that the registration step was done manually and the CI gate is not fully wired.

Consumer-side schema pinning. A consumer that specifies a reader schema explicitly (rather than using the writer schema directly) is more resilient to producer evolution. In Avro, the serialization framework handles writer-to-reader schema projection: if the writer schema has a new field the reader does not know about, the reader ignores it. If the reader expects a field the writer removed, Avro uses the reader schema’s default. This projection is exactly what BACKWARD compatibility guarantees at the registry level; pinning the reader schema in the consumer makes the guarantee concrete.

Schema registries are one layer in a broader data quality and contract stack. It is worth being clear about what they do and do not replace.

dbt tests operate on the transformed data in the warehouse, not on the event schema at the source. A dbt not_null or unique test catches data quality violations in the materialized table; a schema registry catches structural incompatibilities in the event schema before any data is written. They address different failure modes and belong in the same pipeline, not as alternatives.

Great Expectations and Soda Core provide data quality assertions: not-null rates, value range checks, referential integrity. These are runtime checks against actual data, typically run on a schedule or as a pipeline step. They complement the registry’s pre-write structural check and can catch semantic violations (an amount_cents field that contains negative values) that no schema can express. The Data Contract Specification (v1.0, late 2024) defines a YAML format that combines schema, quality assertions, and SLO terms in a single document, and tools like the DataContract CLI can generate schema registry subjects and dbt tests from it.

Pact and consumer contract testing invert the testing direction. Rather than the producer asserting that its schema is compatible, a consumer publishes a “pact” describing the minimum fields it needs from the producer. The producer’s CI verifies that its schema satisfies all registered consumer pacts. This catches the semantic rename case that schema registries miss, because the pact tests the consumer’s actual parsing logic, not just field presence. Pact was designed for REST APIs but the contract-testing pattern applies to event schemas; tooling is less mature on the Kafka side but the concept is sound.

Where it breaks

Schema registries solve a specific problem well. Outside that problem space, they fail in predictable ways.

Semantic breaks pass structural checks. The rename scenario above is the canonical example, but the broader class is: any change that preserves field names and types while changing the meaning or contract of the data. A status field that shifts from free-text to a fixed enum is structurally compatible if the enum is represented as a string; semantically it breaks any consumer that was handling values not in the enum.

Multi-topic consistency is unmanaged. A compatibility check is per-subject. If a business event spans two topics (a payment event and a corresponding audit event, both sharing a payment_id foreign key), the registry checks each independently. There is no cross-subject consistency guarantee. If the payments team changes the payment_id type in one topic and not the other, the registry accepts both changes. Cross-topic consistency is an organizational contract, not a registry feature.

High-throughput subjects exhaust registry capacity. A schema registry is a metadata store, not a data store. But in organizations with thousands of topics and aggressive subject proliferation (one subject per partition, per team, per environment), the registry’s metadata surface grows enough to affect query latency and backup size. Subject naming conventions and namespace governance matter at scale.

Registry unavailability blocks producers. When a producer contacts the registry at startup to register or fetch a schema ID, the registry is on the critical path. A registry outage that prevents schema ID resolution will block producer startup. Mitigations include caching schema IDs locally in the producer and running the registry in a high-availability configuration. Confluent’s hosted Schema Registry offloads this operational concern; self-hosted Karapace requires a HA deployment with a replicated backend.

Transitive mode makes migration painful. BACKWARD_TRANSITIVE is the right mode for a pipeline where consumers may lag by multiple versions, but it also means that a schema change must be compatible with every prior registered version, not just the most recent. An old field removal that was structurally safe two versions ago may become blocked because an even older version relied on it. Teams sometimes respond by registering a new subject and migrating consumers, which is correct but operationally costly.

Schema drift from out-of-band registration. If developers can register schemas manually against a production registry (via the API or the CLI, outside CI), the CI gate is byproduct, not enforcement. Access control on the registry’s REST API, write-only via the CI service account, is the organizational control that keeps the gate meaningful.

Further reading

Start with the Confluent Schema Registry REST API reference [1], which documents every endpoint shown here including the compatibility test path. The Avro specification [2] is the authoritative source for schema evolution rules: default values, union types, and the projection logic between writer and reader schemas. The Confluent documentation on compatibility types [3] explains the non-transitive and transitive modes in detail with examples. For Protobuf schema evolution in the registry, see the Confluent Protobuf serializer docs [4]. The Data Contract Specification [5] is the community standard for full-stack contracts (schema + quality + SLOs). Soda Core [6] and the DataContract CLI [7] are the primary OSS tools for runtime enforcement and CLI-driven contract management. Karapace [8], from Aiven, is the most complete open-source Schema Registry implementation for self-hosted deployments. For consumer contract testing as a complement to structural checks, the Pact documentation [9] is the right starting point. The Martin Fowler article on data contracts [10] remains a useful framing of the ownership and organizational dimensions.

Written with AI assistance, editorially reviewed.

References

[1] Confluent, “Schema Registry API Reference.” https://docs.confluent.io/platform/current/schema-registry/develop/api.html

[2] Apache Software Foundation, “Apache Avro Specification.” https://avro.apache.org/docs/current/specification/

[3] Confluent, “Schema Evolution and Compatibility.” https://docs.confluent.io/platform/current/schema-registry/avro.html

[4] Confluent, “Protobuf Serializer and Deserializer.” https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-protobuf.html

[5] Data Contract Specification, “Data Contract Specification v1.0.” https://datacontract.com/

[6] Soda, “Soda Core Overview.” https://docs.soda.io/soda-core/overview-main.html

[7] DataContract CLI, “DataContract CLI.” https://cli.datacontract.com/

[8] Aiven, “Karapace: Open-Source Schema Registry.” https://karapace.io/

[9] Pact Foundation, “Pact Documentation.” https://docs.pact.io/

[10] Fowler, M., “Data Contract.” https://martinfowler.com/articles/data-contracts.html