In a previous article, I explained how to evolve JSON Schemas, based on whether backward, forward, or full (both backward and forward) compatibility is desired. In JSON Schema, the value of additionalProperties determines whether the JSON content model is open (if the value is true), closed (if the value is false), or partially open (if the value is any other value besides true or false). The discussion in that article led to the following points.
- For backward compatibility, one can add optional properties to an open content model or remove optional properties from a closed content model.
- For forward compatibility, one can add optional properties to a closed content model or remove optiona…
In a previous article, I explained how to evolve JSON Schemas, based on whether backward, forward, or full (both backward and forward) compatibility is desired. In JSON Schema, the value of additionalProperties determines whether the JSON content model is open (if the value is true), closed (if the value is false), or partially open (if the value is any other value besides true or false). The discussion in that article led to the following points.
- For backward compatibility, one can add optional properties to an open content model or remove optional properties from a closed content model.
- For forward compatibility, one can add optional properties to a closed content model or remove optional properties from an open content model.
Because backward and forward compatibility each want the content model to be different when adding or removing optional properties, full compatibility can be difficult to achieve with JSON Schema.
Schema formats such as Avro and Protobuf allow for optional fields to be added and removed in a fully compatible manner. These formats achieve full compatibility when adding and removing optional fields by ensuring the following:
- When a payload is produced using a schema, only the fields in the schema will appear in the payload.
- When a payload is consumed using a schema, any fields in the payload that are not in the schema are ignored.
The above behavior is an example of the Robustness Principle. The Robustness Principle, also known as Postel’s Law, is often stated as
Be conservative in what you send, be liberal in what you accept.
An alternative formulation is
Be tolerant of inputs and strict on outputs.
Using the Robustness Principle
Applications that use JSON Schema are not required to follow the Robustness Principle. However, in practice, many applications do.
For example, the Robustness Principle is often used in REST applications. If a REST service does not receive all of the request properties, it will use default values for the missing properties. If a REST client receives response properties it does not expect, it will ignore the extra properties.
Using the Robustness Principle with JSON Schema ensures that the producer will not add properties to the schema other than those specified in the schema. The producer treats the schema as a closed content model. Likewise, the consumer will tolerate properties that do not appear in the schema by simply ignoring them, rather than raising an error. The consumer treats the schema as an open content model.
If you are sure that your application is following the Robustness Principle, then you can ask Schema Registry to use the lenient policy when performing compatibility checks with JSON Schema1. The lenient policy will allow optional properties to be added and removed in a fully compatible manner, because it assumes your application is following the Robustness Principle. If you do not specify the lenient policy, then the strict policy will be used, which is the default.
Limitations of Full Compatibility
The different schema formats can express both product types and sum types. Product types take the form of records (Avro), messages (Protobuf), and objects (JSON). A product type is comprised of components, which take the form of fields (Avro and Protobuf) and properties (JSON). Sum types take the form of unions (Avro) and oneofs (Protobuf and JSON). A sum type is comprised of variants. While product types are more common, sum types are also important. For example, sum types are needed when representing multiple types within the same Kafka topic.
If you are using the lenient policy with JSON Schema, you may still want to prefer backward compatibility over full compatibility. While both backward compatibility and full compatibility allow you to evolve product types by adding and removing optional components, full compatibility does not allow you to evolve sum types. The most common way to evolve a sum type is by adding a new variant. This is a backward compatible change, but not a fully compatible one.
- The lenient policy for JSON Schema compatibility will be available in CP 8.2.0.