Skip to main content
Version: v1.9.0

EventMesh Schema Registry (OpenSchema)

Overview of Schema and Schema Registry

Schema

A Schema stands for the description of serialization instances(string/stream/file/...) and has two properties. First, it is also in the format of serialization type. Second, it defines what requirements such serialized instances should satisfy.

Besides describing a serialization instance, a Schema may also be used for validating whether an instance is legitimate. The reason is that it defines the type(and other properties) of a serialized instance and inside keys. Taking JSON Schema for example, it could not only be referred when describing a JSON string, but also be used for validating whether a string satisfies properties defined in the schema[1].

Commonly, there are JSON Schema, Protobuf Schema, and Avro Schema.

Schema Registry

Schema Registry is a server provides RESTful interfaces. It could receive and store Schemas from clients, as well as provide intrefaces for other clients to retrieve Schemas from it.

It could be applied to validation process and (de-)serialization process.

Comparison of Schema Registry in Different Projects

ProjectApplication
EMQ[2]Mainly in (de-)serialization process. Use "Schema Registry" and "Rule Matching" to transfer a message from one serialization format to another.
Pulsar[3]Mainly in validation process. Use "Schema Registry" to validate a message.
Confluentinc[4]In both validation and (de-)serialization process.

Overview of OpenSchema

OpenSchema[5] proposes a specification for data schema when exchanging the message and event in more and more modern cloud-native applications. It designs a RESTful interface for storing and retrieving such as Avro, JSON Schema, and Protobuf3 schemas from three aspects(subject/schema/compatibility).

Requirements(Goals)

Requirement IDRequirement DescriptionComments
F-1In transmission, no message needs to contain schema information which bring efficiency.Functionality
F-2The message content from producer could be validated whether serialized correctly according to schema.Functionality

Design Details

Architecture

OpenSchema

Process of Transferring Messages under Schema Registry

Process

The highlevel process of messages transmission contains 10 steps as follows:

  • 1: Consumer subscribes "TOPIC" messages from EventMesh.
  • 2: Producer registers a schema to EventMesh.
  • 3: EventMesh registers a schema to Schema Registry.
  • 4: Schema Registry returns the id of newly created schema; EventMesh caches such id and schema.
  • 5: EventMesh returns the id of schema to Producer.
  • 6: Producer patches the id in front of messages and send messages to EventMesh.
  • 7: EventMesh validates the messages in the entry port and send it to EventStore; EventMesh retrieves messages from EventStore.
  • 8: EventMesh unpatches the id and send it to Schema Registry(if such <id, schema> does not exists in local cache).
  • 9: Schema Registry returns schema and EventMesh caches it.
  • 10: EventMesh patches schema in front of messages and push it to consumer.

Current Progress

Status

Current state: Developing

Discussion thread: ISSUE #339

Proposed Changes

The proposal has two aspects.

First is a separated Open Schema Registry, which includes storage and compatibility check for schema. This proposal is under developing.

Second is the integration of Open Schema in Eventmesh, which includes validation for schema. This proposal is to be developed.

As for the first proposal, some developing statuses are as follows.

Status Code and Exception Code

No.Status CodeException CodeDescriptionstatus
140140101Unauthorized Exception
240440401Schema Non- Exception
3^40402Subject Non-exist Exception
4^40403Version Non-exist Exception
540940901Compatibility Exception
642242201Schema Format Exception
7^42202Subject Format Exception
8^42203Version Format Exception
9^42204Compatibility Format Exception
1050050001Storage Service Exception
11^50002Timeout Exception

API Development Status

No.TypeURLresponseexceptioncodetest
1GET/schemas/ids/{string: id}Schema.class40101\40401\50001
2GET/schemas/ids/{string: id}/subjectsSubjectAndVersion.class40101\40401\50001
3GET/subjectsList\<String>40101\50001
4GET/subjects/{string: subject}/versionsList\<Integer>40101\40402\50001
5DELETE/subjects/(string: subject)List\<Integer>40101\40402\50001
6GET/subjects/(string: subject)Subject.class40101\40402\50001
7GET/subjects/(string: subject)/versions/(version: version)/schemaSubjectWithSchema.class40101\40402\40403\50001
8POST/subjects/(string: subject)/versionsSchemaIdResponse.class40101\40901\42201\50001\50002-
9POST/subjects/(string: subject)/Subject.class40101\40901\42202\50001\50002
10DELETE/subjects/(string: subject)/versions/(version: version)int40101\40402\40403\40901\50001-
11POST/compatibility/subjects/(string: subject)/versions/(version: version)CompatibilityResultResponse.class40101\40402\40403\42201\42203\50001-
12GET/compatibility/(string: subject)Compatibility.class40101\40402\50001
13PUT/compatibility/(string: subject)Compatibility.class40101\40402\40901\42204\50001-

Overall Project Structure

SchemaController.java+SchemaService.java : OpenSchema 7.1.1~7.1.2 (API 1~2)

SubjectController.java+SubjectService.java : OpenSchema 7.2.1~7.2.8 (API 3~10)

CompatibilityController.java+CompatibilityService.java : OpenSchema 7.3.1~7.3.3 (API 11~13) + Check for Compatibility

Project Structure

References

[1][schema validator (github.com)](https://github.com/search?q=schema+validator)

[2][EMQ: Schema Registry](https://www.jianshu.com/p/33e0655c642b)

[3][Pulsar: Schema Registry](https://mp.weixin.qq.com/s/PaB66-Si00cX80py5ig5Mw)

[4][confluentinc/schema-registry](https://github.com/confluentinc/schema-registry)

[5][openmessaging/openschema](https://github.com/openmessaging/openschema)