This recipe addresses the representation of a message broker in DataGalaxy in order to have a robust documentation solution for event management. To understand our approach, we take Kafka as an example. Let's take a look at a simplified diagram of how Kafka works. We can observe the presence of message producers (who will push information to Kafka) and message consumers (who will pull information from Kafka)
We can therefore consider Kakfa as a data hub and that is why we use the dictionary to represent Kafka. For this purpose:
- As messages are often in JSON format, we recommend using NoSQL databases ;
- Topics will be represented by directories
- Messages will then be represented by files
It is of course possible to manage the versions of the messages by integrating a numbering in the file names.
this way you can identify the publications of the producers and the recoveries of the consumers thanks to data processing. Thus it will be possible to identify
- All consumers of a publication
- All the producers used by a consumer. *
This will work in the same way for other message brokers: Rabbit MQ, Red