10. Multiple event types per topic#

Status#

Accepted 2023-05-11

Context#

For the initial implementation of the event bus, we decided to limit each topic to use a single schema. This meant that every signal in openedx-events required a different topic. This worked for our initial use case, course catalog updates, because all changes were considered updates and could be emitted using the same signal. However, other types of events are not so easily grouped into a single signal. For example, there are different signals for XBLOCK_PUBLISHED, XBLOCK_UPDATED, and XBLOCK_DUPLICATED, and indeed different schemas across the three signals. Routing these signals through different topics could lead to events being processed in a nonsensical order, for example an xblock being duplicated before it’s been published. See Should You Put Several Event Types in the Same Kafka Topic? and Putting Several Event Types in the Same Topic – Revisited for more information on why it can be useful to group different event types on the same topic.

Decision#

All implementations of the event bus must support sending multiple event types to a single topic.

Note that here, “topic” refers to however the event bus implementation groups related events (for example, streams in Redis).

Consequences#

  • Event consumers will need to determine the signal from the message headers rather than taking a signal as a passed argument in the management command.

  • We will need to establish new conventions for naming topics, as they can no longer necessarily be named after a single event type.

Rejected Alternatives#

Keeping one event type per topic#

An additional alternative would be to continue writing the events to separate topics and requiring consumers to manage the complexity of ordering across topics. This is possible, but quite complicated, so we plan to avoid it.