OEP-52: Event Bus Architecture#
OEP-52 |
|
Title |
Event Bus Architecture |
Last Modified |
2024-07-09 |
Authors |
Feanil Patel <feanil@edx.org>, Robert Raposa <rraposa@edx.org> |
Arbiter |
Tobias Macey <tmacey@mit.edu> |
Status |
Accepted |
Type |
Architecture |
Created |
2021-08-16 |
Review Period |
2022-06-09 - 2022-06-30 |
References |
Follow up work: event bus project roadmap |
Overview#
Adding an event bus to the Open edX platform allows asynchronous event messaging across services, which enables a number of improvements aligned with our architectural goals.
An abstraction layer will be provided to enable the choice of multiple technologies for implementing an Open edX event bus.
Kafka and Redis streams implementations of the event bus have been implemented.
Event bus events extend the Hooks Extension Framework events, and use an Avro schema to serialize the existing hooks signals.
Context#
The already accepted OEP-41: Asynchronous Server Event Message Format details the general format and conventions the Open edX platform should use for asynchronous event messaging across services. It also provides background on a set of Architectural Goals for the Open edX platform, including:
Align with the Architecture Manifesto themes of decentralization and asynchronous communication in support of loose coupling between services.
Eliminate blocking, synchronous requests.
Eliminate expensive, batch synchronization.
Reduce the need for plugins.
Flexibly integrate with event producers.
Simplify integration to external systems.
However, this earlier OEP explicitly leaves out of scope the specific transport and libraries used for this messaging. At the time, the Open edX platform lacked a reliable way to send events to multiple consumers across services, following the publish-subscribe messaging pattern (pub/sub).
Decision#
Adding an event bus to the Open edX platform allows asynchronous event messaging across services, which enables a number of improvements aligned with our architectural goals.
For the purpose of this OEP, the focus of the event bus is to provide publish-subscribe messaging pattern (pub/sub) and event-driven capabilities.
An abstraction layer will be provided to enable the choice of multiple technologies for implementing an Open edX event bus.
See ADR on Multiple Event Types Per Topic to learn more about the requirement of Open edX event bus implementations to support multiple event types per topic.
An initial event bus implementation has been implemented using Kafka.
See edx.org ADR on Kafka-Based Event Bus when considering other event-bus technologies.
See edx.org ADR on Kafka Managed Hosting if considering Kafka hosting options.
See edx.org ADR on Managing Kafka Consumers if exploring how to deploy consumers for Kafka.
A secondary implementation has been implemented using Redis streams.
Event bus events extend the Hooks Extension Framework events, and use an Avro schema to serialize the existing hooks signals.
See ADR on External event bus and Django Signal events to learn more about how the
OpenEdxPublicSignal
internal used to send the same internal signal-based events across services, and then fire the signal-based events again within consuming services.See ADR on External Event Schema Format and ADR on Event Schema Serialization and Evolution to learn about how the Avro Schema format is used for serializing external events published to Kafka, and consumed from Kafka. Additionally, learn about tooling to automatically handle the translation from
OpenEdxPublicSignal
internal events to Avro Schema formatted events, and back again. Also learn how this explicit schema format can aid in schema evolution.
Note
All decisions related to Open edX Events where the event bus key components are implemented will now be documented and made publicly available through the Architectural Decision Records (ADRs) in the openedx-events repository. For more details, refer to the Open edX Events ADRs.
Consequences#
Individual usage of any new Open edX event bus cannot be required until there is an implementation available as part of a named release of the Open edX platform.
Offering an abstraction layer for the event bus is the current intent, but will require additional work and support. Additionally, any alternative technology implementations (in addition to Kafka and Redis) will require implementation and support.
Although external events offer many benefits, it also requires a different mindset around eventual consistency. Proper education and experience will be needed to support the success of the event bus.
Change History#
2025-01-14#
Added note about decisions related to Open edX Events being documented and made publicly available through the Architectural Decision Records (ADRs) in the openedx-events repository.
2024-07-09#
Removed note about OEP being provisional, as it has been marked as Accepted.
2023-05-15#
Added link to event_bus_redis implementation ADR.
Updated OEP status to Accepted.
2023-05-12#
Added link to new openedx-events ADR
2022-11-29#
Removed disclaimer about provisional ADRs (most have since been accepted) and added follow-up work in References
2022-07-11#
Updated formatting.
Updated note about provisional updates, including reference to roadmap item.
2022-03-30#
Updated draft as follows:
Moved the Kafka technology choice to a separate edx.org ADR.
Added references and overview of a variety of other event bus related ADRs.
Moved ADR on schema representation to openedx-events, near a number of other event bus related ADRs.
2021-08-16#
Initial draft.