4. Kafka Managed Hosting#

Status#

Accepted

Context#

  • Setting up a Kafka-based event bus can be complex since it comes in multiple parts that all need to be deployed and maintained together.

  • The 2U/edX team responsible for this new infrastructure does not have the capacity to manage this in house.

Decision#

Note

This decision is specific to edx.org and is not a requirement or recommendation for how other deployments in the Open edX community manage their brokers. However, the background may still be useful to other community members.

For edx.org, the initial deployment of Kafka-based event bus will use the software as a servics (SaaS) provider Confluent.

Additional Background#

Amazon MSK is an AWS managed service that supplies the Apache Kafka core platform only.

The Confluent Platform adds additional capabilities, some of which are only commercially available.

  • Schema Registry

  • Monitoring and alerting capabilities (Commercial)

  • Self-balancing clusters (Commercial)

  • Tiered storage (Commercial) (future feature of Apache Kafka)

  • Infinite retention (Commercial - Cloud only)

Also see a useful and biased comparison of Apache Kafka vs Vendors by Kai Waehner (of Confluent), comparing various providers and distributions of Kafka and related or competitive services. Or see (Confluent biased) Amazon MSK vs Confluent Cloud.