3. Superset as Data Visualizer#
Status#
Accepted
Context#
As we found in development of Insights, creating web-based visualization of data is a complicated problem requiring a fair amount of resources and technical knowledge of both data and UI design. The complexity and expense of adding new visualizations contributed to the stagnation of the Insights project, in spite of it having a tremendous quantity of data to draw from.
Creating our new analytics system gives us an opportunity to bring in tools that are more flexible and able to support our needs without introducing a lot of new technology for small site operators to manage.
Decision#
The Aspects Analytics system (Aspects) will use Apache Superset as its primary data visualization and exploration tool. When developing a plan to support analytics in Open edX installs we have looked for a solution that offered at least the level of functionality we were able to get from the Insights user interface, as well as the following:
Open source
Deployable and configurable to work with Aspects using Tutor
Able to use the LMS as an authentication and authorization provider
Capable of handling large quantities of data
Minimal new technology / expertise needed beyond the usual Open edX stack
Well supported
Comes with a large set of visualizations to choose from
Presents an API that allows authorized users to further access their data (for example via download or Javascript extensions to the LMS)
Allows extension and customization for novel use cases
Allows operators to create and share queries / visualizations with each other
Allows site operators to choose from a wide variety of analytic database backends if necessary
Superset simple meets all of our needs without ballooning our tech stack or introducing vendor lock-in or expensive paid solutions.
Consequences#
Superset will be integrated into Aspects via a Tutor plugin that:
Allows it to share the existing Tutor redis and MySQL services
Integrates it with our chosen analytic database (detailed in a future ADR)
Creates our default suite of visualizations
Creates OAUTH integration with the LMS, with default permissions allowing different permissions for site operators, course instructors, and analytics users
Any custom UI components that are needed to support our use cases (ex: a template showing all of an instructor’s courses)
Rejected Alternatives#
Rewrite the Insights Dashboard / New Custom UI#
As stated in Context, this is a complicated problem and would be an expensive distraction from our core mission of education. Superset provides all that we could hope to create and much more with a tool that is available today. Several Javascript libraries exist that simplify the visualization aspect, but still require a great deal of work to create and maintain a site framework to drive them.
Grafana#
While Grafana can be used for a variety of data visualization purposes, it is primarily for supporting live services and lacks a lot of the visualizations that Superset has.
Redash#
Redash is similar in many ways to Superset, but seems poorly supported and possibly defunct. There have been few commits and no responses to Github Issues in the last 3 months. It would also seem to require operators to run Postgres in addition to the service itself, which isn’t ideal.
Several other options#
Most other packages in this space are either Javascript front-end components, make assumptions about data sharing that are incompatible with our desired permissions, lack features we need, or do not provide things we need in the open source versions of their products.