6. Areas of Responsibility#

Status#

Superseded by ADR 8 Project structure.

Context#

The Aspects Analytics system (Aspects) consists of several pieces of technology working together via some configuration and scripting. Decisions around where to store data and configuration need to be made so that developers understand where to make changes and look for the causes of issues. These decisions can have wide ranging impact on things such as extensibility and configurability of the system as a whole.

The number of plugins and systems in play can make it difficult to know where to look for configuration or add new features. This ADR hopes to address this by offering the following guiding principles:

  • Each service-based Tutor plugin should be able to be run the service separately from Aspects, and in different configurations

  • Opinionated configuration and non-LMS service associations belong in the tutor-contrib-aspects plugin

Decisions#

The service-based Tutor plugins (tutor-contrib-clickhouse, tutor-contrib-ralph, tutor-contrib-superset) will each be able to be run outside the Aspects context, such that if members of the community want to use them in a modified way, or override Aspects defaults, that will be possible. It is still assumed that the plugins will be used in an Open edX context and they will integrate themselves with the platform where necessary.

Specifically:

  1. tutor-contrib-clickhouse will:
    1. Create a ClickHouse admin user

    2. NOT create any databases or tables

  2. tutor-contrib-ralph will:
    1. Create an Ralph admin user

    2. Create a Ralph user for an LMS to use to insert statements

    3. NOT create any ClickHouse databases, users, or tables

    4. Note: Ralph will fail to start if the configured ClickHouse database and table for xAPI statements has not yet been created

  3. tutor-contrib-superset will:
    1. Create an admin user

    2. Create the Superset-specific MySQL database and permissions for the admin user (Note: By default this uses the LMS MySQL instance to save on system resources)

    3. Create a user in LMS for SSO

    4. Register itself as an SSO target with the LMS

    5. Disable local user management to force SSO login from LMS

    6. Create a set of default roles and permissions based on LMS permissions for each user (course staff, global staff, superuser, etc)

    7. Create row-based filters for queries to enforce course-level permissions (Note: since there are no datasets yet there is nothing to connect them to, but they are created here as they would likely be useful for a non-Aspects use case)

    8. NOT create any ClickHouse databases, users, or tables

The process of tying these plugins together is managed by the tutor-contrib-aspects plugin, which does not provide a running service, but rather the opinionated configuration of the above services and commands to manage them:

  1. In ClickHouse:
    1. Create any necessary databases (currently xAPI and course data)

    2. Create and manage the schemas (currently this is a very basic proof-of-concept, but will soon be managed via dbt)

    3. Create users and permissions for Ralph, Superset, and CMS (for inserting course structure data)

  2. In Ralph:
    1. Configure the ClickHouse user

    2. Configure the ClickHouse database and xAPI statements table to write to

    3. Optionally create test data in ClickHouse via Ralph

  3. In Superset:
    1. Configure the ClickHouse user

    2. Configure default dashboards, datasets, and reports

    3. Attach the row level course filter to the appropriate datasets

  4. In LMS:
    1. Add event-routing-backends as a Python requirement

    2. Create an Aspects user, necessary for the next step…

    3. Configure event-routing-backends to use xAPI and send statements to Ralph using the LMS user created in tutor-contrib-ralph)

The Aspects plugin will also be the conduit for running dbt via Tutor “do” commands. The dbt integration will be covered in a separate ADR.

Consequences#

  • The service-related plugins (ClickHouse, Ralph, Superset) can be run together or separately with configurations different from the Aspects defaults

  • A separate set of plugins could be created to replaces pieces of the stack with different technologies (ex. replace Superset with Tableau, or ClickHouse with other supported Ralph backend)

  • Extensions can be made at any level via additional Tutor plugins or other configuration

  • Configuration of the same services takes place in different plugins, which can be confusing. The hope is that this ADR clarifies where to look and add new functionality.

Rejected Alternatives#

Placing all of each service’s configuration into its plugin

This was considered and briefly implemented, but required tight coupling between the plugins and created a different level of confusion about whether a ClickHouse user for Ralph belonged in the ClickHouse plugin or Ralph’s. It was similarly confusing for shared resources like the ClickHouse schema which Ralph writes to part of, but the rest is managed elsewhere.

This would also leak Aspects implementation details into all of the plugins, reducing their flexibility for other use cases.