Open Response Assessment Data

Open Response Assessment Data#

Warning

This page was last updated in 2020; refer to the Open edX codebase for the latest information on this database schema.

This section describes the data schema for the Open edX open response assessment (ORA) system.

The schema includes a number of tables, and the ORA system offers a range of uses. As a result, the data can be complex. It is recommended that you set up a course on a Tutor instance so that you can create assessments and test the possible learner interactions.

Workflow Tables#

The ORA workflow tables store data about a response that a learner has submitted as it goes through each of the steps defined for the ORA component.

The top level table is AssessmentWorkflow, which specifies the steps that a learner must complete for the ORA component.

Diagram of the assessment workflow and assessment workflow step tables.

The self assessment step in an ORA component is a required, single step process. The AssessmentWorkflowStep table stores data about this step. A component can also include a training step, a peer evaluation step, or both. Additional tables store data for these optional steps.

  • StudentTrainingWorkflow stores data about a learner’s evaluation of a set of instructor-provided samples, including what evaluations the learner has completed and when each evaluation was made.

  • PeerWorkflow stores data about the progress of a response as it undergoes peer grading, which can involve input from one or more peers.

Assessment and Rubric Tables#

The assessment tables store the assessments that have been made for a particular response. Assessments can be self assessments, submitted by learners about their own work, or peer assessments, which are submitted by other learners. Assessments are scored against rubrics.

A Rubric specifies guidelines on how to assess submissions. Each rubric consists of a set of Criterion entries, each of which has a set of CriterionOptions.

For example, a criterion for assessing writing might be, “The essay is well structured”. The options that a learner can choose for that criterion might be, “No evidence of structure; rambles”, “There is an attempt at organizing the material, but it doesn’t flow very well”, and “Clear structure from beginning to end”.

The data for a rubric is written to these tables at the time of assessment. If a course team makes a change to the rubric for a problem, these tables contain different entries for assessments that were done before, and after, the change.

Submission, StudentItem, and Score Tables#

The Submission table represents the responses made by a learner for a given component in a given course. Depending on the configuration of the ORA component, a learner can contribute multiple submissions for a given problem.

Note

The Submission table includes only text responses. The data package does not include files uploaded by learners to accompany their text responses.

The StudentItem table identifies a learner and the ORA component in the course. Rows in this table are linked to all submissions made by that learner for that component, as well as to each Score for that component.

Diagram of the tables used for tracking submissions, student items, scores, and the related score summary table.

Detailed ERD#

This entity relationship diagram (ERD) provides a reference for the tables involved, their columns, and their relationships.

Some of the tables modeled in this diagram have many to many relationships, in which case there is an additional table that stores data to reflect those relationships. An example is the assessment_trainingexample_options_selected table for the relationship between the CriterionOption and TrainingExample tables, identified on the diagram by options_selected (trainingexample).

Diagram of the primary tables used for ORA2 to track submissions, assessments, and rubrics.

You might need to download this image to view it in sufficient detail.

Notes#

  • Anonymous learner IDs are used throughout the ORA2 data. To link to full user data, you must join to the student_anonymoususerid table. For more information, see Columns in the student_anonymoususerid Table.

  • The ERD includes several tables that include AI in the name. These tables were used for an edX prototype of an example based assessment system. That prototype system is not yet in general use.

Maintenance chart

Review Date

Working Group Reviewer

Release

Test situation

2020-10-22

n/a

n/a

Last documentation update on this date
Contents