Interoperability for assessment: Starting the process for Kolibri & hearing your input!

Question banks, exercise types and export standards aren’t always at the top of my mind, but during our team’s first annual hackathon last week, I joined a project to start the process of developing a QTI Standard Compliant Exercise Render so that questions renderable in our open-source education platform, Kolibri, could be compatible with QTI Standards.

The Learning Equality team hacking away — QTI project team on the right.

Let’s start at the beginning. QTI is an abbreviation for Question and Test Interoperability, which is a standard supported by IMS Global. It is what allows for interoperability for assessments — basically the exchange of assessment content between systems. We’re aware of some of the downfalls of committing to QTI, and the various pros and cons to using a standard will not be covered here, but we are motivated to start this transition to QTI compatibility for a few reasons:

• More control over the user experience in Kolibri of exercises created in our curriculum tool, Kolibri Studio, for monitoring, formative and summative assessments
• Increase in range of exercise types that can be created in Kolibri Studio
• Ability to export make Studio-created exercises (particularly those that have been localized) for more widespread use
• Ability to import from standardized question banks
• Increase in potential for discoverability
• More streamlined exercise renderer with fewer software dependencies
• Interest in leveraging existing knowledge in terms of displaying, creating and distributing exercises, while contributing to the open ecosystem by further explaining QTI.

After spending much of the hackathon poring over the implementation guide for QTI and considering the various implementation and content partner needs that we have been hearing over the past few months, I was actually surprised about the apparent flexibility that QTI affords. The various items (definition: “an item is a set of interactions — possibly empty — collected together with any supporting material and an optional set of rules for converting the candidate’s response(s) into assessment outcomes”) seem to cover the exercise types that have been deemed important through various conversations with our partners and the broader community. I had previously gotten the sense that there may be numerous exercise types that weren’t compliant with QTI, but was surprised to discover the flexibility in combinations of exercise types.

Presenting our project to the team.

It seems to me that the biggest challenges with expanding our exercise type options will be around (1) inundating exercise creators on Kolibri Studio with too many options; (2) UI design so that exercise creators are not only aware of how to create the various exercise types, but also understand the pedagogical underpinnings of using these types of exercises; and (3) maintaining the relevant documentation in our Toolkit to support the expanded exercise types.

The Learning Equality team following our Hackathon Oscar-style awards ceremony, complete with red carpet.

Taking a closer look at the QTI items and considering partner needs, we’ve prioritized 10 question types for Kolibri Studio (noting that the first four are currently supported in Kolibri Studio).

Question Type (Item — QTI Standard)

1. Single Selection (Choice)
2. Multiple Selection (Choice Multiple)
3. True / False (Choice)
4. Input (Text Entry)
5. Essay Question (Extended Text + Rubric)
6. Reviewing Stimulus & Answering Questions (Shared Material Object)
7. File Upload (Upload)
8. Self-Assessment (Slider)
9. Fill in the Blank on an Image (Graphic Gap Match)
10. Drawing (Drawing)

We would love your feedback on this prioritization of question types, and your experiences with and need for QTI. Let us know: content@learningequality.org.

And here’s a limerick we presented during the Hackathon about QTI for your pleasure:

QTI has questions for all

Can Kolibri pick up the ball?

With import and export,

Metadata in concert,

We’ll make it ‘cos we’ve got the gall!