Digitizing educational standards to make learning materials reusable across countries
In March 2019, Learning Equality co-convened a two-day design sprint in Paris in collaboration with UNHCR, Google.org, Vodafone Foundation, and UNESCO. We explored the need for automated curriculum alignment in crisis contexts, and the possible role of artificial intelligence (AI) in recognizing curricular mandates and patterns, and recommending pertinent educational content in return. This work is part of a broader collaboration working with refugees and partner organizations to explore utilizing digital education to support learning in these contexts.
In continuation of the Design2Align blog post series, we’re thrilled to bring you a piece by Learning Equality’s Content Library Engineer, Ivan Savov, on technical considerations in OER for curriculum alignment facilitation.
Having previously discussed the implications of both metadata and teacher-generated content annotations to facilitate curriculum alignment, we’re particularly excited about this third post, because its topic will be the focus of a hackathon-type event we are organizing in collaboration with UNHCR, Vodafone Foundation, Google.org and UNESCO. The hackathon, which will take place in San Francisco on October 16–18, 2019, is aimed at prototyping a tool to automate parts of the curriculum alignment process, making it more readily available and feasible for use by countries in emergency contexts. Read on to learn how you can take part in it.
Context
Consider a refugee population coming from country C residing in host country B, with limited or no access to education. The trauma of conflict and displacement, coupled with the difficulty of integration within the host country puts refugee populations at a significant educational disadvantage, so it is worthwhile considering options that could “level the playing field” by providing improved access to education. There is hope that the vast amounts of Open Educational Resources (OER) that are freely available on the internet can play a role in this, in particular in combination with educational platforms like Kolibri. The Kolibri platform aims to provide access to learning opportunities for all and it is particularly suited for the refugee context as the runs-anywhere capabilities of the Kolibri applications allow it to be accessed in computer labs, in the classroom, from phones, and in informal learning centres.
Our experience and work with partners like UNHCR have shown that in emergency and crisis contexts, a key bottleneck is the lack of sufficient educational content aligned to the learning goals of the project. We can subdivide the “content problem” into two parts:
- In the formal learning context, the goal of aligning thousands of content items produced in the context of country A to the target curriculum of country B has proved to be a time consuming task that requires knowledge of the local curriculum. Even if there is a lot of content available, it won’t be immediately useful for teachers and learners unless it is organized according to the local curriculum standards.
- In the non-formal learning context, the curriculum design problem is complicated by lack of pre-existing target curriculum (e.g. for vocational training). This forces us to consider curriculum design questions before we can begin the content alignment process.
Curriculum alignment is in general a complicated multi-faceted process with multiple stakeholders (ministries of education, content producers, OER repository admins, curriculum designers, teachers, implementation partners, etc.). The task of curriculum design is even more complicated when our goal is to build a curriculum based on content produced in one country, adapted for refugees coming from another country, who are living in a third country.
To address this process, we have been working with UNHCR, Vodafone Foundation, Google.org, and UNESCO to build tools to facilitate semi-automated curriculum alignment workflows. The Paris convening was a unique opportunity to gather all the right people in the room and start working on the “spec” for what these tools might look like. Beyond spec-writing and discussion, the design sprint methodology pushed us to build prototypes and do user testing for the proposed solutions. What I intend to share here is background about OER content and curriculum standards for readers not familiar with OER and refugee contexts, and some progress on the initial prototype for a curriculum digitization and alignment tool that our group developed during the two-day design sprint.
The problem space: using OER across countries
There are large amounts of openly-licensed educational materials available on the internet, and the numbers continue to grow every day. Open educational resources include textbooks, video lectures, interactive simulations, documents, exercises, lesson plans, and games designed with specific learning objectives like literacy and numeracy. Teachers and students anywhere in the world can benefit and learn from these materials. To give you an idea of the numbers, Khan Academy has more than 40K content items (videos and exercises), CK-12 offers more than 10K content items (lessons and exercises), OpenStax produces hundreds of textbooks, and these are just some numbers I was able to find from the Kolibri Content Library. Beyond and in parallel to content that is supported in Kolibri, there are OER repositories such as Kiwix library, OER2Go, OER Commons, Global Digital Library, and LibreTexts that contain thousands more content items. Because of the nature of OER, all of these resources are free to use, but also remix, translate, or otherwise adapt.
OERs are usually organized according to the curriculum and educational standards of the country where the content was produced, which poses a significant challenge when trying to use it within the educational system of another country. Even in the best-case scenario where there are no language and cultural appropriateness differences, the mismatch between the organizational structure and sequencing of the curricula for the two countries pose difficulties for content reuse. Basically, we can’t just take a “bucket of content” produced for the U.S. curriculum and bring it to Kenya schools and tell educators “Here, find what you need in there.” In order to make the bucket of content truly useful, we must organize the content into meaningful learning objectives that present the material in a form that is immediately useful for the local educational context.
Each country publishes educational standards and curriculum documents that dictate what subjects, topics, and concepts must be covered in each subject, at each grade level. These curriculum standard documents serve as the blueprint for all parties in the education system: ministries of education, textbook publishers, accreditation bodies, school administrators, and individual teachers. For example, the Kenyan Institute of Curriculum Development (KICD) produces curriculum standards for each subject and grade level that content producers, curriculum designers, and teachers can all consult. Similarly, the Common Core State Standards (CCSS) in the United States define guidelines for what students should be learning in different grades. The curriculum standard documents of different countries in general have different structure, but all of them tend to be organized according to grade levels, subjects, units, and — at the highest level of granularity — specific learning objectives. There is a significant overlap between the sum-total of concepts and learning objectives between different countries, but the instructional sequencing can be very different. If there was a way to identify similar learning objectives in the curricula of countries A and B, this would tremendously speed up the process of aligning content produced for the curriculum of country A and reorganizing it for use within the curriculum of country B.
Exploring solutions for semi-automated curriculum alignment
Having explained the general context of the problem space, let’s now go back to Paris, on the seventh floor of UNESCO HQ, where 40 croissant-fed stakeholders convened at 8am for the Design Sprint on semi-automated curriculum alignment tools co-organized by Learning Equality, UNHCR, Google.org, Vodafone Foundation, and UNESCO. The idea for this gathering is basically to get everyone in the same room and try to come up with different solutions for the challenging multi-layer problem of curriculum alignment. Indeed, like a croissant with many pastry layers, any successful progress towards technology solutions for the curriculum alignment workflows requires work on many layers.
The participants in the event included representatives from ministries of education, content producers, OER repository administrators, curriculum designers, teachers, implementation partners, coaches from refugee camps, EdTech developers — basically a representative sample of all the layers of the croissant. The event was structured as a design sprint, which involves brainstorming and pooling together the past experience from all participants. The best ideas that emerged were further developed into prototypes and tested with target users. The goal was to rapidly explore the solution space together and get immediate feedback about the new ideas. Of the ideas initially proposed, six clusters emerged which were later pursued independently by different groups: five groups within the space in the UNESCO building and a group of participants joining us remotely from Kakuma Refugee Camp.
The focus of the group that I facilitated was the digitization of curriculum standards and the mapping between the curriculum standards. Thanks to the number and variety of experts that were present in the room, we were able to make significant leaps in defining the problem and exploring solutions. We developed a working prototype for a system to record curriculum data, which is an important building block for any curriculum alignment tool that we might want to develop in the future.
Our group worked on a tool for finding links between standards of different countries. We explored three building blocks:
- The process of digitization of curriculum standards.
- Human-centred editing and annotation processes for representing curriculum alignment relations.
- A semi-automated alignment workflow in which a machine learning model is used to recommend links between curriculum standard entries of different countries and presents them for review by curriculum experts.
A data model for curriculum standards
In order to better understand the problem space, we spent the first day discussing the data structure needed to represent curriculum standards of different countries in a common format. We considered the needs for data entry, storage, and representation for curriculum standard documents imported from whatever format they are available, as illustrated below.
A curriculum standard usually consists of 100 to 1000 standard entries, each standard entry specifying a particular learning objective in a given country, for a given subject, at a given grade level. Depending on the country, the standard can be organized into hierarchies with different levels (e.g. CCSS, CCSSM, CSSSM.6, CSSSM.6.EE, CSSSM.6.EE.B, CSSSM.6.EE.B.5). A simple way to think about the curriculum standard as a spreadsheet where each row corresponds to a particular standard entry. We assume each entry specifies the country of the standard, some unique identifier, and a set of learning objectives associated with this standard entry. We were able to validate the suitability of the proposed data structure with curriculum designers and curriculum aligners. Indeed, curriculum aligners already use spreadsheets for this process.
The next question we tackled was the definition for what it means for standard entries between countries to be matched. In the ideal case, the curriculum standards in both countries are organized around a set of well defined learning objectives, which makes it possible to find related standard entries in other countries. Unfortunately there are a number non-ideal-case difficulties that the curriculum alignment process must face including: the variability in the structure and taxonomies of the curriculum standards, the lack of standards in certain countries, and the lack of established taxonomies for representing non-formal learning (vocational and technology training). In the end we decided to keep things simple and defined a single type of relation between standard entries called “is relevant for” to represent links between curriculum standard entries. When a human curriculum expert with knowledge of the curriculum standards for both countries A and B makes the claim “sa is relevant for sb,” this indicates that content items aligned to the learning objectives of standard entry sa are also relevant for teaching the learning objectives of standard entry sb in the curriculum of country B.
In order to explore a concrete use case for the curriculum digitization data structure we manually digitized parts of the Kenya curriculum standards and imported the U.S. Common Core standard from CSV data we obtain from the internet. We were able to identify some “is relevant for” relations like the link between standard entry Kenya:Math.38.1 and the standard entry US:MATH.HSA.REI.B.4. Adding this annotation allows us to recommend 6 math exercises as potentially relevant to the Kenya standard Math.38.1.
This type of standards-alignment recommendation for potentially relevant content items can be considered in addition to any existing workflows for content discovery like browsing through content sources and keyword search, which are known to be time consuming.
While we did not get a chance to build and train any machine learning models for curriculum alignment, we believe there is a lot of potential to automate the process of finding matching entries between the curriculum standards of different countries. One thing that repeatedly came up within our group discussions and also later during the feedback session was the importance of keeping humans in the loop whenever introducing machine learning solutions to a given problem space. Judging whether curriculum standard entry sa is relevant for curriculum standard entry sb requires extensive curriculum expertise, understanding of the local context, and potentially subject matter knowledge. This is a “high-end” cognitive task and the nuances of which will be difficult to capture by a machine learning model. For this reason, whatever machine learning solutions we implement must always be positioned as helper methods for human experts, like a useful source of recommendation, rather than a replacement for human judgment based on curriculum expertise.
Personal takeaways and next steps
Participating in this event was an exhilarating experience for me as I was able to use my technology background to make rapid progress on this important problem that has been the center of many discussions at Learning Equality. Through my experience working as a technologist, I’ve come to the conclusion that software and technology details are the least important aspect of any solution — it’s usually the lack of domain knowledge and the difficulty of human communication that limit the design of software solutions. For this reason it was an amazing experience to work with Judy, Flora, Joe, and Safaa who all brought different expertise to the table. Wanjira, who has supported our project in actually doing curriculum alignment work, deserves an honorary mention as a test user and feedback provider. Thanks to the experts present in the room, I believe we have a clear picture of the data structure and curriculum data we’ll need, and identified a plausible workflow for introducing recommendations from machine learning models to be reviewed by human experts.
I’m excited for the next steps in this project and take it from a proof of concept sketches and turn it into prototypes. The key focus will be to obtain as much curriculum documents from national curricular bodies and other curriculum structures. With enough digital curriculum in place we can then explore different machine learning models to automatically find similarities between curriculum standards, and actually put the tool in the hands of curriculum experts to see if this bulk-alignment works.
I’m happy to announce these specific tasks of digitization of curriculum documents and semi-automated workflows have been chosen as the focus of the hackathon event organized by Learning Equality, UNHCR Vodafone Foundation, Google.org and UNESCO on October 16–18 in San Francisco, California. If you have experience with machine learning, recommender systems, or natural language processing and are interested in being involved, please be in touch to join as space is limited. If you’re interested in participating in these efforts beyond this hackathon, please email us at design2align@learningequality.org. Follow the hashtag #design2align for links to demos and datasets that will be produced during the event.
