Building the Learning Outcomes AI Stack

Embibe has been a data-driven, data-focused, data-hungry company right from its inception, having understood very early that data was the key ingredient in being able to personalise education for every student at scale. And yet, data alone completes only half the picture. The personalisation of education using technology is a challenging problem that requires the interplay of advanced algorithms that can leverage enormous amounts of data across multiple sub-domains.

At Embibe, we believe leaders are not born; they are crafted over time and experience with a thousand little learnings along the journey. Over the past eight years, Embibe has painstakingly honed its data-gathering and interpretation capabilities. It has used this data with advanced machine learning and AI techniques to deliver an unparalleled personalised education experience for students using its platform. In what follows, we lay out Embibe’s philosophy for to data-driven personalisation of EdTech.

Capturing and Collecting Data

Data itself does not have much value if not instrumented properly, collected frequently, and cleaned thoroughly to add value to a user. For instance, the fact that a practice or test question was attempted is not nearly as useful as all the micro-events related to an attempt, like the time of first look of question, time of last save, time of each re-visit, answer choice switches at each visit, hints used during a visit to practice attempt, question out of sequence in the test session, etc. Embibe has invested heavily in capturing granular data over the last eight years. Embibe can capture rich data types not limited to

• User-interaction explicit events — clicks, taps, hovers, scrolls, text updates,
• User-interaction implicit events — cursor position, tap pressure, device orientation,
• System-generated server-side events — page load, session refreshes, API calls,
• System-generated client-side events — system push notifications and triggers.

Domain Expertise

It is generally known to those skilled in the art that data scientists working in silos may be unable to add value to captured data as they don’t have sufficient context and domain knowledge to provide meaningful insights. Embibe understands this and has ensured sufficient overlap between data scientists and academic experts.

Preparation of Academic Data: Embibe has invested time and resources in creating and curating academic data unavailable in the public domain. For instance, a team of 30 faculties over the years has created a Knowledge Tree of close to 62K concepts with hundreds of thousands of interconnections using semi-supervised algorithms, with 426 meta variables at each concept resulting in tens of millions of meta-variables. This team also manually tagged hundreds of thousands of questions to meta tags, such as concepts on the Knowledge Tree, exam syllabus, skills, difficulty level, ideal time, and boom level, to generate training data for downstream algorithms.

Academics + Science: There is a learning curve when data scientists and academicians work together. For instance, scientists must know what parameters academicians consider while setting tests, how they bring in variations, and how they personalise tests for individual students. Translating an academician’s knowledge to algorithmic code is time-consuming and builds domain expertise in the scientist pool. Embibe also used faculty-scientist interactions to generate an academic corpus of tens of thousands of phrases using data mining techniques.

Using Data and AI to Build a Personalized EdTech Platform

First-hand data is acquired over a sufficiently long period of time. Even if someone pours lots of money to acquire users, they may not have sufficient engagement data over time as users may not have interacted enough with the system. Embibe owns all its data and leverages it using advanced algorithms to solve a number of interconnected sub-problems to personalise EdTech.

Smart Tagging: The essence of textual information in the tagging questions to concepts, topics, and other buckets lies in academic keywords. Consolidated academic keyword dictionaries are not available in the public domain.

Moreover, it is tricky to distinguish academic keywords from non-academic ones. A seemingly non-academic word like ‘end’ is actually academic in the appropriate context, for instance, “force applied at the end of string”. Embibe’s smart tagging algorithms can identify the most relevant concepts to tag to a question 82% of the time compared with only 18% for crowd-sourced human faculty.

Automated Test Generation: The scientist-faculty exercise described above resulted in Embibe’s Automated Test Generation module, which was used to create hundreds of test question papers across many exams from lakhs of questions tagged with close to 62,000 concepts, difficulty level, ideal time, bloom level, skills. The module uses a state-of-the-art hybrid algorithm that uses Simulated Annealing and Genetic Algorithms to create new test question papers that match the level of any specified exam at a fraction of the time it would take experienced faculty to do so.

Behavioural Goal Setting and Score Improvement Prediction: With numerous behavioural case studies, faculty domain knowledge and statistical pattern mining, Embibe knows how to measure and improve latent behavioural attributes. Based on hundreds of thousands of statistically valid assessments, we are able to identify the most impacting behavioural attributes, set progressive goals and predict score improvement based on those improvements.

Embibe Score Quotient: Embibe has successfully identified several high-impact features from hundreds of hypotheses over hundreds of thousands of assessments with detailed data signals captured at the event level. We have estimated the score with 94% accuracy and can establish that academic quotient forms 61% and behavioural forms 39% of what impact exam scores. The model was able to converge because of the availability of hundreds of thousands of user tests on the platform. With transfer learning, boot-strapping models for other exams became very easy.

Effort Lead Score Improvement: Embibe’s platform interactions are measured against learning outcomes. With four seasons of historical data and focused research, it validates Embibe’s offering and feeds forward to optimise learning outcomes. Students in the high-effort cohort have achieved ~50% net score improvement.

Content Discovery and Recommendations: Embibe’s search-based UI is powered using the vast amounts of data collected over the past eight years as students interacted with our platform. Our search engine that serves content based on user searches re-ranks relevant results in real-time based on user cohort assignments, historical search trends and content consumption patterns, content difficulty to exam based on and user past user interaction, among 25 such weighting factors for a combined search space of hundreds of millions of potential combinations to choose and re-rank. Additionally, students are shown recommendations for targeted content to focus their time on, based on their past interaction with the platform if sufficient data is available or based on lookalike users in data insufficient cases.

What it Takes To Be an AI Platform for EdTech

There are a number of EdTech companies in the world today. Most of these companies focus only on some subset of the problems that need to be addressed to personalise education and build an AI-driven platform for EdTech. The table below lays out the evolution of an EdTech company as it moves from being a Test Prep portal to being a true EdTech Platform:

Content	Attempt Data	Domain Expertise	Knowledge Graph	Data Science Lab	Possibilities
(1) Average 250+ questions at the chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests					Basic Test Prep
(1) Average 250+ questions at the chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests	Some minimum number of user-level attempts on questions				> Basic Test Prep + Basic User-level Feedback Analytics
(1) Average 250+ questions at chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests, (4) 5 Learn content (videos, text, links) per Chapter	Some minimum number of user-level attempts on questions				Basic Test Prep + Basic Feedback Analytics + Learn
(1) Average 500+ questions at chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests, (4) 5 Learn content per Chapter	Average of 25+ attempts per question for up to ~6 million attempts over all questions	In-house academician for (1) Content Hygiene, (2) quality control on Tests, (3) Doubt Resolution			Basic Test Prep + Detailed Feedback Analytics + Learn + Doubt Resolution
(1) Average 500+ questions at chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests, (4) 5 Learn content per Chapter	Minimum of 25+ attempts per question for a total of ~6 million attempts over all questions	In-house academician for (1) Content Hygiene, (2) quality control on Tests, (3) Doubt Resolution	(1) Basic taxonomy up to topic level (~5 topics per chapter) for up to ~4000 topics		Advanced Test Prep = Basic Test Prep + Detailed Feedback Analytics + Learn + Doubt Resolution + Topic-level Features
(1) Average 500+ questions at chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests, (4) 5 Learn content per Chapter	Average of 100+ attempts per question for a up to ~20 million attempts over all questions	In-house academician for (1) Content Hygiene, (2) quality control on Tests, (3) Doubt Resolution, (4) AI hypotheses generation with Data Scientists	(1) In-depth taxonomy up to concept level (~100 concepts per chapter) for up to ~40K concepts	4 person Data Science Team working for 2+ years	Advanced Test Prep + Personalization + Learning Outcomes
(1) Average 500+ questions at chapter level, (2) at least 3 Chapter Level Tests, (3) 10 Full Tests, (4) At least 5 Learn content per Chapter	Average of 150+ attempts (50 per cohort) per question for up to ~30 million attempts over all questions	In-house academician for (1) Content Hygiene, (2) quality control on Tests, (3) Doubt Resolution, (4) AI hypotheses generation with Data Scientists	(1) In-depth taxonomy up to concept level (~100 concepts per chapter) for up to ~40K concepts	8 person Data Science Team working for 2+ years solving problems like Auto Ingestion (OCR), Auto Tagging (NLP, ML), Packaging (Optimization), Knowledge Graph Generation and Calibration (IR, Graph Mining, ML), Behavior Intervention (ML), Personalization (IRT, ML)	AI Platform for Education = Advanced Test Prep + Personalization + Learning Outcomes + Intelligence-as-a-Service