{"id":112,"date":"2021-11-29T17:09:43","date_gmt":"2021-11-29T17:09:43","guid":{"rendered":"http:\/\/192.168.64.4\/in-en\/?post_type=ai&p=112"},"modified":"2023-06-12T10:09:46","modified_gmt":"2023-06-12T10:09:46","slug":"question-discrimination-factor","status":"publish","type":"ai","link":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/","title":{"rendered":"Question Discrimination Factor"},"content":{"rendered":"

Tests are learners’ most preferred assessment techniques to measure performance against the targeted learning outcomes through deep learning-based methods. So, tests must be fair and effective to identify students\u2019 learning gaps and boost students\u2019 learning. The ability of a test to meet these goals is an aggregation of how relevant each question of the test is. Thus, the reliability of a test can be increased by item analysis, where students\u2019 responses for each question or item are utilized to evaluate test performance. One of the important methods in Item Analysis is Item discrimination which refers to the power of a question to differentiate between different learners. The question Discrimination Factor is an index that measures how well a question can differentiate between different user cohorts. It depicts how top scorers are more likely to get a question correct than low scorers.<\/span><\/p>\n

Embibe has used traditional statistical\u00a0\u2014 Item Point Biserial Correlation and deep learning-based methods to compute the Question Discrimination Factor of questions. Item Point Biserial Correlation is a Pearson-product moment correlation between a student’s question score and total score. So, the higher the difference between the total scores of students who got the question correct and those who got the question incorrect, the higher the Question Discrimination Factor value will be. We also implemented the 2PL model from classical Item Response Theory using Deep Neural Network architecture. Given the students\u2019 attempts data, we derive the question\u2019s difficulty level and discrimination factor from the weights of trained DNN. Here is an example of how the Question Discrimination Factor’s value varies with learners’ question attempt interactions.<\/span><\/p>\n\n\n\n\n\n\n
QDF = 0.11<\/strong><\/td>\nQDF = 0.80<\/strong><\/td>\n<\/tr>\n
Question 1:<\/b><\/p>\n

Alcohols of low molecular weight are<\/span><\/p>\n

a. Soluble in all solvents (Correct Option)<\/span><\/p>\n

b. Soluble in water<\/span><\/p>\n

c. Insoluble in all solvents<\/span><\/p>\n

d. Soluble in water on heating<\/span><\/td>\n

Question 2:<\/b><\/p>\n

Aspirin is also known as<\/span><\/p>\n

a. Acetyl salicylic acid (Correct Option)<\/span><\/p>\n

b. Methyl salicylic acid<\/span><\/p>\n

c. Acetyl salicylate<\/span><\/p>\n

d. Methyl salicylate<\/span><\/td>\n<\/tr>\n

\"\"<\/td>\n\"\"<\/td>\n<\/tr>\n
Table 1: Comparison Between the Distribution of Total Marks for Correct and Incorrect Questions with Low QDF and High QDF Values<\/i><\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Here, the x-axis represents the total marks scored, and the y-axis represents the normalized number of students for gradient-based learning in deep learning. The yellow line denotes the distribution of total marks of students who got the question incorrect. The blue line denotes the distribution of total marks of students who got the question correct. <\/span><\/p>\n

In Question 1, there is a high overlap between the total marks of students who got the question correct. In contrast, in Question 2, the overlap is very less and hence, the value of the Question Discrimination Factor is higher for Question 2 than Question 1. The final Question Discrimination Factor value is the fine-tuned result of the above method and test parameters. <\/span><\/p>\n

Embibe conducted a validation experiment to compare the performance of students in two different tests:<\/span><\/p>\n

    \n
  1. Baseline Policy:<\/b> Questions are selected without bias due to discrimination factors from the ground truth database, ensuring an expected distribution over difficulty levels and syllabus coverage.<\/span><\/li>\n
  2. Discrimination Only Policy:<\/b> Here, questions are selected from the ground truth dataset, ensuring syllabus coverage \u2014 at least one question from each chapter, and ensuring that the overall discrimination factor of the questions is maximized at any difficulty level.<\/span><\/li>\n<\/ol>\n

    For the experiment, a total of 312 students were selected to take a test containing 75 questions. Two statistical metrics compared the performances of the test:<\/span><\/p>\n

      \n
    1. Evaluation using RMSE:<\/b> Using the Item Response Theory model, we predict each student\u2019s probability in the evaluation set of answering the questions correctly and compute the average ability from the scores of the students if they were to attempt the generated test paper. We also determine the ground truth ability of each student from the Item Response Theory model for deep learning collaborative filtering. Finally, we compute the root mean squared error between the ground truth ability and inferred ability to measure the accuracy.<\/span><\/li>\n
    2. Evaluation using Spearman\u2019s \u03c1:<\/b> Here, we sort students\u2019 abilities obtained from the ground truth data and the generated test and determine the rank correlation \u03c1 between the two ranks.<\/span><\/li>\n<\/ol>\n\n\n\n\n\n\n
      Policy<\/strong><\/td>\nRMSE<\/strong><\/td>\nRank corr \u03c1<\/strong><\/td>\n<\/tr>\n
      Baseline Policy\u00a0<\/strong><\/td>\n0.844\u00a0<\/span><\/td>\n0.59\u00a0<\/span><\/td>\n<\/tr>\n
      Discrimination Only Policy<\/strong><\/td>\n0.549\u00a0<\/span><\/td>\n0.83<\/span><\/td>\n<\/tr>\n
      Table 2: Comparison of RMSE (Inferred Ability and Ability from Ground Truth) and Rank Correlation \u03c1 in Tests Generated by Different Policies\u00a0<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Also, we found that the Discrimination Only Policy test gives a 24.8% better spread of scores (score at 95th percentile of students \u2013 score at 5th percentile) than the Baseline Policy test.<\/p>\n

      Hence, the use of high Question Discrimination Factor questions in tests improves the quality of the test in terms of its power to differentiate among students under the same targeted learning goals in deep learning-based methods. Also, these are leveraged to improve content quality where we identify questions with negative Question Discrimination Factors and improve their relevance and clarity.<\/p>\n

      References<\/b><\/h4>\n
        \n
      • Soma Dhavala, Chirag Bhatia, Joy Bose, Keyur Faldu, Aditi Avasthi, \u201cAuto Generation of Diagnostic Assessments and their Quality Evaluation,\u201d July 2020, EDM.<\/span><\/li>\n
      • Vincent LeBlanc, Michael A. A. Cox, \u201cInterpretation of the point-biserial correlation coefficient in the context of a school examination,\u201d January 2017, The Quantitative Methods for Psychology 13(1):46-56<\/span><\/li>\n
      • Linden, W. D., and R. Hambleton. \u201cHandbook of Modern Item Response Theory.\u201d (1997), <\/span>Biometrics<\/span><\/i> 54:1680<\/span><\/li>\n
      • Desai, Nishit, Keyur Faldu, Achint Thomas, and Aditi Avasthi. “System and method for generating an assessment paper and measuring the quality thereof.” U.S. Patent Application 16\/684,434, filed October 1, 2020.<\/span><\/li>\n
      • “Autogeneration of Diagnostic Test and Their Quality Evaluation – EDM:2020”, EDM 2020 presentation, Jul 2020,<\/span> https:\/\/www.youtube.com\/watch?v=7wZz0ckqWFs<\/span><\/li>\n
      • Faldu, Keyur, Achint Thomas, and Aditi Avasthi. “System and method for behavioral analysis and recommendations.” U.S. Patent Application 16\/586,525, filed October 1, 2020.<\/span><\/li>\n<\/ul>\n","protected":false},"featured_media":0,"template":"","yoast_head":"\nImproving Assessment Techniques in AI Education - Embibe<\/title>\n<meta name=\"description\" content=\"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Improving Assessment Techniques in AI Education - Embibe\" \/>\n<meta property=\"og:description\" content=\"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/\" \/>\n<meta property=\"og:site_name\" content=\"EMBIBE - The most powerful AI-powered learning platform\" \/>\n<meta property=\"article:modified_time\" content=\"2023-06-12T10:09:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/exams-assets.embibe.com\/exams\/wp-content\/uploads\/2021\/11\/03011308\/ai18.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/\",\"url\":\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/\",\"name\":\"Improving Assessment Techniques in AI Education - Embibe\",\"isPartOf\":{\"@id\":\"https:\/\/www.embibe.com\/in-en\/#website\"},\"datePublished\":\"2021-11-29T17:09:43+00:00\",\"dateModified\":\"2023-06-12T10:09:46+00:00\",\"description\":\"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.embibe.com\/in-en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Question Discrimination Factor\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.embibe.com\/in-en\/#website\",\"url\":\"https:\/\/www.embibe.com\/in-en\/\",\"name\":\"EMBIBE - The most powerful AI-powered learning platform\",\"description\":\"Just another WordPress site\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.embibe.com\/in-en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Improving Assessment Techniques in AI Education - Embibe","description":"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/","og_locale":"en_US","og_type":"article","og_title":"Improving Assessment Techniques in AI Education - Embibe","og_description":"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.","og_url":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/","og_site_name":"EMBIBE - The most powerful AI-powered learning platform","article_modified_time":"2023-06-12T10:09:46+00:00","og_image":[{"url":"https:\/\/exams-assets.embibe.com\/exams\/wp-content\/uploads\/2021\/11\/03011308\/ai18.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/","url":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/","name":"Improving Assessment Techniques in AI Education - Embibe","isPartOf":{"@id":"https:\/\/www.embibe.com\/in-en\/#website"},"datePublished":"2021-11-29T17:09:43+00:00","dateModified":"2023-06-12T10:09:46+00:00","description":"Learn how Embibe utilizes traditional statistical and deep learning-based methods to measure question discrimination factor (QDF), and how it impacts test outcomes for students.","breadcrumb":{"@id":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.embibe.com\/in-en\/artificial-intelligence-ai-in-education\/question-discrimination-factor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.embibe.com\/in-en\/"},{"@type":"ListItem","position":2,"name":"Question Discrimination Factor"}]},{"@type":"WebSite","@id":"https:\/\/www.embibe.com\/in-en\/#website","url":"https:\/\/www.embibe.com\/in-en\/","name":"EMBIBE - The most powerful AI-powered learning platform","description":"Just another WordPress site","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.embibe.com\/in-en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/www.embibe.com\/in-en\/wp-json\/wp\/v2\/ai\/112"}],"collection":[{"href":"https:\/\/www.embibe.com\/in-en\/wp-json\/wp\/v2\/ai"}],"about":[{"href":"https:\/\/www.embibe.com\/in-en\/wp-json\/wp\/v2\/types\/ai"}],"wp:attachment":[{"href":"https:\/\/www.embibe.com\/in-en\/wp-json\/wp\/v2\/media?parent=112"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}