{"id":1705,"date":"2023-02-02T06:40:16","date_gmt":"2023-02-02T06:40:16","guid":{"rendered":"https:\/\/www.embibe.com\/in-ml\/?post_type=positions&p=1705"},"modified":"2023-02-02T06:42:04","modified_gmt":"2023-02-02T06:42:04","slug":"an-article-on-approaches-to-qdf","status":"publish","type":"positions","link":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/","title":{"rendered":"An Article on Approaches to QDF"},"content":{"rendered":"\n

Tests<\/strong> are the most preferred assessment technique used by learners to measure performance against the targeted learning outcomes.<\/strong> So tests must be fair and effective to identify students\u2019 learning gaps and boost students’ learning. The ability of a test to meet these goals is an aggregation of how relevant each question of the test is. Thus, the reliability of a test can be increased by item analysis,<\/strong> where students’ responses for each question (item) are utilized to evaluate test performance. One of the important methods in Item Analysis is Item discrimination<\/strong> which refers to the power of a question in differentiating between different learners. Question Discrimination Factor (QDF)<\/strong> is an index that measures how well a question can differentiate between different user cohorts. It depicts how top scorers are more likely to get a question correct than low scorers.<\/p>\n\n\n\n

To compute the QDF of questions, Embibe has used traditional statistical (Item Point Biserial Correlation) <\/strong>and deep learning<\/strong>-based methods. Item Point Biserial Correlation is basically Pearson-product moment correlation<\/strong> between question score and total score for a student. So higher the difference between total scores of students who got the question correct and those who got the question incorrect higher will be the QDF value. We also implemented the 2PL<\/strong> model from classical Item Response Theory<\/strong> using Deep Neural Network<\/strong> architecture. Given the students’ attempts data, we derive the question’s difficulty<\/strong> level<\/strong> and discrimination<\/strong> factor<\/strong> from weights of trained DNN. Here is an example of how the value of QDF varies with question attempt interactions of learners.
<\/p>\n\n\n\n\n \n \n \n \n \n
QDF = 0.11<\/th>\n QDF = 0.80<\/th>\n <\/tr>\n
Question
\n 1:Alcohols of low molecular weight are
\n a. Soluble in all solvents (Correct Option)
\n b. Soluble in water
c. Insoluble in all solvents
\n d. Soluble in water on heating<\/td>\n 		Question
\n 2:Aspirin is also known as
\n a. Acetyl salicylic acid (Correct Option)
\n b. Methyl salicylic acid
\n c. Acetyl salicylate
d. Methyl salicylate<\/td>\n <\/tr>\n
\n <\/td>\n \n <\/td>\n <\/tr>\n
Table 1: Comparison between the distribution of total marks for correct and incorrect questions with low\n QDF and high QDF values.<\/td>\n <\/tr>\n <\/tbody>\n<\/table>\n\n\n\n

Here: the x-axis<\/strong> represents the total marks<\/strong> scored,<\/strong> and the y-axis<\/strong> represents the normalized number of students<\/strong>. The yellow line<\/strong> denotes the distribution of total marks of students who got the question incorrect<\/strong>. The blue line <\/strong>denotes the distribution of total marks of students who got the question correct. <\/strong>In Question-1, there is a high overlap between the total marks of students who got the question correct, while in Question-2, the overlap is very less and hence the value of QDF is higher for Question-2 than Question-1. The final QDF value is the fine-tuned result of the above method and test parameters. <\/p>\n\n\n\n

Embibe<\/strong> conducted a validation experiment<\/strong> to compare the performance of students in two different tests : <\/p>\n\n\n\n

    \n
  1. Baseline Policy<\/em>: Here, questions are selected without bias due to discrimination factors<\/strong> from the ground truth database, ensuring an expected distribution over difficulty levels and syllabus coverage.<\/li>\n\n\n\n
  2. Discrimination Only Policy<\/em>:  Here, questions were selected from the ground truth dataset, ensuring syllabus coverage (at least one question from each chapter) and ensuring that the overall discrimination factor of the questions is maximized at any difficulty level.<\/li>\n<\/ol>\n\n\n\n

    For the experiment, a total of 312<\/strong> students were selected to take a test containing 75<\/strong> questions. Two statistical metrics compared the performances of the test:<\/p>\n\n\n\n

      \n
    1. Evaluation using RMSE<\/strong>: Using the IRT model, we predict each student’s probability in the evaluation set answering the questions correctly and compute the average ability from the scores of the students if they were to attempt the generated test paper. We also determine the ground truth ability of each student from the IRT model. Finally, we compute the root mean squared error (RMSE) between the ground truth ability and inferred ability to measure the accuracy.<\/li>\n\n\n\n
    2. Evaluation using Spearman\u2019s \u03c1<\/strong>: Here, we sort students’ abilities obtained from the ground truth data and the generated test and determine the rank correlation \u03c1 between the two ranks.<\/li>\n<\/ol>\n\n\n\n\n\n\n\n\n\n\n
      Policy<\/th>\nRMSE<\/th>\nRank corr p<\/th>\n<\/tr>\n
      Baseline Policy<\/td>\n0.844<\/td>\n0.59<\/td>\n<\/tr>\n
      Discrimination Only Policy<\/td>\n0.549<\/td>\n0.83<\/td>\n<\/tr>\n
      Table 2: Comparison of RMSE (inferred ability and ability from ground truth) and rank correlation \u03c1 in tests generated by different policies Policies RMSE Rank corr \u03c1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n

      Also, we found that the Discrimination Only Policy test gives a 24.8%<\/strong> better spread of scores (score at 95th<\/strong> percentile of students – score at 5th<\/strong> percentile) than the Baseline Policy test.<\/p>\n\n\n\n

      Hence, the use of high QDF questions in tests improves the quality<\/strong> of the test in terms of its power<\/strong> to differentiate<\/strong> among students under the same targeted learning goals. Also, these are leveraged to improve content quality where we identify questions with negative QDF and improve their relevance<\/strong> and clarity<\/strong>.<\/p>\n\n\n\n

      References<\/h4>\n\n\n\n
        \n
      • Soma Dhavala, Chirag Bhatia, Joy Bose, Keyur Faldu, Aditi Avasthi, \u201cAuto Generation of Diagnostic Assessments and their Quality Evaluation,\u201d July 2020, EDM.<\/li>\n\n\n\n
      • Vincent LeBlanc, Michael A. A. Cox, \u201cInterpretation of the point-biserial correlation coefficient in the context of a school examination,\u201d January 2017, The Quantitative Methods for Psychology 13(1):46-56<\/li>\n\n\n\n
      • Linden, W. D., and R. Hambleton. \u201cHandbook of Modern Item Response Theory.\u201d (1997), Biometrics<\/em> 54:1680<\/li>\n<\/ul>\n","protected":false},"template":"","yoast_head":"\nAn Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"An Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform\" \/>\n<meta property=\"og:description\" content=\"Tests are the most preferred assessment technique used by learners to measure performance against the targeted learning outcomes. So tests must be fair and effective to identify students\u2019 learning gaps and boost students’ learning. The ability of a test to meet these goals is an aggregation of how relevant each question of the test is. Thus, the reliability of a test can be increased by item analysis, where students’ responses for each question (item) are utilized to evaluate test performance.....\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/\" \/>\n<meta property=\"og:site_name\" content=\"EMBIBE - The most powerful AI-powered learning platform\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-02T06:42:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/indicmicrosites-assets.embibe.com\/in-en\/wp-content\/uploads\/2023\/01\/09111018\/22646cb3-7310-4cdb-b785-a83990d42595.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/\",\"url\":\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/\",\"name\":\"An Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform\",\"isPartOf\":{\"@id\":\"https:\/\/www.embibe.com\/in-ml\/#website\"},\"datePublished\":\"2023-02-02T06:40:16+00:00\",\"dateModified\":\"2023-02-02T06:42:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.embibe.com\/in-ml\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"An Article on Approaches to QDF\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.embibe.com\/in-ml\/#website\",\"url\":\"https:\/\/www.embibe.com\/in-ml\/\",\"name\":\"EMBIBE - The most powerful AI-powered learning platform\",\"description\":\"Just another WordPress site\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.embibe.com\/in-ml\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"An Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/","og_locale":"en_US","og_type":"article","og_title":"An Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform","og_description":"Tests are the most preferred assessment technique used by learners to measure performance against the targeted learning outcomes. So tests must be fair and effective to identify students\u2019 learning gaps and boost students’ learning. The ability of a test to meet these goals is an aggregation of how relevant each question of the test is. Thus, the reliability of a test can be increased by item analysis, where students’ responses for each question (item) are utilized to evaluate test performance.....","og_url":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/","og_site_name":"EMBIBE - The most powerful AI-powered learning platform","article_modified_time":"2023-02-02T06:42:04+00:00","og_image":[{"url":"https:\/\/indicmicrosites-assets.embibe.com\/in-en\/wp-content\/uploads\/2023\/01\/09111018\/22646cb3-7310-4cdb-b785-a83990d42595.png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/","url":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/","name":"An Article on Approaches to QDF - EMBIBE - The most powerful AI-powered learning platform","isPartOf":{"@id":"https:\/\/www.embibe.com\/in-ml\/#website"},"datePublished":"2023-02-02T06:40:16+00:00","dateModified":"2023-02-02T06:42:04+00:00","breadcrumb":{"@id":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.embibe.com\/in-ml\/joinus\/an-article-on-approaches-to-qdf\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.embibe.com\/in-ml\/"},{"@type":"ListItem","position":2,"name":"An Article on Approaches to QDF"}]},{"@type":"WebSite","@id":"https:\/\/www.embibe.com\/in-ml\/#website","url":"https:\/\/www.embibe.com\/in-ml\/","name":"EMBIBE - The most powerful AI-powered learning platform","description":"Just another WordPress site","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.embibe.com\/in-ml\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/www.embibe.com\/in-ml\/wp-json\/wp\/v2\/positions\/1705"}],"collection":[{"href":"https:\/\/www.embibe.com\/in-ml\/wp-json\/wp\/v2\/positions"}],"about":[{"href":"https:\/\/www.embibe.com\/in-ml\/wp-json\/wp\/v2\/types\/positions"}],"wp:attachment":[{"href":"https:\/\/www.embibe.com\/in-ml\/wp-json\/wp\/v2\/media?parent=1705"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}