Evaluating the Effectiveness of a Faculty Development Program in Enhancing the Quality of Multiple-Choice Questions: A Quantitative Study
DOI:
https://doi.org/10.47489/szmc.v40i2.876Keywords:
Faculty Development, Multiple Choice Questions, Program Evaulation, Assessment, PsychometricsAbstract
Background: Multiple choice questions (MCQs) are widely used in medical education for their objectivity and efficiency. However, poorly constructed questions can compromise the validity and reliability of the assessments. Faculty development programs have emerged as a key strategy to improve the quality of MCQs.
Objective: This study aims to evaluate the impact of structured FDPs on the MCQs construction using objective metrics and participants’ feedback.
Methods: This observational study was conducted over three years period (2022 - 2024) at University College of Medicine & Dentistry. Three workshops on MCQs development were organized for the faculty. Feedback was collected through structured forms using 5- point Likert scale. Additionally, MCQs submitted before and after the workshops were analyzed for item flaws, difficulty & discrimination indices. The study also evaluated the impact of training using the Kirkpatrick Model. The data was computed and analyzed using SPSS version 26.0. The Shapiro Wilk test determined the normality of the data. P value <0.05 was considered as significant.
Result: A significant reduction in test-wiseness flaws (logical cues from 12.9% to 1.3%, p<0.001) and irrelevant difficulty (extraneous details from 7.6% to 0.9%, p<0.001) was observed. The proportion of moderately difficult items increased from 56.9% to 62.2%, while the proportion of very good discriminating items increased from 2.2% to 6.7%; however, neither change reached statistical significance (p=0.062 and p=0.098, respectively). Feedback from the participants revealed increasing satisfaction across workshops, with the highest ratings in workshop 3 (2024). Instructor expertise remained consistently rated above 87.5%.
Conclusion: Incorporation of multiple structured faculty development workshops lead to measurable improvements in MCQ quality. These training sessions can enhance the reliability and validity of medical assessments, benefiting both educators and learners.
References
Beg MA, Tabassum T, Ali S. Role of faculty development workshop for improving MCQS quality in basic medical sciences.Biomedica – Vol. 37, Issue 1, 2021. http://doi.org/10.51441/BioMedica/5170
Ansari M, Sadaf R, Akbar A, Rehman S, Chaudhry ZR, Shakir S. Assessment of distractor efficiency of MCQS in item analysis. The Professional Medical Journal. 2022 Apr 30;29(05):730-4. https://doi.org/10.29309/TPMJ/2022.29.05.6955
Agarwal A, Khattak O, Esmaeel SE, Fahmy E, Bayomy NA, Mehmood SI, Mohamed H, Bawadekji A, Alotibi FA, Alanazi MS, Younes A. Efficacy of Faculty Development Training Workshops (FDTWs) on Writing High-Quality Multiple-Choice Questions at Northern Border University (NBU) in the Kingdom of Saudi Arabia (KSA). Cureus. 2024 Jun 18;16(6).DOI: 10.7759/cureus.62607
Mashood S. Quality Assurance of Multiple-Choice Questions Test Through Item Analysis. Life and Science. 2023 Oct 5;4(4):7-Life & Science 2023 Vol. 4, No. 4.Doi: http://doi.org/10.37185/LnS.1.1.315
Baste VS. Item analysis of MCQs in physiology and its correlation with faculty’s perception of difficulty level of MCQs. National Journal of Physiology, Pharmacy and Pharmacology. 2023 Jun 30;13(7):1444-.DOI: 10.5455/njppp.2023.13.12601202220122022
Kiran F, Ayub R, Rauf A, Qamar K. Evaluating the impact of faculty development programme initiative: are we really improving skills in MCQ writing. J Pak Med Assoc. 2021 Oct 1;71(10):2434-8.DOI: https://doi.org/10.47391/JPMA.1207
Chauhan A, Khaliq F, Nayak KR. Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education. 2025 Apr 7:1-30. https://doi.org/10.1007/s40593-025-00471-z
Ali R, Sultan AS, Zahid N. Evaluating the effectiveness of'MCQ development workshop using cognitive model framework: A pre-post study. JPMA. The Journal of the Pakistan Medical Association. 2021;71(1 (A)):119.DOI: https://doi.org/10.47391/JPMA.1068
Rezigalla AA, Eleragi AM, Elhussein AB, Alfaifi J, ALGhamdi MA, Al Ameer AY, Yahia AI, Mohammed OA, Adam MI. Item analysis: the impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items. BMC Medical Education. 2024 Apr 24;24(1):445. https://doi.org/10.1186/s12909-024-05433-y
Rauf AR, Sultana S. Effect of faculty training on quality of Multiple Choice Questions. Rawal Medical Journal. 2021 Dec;46(2):430.Available at https://www.rmj.org.pk/index.php?mno=133092
Ahmad S, Imtiaz M, Aziz RS, Manzoor N, Ahmed S, Habib B. Kirkpatrick's Approach to Build Faculty Development Programs in Medical Education. Pakistan Journal of Medical & Health Sciences. 2023 Apr 29;17(03):441-.DOI: https://doi.org/10.53350/pjmhs2023173441
Shallal TM, Alkhateeb NE, Al-Dabbagh A. Virtual faculty development program in bioethics evaluated by Kirkpatrick model: A unique opportunity. PloS one. 2023 Oct 30;18(10):e0293008. https://doi.org/10.1371/journal.pone.0293008
Lee HY, Yune SJ, Lee SY, Im S, Kam BS. The impact of repeated item development training on the prediction of medical faculty members’ item difficulty index. BMC Medical Education. 2024 May 30;24(1):599. https://doi.org/10.1371/journal.pone.0293008
Eleragi AM, Miskeen E, Hussein K, Rezigalla AA, Adam MI, Al-Faifi JA, Alhalafi A, Al Ameer AY, Mohammed OA. Evaluating the multiple-choice questions quality at the College of Medicine, University of Bisha, Saudi Arabia: a three-year experience. BMC Medical Education. 2025 Feb 13;25(1):233. https://doi.org/10.1186/s12909-025-06700-2
This work is licensed under a 