Development and Validation of Student Engagement Questionnaire

Introduction: Medical students drop out rate in Pakistan is very high. In our country, only 84% of students graduate, which is significantly lower than the average for developing nations. Class engagement, has been identified as a solution to the failures, dropouts and motivational lacking of students. Being multifaceted construct, the problems of defining engagement have also led to inconsistencies in measurements too.Most of the previous student engagement questionnaires had lack of information regarding their reliability and validity, transparency and auditability. They were long and complex to understand and only measures the behavioral and social engagement. Sothere is a need to develop questionnaires that is short, freely available and can easily be understood by all stakeholders involved in teaching and learning. Aims & Objectives: To develop a questionnaire to assess the engagement of undergraduate medical students during classroom activities . Place and Duration of Study: This was mixed method study, conducted at the University College of Medicine and Dentistry, University of Lahore between November 2022 &May 2023under the guidelines of AMEE Guide 87. Material & Methods: After receiving consent, the first draft of the questionnaire was distributed to 14 clinical and basic science subject experts, as well as medical educators, for qualitative and content validation. To determine the response process and to address any misunderstandings regarding the significance of questionnaire items, cognitive interviews with six students were conducted. Responses from 210 undergraduate medical and dentistry students were used to gauge reliability.SPSS version 23 was used to evaluate the questionnaires, a p-value of (cid:148) 0.05 was considered significant For the scale, an internal consistency of between 0.50 and 0.70 according to Cronbach's alpha was deemed sufficient Results: There were 14 experts who gave qualitative expert validation. The initial student engagement questionnaire contained 49 items to measure 5 theoretical constructs, and were modified to make them more comprehensible and applicable.The questionnaire’s overall scale validity index was 0.84. After the cognitive interviews, there were just twenty items left. Content validation was finished in two rounds with 20 final items, yielding acceptable values of SI-CVI 0.83 & CCA 2.8. After the questionnaire had undergone pilot testing, 210 students filled it out, and the reliability of the survey was evaluated using Cronbach's alpha, which came out to be 0.721. Conclusion: Medical colleges can use this valid and reliable student engagement questionnaire to assess students' emotional, behavioral, and cognitive involvement in class.


INTRODUCTION
Dropout rate of medial students in Pakistan is 16%, compared to USA, Australia and in New Zealand, only 84% of Pakistani medical students graduate, which is significantly lower than the average for developing countries 1 .The authors point out that student engagement is a complex concept with many different facets 2 .Since motivated and engaged students are more likely to participate in academic and institutional activities, they contend that engagement is an important component in fostering academic achievement and lowering dropout rates 3 .The definition and assessment of student engagement have not been agreed upon in the literature, according to the authors 4 .They contend that although university attendance is obligatory, a sense of responsibility and motivation towards learning cannot be commanded, a standard instrument for gauging student engagement in the classroom is lacking.The previously created engagement measures are either ambiguous regarding the engagement aspects they are examining, too long, or neither of those 5 .As a

INTRODUCTION
Globally, acute generalized peritonitis ranks among the top surgical emergencies 1 .It is more common in Third World nations.The prevalence of perforation is low (0.6% -4.9%) in developed nations but high (33% -63%) in West Africa 2 .554 persons were discovered to have peritonitis in a study that took place over three years in India 3 .Researchers in Pakistan have conducted studies with similar methods, with one study reporting 650 cases in a just 9 months 4 .Most cases of peritonitis are caused by a gastrointestinal perforation or anastomotic leak 5 .In the case of peritonitis, anaerobes and gramnegative organisms are mostly responsible for sepsis and morbidity due to the overactive inflammatory cascade brought on by the release of endotoxins 5 .Clinical evidence is used to identify peritonitis.Diagnosis can be achieved via upright plain x-ray of the abdomen, USG, or CT scan.This is often done through diagnostic laparoscopy nowadays 6 .Resuscitation, diagnosis, prompt exploration, treatment of the underlying cause, and extensive surgical peritoneal lavage have always been the cornerstones of peritonitis therapy regimens (IOPL) 7,8 .Regular IOPL is performed to lessen bacterial contamination and burden.Even though large volumes of normal saline are used in IOPL, the rates of sepsis, wound infection, and mortality remain alarmingly high.Another method result, all stakeholders involved in teaching and learning need a questionnaire that is simple to understand and apply 6 .The goal of this study was to create a brief, easily understood, and publicly accessible questionnaire that gauges undergraduate medical students' cognitive, behavioral, and emotional participation in the classroom.The following research questions are addressed in this study based on subject experts (Expert validation): Q1:Which questions in the student engagement questionnaire should be used to gauge undergraduate students' participation in classroom and clinical activities?Q2: What are the questionnaire's content validity?Q3: How do students infer the survey's items (validity of the response process)?Q4: How reliable is the questionnaire?

MATERIAL AND METHODS
It was a mixed method study 7 , conducted at the University College of Medicine and Dentistry, University of Lahore from November 2022 till May 2023 after receiving approval from "Ethical Review Board" under Reference: ERC/07/23/01.Phase 1: Development of questionnaire, Study participants, materials and methods: In this phase, answering our first question, we conducted extensive literature review and two focus groups discussions, separately with students and teachers having 6 participants in each group 8 .Transcripts of the focus group discussion were transcribed by a qualitative software calledATLAS.tiand analyzed for the themes and subthemes 8 .Items were revised and refined by the authors and questionnaire version 1 was developed.Phases of the study are shown in Fig- 1 below.Phase 2: Expert validation, Study participants, materials and methods: Out of 14 basic and clinical science experts with additional degree of medical education were invited using purposive sampling 9 to rank the items for grammatical errors, duplication, clarity and content relevance , all of them (100%) participated.Likert scales were included in the questionnaire to gauge the items' relevance and clarity.We utilized a scale for relevance, with 4 denoting a lack of relevance and 1 denoting a high level of relevance.We used the following scale for clarity: 3 for highly clear, 2 for item that need modification, and 1 for unclear.Data Analysis: The content validity index (CVI) for the quantitative component was calculated for each individual item (I-CVI) and for the scale (S-CVI) based on the expert ratings 10 .
The percentage of experts who concurred across all items was used to determine the I-CVI, and the average of all CVI values across all items was used to calculate the S-CVI.While relevance ratings of 3 or 4 were recoded as 1, item rankings of 1 or 2 were changed to 0. The 1s for each item were added up and divided by the total number of experts to produce the I-CVI.Using a 3-point Likert scale, the average content clarity was found to improve the clarity of the items.Each item's average clarity was calculated by adding up all of the values assigned to it, then dividing that amount by the number of items.To be considered highly clear, a clarity of 2.4 (80%) or greater was required 10 .Based on the comments, the items were modified, grammatical errors were corrected and version2 of the questionnaire having 29 items was developed.Phase 3: Demonstrating the validity of questionnaire, Study participants, materials and methods: Cognitive interviews with six students (four from 4th year MBBS and two from 2nd year BDS) were conducted to demonstrate response process.'Think aloud' and'verbal probing' approaches were combined.The participants were instructed to read the item aloud after reading it silently to the group 11 .Following the participant's reading of an item, we used scripted and unscripted questions in verbal probing.The cognitive interview took place for about 50 minutes.Another researcher was also present during the session in order to strengthen the validity of the interview technique and minimize bias.Data analysis: After considering the comments, interpretations, suggestions and ideas of the students, we came up with third version of questionnaire (having 25 items).We made the decision to have version 3's items' relevance and clarity verified by experts 12 .Content validity of the questionnaire version 3 was established again.A request to give feedback on relevance and clarity wassent again via google forms to 14 experts who submitted feedback in Phase 2 earlier.Out of 14 participants, 10 (80%) responded.The filled google forms sent by all participants were complete and included in the study.Phase 4: Demonstrating the reliability of the questionnaire, Study participants, materials and methods:Based on reports from the literature that an acceptable sample size (minimum of 10 participants each item) was 210 students (2nd & 4th year MBBS, 2nd year BDS) for 20 items. 13.A 5point Likert scale was used to grade the responses: 1 denoting strongly disagree, 2 denoting somewhat disagree, 3 denoting neither agree nor disagree, 4 denoting somewhat agree, and 5 denoting strongly agree.The items were mixed up so that none could be categorized according to the proposed structures.Data analysis: Internal consistency was evaluated using Cronbach's Alpha 14 SPSS version 23 was used to evaluate the questionnaires.For the scale, an internal consistency of between 0.50 and 0.70 according to Cronbach's alpha was deemed sufficient 14 .

DISCUSSION
The study's primary goal was to create a valid and reliable questionnaire that can assess student engagement.This questionnaire will be used by all stakeholders involved in learning and teaching, including teachers, students, institutional administrators, policymakers, and curriculum specialists, to gather accurate data on where students' engagement levels in the classroom may need improvement. 15.The majority of the self-reporting questionnaires created between 2006 and 2020 were designed to gauge participants' level of cognitive and behavioral engagement 6 .The cognitive component is covered by the Situational Cognitive Engagement Questionnaire (SCEQ) 16 , the Survey of Student Engagement 17 , and the Student Engagement Instrument, or SEI 18 .To gauge the behavioral elements, the Student Engagement Scale (SES) 19 and Student Self-Report of Engagement (SSRE) 20 were established.According to previous studies, there was a paucity of information regarding their validity, dependability, and auditability 6 .Few were lengthy, difficult to comprehend, and expensive, including the Higher Education Student Engagement Scale (HESES) 21 , AUSSE 22 , which caused student bias because students could'ntcomprehendit properly 15 and many items lacked theoretical justification and had no predictive validity 6 .In many of the measures, the context of the engagement, for example university or classroom remained unspecified 23 .This instrument's content validation demonstrates its dependability and internal consistency is comparable with alpha values from earlier research on engagement. 6ur study's thorough approach to generating the questionnaires in accordance with the procedures and criteria described in the literature was one of its key strengths.Cronbach's alpha analysis of internal consistency revealed a high level of internal consistency for the entire scale of 0.721 13 .The study had a few drawbacks.We recruited people at a ratio of 1:10, which is thought to be adequate to good for the sample size 14 .Construct validity of the tool is lacking.

CONCLUSION
This study has produced a valid, reliable, userfriendly, brief student engagement questionnaire that is free to the public domain and can be used to gauge undergraduate medical students' participation in the classroom.

Fig- 1 : 1 :
Fig-1: Phases of the study showing development & validation of student engagement questionnaire.

Table - 1: Modifications done in different versions of student engagement questionnaire. Phase 3: Demonstrating the validity of the questionnaire
Table-1.

Demonstrating the reliability of the questionnaire
The final questionnaire with 20 items measuring 3 constructs along with the Cronbach's alpha rating as shown inTable-2.