Introduction
Practical examination in preclinical subjects in Physiotherapy has been a challenge. Three independent variables challenges the objectivity in assessment of performance during the traditional practical examination: the student, the examiner and the patient. Of which, the examiner and the patient are the prudent sources of variability which may influence the assessment of students.1 The patient component is controlled by using the students’ peers or healthy support staff of institute as models during practical examinations. This gives a chance to all students to be able to demonstrate requisite skills as taught during the entire academic year on models. The examiner component of variability is taken care by OSPE method of assessment by providing very specific criteria for assigning marks in the form of checklist.1 The OSPE is an excellent solution as non-standardised practical tasks present major problems in achieving objective levels of assessment since the criteria for performance may vary.2
There are many skills which are taught during the regular study but cannot be assessed fully due to lack of time and restricted number of examiners during summative examination. There are 10 skills taught in first year subject of Kinesiotherapy. However, prescribed exam pattern allows only 4 skills to be assessed during the summative assessments.3
To overcome these issues, Objectively Structured Practical Examination (OSPE) has been suggested to be a useful tool to overcome the lacunae in traditional examination pattern. OSPE consists of stations that test various techniques to confirm practical capability. It has been studied and proven to be an effective, valid, reliable nursing4, 5 physiology6 and oral surgery.7 Its use in Physiotherapy has also been documented in literature as introduction of OSPE and comparison of traditional and OSPE method. However these are cross sectional studies conducted at one time point.8, 9, 10
With already proven advantages of OSPE over traditional examination, it is necessary that it becomes a routine part of assessment methods. This article is an attempt to share the analysis of OSPE implementation for 3 years at our institute which is still being continued.
Methodology
It was a longitudinal study conducted between 2021- 2023 and is being continued still, on first year Physiotherapy students in the subject of fundamentals of kinesiology and kinesiotherapy. It was conducted in 2 parts. First part being creating the OSPE stations, with checklist and questionnaire for collecting the perception about examiners and students.
For first step focus group discussion, 4 experts involved in teaching the subject, 1 expert in educational technology and administrator of the institute were invited by the principal investigator. As per the subject content given by the parent university, it was decided to prepare 12 stations for adequate sampling of the content. All stations were based on psychomotor skill with a component of cognition and affective domain included in it. Details of stations were prepared with instructions to students along with the details of the tasks to be performed by the student. The checklists in the form of marks distribution were prepared for each station. Time required for performing each stations and overall total time required for the conduct of examination for one batch of students was calculated. Along with this, separate two questionnaires with 8 close ended and 2 open ended questions each were prepared for recording the perception of students and teachers were prepared.
Pilot testing of the stations, checklists and questionnaires was done on a random 10 students of first year batch who were not due to appear in the university examination. Stations were reported to be appropriate with clear instructions. However, students suggested modifications in 2 questions as the meaning was not clear to them and hence were modified.
After pilot testing, examiners were identified from the available pool of teachers. A briefing was made about the expected behavior of teachers during the examination. All station details were discussed with them, along with instructions to students and marking system.
This exam was conducted 10 days before the preliminary examination. Instructions were given to the students by subject teacher about this new pattern of examination. Orientation about stations was given to the students one day before the exam by actual visit to the examination area. On the day of exam students were assigned to each station by the coordinator. Instructions were available at the entry point at each station. Each student performed the task according to the instructions on the given model infront of the examiner who scored the student according to the available checklist. The time required for the exam was as anticipated and there were relapses. Perception of teachers and examiners was obtained through the questionnaires on the day of examination itself to avoid any recall bias. After 2 days of the exam, all students were asked to gather in the classroom and overall feedback about the performance was given to the students. Individual marks and feedback was given on one to one basis by the subject teacher.
This process was repeated next two years with same set of stations and questionnaires. However, the exam was taken twice in a year in the next two years with half stations in the first term and all stations in the 2nd term. Examiners who are also the teachers expressed greater satisfaction about the exam as it made the task easier to evaluate each student on each skill and hence to give feedback so as to improve the skill. Time of the conduct of examination was found to be suitable as it provided them breathing time and correction time for the final university examination. This led to better marks in the final examination.
Result
40 students participated in 2021, 2022 and 60 participated in 2023, with 10 examiners in each year. 4 experts were invited to the part of Focus group discussion. These were the subject teachers involved in teaching first year students since minimum 5 years and had the experience of conducting the examination as well. Few minor modifications were done in each year in the OSPE stations with the experience of previous year. The number of stations was kept constant. However the content was enhanced for better understanding of students with more clarity and specificity. List of 10 stations is given in (Table 2) prepared according to the content of syllabus.
Table 1
Demographics of examiners
|
N=10 |
2021 |
2022 |
2023 |
|
Age |
24-35 |
24-35 |
22-36 |
|
Gender |
10 Females |
8 Females |
9 Females |
|
2 Males |
1 Male |
||
|
Average Years of experience |
7 |
6 |
8 |
Table 2
OSPE stations
Students and examiners feedback was obtained through structured questionnaire each year. Their response details are given in (Table 3, Table 4). Overall, there was a great satisfaction about the entire process. However, suggestions were also given and the specific ones will be complied during the implementation of next session. For example students had mentioned about lack of clarity for 2 stations and examiners had expressed that more time should be allowed for 2 stations. Accordingly, the modifications were done in the subsequent sessions.
Table 3
Students perception about the examination pattern
Table 4
Examiners and teachers perception about the examination pattern
Discussion
When OSPE was first introduced in our institute in 2021, it was thought to be conducted for only one year. However, when it was noticed by teachers that the biggest advantage of OSPE is objectivity in marks awarded and less time consuming, it was repeated for next years. And now it has become a routine for formative assessments in the subject of Kinesiotherapy for first year and second year of Physiotherapy education.
The consistency in teachers and students perspectives suggests the positive appraoch of these important stakeholders.
The primary purpose of practical examination is evaluation of all psycomotor skills taught in that professional year.2 However the traditional examination pattern includes a sampling of only 3 or 4 skills. Thus, the evaluation is not through 360 degree. Students also work hard through out the year practicing these skills. However at the time of examination gets to select only 3 skills. Thus he/she is not assessed for all skills. OSPE allows assessment of maximum no of skills though the stations. This will help teachers to give precise feedback to the students in the direction of improving their skills.
The checklist helps to assess all steps to be included in performing the task which may help examiner to pinpoint the mistakes but also my result in reduction of marks.9 However the good part is, it becomes uniform for all students as the same checklist is used for all.11
It may be hypothesised that more number of stations helps to compensate poor performance in one skill by better performance in other skills. Thus the students may look forward to these stations. This hypothesis is based on the informal communication with students while discussing their performnace . And hence may be because of this reason students must have opined that it is easier to pass through this method.
It is suggested to be time saving method by teachers. Distributing the content of syllabus among more no of stations with fixed time duration reduces the overall time required for conducting the exam for a specific number of students in one batch. One of the drawback of practical examination as mentioned in literature, that sometimes it turns into an extended theory examination where teacher asking theory based questions. With the specific instructions about the task to be performed along with checklist and next student waiting outside the station, compels every examiner to follow strict time schedule. This allows for equal amont of time for each student’s examintion. Thus there is least chance of getting the examination delayed and hence can be completed in stipulated less time period.
Based on individual score sheet, it is easier for teachers to give feedback to students about the performance which is the purpose of formative assessment. Feedback is very important for stimulating the learning among students. Only care that the teacher should take is to follow the rules of giving feedback.12
There is found to be greater reliability between micro-skills where dichotomous mark allocation is followed. However it is difficult to grade the skill with only 2 options as students may show diverse levels of performance. Hence training of teachers in preparing and applying check list is suggested.13, 14 It was found that examiners with a similar number of years of experience scored students with almost similar scores as evident by better correlation.15 Hence it is suggested to keep the homogeneity in the levels of examiners to get more reliable assessment or specific adequate training be provided to them.
With clear instructions from university regarding the summative examination pattern,3 and more number of students and less number of examiners appointed by university, implementation of OSPE at University level exam may seem difficult. However, each institute may train their own teachers for this mehod of assessment and apply it for formative assessments so that more effective feedback can be given to students for their performance in every skill mentioned in subject syllabus.