2017 IEEE 9th International Conference on Engineering Education (ICEED), Kanazawa, Japan, November 9-10, 2017

Assessment on Course Outcome Performance Using

Rasch Measurement Model
Azianti Ismail, Liyana Roslan
Ahmad Naufal Adnan
Faculty of Mechanical Engineering,
UiTM Johor, Pasir Gudang Campus
Johor, Malaysia
[email protected]

Abstract— Measurements on students’ level of understanding knowledge within. Nonetheless, measurement of COs can
in the Mechanical Engineering program fundamental courses is provide information related to the student’s level of knowledge
crucial by determining the level of students’ performance prior and skill to assist lecturers in deciding relevant teaching and
on proceeding to advanced courses. The level of students’ learning method for the course. The student’s ability in
understanding can be identified from the assessment of the achieving the COs can indicate how far the students can
course outcomes (CO) as it used to indicate the expected implement the required skills and also to measure the
knowledge that the students supposed to acquire at the end of the effectiveness of teaching [1].
semester, and it shall be outlined according to the Bloom’s
Taxonomy. Therefore, this paper presents the level of students’ Statics is a fundamental subject or concept in the
competency of the Statics course in a local Malaysian university. mechanical engineering program and normally offered in the
The Statics course was chosen as it is a core course for the first- first year of the program. This subject is a pre-requisite to other
year students in this program. The measurement of students’ core subjects such as Dynamics, Mechanics of Material and
understanding level was developed through Rasch measurement Mechanical Engineering Design. An investigation on the
model. The results obtained were assessed against the course program’s performance revealed that Statics is the main reason
outcomes (CO) maps for consistency. The study showed that the students failed to graduate on time due to the poor
Rasch measurement model can classify grades into course performance and it contributes to the pre-requisite subjects. A
outcomes (CO) more accurately and can segregate the students
deeper understanding of key concepts in Statics is needed in
based on their understanding level. The results from this
the follow-on subjects [2]. Hence, it is crucial to find a
measurement can provide useful information regarding the
students’ capabilities in certain fundamental concepts that will be
comprehensive approach to overcome the problem of poor
used in the other pre-requisite or co-requisite courses such as performance. This can be started with investigating the CO’s
Dynamics and Mechanics of Materials. performance that will be narrowed down to a specific chapter
and concept in Statics. The finding is useful for the lecturers to
Keywords— Course Outcome; Rasch Model; performance make improvement in understanding of that particular concept
measurement or overall teaching and learning method.
Nowadays, Rasch measurement model has been widely
I. INTRODUCTION applied in examining and validating psychometric proportion
Since 2004, implementation of Outcome-Based Education of measurement instrument [3] [4] [5] [6]. Rasch model is
(OBE) by Engineering Accreditation Council (EAC) has found to be suitable to examine and validate item difficulty
shifted the direction of measurement dan assessment in (COs) and persons’ (students) competency in response to their
teaching and learning as continuous quality improvement [1]. grades for the course.
By mapping of Program Outcomes (PO) and Course Outcomes A graphical output known as Person and Items Distribution
(CO), the OBE approach must always be assessed, measured Map (PIDM) can give a precise overview of the student’s
and monitored through varies type of assessment. achievement on a linear scale of measurement [7]. PIDM is a
OBE is capable of measuring the overall student’s useful tool which gives clarity for fast and easy decision
competency in which can be demonstrated in various skills making [8]. Instead of heavily rely on traditional test method
such communication and teamwork (presentation and report using grade raw score, Rasch model stands on the concept that
writing), synthesis (case study analysis), practical skills (lab probability which offers better measurement method construct
work and workshop), etc. These skills are specified clearly in in education longitudinal research.
the COs that are divided to several selected assessment The aim of this study is to develop a Rasch measurement
methods. Then, the assessment is constructed based on scale to identify the students’ competency in each CO. The
Bloom’s Taxonomy generic skills (cognitive, psychomotor and analysis of students’ competency and difficulty of the CO can
affective) and assigned with marks. be tabulated in the same scale. Based on the results, proper
The traditional education system based only on the final improvement actions can be planned for the weak students to
grade often fails to measure to other skills which balance the prepare them for the advanced courses. Hence, the information

978-1-4673-6956-5/17/$31.00 ©2017 IEEE 110

 2017 IEEE 9th International Conference on Engineering Education (ICEED), Kanazawa, Japan, November 9-10, 2017

can be used as a yardstick for the lecturers to further improve Then, marks for each CO were then assigned per grade
the teaching and learning method for fundamental courses in based on the mark distribution (Table IV).
this program.
TABLE IV. GRADE’S CATEGORY
II. METHOD Category (x = %) Grade
0 ” x < 40 0
A group of 28 students who enrolled in Statics course
40 ” x < 50 1
during the second semester of the 2015/2016 was selected as 50 ” x < 60 2
the samples in the study. The Statics course was chosen as it is 60 ” x < 70 3
a core course for the first-year students in this program. This 70 ” x < 80 4
course is taught in English by local lecturers for the 14-week 80 ” x < 100 5
study period. The course learning outcome (CO) for this course
with the Bloom’s Taxonomy cognitive levels were examined B. Measurement
and showed in Table I. The CO marks based on the grade’s category as shown in
Table V were then used as the input to be processed in the
TABLE I. CO FOR STATICS Rasch Analysis Software (WINSTEP) to obtain the results.
Description for Course Learning Bloom’s Table V shows a sample of 10 students’ marks based on grade.
CO
Outcome Taxonomy
CO1 Explain the basic concepts and principles Comprehension TABLE V. MARKS BASED ON GRADE
in statics (C2)
CO2 Apply the basic principles to solve Application CO Mark According to
statics problems of particles and rigid (C3) Student Grade
bodies CO1 CO2 CO3
CO3 Apply and solve statics problems of Synthesis (C5) EM01 1 0 0
simple engineering structure in a EM02 3 2 1
systematic and logical manner EM03 3 0 1
EM04 1 0 0
EM05 4 1 1
All the questions used in the quizzes, assignments, tests and EM06 4 2 1
final examinations are identified and classified per CO as EM07 4 2 2
shown in Table II. EF04 4 2 3
EF05 3 2 2
EF06 2 3 3
TABLE II. PERCENTAGE DISTRIBUTION ACCORDING TO CO FOR EACH EF07 3 4 2
TYPE OF ASSESSMENTS

Final
Quiz Assignment Test Total
Exam
(5%) (5%) (30%) (100%) III. RESULTS AND DISCUSSION
(60%)
CO1 0.33 0.10 0.15 0.08 12.9
Table VI and VII show the summary statistics for person
CO2 0.67 0.53 0.57 0.52 55.2
CO3 0.00 0.37 0.28 0.40 31.9
and item category for the course. The value of Cronbach alpha
is 0.87 is considered as a good score based on the instrument
A. Sample and Classification scale rating by [9]. It must be above 0.6 to be acceptable. Thus,
For the input data, each evaluation marks for these this validates the model is acceptable.
students (in percentage distribution) for each CO was summed The Meanperson is -1.74 which below the Meanitem =0.00
up and divided by the whole total marks for each CO. Table (threshold value). It means that the students found to be below
III shows a sample of 10 students marks based on marks the expected performance. The negative value shows that the
distribution. students are having problems in fulfilling some of the COs.

TABLE III. MARKS DISTRIBUTION ACCORDING TO CO In table VI, the value for person separation is 2.64 and it
shows that the students’ performance can be categories into
CO Achievement three groups. The students’ ability range is from -5.98 logit to
Student
CO1 CO2 CO3 +3.79 logit with SD of 2.02.
EM01 47 32 39
In table VII, Item summary shows very high reliability of
EM02 64 50 42
0.97 with six separations. It has 5.26 logit with SD of 1.71. It
EM03 68 39 48
EM04 48 21 32 shows the COs difficulty level is spread over a good range. In
EM05 70 42 44 order to confirm for fit and misfit persons or items, these
EM06 74 57 48 criteria must be fulfilled:
EM07 74 53 52
EF04 78 58 64 Outfit Mean Square 0.5<y<1.5 and Outfit Z-Standard -
EF05 68 56 52 2.0<Z<+2.0 [9]. From Table VI and VII, Outfit Mean Square
EF06 55 65 67 person and item, both are 0.89. Outfit Z-Standard for Person
EF07 65 79 58 and Item both are -0.2. Since all the values have fulfilled the

111
 2017 IEEE 9th International Conference on Engineering Education (ICEED), Kanazawa, Japan, November 9-10, 2017

criteria, it can be concluded that the data do fit the Rasch model parts to answer. This CO2 covers basic principles to solve
reasonably well. statics problems of particles and rigid bodies. This topic will
be learned further in advanced courses such as Dynamics and
TABLE VI. SUMMARY STATISTIC: PERSON Mechanics of Materials.

Five students can score for all COs without any problem.
This group of students can be appointed as tutors in the peer
tutoring program to assist the weak students. Lecturers also
need to revise the questions set for CO2. Some alteration in
the sentences used in the questions can help the students to
comprehend better.

Lecture on related topics to CO2 can be accessed easily

and continuously via online as for student to have ample time
to digest the concepts and theories. Based on spaced repetition
learning theory, spaced practice enhances memory, problem-
solving, and transfer of learning to new contexts. Spaced
practice offers great potential for improving students’
understanding of difficult topics [10].

CO1 is considered as an easy item and it is measured in the

quizzes. Students were given a quiz at the end of each chapter.
More than 78% of students can answer the quizzes without
any difficulty for CO1. Six students cannot fulfill the easiest
TABLE VII. SUMMARY STATISTIC: ITEM
CO1. These students need to be given special attention by
providing extra help in understanding the course. Two
students can achieve all COs outstandingly. EF02 is
considered as the best student for this course since she is at the
top of the class. Even though this course is male-dominant,
she has effortlessly outshined the others.

IV. CONCLUSIONS

The application of Rasch Model enables the identification

of the students’ competency in next advanced such as
Dynamics and Mechanics of Materials, whom may need
special attention and extra helping hands. Rasch analysis can
assist lecturers to monitor the ability of each student in
achieving each CO.
The results reveal how well each CO is achieved by each
student. Lecturers are encouraged to develop more effective
teaching and learning process such more interactive and
creative approach through the massive online open course
Person-Item Distribution Map (PIDM) is shown in Fig.1. It (MOOC). Moreover, MOOC can help the students to access
represents the findings by showing along vertical ruler the materials of the course easily and repeatedly. It may help
measured in logit the distribution of the persons and items. In the students to have extra time to understand difficult concepts
Fig. 1, the achievement of each student and CO is tabulated. and theories in the course. Students need to understand the
On the left side of PIDM, the persons are arranged from the concepts and theories beforehand to ensure they can answer
bottom (weak performance) to top (excellent performance). questions for CO2.
For items on the right side of PIDM, the ordering goes
from the bottom which is easiest to difficult. 21 out of 28 By using Rasch model to measure the actual performance
students (75%) were found to be below Meanitem while the rest of the CO, improvement plans to assist and support weak
is above average (25%). Most difficult item is CO2 in which is students can be taken such as peer tutoring program and video
tested in quizzes, assignments and final examination as in C3 of lectures available online.
level (application). Students need to understand the real-life
scenario type of question and to separate the question into few

112
 2017 IEEE 9th International Conference on Engineering Education (ICEED), Kanazawa, Japan, November 9-10, 2017

REFERENCES

[1] K. Ahmad, N. Mohamad Ali and S. Zainudin, "An Improved Course

Assessment Measurement for Analyzing Learning Outcomes
Performance Using Rasch Model," Procedia - Social and Behavioral
Sciences, vol. 18, pp. 442-449, 2011.
[2] H. Haron and A. Mohd. Shaharoun, “The pedagogical issues in
engineering statics”, in International Conference on Engineering and
Education in the 21St Century (ICEE 2009), 2009.
[3] N. Che Musa, Z. Mahmud and N. Md Ghani, "Application of Rasch
Measurement Model in Validation and Analysis of Measurement
Instruments in Statistical Education Research," Recent Researches in
Education, pp. 123-129, 2011.
[4] N. Hamzah, M. Khoiry, S. Osman, R. Hamid, O. Jaafar and I. Arshad,
"Poor Student Performance in Project Management Course: Was the
Exam Wrongly Set?," Recent Researches in Education, pp. 154-160,
2011.
[5] S. Osman, M. Khoiry, W. Badaruzzaman and A. Mutalib, "Measurement
of students' understanding in final examination of Statics and Dynamics
Course using Rasch Measurement Model," in IEEE International
Conference onTeaching, Assessment and Learning for Engineering
(TALE), 2013.
[6] S. Osman, S. Naam, O. Jaafar, W. Badaruzzaman and R. Rahmat,
"Application of Rasch model in measuring students’ performance in
Civil Engineering Design II Course," Procedia-Social and Behavioral
Sciences, vol. 56, pp. 59-66, 2012.
[7] Saidfudin, M., Azrilah, A.A., Rodzo’an, N. A., Omar, M.Z., Zaharim, A.
& Basri, H., “Easier Learning Outcomes Analysis using Rasch Model in
Engineering Education Research,” EDUCATION'10 Proceedings of the
7th WSEAS International Conference on Engineering Education, pp.
442-447, 2010.
[8] M. Saidfudin, A. Azrilah, N. Rodzo'An, M. Omar, A. Zaharim and H.
Basri, "Use of Rasch analysis to measure students performance in
engineering education," in 7th WSEAS international conference on
Engineering education, 2010.
[9] W. Fisher, "Fisher, W. P. "Rating scale instrument quality criteria,"
Fig. 1. Person-Item Distribution Map (PIDM) Rasch Measurement Transactions, vol. 21, no. 1, p. 1095, 2007.
[10] S. Kang, "Spaced Repetition Promotes Efficient and Effective Learning:
Policy Implications for Instruction," Policy Insights from the Behavioral
ACKNOWLEDGMENT and Brain Sciences, vol. 3, no. 1, pp. 12-19, 2016.
The authors would like to thank the Ministry of Higher
Education for funding this research 600-RMI/RAGS5/3
(5/2015).

113