Running head: CASE MANAGEMENT MODULE EVALUATION 1

Case Management Module Evaluation

Nuong Nguyen, Diego Barraza

California State University, Monterey Bay

Assessment and Evaluation

Dr. Bude Su

July 25, 2017

Table of Contents
I. Introduction ............................................................................................................................. 3
II. Metholodogy .......................................................................................................................... 3
a. Prototype ............................................................................................................................. 3
b. Learners ............................................................................................................................... 5
c. Tryout Process ..................................................................................................................... 6
1) Survey ......................................................................................................................... 6
2) Pre-Test ....................................................................................................................... 6
3) Learning Module ......................................................................................................... 7
4) Post-Test ..................................................................................................................... 7
5) Questionnaire .............................................................................................................. 7
6) Direct Observation ...................................................................................................... 7
d. Tryout Conditions ............................................................................................................... 8
III. Results .................................................................................................................................. 8
a. Entry Conditions .................................................................................................................. 9
b. Instruction ........................................................................................................................... 9
c. Outcomes ............................................................................................................................. 9
1) Pre-Test and Post-Test Results .................................................................................... 9
2) Questionnaire Results ................................................................................................ 11
3) T-Test Analysis ......................................................................................................... 15
d. Recommendations.............................................................................................................. 15
IV. Summary............................................................................................................................. 16
V. Appendices ........................................................................................................................... 16
A. Survey Questions .............................................................................................................. 16
B. Pre-Test and Post-Test ...................................................................................................... 17
C. Questionnaire Results ........................................................................................................ 19
D. Observation Checklist ....................................................................................................... 21
E. Learning Module Link ...................................................................................................... 23
F. Pre-Test and Post-Test Results........................................................................................... 23
G. Descriptive Statistics of Pre-Test and Post-Test................................................................. 24
CASE MANAGEMENT MODULE EVALUATION 3

Section I
Introduction

The Case Management learning module was created as part of a capstone project. The

scope of this module was to cover a ten-minute lesson that explained the process of case

management. The module was created with the objective of having learners go through the

phases of case management before moving on with the rest of the capstone project, which

consists of creating new objectives on SIRAS Systems. SIRAS Systems is a county software

used by the Special Education department staff to submit and manage paperwork, like students

individualized education plan (IEP).

Case management is a process that each staff member takes part in when handling the

caseload. This has become an issue as newly hired staff do not have experience handling client

cases. This module aims to introduce a quick but effective way to approach this issue, which is

currently being handle with a face-to-face approach.

The ultimate goal is to have an eLearning module that new staff can access immediately

at the time of hire. The new staff can then apply their knowledge and not have to wait for a

training to take place in person. An important skill for special education staff is to be able to

create new goals within SIRAS Systems. However, it is necessary to know how to manage a case

before doing so.

Section 2

Methodology

a. Prototype

My capstone project is to design an eLearning class that focuses on helping staff

members create new goals for students within SIRAS Systems. The prototype module used for

this evaluation, which is part of a bigger project, teaches the basics on case management. Special
CASE MANAGEMENT MODULE EVALUATION 4

Education department members are expected to set new goals for students on SIRAS Systems;

however, skills on case management are also necessary to accomplish such objective. The

prototype addresses a quick but effective way to introduce case management.

The prototype was created in Captivate and can be accessed through a webpage link on

the computer and mobile device. The duration of the prototype is approximately ten minutes. At

the start of the module, learners are able to see the lesson, practice, and test options. The module

has narration and closed caption. The module is designed so that any first-time user can simply

follow the directions, go through the lesson, take the practice, and finally take an assessment that

will provide them with a pass or fail score at the end.

The prototype was created with the intention of addressing the knowledge gap between

the main topic of the capstone project and what is needed to get there. Case management skills

are needed in order to be able to set new goals into SIRAS

Systems.

Figure 1
CASE MANAGEMENT MODULE EVALUATION 5

The module begins with a straight forward menu where the learner can access the lesson,

practice, and test. The prototype includes audio, narration, and engaging multimedia content. See

examples of prototype slides in figure 1 and 2.

Figure 2

b. Learners

The learners represent a wide age range, occupations, and ethnicities. The Special

Education department is represented by teachers, psychologists, and speech language

pathologists. Everyone in the department is required to set new goals for students into SIRAS

Systems. The learners level of experience in SIRAS Systems varies from inexperienced to

experienced. This module and ultimately this capstone project will target non-experienced

members of the department.

Members education for this evaluation ranges from bachelors to masters degrees. The

average age for the learners in this project is thirty-four years old. The learners are comprised of

five males and five females (see Appendix A for survey data).
CASE MANAGEMENT MODULE EVALUATION 6

Four learners claimed that they have case management experience. The learners represent

a diverse group composed of Hispanics, Asians, and African Americans. There were seven

people with a bachelors degree and three people with a masters degree in our learners group

(see Appendix A).

c. Tryout Process

The tryout process consists of six different parts, which includes the 1) survey, 2) pre-

test, 3) learning module, 4) post-test, 5) questionnaire, and 6) observation. Throughout the

whole process, the evaluator observes and documents the usability and learning process. The ten

learners are instructed at the beginning to complete all five parts in sequential order.

1. Survey: The survey questions were asked in person (Appendix A). The survey

had demographic questions that includes age, gender, ethnicity, education, and

occupation. Additional questions were asked in regards to their interaction with

the case manager and for the extent of time. This helps the evaluators to know the

learners experience with the case management process.

2. Pre-test: Next, the learners took the pre-test. The pre-test helps to assess the

learner's knowledge on the seven phases of case management. The pre-test and

post-test are the same in order to measure how much was gained from the training

(Appendix B). The learners access the pre-test through learning module from the

link or the host computer. The answers were recorded on paper by the

evaluator. There were seven questions altogether with three multiple choice, two

true and false, and two fill-in-the-blanks. The variety of questions help check the

learner's understanding of the case management process and whether they can

recall the information without referring back to the learning module.

3. Learning Module: After the learners took the pre-test, they would click on the

begin button to start the learning module. The directions are given from the

beginning of the module to the end with audio and closed caption. The learners

read all seven slides and then practice what they have learned through a matching

exercise. In order to match the words, they click and drag each word to the

correct definition or description as stated in the audio and closed caption. The

tasks takes on average of ten minutes and depends on how quickly the learners

read and comprehend.

4. Post-Test: Afterwards the learner took the post-test, but they were not allowed to

refer back to the learning module. The post-test is the same as the pre-test and

access through the main menu. For the post-test, the learners would submit their

answers and get immediate feedback on whether they answered them correctly or

not. If the answer was left blank, then an error message would pop up saying

you must answer the question before continuing. At the end of the test, the

results are shown with the percentage answered correctly.

5. Questionnaire: Finally, the learners were asked to fill out the questionnaire on

paper in regards to the whole learning process as well as their confidence level on

applying what they had learned. There are six questions with a rating scale and

two open-ended questions that allow the learners to elaborate on what they like

and dislike about the training. The feedback will be helpful for the purpose of

training improvement.

6. Direct Observation: While each learner completed the training in a quiet setting,

the evaluator would sit beside him or her with an observation checklist
CASE MANAGEMENT MODULE EVALUATION 8

(Appendix D). The ten learners were observed separately to allow for more in-

depth notetaking. The observer noted on the checklist if the learners were

struggling with accessibility, accessing the different components using the main

menu, and navigating through the learning module. Then each item was marked

with a yes or no if it was observed or not. Since the observation checklist

was based solely on what the evaluator sees, for more in-depth questions, the

other form, which is the questionnaire was used to describe the learners

experience. The observation allowed the evaluator to note where the learners

struggled and how the learning modules usability can be improved to enhance

learning.

d. Tryout Conditions

The tryout conditions are similar to the training conditions. The learners were asked to

bring their own laptops or one was provided for them. In the real training settings, the district

will have to decide whether to use the professional development center where the special

education staff will have to bring their laptops, or at a school site where computers are provided

for the staff. For the tryout conditions, a quiet room was used with minimal distractions. The

learners had to access the learning module through a web page link or a host computer with the

module already on the computer. The real training will have the module on a learning

management system. The learners began by answering the survey questions. Then instructions

were given to take the pretest, then complete the learning module and post-test on the

computer. Afterwards, the learners had to answer the questionnaire.

Section 3

Results
CASE MANAGEMENT MODULE EVALUATION 9

a. Entry conditions

The intended and observed were similar. The learners did not have any issues accessing

the module through the link. Some learners also took the module through a host computer where

the elearning module was hosted. Most of the learners had some experience with online and

computer based modules. One challenge for one learner was the font in the practice section of

the module. The learner stated that the font was too small to read. Another challenge for three of

the learners was finding the right time to take the module. Since they all had very busy work

schedules, they had to take the course during the weekend using their personal time. We did

expect some learners to have trouble finding the right time to take this module, however, they

manage to solve this time constrain.

b. Instruction

The intended and observed instruction were consistent with each other. Since the

learning module is self-contained, the directions are included and the learners are able to learn

the content, practice, and take the test independently. However, one learner turned off the audio,

which includes the directions, and as a result, she did not know what to do. Thus, it is important

to also have written directions at the top of each module. The other nine learners did not seem to

be struggling with the instructions and they were able to complete the learning module at a good

pace.

c. Outcomes

1) Pretest and Post-Test Results: Both tests were identical with seven questions. On

the pre-test, only two people received a 71% and the rest scored below. However, on the post-

test all ten learners received a higher score as seen in table 1. Four learners received 71% and six

learner received a score of 86%. The learners only missed one or two questions on the post-
CASE MANAGEMENT MODULE EVALUATION 10

test. The mean for the pre-test was 3.8 with a standard deviation of 0.92 and the mean for the

post-test was 5.6 with a standard deviation of 0.52. Overall, there was an improvement in the

scores from the training.

Pre-Test Post-Test
Scores Scores

4 6

3 5

4 6

2 6

4 5

5 6

4 6

4 5

5 6

3 5

Table 1

The learners struggled most with the fill-in-the blank questions, six and seven, as well as

the multiple choice question number two on the pre-test (see Figure 3). However, after the

training, the learners showed the most improvement on the fill-in-the-blank questions and the

majority of the learners were able to fill in the correct terms. 100% of the learners were able to

answer number seven correctly.

Figure 3

2) Questionnaire Results: The learners felt the instructions for the module was easy or

very easy to follow. 20% of the learners said it was easy, and 80% said it was very easy. The

learners did not have any difficulties that impeded the learning process (see Figure 4). 30%

became familiar and 70% became very familiar with the case management process. The training

had a positive result in teaching the participants more about case management (see Figure 5).

The learners felt more confident about managing a client. 50% of the learners felt

confident and 50% felt very confident about managing a client. This demonstrates that the

learners felt positive about their ability to apply the learned skill (see Figure 6). The learners

thought that the content was relevant to the topic. 10% of the learners said it was relevant and

90% said very relevant. The learning module taught the learners about the case management

process (see Figure 7).

The learners level of frustration varies. 20% felt some frustration and 80% hardly felt

any frustration. One learner mentioned the frustration was due to learning new content, and
CASE MANAGEMENT MODULE EVALUATION 12

another learner mentioned learning on the computer was a new experience for her, which

resulted in some frustration (see Figure 8). The level of difficulty of the quizzes for the learners

varied. 60% thought it was difficult and 40% thought it was easy. Even though the learners

thought it was difficult, the results from the post-test showed that the learners scores improved,

and the instruction was effective (see Figure 9).

Figure 4

Figure 5
CASE MANAGEMENT MODULE EVALUATION 13

Figure 6

Figure 7
CASE MANAGEMENT MODULE EVALUATION 14

Figure 8

Figure 9

From the open-ended questions, learners were able to verbalize what they like and dislike

about the instruction. The comments were all positive and there was a consensus that the

instruction was easy to understand. What the learners disliked about the instruction are: the

directions should be at the top instead of the bottom, a repeat button would be helpful, a go back

button to access the terms is needed, and closed caption that can easily be turned on and
CASE MANAGEMENT MODULE EVALUATION 15

off. Even though some functions need to be added, all the learners were able to complete the

learning module (see Appendix C).

3) T-test Analysis: Our hypothesis was that the training will enhance the learners

knowledge about case management. The dependent sample t-test was used to analyze the pre-

test and post-test for the same group of learners. The Paired Two Sample for Means is shown in

table 2. We used the one-tail results since the hypothesis was directional. The absolute value of

the t-stat result is greater than the critical value 6.19>1.83. So, we reject the null hypothesis and

support the research hypothesis that the learning module will enhance the learners knowledge

about case management. This shows that the training was effective in teaching learners about the

case management process.

Table 2
d. Recommendation
These are our recommendations based on the evaluation results:

1. Enlarge Font. Learners should be able to read the text in every section of the module

without any effort. The practice section shows the font smallers than the rest of the

section. This will need to be fixed to ensure all learners are able to read all section.
CASE MANAGEMENT MODULE EVALUATION 16

2. Drag and Drop Activity. This activity was fun and interactive for the learners. However,

the drag and drop items drop in the middle of the sentence and obstruct the view for the

learners. The item should drop on the left side of the sentence where it does not obstruct

the view for the learners.

3. Improve navigation. The learners experience a loop in the module. This loop brings the

practice section right after the quiz and does not give the results for the quiz. This loop is

due to the SCORM file that had to be activated for this module in order to be used in a

Learning Management System. A work around will need to be applied to solve this issue.

Section 4

Summary

The research hypothesis was that the Case Management module would introduce Case

Management and its seven phases to newly hired members in the department. Based on the

evaluation of the module, learners got familiar with the Case Management process. The training

module significantly contributed to the improvement of post-test scores. Learners showed

understanding after taking the training module. The training module showed that newly hired

members would benefit from taking this course at the time of hire. Recommendations will be

incorporated to the training for capstone purposes.

Section 5 Appendices

Appendix A

Survey Questions

What is your age? 29, 30, 30, 32, 32, 33, 38, 38, 40, 40
CASE MANAGEMENT MODULE EVALUATION 17

What is your gender? 5 females, 5 males

What is your ethnicity? 1 African American, 1 Asian, 8 Hispanics

What is your highest level of 3 Masters Degree, 7 Bachelors degree,

education or highest degree

you have received?

What is your job title? 5 teachers, 1 case manager/resource specialist, 1 screen

printer, 3 computer technicians

Have you ever worked with a 4 yes, 6 no

case manager?

If yes, how many years? 3 years (2 people), 2 years (2 people)

If yes, describe your (Total of 4 people)

interaction. 3 people said they had special needs students with

individualized education program (IEP). 1 person said she is

a case manager and she interacts with other case managers.

Appendix B

Pre-Test and Post-Test

Multiple Choice

1. It is an active and ongoing process where aspects of the planning and implementation

phases are reviewed. (The correct answer is C .)

A) Planning
CASE MANAGEMENT MODULE EVALUATION 18

B) Monitoring

2. Influenced by a number of factors. These factors may relate to organizational and/or

agency requirements in which a limited time frame is identified for case manager

involvement with any one client. (The correct answer is A.)

A) Closure

B) Engagement

3. It helps ensure that outcomes for people are relevant to their needs and include a focus

on community inclusion and participation. (The correct answer is A.)

A) Review

B) Implementation

True/False

4. Implementation is the process of putting into action the plan developed by the case

manager, together with the person with a disability, and their family. (The correct answer

is A.)

A) True

B) False

5. Assessment is a map of actions that documents the interventions, actions, responsibilities

and timeframes needed to meet the identified goals. (The correct answer is B.)

A) True

B) False

6. Complete the sentence below by filling in the blanks.

_________ establishes the relationship and sets the ground rules which are enhanced and

reviewed throughout the phases of case management. (The correct answer is Engagement.)
CASE MANAGEMENT MODULE EVALUATION 19

Type the _________ phase

7. A __________ is a map of actions that documents the interventions, actions,

responsibilities and timeframes needed to meet the identified goals. (The correct answer is

Plan.)

Type the _________ phase

Appendix C

Questionnaire Results

1. Were the instructions for the module easy to follow?

Very Easy (2) Easy (8) Neutral Difficult Very Difficult

2. How familiar did you become with case management?

Very Familiar Familiar (3) Undecided Unfamiliar Very Unfamiliar

(7)

3. How confident you feel to manage your first case for a client?

Very Confident Confident (5) Undecided Unconfident Very

(5) Unconfident

4. Did you find the content relevant to the topic?

Very Relevant Relevant (1) Undecided Irrelevant Very Irrelevant

(9)

5. Did you experience any frustration while taking this module?

Lots of Some Frustration Unsure Hardly Any No Frustration

Frustration (2) Frustration (8)

6. How would you describe the level of difficulty of the quizzes?

Very Easy Easy (4) Undecided Difficult (6) Very Difficult

7. What did you like about the instruction? Please type your answer.

-Instruction was easy to follow. Can just click on the button. Easy to navigate.

-You had clear instruction.

-I can read and hear the instructions.

-It was easy to understand.

-I had no difficulties with following the instruction.

-The instructions were clear (3)

-The instruction were easy to understand (2)

8. What did you dislike about the instruction? Please type your answer.

-The instructions were clear but need to be able to repeat.

-The instructions should also be at the top of the page.

-If you close the captions, then the directions are gone.

-The instruction for the practice was unclear. There should be a box next to the definition that

I can drop the word into.

-There should be a go back button to access the list of terms at any time without having to wait

for the audio to finish playing.

-The instructions were clear and concise (5)

Appendix D

Observation Checklist

Observation Checklist Yes No Comments

Accessibility

1. Is the font size large enough 9 1 The learner would squint and lean closer to the

for the user to read? computer screen to read in the practice section

of the learning module.

2. Is the user able to read the text 10

on the contrasting background

color?
CASE MANAGEMENT MODULE EVALUATION 22

3. Does the user know how to 10

disable closed captions?

Main Menu

4. Does the user understand the 9 1 One learner close the closed caption, which has

directions? the directions. The learner looked confused on

what to do.

5. Does the user know how to 10

access the learning module?

6. Does the user know how to 10

access the test?

7. Does the user know how to 10

access the practice portion of the

module?

Navigation

8. Does the user know how to 9 1 The user struggle with using the main menu

navigate in the main menu? when progressing through the learning module.

9. Is the user able to go back and 7 3 The learners had trouble going back and forth

forth between slides? using the go back button or the rewind and

fast forward buttons at the bottom.

Is the user able to drag and drop 8 2 Some learners wanted to undo what they

the words to the corresponding dragged, but were unable to.

definitions?

Appendix E

Learning Module Link

http://itcdland.csumb.edu/~dbarraza/labs/Case_Management/

Appendix F

Pre-Test and Post-Test Results

Appendix G

Descriptive Statistics of Pre-Test and Post-Test