CHAPTER III

RESEARCH DESIGN AND METHODOLOGY

This chapter presents the research design, locale of the study, participants and

their population size, research instrument, data gathering procedure and statistical

treatment applied in the analysis of the data.

Research Design

The study used the Qualitative and Experimental research designs. Experimental

research design provided tangible and concrete evidence that subtitles improved the

comprehension of the viewers through reading and listening. Under this design, the data

were interpreted from the tabulated scores using numerical descriptions that answered the

research questions.

The type of design is a True Experimental Pretest-Posttest Design (TEPPD). It

was a true experiment because even though the participants were not assigned to groups,

their academic performance were at par with each other. It was revealed in the t-test that

their academic performance was not significantly different from each other. Thus, the

performance of both groups was compared to each other in terms of their performance

in the experiment. However, the pretest and posttest of each group was statistically

treated separately as it employed criterion referenced test. It was due to the percentage

of participants in both groups which has difference in terms of comprehension level.

Criterion-referenced tests compared the learner’s comprehension against a

predetermined standard which were identified as High, Average, and Low level.

Criterion-referenced tests, each learner’s performance was compared directly to the said
33
standard, without considering how other students from the other group performed on the

test.

The other research design utilized is Qualitative. In-depth answers from the semi-

structured interview revealed the rise and fall of the scores was attributed to the subtitles.

This could not be achieved when a mere survey method is employed. Moreover, it was

necessary for the study to have scrutinized responses from the selected participants so

as to have thorough analysis on the treatment conducted in the experiment.

Participants of the Study

There were two sections in Grade 11 that participated in the study (see Appendix

V). The participants were from Metropolitan Medical Center – College of Arts, Science

and Technology, Senior High School Department. It is where the researcher has formerly

taught. The two sections were ABM 1, as the control group (CG), and ABM 2 as the

experimental group (EG). The CG consisted of thirty-three (33) students, while the EG

consisted of thirty-five (35).

The selection of the participants of the study was based on the following criteria

set by the researcher to lessen the confounding variables that could affect the result of

the study. Since the type of design is TEPPD, the researcher ensured that the groups are

as similar as possible.

(i) The researcher selected two classes in the same Academic Track;

(ii) The students were enrolled in the same school;

(iii) The sections were at the same grade level;

(iv) The students in the two classes had the same teacher, therefore

exposed to similar teaching styles.

34
 Aside from identifying the academic performance of each group to know whether

they are comparable, taking such steps increased the internal validity of the study

because it eliminated some of the most important confounding variables.

Research Procedure

The researcher utilized a researcher-made test for the pretest and posttest of the

study. The pretest measured current reading and listening comprehension level of the

participants of the CG and EG. It is necessary for the reading comprehension and

listening comprehension to be in one research instrument because viewing the

audiovisual aid with subtitles requires the learners to listen and read the subtitles at the

same time. This underlying principle follows the Theoretical Framework of the study,

“The Cognitive Theory of Multimedia Learning” (see Figure 1).

In the experiment, audiovisual aids without subtitles exposed the learners to

visual and aural cues only, whereas, the audiovisual aids with subtitles presented visual,

aural, and written cues.

After viewing an audiovisual aid for each day, both groups answered researcher-

made comprehension questions that are based from the material that they watched (see

Appendix E). The CG viewed audiovisual aids without subtitles to test their

comprehension through dual channels: listening and viewing comprehension. On the

other hand, the EG tested their comprehension through multi-modal channels: reading,

listening and viewing. They repeated the process once a week in two and a half months

for a total of ten (10) audiovisual aids. After which, the posttest was administered to the

35
two groups. Same test was administered after a period of two (2) and a half months after

the experiment to know how both groups fared in terms of their comprehension.

Then, the researcher selected the students interviewed. He interviewed students

from the CG and EG who achieved an improvement on their score. An interview was

also conducted to students in both groups who got a lower score to acquire an explanation

to such phenomenon (see Appendix K). Other informal interviews were conducted after

the researcher tabulated the scores and identified learners he deemed necessary to

interview to provide appropriate interpretation of the numerical data.

The data gathered were the answers from the objective type of questions after

the experiment, scores from the pretest-posttest and the transcript of answers from the

interview. Hence, there were three sets of questionnaires that the researcher constructed

which were validated by a team of experts (see Appendix H). Two are for the

experimental design which include the pretest-posttest listening cloze test and the

comprehension questions for the daily experiment; and the last one is the interview

guide.

Research Instrumentation and Validation

The researcher-made test used in the pretest and posttest was validated by experts

in the field of language (see Appendix I). The researcher chose a descriptive essay

entitled, “Where's the Patis?” by Carmen Guerrero Nakpil (see Appedix D). It discovers

the phenomenon of how “travel has become the great Filipino dream.” Filipinos go

abroad either to visit overseas sceneries and experience new cultures or to work abroad

so that they can provide their loved ones an improved life. This was considered to be in

the interest level of the participants since it featured a local context and relevant Filipino
36
experiences. It was also age and grade appropriate for the Grade 11 Senior High School

students, as revealed by the Fry Graph readability test.

Source: https://readabilityformulas.com/freetests/fry-graph.php

Figure 2: Fry Graph result of the Research Instrument Readabillity

The formula and graph were used to provide a common standard by which the

readability of the material can be measured. The zone where the two coordinates meet

shows the grade score of nine (9) to ten (10). Since Fry Graph is used on Filipino students

who are second language learners of English, the researcher deemed it appropriate to

grade eleven learners even if the grade result is one to two steps lower.

37
 Figure 3: Cross-over Examination using Different Readability Indices

The test result of Flesch-Kincaid deemed it appropriate for learners aged fourteen

(14) to fifteen (15) years old in grade level nine (9). The Gunning-Fog Score, on the

other hand, categorized it in grade level ten (10); SMOG Index on grade level eight and

two tenths (8.2); Coleman Liau Index on grade level ten and six tenths (10.6); while the

automated readability index classified the reading material as readable to grade eight (8)

learners. Since the learners are Filipinos, the researcher considered it appropriate for the

participants of the study with sixteen (16) to seventeen (17) years of age and one to two

steps lower in terms of grade level, that is, grade eleven in the Philippine setting.

38
Table 3
Word Statistics of the Research Instrument
Description Count Percentage

Total number of words 709

Total number of unique words 382 54% of the total text

Total number of repeat words 327 46% of the total text

Average number of words per 16

sentence

Total number of sentences 43

Total number of characters 3163

Average number of characters per 4.5

word

Average number of syllables per word 1

Total syllables in text 1085

Total number of words with double 149 21% of the total text

syllables

Total number of words with single 481 68% of the total text

syllables

Total number of words with 3+ 79 11% of the total text

syllables

39
 This instrument, transformed into a listening cloze test, was answered by the

students as their pretest and posttest (see Appendix A). It was iperformed iby listening to

the entire essay and then answering the cloze with the seventh (7th) word removed.

In order to measure their listening comprehension, same essay was narrated and

recorded by a Filipina with an American-like acccent and then played before the learners

(see Appendix G). The essay has some Filipino words which could not be pronounced

accurately if it is spoken by a native speaker. This is why, the researcher sought help

from his friend to narrate the story with a native-like American accent. It was recorded

by a call center agent, who in her work, talks to Americans everyday and seemed to

influence her speaking skills in English. The rationale of this decision was for the

researcher to test how much the learners can comprehend English as spoken, not in

Standard Filipino-English, but in its native-like American way. Therefore, the listening

cloze test, as a hybrid task, combined two activities, testing reading and listening

comprehension.

Vargas (2015) also did this test in her research and she claimed that by listening

to the material first, the language learners will have to rely on it to understand the passage

before being able to answer the cloze test. Moreover, the study conducted by Lewkowicz

(2017) used this test where it used two tasks rather than one. The purpose of his study

was to see how well a test of this nature performed in comparison with the more

traditional type of listening comprehension tests both in terms of reliability as well as

validity.

Combining the listening and reading comprehension followed the notion that

language cannot be fragmented into elements (Sulistyo, 2017) especially that the variable
40
used in this study is audiovisual aids with subtitles. Reading the subtitles is

simultaneously done by listening to the spoken dialogues. In support to this, Oller (2019)

argued that language competence is a unified set of interacting abilities that cannot be

tested separately. Thus, he called it integrative testing.

In the pretest, the participants from both group ifilled iin ithe imissing words after

listening to the passage. iThe frequency of correct response iwas ithe measure of ihow

comprehensible ithe itext was. iThe ihigher ithe ipercentage, ithe imore comprehensible

ithe itext is. Therefore, the researcher decided to have this as an evaluation of the reading

and listening comprehension level of the learners since listening cloze itest imeasures ithe

icomprehensibility iof ia itext.

If imore ithan sixty percent iof ithe iblanks iare ifilled iin icorrectly iby ithe

iparticipants,ithe itext iis iconsidered ito ibe ieasy or the learner is in the high

comprehension level. iIf forty-one percent ito sixty percentiof ithe iblanks iare ifilled iin

icorrectly, ithen ithe itext has Iaverage comprehensibility , and the learner is in average

comprehension level. iIf iless ithan forty percent of ithe iblanks iare ifilled iin icorrectly,

ithen ithe itext iis iconsidered idifficult ito ithem, thus the learners are in the low

comprehension leveli(Nielsen , i2011).

Table 4:
Comprehension Level Measurement of the Research Instrument
Percent Correct Comprehensibility Comprehension Level

61% and above Easy High

41%-60% Average Average

40% and below Difficult Low

41
 The abovementioned description of the comprehension of the instrument served

as the evaluation for learners in terms of their reading and listening comprehension level.

To iprovide icontext, ithis paper istudied ion ithe ieffects iof isubtitles ispoken in

English by a native speaker, called intralingual or same-language subtitles, instead of

i subtitles iin ione’s iL2, or interlingual subtitles. This is why the research instrument is also

spoken and written in English.

The cloze test was then administered after listening to the audio recording. This

tested both their reading and listening comprehension.

The research instrument used in the pretest and posttest is a descriptive type of

essay, whereas the comprehension tests are composed of different types of academically-

inclined topics. The purpose of which follows the general premise of the experiment

which is to enhance the language skills of the students. Language, in general sense, is a

tool that people use to acquire knowledge in different fields. Thus, the findings of the

study would prove or disprove the effectiveness of the subtitles in audiovisual aids to

enhance the reading and listening comprehension regardless of the test format of the

research instrument.

For the comprehension tests in the experiment, the Revised Bloom’s Taxonomy

of Cognitive Domain was considered to have reasonable number of items for lower order

thinking skills and higher order thinking skills. This helped the researcher to analyze the

cognitive effectiveness of English subtitles in audiovisual aids.

42
 Distribution of Cognitive Domain Levels
30

24
25
21
20 20
20

14
15
11
10

0
1 2 3 4 5 6
Remembering Understanding Applying Analyzing Evaluating Creating

Figure 4: Numeric Distribution of Cognitive Domain Dimensions of the Revised

Bloom’s Taxonomy used in the Experiment

This figure shows the numeric distribution of the cognitive domain levels of the

revised Bloom’s version used in the study. There were twenty-one (21) questions in the

Remembering, twenty-four (24) questions in Understanding, twenty (20) questions both

in Applying and Analyzing, fourteen (14) questions in Evaluating and eleven 11

questions in Creating. To sum up, there were a total of one hundred ten (110) questions.

All questions were distributed across the whole ten-week experiment period. The topics

and the corresponding questions were based on the contents of their subject, Practical

Research 1, which were duly authorized and noted by the school principal (see Appendix

N).

43
Table 5
Summary of Numeric Distribution of Cognitive Dimensions

BLOOM'S TAXONOMY ON COGNITIVE

DOMAIN

Number of LOTS Questions Number of HOTS Questions Total

Number
of
Items
Remem Underst Evalua
Analyzin Creating
bering anding Applying
g ting
Experiment
Day An example of
4 3 0 2 1 0 10
1 Applied Research
Day Different Kinds of
4 2 0 3 1 0 10
2 Literature
Day
Language in L1 4 2 10 3 1 0 20
3
Day Paraphrasing Skills:
1 1 0 2 1 5 10
4 The Basic Steps
Day How to write a
0 4 0 2 1 3 10
5 Research Proposal
Day Research Questions
1 2 1 2 1 3 10
6 Worth Answering
Writing the
Day
Introduction of a 1 3 2 2 2 5 10
7
Research Paper
Day Theoretical
0 2 5 1 2 0 10
8 Framework
Difference Between
Day
Theory and 3 2 2 2 1 0 10
9
Scientific Law
Day What is Basic
3 3 0 1 3 0 10
10 Research
TOTAL 21 24 20 20 14 11 110

After completing the 10 day experiment in a ten (10) week period, the posttest

was administered to the students. The conduct of the test was the same as how the pretest

was conducted. This is the time when the progress or improvement of the learners was

statistically treated by the researcher.

44
 After knowing the result, the researcher conducted an interview to verify the

reasons of such decline or improvement of scores (see Appendix U). Questions in the

interview guide (See Appendix K) compared the effects of the use and nonuse of subtitles

in audiovisual aids but the researcher did not prompt the learners to have an idea on the

positive or negative exposure of subtitles in the audiovisual aids.

Statistical Treatment

The data from the experiment were treated statistically by the sig-test to identify

if the pretest and posttest of separate group has significant difference. Therefore, there

were two results: a result for the CG, and another for the EG. Significance level adopted

was p = 0.05 with 95% confidence intervals. Given samples from two normal populations

of size n1 and n2 with unknown means and and known standard

deviations and , the test statistic, which has the standard normal

distribution (N(0,1)), compares the means which is known as the two-sample z statistic.

In addition, descriptive measures such as frequency and percentage were also

obtained to describe the results of the study. Percentage frequency distribution was

applied in determining levels of comprehension when data were analyzed based on their

scores in pretest, experiment and posttest. Below is the formula where P is the percentage

45
being computed, f for the frequency and N for the total number multiplied by one

hundred percent (100%).

46