Running Head: CRITICAL ANALYSIS OF ARTICLE 1

Critical Analysis of Article

Joshua Croft
Mabel Duran
Christine Flanders
Michelle Henderson
Kristen Tatroe

California State University, Monterey Bay

April 14, 2020

Abstract
CRITICAL ANALYSIS OF ARTICLE 2

This paper is a critical analysis of a research study published by The International Review

of Research in Open and Distributed Learning entitled Coordinated Implicitly? An Empirical

Study on the Role of Social Media in Collaborative Learning. This analysis will include a

discussion of research purpose, biases, procedures, results, and the study’s application to

academic communities and future research.

Introduction

The critical analysis is on a study on the role of social media on collaborative learning. It

is an empirical study based on the observation of groups of students that are participating in a
CRITICAL ANALYSIS OF ARTICLE 3

collaborative class assignment, using online social tools to communicate and produce their work.

The purpose of this study was to test the effectiveness of team communication and socialization

on how productive it makes the team run and complete work. The stated hypotheses within the

study are measurable only within an observational arena. They are clearly stated, but seem to be

already undeniable components of social interaction and not really well thought out. Researchers

of this article seemed to have a positive attitude toward the subject matter. The goals of the

research were geared not toward the opinions of does it work but rather, how well does it work.

We do not believe this study, in and of itself, a complete study on social collaboration. We

believe it provides a start to further research on the subject.

Research Procedures

This was a quantitative study conducted using graduate students from an eight-week

business analytics course. The average student age was 22.74 years. The students were mostly

Chinese; however, there were also 11 international students. After removing 25 participants from

the study, the authors were left with results from 135 participants in 40 teams of three to four

people. The participants were randomly assigned to 40 four-member teams. This was an

adequate sample size to conduct this study and was representative of the target population

(Chinese graduate students); however, it may not adequately represent graduate students in the

U.S. or older graduate students.

The study method required each team to collaborate via social media to complete a

research report. Zhang, Chen, Pablos, Lytras, & Sun state that the participants “were encouraged

to progress the collaboration process on specific social media platforms Slack and WeChat”;

however, it was noted that students often used additional social media (2016). The impact and

degree of other social media use is unknown.

Data was collected by survey at both the fourth and eighth weeks to mitigate common

method bias, as some students did not complete either one or both surveys. Assessment was

determined using a seven-point Likert scale. Further, the data for the independent variables was

collected during the first survey and the data for the dependent variables was collected during the

second survey. The authors ensured validity and reliability by assessing the variables using

previously established scales adapted from the validated work of Zellmer-Bruhn and Gibson;

Hoegl and Gemuenden; Borgatti and Cross; and Ko, Cho, and Roberts.

Per the Salkind guide, this research is quasi-experimental because it is testing the causal

relationship between information processing and Transactive Memory Systems (TMSs),

information processing and communication quality, social connection and TMSs, social

connection and communication quality, TMS and task performance, TMS and personal success,

communication and task performance, and communication and personal success. This research

was based on previously validated studies and relied heavily on past research to formulate its

hypotheses and collect data. This is evident in the extensive literature review and use of

established assessment methods as noted above. As a result, the research procedures appear

appropriate and clearly described so that others can replicate them.

It should be noted that after consultation with our co-teacher, Dr. Fischer, our team has

concluded that, though the methodology appears valid and the researchers took measures to

employ previously established and validated assessment methods, the conclusions drawn by the

researchers are not valid:

However, the conclusions drawn by the researchers did not include applying a multiple

test/simultaneous testing procedure such as the Bonferroni Correction which would

reduce the likelihood of obtaining statistical significance when in fact the null hypothesis
CRITICAL ANALYSIS OF ARTICLE 5

of no effect should not be rejected. Instead of a critical value, alpha, being .05; the value

should have been .05/15 = .0033, with 15 being the number of tests of hypotheses that

was made. Thus, to achieve significance, the p level obtained through the Analysis of

Variance/F-test would have to be less than .0033. Only one test resulted in a p value less

than .001 (which still might not have been less than the critical value of .0033). (D.

Fischer, personal communication, April 12, 2020)

As a result of ongoing conversations with Dr. Fischer, our team agrees with him that this study

should be reviewed and the conclusions and summaries modified accordingly.

Research Results

For this research study, Zhang et al., (2016) analyzed data using the Partial least squares

(PLS) analysis. This statistical technique, commonly used for small- to medium-sized samples,

was appropriately used as a method for constructing predictive models when factors are many

and highly collinear (Tobias, 2016). In this study, a singular model could not be used to explain

the relationship of the multiple variables. Partial least squares serves as a useful multivariate

technique. It is a solution for research questions that are focused on analytical prediction and

correlation without sacrificing multiple factors, as seen in this study.

The researchers also used Cronbach’s alpha (Cα) value, which tests the estimated

reliability of the psychometric variables. It is used under the assumption that multiple items are

measuring the same underlying construct. The study items included people-related success

related to teamwork, including work satisfaction and personal learning effectiveness. It also

included social media activities such as information processing and social connection.

The Cα values of the variables indicated reliable data (Zhang et al., 2016). The variables

emerged in meaningful ways from the data because it showed that each variable had a
CRITICAL ANALYSIS OF ARTICLE 6

statistically significant association with another variable. The results of the PLS analysis showed

the relationships between people-related success items and the significance of social connection

activities (Zhang et al., 2016).

Discussion of Results

The results largely confirmed the hypothesis of seeing increased TMSs between

participants. Surveys revealed a positive outlook on the use of social media and an overall

feeling of success in task completion. A metric and mathematical baseline ensured a level of

success could be measurable with survey results. Based on the team’s analysis, those results

averaged scoring over the threshold of a hypothetical success rate. These results were also

backed and checked by a theorem of statistical reliability to ensure validity. Task performance

resulted in the highest perceived area of success. It is not stated in the study’s discussion, but this

result determines a high level of success in the use of social media. The participants gain TMSs

and productivity but overall felt this media was successful as in-person collaboration.

Accessibility of social media allowed relief from time constraints of in-person meetings and

inherently increased efficiency, if the performance task showed to be as complete as traditional

methods. This point of view only came from the student participants. At this point, the reader

does not know the quality of the work produced and this is an issue. Many students proceed on

with a task under false pretenses or in misunderstanding, this is in fact very common in K-12

education. Without input from the professor, it is difficult to proceed with a discussion of the

students’ work, or effectiveness of social media on their work, if we don’t know the end result of

the students’ efforts.

Summary
CRITICAL ANALYSIS OF ARTICLE 7

With current prevalent use of social media in personal, business, and academic arenas,

the relevance of this research paper seems timely and useful. It certainly offers relevance to us

students in the MIST program, where we communicate via Zoom, Google Docs, and text

message. We found, however, that while Zhang et al. provided a start to the research on this

topic, further research should be conducted with graduate students in the U.S., Europe, and

elsewhere, as well as with older graduate students. This new study could produce even more

relevant results if all students used the same social media to collaborate. Alternatively, a separate

study could be conducted regarding the efficacy of various types of social media.

References

Salkind, N. (2009). Exploring research. Essex, England: Pearson Prentice Hall.

Tobias, Randall. (2016). An introduction to partial least square regression. UCLA Institute for

Digital Research and Education: Statistical Consulting. Retrieved from

https://stats.idre.ucla.edu/wp-content/uploads/2016/02/pls.pdf

Zhang, X., Chen, H., Pablos, P. O. D., Lytras, M. D., & Sun, Y. (2016). Coordinated Implicitly?

An Empirical Study on the Role of Social Media in Collaborative Learning. The

International Review of Research in Open and Distributed Learning, 17(6). doi:

10.19173/irrodl.v17i6.2622