Paper ID #11849

Development and Usability Testing of a Student Mobile Application for the

AIChE Concept Warehouse
Rachel M. White, Oregon State University
Rachel White is a senior in chemical engineering at Oregon State University. Her interest in engineering
education comes from student experience and observing fellow classmates struggling with their studies.
She is interested in promoting conceptual understanding in the chemical engineering core curriculum so
that students can perform better both in the classroom and beyond.
Dr. Bill Jay Brooks, Oregon State University
Bill Brooks is a Postdoctoral Scholar in the School of Chemical, Biological and Environmental Engi-
neering at Oregon State University. As an undergraduate he studied hardware, software, and chemical
engineering. He ultimately received his Ph.D. from Oregon State University in Chemical Engineering.
He is currently interested in the development of technology to study and promote STEM learning.
Dr. Milo Koretsky, Oregon State University
Milo Koretsky is a Professor of Chemical Engineering at Oregon State University. He received his B.S.
and M.S. degrees from UC San Diego and his Ph.D. from UC Berkeley, all in Chemical Engineering.
He currently has research activity in areas related engineering education and is interested in integrating
technology into effective educational practices and in promoting the use of higher-level cognitive skills
in engineering problem solving. His research interests particularly focus on what prevents students from
being able to integrate and extend the knowledge developed in specific courses in the core curriculum to
the more complex, authentic problems and projects they face as professionals. Dr. Koretsky is one of the
founding members of the Center for Lifelong STEM Education Research at OSU.

American
c Society for Engineering Education, 2015
 Development and Usability Testing of a Student Mobile Application for
the AIChE Concept Warehouse
Abstract
Incorporating user feedback is imperative for the adoption and continued usage of educational
innovations in the classroom. We report on the development of an Android-based student
mobile application, a user-suggested improvement for the AIChE Concept Warehouse. An
Apple-based mobile application is also in development. Our intent is to share the applications’
development and improvement process in the hopes that other innovators can benefit from the
lessons learned through our experience.

The AIChE Concept Warehouse was developed with the intent of fostering a community of
learning within chemical engineering. The Concept Warehouse is a web-enabled database
infrastructure that is designed to promote concept-based instruction through the use of concept
questions. These concept questions are used in core curriculum courses like Material/Energy
Balances, Thermodynamics, Transport Phenomena, Kinetics and Reactor Design, and Materials
Science. Availability of high quality concept questions can lower the barrier for faculty to use
concept-based instruction and assessment. This software allows the instructor to engage students
and evaluate student learning in real-time. The instructor is then able to adjust the pace of
lecture in response to student understanding. The Concept Warehouse also allows for reflective
assessments such as “the muddiest point.”

The student mobile application was developed to make it easier for students to submit answers
and written explanations to these assessments using mobile devices. Originally, students could
submit their answers to conceptual questions using clickers, mobile browsers on smartphones,
and laptops. Input via smartphones, however, proved cumbersome because it depended on the
quality of the student’s mobile browser and utilized the full-size webpage interface. The
improved mobile student interface facilitates participation by making responses via smartphone
more user-friendly.

After the development of the application, we conducted initial usability tests with students who
had previously used the web-based options for answer submission. In order to gauge usability,
we collected usage statistics from student responses to a usability survey. Survey responses were
used to identify student likes and dislikes as well as to compare different available options for
answer submission. These results will be used to improve the design of the current application
as well as guide our design decisions for the development of the iOS version of the student app.

Introduction
Engineering educators and industry professionals alike have expressed a need for students to
have the ability to apply what they learn to novel and challenging problems1. Traditional lecture-
based courses, however, commonly promote rote memorization over the conceptual
understanding needed to apply knowledge to these new situations2,3. Instructors need to place a
greater emphasis on conceptual understanding in their classrooms in order to adequately prepare
students for practice.
The AIChE Concept Warehouse was designed to lower one of the biggest barriers that prevents
instructors from using concept-based instruction: access to high quality concept questions.
Developing these questions is difficult and time-consuming; these questions must focus on a
single important concept, require thought from the student, and be neither too easy nor too
difficult to answer4. The Concept Warehouse alleviates these barriers by giving instructors
access to over 2000 concept questions for core chemical engineering courses and allowing
instructors to contribute their own questions. The Concept Warehouse also provides many
different methods of delivery for these questions. Instructors can assign these questions as
homework or use them in class as part of active learning pedagogies. Students can respond to in-
class questions using their clickers, laptops, or smartphones and instructors will receive a
distribution of student responses, their written explanations, and summary word clouds in real-
time. This feature is useful for determining student understanding of new topics and whether
students hold misconceptions about this content.

Many higher education institutions have chosen to adopt clickers into their curriculums. This
technology carries with it a set of disadvantages. Clicker limitations include the inability to
assess students’ ability to answer open-ended questions involving higher level thinking5,
distortion of student responses by providing a set list of answer possibilities, and the inability to
simulate situations similar to industry6. Most clickers are not compatible with the AIChE
Concept Warehouse’s written explanations feature, which allows students to include an
explanation of their answer choice to a multiple-choice question. While laptops allow students
to provide these explanations, many students have reported that carrying their laptops from class
to class is inconvenient7. Thus, mobile devices such as smartphones and tablets are attractive
alternatives.

Previously, use of the AIChE Concept Warehouse interface on smartphones proved cumbersome
and inconvenient for students. The webpages were not optimized for mobile devices, and
students had to manually resize pages in order to read and answer concept questions. The
AIChE Concept Warehouse Student App seeks to improve the student user experience with
optimization for small screens and touch-friendly navigation. In this paper, we present a
description of the student application. We also provide a detailed description of the design and
development process to provide a reference for future design projects. Finally, we report on
future plans and activities for the student application.

Related Work
Mobile device applications have been used as teaching and learning aids before. EvaluA+ is an
iPad app developed to aid instructors in creating and using rubrics to grade assignments and
presentations. It allows instructors to create a rubric, import assignments into that rubric, and
then view the rubric and assignment simultaneously for grading in both online and offline
modes. Once created, these rubrics can also be shared with students to use as a reference while
working on the assignment. The app also has a presentation mode where the instructor can use
the rubric to grade a presentation in real-time8.

Pikme is an iPhone app designed to help the instructor manage class lists to learn student names,
randomly select students to participate in class discussion, and rate solicited student responses.
The app then stores these ratings so the instructor can use them in assigning grades. Instructors
reported that use of the app in class led to increased student motivation, increased participation,
improved engagement, and more balanced feedback about course material. Students felt that the
random selection for participation in class was fair and provided incentive to prepare outside of
class. Some students even reported an increased willingness to volunteer regardless of whether
they were chosen by the random selection tool or not9.

EvaluA+ and Pikme are examples of mobile apps designed to aid the instructor, but mobile apps
have also been developed with the intent of providing formative assessment of student
understanding in real-time. InkSurvey is a free, web-based software that collects student
responses to open format questions. Students “ink” their responses with pen-enabled Android
devices, iPads, iPhones, or tablet PCs. Students can respond to in-class questions with words,
drawings, graphs, or equations. Creating these responses gives an opportunity to interact with
the subject material and increase metacognition. The instructor gains real-time feedback about
what students are thinking and can address misconceptions and questions10.

Mobile apps like InkSurvey help promote active learning by encouraging students to reflect on
subject material and explain concepts in their own words. Studies of more than 5,000 science
and engineering students found that active learning methods double conceptual learning gains11
and give a 25% higher pass rate than traditional lecture12. Active learning methods help place
greater emphasis on conceptual understanding. We consider this type of app development a key
technology-based enabler for active engagement and learning.

The Student App Design Process

Develop a Function List for Each Screen
The AIChE Concept Warehouse Student App is designed to have the same functionality as the
student version of the web interface. Thus, the app has to be capable of presenting the student
with assigned question sets, allowing the student to select an answer, provide a written
explanation and confidence follow-up if prompted, and then submit the answer to record
participation. Figure 1 is a graphical representation of the relationship between features for the
student app.

Figure 1: Graphical representation of the elements of the AIChE Concept Warehouse Student App.
Create Mock-Ups
Mockups of each screen in the app were created to determine the
layout as well as the look and feel. The application is optimized
to run on mobile devices, and this was the driving factor in
designing the user interface. Screen space is at a premium on
smartphones, so the menu is hidden when not needed. When the
student is viewing and answering questions, the question text,
image, and answer options are the focus of what is displayed on
screen. Menus and navigation were designed for touch input.
Figure 2 provides an example of the mock-ups produced when
designing the app.

Build a Live Version and Conduct Internal Testing

The AIChE Concept Warehouse Student App was written in Java
using the Android Software Development Kit (SDK) and the
Eclipse Integrated Development Environment (IDE). Java, the
Android SDK, and Eclipse IDE are all available online for no
charge. The app was then tested using an Android Figure 2: Mock-up of the questions screen
interface.
emulator and developer-enabled phone. These allowed for
continued refinement and assurance of stability before releasing the app on a larger scale.

Test Usability in the Classroom

The AIChE Concept Warehouse Student App was tested in a sophomore level energy balances
course. Students were encouraged to download the app at the start of their term and answer in-
class questions using their smartphones. They were given a usability survey through the Concept
Warehouse to share their experiences and rank which device they preferred to use.

Functionality
The AIChE Concept Warehouse Student App is intended to mimic the functionality of the
webpage student interface13. Students must first log in. They begin at the home screen once
they have supplied their credentials. If questions have been assigned to the student, these
question sets are listed by class with number of questions available to answer (Figure 3a). If no
questions have been assigned, a message will appear on the home screen explaining this (Figure
3b). The app supports multiple question types; instructors can assign multiple choice questions,
multiple choice questions where students must select all that apply, and questions where answers
must be ranked.

The home screen is also the first opportunity students have to use the hidden menu (Figure 4).
The menu exists as a small image at the top of the screen to conserve space. Students can either
tap the image or swipe from left to right to bring up the menu. This menu serves as the main
method of navigation through the app; students have access to their profile, settings, and
informed consent information. The profile screen allows students to update their demographic
information. This includes birth year, year started at the university, gender, race, and major. The
settings screen allows students to register their clickers as a secondary method of submitting
answer choices. The informed consent screen provides students with information about the
ongoing AIChE Concept Warehouse Study, Integration of Conceptual Learning Throughout the
Core Chemical Engineering Curriculum and allows students to opt in.
 Figure 3: The home screen (a) with questions assigned and (b) when no questions have
been assigned.

When questions are assigned, students can navigate to the questions screen (Figure 5) by either
clicking on the names of the classes with questions assigned or by clicking the questions button
in the hidden menu. The questions screen presents the question text, associated picture or
diagram if applicable, and the answer options. If the instructor has chosen to enable written
explanations and/or confidence follow-ups, these text input fields appear as well. Students can
use the drop-down menu to navigate between classes that have questions assigned.
 Figure 5: Students can swipe from left to Figure 4: A concept question as it appears
right or tap the button in the upper left- on the student’s device.
hand corner to access the hidden menu.

Student Usability Feedback

The AIChE Concept Warehouse Student App was tested by students of a sophomore level
engineering course. The app was made available on the Google Play Store, and students with
Android devices were encouraged to download and test the device in a four week period.
Students were then assigned a 12 question survey to answer through the Concept Warehouse
about their experiences. Table 1 shows the survey questions.

The survey data includes 178 student responses, though not every student answered every
question. This survey was provided to all students in the course, regardless of whether they
tested the Student App or not. Students were allowed to choose the method of answering in-
class concept questions using either their laptop, clicker, smartphone/tablet mobile browser, or
the Android app on a supported device. Of the 178 students that participated, 156 (88%)
indicated that they owned a smartphone. All 178 students ranked the different methods of
answer concept questions in the Concept Warehouse in the following manner, from most to least
frequently used:

1. Laptops
2. Smartphone mobile browser
3. Tablet mobile browser
4. Using the app on a smartphone
5. Using the app on a tablet
6. Clickers
Table 1: Survey questions intended to elicit student feedback about the AIChE Concept Warehouse Student App user experience.

Usability Survey Questions

Question Question Type
In how many of your courses have you used the AIChE Concept Multiple Choice
Warehouse?
Rank the following methods of using the AIChE Concept Multiple Choice – Ranking
Warehouse from most frequently used to least frequently used.
Do you own a smartphone? Multiple Choice
How often do you use your smartphone for class? Likert
In what different ways have you used your smartphone for Open-Ended
class?
Have you ever used a smartphone for educational purposes? Open-Ended
I will likely continue using this app to answer concept Likert
questions through the AIChE Concept Warehouse.
If I make a mistake in using the app (navigation, inputting Likert
information, etc.), I can recover from my mistake easily.
I would recommend using the app to my friends. Likert
What additional features would you include in the student Open-Ended
mobile app?
What other aspects of the app would you change? Open-Ended
Please include any additional comments about the app. Open-Ended

Nineteen students (11%) mentioned in their survey that they chose not to test the student app. It
is likely that the number of students who answered the survey but did not participate in testing is
higher given the popularity of iOS devices. Three students mentioned in the open-ended section
of the survey that an iOS version of the app would be beneficial. The instructor of the course
noted that a large percentage of the class owned iOS devices. An iOS version of the student app
is in development to improve student access.

Five of the students provided suggestions for features to add in later versions of the AIChE
Concept Warehouse Student App. One student mentioned that they preferred their laptop so that
they could have the question window and a reference table open at the same time. Adding multi-
window support to the app or allowing students to access important tables of information in the
Concept Warehouse is contrary to the intent of the concept questions. Such questions are meant
to be computationally simple and conceptually challenging. Two students indicated a desire to
be able to change answers after submission; the Concept Warehouse does not support this on the
web page or the student app. Two students indicated that the formatting of the question
information and answers could be improved. One student mentioned that it was difficult to tell
when the confidence follow-up was being used by the instructor, and the other student explained
that the formatting of question text (use of bold or italicized characters) did not always function
properly. These suggestions are worth further investigation.

Student suggestions indicate that the app effectively allows students to answer conceptual
questions in class as well as provide a written explanation of answer selection and an indication
of confidence in the selected answer. Based on the rankings from the usability survey, the app
has successfully replaced clickers as an answer input method. This app enables features often
not provided by clickers, including written explanations and a confidence follow-up. The AIChE
Concept Warehouse Student App provides an example of successfully developing and
implementing a mobile application in an educational setting.

Future Work
While the app is fully functional and successfully allows students to answer concept questions
in-class, some features are missing that could improve the user experience. These features
include the ability to save one’s username and password, checking network connectivity prior to
logging in, and giving the app explicit instructions for operating in a low-power setting.

The usability survey successfully engaged 178 students out of a course of approximately 300,
indicating a survey completion rate of about 59%. It would be beneficial to improve the survey
for use in later terms. New questions should elicit whether students felt their learning improved
as a result of using the app, what features they liked, and what devices (the type of smartphone or
tablet) they were using with the app. It might also be beneficial to develop a survey for
instructors as well to capture usage trends and other aspects of the class as a whole.

The lack of an iOS version of the app was a major barrier to complete replacement of other
devices for answering in-class questions. An iOS version of the AIChE Concept Warehouse
Student App is currently in development using the new Swift programming language and xCode
IDE. This version of the app will be available in Summer 2015 pending Apple’s approval.

Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No.
DUE 1023099. Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of the National Science
Foundation.

References
1. Educating the Engineer of 2020: Adapting Engineering Education to the New Century. The National
Academies Press: Washington DC, 2005.
2. Elby, A., “Another Reason that Physics Students Learn by Rote”, American Journal of Physics, 1999.
3. Felder, R. M. and Brent, R., “Understanding Student Differences”, Journal of Engineering Education, 57 –
72, 2005.
4. Crouch, C., Watkins, J., Fagen, A., Mazur, E., “Peer Instruction: Engaging Students One-on-One, All at
Once”, Research-Based Reform of University Physics, 2007.
5. Yorke, M., “Formative Assessment in Higher Education: Moves towards Theory and the Enhancement of
Pedagogic Practice”, Higher Education, 477-501, 2003.
6. Kowalski, S. E., Kowalski, F. V., Hoover, E., “Using InkSurvey: A Free Web-Based Tool for Open-Ended
Questioning to Promote Active Learning and Real-Time Formative Assessment of Tablet PC-Equipped
Engineering Students”, Proceedings of the 2007 ASEE Conference and Exposition, Honolulu, HI, June
2007.
7. Koretsky, M. D. and Brooks, B., “Student Attitudes in the Transition to an Active-Learning Technology”,
Chemical Engineering Education, 41-49, 2012
8. Bakrania, S., “A rubric-based grading app for iPads”, Proceedings of the 2013 ASEE Conference and
Exposition, Atlanta, GA, June 2013.
9. Bakrania, S., “Getting Students Involved in a Classroom with an iPhone App”, Proceedings of the 2012
ASEE Conference and Exposition, San Antonio, TX, June 2012.
10. Kowalski, F. V., Kowalski, S. E., Gardner, T. Q., “Using Mixed Mobile Computing Devices for Real-Time
Formative Assessment”, Proceedings of the 2013 ASEE Conference and Exposition, Atlanta, GA, June
2013.
11. Hake, R. R., “Interactive-engagement versus traditional methods: A six-thousand-student survey of
mechanics test data for introductory physics courses, American Journal of Physics, 66-74, 1998.
12. Poulis, J., Massen, C., Robens, E., Gilbert, M., “Physics lecturing with audience paced feedback”,
American Journal of Physics, 439-441, 1998.
13. Brooks, B. J., Gilbuena, D., Falconer, J. L., Silverstein, D. L., Miller, R. L., Koretsky, M. D., “Preliminary
Development of the AIChE Concept Warehouse”, Proceedings of the 2012 ASEE Conference and
Exposition, San Antonio, TX, 2012.