AC 2012-3821: INTEGRATING PROJECT MANAGEMENT, LEAN-SIX

SIGMA, AND ASSESSMENT IN AN INDUSTRIAL ENGINEERING CAP-

STONE COURSE
Dr. Ana Vila-Parrish, North Carolina State University

Ana ”Anita” Vila-Parrish is a Teaching Assistant Professor and Director of Undergraduate Programs in
the Edward P. Fitts Department of Industrial and Systems Engineering.

Dr. Dianne Raubenheimer, Meredith College

c American Society for Engineering Education, 2012

 Integrating Project Management & Lean-Six Sigma Methodologies in an
Industrial Engineering Capstone Course
Abstract

The ability to effectively manage large, complex projects is a skill required of most engineers in
industry today. The skills required to be an effective project manager go far beyond technical
engineering content taught in engineering curriculums. Industry sponsored capstone project
experiences require students to communicate, deconstruct the large project into manageable
pieces, and manage risk and uncertainty. This is a departure from traditional engineering
problems which have a right answer and typically have a prescribed solution method. In this
paper we discuss a pilot study that evaluates how a structured framework of project management,
mimicking industry practices, affects the problem solving process throughout the project life
cycle. Specifically we develop an approach that integrates project management methods and
tools with Lean-Six Sigma methods. An additional objective of this research is to develop a
better understanding of the unique aspects of the engineering problem solving process. We
assessed the student’s problem solving strategies, products, and design process reflections using
Wolcott’s “Steps for Better Thinking” rubric 1.

Introduction

Capstone courses give students the opportunity to solve large, unstructured problems in a
classroom setting. These team-based projects mimic the industrial setting that most students will
enter upon graduation. Throughout the capstone experience students find themselves faced with
complexities not found in a traditional course, especially when the projects are industry
sponsored. Student teams face challenges when defining objectives for an ambiguous project,
controlling scope creep, achieving buy-in, and selling their results to the sponsor. These are
skills that are not taught in most engineering curriculums prior to the capstone course(s). Our
observations are consistent with the observations by other researchers who have studied the
design process. Wilson et. al 2 highlighted how students in the capstone course setting struggle
with setting milestones and soliciting feedback at the right times. Developing and effectively
communicating the project plan and status are critical to the success of the project. As noted by
Yildirim3, there is a need for understanding the relationships between design activities, the
evolution of the design process, and the solution or outcome. We share in this sentiment and feel
that developing a framework that explicitly links the design process with design activities is
beneficial for the students and the instructor. Given a design process framework, the students
establish a sense of the key inputs and outputs needed to identify and solve their problem. The
instructor now has a course structure they can exploit for assessments that can be timed at key
intervals throughout the design process. Our solution was to leverage standard industry
practices, specifically project management and Lean-Six Sigma, and overlay these
methodologies on top of the capstone course. In the following section we describe the course
content enhancements made in the Fall of 2011.

Course Enhancements: Using Project Management & Lean-Six Sigma Tools

In the Fall of 2011 changes were made in the capstone project course in the Industrial and
Systems Engineering (ISE) department at North Carolina State University (NCSU) which were
intended to both provide a framework for the design cycle and assess the problem solving
process throughout each phase of the process. Our research interests are two-fold: (1) to describe
an application of enhancing the capstone experience by aligning the content with industry project
life cycle practices and (2) to assess the evolution of a student’s problem solving skills through
the course of the project. The capstone project course in the ISE department is a single semester
course in which 100% of the projects are industry sponsored. In the Fall 2011 cohort there were
11 total students that were assigned in teams of 2 or 3 to four unique projects. The majority of
class lecture time was spent introducing project management methods and Lean-Six Sigma
(LSS) tools. From a project management perspective the course was framed as having four
consecutive phases: Project Initiation, Planning, Implementation/Execution, and Closeout.
Overlaid on top of the project management phases we introduced LSS methodologies (define,
measure, analyzed and improve, control) and tools. Figure 1 shows the overlap in these project
life cycle phases.

Define Measure Analyze & Control

Improve

Project Project Close-

Planning Implementation
Initiation out

Figure 1: Relationship between Six Sigma phases and Project Life Cycle phases used in the ISE
498 Senior Design Project course

Various tools linked to project phases were discussed throughout the course prior to the
estimated time in which the tool would be needed by the student team. Table 2 lists the tools that
align with each phase. The tools with an asterisk were assignments that were graded while the
others were optional. Since projects evolve at different rates, short, in-class cases were provided
to the students to practice their design process skills. The students would work with their project
teams during these sessions and then outcomes were discussed with the entire class. Providing
in-class cases helped the instructor to informally assess the understanding of critical problem
solving approaches.
Tools and methods alone do not result in successful projects. To further emulate industry
practice, we incorporated two mechanisms to control the design process. First is the requirement
that each team submit a Status Report every two weeks to the instructor and the industry sponsor.
Figure 2 shows a sample status report. The intention of the report is to familiarize students with
the process of representing project status in a single page with visual cues (red, yellow, green).
We defined red as a milestone or deliverable that was missed, yellow indicated a concern that
may jeopardize the success of the corresponding milestone, and green indicated that the item was
on track or complete. The status report keeps the team and sponsor accountable for action items,
milestones, and focuses the team on project risks and concerns. The report is a quick means of
formal communication between the student team and the sponsor team that can be used as a basis
for follow-up discussion.

Project Name: Project X Date: 9/15/2011

Overall Status: R,Y, G Project Sponsor: Co. XYZ

PM Phase SS Phase Milestone Owner Plan Status Major Issue/Concern Owner Target Status
Planning Measure Draft Schedule Team 9/25/2011 G
Planning Measure
Planning Measure
Execution Analyze
Execution Analyze
Execution Improve

Risk Mitigations Plan Owner Status

Figure 2: Sample Status Report

The second project control tool that we employed is the phase-gate process. Similar to industry
practice, our phase-gate process was implemented by developing deliverables that are reviewed
during each project lifecycle phase. In this context the syllabus contained deliverables at key
points during the project so the instructor, students, and sponsor could monitor the project
progress. These deliverables typically took the form of a document (e.g., project charter) or
presentation (e.g., project proposal presentation) or a combination of the two. The following are
a list of the major deliverables that occur throughout the semester in order of their occurrence:

• Project Charter (presentation and document)

• Project Failure Modes and Effects Analysis (FMEA)
• Project Proposal Report and Presentation
• Mid-phase Presentation
• Final Project Report and Presentation
While in industry these phase gates are used to make go/no-go decisions, in the context of the
capstone course they were used by the instructor to give the student teams’ feedback on project
direction, approach, etc…

These structured project management and six-sigma methods provide a guide during the project
lifecycle which can be an overwhelming and ambiguous experience. However, developing
course content that enhances the student’s exposure to these topics is not enough. We must
assess not only the student’s application of the concepts but also the potential impact these
concepts have on their ability to solve problems. In order to do so we developed design
reflection assignments at critical phases of the project (Planning Phase Exit, Mid-Implementation
Review, and Project Closeout). Throughout these reflections we asked the students questions in
order to discern how they were using information to solve their problem during that particular
phase. Finally, we administered a skills survey pre- and post-capstone that measured self-
assessment on a variety of measures. The same survey was sent to the sponsors who rated their
expectation prior to the course and the reality post-project completion. We will describe this
reflections assessment and rubric as well as the results of the pre/post skills survey in more detail
in the following section.

Assessment Mechanisms

Now that we have set the context of the course content and structure, we will focus on the
various modes of assessment. The assessments had two main objectives and the assessment
timeline is shown according to the project lifecycle phase in Figure 3.

Pre- Capstone Skills Post- Capstone Skills

Project Mid-Phase
Assessment Assessment
Proposal Presentation

Project Project Close-

Planning Implementation
Initiation out

Project
Final Project Report
Charter

Figure 3: Assessment timeline where the asterisk denotes the design reflection time points in
relationship to the project life cycle phase, the gray arrows indicate key deliverables, and the
black arrows indicate pre/post surveys.

The first objective was to develop an understanding in the gap between industry expectations and
student preparation on a number of measures (from technical to professional).
In conducting the comparable surveys before and after the course, we developed the following
research questions:

1. How do companies’ baseline expectations differ in regards to the following categories:

• A student team’s ability to communicate
• A student teams’ ability to solve problems
• A student teams’ ability to develop a design strategy throughout the project life cycle
• A student teams’ ability to communicate in various forms
• A student teams’ ability to function effectively as a team
• A student teams’ knowledge of project management methods
• A student teams’ knowledge of lean-six sigma methods
2. Are companies’ expectations a representation of reality?
3. How do students’ rate themselves on these same measures before and after their senior
design experience?
• Which measures appear to be impacted by the capstone experience?

Our second objective was to evaluate the evolution of the problem solving process using
Wolcott’s “Steps for Better Thinking” rubric (see Appendix 1). This rubric places students on a
scale from less complex performance patters (e.g., confused fact finder) to more complex
performance patterns (e.g., strategic re-visioner) during each of the 4 pre-defined problem
solving steps. This rubric was used to assess student’s problem solving capabilities using
computational tools in engineering classes 4 and while these authors found the rubric to be a
valuable assessment tool, they identified the need to revise the rubric to better align with
engineering problem contexts and nomenclature. We set out to identify the limitations of the
rubric and to revise it accordingly, so as to make it more relevant to the engineering problem
solving and design context.

At three key points during the semester students were asked to write reflective accounts of their
problem solving and design processes (see Figure 3). These reflective questions were based on
the different stages in the Wolcott rubric. The two project researchers coded student reflective
responses using the Wolcott rubric, and noted areas for improvement of the rubric as they
proceeded with the coding. The focus of these improvements is on developing descriptions in
each of the cells (see Appendix 1) that are more appropriate to the engineering problem solving
design process.

Assessment Results

Assessment results are presented in the next sections.

Student’s Skills Self-Assessment

To assess students’ perceptions about gains made in the course, pre- and post-surveys were used
and students were asked to rate themselves on common questions about skills. A five point scale
was used with the following anchors:

1 - Novice = you are new to the concept/skill.

2 - Advanced beginner = you have some exposure to the concept/skill, but have not used it
frequently and need to learn much more.

3 - Competent = you have had reasonable exposure to and use of the concept/skill, but can still
develop further.

4 - Proficient = you have had a lot of exposure to and regularly use the concept/skill but there are
a few areas for advancement.

5 - Expert = you have had significant exposure to and use the concept/skill routinely, and there is
little or nothing to improve.

Due to the small sample size the Mann-Whitney U-test was used to establish significant response
differences. Table 1 below shows all questions asked, with significantly different changes
highlighted in gray using an alpha of 0.05.

Table 1: Student pre- and post-survey scores for surveys conducted at the beginning and end of
semester.
Topic Pre- Post- Question
test test
Mean Mean
Problem Solving u-value
Q1 3.73 3.73 Differentiating between useful and irrelevant information .9
Q2 3.27 3.64 Identifying uncertainties and limitations 0.3
Q3 3.91 3.64 Considering different perspectives or viewpoints in 0.56
solving a problem
Q4 3.64 3.45 Identifying strengths and weaknesses of different 0.9
perspectives
Q5 4.00 3.73 Interpreting and organizing information 0.85
Q6 3.27 3.73 Generating different solutions to a problem 0.27
Q7 3.55 3.91 Deciding on the most appropriate solution 0.19
Q8 3.91 3.82 Generating a plan to solve the problem 0.95
Q9 3.36 3.50 Communicating decisions about the best solution 0.56
Q10 3.18 3.27 Dealing with uncertainties, limitations and tradeoffs 0.95
Q11 3.82 3.91 Looking for ways to improve solutions and processes 0.85
Section Mean 3.60 3.67
Topic Pre-test Post-test Question
Design u-test
Q12 3.64 3.40 Generating appropriate goals and objectives 0.71
Q13 3.55 3.80 Identifying key design issues 0.17
Q14 3.73 4.00 Identification of necessary tasks and actions 0.43
Q15 3.18 3.50 Identifying deliverables 0.51
Q16 3.27 3.50 Creating a project schedule 0.61
Q17 3.36 3.70 Identifying resources and constraints 0.25
Q18 3.55 3.90 Implementing the process 0.25
Q19 3.82 3.80 Organizing and managing tasks .71
Q20 3.18 3.50 Monitoring and evaluating progress 0.51
Section Mean 3.47 3.68
Communication
Q21 3.09 3.64 Writing technical documents (proposals, reports, etc.) 0.24
Q22 3.27 3.55 Making oral presentations 0.61
Q23 3.64 4.09 Written professional communications (e.g. email, letters) 0.19
Q24 3.10 3.91 Professional oral communication (e.g. phone or other 0.04
medium)
Section Mean 3.30 3.80
Team work
Q25 4.27 4.09 Attending meetings 0.95
Q26 4.00 4.00 Contributing to idea generation .8
Q27 4.00 4.09 Contributing to decision making 0.52
Q28 4.18 4.18 Communicating with team members .44
Q29 4.00 4.18 Taking on a fair share of tasks 0.33
Q30 4.09 4.18 Dependability and preparedness 0.61
Q31 4.73 4.36 Team spirit and respect for others 0.85
Q32 4.27 4.09 Negotiation and conflict management skills 0.9
Q33 4.09 4.00 Time management 1.0
Q34 4.09 4.36 Completion of work assumed/assigned 0.19
Section Mean 4.17 4.20
Project Management
Q35 3.64 3.82 Assuming the role of a project manager 0.52
Q36 3.09 3.55 Developing a design roadmap 0.15
Q37 2.55 3.27 Developing risk management plans 0.06
Q38 3.18 3.73 Managing a project through each project life cycle phase: 0.08
(1) Planning Phase
Q39 3.00 4.00 Managing a project through each project life cycle phase: 0.004
(2)Implementation Phase
Q40 3.27 3.64 Managing a project through each project life cycle phase: 0.17
(3) Close-out Phase
Q41 2.18 3.91 Using the six sigma methodologies (DMAIC) to solve 0.000
problems
Q42 2.27 4.00 Using the Lean methodologies to eliminate waste 0.000
Section Mean 2.90 3.89
Problem solving and design skills generally stayed the same or improved throughout the
capstone experience. There are some interesting results where the skill level is reported to have
decreased, namely Q2-5 and Q12. Though our sample size is limited we believe there is an
explanation for this regression in skill level. Questions 2-4 focus on working with others and
integrating different perspectives in the problem solving process. Students may have
overestimated their ability based on a lack of experience working with people from different
backgrounds. The result in Question 5 regarding interpreting and organizing information leads
to an inherent challenge in a capstone course that incorporates real-world problems. Since data
and information are not provided to the students in a formal classroom-like setting the instructor
believes that the students may have encountered a significant challenge that they had not
anticipated. Many students in the course expressed the difficulty of defining project goals
(Question 12) and most groups anticipated that the sponsors would define the objectives. These
sentiments were discussed during the Lessons Learned lecture at the end of the course.

Given the structure of the course it was not surprising to find that the skills defined in the
communication and project management increased significantly throughout the course of the
semester. We can observe that all skills in that category increased and almost all were
significant. As discussed in the Course Enhancements section, the instructor delivered content
and assigned deliverables specific to project management. The students had the opportunity to
communicate using many modes: email, phone, and formal presentations (3-5 per team). The
spring 2012 senior design course will further emphasize these topics. Given the small class size
in fall 2011, the researchers will extend the same pilot survey to approximately 30 students in
order to validate the results and observations from this study.

Industry Expectations

In order to assess the performance of students and appropriateness of course content it is critical
to understand industry expectations. The pre-capstone skills assessment’s objective was to set
the baseline expectations on the measures discussed in the previous section before the sponsor
interacts with the student team. The instructor also completed a parallel survey assessing the
same constructs as those assessed in the student survey.

For the industry ratings, no constructs resulted in statistically significant difference pre/post
capstone. However, there were several constructs that were significant at the alpha = 0.06 to 0.1
range and we will discuss a few instances that fall into this category. For example, the
companies lowered their score on Q2 noting that students did not deal with uncertainty as well as
they expected (alpha = 0.06). Q14 (identification of necessary tasks and actions) had an alpha of
.1 and highlighted a weakness in the students’ ability to define tasks and actions. We believe this
may be related to a weakness in planning and solving problems under uncertainty, thus
combining the trend seen in both Q2 and Q14. Finally, the industry representatives had higher
expectations related to the students’ ability to assume the role of the project manager (Q35).
This result further emphasizes the need for instruction in project management at the
undergraduate level.

Another interesting result of this analysis was the variation in expectations between companies.
For example, Figure 4 shows the industry assessment on the problem solving skills question set
(Q1-Q11). In the post-capstone, survey “Company 3” had three employees who participated in
the survey. All other company representatives stayed the same and completed both
questionnaires. As previously mentioned, there were no significant differences in the industry
assessment pre/post capstone. However, we hypothesize they may be due to the averaging of the
scores and small sample size. Many of the scores pre and post-capstone varied by as much as
two to three points on the likert scale between companies. In general, companies did not have
high expectations (i.e., most Skill Level ratings ≤ 3) pre-capstone. Regardless of the expectation
level entered during the pre-capstone survey, the sponsors’ expectations seem to be met (i.e., a
similar post-capstone score was recorded). We hypothesize that there could be a correlation
between the expectation level of the company and student performance. This may be due to the
level of support and pressure to succeed and is an area that we are interested in exploring in the
future.

problem solving
4.5

3.5

3
Company 1
2.5
Axis Title

Company 2
2 Company 3
1.5 Company 2b

0.5

0
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11

Figure 4: Industry pre-capstone scores on questions 1 through 11, which reflected problem
solving skills.
 Skill: Problem Solving Post-Capstone

5
4.5
4
Company 1
Skill Level (Scale 1-5)

3.5
3 Company 2

2.5 Company 2b

2 Company 3
1.5 Company 3b
1 Company 3c
0.5
0
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11

Figure 5: Industry post-capstone scores on questions 1 through 11, which reflected problem
solving skills.

In addition we compared the post-capstone responses between industry and student respondents.
The significant differences are summarized as follows:
• Written professional communications
• Team spirit
• Completion of work assumed
• Managing projects through implementation and close out phase

Since there were three companies (and four respondents) in fall 2011 we will again be validating
these results in the Spring 2012 semester with a larger sample size. In the spring of 2012, there
will be nine distinct companies involved and all new companies will take the pre-survey. At the
completion of the semester all companies will respond to the post-capstone survey. With this
larger sample size we will reanalyze the data in order to draw stronger conclusions and validate
the findings from the pilot study. In addition any themes that emerge will be investigated for
further course enhancements.

Scoring of Student’s Design Reflections

As previously discussed the students submitted three design reflections at key milestones (Refer
to Figure 3). We constructed the survey such that the questions mapped to a specific design
phase in Wolcott’s rubric in order to assess the individual student’s performance. For example,
the following questions represent a subset of the questions posed in the planning phase exit
survey which occurs immediately following the project proposal submission:

• What was the most useful information used to identify project objectives? What
information was not useful?
• What assumptions did you make? Why are these assumptions necessary?
• Is there one correct way to solve the problem you are working on? Explain your answer.

These questions corresponded to the Identify step in Wolcott’s rubric. Both researchers worked
together to score the student design reflections at these milestones, by discussing and reaching
consensus on the score for each reflection. For each step and sub category (e.g., Step: Identify,
sub-category: A) the performance pattern was scored from 0 (Confused Fact Finder) to 4
(Strategic Revisioner). Student reflections showed a transition from lower levels in the rubric at
the start of the semester, to higher levels over time (see Figure 5 below). The responses to the
reflection questions were useful to the instructor in developing interventions and additional
course content. For example, there were individuals within one team that had not identified any
project assumptions. The instructor called a meeting with the team and it was discovered that
they had not established a good understanding of the project objectives set by the project
sponsor. This finding led to two changes implemented in the Spring 2012 capstone course.
First, the project charter is developed in two phases allowing time for feedback from both the
instructor and the sponsor. Second, content has been added to the corresponding lecture to
discuss the difficulties that past teams have exhibited in developing project objectives,
assumptions, and constraints. The reflections enable the instructor to intervene in the design
process at critical points and provide targeted direction and advice.
 ←Less Complex Performance More Complex Performance
Steps for Better Patterns Patterns→
Thinking SKILLS "Confused Fact "Biased Jumper" "Perpetual Analyzer" "Pragmatic "Strategic Re-
Finder" Performance Pattern Performance Pattern Performer" Visioner"
Performance 1—-How 2—-How Performance Pattern Performance
Pattern 0—How performance might performance might 3—-How Pattern 4—-How
performance might appear when Step 1 appear when Step 1 performance might performance might
appear when Step skills are adequate, and 2 skills are appear when Step 1, appear when one
1, 2, 3, and 4 skills but Step 2, 3, and 4 adequate, but Step 3 2, and 3 skills are has strong Step 1,
are weak skills are weak and 4 skills are weak adequate, but Step 4 2, 3, and 4 skills
skills are weak
Step 1:
IDENTIFY A— Overall, FIRST Overall, FIRST Overall, SECOND
Identify and use reflections reflections reflections located
relevant information
located in these located in these in these cells
B—Articulate
uncertainties cells cells

Step 2:
EXPLORE C— Overall, FIRST
Integrate multiple reflections Overall, SECOND
perspectives and
located in these reflections located
clarify assumptions
D—Qualitatively cells in these cells
interpret information
and create a Overall, SECOND
meaningful reflections
organization
located in these
cells
Step 3:
PRIORITIZE E—
Use guidelines or Overall, THIRD Overall, THIRD
principles to judge
reflections located reflections located
objectively across the
various options F— in these cells in these cells
Implement and
communicate
conclusions for the
setting and audience
Step 4:
ENVISION G—
Acknowledge and Overall, THIRD Overall, THIRD Overall, THIRD
monitor solution
reflections reflections located reflections located
limitations through
next steps H—Overall located in these in these cells in these cells
approach to the cells
problem

Figure 6: Overall classification of students on the Wolcott rubric at each of the three phases in
the design process.

However, the application of the Steps for Better Thinking rubric does not map exactly for an
engineering context. The researchers identified a need to revise the rubric to contain
terminology for an engineering problem solving context. Further, we observed the need to
revisit steps in Walcott’s rubric. For example, students (and engineers) revisit problem solving
phases throughout the project life cycle. As objectives are being defined and understood, it may
be necessary to revisit the activities undertaken in the Identify step. A revised rubric which
reflects this iterative nature would be of significant value to instructors and researchers who are
interested in the progression of problem solving throughout the design process. We are currently
working to extend Wolcott’s rubric to address some of the limitations that we identified in this
pilot study.

Conclusions

The study presented in this paper is a first step towards developing a deeper understanding of the
design process using a combination of structured approaches: project management and Lean Six-
Sigma and assessment methods. Project management and Lean Six-Sigma tools provided a
structure to an otherwise overwhelming project environment. We are continuing to refine these
tools and develop new deliverables that will help the students overcome the challenges we have
observed in the Fall of 2011.

This analysis has revealed the need for an assessment rubric that shows the iterative design
process along a spectrum of project performance. While we were able to capture the general
trajectory of students thinking throughout the design process using Wolcott’s rubric, it was
difficult to translate some of the terminology in an engineering context. A correlation study
between the outcomes from the rubric, the pre/post surveys (industry and student), and the
individual’s (team’s) performance on the final project will be conducted with the additional data
that is being gathered in the Spring of 2012.

References:
[1] Wolcott, S.K. (2006). Steps for better thinking: Developmental problem solving process. Downloaded
6/20/2007 from http://www.wolcottlynch.com/EducatorResources.html
[2] Wilson, S. and McIntyre, M. (2010). Assessing and improving a capstone design sequence with industrial
project management techniques. Proceedings of the 2011 American Society for Engineering Education
International Conference, Vancouver, British Columbia, Canada.
[3] PinarYildirim, T., Besterfield-Sacre, M., Shuman, L. Wolfe, H., and Clark, R. (2007). An Analysis of
Engineering Design using Markov Chains. Proceedings of the Institute of Industrial Engineers 57th Annual
Research Conference, Nashville, TN.
[4] Raubenheimer, C.D., Craig, A., and Joines, J. (June 2009). Using computational tools to enhance problem
solving. Proceedings of the 2009 American Society for Engineering Education International Conference,
Austin,TX. Also published in Computers in Education Journal, (2010), vol20.