Paper Title: Identifying key value drivers of lessons learned in projects

Presentation Title: How to teach a new dog old tricks

Abstract

Project managers are exhorted to perform “Lessons Learned” exercises at least at the
end of each project or project phase. In reality these either don’t happen at all or the
findings are not seen as very valuable, and thus not recorded, shared or capitalised.

In addition, in the increasingly complex world and accelerated timescales that project
managers have to deal, not only can we ill-afford to repeat our mistakes but we need to
find faster, more efficient methods of developing robust, reliable plans for our projects.

In my research into the value of lessons learned, I have discovered that not only is the
value not well understood but also one of the reasons for an apparent reluctance to
perform the lessons process in the first place is that this process of identifying the real
“lesson” is often not well understood.

This paper attempts to both uncover some key values of project lessons as well as show
some simple yet robust mechanisms for the process of identification, codification and
sharing of lessons. The primary objectives are to help project managers with an end-to-
end process that will reduce the repeating of mistakes from the past, thus improving the
profession of project management as perceived by project stakeholders.

Introduction
The main driver for doing this research is, quite simply, a desire to see the
profession of project management taken more seriously. One way to improve the
image of PM is to improve the probability of success for projects, reduce the number
of typical failure modes and improve risk management through improving the
approach to dealing with uncertainty. One way of dealing with the unknown is to
base our decision making approach on experience – otherwise known as lessons
learned.

Most evidence of lessons learned practices in project management today points to a

tendency to apply a mechanistic approach (where an approach exists at all). The
reality is that knowledge management, of which lessons learned is one instance, is a
very holistic concept. Our tendency to measure and rationalise the approach to
lessons learned often leads to a technology driven solution - which more often than
not fails (Baets, Knowledge Management and Management Learning)

In this paper I will present some of the definitions and usages of lessons in the
context of project management, identifying where possible the value drivers in each
case. Building on this I will try to show where organisations have made use of
specific approaches or methods to support the identification, capturing and sharing
of lessons across their project management communities, highlighting the usage of
best practices where possible as well as the key value drivers in each case. Further
research into specific practices in organisations is used to support, or to refute, the
key arguments in the various theoretical viewpoints.

This research builds on, the work already done in my previous paper “You’ve learned
your lesson – was it worth it” (Gray, You’ve learned your lesson – was it worth it) in
which I noted that although there is no magic formula for identifying the value of a
lesson, there is already some evidence of links between lessons and value to the
organisation. I will therefore make an attempt at identifying some value drivers as
well as best practice approaches in organisations today.

Problem statement
Einstein was reported to have said that if you understand the question then you
already have 80% of the answer. In the case of this research, the question of how to
maximise the return on investment on lessons learned seems at first glance to be
quite straightforward – until we start trying to clarify what is meant by “value”, what
defines a lesson and what is or can be done to ensure the value is maximised.

Therefore I will break this down into four key parts:

- Lessons as Knowledge: What are the knowledge components of lessons, and

how are they valued
- Lessons as Continuous Improvement: What are the change drivers in lessons
- Lessons as Risk Management: What is the relationship between lessons and
avoiding repeated mistakes
- Lessons as Best Practices: What is the relationship between lessons and
doing things better

Contents
I will set the stage with a review of what is known in accessible literature on the
subject of knowledge management in relation to organisational success, and what is
meant by “learning organisations”.
Building on this we will take a look at some specific case examples of what is done in
a few well-known international organisations – both from a perspective of what works
in addition to what doesn’t (on the grounds that we are trying to learn something from
others here).

Finally I will make some recommendations on what organisations can do in order to

promote the lessons learned approach in their organisation and – through this –
improve the probability of project success in the organisation and the maturity of
project management at the same time...

Literature Review
Lessons as Knowledge
It is well accepted that lessons are a form of knowledge; terms such as “Intellectual
Capital”, “Organisational Competence”, “Competence Baseline” and various others
are used in connection with the cumulative knowledge in organisations, and
sometimes in project management as well. It would seem likely then that this
“knowledge” has some perceived value at individual, organisational and professional
levels.

Baets, in his compiled book on knowledge management points out that the key
reason for managing knowledge is to reduce the possibility of making the same
mistake twice – as well as learning from the past to be more effective and efficient in
the future. He goes on to argue that knowledge and in particular the capability to
manage knowledge is at the centre of successful companies. We can see examples
of this in companies that are valued more on their intellectual property and what they
can do with it, rather than on their physical assets.

Baets defines three main categories of knowledge:

- Tacit knowledge, based on lived experiences
- Explicit knowledge, based on the rules and procedures that the company
follows
- Cultural knowledge, defined as the environment in which the company and
the individual operates

From this we can easily see a link between lessons learned and tacit knowledge.
Lessons are by their very definition based on past experiences, either our own or
those of people (PMs) around us.

Less obvious is the link between lessons as explicit knowledge, unless we follow
through on lessons and identify those which lead to changes in the way of working of
the organisation.

In terms of cultural knowledge, I would argue that lessons transmitted across PM

communities (PMI etc) add to the overall cultural knowledge base – especially if the
if the lesson can be identified as a best practice.

Furthermore, a learning organisation is one in which all its members are supported
and encouraged to learn from internal as well as external experiences, allowing the
organisation to adapt and improve on a continuous basis (Baets, deGeus). In fact
deGeus (Arie de Geus – The Living Company, Habits for Survival in a Turbulent
Business Environment) draws the analogy of the organisation as a living entity in
which the individuals are the knowledge carriers, clearly indicating a need to transmit
knowledge from one carrier to the next in the form of experiences or lessons. In this
way the knowledge becomes a transferrable commodity (Baets), and it is generally
accepted that a commodity has some value, be it tangible or intangible.

Knowledge at a “professional” level can and should be transferrable as well –

consider the following two examples:
- Medical practitioners: Doctors will make a diagnosis of a patient’s illness
based on their taught knowledge added to their own experience. In the case
where the symptoms appear to be outside the current knowledge sphere of
the doctor then they will normally either refer to a specialist (peer) or to case
history held in the medical journals. New discoveries are also subject to
extensive documentation and diffusion to practitioners.
- Practitioners of Law: Lawyers and in particular civil case lawyers make
extensive use of case history to argue or support their points in legal
 challenges. This is referred to as quoting precedent and forms the keystone of
modern civil law.

In both of these examples (there are others) the common viewpoint is that these are
professions in which individuals are taught, licensed to practice, and make extensive
use of past knowledge or lessons in their day-to-day work. In each of these, the
value of the knowledge attained is typically recognised in the “professional level”
salary paid to the practitioners, as well as their “social status”

Lessons as Continuous Improvement

Various writings on Continuous Improvement (CI) come up with a common driving
message: The need to move from a reactive to a more proactive approach to
improvement.

Bessant (John Bessant – Technology Management in the UK ), in his report to the

European Technology Management Initiative in 1992, points out that competitive
success comes not only from having a stream of new and innovative products but
also from improvements in the speed, quality and reliability of their delivery. Although
the discussion is focussed largely on improvements in the manufacturing process,
there is significant amount of attention paid to the idea of improvements in the
overlying business process or control elements. Key to the idea of CI are the
following three prerequisites:
- Active involvement of all employees in the innovation, and by association, the
improvement process
- Positive support from the structure (organisation) in which the CI initiatives are
being driven
- Investment in training and development of the employees involved in the
process

Applying the same approach to Lessons Learned indicates a similar set of key
elements:
- Involvement of the entire project team in the lessons learned process is not
enough; we need to bring in other members of the organisation as well.
- Management support for the lessons learned process is requirement for
success
- The organisation needs to invest time and energy in developing the lessons
learned process and also in supporting the process going forward – including
training support for identified competence gaps.

Organisations that apply ISO or CMMI approaches to quality should already be in the
mode of identification and application of improvements in their processes.

In their paper on CI behaviour, Bessant, Caffyn & Gallagher (John Besant, Sarah
Caffyn, Maeve Gallagher – An Evolutionary Model of Continuous Improvement
Behaviour) identified several common themes or ideas in their research case
studies:
- CI involves a suit of behaviours that evolve over time
- These behaviours cluster around several core themes – for example,
behaviours associated with systematic finding and solving of problems,
monitoring and measuring processes, etc
 - There appears to be a correlation between performance and the extent of
development of these routines
- Developing routines involves two kinds of learning – improving and reinforcing
behaviours within a particular routine cluster and adding new routines to the
repertoire.
- Although the development of CI involves a behavioural learning process
which takes place over time, there is no correlation between length of time
and degree of success. Rather the key variable seems to be the amount of
management effort put in to build and maintain the CI behaviour patterns.

In the conclusion of their paper, Bessant, Caffyn and Gallagher not that CI is of
“considerable strategic importance, but that it’s management is often poorly
understood”.

Making the link between lessons learned and CI is quite easy, and we can
immediately note similar requirements and issues facing the LL initiatives.

Lessons as Risk Management

Risks are defined as “uncertain events that, if they occur, may have either a positive
or negative effect on the project objectives.” (PMI – PMBoK Guide, 3rd Edition ).

Although it is well recognised that uncertainty is a key driver in innovation and

creativity, the role of the project manager is to reduce the levels of uncertainty in the
risk domain wherever possible.

Hillson and Simon in their book on the ATOM Methodology (David Hillson & Peter
Simon – Practical Project Risk Management) of risk management point out the
importance of using a common process to manage uncertainties that can be
identified as either threats or opportunities in the project. They go on to show
evidence from benchmarking of organisations (based on research by Terry Cooke-
Davies) indicating that risk management is the single most influential factor in project
success. Furthermore they list key criteria needed to support a consistent, structured
& formalised risk approach:
- A supportive organisation
- Competent people
- Appropriate supporting infrastructure
- A simple-to-use, scalable and documented process.

They also put forward some of the common reasons why risk management is often
paid only token attention in an organisation, including:
- Cost of the process, and of applying the process
- Disbelief in the value of the process
- Lack of time to implement
- It should be common sense

In a previous paper (Gray, PMI Congress 2008) I argued that lessons can play an
active role in risk management through:
- Mitigation – by observing the actions and their results from a previous project
- Avoidance – by observing how risks were overcome (or not, as the case may
be)
 - Identification – lessons from previous projects may help to highlight specific
risks through, for example, a standardised risk list

Building on this and blending in ideas from Hillson & Simon, I would expand the
influence of lessons learned into the positive side as well – lessons from past
projects can and should highlight opportunities in downstream projects.

Lessons as Best Practices

Lessons that are re-used in downstream projects and that have a positive outcome
on results can, and often are, considered as best practices. The Organisational
Project Management Maturity Model (OPM3) standard has produced a set of Best
Practice “elements” that are seen to contribute to the organisation’s maturity. These
are localised around three key questions:
- Do they contribute to an organisation’s project management maturity?
- Can an organisation implement them directly, without prerequisites?
- Are they conducive to performance criteria to measure effectiveness of
implementation

Applying the same criteria to lessons in the sense that lessons are potential Best
Practices can help to identify the value-added elements of LL’s. Re-writing these
three questions in this context gives rise to the following:
- Does the lesson lead to a change to the project management processes or
methodology used by the organisation?
- Can other projects make direct use of these lessons?
- Is there any metric that can be applied to measure the effectiveness of doing
something differently?

Answering the first of these questions is relatively easy – if we identify a systemic

issue in the way of working then the lesson learned should immediately lead to a
process change request, thus helping to ensure this is deployed to downstream
projects. To make this effective would require the involvement of the quality team in
the project debrief to help capture and filter the lessons into this category. The
outcome would be improvement actions on the quality system, probably with a
follow-up review process independent of the source project. This is also an
opportunity to perform selection of lessons or best practices to manage the
complexity of the quality system.

The second question is somewhat less easy – since we have the argument that each
project tends to be unique, we will encounter the issue that the lesson is not
compatible. Therefore it is my opinion that these lessons will be those which are
either applied directly into a new project that is sufficiently similar to allow direct
application, or that they fall into the “Eureka” category. Unfortunately the latter tend
to be somewhat scarce.

The final question – regarding value – is key to the subject of this paper. How can we
identify the value-added of a lesson and how can we ensure we maximise this value.

Kerzner (Harold Kerzner - Project Management, A Systems Approach to Planning,

Scheduling and Controlling – 9th Edition) discusses the variables of success in his
topic on Best Practices. He notes that there are multiple definitions of success
including
- Something that works
- Something that works well
- Something that works well on a repetitive basis
- Something that leads to a competitive advantage
- Something that can be identified in a proposal to generate business

Kerzner also goes on to ask the critical questions of

- Who identifies a best practice?
- How will we know if it really is a best practice?
- Is the best practice recognised by the management as being of value?
- Is the practice transferrable to other projects?
- Does the practice help to avoid future issues?

Different organisations have different approaches to these points but all come back
to the common theme of “usefulness” or “value-added”

Summary of Literature Research

From the research covered so far in this paper, we can draw a few key elements of
lessons:
- Lessons as Knowledge: Lessons can clearly be identified as a type of
transportable knowledge, even if generally this is in the form of experience at
the individual level. This knowledge or intellectual capital can be attributed a
certain value.
- Lessons as Continuous Improvement: For lessons to become learned there
needs to be simple but robust underlying process that promotes the
identification, capturing and sharing of lessons across the organisation. The
lessons that are used to drive improvement in the organisational processes
become part of the organisations experience
- Lessons as Risk Management: Lessons that enable improved risk
management approaches are highly likely to become part of the
organisational knowledge base, supporting a robust approach to dealing with
recognised uncertainties.
- Lessons as Best Practices: Where lessons lead to new opportunities or better
ways of doing things, the organisation shows an agility in dealing with change
and will develop competitive approaches to problem solving

Cases from industry

NASA – Lessons as Knowledge / Risk Management components
From a report by R Gilman (Lessons Learned – A Value Added Product of the
Project Lifecycle, Keane group, 19th April 2006):
- Lessons learned are knowledge or understandings gained from the results of
either a positive or negative experience in the project that are significant to the
project effort and are not part of the (current) business processes.
- Determining a the presence of a lesson involves root-cause analysis of the
incident, evaluation of the impact, assessment of recommended actions and
diffusion of the result.
 - Common root causes are identified as coming from project assumptions,
overlooked issues in the planning process and communication errors –
amongst others.
- Gilman notes three main value elements of lessons learned; Improved
organisational processes, risk mitigation and identification of common best
practice
- Gilman goes on to note that the lessons learned process should be integrated
into the whole project lifecycle (not just left to the closure phase) and that it
requires the active involvement and support of management.
- In closing, Gilman notes that in order to obtain value from lessons learned,
there needs to be an approach that includes:
o Identification of lessons through strong critical thinking skills
o Development of clear improvement recommendations
o Communication
o Recording, storing and dissemination of lessons
o Use of the lessons learned data
o Measurable lessons learned process

It should be noted here that although Gilman gives an example of a lesson learned
and how the reaction effort enabled a scheduled launch to be maintained, he does
not offer a metric for the value of the lesson or the effectiveness of the process – one
would have to draw the conclusion that the data is empirical in nature only. However,
taking the examples noted in the presentation, the main result of lessons learned at
NASA would appear to be in working to reduce the level of risk to the future
missions.

IBM – Lessons as part of Continuous Improvement

Jim De Piante, Executive Project Manager at IBM
At IBM there are various approaches to managing lessons learned, linked largely to
the CMMI level – basically those parts or divisions of the organisation that are at
CMMI Level 5 are in the habit of continuous review and improvement.
Jim draws the parallel between the process of individual learning and organisational
learning.
In short, the learning process can be described in just four steps:
- Exposure to an event or action
- Recognition that there is knowledge to be acquired through learning
- Internalisation and retention of the new knowledge
- Usage of this knowledge in future situations

For the organisation the process is almost identical to that of individual learning
except that the retention part is not in the heads of individuals – individuals are in
effect transient in an organisation.

Drawing from this, Jim’s view is that the only way for an organisation to retain
knowledge (learn from lessons) is to capture them in the organisational processes. In
this way they become institutionalised. If the knowledge is kept only in the heads of
the individuals then, when these individuals leave the organisation, they take the
acquired knowledge with them.
Jim’s recommendations for achieving organisational level improvement is to ensure
that there is support for process improvement , moving the lessons from “ad-hoc”
learning to continuous improvement.

One way in which this can be achieved is through the use, and re-use, of checklists.
In the planning stage of a project, the checklists can help avoid overlooking or
forgetting important items; during the execution phase, checklists can become a key
element of contingency plans. Jim in fact draws the parallel with pre-flight checklists
and in-flight trained responses; Prior to take-off, the pilot and crew run through a
specific set of checks to ensure the safety of the aircraft and crew; if, during the
flight, there is an emergency situation then the crew will split the effort into immediate
response actions based on training and at the same time will run through the
appropriate checklists to identify and manage the problem.
Clearly IBM is using an approach based on continuous improvement – in fact the
application of CMMI processes underscores this point quite neatly.

NXP Semiconductors – Lessons as Best Practice

At NXP semiconductors the approach to lessons learned has been both semi-formal
and ad-hoc;
- Semi-formal in that the organisation established a standardised project
management process which includes requirements for review and reporting of
lessons
- Ad-hoc in that the project managers themselves built cross-division
communities of practice in which to share their experiences.

In the case of the standardised process, there is a clear link to continuous

improvement in the same style as CMMI (in fact a large driver for the process
development came from identified needs in CMMI and ISO9001 audits). During
review of projects, lessons are identified and classified according to impact and
effect. Where there is a clear systemic issue, the reviewing team should make a
recommendation to update the organisational process to deal with this in future
projects. Often, however, the root cause shows clear links to risk elements and the
result is that these lessons are added into standard risk lists – at least for that part of
the organisation running the project under review.

In the case of the ad-hoc approach, the project management communities of practice
(CoP) uses three key tools to identify and share knowledge or lessons;
- Project post-phase reviews are shared with the community, along with any
recommendations for improvement in the project management methodology
- Peer reviews are set up in a “multi-brained project manager” (Hydra)
approach where a PM will call on several peers to review and offer advice on
the planning phase of the project.
- Internal training ranging from formalised training sessions to round-table
discussions on chosen topics are run as workshops on a regular basis
(monthly in most sites)

Although the CoP is a somewhat informal approach, the power of doing this became
evident when the international network of project managers inside NXP were able to
identify and drive significant improvement actions in the organisational project
management processes. In effect, the actors took ownership of the process along
with the responsibility of driving towards excellence.

It should be noted that this approach of CoP is not unique to NXP – Wenger et al
discuss this quite extensively in their book “Cultivating Communities of Practice” ,
with examples from various organisations used for illustration. There is an interesting
argument for calculating the ROI from communities which equates things like the
numeric savings (due to actions coming from the community) factored by the share
of the community’s contribution and a degree of certainty that the community was the
source.

Numeric value * share of community * degree of certainty = attributed saving

reported

At NXP we observed several actions over the last two years where there were
distinct savings made as a direct result of lessons transmitted through the CoP –
with the conclusion that the initiative really does contribute to the organisational
success.

NXP is showing some evidence of identification and application of best practice – in

a somewhat ad-hoc approach but nonetheless with clear results in the direction of
doing things better.

Conclusion
In reviewing the literature and combining this with observed approaches in various
industries, we get the clear message that “the lessons learned process should be
done”. In fact Kerzner has learning and improvement as the highest level of project
management maturity, CMMI has several explicit references to the need to review
lessons learned in improving the processes and various researchers (Robert Cook,
etc) show correlations between the use of project management practices and
organisational success.

Taking this clearly stated need to use robust processes or methods in project
management in conjunction with the supposition that each project is an “unique
endeavour” shows a requirement to continuously assess, review and update our
processes and methodology based on lessons from current a past projects – the
environment changes, we need to adapt with it.

These arguments give a clear indication of the value of having a process in which we
learn lessons from our experiences – but does not show a clear link between “value”
and any specific “lesson”. The only places where we can see a clear and quantifiable
link is in the case where we assess the value as being the differentiation factor on a
risk mitigation action.

In all other cases, the value is not only qualitative in nature, but also tends to be
longer term – for example in process improvement actions. The only counter-
argument to this that I have discovered to date is that, through implementing lessons
learned and CI we improve the overall effectiveness of our organisation by reducing
the cost of non-quality.

This therefore leads me to the conclusion that although there may not be a tangible,
clearly measurable result of a lesson or a lesson learned process; failure to build this
into our ongoing project management approach can significantly contribute to the
possibility of failure at organisational, project and also individual levels. The
additional investment, in terms of time or effort spent to run the process, along with
costs of any supporting tools, is negligible in relation to the overall cost to the
company of possible failure. Conversely, taking an active stance and investing a
small amount of time and energy into the process as part of the day-to-day project
management methodology, can lead to a significant competitive advantage for the
corporation, can secure the project outcomes and can improve the individual project
manager’s competence levels.

Therefore, although the value of the lessons may be un-measurable in anything

other than qualitative terms, the return on the investment in putting in place a robust
lesson learned process could be immeasurable – while the impact of doing nothing
can be easily measured in terms of organisational failure.

References:
• Walter Baets - Knowledge Management and Management Learning, Springer
Press 2005
• Arie de Geus – The Living Company, Habits for Survival in a Turbulent Business
Environment, Harvard Business School Press, 2002
• John Bessant – Technology Management in the UK – A report to the ETMI,
December 1992
• Wenger, McDermott & Snyder – Cultivating Communities of Practice, Harvard
Business School Press
• John Besant, Sarah Caffyn, Maeve Gallagher – An Evolutionary Model of
Continuous Improvement Behaviour; Technovation 21 (2001)
• PMI – PMBoK Guide, 3rd Edition
• David Hillson & Peter Simon – Practical Project Risk Management The ATOM
Methodology, Management Concepts 2007