Data, Information And Knowledge Management Framework And The Data Management Book Of Knowledge (Dmbok)

Structured and
Comprehensive Approach to
Data Management and the
Data Management Book of
Knowledge (DMBOK)

Alan McSweeney

Objectives

• To provide an overview of a structured approach to
developing and implementing a detailed data management
policy including frameworks, standards, project, team and
maturity

March 8, 2010 2

Agenda

• Introduction to Data Management
• State of Information and Data Governance
• Other Data Management Frameworks
• Data Management and Data Management Book of
Knowledge (DMBOK)
• Conducting a Data Management Project
• Creating a Data Management Team
• Assessing Your Data Management Maturity

March 8, 2010 3

Preamble

• Every good presentation should start with quotations from
The Prince and Dilbert

March 8, 2010 4

Management Wisdom

• There is nothing more difficult to take in hand, more perilous to conduct or more
uncertain in its success than to take the lead in the introduction of a new order of
things.
− The Prince

• Never be in the same room as a decision. I'll illustrate my point with a puppet
show that I call "Journey to Blameville" starring "Suggestion Sam" and "Manager
Meg.“
• You will often be asked to comment on things you don't understand. These
handouts contain nonsense phrases that can be used in any situation so, let's
dominate our industry with quality implementation of methodologies.
• Our executives have started their annual strategic planning sessions. This involves
sitting in a room with inadequate data until an illusion of knowledge is attained.
Then we'll reorganise, because that's all we know how to do.
− Dilbert

March 8, 2010 5

Information

• Information in all its forms –
input, processed, outputs – is a
Applications core component of any IT
system
• Applications exist to process
data supplied by users and
other applications
Processes Information
• Data breathes life into
applications
IT Systems
• Data is stored and managed by
infrastructure – hardware and
software
• Data is a key organisation asset
with a substantial value
People Infrastructure • Significant responsibilities are
imposed on organisations in
managing data

March 8, 2010 6

Data, Information and Knowledge

• Data is the representation of facts as text, numbers, graphics,
images, sound or video
• Data is the raw material used to create information
• Facts are captured, stored, and expressed as data
• Information is data in context
• Without context, data is meaningless - we create meaningful
information by interpreting the context around data
• Knowledge is information in perspective, integrated into a viewpoint
based on the recognition and interpretation of patterns, such as
trends, formed with other information and experience
• Knowledge is about understanding the significance of information
• Knowledge enables effective action

March 8, 2010 7

Data, Information, Knowledge and Action

Knowledge Action

Information
Data

March 8, 2010 8

Information is an Organisation Asset

• Tangible organisation assets are seen as having a value and
are managed and controlled using inventory and asset
management systems and procedures
• Data, because it is less tangible, is less widely perceived as
a real asset, assigned a real value and managed as if it had
a value
• High quality, accurate and available information is a pre-
requisite to effective operation of any organisation

March 8, 2010 9

Data Management and Project Success

• Data is fundamental to the effective and efficient
operation of any solution
− Right data
− Right time
− Right tools and facilities
• Without data the solution has no purpose
• Data is too often overlooked in projects
• Project managers frequently do not appreciate the
complexity of data issues

March 8, 2010 10

Generalised Information Management Lifecycle

Enter, Create, Acquire, • Generalised lifecycle that
Derive, Update, Capture
differs for specific
information types
Store, Manage, M
an
Replicate and Distribute ag
e,
Co
nt
ro
la
nd
Ad
Protect and Recover mi
n is
t er

• Design, define and implement
framework to manage Archive and Recall
information through this
lifecycle
Delete/Remove

March 8, 2010 11

Expanded Generalised Information Management
Lifecycle
Plan, Design and
Specify
De
Implement sig
Underlying n,
Im
Infrastructure ple
m en
Enter, Create, t, M
Acquire, Derive, an
ag
Update, Capture e,
Co
nt
Store, Manage, ro
la
Replicate and nd
Distribute Ad
mi
ni ste
r
• Include phases for information Protect and Recover
management lifecycle design
and implementation of Archive and Recall
appropriate hardware and
software to actualise lifecycle
Delete/Remove

March 8, 2010 12

Data and Information Management

• Data and information management is a business process
consisting of the planning and execution of policies,
practices, and projects that acquire, control, protect,
deliver, and enhance the value of data and information
assets

March 8, 2010 13

Data and Information Management

To manage and utilise information as a strategic asset

To implement processes, policies, infrastructure and solutions to
govern, protect, maintain and use information

To make relevant and correct information available in all business
processes and IT systems for the right people in the right context at
the right time with the appropriate security and with the right
quality

To exploit information in business decisions, processes and
relations

March 8, 2010 14

Data Management Goals

• Primary goals
− To understand the information needs of the enterprise and all its
stakeholders
− To capture, store, protect, and ensure the integrity of data assets
− To continually improve the quality of data and information,
including accuracy, integrity, integration, relevance and
usefulness of data
− To ensure privacy and confidentiality, and to prevent
unauthorised inappropriate use of data and information
− To maximise the effective use and value of data and information
assets

March 8, 2010 15

Data Management Goals

• Secondary goals
− To control the cost of data management
− To promote a wider and deeper understanding of the value of
data assets
− To manage information consistently across the enterprise
− To align data management efforts and technology with business
needs

March 8, 2010 16

Triggers for Data Management Initiative

• When an enterprise is about to undertake architectural
transformation, data management issues need to be
understood and addressed
• Structured and comprehensive approach to data
management enables the effective use of data to take
advantage of its competitive advantages

March 8, 2010 17

Data Management Principles

• Data and information are valuable enterprise assets
• Manage data and information carefully, like any other
asset, by ensuring adequate quality, security, integrity,
protection, availability, understanding and effective use
• Share responsibility for data management between
business data owners and IT data management
professionals
• Data management is a business function and a set of
related disciplines

March 8, 2010 18

Organisation Data Management Function

• Business function of planning for, controlling and
delivering data and information assets
• Development, execution, and supervision of plans,
policies, programs, projects, processes, practices and
procedures that control, protect, deliver, and enhance the
value of data and information assets
• Scope of the data management function and the scale of
its implementation vary widely with the size, means, and
experience of organisations
• Role of data management remains the same across
organisations even though implementation differs widely
March 8, 2010 19

Scope of Complete Data Management Function

Data Management

Data Governance Data Architecture Management

Data Development Data Operations Management

Data Security Management Data Quality Management

Reference and Master Data Data Warehousing and Business
Management Intelligence Management

Document and Content Management Metadata Management

March 8, 2010 20

Shared Role Between Business and IT

• Data management is a shared responsibility between data
management professionals within IT and the business data
owners representing the interests of data producers and
information consumers
• Business data ownership is the concerned with
accountability for business responsibilities in data
management
• Business data owners are data subject matter experts
• Represent the data interests of the business and take
responsibility for the quality and use of data

March 8, 2010 21

Why Develop and Implement a Data Management
Framework?
• Improve organisation data management efficiency
• Deliver better service to business
• Improve cost-effectiveness of data management
• Match the requirements of the business to the management of the
data
• Embed handling of compliance and regulatory rules into data
management framework
• Achieve consistency in data management across systems and
applications
• Enable growth and change more easily
• Reduce data management and administration effort and cost
• Assist in the selection and implementation of appropriate data
management solutions
• Implement a technology-independent data architecture
March 8, 2010 22

Data Management Issues

March 8, 2010 23

Data Management Issues

• Discovery - cannot find the right information
• Integration - cannot manipulate and combine information
• Insight - cannot extract value and knowledge from
information
• Dissemination - cannot consume information
• Management – cannot manage and control information
volumes and growth

March 8, 2010 24

Data Management Problems – User View

• Managing Storage Equipment
• Application Recoveries / Backup Retention
• Vendor Management
• Power Management
• Regulatory Compliance
• Lack of Integrated Tools
• Dealing with Performance Problems
• Data Mobility
• Archiving and Archive Management
• Storage Provisioning
• Managing Complexity
• Managing Costs
• Backup Administration and Management
• Proper Capacity Forecasting and Storage Reporting
• Managing Storage Growth
March 8, 2010 25

Information Management Challenges

• Explosive Data Growth
− Value and volume of data is overwhelming
− More data is see as critical
− Annual rate of 50+% percent
• Compliance Requirements
− Compliance with stringent regulatory requirements and audit
procedures
• Fragmented Storage Environment
− Lack of enterprise-wide hardware and software data storage
strategy and discipline
• Budgets
− Frozen or being cut

March 8, 2010 26

Data Quality

• Poor data quality costs real money
• Process efficiency is negatively impacted by poor data
quality
• Full potential benefits of new systems not be realised
because of poor data quality
• Decision making is negatively affected by poor data quality

March 8, 2010 27

State of Information and Data Governance

• Information and Data Governance Report, April 2008
− International Association for Information and Data Quality (IAIDQ)
− University of Arkansas at Little Rock, Information Quality Program
(UALR-IQ)

March 8, 2010 28

Your Organisation Recognises and Values Information as a
Strategic Asset and Manages it Accordingly

Strongly Disagree 3.4%

Disagree 21.5%

Neutral 17.1%

Agree 39.5%

Strongly Agree 18.5%

0% 10% 20% 30% 40% 50%

March 8, 2010 29

Direction of Change in the Results and Effectiveness of the
Organisation's Formal or Informal Information/Data
Governance Processes Over the Past Two Years

Results and Effectiveness Have Significantly
8.8%
Improved

Results and Effectiveness Have Improved 50.0%

Results and Effectiveness Have Remained
31.9%
Essentially the Same

Results and Effectiveness Have Worsened 3.9%

Results and Effectiveness Have Significantly
0.0%
Worsened

Don’t Know 5.4%

0% 10% 20% 30% 40% 50% 60% 70%

March 8, 2010 30

Perceived Effectiveness of the Organisation's Current
Formal or Informal Information/Data Governance Processes

Excellent (All Goals are
2.5%
Met)

Good (Most Goals are
21.1%
Met)

OK (Some Goals are Met) 51.5%

Poor (Few Goals are Met) 19.1%

Very Poor (No Goals are
3.9%
Met)

Don’t Know 2.0%

0% 10% 20% 30% 40% 50% 60% 70%

March 8, 2010 31

Actual Information/Data Governance Effectiveness
vs. Organisation's Perception

It is Better Than Most
20.1%
People Think

It is the Same as Most
32.4%
People Think

It is Worse Than Most
35.8%
People Think

Don’t Know 11.8%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

March 8, 2010 32

Current Status of Organisation's Information/Data
Governance Initiatives
Started an Information/Data Governance Initiative, but
1.5%
Discontinued the Effort
Considered a Focused Information/Data Governance
0.5%
Effort but Abandoned the Idea

None Being Considered - Keeping the Status Quo 7.4%

Exploring, Still Seeking to Learn More 20.1%

Evaluating Alternative Frameworks and Information
23.0%
Governance Structures

Now Planning an Implementation 13.2%

First Iteration Implemented the Past 2 Years 19.1%

First Interation"in Place for More Than 2 Years 8.8%

Don’t Know 6.4%

0% 5% 10% 15% 20% 25% 30%

March 8, 2010 33

Expected Changes in Organisation's Information/Data
Governance Efforts Over the Next Two Years

Will Increase Significantly 46.6%

Will Increase Somewhat 39.2%

Will Remain the Same 10.8%

Will Decrease Somewhat 1.0%

Will Decrease Significantly 0.5%

Don’t Know 2.0%

0% 10% 20% 30% 40% 50% 60%
March 8, 2010 34

Overall Objectives of Information / Data Governance
Efforts
Improve Data Quality 80.2%

Establish Clear Decision Rules and Decisionmaking
65.6%
Processes for Shared Data

Increase the Value of Data Assets 59.4%

Provide Mechanism to Resolve Data Issues 56.8%

Involve Non-IT Personnel in Data Decisions IT Should
55.7%
not Make by Itself
Promote Interdependencies and Synergies Between
49.6%
Departments or Business Units

Enable Joint Accountability for Shared Data 45.3%

Involve IT in Data Decisions non-IT Personnel Should
35.4%
not Make by Themselves

Other 5.2%

None Applicable 1.0%

Don't Know 2.6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100
%
March 8, 2010 35

Change In Organisation's Information / Data Quality
Over the Past Two Years
Information / Data Quality
10.5%
Has Significantly Improved

68.4%
Has Improved

Has Remained Essentially 15.8%
the Same

3.5%
Has Worsened

0.0%
Has Significantly Worsened

Don’t Know 1.8%

0% 10% 20% 30% 40% 50% 60% 70% 80%

March 8, 2010 36

Maturity Of Information / Data Governance Goal
Setting And Measurement In Your Organisation

5 - Optimised 3.7%

4 - Managed 11.8%

3 - Defined 26.7%

2 - Repeatable 28.9%

1 - Ad-hoc 28.9%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

March 8, 2010 37

Maturity Of Information / Data Governance
Processes And Policies In Your Organisation
5 - Optimised 1.6%

4 - Managed 4.8%

3 - Defined 24.5%


1 - Ad-hoc 22.9%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

March 8, 2010 38

Maturity Of Responsibility And Accountability For
Information / Data Governance Among Employees In Your
Organisation
5 - Optimised 6.9%

4 - Managed 3.2%

3 - Defined 31.7%


1 - Ad-hoc 32.8%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

March 8, 2010 39

Other Data Management Frameworks

March 8, 2010 40

Other Data Management-Related Frameworks

• TOGAF (and other enterprise architecture standards) define a
process for arriving an at enterprise architecture definition, including
data
• TOGAF has a phase relating to data architecture
• TOGAF deals with high level
• DMBOK translates high level into specific details
• COBIT is concerned with IT governance and controls:
− IT must implement internal controls around how it operates
− The systems IT delivers to the business and the underlying business processes
these systems actualise must be controlled – these are controls external to IT
− To govern IT effectively, COBIT defines the activities and risks within IT that
need to be managed
• COBIT has a process relating to data management
• Neither TOGAF nor COBIT are concerned with detailed data
management design and implementation

March 8, 2010 41

DMBOK, TOGAF and COBIT
Can be a DMBOK Is a Specific and
Precursor to Comprehensive Data
Implementing Oriented Framework
Data
Management DMBOK Provides Detailed
for Definition,
Implementation and
TOGAF Defines the Process Operation of Data
for Creating a Data Management and Utilisation
Architecture as Part of an
Overall Enterprise
Architecture
Can Provide a Maturity
Model for Assessing
Data Management

COBIT Provides Data
Governance as Part of
Overall IT Governance

March 8, 2010 42

DMBOK, TOGAF and COBIT – Scope and Overlap
DMBOK
Data Development
Data Operations Management
Reference and Master Data Management
Data Warehousing and Business Intelligence Management
TOGAF Document and Content Management
Metadata Management
Data Quality Management

Data Architecture Management
Data Management
Data Migration

Data
Governance
Data Security COBIT
Management

March 8, 2010 43

TOGAF and Data Management
• Phase C1 (subset of
Phase C) relates to
Phase A:
Architecture defining a data
Vision
Phase H:
Phase B:
architecture
Architecture
Business
Change
Architecture
Management
Phase C1:
Data
Architecture
Phase G: Phase C:
Requirements Information
Implementation
Management Systems
Governance Architecture
Phase C2:
Solutions and
Application
Phase F: Phase D: Architecture
Migration Technology
Planning Architecture
Phase E:
Opportunities
and Solutions

March 8, 2010 44

TOGAF Phase C1: Information Systems Architectures
- Data Architecture - Objectives
• Purpose is to define the major types and sources of data
necessary to support the business, in a way that is:
− Understandable by stakeholders
− Complete and consistent
− Stable
• Define the data entities relevant to the enterprise
• Not concerned with design of logical or physical storage
systems or databases

March 8, 2010 45

- Data Architecture - Overview
Phase C1: Information Systems
Architectures - Data Architecture

Approach Elements Inputs Steps Outputs

Key Considerations for Data Reference Materials External to the Select Reference Models,
Architecture Enterprise Viewpoints, and Tools

Develop Baseline Data Architecture
Architecture Repository Non-Architectural Inputs
Description

Develop Target Data Architecture
Architectural Inputs
Description

Perform Gap Analysis

Define Roadmap Components

Resolve Impacts Across the
Architecture Landscape

Conduct Formal Stakeholder
Review

Finalise the Data Architecture

Create Architecture Definition
Document
March 8, 2010 46

TOGAF Phase C1: Information Systems Architectures - Data
Architecture - Approach - Key Considerations for Data
Architecture
• Data Management
− Important to understand and address data management issues
− Structured and comprehensive approach to data management enables the
effective use of data to capitalise on its competitive advantages
− Clear definition of which application components in the landscape will serve as
the system of record or reference for enterprise master data
− Will there be an enterprise-wide standard that all application components,
including software packages, need to adopt
− Understand how data entities are utilised by business functions, processes, and
services
− Understand how and where enterprise data entities are created, stored,
transported, and reported
− Level and complexity of data transformations required to support the
information exchange needs between applications
− Requirement for software in supporting data integration with external
organisations

March 8, 2010 47

Architecture
• Data Migration
− Identify data migration requirements and also provide indicators
as to the level of transformation for new/changed applications
− Ensure target application has quality data when it is populated
− Ensure enterprise-wide common data definition is established to
support the transformation

March 8, 2010 48

Architecture
• Data Governance
− Ensures that the organisation has the necessary dimensions in
place to enable the data transformation
− Structure – ensures the organisation has the necessary structure
and the standards bodies to manage data entity aspects of the
transformation
− Management System - ensures the organisation has the
necessary management system and data-related programs to
manage the governance aspects of data entities throughout its
lifecycle
− People - addresses what data-related skills and roles the
organisation requires for the transformation

March 8, 2010 49

- Data Architecture - Outputs
• Refined and updated versions of the Architecture Vision phase deliverables
− Statement of Architecture Work
− Validated data principles, business goals, and business drivers
• Draft Architecture Definition Document
− Baseline Data Architecture
− Target Data Architecture
• Business data model
• Logical data model
• Data management process models
• Data Entity/Business Function matrix
• Views corresponding to the selected viewpoints addressing key stakeholder concerns
− Draft Architecture Requirements Specification
• Gap analysis results
• Data interoperability requirements
• Relevant technical requirements
• Constraints on the Technology Architecture about to be designed
• Updated business requirements
• Updated application requirements
− Data Architecture components of an Architecture Roadmap
March 8, 2010 50

COBIT Structure
COBIT

Plan and Organise (PO) Acquire and Implement (AI) Deliver and Support (DS) Monitor and Evaluate (ME)

DS1 Define and manage service ME1 Monitor and evaluate IT
PO1 Define a strategic IT plan AI1 Identify automated solutions
levels performance

PO2 Define the information AI2 Acquire and maintain ME2 Monitor and evaluate
DS2 Manage third-party services
architecture application software internal control

PO3 Determine technological AI3 Acquire and maintain DS3 Manage performance and ME3 Ensure regulatory
direction technology infrastructure capacity compliance

PO4 Define the IT processes,
AI4 Enable operation and use DS4 Ensure continuous service ME4 Provide IT governance
organisation and relationships

PO5 Manage the IT investment AI5 Procure IT resources DS5 Ensure systems security

PO6 Communicate management
AI6 Manage changes DS6 Identify and allocate costs
aims and direction

AI7 Install and accredit solutions
PO7 Manage IT human resources DS7 Educate and train users
and changes

DS8 Manage service desk and
PO8 Manage quality
incidents

PO9 Assess and manage IT risks DS9 Manage the configuration

PO10 Manage projects DS10 Manage problems

DS11 Manage data
DS12 Manage the physical
environment

DS13 Manage operations

March 8, 2010 51

COBIT and Data Management

• COBIT objective DS11 Manage Data within the Deliver and
Support (DS) domain
• Effective data management requires identification of data
requirements
• Data management process includes establishing effective
procedures to manage the media library, backup and
recovery of data and proper disposal of media
• Effective data management helps ensure the quality,
timeliness and availability of business data

March 8, 2010 52

COBIT and Data Management

• Objective is the control over the IT process of managing data that
meets the business requirement for IT of optimising the use of
information and ensuring information is available as required
• Focuses on maintaining the completeness, accuracy, availability and
protection of data
• Involves taking actions
− Backing up data and testing restoration
− Managing onsite and offsite storage of data
− Securely disposing of data and equipment
• Measured by
− User satisfaction with availability of data
− Percent of successful data restorations
− Number of incidents where sensitive data were retrieved after media were
disposed of

March 8, 2010 53

COBIT Process DS11 Manage Data
• DS11.1 Business Requirements for Data Management
− Establish arrangements to ensure that source documents expected from the business are received, all data received from the
business are processed, all output required by the business is prepared and delivered, and restart and reprocessing needs are
supported
• DS11.2 Storage and Retention Arrangements
− Define and implement procedures for data storage and archival, so data remain accessible and usable
− Procedures should consider retrieval requirements, cost-effectiveness, continued integrity and security requirements
− Establish storage and retention arrangements to satisfy legal, regulatory and business requirements for documents, data, archives,
programmes, reports and messages (incoming and outgoing) as well as the data (keys, certificates) used for their encryption and
authentication
• DS11.3 Media Library Management System
− Define and implement procedures to maintain an inventory of onsite media and ensure their usability and integrity
− Procedures should provide for timely review and follow-up on any discrepancies noted
• DS11.4 Disposal
− Define and implement procedures to prevent access to sensitive data and software from equipment or media when they are
disposed of or transferred to another use
− Procedures should ensure that data marked as deleted or to be disposed cannot be retrieved.
• DS11.5 Backup and Restoration
− Define and implement procedures for backup and restoration of systems, data and documentation in line with business
requirements and the continuity plan
− Verify compliance with the backup procedures, and verify the ability to and time required for successful and complete restoration
− Test backup media and the restoration process
• DS11.6 Security Requirements for Data Management
− Establish arrangements to identify and apply security requirements applicable to the receipt, processing, physical storage and
output of data and sensitive messages
− Includes physical records, data transmissions and any data stored offsite

March 8, 2010 54

COBIT Data Management Goals and Metrics
Activity Goals Process Goals Activity Goals

•Backing up data and testing •Maintain the completeness, •Backing up data and testing
restoration accuracy, validity and restoration
•Managing onsite and offsite accessibility of stored data •Managing onsite and offsite
storage of data •Secure data during disposal storage of data
•Securely disposing of data of media •Securely disposing of data
and equipment •Effectively manage storage and equipment
media

Are Measured Are Measured Are Measured
By Drive By Drive By

Key Performance Process Key Goal IT Key Goal Indicators
Indicators Indicators
•% of successful data •Occurrences of inability to
restorations recover data critical to
•Frequency of testing of •# of incidents where business process
backup media sensitive data were retrieved •User satisfaction with
•Average time for data after media were disposed of availability of data
restoration •# of down time or data •Incidents of noncompliance
integrity incidents caused by with laws due to storage
insufficient storage capacity management issues

March 8, 2010 55

Data Management Book of Knowledge (DMBOK)

March 8, 2010 56


• DMBOK is a generalised and comprehensive framework for
managing data across the entire lifecycle
• Developed by DAMA (Data Management Association)
• DMBOK provides a detailed framework to assist
development and implementation of data management
processes and procedures and ensures all requirements
are addressed
• Enables effective and appropriate data management
across the organisation
• Provides awareness and visibility of data management
issues and requirements
March 8, 2010 57


• Not a solution to your data management needs
• Framework and methodology for developing and
implementing an appropriate solution
• Generalised framework to be customised to meet specific
needs
• Provide a work breakdown structure for a data
management project to allow the effort to be assessed
• No magic bullet

March 8, 2010 58

Scope and Structure of Data Management Book of
Knowledge (DMBOK)

Data Management
Environmental Elements

Data
Management
Functions

March 8, 2010 59

DMBOK Data Management Functions
Data Management
Functions

Data Governance Data Architecture Management

Data Development Data Operations Management

Data Security Management Data Quality Management

Data Warehousing and Business
Intelligence Management

Document and Content Management Metadata Management

March 8, 2010 60

DMBOK Data Management Functions

• Data Governance - planning, supervision and control over data management and
use
• Data Architecture Management - defining the blueprint for managing data assets
• Data Development - analysis, design, implementation, testing, deployment,
maintenance
• Data Operations Management - providing support from data acquisition to
purging
• Data Security Management - Ensuring privacy, confidentiality and appropriate
access
• Data Quality Management - defining, monitoring and improving data quality
• Reference and Master Data Management - managing master versions and
replicas
• Data Warehousing and Business Intelligence Management - enabling reporting
and analysis
• Document and Content Management - managing data found outside of databases
• Metadata Management - integrating, controlling and providing metadata

March 8, 2010 61

DMBOK Data Management Environmental Elements
Data Management

Goals and Principles Activities

Primary Deliverables Roles and Responsibilities

Practices and Techniques Technology

Organisation and Culture

March 8, 2010 62

DMBOK Data Management Environmental Elements

• Goals and Principles - directional business goals of each function and the fundamental
principles that guide performance of each function
• Activities - each function is composed of lower level activities, sub-activities, tasks and
steps
• Primary Deliverables - information and physical databases and documents created as
interim and final outputs of each function. Some deliverables are essential, some are
generally recommended, and others are optional depending on circumstances
• Roles and Responsibilities - business and IT roles involved in performing and supervising
the function, and the specific responsibilities of each role in that function. Many roles will
participate in multiple functions
• Practices and Techniques - common and popular methods and procedures used to perform
the processes and produce the deliverables and may also include common conventions,
best practice recommendations, and alternative approaches without elaboration
• Technology - categories of supporting technology such as software tools, standards and
protocols, product selection criteria and learning curves
• Organisation and Culture – this can include issues such as management metrics, critical
success factors, reporting structures, budgeting, resource allocation issues, expectations
and attitudes, style, cultural, approach to change management

March 8, 2010 63

DMBOK Data Management Functions and
Goals and Activities Primary Roles and Practices and Technology Organisation
Principles Deliverables Responsibilities Techniques and Culture
Data
Governance
Data
Architecture
Management
Data
Development
Data
Operations
Management
Scope of Each Data Management Function
Data Security
Management
Data Quality
Management
Reference and
Master Data
Management
Data
Warehousing
and Business
Intelligence
Management
Document and
Content
Management
Metadata
Management
March 8, 2010 64

Scope of Data Management Book of Knowledge
(DMBOK) Data Management Framework
• Hierarchy
− Function
• Activity
− Sub-Activity (not in all cases)
• Each activity is classified as one (or more) of:
− Planning Activities (P)
• Activities that set the strategic and tactical course for other data management
activities
• May be performed on a recurring basis
− Development Activities (D)
• Activities undertaken within implementation projects and recognised as part of the
systems development lifecycle (SDLC), creating data deliverables through analysis,
design, building, testing, preparation, and deployment
− Control Activities (C)
• Supervisory activities performed on an on-going basis
− Operational Activities (O)
• Service and support activities performed on an ongoing basis

March 8, 2010 65

Activity Groups Within Functions

• Activity groups are
classifications of data
management
Planning Development
activities
Activities Activities • Use the activity
groupings to define
the scope of data
management sub-
projects and identify
the appropriate tasks:
Control Operational
Activities − Analysis and design
Activities
− Implementation
− Operational
improvement
− Management and
administration

March 8, 2010 66

DMBOK Function and Activity Structure
Data
Management

Reference and Document and
Data Architecture Data Operations Data Security Data Quality DW and BI Metadata
Data Governance Data Development Master Data Content
Management Management Management Management Management Management
Management Management

Understand Data
Data Modeling, Develop and Promote Understand Reference Understand Business
Data Management Understand Enterprise Security Needs and Documents / Records Understand Metadata
Analysis, and Solution Database Support Data Quality and Master Data Intelligence
Planning Information Needs Regulatory Management Requirements
Design Awareness Integration Needs Information Needs
Requirements

Identify Master and
Develop and Maintain Define and Maintain
Data Management Data Technology Define Data Security Define Data Quality Reference Data Define the Metadata
the Enterprise Data Detailed Data Design the DW / BI Content Management
Control Management Policy Requirement Sources and Architecture
Model Architecture
Contributors

Analyse and Align Data Model and Define and Maintain Implement Data
Define Data Security Profile, Analyse, and Develop and Maintain
With Other Business Design Quality the Data Integration Warehouses and Data
Standards Assess Data Quality Metadata Standards
Models Management Architecture Marts

Implement Reference
Define and Maintain Define Data Security Implement a Managed
Define Data Quality and Master Data Implement BI Tools
the Database Data Implementation Controls and Metadata
Metrics Management and User Interfaces
Architecture Procedures Environment
Solutions

Define and Maintain Manage Users,
Define Data Quality Define and Maintain Process Data for Create and Maintain
the Data Integration Passwords, and Group
Business Rules Match Rules Business Intelligence Metadata
Architecture Membership

Define and Maintain Monitor and Tune
Manage Data Access Test and Validate Data Establish “Golden”
the DW / BI Data Warehousing Integrate Metadata
Views and Permissions Quality Requirements Records
Architecture Processes

Define and Maintain Monitor User Define and Maintain Monitor and Tune BI
Set and Evaluate Data Manage Metadata
Enterprise Taxonomies Authentication and Hierarchies and Activity and
Quality Service Levels Repositories
and Namespaces Access Behaviour Affiliations Performance

Define and Maintain Continuously Measure Plan and Implement
Classify Information Distribute and Deliver
the Metadata and Monitor Data Integration of New
Confidentiality Metadata
Architecture Quality Data Sources

Replicate and
Manage Data Quality Query, Report, and
Audit Data Security Distribute Reference
Issues Analyse Metadata
and Master Data

Clean and Correct Data Manage Changes to
Quality Defects Reference and Master
Data

Design and Implement
Operational DQM
Procedures

Monitor Operational
DQM Procedures and
Performance
March 8, 2010 67

DMBOK Function and Activity - Planning Activities
Data
Management

Understand Data Understand
Understand Data Modeling, Develop and Promote Understand Business Understand
Data Management Security Needs and Reference and Documents / Records
Enterprise Analysis, and Database Support Data Quality Intelligence Metadata
Planning Regulatory Master Data Management
Information Needs Solution Design Awareness Information Needs Requirements
Requirements Integration Needs
Develop and Identify Master and
Define and Maintain
Data Management Maintain the Data Technology Define Data Security Define Data Quality Reference Data Content Define the Metadata
Detailed Data Design the DW / BI
Control Enterprise Data Management Policy Requirement Sources and Management Architecture
Architecture
Model Contributors

Analyse and Align Data Model and Define and Maintain Implement Data Develop and
Define Data Security Profile, Analyse, and
With Other Business Design Quality the Data Integration Warehouses and Maintain Metadata
Standards Assess Data Quality
Models Management Architecture Data Marts Standards

Implement Reference
Define and Maintain Define Data Security Implement a
the Database Data Implementation Controls and Managed Metadata
Solutions

the Data Integration Passwords, and
Architecture Group Membership

Define and Maintain Manage Data Access Test and Validate Monitor and Tune
Establish “Golden”
the DW / BI Views and Data Quality Data Warehousing Integrate Metadata
Records
Architecture Permissions Requirements Processes

Define and Maintain
Monitor User Set and Evaluate Define and Maintain Monitor and Tune BI
Enterprise Manage Metadata
Authentication and Data Quality Service Hierarchies and Activity and
Taxonomies and Repositories
Access Behaviour Levels Affiliations Performance
Namespaces

Define and Maintain Continuously Plan and Implement
Classify Information Distribute and
the Metadata Measure and Monitor Integration of New
Confidentiality Deliver Metadata
Architecture Data Quality Data Sources

Replicate and
and Master Data

Clean and Correct Manage Changes to
Data Quality Defects Reference and
Master Data

Design and
Implement
Operational DQM
Procedures

Monitor Operational
DQM Procedures and
Performance

March 8, 2010 68

DMBOK Function and Activity - Control Activities
Data
Management


Understand Data
Requirements

Identify Master and
Model Architecture
Contributors


Implement Reference
Solutions





Replicate and
and Master Data

Data

Operational DQM
Procedures

Monitor Operational
DQM Procedures and
Performance
March 8, 2010 69

DMBOK Function and Activity - Development
Activities Data
Management


Understand Data
Requirements

Identify Master and
Model Architecture
Contributors


Implement Reference
Solutions





Replicate and
and Master Data

Data

Operational DQM
Procedures

Monitor Operational
DQM Procedures and
Performance
March 8, 2010 70

DMBOK Function and Activity - Operational
Activities Data
Management


Understand Data
Requirements

Identify Master and
Model Architecture
Contributors


Implement Reference
Solutions





Replicate and
and Master Data

Data

Operational DQM
Procedures

Monitor Operational
DQM Procedures and
Performance
March 8, 2010 71

DMBOK Environmental Elements Structure
Data Management
Environmental
Elements

Goals and Primary Roles and Practices and Organisation and
Activities Technology
Principles Deliverables Responsibilities Techniques Culture

Phases. Tasks, Inputs and Recognised Best Critical Success
Vision and Mission Individual Roles Tool Categories
Steps Outputs Practices Factors

Standards and Common Reporting
Business Benefits Dependencies Information Organisation Roles
Protocols Approaches Structures

Sequence and Business and IT Alternative Management
Strategic Goals Documents Selection Criteria
Flow Roles Techniques Metrics

Use Cases and Qualifications and Values, Beliefs,
Specific Objectives Databases Learning Curves
Scenarios Skills Expectations

Attitudes. Styles,
Guiding Principles Trigger Events Other Resources
Preferences

Teamwork, Group
Dynamics,
Authority,
Empowerment.

Contracting
Strategies

Change
Management
Approach
March 8, 2010 72

DMBOK Environmental Elements

March 8, 2010 73

Data Governance

March 8, 2010 74

Data Governance

• Core function of the Data Management Framework
• Interacts with and influences each of the surrounding ten data
management functions
• Data governance is the exercise of authority and control (planning,
monitoring, and enforcement) over the management of data assets
• Data governance function guides how all other data management
functions are performed
• High-level, executive data stewardship
• Data governance is not the same thing as IT governance
• Data governance is focused exclusively on the management of data
assets

March 8, 2010 75

Data Governance – Definition and Goals

• Definition
− The exercise of authority and control (planning, monitoring, and
enforcement) over the management of data assets
• Goals
− To define, approve, and communicate data strategies, policies,
standards, architecture, procedures, and metrics
− To track and enforce regulatory compliance and conformance to
data policies, standards, architecture, and procedures
− To sponsor, track, and oversee the delivery of data management
projects and services
− To manage and resolve data related issues
− To understand and promote the value of data assets

March 8, 2010 76

Data Governance - Overview
Inputs Primary Deliverables

•Business Goals •Data Policies
•Business Strategies •Data Standards
•IT Objectives •Resolved Issues
•IT Strategies •Data Management Projects and
•Data Needs Services
•Data Issues •Quality Data and Information
•Regulatory Requirements •Recognised Data Value

Suppliers Data Governance Consumers

•Business Executives •Data Producers
•IT Executives •Knowledge Workers
•Data Stewards •Managers and Executives
•Regulatory Bodies •Data Professionals
•Customers

Participants Tools Metrics

•Executive Data Stewards •Intranet Website •Data Value
•Coordinating Data Stewards •E-Mail •Data Management Cost
•Business Data Stewards •Metadata Tools •Achievement of Objectives
•Data Professionals •Metadata Repository •# of Decisions Made
•DM Executive •Issue Management Tools •Steward Representation / Coverage
•CIO •Data Governance KPI •Data Professional Headcount
•Dashboard •Data Management Process Maturity

March 8, 2010 77

Data Governance Function, Activities and Sub-
Activities
Data Governance

Data Management Planning Data Management Control

Supervise Data Professional Organisations
Understand Strategic Enterprise Data Needs
and Staff

Develop and Maintain the Data Strategy Coordinate Data Governance Activities

Establish Data Professional Roles and
Manage and Resolve Data Related Issues
Organisations

Identify and Appoint Data Stewards Monitor and Ensure Regulatory Compliance

Establish Data Governance and Stewardship Monitor and Enforce Conformance with Data
Organisations Policies, Standards and Architecture

Develop and Approve Data Policies, Oversee Data Management Projects and
Standards, and Procedures Services

Communicate and Promote the Value of Data
Review and Approve Data Architecture
Assets

Plan and Sponsor Data Management Projects
and Services

Estimate Data Asset Value and Associated
Costs

March 8, 2010 78

Data Governance

• Data governance is accomplished most effectively as an
on-going program and a continual improvement process
• Every data governance programme is unique, taking into
account distinctive organisational and cultural issues, and
the immediate data management challenges and
opportunities
• Data governance is at the core of managing data assets

March 8, 2010 79

Data Governance - Possible Organisation Structure

Data Governance Structure

Organisation Data Governance
CIO
Council

Data Governance Office Data Management Executive

Business Unit Data Governance
Data Technologists
Councils

Data Stewardship Committees

Data Stewardship Teams

March 8, 2010 80

Data Governance Shared Decision Making
Business Decisions Shared Decision Making IT Decisions

Enterprise
Business Operating Enterprise Information Database
Model Information Model Management Architecture
Strategy
Enterprise
Information Needs Information Data Integration
IT Leadership Management Architecture
Policies
Enterprise Data Warehousing
Information Information and Business
Capital Investments Specifications Management Intelligence
Standards Architecture

Research and Enterprise
Quality Information Metadata
Development Requirements Management Architecture
Funding Metrics

Enterprise
Data Governance Issue Resolution Information Technical Metadata
Model Management
Services

March 8, 2010 81

Data Stewardship

• Formal accountability for business responsibilities ensuring effective
control and use of data assets
• Data steward is a business leader and/or recognised subject matter
expert designated as accountable for these responsibilities
• Manage data assets on behalf of others and in the best interests of
the organisation
• Represent the data interests of all stakeholders, including but not
limited to, the interests of their own functional departments and
divisions
• Protects, manages, and leverages the data resources
• Must take an enterprise perspective to ensure the quality and
effective use of enterprise data

March 8, 2010 82

Data Stewardship - Roles

• Executive Data Stewards – provide data governance and
make of high-level data stewardship decisions
• Coordinating Data Stewards - lead and represent teams of
business data stewards in discussions across teams and
with executive data stewards
• Business Data Stewards - subject matter experts work
with data management professionals on an ongoing basis
to define and control data

March 8, 2010 83

Data Stewardship Roles Across Data Management
Functions - 1
All Data Stewards Executive Data Stewards Coordinating Data Business Data Stewards
Stewards
Data Architecture Review, validate, approve, Review and approve the Integrate specifications, Define data requirements
Management maintain and refine data enterprise data resolving differences specifications
architecture architecture
Data Development Validate physical data Define data requirements
models and database and specifications
designs, participate in
database testing and
conversion
Data Operations Define requirements for
Management data recovery, retention
and performance
Help identify, acquire, and
control externally sourced
data
Data Security Management Provide security, privacy
and confidentiality
requirements, identify and
resolve data security
issues, assist in data
security audits, and classify
information confidentiality
Reference and Master Data Control the creation,
Management update, and retirement of
code values and other
reference data, define
master data management
requirements, identify and
help resolve issues

March 8, 2010 84

Data Stewardship Roles Across Data Management
Functions - 2
All Data Stewards Executive Data Stewards Coordinating Data Business Data Stewards
Stewards
Data Warehousing and Provide business
Business Intelligence intelligence requirements
Management and management metrics,
and they identify and help
resolve business
intelligence issues
Document and Content Define enterprise
Management taxonomies and resolve
content management
issues
Metadata Management Create and maintain
business metadata (names,
meanings, business rules),
define metadata access
and integration needs and
use metadata to make
effective data stewardship
and governance decisions
Data Quality Management Define data quality
requirements and business
rules, test application edits
and validations, assist in
the analysis, certification,
and auditing of data
quality, lead clean-up
efforts, identify ways to
solve causes of poor data
quality, promote data
quality awareness
March 8, 2010 85

Data Strategy

• High-level course of action to achieve high-level goals
• Data strategy is a data management program strategy a
plan for maintaining and improving data quality, integrity,
security and access
• Address all data management functions relevant to the
organisation

March 8, 2010 86

Elements of Data Strategy

• Vision for data management
• Summary business case for data management
• Guiding principles, values, and management perspectives
• Mission and long-term directional goals of data management
• Management measures of data management success
• Short-term data management programme objectives
• Descriptions of data management roles and business units along
with a summary of their responsibilities and decision rights
• Descriptions of data management programme components and
initiatives
• Outline of the data management implementation roadmap
• Scope boundaries
March 8, 2010 87

Data Strategy

Data Management
Programme Charter
Data Management Data Management
Scope Statement Overall vision, business case,
goals, guiding principles, Implementation
measures of success, critical Roadmap
Goals and objectives for a success factors, recognised risks
defined planning horizon and the
Identifying specific programs,
roles, organisations, and
projects, task assignments, and
individual leaders accountable
delivery milestones
for achieving these objectives

March 8, 2010 88

Data Policies

• Statements of intent and fundamental rules governing the
creation, acquisition, integrity, security, quality, and use of
data and information
• More fundamental, global, and business critical than data
standards
• Describe what to do and what not to do
• Should be few data policies stated briefly and directly

March 8, 2010 89

Data Policies

• Possible topics for data policies
− Data modeling and other data development activities
− Development and use of data architecture
− Data quality expectations, roles, and responsibilities
− Data security, including confidentiality classification policies,
intellectual property policies, personal data privacy policies,
general data access and usage policies, and data access by
external parties
− Database recovery and data retention
− Access and use of externally sourced data
− Sharing data internally and externally
− Data warehousing and business intelligence
− Unstructured data - electronic files and physical records

March 8, 2010 90

Data Architecture

• Enterprise data model and other aspects of data
architecture sponsored at the data governance level
• Need to pay particular attention to the alignment of the
enterprise data model with key business strategies,
processes, business units and systems
• Includes
− Data technology architecture
− Data integration architecture
− Data warehousing and business intelligence architecture
− Metadata architecture

March 8, 2010 91

Data Standards and Procedures

• Include naming standards, requirement specification
standards, data modeling standards, database design
standards, architecture standards and procedural
standards for each data management function
• Must be effectively communicated, monitored, enforced
and periodically re-evaluated
• Data management procedures are the methods,
techniques, and steps followed to accomplish a specific
activity or task

March 8, 2010 92

Data Standards and Procedures

• Possible topics for data standards and procedures
− Data modeling and architecture standards, including data naming conventions,
definition standards, standard domains, and standard abbreviations
− Standard business and technical metadata to be captured, maintained, and
integrated
− Data model management guidelines and procedures
− Metadata integration and usage procedures
− Standards for database recovery and business continuity, database
performance, data retention, and external data acquisition
− Data security standards and procedures
− Reference data management control procedures
− Match / merge and data cleansing standards and procedures
− Business intelligence standards and procedures
− Enterprise content management standards and procedures, including use of
enterprise taxonomies, support for legal discovery and document and e-mail
retention, electronic signatures, report formatting standards and report
distribution approaches

March 8, 2010 93

Regulatory Compliance

• Most organisations are is impacted by government and
industry regulations
• Many of these regulations dictate how data and
information is to be managed
• Compliance is generally mandatory
• Data governance guides the implementation of adequate
controls to ensure, document, and monitor compliance
with data-related regulations.

March 8, 2010 94

Regulatory Compliance

• Data governance needs to work the business to find the best
answers to the following regulatory compliance questions
− How relevant is a regulation?
− Why is it important for us?
− How do we interpret it?
− What policies and procedures does it require?
− Do we comply now?
− How do we comply now?
− How should we comply in the future?
− What will it take?
− When will we comply?
− How do we demonstrate and prove compliance?
− How do we monitor compliance?
− How often do we review compliance?
− How do we identify and report non-compliance?
− How do we manage and rectify non-compliance?
March 8, 2010 95

Issue Management

• Data governance assists in identifying, managing, and resolving data
related issues
− Data quality issues
− Data naming and definition conflicts
− Business rule conflicts and clarifications
− Data security, privacy, and confidentiality issues
− Regulatory non-compliance issues
− Non-conformance issues (policies, standards, architecture, and procedures)
− Conflicting policies, standards, architecture, and procedures
− Conflicting stakeholder interests in data and information
− Organisational and cultural change management issues
− Issues regarding data governance procedures and decision rights
− Negotiation and review of data sharing agreements

March 8, 2010 96

Issue Management, Control and Escalation

• Data governance implements issue controls and
procedures
− Identifying, capturing, logging and updating issues
− Tracking the status of issues
− Documenting stakeholder viewpoints and resolution alternatives
− Objective, neutral discussions where all viewpoints are heard
− Escalating issues to higher levels of authority
− Determining, documenting and communicating issue resolutions.

March 8, 2010 97

Data Management Projects

• Data management roadmap sets out a course of action for
initiating and/or improving data management functions
• Consists of an assessment of current functions, definition
of a target environment and target objectives and a
transition plan outlining the steps required to reach these
targets including an approach to organisational change
management
• Every data management project should follow the project
management standards of the organisation

March 8, 2010 98

Data Asset Valuation

• Data and information are truly assets because they have
business value, tangible or intangible
• Different approaches to estimating the value of data assets
• Identify the direct and indirect business benefits derived
from use of the data
• Identify the cost of data loss, identifying the impacts of not
having the current amount and quality level of data

March 8, 2010 99


March 8, 2010 100


• Concerned with defining and maintaining specifications
that
− Provide a standard common business vocabulary
− Express strategic data requirements
− Outline high level integrated designs to meet these requirements
− Align with enterprise strategy and related business architecture
• Data architecture is an integrated set of specification
artifacts used to define data requirements, guide
integration and control of data assets and align data
investments with business strategy
• Includes formal data names, comprehensive data
definitions, effective data structures, precise data integrity
rules, and robust data documentation
March 8, 2010 101

Data Architecture Management – Definition and
Goals
• Definition
− Defining the data needs of the enterprise and designing the
master blueprints to meet those needs
• Goals
− To plan with vision and foresight to provide high quality data
− To identify and define common data requirements
− To design conceptual structures and plans to meet the current
and long-term data requirements of the enterprise

March 8, 2010 102

Data Architecture Management - Overview
•Business Goals •Enterprise Data Model Information
•Business Strategies Value Chain Analysis
•Business Architecture •Data Technology Architecture
•Process Architecture •Data Integration / MDM Architecture
•IT Objectives •DW / BI Architecture
•IT Strategies •Metadata Architecture
•Data Strategies •Enterprise Taxonomies and
•Data Issues and Needs Namespaces
•Technical Architecture •Document Management Architecture
•Metadata
Data Architecture
Suppliers Consumers
Management
•Executives •Data Producers
•Data Stewards •Knowledge Workers
•Data Producers •Managers and Executives
•Information Consumers •Data Professionals
•Customers

•Data Stewards
•Subject Matter Experts (SMEs) Data •Data Value
Architects •Data Modeling Tools •Data Management Cost
•Data Analysts and Modelers Other •Model Management Tool •Achievement of Objectives
Enterprise Architects •Metadata Repository Office •# of Decisions Made
•DM Executive and Managers •Productivity Tools •Steward Representation / Coverage
•CIO and Other Executives •Data Professional Headcount
•Database Administrators •Data Management Process Maturity
•Data Model Administrator
March 8, 2010 103

Enterprise Data Architecture

• Integrated set of specifications and documents
− Enterprise Data Model - the core of enterprise data architecture
− Information Value Chain Analysis - aligns data with business
processes and other enterprise architecture components
− Related Data Delivery Architecture - including database
architecture, data integration architecture, data warehousing /
business intelligence architecture, document content
architecture, and metadata architecture

March 8, 2010 104

Data Architecture Management Activities

• Understand Enterprise Information Needs
• Develop and Maintain the Enterprise Data Model
• Analyse and Align With Other Business Models
• Define and Maintain the Database Architecture
• Define and Maintain the Data Integration Architecture
• Define and Maintain the Data Warehouse / Business
Intelligence Architecture
• Define and Maintain Enterprise Taxonomies and
Namespaces
• Define and Maintain the Metadata Architecture

March 8, 2010 105

Understanding Enterprise Information Needs

• In order to create an enterprise data architecture, the
organisation must first define its information need
• An enterprise data model is a way of capturing and
defining enterprise information needs and data
requirements
• Master blueprint for enterprise-wide data integration
• Enterprise data model is a critical input to all future
systems development projects and the baseline for
additional data requirements analysis
• Evaluate the current inputs and outputs required by the
organisation, both from and to internal and external
targets
March 8, 2010 106

Develop and Maintain the Enterprise Data Model

• Data is the set of facts collected about business entities
• Data model is a set of data specifications that reflect data
requirements and designs
• Enterprise data model is an integrated, subject-oriented
data model defining the critical data produced and
consumed across the organisation
• Define and analyse data requirements
• Design logical and physical data structures that support
these requirements

March 8, 2010 107

Enterprise Data Model

Enterprise Data
Model

Other Enterprise
Conceptual Data Enterprise Logical
Subject Area Model Data Model
Model Data Models
Components

Data Steward
Valid Reference Data Quality
Responsibility Entity Life Cycles
Data Values Specifications
Assignments

March 8, 2010 108

Enterprise Data Model

• Build an enterprise data model in layers
• Focus on the most critical business subject areas

March 8, 2010 109

Subject Area Model

• List of major subject areas that collectively express the
essential scope of the enterprise
• Important to the success of the entire enterprise data
model
• List of enterprise subject areas becomes one of the most
significant organisation classifications
• Acceptable to organisation stakeholders
• Useful as the organising framework for data governance,
data stewardship, and further enterprise data modeling

March 8, 2010 110

Conceptual Data Model

• Conceptual data model defines business entities and their
relationships
• Business entities are the primary organisational structures in a
conceptual data model
• Business needs data about business entities
• Include a glossary containing the business definitions and other
metadata associated with business entities and their relationships
• Assists improved business understanding and reconciliation of terms
and their meanings
• Provide the framework for developing integrated information
systems to support both transactional processing and business
intelligence.
• Depicts how the enterprise sees information

March 8, 2010 111

Enterprise Logical Data Models

• Logical data model contain a level of detail below the
conceptual data model
• Contain the essential data attributes for each entity
• Essential data attributes are those data attributes without
which the enterprise cannot function – can be a subjective
decision

March 8, 2010 112

Other Enterprise Data Model Components

• Data Steward Responsibility Assignments- for subject
areas, entities, attributes, and/or reference data value sets
• Valid Reference Data Values - controlled value sets for
codes and/or labels and their business meaning
• Data Quality Specifications - rules for essential data
attributes, such as accuracy / precision requirements,
currency (timeliness), integrity rules, nullability,
formatting, match/merge rules, and/or audit requirements
• Entity Life Cycles - show the different lifecycle states of
the most important entities and the trigger events that
change an entity from one state to another
March 8, 2010 113

Analyse and Align with Other Business Models

• Information value-chain analysis maps the relationships
between enterprise model elements and other business
models
• Business value chain identifies the functions of an
organisation that contribute directly or indirectly to the
organisation’s goals

March 8, 2010 114

Define and Maintain the Data Technology
Architecture
• Data technology architecture guides the selection and integration of
data-related technology
• Data technology architecture defines standard tool categories,
preferred tools in each category, and technology standards and
protocols for technology integration
• Technology categories include
− Database management systems (DBMS)
− Database management utilities
− Data modelling and model management tools
− Business intelligence software for reporting and analysis
− Extract-transform-load (ETL), changed data capture (CDC), and other data
integration tools
− Data quality analysis and data cleansing tools
− Metadata management software, including metadata repositories
March 8, 2010 115

Define and Maintain the Data Technology
Architecture
• Classify technology architecture components as
− Current - currently supported and used
− Deployment - deployed for use in the next 1-2 years
− Strategic - expected to be available for use in the next 2+ years
− Retirement - the organisation has retired or intends to retire this
year
− Preferred - preferred for use by most applications.
− Containment - limited to use by certain applications
− Emerging - being researched and piloted for possible future
deployment

March 8, 2010 116

Define and Maintain the Data Integration
Architecture
• Defines how data flows through all systems from
beginning to end
• Both data architecture and application architecture,
because it includes both databases and the applications
that control the data flow into the system, between
databases and back out of the system

March 8, 2010 117

Define and Maintain the Data Warehouse / Business
Intelligence Architecture
• Focuses on how data changes and snapshots are stored in
data warehouse systems for maximum usefulness and
performance
• Data integration architecture shows how data moves from
source systems through staging databases into data
warehouses and data marts
• Business intelligence architecture defines how decision
support makes data available, including the selection and
use of business intelligence tools

March 8, 2010 118

Define and Maintain Enterprise Taxonomies and
Namespaces
• Taxonomy is the hierarchical structure used for outlining
topics
• Organisations develop their own taxonomies to organise
collective thinking about topics
• Overall enterprise data architecture includes
organisational taxonomies
• Definition of terms used in such taxonomies should be
consistent with the enterprise data model

March 8, 2010 119

Define and Maintain the Metadata Architecture

• Metadata architecture is the design for integration of
metadata across software tools, repositories, directories,
glossaries, and data dictionaries
• Metadata architecture defines the managed flow of
metadata
• Defines how metadata is created, integrated, controlled,
and accessed
• Metadata repository is the core of any metadata
architecture
• Focus of metadata architecture is to ensure the quality,
integration, and effective use of metadata
March 8, 2010 120

Data Architecture Management Guiding Principles

• Data architecture is an integrated set of specification master blueprints used to
define data requirements, guide data integration, control data assets, and align
data investments with business strategy
• Enterprise data architecture is part of the overall enterprise architecture, along
with process architecture, business architecture, systems architecture, and
technology architecture
• Enterprise data architecture includes three major categories of specifications: the
enterprise data model, information value chain analysis, and data delivery
architecture
• Enterprise data architecture is about more than just data - it helps to establish a
common business vocabulary
• An enterprise data model is an integrated subject-oriented data model defining
the essential data used across an entire organisation
• Information value-chain analysis defines the critical relationships between data,
processes, roles and organisations and other enterprise elements
• Data delivery architecture defines the master blueprint for how data flows across
databases and applications
• Architectural frameworks like TOGAF help organise collective thinking about
architecture
March 8, 2010 121

Data Development

March 8, 2010 122

Data Development

• Analysis, design, implementation, deployment, and
maintenance of data solutions to maximise the value of
the data resources to the enterprise
• Subset of project activities within the system development
lifecycle focused on defining data requirements, designing
the data solution components, and implementing these
components
• Primary data solution components are databases and
other data structures

March 8, 2010 123

Data Development – Definition and Goals

• Definition
− Designing, implementing, and maintaining solutions to meet the
data needs of the enterprise
• Goals
− Identify and define data requirements
− Design data structures and other solutions to these requirements
− Implement and maintain solution components that meet these
requirements
− Ensure solution conformance to data architecture and standards
as appropriate
− Ensure the integrity, security, usability, and maintainability of
structured data assets

March 8, 2010 124

Data Development - Overview
•Data Requirements and Business
•Business Goals and Strategies Rules
•Data Needs and Strategies •Conceptual Data Models
•Data Standards •Logical Data Models and
•Data Architecture Specifications
•Process Architecture •Physical Data Models and
•Application Architecture Specifications
•Technical Architecture •Metadata (Business and Technical)
•Data Modeling and DB Design
Standards
Suppliers Data Development •Data Model and DB Design Reviews
•Version Controlled Data Models
•Test Data
•Data Stewards •Development and Test Databases
•Subject Matter Experts •Information Products
•IT Steering Committee •Data Access Services
•Data Governance Council •Data Integration Services
•Data Architects and Analysts •Migrated and Converted Data
•Software Developers
•Data Producers
•Information Consumers

Participants Tools Consumers
•Data Stewards and SMEs •Data Modeling Tools
•Data Architects and Analysts •Database Management Systems •Data Producers
•Database Administrators •Software Development Tools •Knowledge Workers
•Data Model Administrators •Testing Tools •Managers and Executives
•Software Developers •Data Profiling Tools •Customers
•Project Managers •Model Management Tools •Data Professionals
•DM Executives and Other IT •Configuration Management Tools •Other IT Professionals
Management •Office Productivity Tools

March 8, 2010 125

Data Development Function, Activities and Sub-
Activities
Data Development

Data Modelling,
Data Model and Design
Analysis and Solution Detailed Data Design Data Implementation
Quality Management
Design

Implement
Analyse Information Design Physical Develop Data Modeling
Development / Test
Requirements Databases and Design Standards
Database Changes

Develop and Maintain
Physical Database Review Data Model and Create and Maintain
Conceptual Data
Design Database Design Quality Test Data
Models

Performance Conceptual and Logical Migrate and Convert
Entities
Modifications Data Model Reviews Data

Physical Database Physical Database Build and Test
Relationships
Design Documentation Design Review Information Products

Develop and Maintain Design Information Build and Test Data
Data Model Validation
Logical Data Models Products Access Services

Manage Data Model
Design Data Access Validate Information
Attributes Versioning and
Services Requirements
Integration

Design Data Integration Prepare for Data
Domains
Services Deployment

Keys

Develop and Maintain
Physical Data Models

March 8, 2010 126

Data Development - Principles

• Data development activities are an integral part of the software development lifecycle
• Data modeling is an essential technique for effective data management and system design
• Conceptual and logical data modeling express business and application requirements while
physical data modeling represents solution design
• Data modeling and database design define detail solution component specifications
• Data modeling and database design balances tradeoffs and needs
• Data professionals should collaborate with other project team members to design
information products and data access and integration interfaces
• Data modeling and database design should follow documented standards
• Design reviews should review all data models and designs, in order to ensure they meet
business requirements and follow design standards
• Data models represent valuable knowledge resources and so should be carefully managed
and controlled them through library, configuration, and change management to ensure
data model quality and availability
• Database administrators and other data professionals play important roles in the
construction, testing, and deployment of databases and related application systems

March 8, 2010 127

Data Modeling, Analysis, and Solution Design

• Data modeling is an analysis and design method used to
define and analyse data requirements, and design data
structures that support these requirements
• A data model is a set of data specifications and related
diagrams that reflect data requirements and designs
• Data modeling is a complex process involving interactions
between people and with technology which do not
compromise the integrity or security of the data
• Good data models accurately express and effectively
communicate data requirements and quality solution
design
March 8, 2010 128

Data Model

• The purposes of a data model are:
− Communication - a data model is a bridge to understanding data between
people with different levels and types of experience. Data models help us
understand a business area, an existing application, or the impact of modifying
an existing structure. Data models may also facilitate training new business
and/or technical staff
− Formalisation - a data model documents a single, precise definition of data
requirements and data related business rules
− Scope – a data model can help explain the data context and scope of
purchased application packages
• Data models that include the same data may differ by:
− Scope - expressing a perspective about data in terms of function (business view
or application view), realm (process, department, division, enterprise, or
industry view), and time (current state, short-term future, long-term future)
− Focus - basic and critical concepts (conceptual view), detailed but independent
of context (logical view), or optimised for a specific technology and use
(physical view)

March 8, 2010 129

Analyse Information Requirements

• Information is relevant and timely data in context
• To identify information requirements, first identify business
information needs, often in the context of one or more business
processes
• Business processes (and the underlying IT systems) consume
information output from other business processes
• Requirements analysis includes the elicitation, organisation,
documentation, review, refinement, approval, and change control of
business requirements
• Some of these requirements identify business needs for data and
information
• Logical data modeling is an important means of expressing business
data requirements

March 8, 2010 130

Develop and Maintain Conceptual Data Models

• Visual, high-level perspective on a subject area of
importance to the business
• Contains the basic and critical business entities within a
given realm and function with a description of each entity
and the relationships between entities
• Define the meanings of the essential business vocabulary
• Reflect the data associated with a business process or
application function
• Independent of technology and usage context

March 8, 2010 131

Develop and Maintain Conceptual Data Models

• Entities
− A data entity is a collection of data about something that the
business deems important and worthy of capture
− Entities appear in conceptual or logical data models
• Relationships
− Business rules define constraints on what can and cannot be done
• Data Rules – define constraints on how data relates to other data
• Action Rules - instructions on what to do when data elements contain
certain values

March 8, 2010 132

Develop and Maintain Logical Data Models

• Detailed representation of data requirements and the
business rules that govern data quality
• Independent of any technology or specific implementation
technical constraints
• Extension of a conceptual data model
• Logical data models transform conceptual data model
structures by normalisation and abstraction
− Normalisation is the process of applying rules to organise business
complexity into stable data structure
− Abstraction is the redefinition of data entities, elements, and
relationships by removing details to broaden the applicability of
data structures to a wider class of situations

March 8, 2010 133

Develop and Maintain Physical Data Models

• Physical data model optimises the implementation of
detailed data requirements and business rules in light of
technology constraints, application usage, performance
requirements, and modeling standards
• Physical data modeling transforms the logical data model
• Includes specific decisions
− Name of each table and column or file and field or schema and
element
− Logical domain, physical data type, length, and nullability of each
column or field
− Default values
− Primary and alternate unique keys and indexes
March 8, 2010 134

Detailed Data Design

• Detailed data design activities include
− Detailed physical database design, including views, functions,
triggers, and stored procedures
− Definition of supporting data structures, such as XML schemas
and object classes
− Creation of information products, such as the use of data in
screens and reports
− Definition of data access solutions, including data access objects,
integration services, and reporting and analysis services

March 8, 2010 135

Design Physical Databases

• Create detailed database implementation specifications
• Ensure the design meets data integrity requirements
• Determine the most appropriate physical structure to house and organise the data,
such as relational or other type of DBMS, files, OLAP cubes, XML, etc.
• Determine database resource requirements, such as server size and location, disk
space requirements, CPU and memory requirements, and network requirements
• Creating detailed design specifications for data structures, such as relational
database tables, indexes, views, OLAP data cubes, XML schemas, etc.
• Ensure performance requirements are met, including batch and online response
time requirements for queries, inserts, updates, and deletes
• Design for backup, recovery, archiving, and purge processing, ensuring availability
requirements are met
• Design data security implementation, including authentication, encryption needs,
application roles and data access and update permissions
• Review code to ensure that it meets coding standards and will run efficiently

March 8, 2010 136

Physical Database Design

• Choose a database design based on both a choice of architecture
and a choice of technology
• Base the choice of architecture (for example, relational, hierarchical,
network, object, star schema, snowflake, cube, etc.) on data
considerations
• Consider factors such as how long the data needs to be kept,
whether it must be integrated with other data or passed across
system or application boundaries, and on requirements of data
security, integrity, recoverability, accessibility, and reusability
• Consider organisational or political factors, including organisational
biases and developer skill sets, that lean toward a particular
technology or vendor

March 8, 2010 137

Physical Database Design - Principles

• Performance and Ease of Use - Ensure quick and easy access to data
by approved users in a usable and business-relevant form
• Reusability - The database structure should ensure that, where
appropriate, multiple applications would be able to use the data
• Integrity - The data should always have a valid business meaning and
value, regardless of context, and should always reflect a valid state
of the business
• Security - True and accurate data should always be immediately
available to authorised users, but only to authorised users
• Maintainability - Perform all data work at a cost that yields value by
ensuring that the cost of creating, storing, maintaining, using, and
disposing of data does not exceed its value to the organisation

March 8, 2010 138

Physical Database Design - Questions

• What are the performance requirements? What is the maximum permissible time for a
query to return results, or for a critical set of updates to occur?
• What are the availability requirements for the database? What are the window(s) of time
for performing database operations? How often should database backups and transaction
log backups be done (i.e., what is the longest period of time we can risk non-recoverability
of the data)?
• What is the expected size of the database? What is the expected rate of growth of the
data? At what point can old or unused data be archived or deleted? How many concurrent
users are anticipated?
• What sorts of data virtualisation are needed to support application requirements in a way
that does not tightly couple the application to the database schema?
• Will other applications need the data? If so, what data and how?
• Will users expect to be able to do ad-hoc querying and reporting of the data? If so, how and
with which tools?
• What, if any, business or application processes does the database need to implement?
(e.g., trigger code that does cross-database integrity checking or updating, application
classes encapsulated in database procedures or functions, database views that provide
table recombination for ease of use or security purposes, etc.).
• Are there application or developer concerns regarding the database, or the database
development process, that need to be addressed?
• Is the application code efficient? Can a code change relieve a performance issue?
March 8, 2010 139

Performance Modifications

• Consider how the database will perform when applications
make requests to access and modify data
• Indexing can improve query performance in many cases
• Denormalisation is the deliberate transformation of a
normalised logical data model into tables with redundant
data

March 8, 2010 140

Physical Database Design Documentation

• Create physical database design document to assist
implementation and maintenance

March 8, 2010 141

Design Information Products

• Design data-related deliverables
• Design screens and reports to meet business data requirements
• Ensure consistent use of business data terminology
• Reporting services give business users the ability to execute both pre-developed
and ad-hoc reports
• Analysis services give business users to ability slice and dice data across multiple
dimensions
• Dashboards display a wide array of analytics indicators, such as charts and graphs,
efficiently
• Scorecard display information that indicates scores or calculated evaluations of
performance
• Use data integrated from multiple databases as input to software for business
process automation that coordinates multiple business processes across disparate
platforms
• Data integration is a component of Enterprise Application Integration (EAI)
software, enabling data to be easily passed from application to application across
disparate platforms

March 8, 2010 142

Design Data Access Services

• May be necessary to access and combine data from
remote databases with data in the local database
• Goal is to enable easy and inexpensive reuse of data across
the organisation preventing, wherever possible, redundant
and inconsistent data
• Options include
− Linked database connections
− SOA web services
− Message brokers
− Data access classes
− ETL
− Replication

March 8, 2010 143

Design Data Integration Services

• Critical aspect of database design is determining
appropriate update mechanisms and database transaction
for recovery
• Define source-to-target mappings and data transformation
designs for extract-transform-load (ETL) programs and
other technology for ongoing data movement, cleansing
and integration
• Design programs and utilities for data migration and
conversion from old data structures to new data structures

March 8, 2010 144

Data Model and Design Quality Management

• Balance the needs of information consumers (the people
with business requirements for data) and the data
producers who capture the data in usable form
• Time and budget constraints
• Ensure data resides in data structures that are secure,
recoverable, sharable, and reusable, and that this data is
as correct, timely, relevant, and usable as possible
• Balance the short-term versus long-term business data
interests of the organisation

March 8, 2010 145

Develop Data Modeling and Design Standards

• Data modeling and database design standards serve as the guiding
principles to effectively meet business data needs, conform to data
architecture, and ensure data quality
• Data modeling and database design standards should include
− A list and description of standard data modeling and database design
deliverables
− A list of standard names, acceptable abbreviations, and abbreviation rules for
uncommon words, that apply to all data model objects
− A list of standard naming formats for all data model objects, including attribute
and column class words
− A list and description of standard methods for creating and maintaining these
deliverables
− A list and description of data modeling and database design roles and
responsibilities
− A list and description of all metadata properties captured in data modeling and
database design, including both business metadata and technical metadata,
with guidelines defining metadata quality expectations and requirements
− Guidelines for how to use data modeling tools
− Guidelines for preparing for and leading design reviews
March 8, 2010 146

Review Data Model and Database Design Quality

• Conduct requirements reviews and design reviews,
including a conceptual data model review, a logical data
model review, and a physical database design review

March 8, 2010 147

Conceptual and Logical Data Model Reviews

• Conceptual data model and logical data model design
reviews should ensure that:
− Business data requirements are completely captured and clearly
expressed in the model, including the business rules governing
entity relationships
− Business (logical) names and business definitions for entities and
attributes (business semantics) are clear, practical, consistent,
and complementary
− Data modeling standards, including naming standards, have been
followed
− The conceptual and logical data models have been validated

March 8, 2010 148

Physical Database Design Review

• Physical database design reviews should ensure that:
− The design meets business, technology, usage, and performance
requirements
− Database design standards, including naming and abbreviation
standards, have been followed
− Availability, recovery, archiving, and purging procedures are
defined according to standards
− Metadata quality expectations and requirements are met in order
to properly update any metadata repository
− The physical data model has been validated

March 8, 2010 149

Data Model Validation

• Validate data models against modeling standards, business
requirements, and database requirements
• Ensure the model matches applicable modeling standards
• Ensure the model matches the business requirements
• Ensure the model matches the database requirements

March 8, 2010 150

Manage Data Model Versioning and Integration

• Data models and other design specifications require
change control
− Each change should include
− Why the project or situation required the change
− What and how the object(s) changed, including which tables had
columns added, modified, or removed, etc.
− When the change was approved and when the change was made
to the model
− Who made the change
− Where the change was made

March 8, 2010 151

Data Implementation

• Data implementation consists of data management
activities that support system building, testing, and
deployment
− Database implementation and change management in the
development and test environments
− Test data creation, including any security procedures
− Development of data migration and conversion programs, both
for project development through the SDLC and for business
situations
− Validation of data quality requirements
− Creation and delivery of user training
− Contribution to the development of effective documentation

March 8, 2010 152

Implement Development / Test Database Changes

• Implement changes to the database that are required
during the course of application development
• Monitor database code to ensure that it is written to the
same standards as application code
• Identify poor SQL coding practices that could lead to errors
or performance problems

March 8, 2010 153

Create and Maintain Test Data

• Populate databases in the development environment with
test data
• Observe privacy and confidentiality requirements and
practices for test data

March 8, 2010 154

Migrate and Convert Data

• Key component of many projects is the migration of legacy
data to a new database environment, including any
necessary data cleansing and reformatting

March 8, 2010 155

Build and Test Information Products

• Implement mechanisms for integrating data from multiple
sources, along with the appropriate metadata to ensure
meaningful integration of the data
• Implement mechanisms for reporting and analysing the
data, including online and web-based reporting, ad-hoc
querying, BI scorecards, OLAP, portals, and the like
• Implement mechanisms for replication of the data, if
network latency or other concerns make it impractical to
service all users from a single data source

March 8, 2010 156

Build and Test Data Access Services

• Develop, test, and execute data migration and conversion
programs and procedures, first for development and test
data and later for production deployment
• Data requirements should include business rules for data
quality to guide the implementation of application edits
and database referential integrity constraints
• Business data stewards and other subject matter experts
should validate the correct implementation of data
requirements through user acceptance testing

March 8, 2010 157

Validate Information Requirements

• Test and validate that the solution meets the
requirements, and plan deployment, developing training,
and documentation.
• Data requirements may change abruptly, in response to
either changed business requirements, invalid
assumptions regarding the data or reprioritisation of
existing requirements
• Test the implementation of the data requirements and
ensure that the application requirements are satisfied

March 8, 2010 158

Prepare for Data Deployment

• Leverage the business knowledge captured in data modeling to
define clear and consistent language in user training and
documentation
• Business concepts, terminology, definitions, and rules depicted in
data models are an important part of application user training
• Data stewards and data analysts should participate in deployment
preparation, including development and review of training materials
and system documentation, especially to ensure consistent use of
defined business data terminology
• Help desk support staff also require orientation and training in how
system users appropriately access, manipulate, and interpret data
• Once installed, business data stewards and data analysts should
monitor the early use of the system to see that business data
requirements are indeed met

March 8, 2010 159


March 8, 2010 160


• Management is the development, maintenance, and
support of structured data to maximise the value of the
data resources to the enterprise and includes
− Database support
− Data technology management

March 8, 2010 161

Data Operations Management – Definition and
Goals
• Definition
− Planning, control, and support for structured data assets across
the data lifecycle, from creation and acquisition through archival
and purge
• Goals
− Protect and ensure the integrity of structured data assets
− Manage the availability of data throughout its lifecycle
− Optimise performance of database transactions

March 8, 2010 162

Data Operations Management - Overview

•DBMS Technical Environments
•Data Requirements •Dev/Test, QA, DR, and Production
•Data Architecture Databases
•Data Models •Externally Sourced Data
•Legacy Data •Database Performance
•Service Level Agreements •Data Recovery Plans
•Business Continuity
•Data Retention Plan
Data Operations •Archived and Purged Data
Suppliers
Management
Consumers
•Executives
•IT Steering Committee
•Data Governance Council
•Data Stewards •Data Creators
•Data Architects and Modelers •Information Consumers
•Software Developers •Enterprise Customers
•Data Professionals
•Other IT Professionals

Participants Tools
•Database Administrators Metrics
•Software Developers
•Project Managers •Database Management Systems
•Data Stewards •Data Development Tools
•Data Architects and Analysts •Database Administration Tools •Availability
•DM Executives and Other IT •Office Productivity Tools •Performance
Management
•IT Operators

March 8, 2010 163

Data Operations Management Function, Activities
and Sub-Activities

Database Support Data Technology Management

Implement and Control Database
Understand Data Technology Requirements
Environments

Obtain Externally Sourced Data Define the Data Technology Architecture

Plan for Data Recovery Evaluate Data Technology

Backup and Recover Data Install and Administer Data Technology

Set Database Performance Service Levels Inventory and Track Data Technology Licenses

Monitor and Tune Database Performance Support Data Technology Usage and Issues

Plan for Data Retention

Archive, Retain, and Purge Data

Support Specialised Databases
March 8, 2010 164

Data Operations Management - Principles

• Write everything down
• Keep everything
• Whenever possible, automate a procedure
• Focus to understand the purpose of each task, manage scope,
simplify, do one thing at a time
• Measure twice, cut once
• React to problems and issues calmly and rationally, because panic
causes more errors
• Understand the business, not just the technology
• Work together to collaborate, be accessible, share knowledge
• Use all of the resources at your disposal
• Keep up to date
March 8, 2010 165

Database Support - Scope

• Ensure the performance and reliability of the database,
including performance tuning, monitoring, and error
reporting
• Implement appropriate backup and recovery mechanisms
to guarantee the recoverability of the data in any
circumstance
• Implement mechanisms for clustering and failover of the
database, if continual data availability data is a
requirement
• Implement mechanisms for archiving data operations
management
March 8, 2010 166

Database Support - Deliverables

• A production database environment, including an instance of the
DBMS and its supporting server, of a sufficient size and capacity to
ensure adequate performance, configured for the appropriate level
of security, reliability and availability
• Mechanisms and processes for controlled implementation and
changes to databases into the production environment
• Appropriate mechanisms for ensuring the availability, integrity, and
recoverability of the data in response to all possible circumstances
that could result in loss or corruption of data
• Appropriate mechanisms for detecting and reporting any error that
occurs in the database, the DBMS, or the data server
• Database availability, recovery, and performance in accordance with
service level agreements
March 8, 2010 167

Implement and Control Database Environments

• Updating DBMS software
• Maintaining multiple installations, including different DBMS versions
• Installing and administering related data technology, including data
integration software and third party data administration tools
• Setting and tuning DBMS system parameters
• Managing database connectivity
• Tune operating systems, networks, and transaction processing
middleware to work with the DBMS
• Optimise the use of different storage technology for cost-effective
storage

March 8, 2010 168

Obtain Externally Sourced Data

• Managed approach to data acquisition centralises
responsibility for data subscription services
• Document the external data source in the logical data
model and data dictionary
• Implement the necessary processes to load the data into
the database and/or make it available to applications

March 8, 2010 169

Plan for Data Recovery

• Establish service level agreements (SLAs) with IT data management
services organisations for data availability and recovery
• SLAs set availability expectations, allowing time for database
maintenance and backup, and set recovery time expectations for
different recovery scenarios, including potential disasters
• Ensure a recovery plan exists for all databases and database servers,
covering all possible scenarios
− Loss of the physical database server
− Loss of one or more disk storage devices
− Loss of a database, including the DBMS master database, temporary storage
database, transaction log segment, etc.
− Corruption of database index or data pages
− Loss of the database or log segment file system
− Loss of database or transaction log backup files
March 8, 2010 170

Backup and Recover Data

• Make regular backups of database and the database
transaction logs
• Balance the importance of the data against the cost of
protecting it
• Databases should reside on some sort of managed storage
area
• For critical data, implement some sort of replication facility

March 8, 2010 171

Set Database Performance Service Levels

• Database performance has two components - availability
and performance
• An unavailable database has a performance measure of
zero
• SLAs between data management services organisations
and data owners define expectations for database
performance
• Availability is the percentage of time that a system or
database can be used for productive work
• Availability requirements are constantly increasing, raising
the business risks and costs of unavailable data

March 8, 2010 172


• Factors affecting availability include
− Manageability - ability to create and maintain an effective environment
− Recoverability - ability to reestablish service after interruption, and correct
errors caused by unforeseen events or component failures
− Reliability - ability to deliver service at specified levels for a stated period
− Serviceability - ability to determine the existence of problems, diagnose their
causes, and repair / solve the problems
• Tasks to ensure databases stay online and operational
− Running database backup utilities
− Running database reorganisation utilities
− Running statistics gathering utilities
− Running integrity checking utilities
− Automating the execution of these utilities
− Exploiting table space clustering and partitioning
− Replicating data across mirror databases to ensure high availability

March 8, 2010 173


• Cause of loss of database availability
− Planned and unplanned outages
− Loss of the server hardware
− Disk hardware failure
− Operating system failure
− DBMS software failure
− Application problems
− Network failure
− Data center site loss
− Security and authorisation problems
− Corruption of data (due to bugs, poor design, or user error)
− Loss of database objects
− Loss of data
− Data replication failure
− Severe performance problems
− Recovery failures
− Human error

March 8, 2010 174

Monitor and Tune Database Performance

• Optimise database performance both proactively and reactively, by
monitoring performance and by responding to problems quickly and
effectively
• Run activity and performance reports against both the DBMS and
the server on a regular basis including during periods of heavy
activity
• When performance problems occur, use the monitoring and
administration tools of the DBMS to help identify the source of the
problem
− Memory allocation (buffer / cache for data)
− Locking and blocking
− Failure to update database statistics
− Poor SQL coding
− Insufficient indexing
− Application activity
− Increase in the number, size, or use of databases
− Database volatility

March 8, 2010 175

Support Specialised Databases

• Some specialised situations require specialised types of
databases

March 8, 2010 176

Data Technology Management

• Managing data technology should follow the same
principles and standards for managing any technology
• Use a reference model for technology management such
as Information Technology Infrastructure Library (ITIL)

March 8, 2010 177

Understand Data Technology Requirements

• Understand the data and information needs of the business
• Understand the best possible applications of technology to solve business
problems and take advantage of new business opportunities
• Understand the requirements of a data technology before determining what
technical solution to choose for a particular situation
− What problem does this data technology mean to solve?
− What does this data technology do that is unavailable in other data technologies?
− What does this data technology not do that is available in other data technologies?
− Are there any specific hardware requirements for this data technology?
− Are there any specific Operating System requirements for this data technology?
− Are there any specific software requirements or additional applications required for this
data technology to perform as advertised?
− Are there any specific storage requirements for this data technology?
− Are there any specific network or connectivity requirements for this data technology?
− Does this data technology include data security functionality? If not, what other tools
does this technology work with that provides for data security functionality?
− Are there any specific skills required to be able support this data technology? Do we
have those skills in-house or must we acquire them?

March 8, 2010 178

Define the Data Technology Architecture

• Data technology architecture addresses three core questions
− What technologies are standard (which are required, preferred, or
acceptable)?
− Which technologies apply to which purposes and circumstances?
− In a distributed environment, which technologies exist where, and how does
data move from one node to another?
• Technology is never free - even open-source technology requires
maintenance
• Technology should always be regarded as the means to an end,
rather than the end itself
• Buying the same technology that everyone else is using, and using it
in the same way, does not create business value or competitive
advantage for the organisation

March 8, 2010 179


• Technology categories include
− Database management systems (DBMS)
− Database management utilities
− Data modelling and model management tools
− Extract-transform-load (ETL), changed data capture (CDC), and
other data integration tools

March 8, 2010 180


• Classify technology architecture components as
− Current - currently supported and used
− Deployment - deployed for use in the next 1-2 years
− Strategic - expected to be available for use in the next 2+ years
− Retirement - the organisation has retired or intends to retire this
year
− Preferred - preferred for use by most applications.
− Containment - limited to use by certain applications
− Emerging - being researched and piloted for possible future
deployment
• Create road map for the organisation consisting of these
components to helps govern future technology decisions
March 8, 2010 181

Evaluate Data Technology

• Selecting appropriate data related technology, particularly the
appropriate database management technology, is an important data
management responsibility
• Data technologies to be researched and evaluated include:
− Database management systems (DBMS) software
− Database utilities, such as backup and recovery tools, and performance
monitors
− Data modeling and model management software
− Database management tools, such as editors, schema generators, and
database object generators
− Extract-transfer-load (ETL) and other data integration tools
− Data virtualisation technology

March 8, 2010 182


• Use a standard technology evaluation process
− Understand user needs, objectives, and related requirements
− Understand the technology in general
− Identify available technology alternatives
− Identify the features required
− Weigh the importance of each feature
− Understand each technology alternative
− Evaluate and score each technology alternative’s ability to meet
requirements
− Calculate total scores and rank technology alternatives by score
− Evaluate the results, including the weighted criteria
− Present the case for selecting the highest ranking alternative

March 8, 2010 183


• Selecting strategic DBMS software is very important
• Factors to consider when selecting DBMS software include:
− Product architecture and complexity
− Application profile, such as transaction processing, business intelligence, and
personal profiles
− Organisational appetite for technical risk
− Hardware platform and operating system support
− Availability of supporting software tools
− Performance benchmarks
− Scalability
− Software, memory, and storage requirements
− Available supply of trained technical professionals
− Cost of ownership, such as licensing, maintenance, and computing resources
− Vendor reputation
− Vendor support policy and release schedule
− Customer references

March 8, 2010 184

Install and Administer Data Technology

• Need to deploy new technology products in development /
test, QA / certification, and production environments
• Create and document processes and procedures for
administering the product
• Cost and complexity of implementing new technology is
usually underestimated
• Features and benefits are usually overestimated
• Start with small pilot projects and proof-of-concept (POC)
implementations to get a good idea of the true costs and
benefits before proceeding with larger production
implementation
March 8, 2010 185

Inventory and Track Data Technology Licenses

• Comply with licensing agreements and regulatory
requirements
• Track and conduct yearly audits of software license and
annual support costs
• Track other costs such as server lease agreements and
other fixed costs
• Use data to determine the total cost-of-ownership (TCO)
for each type of technology and technology product
• Evaluate technologies and products that are becoming
obsolete, unsupported, less useful, or too expensive

March 8, 2010 186

Support Data Technology Usage and Issues

• Work with business users and application developers to
− Ensure the most effective use of the technology
− Explore new applications of the technology
− Address any problems or issues that surface from its use
• Training is important to effective understanding and use of
any technology

March 8, 2010 187

Data Security Management

March 8, 2010 188


• Planning, development, and execution of security policies
and procedures to provide proper authentication,
authorisation, access, and auditing of data and information
assets
• Effective data security policies and procedures ensure that
the right people can use and update data in the right way,
and that all inappropriate access and update is restricted
• Effective data security management function establishes
governance mechanisms that are easy enough to abide by
on a daily operational basis

March 8, 2010 189

Data Security Management – Definition and Goals

• Definition
− Planning, development, and execution of security policies and
procedures to provide proper authentication, authorisation,
access, and auditing of data and information.
• Goals
− Enable appropriate, and prevent inappropriate, access and
change to data assets
− Meet regulatory requirements for privacy and confidentiality
− Ensure the privacy and confidentiality needs of all stakeholders
are met

March 8, 2010 190


• Protect information assets in alignment with privacy and
confidentiality regulations and business requirements
− Stakeholder Concerns - organisations must recognise the privacy and
confidentiality needs of their stakeholders, including clients, patients, students,
citizens, suppliers, or business partners
− Government Regulations - government regulations protect some of the
stakeholder security interests. Some regulations restrict access to information,
while other regulations ensure openness, transparency, and accountability
− Proprietary Business Concerns - each organisation has its own proprietary data
to protect - ensuring competitive advantage provided by intellectual property
and intimate knowledge of customer needs and business partner relationships
is a cornerstone in any business plan
− Legitimate Access Needs - data security implementers must also understand
the legitimate needs for data access

March 8, 2010 191

Data Security Requirements and Procedures

• Data security requirements and the procedures to meet
these requirements
− Authentication - validate users are who they say they are
− Authorisation - identify the right individuals and grant them the
right privileges to specific, appropriate views of data
− Access - enable these individuals and their privileges in a timely
manner
− Audit - review security actions and user activity to ensure
compliance with regulations and conformance with policy and
standards

March 8, 2010 192

Data Security Management - Overview

•Business Goals
•Business Strategy
•Business Rules
•Business Process •Data Security Policies
•Data Strategy •Data Privacy and Confidentiality
•Data Privacy Issues Standards
•Related IT Policies and Standards •User Profiles, Passwords and
Memberships
Data Security •Data Security Permissions
•Data Security Controls
Suppliers
Management •Data Access Views
•Document Classifications
•Authentication and Access History
•Data Security Audits
•Data Stewards
•IT Steering Committee
•Data Stewardship Council
•Government
•Customers

Participants Tools Consumers
•Data Stewards
•Data Security Administrators •Data Producers
•Database Administrators •Database Management System •Knowledge Workers
•BI Analysts •Business Intelligence Tools •Managers
•Data Architects •Application Frameworks •Executives
•DM Leader •Identity Management Technologies •Customers
•CIO/CTO •Change Control Systems •Data Professionals
•Help Desk Analysts

March 8, 2010 193

Data Security Management Function, Activities and
Sub-Activities
Data Security
Management

Understand
Define Data Manage Users, Monitor User
Data Security Define Data Manage Data Classify
Define Data Security Passwords, and Authentication Audit Data
Needs and Security Access Views Information
Security Policy Controls and Group and Access Security
Regulatory Standards and Permissions Confidentially
Procedures Membership Behaviour
Requirements

Password
Business
Standards and
Requirements
Procedures

Regulatory
Requirements

March 8, 2010 194

Data Operations Management - Principles
• Be a responsible trustee of data about all parties. Understand and respect the privacy and confidentiality needs of all
stakeholders, be they clients, patients, students, citizens, suppliers, or business partners
• Understand and comply with all pertinent regulations and guidelines
• Data-to-process and data-to-role relationship (CRUD Create, Read, Update, Delete) matrices help map data access
needs and guide definition of data security role groups, parameters, and permissions
• Definition of data security requirements and data security policy is a collaborative effort involving IT security
administrators, data stewards, internal and external audit teams, and the legal department
• Identify detailed application security requirements in the analysis phase of every systems development project
• Classify all enterprise data and information products against a simple confidentiality classification schema
• Every user account should have a password set by the user following a set of password complexity guidelines, and
expiring every 45 to 60 days
• Create role groups; define privileges by role; and grant privileges to users by assigning them to the appropriate role
group. Whenever possible, assign each user to only one role group
• Some level of management must formally request, track, and approve all initial authorisations and subsequent
changes to user and group authorisations
• To avoid data integrity issues with security access information, centrally manage user identity data and group
membership data
• Use relational database views to restrict access to sensitive columns and / or specific rows
• Strictly limit and carefully consider every use of shared or service user accounts
• Monitor data access to certain information actively, and take periodic snapshots of data access activity to understand
trends and compare against standards criteria
• Periodically conduct objective, independent, data security audits to verify regulatory compliance and standards
conformance, and to analyse the effectiveness and maturity of data security policy and practice
• In an outsourced environment, be sure to clearly define the roles and responsibilities for data security and
understand the chain of custody data across organisations and roles.

March 8, 2010 195

Understand Data Security Needs and Regulatory
Requirements
• Distinguish between business rules and procedures and
the rules imposed by application software products
• Common for systems to have their own unique set of data
security requirements over and above those required
business processes

March 8, 2010 196

Business Requirements

• Implementing data security within an enterprise requires
an understanding of business requirements
• Business needs of an enterprise define the degree of
rigidity required for data security
• Business rules and processes define the security touch
points
• Data-to-process and data-to-role relationship matrices are
useful tools to map these needs and guide definition of
data security role-groups, parameters, and permissions
• Identify detailed application security requirements in the
analysis phase of every systems development project
March 8, 2010 197

Regulatory Requirements

• Organisations must comply with a growing set of
regulations
• Some regulations impose security controls on information
management

March 8, 2010 198

Define Data Security Policy

• Definition of data security policy based on data security
requirements is a collaborative effort involving IT security
administrators, data stewards, internal and external audit
teams, and the legal department
• Enterprise IT strategy and standards typically dictate high-
level policies for access to enterprise data assets
• Data security policies are more granular in nature and take
a very data-centric approach compared to an IT security
policy

March 8, 2010 199

Define Data Security Standards

• No one prescribed way of implementing data security to meet
privacy and confidentiality requirements
• Regulations generally focus on ensuring achieving an end without
defining them means for achieving it
• Organisations should design their own security controls,
demonstrate that the controls meet the requirements of the law or
regulations and document the implementation of those controls
• Information technology security standards can also affect
− Tools used to manage data security
− Data encryption standards and mechanisms
− Access guidelines to external vendors and contractors
− Data transmission protocols over the internet
− Documentation requirements
− Remote access standards
− Security breach incident reporting procedures
March 8, 2010 200

Define Data Security Standards

• Consider physical security, especially with the explosion of portable
devices and media, to formulate an effective data security strategy
− Access to data using mobile devices
− Storage of data on portable devices such as laptops, DVDs, CDs or USB drives
− Disposal of these devices in compliance with records management policies
• An organisation should develop a practical, implementable security
policy including data security guiding principles
• Focus should be on quality and consistency not creating a lengthy
body of guidelines
• Execution of the policy requires satisfying the elements of securing
information assets: authentication, authorisation, access, and audit
• Information classification, access rights, role groups, users, and
passwords are the means to implementing policy and satisfying
these elements

March 8, 2010 201

Define Data Security Controls and Procedures

• Implementation and administration of data security policy
is primarily the responsibility of security administrators
• Database security is often one responsibility of database
administrators
• Implement proper controls to meet the objectives of
relevant laws
• Implement a process to validate assigned permissions
against a change management system used for tracking all
user permission requests

March 8, 2010 202

Manage Users, Passwords, and Group Membership

• Role groups enable security administrators to define
privileges by role and to grant these privileges to users by
enrolling them in the appropriate role group
• Data consistency in user and group management is a
challenge in a mixed IT environment
• Construct group definitions at a workgroup or business
unit level
• Organise roles in a hierarchy, so that child roles further
restrict the privileges of parent roles

March 8, 2010 203

Password Standards and Procedures

• Passwords are the first line of defense in protecting access
to data
• Every user account should be required to have a password
set by the user with a sufficient level of password
complexity defined in the security standards

March 8, 2010 204

Manage Data Access Views and Permissions

• Data security management involves not just preventing
inappropriate access, but also enabling valid and appropriate access
to data
• Most sets of data do not have any restricted access requirements
• Control sensitive data access by granting permissions - opt-in
• Access control degrades when achieved through shared or service
accounts
− Implemented as convenience for administrators, these accounts often come
with enhanced privileges and are untraceable to any particular user or
administrator
− Enterprises using shared or service accounts run the risk of data security
breaches
− Evaluate use of such accounts carefully, and never use them frequently or by
default
March 8, 2010 205

Monitor User Authentication and Access Behaviour

• Monitoring authentication and access behaviour is critical
because:
− It provides information about who is connecting and accessing
information assets, which is a basic requirement for compliance
auditing
− It alerts security administrators to unforeseen situations,
compensating for oversights in data security planning, design, and
implementation
• Monitoring helps detect unusual or suspicious transactions
that may warrant further investigation and issue resolution
• Perform monitoring either actively or passively
• Automated systems with human checks and balances in
place best accomplish both methods
March 8, 2010 206

Classify Information Confidentiality

• Classify an organisation’s data and information using a simple
confidentiality classification schema
• Most organisations classify the level of confidentiality for
information found within documents, including reports
• A typical classification schema might include the following five
confidentiality classification levels:
− For General Audiences: Information available to anyone, including the general
public
− Internal Use Only: Information limited to employees or members, but with
minimal risk if shared
− Confidential: Information which should not be shared outside the organisation.
Client Confidential information may not be shared with other clients
− Restricted Confidential: Information limited to individuals performing certain
roles with the need to know
− Registered Confidential: Information so confidential that anyone accessing the
information must sign a legal agreement to access the data and assume
responsibility for its secrecy

March 8, 2010 207

Audit Data Security

• Auditing data security is a recurring control activity with
responsibility to analyse, validate, counsel, and recommend policies,
standards, and activities related to data security management
• Auditing is a managerial activity performed with the help of analysts
working on the actual implementation and details
• The goal of auditing is to provide management and the data
governance council with objective, unbiased assessments, and
rational, practical recommendations
• Auditing data security is no substitute for effective management of
data security
• Auditing is a supportive, repeatable process, which should occur
regularly, efficiently, and consistently

March 8, 2010 208

Audit Data Security

• Auditing data security includes
− Analysing data security policy and standards against best practices and needs
− Analysing implementation procedures and actual practices to ensure
consistency with data security goals, policies, standards, guidelines, and
desired outcomes
− Assessing whether existing standards and procedures are adequate and in
alignment with business and technology requirements
− Verifying the organisation is in compliance with regulatory requirements
− Reviewing the reliability and accuracy of data security audit data
− Evaluating escalation procedures and notification mechanisms in the event of a
data security breach
− Reviewing contracts, data sharing agreements, and data security obligations of
outsourced and external vendors, ensuring they meet their obligations, and
ensuring the organisation meets its obligations for externally sourced data
− Reporting to senior management, data stewards, and other stakeholders on
the state of data security within the organisation and the maturity of its
practices
− Recommending data security design, operational, and compliance
improvements

March 8, 2010 209

Data Security and Outsourcing

• Outsourcing IT operations introduces additional data security
challenges and responsibilities
• Outsourcing increases the number of people who share
accountability for data across organisational and geographic
boundaries
• Previously informal roles and responsibilities must now be explicitly
defined as contractual obligations
• Outsourcing contracts must specify the responsibilities and
expectations of each role
• Any form of outsourcing increases risk to the organisation
• Data security risk is escalated to include the outsource vendor, so
any data security measures and processes must look at the risk from
the outsource vendor not only as an external risk, but also as an
internal risk
March 8, 2010 210

Data Security and Outsourcing

• Transferring control, but not accountability, requires tighter risk
management and control mechanisms:
− Service level agreements
− Limited liability provisions in the outsourcing contract
− Right-to-audit clauses in the contract
− Clearly defined consequences to breaching contractual obligations
− Frequent data security reports from the service vendor
− Independent monitoring of vendor system activity
− More frequent and thorough data security auditing
− Constant communication with the service vendor
• In an outsourced environment, it is important to maintain and track
the lineage, or flow, of data across systems and individuals to
maintain a chain of custody

March 8, 2010 211


March 8, 2010 212


• Reference and Master Data Management is the ongoing
reconciliation and maintenance of reference data and master data
− Reference Data Management is control over defined domain values (also
known as vocabularies), including control over standardised terms, code values
and other unique identifiers, business definitions for each value, business
relationships within and across domain value lists, and the consistent, shared
use of accurate, timely and relevant reference data values to classify and
categorise data
− Master Data Management is control over master data values to enable
consistent, shared, contextual use across systems, of the most accurate,
timely, and relevant version of truth about essential business entities
• Reference data and master data provide the context for transaction
data

March 8, 2010 213

Reference and Master Data Management –
Definition and Goals
• Definition
− Planning, implementation, and control activities to ensure
consistency with a golden version of contextual data values
• Goals
− Provide authoritative source of reconciled, high-quality master
and reference data
− Lower cost and complexity through reuse and leverage of
standards
− Support business intelligence and information integration efforts

March 8, 2010 214

Reference and Master Data Management - Overview

•Business Drivers
•Data Requirements Policy and •Master and Reference Data
Regulations Requirements
•Standards •Data Models and Documentation
•Code Sets •Reliable Reference and Master Data
•Master Data •Golden Record Data Lineage
•Transactional Data •Data Quality Metrics and Reports
Reference and •Data Cleansing Services

Suppliers Master Data
Management Consumers
•Steering Committees
•Business Data Stewards
•Subject Matter Experts •Application Users
•Data Consumers •BI and Reporting Users
•Standards Organisations Tools •Application Developers and Architects
•Data Providers •Data Integration Developers and
Architects
•Reference Data Management •BI Developers and Architects
Applications •Vendors, Customers, and Partners
Participants •Master Data Management
Applications
•Data Stewards •Data Modeling Tools Metrics
•Subject Matter Experts •Process Modeling Tools
•Data Architects •Metadata Repositories •Reference and Master Data Quality
•Data Analysts •Data Profiling Tools •Change Activity
•Application Architects •Data Cleansing Tools •Issues, Costs, Volume
•Data Governance Council •Data Integration Tools •Use and Re-Use
•Data Providers •Business Process and Rule Engines •Availability
•Other IT Professionals Change Management Tools •Data Steward Coverage

March 8, 2010 215

Reference and Master Data Management Function,
Activities and Sub-Activities
Reference
and Master
Data
Management

Understand Identify Implement
Define and Define and Plan and Replicate and Manage
Reference Reference Reference
Maintain the Define and Establish Maintain Implement Distribute Changes to
Reference and Master and Master and Master
Master Data Data Maintain Golden Hierarchies Integration of Reference Reference
Data Data Data Sources Data
integration Match Rules Records and New Data and Master and Master
Integration and Management
Architecture Affiliations Sources Data Data
Needs Contributors Solutions

Vocabulary
Management
Party Master
and
Data
Reference
Data

Defining
Financial Golden
Master Data Master Data
Values

Product
Master Data

Location
Master Data

March 8, 2010 216

Reference and Master Data Management -
Principles
• Shared reference and master data belongs to the organisation, not to a particular
application or department
• Reference and master data management is an on-going data quality improvement
program; its goals cannot be achieved by one project alone
• Business data stewards are the authorities accountable for controlling reference
data values. Business data stewards work with data professionals to improve the
quality of reference and master data
• Golden data values represent the organisation’s best efforts at determining the
most accurate, current, and relevant data values for contextual use. New data
may prove earlier assumptions to be false. Therefore, apply matching rules with
caution, and ensure that any changes that are made are reversible
• Replicate master data values only from the database of record
• Request, communicate, and, in some cases, approve of changes to reference data
values before implementation

March 8, 2010 217

Reference Data

• Reference data is data used to classify or categorise other
data
• Business rules usually dictate that reference data values
conform to one of several allowed values
• In all organisations, reference data exists in virtually every
database
• Reference tables link via foreign keys into other relational
database tables, and the referential integrity functions
within the database management system ensure only valid
values from the reference tables are used in other tables

March 8, 2010 218

Master Data

• Master data is data about the business entities that
provide context for business transactions
• Master data is the authoritative, most accurate data
available about key business entities, used to establish the
context for transactional data
• Master data values are considered golden
• Master Data Management is the process of defining and
maintaining how master data will be created, integrated,
maintained, and used throughout the enterprise

March 8, 2010 219

Master Data Challenges

• What are the important roles, organisations, places, and things referenced
repeatedly?
• What data is describing the same person, organisation, place, or thing?
• Where is this data stored? What is the source for the data?
• Which data is more accurate? Which data source is more reliable and credible?
Which data is most current?
• What data is relevant for specific needs? How do these needs overlap or conflict?
• What data from multiple sources can be integrated to create a more complete
view and provide a more comprehensive understanding of the person,
organisation, place or thing?
• What business rules can be established to automate master data quality
improvement by accurately matching and merging data about the same person,
organisation, place, or thing?
• How do we identify and restore data that was inappropriately matched and
merged?
• How do we provide our golden data values to other systems across the
enterprise?
• How do we identify where and when data other than the golden values is used?
March 8, 2010 220

Party Master Data

• Includes data about individuals, organisations, and the roles they
play in business relationships
• Customer relationship management (CRM) systems perform MDM
for customer data (also called Customer Data Integration (CDI))
• Focus is to provide the most complete and accurate information
about each and every customer
• Need to identify duplicate, redundant and conflicting data
• Party master data issues
− Complexity of roles and relationships played by individuals and organisations
− Difficulties in unique identification
− High number of data sources
− Business importance and potential impact of the data

March 8, 2010 221

Financial Master Data

• Includes data about business units, cost centers, profit
centers, general ledger accounts, budgets, projections, and
projects
• Financial MDM solutions focus on not only creating,
maintaining, and sharing information, but also simulating
how changes to existing financial data may affect the
organisation’s bottom line

March 8, 2010 222

Product Master Data

• Product master can consists of information on an
organisation’s products and services or on the entire
industry in which the organisation operates, including
competitor products, and services
• Product Lifecycle Management (PLM) focuses on managing
the lifecycle of a product or service from its conception
(such as research), through its development,
manufacturing, sale / delivery, service, and disposal

March 8, 2010 223

Location Master Data

• Provides the ability to track and share reference
information about different geographies, and create
hierarchical relationships or territories based on
geographic information to support other processes
• Different industries require specialised earth science data
(geographic data about seismic faults, flood plains, soil,
annual rainfall, and severe weather risk areas) and related
sociological data (population, ethnicity, income, and
terrorism risk), usually supplied from external sources

March 8, 2010 224

Understand Reference and Master Data Integration
Needs
• Reference and master data requirements are relatively
easy to discover and understand for a single application
• Potentially much more difficult to develop an
understanding of these needs across applications,
especially across the entire organisation
• Analysing the root causes of a data quality problem usually
uncovers requirements for reference and master data
integration
• Organisations that have successfully managed reference
and master data typically have focused on one subject
area at a time
− Analyse all occurrences of a few business entities, across all
physical databases and for differing usage patterns
March 8, 2010 225

Identify Reference and Master Data Sources and
Contributors
• Successful organisations first understand the needs for
reference and master data
• Then trace the lineage of this data to identify the original
and interim source databases, files, applications,
organisations and the individual roles that create and
maintain the data
• Understand both the upstream sources and the
downstream needs to capture quality data at its source

March 8, 2010 226

Define and Maintain the Data integration
Architecture
• Effective data integration architecture controls the shared access, replication, and
flow of data to ensure data quality and consistency, particularly for reference and
master data
• Without data integration architecture, local reference and master data
management occurs in application silos, inevitably resulting in redundant and
inconsistent data
• The selected data integration architecture should also provide common data
integration services
− Change request processing, including review and approval
− Data quality checks on externally acquired reference and master data
− Consistent application of data quality rules and matching rules
− Consistent patterns of processing
− Consistent metadata about mappings, transformations, programs and jobs
− Consistent audit, error resolution and performance monitoring data
− Consistent approach to replicating data
• Establishing master data standards can be a time consuming task as it may involve
multiple stakeholders.
• Apply the same data standards, regardless of integration technology, to enable
effective standardisation, sharing, and distribution of reference and master data

March 8, 2010 227

Data Integration Services Architecture

Data Acquisition, File Data Standardisation Replication
Management and Cleansing and Management
Audit Matching

Source Data Rules Reconciled
Master Data

Archives Errors Subscriptions

Staging

MetaData Management
Business Integration Job Flow and
Metadata Metadata Statistics

March 8, 2010 228

Implement Reference and Master Data
Management Solutions
• Reference and master data management solutions are
complex
• Given the variety, complexity, and instability of
requirements, no single solution or implementation
project is likely to meet all reference and master data
management needs
• Organisations should expect to implement reference and
master data management solutions iteratively and
incrementally through several related projects and phases

March 8, 2010 229

Define and Maintain Match Rules

• Matching, merging, and linking of data from multiple systems about
the same person, group, place, or thing is a major master data
management challenge
• Matching attempts to remove redundancy, to improve data quality,
and provide information that is more comprehensive
• Data matching is performed by applying inference rules
− Duplicate identification match rules focus on a specific set of fields that
uniquely identify an entity and identify merge opportunities without taking
automatic action
− Match-merge rules match records and merge the data from these records into
a single, unified, reconciled, and comprehensive record.
− Match-link rules identify and cross-reference records that appear to relate to a
master record without updating the content of the cross-referenced record

March 8, 2010 230

Establish Golden Records

• Establishing golden master data values requires more
inference, application of matching rules, and review of the
results

March 8, 2010 231

Vocabulary Management and Reference Data

• A vocabulary is a collection of terms / concepts and their
relationships
• Vocabulary management is defining, sourcing, importing,
and maintaining a vocabulary and its associated reference
data
− See ANSI/NISO Z39.19 - Guidelines for the Construction, Format,
and Management of Monolingual Controlled Vocabularies -
http://www.niso.org/kst/reports/standards?step=2&gid=&project
_key=7cc9b583cb5a62e8c15d3099e0bb46bbae9cf38a
• Vocabulary management requires the identification of the
standard list of preferred terms and their synonyms
• Vocabulary management requires data governance,
enabling data stewards to assess stakeholder needs
March 8, 2010 232

Vocabulary Management and Reference Data

• Key questions to ask to enable vocabulary management
− What information concepts (data attributes) will this vocabulary support?
− Who is the audience for this vocabulary? What processes do they support, and
what roles do they play?
− Why is the vocabulary needed? Will it support applications, content
management, analytics, and so on?
− Who identifies and approves the preferred vocabulary and vocabulary terms?
− What are the current vocabularies different groups use to classify this
information? Where are they located? How were they created? Who are their
subject matter experts? Are there any security or privacy concerns for any of
them?
− Are there existing standards that can be leveraged to fulfill this need? Are
there concerns about using an external standard vs. internal? How frequently
is the standard updated and what is the degree of change of each update? Are
standards accessible in an easy to import / maintain format in a cost efficient
manner?

March 8, 2010 233

Defining Golden Master Data Values

• Golden data values are the data values thought to be the most
accurate, current, and relevant for shared, consistent use across
applications
• Determine golden values by analyssing data quality, applying data
quality rules and matching rules, and incorporating data quality
controls into the applications that acquire, create, and update data
• Establish data quality measurements to set expectations, measure
improvements, and help identify root causes of data quality
problems
• Assess data quality through a combination of data profiling activities
and verification against adherence to business rules
• Once the data is standardised and cleansed, the next step is to
attempt reconciliation of redundant data through application of
matching rules

March 8, 2010 234

Define and Maintain Hierarchies and Affiliations

• Vocabularies and their associated reference data sets are
often more than lists of preferred terms and their
synonyms
• Affiliation management is the establishment and
maintenance of relationships between master data
records

March 8, 2010 235

Plan and Implement Integration of New Data
Sources
• Integrating new reference data sources involves
− Receiving and responding to new data acquisition requests from
different groups
− Performing data quality assessment services using data cleansing
and data profiling tools
− Assessing data integration complexity and cost
− Piloting the acquisition of data and its impact on match rules
− Determining who will be responsible for data quality
− Finalising data quality metrics

March 8, 2010 236

Replicate and Distribute Reference and Master Data

• Reference and master data may be read directly from a
database of record, or may be replicated from the
database of record to other application databases for
transaction processing, and data warehouses for business
intelligence
• Reference data most commonly appears as pick list values
in applications
• Replication aids maintenance of referential integrity

March 8, 2010 237

Manage Changes to Reference and Master Data

• Specific individuals have the role of a business data
steward with the authority to create, update, and retire
reference data
• Formally control changes to controlled vocabularies and
their reference data sets
• Carefully assess the impact of reference data changes

March 8, 2010 238

Data Warehousing and Business Intelligence
Management

March 8, 2010 239

Management
• A Data Warehouse is a combination of two primary
components
− An integrated decision support database
− Related software programs used to collect, cleanse, transform,
and store data from a variety of operational and external sources
• Both components combine to support historical, analytical,
and business intelligence (BI) requirements
• A Data Warehouse may also include dependent data
marts, which are subset copies of a data warehouse
database
• A Data Warehouse includes any data stores or extracts
used to support the delivery of data for BI purposes

March 8, 2010 240

Management
• Data Warehousing means the operational extract, cleansing,
transformation, and load processes and associated control processes
that maintain the data contained within a data warehouse
• Data Warehousing process focuses on enabling an integrated and
historical business context on operational data by enforcing business
rules and maintaining appropriate business data relationships and
processes that interact with metadata repositories
• Business Intelligence is a set of business capabilities including
− Query, analysis, and reporting activity by knowledge workers to monitor and
understand the financial operation health of, and make business decisions
about, the enterprise
− Strategic and operational analytics and reporting on corporate operational
data to support business decisions, risk management, and compliance

March 8, 2010 241

Management
• Together Data Warehousing and Business Intelligence
Management is the collection, integration, and
presentation of data to knowledge workers for the
purpose of business analysis and decision-making
• Composed of activities supporting all phases of the
decision support life cycle that provides context
− Moves and transforms data from sources to a common target
data store
− Provides knowledge workers various means of access,
manipulation
− Reporting of the integrated target data

March 8, 2010 242

Management – Definition and Goals
• Definition
− Planning, implementation, and control processes to provide
decision support data and support knowledge workers engaged in
reporting, query and analysis
• Goals
− To support and enable effective business analysis and decision
making by knowledge workers
− To build and maintain the environment / infrastructure to support
business intelligence activity, specifically leveraging all the other
data management functions to cost effectively deliver consistent
integrated data for all BI activity

March 8, 2010 243

Management - Overview
•Business Drivers
•BI Data and Access Requirements
•Data Quality Requirements •DW/BI Architecture
•Data Security Requirements •Data Warehouses
•Data Architecture •Data Marts and OLAP Cubes
•Technical Architecture •Dashboards and Scorecards
•Data Modeling Standards and Guidelines
•Transactional Data
Data Warehousing •Analytic Applications
•File Extracts (for Data Mining/Statistical
•Master and Reference Data
•Industry and External Data and Business Tools)
•BI Tools and User Environments
•Data Quality Feedback Mechanism/Loop
Suppliers
Intelligence
•Executives and Managers
Management
•Subject Matter Experts Consumers
•Data Governance Council
•Information Consumers (Internal and
External)
•Data Producers •Knowledge Workers
•Data Architects and Analysts Tools •Managers and Executives
•External Customers and Systems
•Internal Customers and Systems
Participants •Data Professionals Other IT Professionals
•Database Management Systems
•Business Executives and Managers •Data Profiling Tools
•DM Execs and Other IT Management •Data Integration Tools
•BI Program Manage •Data Cleansing Tools
•SMEs and Other Information Consumers •Business Intelligence Tools Metrics
•Data Stewards •Analytic Applications
•Project Managers •Data Modeling Tools
•Data Architects and Analysts •Performance Management Tools
•Data Integration (ETL) Specialists •Usage Metrics
•Metadata Repository •Customer/User Satisfaction
•BI Specialists •Data Quality Tools
•Database Administrators •Subject Area Coverage %
•Data Security Tools •Response/Performance Metrics
•Data Security Administrators
•Data Quality Analysts

March 8, 2010 244

Management Objectives
• Providing integrated storage of required current and historical data, organised by
subject areas
• Ensuring credible, quality data for all appropriate access capabilities
• Ensuring a stable, high-performance, reliable environment for data acquisition,
data management, and data access
• Providing an easy-to-use, flexible, and comprehensive data access environment
• Delivering both content and access to the content in increments appropriate to
the organisation’s objectives
• Leveraging, rather than duplicating, relevant data management component
functions such as Reference and Master Data Management, Data Governance,
Data Quality, and Metadata
• Providing an enterprise focal point for data delivery in support of the decisions,
policies, procedures, definitions, and standards that arise from DG
• Defining, building, and supporting all data stores, data processes, data
infrastructure, and data tools that contain integrated, post-transactional, and
refined data used for information viewing, analysis, or data request fulfillment
• Integrating newly discovered data as a result of BI processes into the DW for
further analytics and BI use.

March 8, 2010 245

Management Function, Activities and Sub-Activities Data Warehousing
and Business
Intelligence
Management

Understand Business Define and Maintain Implement Data Implement Business Monitor and Tune Monitor and Tune BI
Process Data for
Intelligence the DW-BI Warehouses and Data Intelligence Tools and Data Warehousing Activity and
Business Intelligence
Information Needs Architecture Marts User Interfaces Processes Performance

Query and Reporting
Staging Areas
Tools

On Line Analytical
Mapping Sources and
Processing (OLAP)
Targets
Tools

Data Cleansing and
Analytic Applications Transformations
(Data Acquisition)

Implementing
Management
Dashboards and
Scorecards

Performance
Management Tools

Predictive Analytics
and Data Mining Tools

Advanced
Visualisation and
Discovery Tools
March 8, 2010 246

Management Principles
• Obtain executive commitment and support as these projects are labour intensive
• Secure business SMEs as their support and high availability are necessary for getting the correct data
and useful BI solution
• Be business focused and driven. Make sure DW / BI work is serving real priority business needs and
solving burning business problems. Let the business drive the prioritisation
• Demonstrable data quality is essential
• Provide incremental value. Ideally deliver in continual 2-3 month segments
• Transparency and self service. The more context (metadata of all kinds) provided, the more value
customers derive. Wisely exposing information about the process reduces calls and increases
satisfaction.
• One size does not fit all. Make sure you find the right tools and products for each of your customer
segments
• Think and architect globally, act and build locally. Let the big-picture and end- vision guide the
architecture, but build and deliver incrementally, with much shorter term and more project-based
focus
• Collaborate with and integrate all other data initiatives, especially those for data governance, data
quality, and metadata
• Start with the end in mind. Let the business priority and scope of end-data- delivery in the BI space
drive the creation of the DW content. The main purpose for the existence of the DW is to serve up data
to the end business customers via the BI capabilities
• Summarise and optimise last, not first. Build on the atomic data and add aggregates or summaries as
needed for performance, but not to replace the detail.

March 8, 2010 247

Understand Business Intelligence Information Needs

• All projects start with requirements
• Gathering requirements for DW-BIM projects has both similarities to and differences from
gathering requirements for other projects
• For DW-BIM projects, it is important to understand the broader business context of the
business area targeted as reporting is generalised and exploratory
• Capturing the actual business vocabulary and terminology is a key to success
• Document the business context, then explore the details of the actual source data
• Typically, the ETL portion can consume 60%-70% of a DW-BIM project’s budget and time
• The DW is often the first place where the pain of poor quality data in source systems and /
or data entry functions becomes apparent
• Creating an executive summary of the identified business intelligence needs is a best
practice
• When starting a DW-BIM programme, a good way to decide where to start is using a simple
assessment of business impact and technical feasibility
− Technical feasibility will take into consideration things like complexity, availability and state of the
data, and the availability of subject matter experts
− Projects that have high business impact and high technical feasibility are good candidates for
starting.

March 8, 2010 248

Define and Maintain the DW-BI Architecture

• Successful DW-BIM architecture requires the identification and
bringing together of a number of key roles
− Technical Architect - hardware, operating systems, databases and DW-BIM
architecture
− Data Architect - data analysis, systems of record, data modeling and data
mapping
− ETL Architect / Design Lead - staging and transform, data marts, and schedules
− Metadata Specialist - metadata interfaces, metadata architecture and
contents
− BI Application Architect / Design Lead - BI tool interfaces and report design,
metadata delivery, data and report navigation and delivery
• Technical requirements including performance, availability, and
timing needs are key drivers in developing the DW-BIM architecture
• The design decisions and principles for what data detail the DW
contains is a key design priority for DW-BIM architecture
• Important that the DW-BIM architecture integrate with the overall
corporate reporting architecture

March 8, 2010 249

Define and Maintain the DW-BI Architecture

• No DW-BIM effort can be successful without business acceptance of
data
• Business acceptance includes the data being understandable, having
verifiable quality and having a demonstrable origin
• Sign-off by the Business on the data should be part of the User
Acceptance Testing
• Structured random testing of the data in the BIM tool against data in
the source systems over the initial load and a few update load cycles
should be performed to meet sign-off criteria
• Meeting these requirements is paramount for every DW-BIM
architecture

March 8, 2010 250

Implement Data Warehouses and Data Marts

• The purpose of a data warehouse is to integrate data from multiple
sources and then serve up that integrated data for BI purposes
• Consumption is typically through data marts or other systems
• A single data warehouse will integrate data from multiple source
systems and serve data to multiple data marts
• Purpose of data marts is to provide data for analysis to knowledge
workers
• Start with the end in mind - identify the business problem to solve,
then identify the details and what would be used and continue to
work back into the integrated data required and ultimately all the
way back to the data sources.

March 8, 2010 251

Implement Business Intelligence Tools and User
Interfaces
• Well defined set of well-proven BI tools
• Implementing the right BI tool or User Interface (UI) is
about identifying the right tools for the right user set
• Almost all BI tools also come with their own metadata
repositories to manage their internal data maps and
statistics

March 8, 2010 252

Query and Reporting Tools

• Query and reporting is the process of querying a data
source and then formatting it to create a report
• With business query and reporting the data source is more
often a data warehouse or data mart
• While IT develops production reports, power users and
casual business users develop their own reports with
business query tools
• Business query and reporting tools enable users who want
to author their own reports or create outputs for use by
others

March 8, 2010 253

Query and Reporting Tools Landscape

Customers, Suppliers
and Regulators

Published
Frontline Workers Reports

Embedded BI Executives and
Managers
Scorecards
Analysts and
Dashboards Information Workers
Interactive
IT Developers Fixed
OLAP Reports

BI Spreadsheets Business
Production Reporting Tools Statistics Query

Commonly Commonly
Specialist Tools
Used Tools Used Tools
March 8, 2010 254

On Line Analytical Processing (OLAP) Tools

• OLAP provides interactive, multi-dimensional analysis with different
dimensions and different levels of detail
• The value of OLAP tools and cubes is reduction of the chance of
confusion and erroneous interpretation by aligning the data content
with the analyst's mental model
• Common OLAP operations include slice and dice, drill down, drill up,
roll up, and pivot
− Slice - a slice is a subset of a multi-dimensional array corresponding to a single
value for one or more members of the dimensions not in the subset
− Dice - the dice operation is a slice on more than two dimensions of a data
cube, or more than two consecutive slices
− Drill Down / Up - drilling down or up is a specific analytical technique whereby
the user navigates among levels of data, ranging from the most summarised
(up) to the most detailed (down)
− Roll-Up – a roll-up involves computing all of the data relationships for one or
more dimensions. To do this, define a computational relationship or formula
− Pivot - to change the dimensional orientation of a report or page display

March 8, 2010 255

Analytic Applications

• Analytic applications include the logic and processes to
extract data from well-known source systems, such as
vendor ERP systems, a data model for the data mart, and
pre-built reports and dashboards
• Analytic applications provide businesses with a pre-built
solution to optimise a functional area or industry vertical
• Different types of analytic applications include customer,
financial, supply chain, manufacturing, and human
resource applications

March 8, 2010 256

Implementing Management Dashboards and
Scorecards
• Dashboards and scorecards are both ways of efficiently
presenting performance information
• Dashboards are oriented more toward dynamic
presentation of operational information while scorecards
are more static representations of longer-term
organisational, tactical, or strategic goals
• Typically, scorecards are divided into 4 quadrants or views
of the organisation such as Finance, Customer,
Environment, and Employees, each with a number of
metrics

March 8, 2010 257

Performance Management Tools

• Performance management applications include budgeting,
planning, and financial consolidation

March 8, 2010 258

Predictive Analytics and Data Mining Tools

• Data mining is a particular kind of analysis that reveals
patterns in data using various algorithms
• A data mining tool will help users discover relationships or
show patterns in more exploratory fashion

March 8, 2010 259

Advanced Visualisation and Discovery Tools

• Advanced visualisation and discovery tools allow users to
interact with the data in a highly visual, interactive way
• Patterns in a large dataset can be difficult to recognise in a
numbers display
• A pattern can be picked up visually fairly quickly when
thousands of data points are loaded into a sophisticated
display on a single page of display

March 8, 2010 260

Process Data for Business Intelligence

• Most of the work in any DW-BIM effort involves in the
preparation and processing of the data

March 8, 2010 261

Staging Areas

• A staging area is the intermediate data store between an
original data source and the centralised data repository
• All required cleansing, transformation, reconciliation, and
relationships happen in this area

March 8, 2010 262

Mapping Sources and Targets

• Source-to-target mapping is the documentation activity that defines
data type details and transformation rules for all required entities
and data elements and from each individual source to each
individual target
• DW-BIM adds additional requirements to this source-to-target
mapping process encountered as a component of any typical data
migration
• One of the goals of the DW-BIM effort should be to provide a
complete lineage for each data element available in the BI
environment all the way back to its respective source(s)
• A solid taxonomy is necessary to match the data elements in
different systems into a consistent structure in the EDW

March 8, 2010 263

Data Cleansing and Transformations (Data
Acquisition)
• Data cleansing focuses on the activities that correct and enhance the
domain values of individual data elements, including enforcement of
standards
• Cleansing is particularly necessary for initial loads where significant
history is involved
• The preferred strategy is to push data cleansing and correction
activity back to the source systems whenever possible
• Data transformation focuses on activities that provide organisational
context between data elements, entities, and subject areas
• Organisational context includes cross- referencing, reference and
master data management and complete and correct relationships
• Data transformation is an essential component of being able to
integrate data from multiple sources

March 8, 2010 264

Monitor and Tune Data Warehousing Processes

• Processing should be monitored across the system for
bottlenecks and dependencies among processes
• Database tuning techniques should be employed where
and when needed, including partitioning, tuned backup
and recovery strategies
• Archiving is a difficult subject in data warehousing
• Users often consider the data warehouse as an active
archive due to the long histories that are built, and are
unwilling, particularly if the OLAP sources have dropped
records, to see the data warehouse engage in archiving

March 8, 2010 265

Monitor and Tune BI Activity and Performance

• A best practice for BI monitoring and tuning is to define
and display a set of customer- facing satisfaction metrics
• Average query response time and the number of users per
day / week / month, are examples of useful metrics to
display
• Regular review of usage statistics and patterns is essential
• Reports providing frequency and resource usage of data,
queries, and reports allow prudent enhancement
• Tuning BI activity is analogous to the principle of profiling
applications in order to know where the bottlenecks are
and where to apply optimisation efforts
March 8, 2010 266

Document and Content Management

March 8, 2010 267


• Document and Content Management is the control over capture,
storage, access, and use of data and information stored outside
relational databases
• Strategic and tactical focus overlaps with other data management
functions in addressing the need for data governance, architecture,
security, managed metadata, and data quality for unstructured data
• Document and Content Management includes two sub-functions:
− Document management is the storage, inventory, and control of electronic and
paper documents. Document management encompasses the processes,
techniques, and technologies for controlling and organising documents and
records, whether stored electronically or on paper
− Content management refers to the processes, techniques, and technologies for
organising, categorising, and structuring access to information content,
resulting in effective retrieval and reuse. Content management is particularly
important in developing websites and portals, but the techniques of indexing
based on keywords, and organising based on taxonomies, can be applied
across technology platforms.
March 8, 2010 268

Document and Content Management – Definition
and Goals
• Definition
− Planning, implementation, and control activities to store, protect,
and access data found within electronic files and physical records
(including text, graphics, images, audio, and video)
• Goals
− To safeguard and ensure the availability of data assets stored in
less structured formats
− To enable effective and efficient retrieval and use of data and
information in unstructured formats
− To comply with legal obligations and customer expectations
− To ensure business continuity through retention, recovery, and
conversion
− To control document storage operating costs

March 8, 2010 269

Document and Content Management - Overview
•Text Documents •Managed Records in Many Media
•Reports Formats
•Spreadsheets •E-discovery Records
•Email •Outgoing Letters and Emails
•Instant Messages •Contracts and Financial Documents
•Faxes •Policies and Procedures
•Voicemail •Audit Trails and Logs
•Images •Meeting Minutes
•Video Recordings
•Audio Recordings
Document and •Formal Reports
•Significant Memoranda
•Printed Paper Files
•Microfiche/Microfilm
Content
•Graphics Management Consumers
Suppliers
•Business and IT Users
•Employees Tools •Government Regulatory Agencies
•External Parties • Senior Management
•External Customers

•Stored Documents
Participants •Office Productivity Tools
•All Employees •Image and Workflow
•Data Stewards Management Tools Metrics
•DM Professionals •Records Management Tools
•Records Management Staff •XML Development Tools
•Other IT Professionals •Collaboration Tools •Return on investment
•Data Management Executive •Internet •Key Performance Indicators
•Other IT Managers •Email Systems •Balanced Scorecards
•Chief Information Officer
•Chief Knowledge Officer
March 8, 2010 270

Document and Content Management Function,
Activities and Sub-Activities

Document / Record Management Content Management

Define and Maintain Enterprise
Plan for Managing Documents / Records Taxonomies (Information Content
Architecture)

Implement Document / Record
Document / Index Information Content
Management Systems for Acquisition,
Metadata
Storage, Access, and Security Controls

Backup and Recover Documents /
Provide Content Access and Retrieval
Records

Retention and Disposition of Documents
Govern for Quality Content
/ Records

Audit Document / Records Management

March 8, 2010 271

Document and Content Management - Principles

• Everyone in an organisation has a role to play in protecting
its future. Everyone must create, use, retrieve, and dispose
of records in accordance with the established policies and
procedures
• Experts in the handling of records and content should be
fully engaged in policy and planning. Regulatory and best
practices can vary significantly based on industry sector
and legal jurisdiction
• Even if records management professionals are not
available to the organisation, everyone can be trained and
have an understanding of the issues. Once trained,
business stewards and others can collaborate on an
effective approach to records management

March 8, 2010 272


• A document management system is an application used to track and
store electronic documents and electronic images of paper
documents
• Document management systems commonly provide storage,
versioning, security, metadata management, content indexing, and
retrieval capabilities
• A content management system is used to collect, organise, index,
and retrieve information content; storing the content either as
components or whole documents, while maintaining links between
components
• While a document management system may provide content
management functionality over the documents under its control, a
content management system is essentially independent of where
and how the documents are stored

March 8, 2010 273

Document / Record Management

• Document / Record Management is the lifecycle management of the
designated significant documents of the organisation
• Records can
− Physical such as documents, memos, contracts, reports or microfiche
− Electronic such as email content, attachments, and instant messaging
− Content on a website
− Documents on all types of media and hardware
− Data captured in databases of all kinds
• More than 90% of the records created today are electronic
• Growth in email and instant messaging has made the management
of electronic records critical to an organisation

March 8, 2010 274

Document / Record Management

• The lifecycle of Document / Record Management includes:
− Identification of existing and newly created documents / records
− Creation, Approval, and Enforcement of documents / records
policies
− Classification of documents / records
− Documents / Records Retention Policy
− Storage: Short and long term storage of physical and electronic
documents / records
− Retrieval and Circulation: Allowing access and circulation of
documents / records in accordance with policies, security and
control standards, and legal requirements
− Preservation and Disposal: Archiving and destroying documents /
records according to organisational needs, statutes, and
regulations

March 8, 2010 275

Plan for Managing Documents / Records

• Plan document lifecycle from creation or receipt, organisation for
retrieval, distribution and archiving or disposition
• Develop classification / indexing systems and taxonomies so that the
retrieval of documents is easy
• Create planning and policy around documents and records on the
value of the data to the organisation and as evidence of business
transactions
• Identify the responsible, accountable organisational unit for
managing the documents / records
• Develop and execute a retention plan and policy to archive, such as
selected records for long-term preservation
• Records are destroyed at the end of their lifecycle according to
operational needs, procedures, statutes and regulations

March 8, 2010 276

Implement Document / Record Management Systems for
Acquisition, Storage, Access, and Security Controls

• Documents can be created within a document management system
or captured via scanners or OCR software
• Electronic documents must be indexed via keywords or text during
the capture process so that the document can be found
• A document repository enables check-in and check-out features,
versioning, collaboration, comparison, archiving, status state(s),
migration from one storage media to another and disposition
• Document management can support different types of workflows
− Manual workflows that indicate where the user sends the document
− Rules-based workflow, where rules are created that dictate the flow of the
document within an organisation
− Dynamic rules that allow for different workflows based on content

March 8, 2010 277

Backup and Recover Documents / Records

• The document / record management system needs to be
included as part of the overall corporate backup and
recovery activities for all data and information
• Document / records manager be involved in risk mitigation
and management, and business continuity especially
regarding security for vital records
• A vital records program provides the organisation with
access to the records necessary to conduct its business
during a disaster and to resume normal business afterward

March 8, 2010 278

Retention and Disposition of Documents / Records

• Defines the period of time during which documents /
records for operational, legal, financial or historical value
must be maintained
• Specifies the processes for compliance, and the methods
and schedules for the disposition of documents / records
• Must deal with privacy and data protection issues
• Legal and regulatory requirements must be considered
when setting up document record retention schedules

March 8, 2010 279

Audit Document / Records Management

• Document / records management requires auditing on a periodic basis to ensure
that the right information is getting to the right people at the right time for
decision making or performing operational activities
− Inventory - Each location in the inventory is uniquely identified
− Storage - Storage areas for physical documents / records have adequate space to
accommodate growth
− Reliability and Accuracy - Spot checks are executed to confirm that the documents /
records are an adequate reflection of what has been created or received
− Classification and Indexing Schemes - Metadata and document file plans are well
described
− Access and Retrieval - End users find and retrieve critical information easily
− Retention Processes - Retention schedule is structured in a logical way
− Disposition Methods - Documents / records are disposed of as recommended
− Security and Confidentiality - Breaches of document / record confidentiality and loss of
documents / records are recorded as security incidents and managed appropriately
− Organisational Understanding of Documents / Records Management - Appropriate
training is provided to stakeholders and staff as to the roles and responsibilities related
to document / records management

March 8, 2010 280

Content Management

• Organisation, categorisation, and structure of data /
resources so that they can be stored, published, and
reused in multiple ways
• Includes data / information, that exists in many forms and
in multiple stages of completion within its lifecycle
• Content management systems manage the content of a
website or intranet through the creation, editing, storing,
organising, and publishing of content

March 8, 2010 281

Define and Maintain Enterprise Taxonomies
(Information Content Architecture)
• Process of creating a structure for a body of information or
content
• Contains a controlled vocabulary that can help with
navigation and search systems
• Content Architecture identifies the links and relationships
between documents and content, specifies document
requirements and attributes and defines the structure of
content in a document or content management system

March 8, 2010 282

Document / Index Information Content Metadata

• Development of metadata for unstructured data content
• Maintenance of metadata for unstructured data becomes
the maintenance of a cross-reference of various local
schemes to the official set of organisation metadata

March 8, 2010 283

Provide Content Access and Retrieval

• Once the content has been described by metadata / key
word tagging and classified within the appropriate
Information Content Architecture, it is available for
retrieval and use
• Finding unstructured data can be eased through portal
technology

March 8, 2010 284

Govern for Quality Content

• Managing unstructured data requires effective
partnerships between data stewards, data professionals,
and records managers
• The focus of data governance can include document and
record retention policies, electronic signature policies,
reporting formats, and report distribution policies
• High quality, accurate, and up-to-date information will aid
in critical business decisions
• Timeliness of the decision-making process with high
quality information may increase competitive advantage
and business effectiveness
March 8, 2010 285

Metadata Management

March 8, 2010 286

Metadata Management

• Metadata is data about data
• Metadata Management is the set of processes that ensure proper creation,
storage, integration, and control to support associated usage of metadata
• Leveraging metadata in an organisation can provide benefits
− Increase the value of strategic information by providing context for the data, thus aiding
analysts in making more effective decisions
− Reduce training costs and lower the impact of staff turnover through thorough
documentation of data context, history, and origin
− Reduce data-oriented research time by assisting business analysts in finding the
information they need, in a timely manner
− Improve communication by bridging the gap between business users and IT
professionals, leveraging work done by other teams, and increasing confidence in IT
system data
− Increase speed of system development time-to-market by reducing system
development life-cycle time
− Reduce risk of project failure through better impact analysis at various levels during
change management
− Identify and reduce redundant data and processes, thereby reducing rework and use of
redundant, out-of-date, or incorrect data
March 8, 2010 287

Metadata Management – Definition and Goals

• Definition
− Planning, implementation, and control activities to enable easy
access to high quality, integrated metadata
• Goals
− Provide organisational understanding of terms, and usage
− Integrate metadata from diverse source
− Provide easy, integrated access to metadata
− Ensure metadata quality and security

March 8, 2010 288

Metadata

• Metadata is information about the physical data, technical and business processes, data rules and
constraints, and logical and physical structures of the data, as used by an organisation
• Descriptive tags describe data, concepts and the connections between the data and concepts
− Business Analytics: Data definitions, reports, users, usage, performance
− Business Architecture: Roles and organisations, goals and objectives
− Business Definitions: The business terms and explanations for a particular concept, fact, or other item
found in an organisation
− Business Rules: Standard calculations and derivation methods
− Data Governance: Policies, standards, procedures, programs, roles, organisations, stewardship
assignments
− Data Integration: Sources, targets, transformations, lineage, ETL workflows, EAI, EII, migration /
conversion
− Data Quality: Defects, metrics, ratings
− Document Content Management: Unstructured data, documents, taxonomies, name sets, legal
discovery, search engine indexes
− Information Technology Infrastructure: Platforms, networks, configurations, licenses
− Logical Data Models: Entities, attributes, relationships and rules, business names and definitions
− Physical Data Models: Files, tables, columns, views, business definitions, indexes, usage, performance,
change management
− Process Models: Functions, activities, roles, inputs / outputs, workflow, business rules, timing, stores
− Systems Portfolio and IT Governance: Databases, applications, projects and programs, integration
roadmap, change management
− Service-Oriented Architecture (SOA) Information: Components, services, messages, master data
− System Design and Development: Requirements, designs and test plans, impact
− Systems Management: Data security, licenses, configuration, reliability, service levels

March 8, 2010 289

Metadata Management - Overview
•Metadata Requirements
•Metadata Issues •Metadata Repositories
•Data Architecture •Quality Metadata
•Business Metadata •Metadata Models and Architecture
•Technical Metadata •Metadata Management
•Process Metadata •Operational Analysis
•Operational Metadata •Metadata Analysis
•Data Stewardship Metadata •Data Lineage
•Change Impact Analysis
Metadata •Metadata Control Procedures
Suppliers
Management
•Data Stewards Consumers
•Data Architects •Data Stewards
•Data Modelers •Data Professionals
•Database Administrators •Other IT Professionals
•Other Data Professionals •Knowledge Workers
•Data Brokers Tools •Managers and Executives
•Government and Industry Regulators •Customers and Collaborators
•Metadata Repositories •Business Users
•Data Modeling Tools
•Database Management Systems
Participants •Data Integration Tools Metrics
•Business Intelligence Tools
•Metadata Specialist •System Management Tools •Meta Data Quality
•Data Integration Architects •Object Modeling Tools •Master Data Service Data Compliance
•Data Stewards •Process Modeling Tools •Metadata Repository Contribution
•Data Architects and Modelers •Report Generating Tools Metadata Documentation Quality
•Database Administrators •Data Quality Tools Steward Representation / Coverage
•Other DM Professionals •Data Development and •Metadata Usage / Reference
•Other IT Professionals Administration Tools •Metadata Management Maturity
•DM Executive •Reference and Master Data •Metadata Repository Availability
•Business Users Management Tools
March 8, 2010 290

Metadata Management Function, Activities and Sub-
Activities
Metadata
Management

Develop and Implement a
Understand Define the Create and Manage Distribute and Query, Report
Maintain Managed Integrate
Metadata Metadata Maintain Metadata Deliver and Analyse
Metadata Metadata Metadata
Requirements Architecture Metadata Repositories Metadata Metadata
Standards Environment

Industry /
Centralised
Business User Consensus Metadata
Metadata
Requirements Metadata Repositories
Architecture
Standards

Directories,
Distributed International Glossaries and
Technical User
Metadata Metadata Other
Requirements
Architecture Standards Metadata
Stores

Hybrid Standard
Metadata Metadata
Architecture Metrics

March 8, 2010 291

Metadata Management - Principles
• Establish and maintain a metadata strategy and appropriate policies, especially clear goals and objectives for
metadata management and usage
• Secure sustained commitment, funding, and vocal support from senior management concerning metadata
management for the enterprise
• Take an enterprise perspective to ensure future extensibility, but implement through iterative and
incremental delivery
• Develop a metadata strategy before evaluating, purchasing, and installing metadata management products
• Create or adopt metadata standards to ensure interoperability of metadata across the enterprise
• Ensure effective metadata acquisition for both internal and external metadata
• Maximise user access, since a solution that is not accessed or is under-accessed will not show business value
• Understand and communicate the necessity of metadata and the purpose of each type of metadata;
socialisation of the value of metadata will encourage business usage
• Measure content and usage
• Leverage XML, messaging, and Web services
• Establish and maintain enterprise-wide business involvement in data stewardship, assigning accountability for
metadata
• Define and monitor procedures and processes to ensure correct policy implementation
• Include a focus on roles, staffing, standards, procedures, training, and metrics
• Provide dedicated metadata experts to the project and beyond
• Certify metadata quality

March 8, 2010 292

Understand Metadata Requirements

• Metadata management strategy must reflect an
understanding of enterprise needs for metadata
• Gather requirements to confirm the need for a metadata
management environment, to set scope and priorities,
educate and communicate, to guide tool evaluation and
implementation, guide metadata modeling, guide internal
metadata standards, guide provided services that rely on
metadata, and to estimate and justify staffing needs
• Gather requirements from business and technical users
• Summarise the requirements from an analysis of roles,
responsibilities, challenges, and the information needs of
selected individuals in the organisation

March 8, 2010 293

Business User Requirements

• Business users require improved understanding of the
information from operational and analytical systems
• Business users require a high level of confidence in the
information obtained from corporate data warehouses,
analytical applications, and operational systems
• Need appropriate access to information delivery methods,
such as reports, queries, ad-hoc, OLAP, dashboards with a
high degree of quality documentation and context
• Business users must understand the intent and purpose of
metadata management

March 8, 2010 294

Technical User Requirements

• Technical requirement topics include
− Daily feed throughput: size and processing time
− Existing metadata
− Sources - known and unknown
− Targets
− Transformations
− Architecture flow logical and physical
− Non-standard metadata requirements
• Technical users must understand the business context of
the data at a sufficient level to provide the necessary
support, including implementing the calculations or
derived data rules
March 8, 2010 295

Define the Metadata Architecture

• Metadata management solutions consist of
− Metadata creation / sourcing
− metadata integration
− Mmetadata repositories
− Metadata delivery
− Metadata usage
− Metadata control / management

March 8, 2010 296

Centralised Metadata Architecture

• Single metadata repository that contains copies of the live metadata
from the various sources
• Advantages
− High availability, since it is independent of the source systems
− Quick metadata retrieval, since the repository and the query reside together
− Resolved database structures that are not affected by the proprietary nature of
third party or commercial systems
− Extracted metadata may be transformed or enhanced with additional
metadata that may not reside in the source system, improving quality
• Disadvantages
− Complex processes are necessary to ensure that changes in source metadata
quickly replicate into the repository
− Maintenance of a centralised repository can be substantial
− Extraction could require custom additional modules or middleware
− Validation and maintenance of customised code can increase the demands on
both internal IT staff and the software vendors
March 8, 2010 297

Distributed Metadata Architecture

• Metadata retrieval engine responds to user requests by retrieving
data from source systems in real time with no persistent repository
• Advantages
− Metadata is always as current and valid as possible
− Queries are distributed, possibly improving response / process time
− Metadata requests from proprietary systems are limited to query processing
rather than requiring a detailed understanding of proprietary data structures,
therefore minimising the implementation and maintenance effort required
− Development of automated metadata query processing is likely simpler,
requiring minimal manual intervention
− Batch processing is reduced, with no metadata replication or synchronisation
processes
• Disadvantages
− No enhancement or standardisation of metadata is possible between systems
− Query capabilities are directly affected by the availability of the participating
source systems
− No ability to support user-defined or manually inserted metadata entries since
there is no repository in which to place these additions

March 8, 2010 298

Hybrid Metadata Architecture

• Hybrid architecture where metadata still moves directly from the source systems
into a repository but the repository design only accounts for the user-added
metadata, the critical standardised items and the additions from manual sources
• Advantages
− Near-real-time retrieval of metadata from its source and enhanced metadata to meet
user needs most effectively, when needed
− Lowers the effort for manual IT intervention and custom-coded access functionality to
proprietary systems.
• Disadvantages
− Source systems must be available because the distributed nature of the back-end
systems handles processing of queries
− Additional overhead is required to link those initial results with metadata augmentation
in the central repository before presenting the result set to the end user
− Design forces the metadata repository to contain the latest version of the metadata
source and forces it to manage changes to the source, as well
− Sets of program / process interfaces to tie the repository back to the metadata
source(s) must be built and maintained

March 8, 2010 299

Develop and Maintain Metadata Standards

• Check industry or consensus standards and international
standards
• International standards provide the framework from which
the industry standards are developed and executed

March 8, 2010 300

Industry / Consensus Metadata Standards

• Understanding the various standards for the implementation and management of meta-
data in industry is essential to the appropriate selection and use of a metadata solution for
an enterprise
− OMG (Object Management Group) specifications
• Common Warehouse Metadata (CWM)
• Information Management Metamodel (IMM)
• MDC Open Information Model (OIM)
• Extensible Markup Language (XML)
• Unified Modeling Language (UML)
• Extensible Markup Interface (XMI)
• Ontology Definition Metamodel (ODM)
− World Wide Web Consortium (W3C) RDF (Relational Definition Framework) for describing and
interchanging metadata using XML
− Dublin Core Metadata Initiative (DCMI) interoperable online metadata standard using RDF
− Distributed Management Task Force (DTMF) Web-Based Enterprise Management (WBEM)
Common Information Model (CIM) standards-based management tools facilitating the exchange of
data across otherwise disparate technologies and platforms
− Metadata standards for unstructured data
• ISO 5964 - Guidelines for the establishment and development of multilingual thesauri
• ISO 2788 - Guidelines for the establishment and development of monolingual thesauri
• ANSI/NISO Z39.1 - American Standard Reference Data and Arrangement of Periodicals
• ISO 704 - Terminology work Principles and methods

March 8, 2010 301

International Metadata Standards

• ISO / IEC 11179 is an international metadata standard for
standardising and registering of data elements to make
data understandable and shareable

March 8, 2010 302

Standard Metadata Metrics

• Controlling the effectiveness of the metadata deployed
environment requires measurements to assess user
uptake, organisational commitment, and content coverage
and quality
− Metadata Repository Completeness
− Metadata Documentation Quality
− Master Data Service Data Compliance
− Steward Representation / Coverage
− Metadata Usage / Reference
− Metadata Management Maturity
− Metadata Repository Availability

March 8, 2010 303

Implement a Managed Metadata Environment

• Implement a managed metadata environment in
incremental steps in order to minimise risks to the
organisation and to facilitate acceptance
• First implementation is a pilot to prove concepts and learn
about managing the metadata environment

March 8, 2010 304

Create and Maintain Metadata

• Metadata creation and update facility provides for the
periodic scanning and updating of the repository in
addition to the manual insertion and manipulation of
metadata by authorised users and program
• Audit process validates activities and reports exceptions
• Metadata is the guide to the data in the organisation so its
quality is critical

March 8, 2010 305

Integrate Metadata

• Integration processes gather and consolidate metadata from across
the enterprise including metadata from data acquired outside the
enterprise
• Challenges will arise in integration that will require resolution
through the governance process
• Use a non-persistent metadata staging area to store temporary and
backup files that supports rollback and recovery processes and
provides an interim audit trail to assist repository managers when
investigating metadata source or quality issues
• ETL tools used for data warehousing and Business Intelligence
applications are often used effectively in metadata integration
processes

March 8, 2010 306

Manage Metadata Repositories

• Implement a number of control activities in order to
manage the metadata environment
• Control of repositories is control of metadata movement
and repository updates performed by the metadata
specialist

March 8, 2010 307

Metadata Repositories

• Metadata repository refers to the physical tables in which
the metadata are stored
• Generic design and not merely reflecting the source
system database designs
• Metadata should be as integrated as possible this will be
one of the most direct valued-added elements of the
repository

March 8, 2010 308

Directories, Glossaries and Other Metadata Stores

• A Directory is a type of metadata store that limits the
metadata to the location or source of data in the
enterprise
• A Glossary typically provides guidance for use of terms
• Other Metadata stores include specialised lists such as
source lists or interfaces, code sets, lexicons, spatial and
temporal schema, spatial reference, and distribution of
digital geographic data sets, repositories of repositories
and business rules

March 8, 2010 309

Distribute and Deliver Metadata

• Metadata delivery layer is responsible for the delivery of
the metadata from the repository to the end users and to
any applications or tools that require metadata feeds to
them

March 8, 2010 310

Query, Report and Analyse Metadata

• Metadata guides management and use of data assets
• A metadata repository must have a front-end application
that supports the search-and- retrieval functionality
required for all this guidance and management of data
assets

March 8, 2010 311


March 8, 2010 312


• Critical support process in organisational change management
• Data quality is synonymous with information quality since poor data
quality results in inaccurate information and poor business
performance
• Data cleansing may result in short-term and costly improvements
that do not address the root causes of data defects
• More rigorous data quality program is necessary to provide an
economic solution to improved data quality and integrity
• Institutionalising processes for data quality oversight, management,
and improvement hinges on identifying the business needs for
quality data and determining the best ways to measure, monitor,
control, and report on the quality of data
• Continuous process for defining the parameters for specifying
acceptable levels of data quality to meet business needs, and for
ensuring that data quality meets these levels

March 8, 2010 313

Data Quality Management – Definition and Goals

• Definition
− Planning, implementation, and control activities that apply quality
management techniques to measure, assess, improve, and ensure
the fitness of data for use
• Goals
− To measurably improve the quality of data in relation to defined
business expectations
− To define requirements and specifications for integrating data
quality control into the system development lifecycle
− To provide defined processes for measuring, monitoring, and
reporting conformance to acceptable levels of data quality

March 8, 2010 314


• Data quality expectations provide the inputs necessary to
define the data quality framework
• Framework includes defining the requirements, inspection
policies, measures, and monitors that reflect changes in
data quality and performance
• Requirements reflect three aspects of business data
expectations
− Way to record the expectation in business rules
− Way to measure the quality of data within that dimension
− Acceptability threshold

March 8, 2010 315

Data Quality Management Approach

• Planning for the assessment of the current state and
identification of key metrics for measuring data quality
• Deploying processes for measuring and improving the
quality of data
• Monitoring and measuring the levels in relation to the
defined business expectations
• Acting to resolve any identified issues to improve data
quality and better meet business expectations

March 8, 2010 316

Data Quality Management - Overview

•Business Requirements
•Data Requirements •Improved Quality Data
•Data Quality Expectations •Data Management
•Data Policies and Standards •Operational Analysis
•Business metadata •Data Profiles
•Technical metadata •Data Quality Certification Reports
•Data Sources and Data Stores •Data Quality Service Level
Agreements

Suppliers

•External Sources Consumers
•Regulatory Bodies Data Quality
•Business Subject Matter Experts
•Information Consumers Management •Data Stewards
•Data Professionals
•Data Producers •Other IT Professionals
•Data Architects •Knowledge Workers
•Data Modelers •Managers and Executives Customers


•Data Quality Analysts
•Data Analysts •Data Profiling Tools •Data Value Statistics
•Database Administrators •Statistical Analysis Tools •Errors / Requirement Violations
•Data Stewards •Data Cleansing Tools •Conformance to Expectations
•Other Data Professionals •Data Integration Tools •Conformance to Service Levels
•DRM Director •Issue and Event Management Tools
•Data Stewardship Council

March 8, 2010 317

Data Quality Management Function, Activities and
Sub-Activities
Data Quality
Management

Monitor
Design and
Develop and Profile, Define Data Test and Set and Continuously Clean and Operational
Define Data Define Data Implement
Promote Data Analyse and Quality Validate Data Evaluate Data Measure and Manage Data Correct Data DQM
Quality Quality Operational
Quality Assess Data Business Quality Quality Monitor Data Quality Issues Quality Procedures
Requirements Metrics DQM
Awareness Quality Rules Requirements Service Levels Quality Defects and
Procedures
Performance

March 8, 2010 318

Data Quality Management - Principles

• Manage data as a core organisational asset
• All data elements will have a standardised data definition, data type, and
acceptable value domain
• Leverage Data Governance for the control and performance of DQM
• Use industry and international data standards whenever possible
• Downstream data consumers specify data quality expectations
• Define business rules to assert conformance to data quality expectations
• Validate data instances and data sets against defined business rules
• Business process owners will agree to and abide by data quality SLAs
• Apply data corrections at the original source, if possible
• If it is not possible to correct data at the source, forward data corrections to the
owner of the original source whenever possible
• Report measured levels of data quality to appropriate data stewards, business
process owners, and SLA managers
• Identify a gold record for all data elements

March 8, 2010 319

Develop and Promote Data Quality Awareness

• Promoting data quality awareness means more than ensuring that
the right people in the organisation are aware of the existence of
data quality issues
• Establish a data governance framework for data quality
− Set priorities for data quality
− Develop and maintain standards for data quality
− Report relevant measurements of enterprise-wide data quality
− Provide guidance that facilitates staff involvement
− Establish communications mechanisms for knowledge sharing
− Develop and apply certification and compliance policies
− Monitor and report on performance
− Identify opportunities for improvements and build consensus for approval
− Resolve variations and conflicts

March 8, 2010 320

Define Data Quality Requirements

• Applications are dependent on the use of data that meets specific needs associated with
the successful completion of a business process
• Data quality requirements are often hidden within defined business policies
− Identify key data components associated with business policies
− Determine how identified data assertions affect the business
− Evaluate how data errors are categorised within a set of data quality dimensions
− Specify the business rules that measure the occurrence of data errors
− Provide a means for implementing measurement processes that assess conformance to those
business rules
• Dimensions of data quality
− Accuracy
− Completeness
− Consistency
− Currency
− Precision
− Privacy
− Reasonableness
− Referential Integrity
− Timeliness
− Uniqueness
− Validity

March 8, 2010 321

Profile, Analyse and Assess Data Quality

• Perform an assessment of the data using two different approaches,
bottom-up and top-down
• Bottom-up assessment of existing data quality issues involves
inspection and evaluation of the data sets themselves
• Top-down approach involves understanding how their processes
consume data, and which data elements are critical to the success of
the business application
− Identify a data set for review
− Catalog the business uses of that data set
− Subject the data set to empirical analysis using data profiling tools and
techniques
− List all potential anomalies, review and evaluate
− Prioritise criticality of important anomalies in preparation for defining data
quality metrics

March 8, 2010 322

Define Data Quality Metrics

• Poor data quality affects the achievement of business objectives
• Seek and use indicators of data quality performance to report the
relationship between flawed data and missed business objectives
• Measuring quality similarly to monitoring any type of business
performance activity
• Data quality metrics should be reasonable and effective
− Measurability
− Business Relevance
− Acceptability
− Accountability / Stewardship
− Controllability
− Trackability

March 8, 2010 323

Define Data Quality Business Rules

• Measurement of conformance to specific business rules
requires definition
• Monitoring conformance to these rules requires
• Segregating data values, records, and collections of
records that do not meet business needs from the valid
ones
• Generating a notification event alerting a data steward of a
potential data quality issue
• Establishing an automated or event driven process for
aligning or possibly correcting flawed data within business
expectations
March 8, 2010 324

Test and Validate Data Quality Requirements

• Data profiling tools analyse data to find potential anomalies
• Data profiling tools allow data analysts to define data rules for
validation, assessing frequency distributions and corresponding
measurements and then applying the defined rules against the data
sets
• Characterising data quality levels based on data rule conformance
provides an objective measure of data quality
• By using defined data rules to validate data, an organisation can
distinguish those records that conform to defined data quality
expectations and those that do not
• In turn, these data rules are used to baseline the current level of
data quality as compared to ongoing audits

March 8, 2010 325

Set and Evaluate Data Quality Service Levels

• Data quality SLAs specify the organisation’s expectations for response and
remediation
• Having data quality inspection and monitoring in place increases the likelihood of
detection and remediation of a data quality issue before a significant business
impact can occur
• Operational data quality control defined in a data quality SLA includes
− The data elements covered by the agreement
− The business impacts associated with data flaws
− The data quality dimensions associated with each data element
− The expectations for quality for each data element for each of the identified dimensions
in each application or system in the value chain
− The methods for measuring against those expectations
− The acceptability threshold for each measurement
− The individual(s) to be notified in case the acceptability threshold is not met. The
timelines and deadlines for expected resolution or remediation of the issue
− The escalation strategy and possible rewards and penalties when the resolution times
are met.

March 8, 2010 326

Continuously Measure and Monitor Data Quality

• Provide continuous monitoring by incorporating control
and measurement processes into the information
processing flow
• Incorporating the results of the control and measurement
processes into both the operational procedures and
reporting frameworks enable continuous monitoring of the
levels of data quality

March 8, 2010 327

Manage Data Quality Issues

• Supporting the enforcement of the data quality SLA requires a
mechanism for reporting and tracking data quality incidents and
activities for researching and resolving those incidents
• Data quality incident reporting system provides this capability
• Tracking of data quality incidents provides performance reporting
data, including mean-time-to-resolve issues, frequency of
occurrence of issues, types of issues, sources of issues and common
approaches for correcting or eliminating problems
• Data quality incident tracking also requires a focus on training staff
to recognise when data issues appear and how they are to be
classified, logged and tracked according to the data quality SLA
• Implementing a data quality issues tracking system provides a
number of benefits
− Information and knowledge sharing can improve performance and reduce
duplication of effort
− Analysis of all the issues will help data quality team members determine any
repetitive patterns, their frequency, and potentially the source of the issue
March 8, 2010 328

Clean and Correct Data Quality Defects

• Perform data correction in three general ways
− Automated correction - Submit the data to data quality and data
cleansing techniques using a collection of data transformations
and rule-based standardisations, normalisations, and corrections
− Manual directed correction - Use automated tools to cleanse and
correct data but require manual review before committing the
corrections to persistent storage
− Manual correction: Data stewards inspect invalid records and
determine the correct values, make the corrections, and commit
the updated records

March 8, 2010 329

Design and Implement Operational DQM Procedures

• Using defined rules for validation of data quality provides a
means of integrating data inspection into a set of
operational procedures associated with active DQM
• Design and implement detailed procedures for
operationalising activities
− Inspection and monitoring
− Diagnosis and evaluation of remediation alternatives
− Resolving the issue
− Reporting

March 8, 2010 330

Monitor Operational DQM Procedures and
Performance
• Accountability is critical to the governance protocols
overseeing data quality control
• Issues must be assigned to some number of individuals,
groups, departments, or organisations
• Tracking process should specify and document the
ultimate issue accountability to prevent issues from
dropping through the cracks
• Metrics can provide valuable insights into the effectiveness
of the current workflow, as well as systems and resource
utilisation and are important management data points that
can drive continuous operational improvement for data
quality control

March 8, 2010 331

Conducting a Data Management Project

March 8, 2010 332

Conducting a Data Management Project

• Data management project depends on:
− Scope of the Project – data management functions to be
encompassed
− Type of Project – from architecture to analysis to implementation
− Scope Within the Organisation – one or more business units or
the entire organisation

March 8, 2010 333

Data Management Function and Project Type

Scope of Project
Data Data Data Data Data Security Reference and Data Document Metadata Data Quality
Type of Governance Architecture Development Operations Management Master Data Warehousing and Content Management Management
Project Management Management Management and Business
Intelligence
Management

Management
Architecture

Analysis and
Design

Implementation

Operational
Improvement

Management
and
Administration

March 8, 2010 334

Mapping the Path Through the Selected Data
Management Project
• Use the framework
to define the
breakdown of the
selected project

March 8, 2010 335

Project Elements – Data Management Functions,
Type of Project, Organisational Scope

Organisational
Scope of Project

Data
Management
Functions
Within Scope
of Project

Type of
Project

• Select the project building blocks based on the project scope
March 8, 2010 336

Creating a Data Management Team

March 8, 2010 337

Creating a Data Management Team

• Having implemented a data management framework,
must be monitored, managed and constantly improved
• Need to consolidate and coordinate data management and
governance efforts to meet the challenges of
− Demand for performance management data
− Complexity in systems and processes
− Greater regulatory and compliance requirements
• Build a Data Management Center of Excellence (DMCOE)

March 8, 2010 338

Data Management Center of Excellence

• Separate business units with the organisation generally implement their own
solutions
• Each business unit will have different IT systems, data warehouses/data marts and
business intelligence tools
• Organisation-wide coordination of data resources requires a centralised dedicated
structure like the DMCOE providing data services
• Leads a organisation to business benefits through continuous improvement of
data management
• DMCOE functions need to focus on leveraging organisational knowledge and skills
to maximise the value of data to the organisation
• Maximise technology investment while decreasing costs and increasing efficiency,
centralise best practices and standards and empower knowledge workers with
information and provide thought leadership to the entire company
• DMCOE does not exist in isolation to other operations and service management
functions

March 8, 2010 339

DMCOE Functions

• Maximise the value of the data technology investment to
the organisation by taking a portfolio approach to increase
skills and leverage and to optimise the infrastructure
• Focus on project delivery and information asset creation
with an emphasis on reusability and knowledge
management along with solution delivery
• Ensure the integrity of the organisation’s business
processes and information systems
• Ensure the quality compliance effort related to the
configuration, development, and documentation of
enhancements
• Develop information learning and effective practices

March 8, 2010 340

Data Charter

• Create charter that lists the fundamental principles of data
management the DMCOE will adhere to:
− Data Strategy - Create a data blueprint, based upon business
functions to facilitate data design
− Data Sharing - Promote the sharing of data across the
organisation and reduce data redundancy
− Data Integrity - Ensure the integrity of data from design and
availability perspectives
− Technical Expertise - Provide the expertise for the development
and support of data systems
− High Availability and Optimal Performance - Ensure consistent
high availability of data systems through proper design and use
and optimise performance of the data systems
March 8, 2010 341

DMCOE Skills

• DMCOE needs skills across three dimensions
− Specific data management functions
− Business management and administration
− Technology and service management

March 8, 2010 342

DMCOE Skills
Data Management
Design and
Development
Data Management
Data Management Business Skills
Process Management

Personnel Management

Data Management
Portfolio Management
Data
Reference
Data Data Warehousing Document
Data Management Data Data Data Security and Master Metadata Data Quality
Architecture Operations and Business and Content
Strategy Governance Development Management Data Management Management
Management Management Intelligence Management
Management
Management

Data Management
Environment and Specific Functions
Infrastructure
Management

Service Management • Idealised set of DMCOE skills that need to
Data
and Support
be customised to suit specific organisation
Management Application Deployment needs
Technology and and Data Migration
Service
Functions • Just one view of a DMCOE
Technical Architecture

March 8, 2010 343

DMCOE Business Management and Administration
Skills
DMCOE Business
Management and
Administration

Data Management Data Management Data Management
Data Management Personnel
Portfolio Process Design and
Strategy Management
Management Management Development

Management of Education and
Creation and Requirements
Strategic Planning Portfolio of Data Skills
Enforcement of Definition and
Processes Management Identification and
Process Standards Management
Initiatives Development

Co-ordination of
Resource
Data Management Management of Analysis and
Management and
Systems and Data Processes Design
Allocation
Initiatives

Creation and
Enforcement of Vendor Performance Development
Data Principles Management Management Standards
and Standards

Solution
Data Usage
Data Quality Development and
Strategy
Deployment

March 8, 2010 344

DMCOE Technology and Service Management Skills
DMCOE Technology
and Service
Management

Environment and Application
Service Management
Infrastructure Deployment and Data Technical Architecture
and Support
Management Migration

Change Management Application Infrastructure
Service Desk
and Control Deployment Architecture

Test Management –
Version Management Service Level Application and Tools
System, Integration,
and Control Management Architecture
UAT, UAT Support

Performance
Data Migration Data and Content
Monitoring and Security Management
Management Architecture
Management

Integration
Reporting System Maintenance
Architecture

March 8, 2010 345

Benefits of DMCOE

• Consistent infrastructure that reduces time to analyse and
design and implement new IT solutions
• Reduced data management costs through a consistent
data architecture and data integration infrastructure -
reduced complexity, redundancy, tool proliferation
• Centralised repository of the organisation's data
knowledge
• Organisation-wide standard methodology and processes to
develop and maintain data infrastructure and procedures
• Increased data availability
• Increased data quality

March 8, 2010 346

Assessing Your Data Management Maturity

March 8, 2010 347

Assessing Your Data Management Maturity

• A Data Management Maturity Model is a measure of and then a
process for determining the level of maturity that exists within an
organisation’s data management function
• Provides a systematic framework for improving data management
capability and identifying and prioritising opportunities, reducing
cost and optimising the business value of data management
investments
• Measure of data management maturity so that:
− It can be tracked over time to measure improvements
− It can be use to define project for data management maturity improvements
within costs, time, and return on investment constraints
• Enables organisations to improve their data management function
so that they can increase productivity, increase quality, decrease
cost and decrease risk

March 8, 2010 348

Data Management Maturity Model

• Assesses data management maturity on a level of 1 to 5
across a number of data management capabilities
Level Title Description
1 Initial Data management is ad hoc and localised. Everybody has their own
approach that is unique and not standardised except for local
initiatives.
2 Repeatable and Data management has become independent of the person or
Reactive business unit administering and is standardised.
3 Defined and Data management is fully documented, determined by subject
Standardised matter experts and validated.
4 Managed and Data management results and outcomes are stored and pro-
Predictable actively cross-related within and between business units. The data
management function actively exploit benefits of standardisation.
5 Optimising and As time, resources, technology, requirements and business
Innovating landscape changes the data management function is able to be
easily and quickly adjusted to fit new needs and environments

March 8, 2010 349

Maturity Level 1 - Initial

• Data management processes are mostly disorganised and generally performed on
an ad hoc or even even chaotic basis
• Data is considered as general purpose and is not viewed by either business or
executive management to be a problem or a priority
• Data is accessible but not always available and is not secure or auditable
• No data management group and no one owns the responsibility for ensuring the
quality, accuracy or integrity of the data
• Data management (to the degree that it is done at all) is reliant on the efforts and
competence of individuals
• Data proliferates without control and the quality is inconsistent across the various
business and applications silos
• Data exists in unconnected databases and spreadsheets using multiple formats
and inconsistent definitions
• Little data profiling or analysis and data is not considered or understood as a
component of linked processes
• No formal data quality processes and the processes that do exist are not
repeatable because they are neither well defined nor well documented

March 8, 2010 350

Maturity Level 2 - Repeatable and Reactive

• Fundamental data management practices are established, defined, documented
and can be repeated
• Data policies for creation and change management exist, but still rely on
individuals and are not institutionalised throughout the organisation
• Data as valuable asset is a concept understood by some, but senior management
support is lacking and there is little organisational buy-in to the importance of an
enterprise-wide approach to managing data
• data is stored locally and data quality is reactive to circumstances
• Requirements are known and managed at the business unit and application level
• Procurement is ad hoc based on individual needs and data duplication is mostly
invisible
• Data quality varies among business units and data failures occur on a cross-
functional basis.
• Most data is integrated point-to-point and not across business units

March 8, 2010 351

Maturity Level 3 - Defined and Standardised

• Business analysts begin to control the data management process with IT playing a
supporting role
• Data is recognised as a business enabler and moves from an undervalued commodity to an
enterprise asset but there are still limited controls in place
• Executive management appreciates and understands the role of data governance and
commits resources to its management
• Data administrative function exists as a complement to the database administration
function and data is present for both business and IT related development discussions
• Some core data has defined policy that it is documented as part of the applications
development lifecycle and the policies are enforced to a limited extent and testing is
performed to ensure that data quality requirements are being achieved
• Data quality is not fully defined and there are multiple views of what quality
• Metadata repository exists and a data group maintains corporate data definitions and
business rules
• A centralised platform for managing data is available at the group level and feeds analytical
data marts
• Data is available to business users and can be audited

March 8, 2010 352

Maturity Level 4 - Managed and Predictable

• Data is treated as a critical corporate asset and viewed as equivalent to other enterprise wide assets
• Unified data governance strategy exists throughout the enterprise with executive level and CEO
support
• Data management objectives are reviewed by senior management
• Business process interaction is completely documented and planning is centralised
• Data quality control, integration and synchronisation are integral parts of all business processes
• Content is monitored and corrected in real time to manage the reliability of the data manufacturing
process and is based on the needs of customers, end users and the organisation as a whole
• Data quality is understood in statistical terms and managed throughout the transactions lifecycle
• Root cause analysis is well established and proactive steps are taken to prevent and not just correct
data inconsistencies
• A centralised metadata repository exists and all changes are synchronised
• Data consistency is expected and achieved
• Data platform is managed at the enterprise level and feeds all reference data repositories
• Advanced platform tools are used to manage the metadata repository and all data transformation
processes
• Data quality and integration tools are standardised across the enterprise.

March 8, 2010 353

Maturity Level 5 - Optimising and Innovating

• The organisation is in continuous improvement mode
• Process enhancements are managed through monitoring feedback and a
quantitative understanding of the causes of data inconsistencies
• Enterprise wide business intelligence is possible
• Organisation is agile enough to respond to changing circumstances and evolving
business objectives
• Data is considered as the key resource for process improvement
• Data requirements for all projects are defined and agreed prior to initiation
• Development stresses the re-use of data and is synchronised with the
procurement process
• Process of data management is continuously being improved
• Data quality (both monitoring and correction) is fully automated and adaptive
• Uncontrolled data duplication is eliminated and controlled duplication must be
justified
• Governance is data driven and the organisation adopts a “test and learn”
philosophy

March 8, 2010 354

Data Management Maturity Evaluation - Key
Capabilities and Maturity Levels
Level 1 Level 2 Level 3 Level 4 Level 5

Data Governance

< Description of
Data Architecture Management capability associated
with maturity level >

Data Development



Reference and Master Data
Management

Data Warehousing and Business
Intelligence Management

Document and Content
Management

Metadata Management


March 8, 2010 355

More Information

Alan McSweeney
alan@alanmcsweeney.com

March 8, 2010 356

Data, Information And Knowledge Management Framework And The Data Management Book Of Knowledge (Dmbok)

Recommended

More Related Content

What's hot (20)

Similar to Data, Information And Knowledge Management Framework And The Data Management Book Of Knowledge (Dmbok) (20)

More from Alan McSweeney (20)

Recently uploaded (20)

Data, Information And Knowledge Management Framework And The Data Management Book Of Knowledge (Dmbok)