April 24th, 2006

ORACLE 10g DATA PUMP

Inderpal S. Johal
Principal Consultant
Data Softech Inc.
 AGENDA

Oracle Data Loading/Unloading options

Why we use Data pump or exp/imp

Introduction to Data Pump

Data Pump Architecture & Implementation

Benefits of Data Pump Utility

Data Pump vs. Exp/Imp

DBMS_DATAPUMP package

Questions & Answers
Oracle Data Loading/Unloading Options

INSERT Sql Command

MERGE Sql Command

Exp/Imp utility

PL/SQL API UTL_FILE

SQL*Loader

External Tables

PL/SQL Bulk Collect … ForAll

Data Pump
 Why Export/Import or DataPump

For Moving Data between Database
Development QA Production

Taking Logical Backup

Archiving Data for future Reference

Migrating Database
Operating System[MS] Operating System [Linux]
Oracle Version [9i] 10g
One Character Set Other Character Set
 Export/Import Restrictions

Cannot use for Large Databases

Slow Data Loading and Unloading

Limited fine-grained object selection

Limited Job Monitoring

No callable API

Limited Maintenance
 DATA PUMP - Introduction

Oracle Data/Metadata Loading/Unloading utility

Use PL/SQL API

DBMS_DATAPUMP for Data written in direct path stream format

DBMS_METADATA for Metadata written in XML format

New client expdp and impdp

High speed server based utility

Create platform independent dump file in binary
proprietary format

Support parallel, bulk data & metadata movement

Available with all Oracle 10g version for all supported
platform

Can be monitored remotely

Stop and Restart anytime

Allow Data movement thru database links
 DATA PUMP Architecture

SQL Expdp Impdp Grid Control Other

Loader Client Client DB Control Clients

DATA PUMP

DBMS_DATAPUMP
Data / MetaData Movement Engine

ORACLE_ ORACLE_
LOADER DATAPUMP DIRECT METADATA API:
Read Read/Write PATH API DBMS_METADATA
External Table API
 DATA PUMP – New Export Client
Server1>expdp indy/indy@server1
Server2>expdp indy/indy network_link=server1
dumpfile=indy.dmp logfile=indy.log
dumpfile=indy.dmp
dumpfile=indy.dmp full=y
logfile=indy.log full=y

Expdp
Client

NETWORK_LINK
Server 1 Server 2
DbLink
DATA PUMP
Server JOB
Process
DATA PUMP JOB

Dump File Dump File

Set Set
 DATA PUMP – New Export Client
Server2>impdp
Server1>impdp
Server1>
Server2> impdp indy/indy@server1
Server2>impdp indy/indy@server1
indy/indy dumpfile=indy.dmp
dumpfile=indy.dmp
logfile=indy.log
network_link=server1 full=y
dumpfile=indy.dmp
logfile=indy.log full=y
logfile=indy.log full=y

Impdp
Client

NETWORK_LINK
Server 1 Server 2
DbLink
DATA PUMP
Server JOB
Process
DATA PUMP JOB

Dump File
Set
DATA PUMP – Data Unloading Features
Source Database

Data-Access
Methods

Direct External
Path Tables

Dump File
Set
When DIRECT Path is not used by DATAPUMP

Loading tables with Global index on partitioned tables
exists

Loading tables with domain index exists for a LOB column

Loading Clustered tables

Loading tables with active Triggers

Loading and Unloading tables with encrypted columns

Loading tables with fine-
fine-grained access control enabled for
inserts

Loading table containing BFILE or VARRAY columns
 DATA PUMP – Data Unloading Features
Source Database
Include=function
Include=procedure
Include=trigger
Data-Access Fine-Grained
Methods Filtering Include=view
include=table:”in (‘emp’,’dept’)”
Procedure/Function
Package/Trigger/views
Grant/Constraints
EXCLUDE/INCLUDE
Exclude=index
Exclude=trigger
Exclude=view:”like ‘app%’”

Dump File
Set
 DATA PUMP – Data Unloading Features
Source Database

expdp hr/hr parallel=3

Data-Access Fine-Grained
Parallelism
Methods Filtering

PARALLEL
To specify the number of
Worker processes

Dump File Set

 DATA PUMP – Data Unloading Features
Source Database

Data-Access Fine-Grained
Parallelism Estimate
Methods Filtering

ESTIMATE -- Specifies space used by each Table data

--Only for Data and not for Metadata
--Available in the Logfile or output device
... exported “HR"."PRODUCT" 136.4 MB 649114 rows
ESTIMATE_ONLY -- To determine space required for Export
-- No Data is actually exported
$ expdp scott/tiger ESTIMATE_ONLY=Y
 DATA PUMP – Data Unloading Features
Source Database

Data-Access Fine-Grained Monitor

Parallelism Estimate
Methods Filtering Long Jobs

JOB_NAME -- To Specify the Name to the Job

If not given, System generated name is given
expdp hr/hr JOB_NAME=fullScott ……

ATTACH -- To Attach to existing Job

expdp hr/hr ATTACH= fullScott
 DATA PUMP – Dictionary Views Monitor
Long Jobs

DBA_DATAPUMP_JOBS

OWNER_NAME,
JOB_NAME

DBA_DATAPUMP_SESSIONS

SADDR

V$SESSION

V$SESSION_LONOPS Monitor the Progress of Long-

running Data Pump Job
 DATA PUMP IMPLEMENTATION
DBMS_DATAPUMP.OPEN
DBA Dump File Set
Master
Data,Metadata,
Table Master Table
Client
A’s
Shadow
Process
Worker 1:
Master Metadata
Status Queue
Control
Process Worker 2:
Direct Path
Dynamic
Commands
Client [Stop,Start]
B’s Worker 3: P1
Shadow External Table
Process DBA
DATAPUMP Command P2
JOBS & Control
Queue
Log File
DBA
 DATA PUMP Process Components
Client
A’s

Foreground process created with datapump calls
Shadow like dbms_datapump.open
Process

Create a Job based on JOB_NAME parameter

Create Master Table

Create 2 Queues namely
Master
Table
Status Queue and

Command & Control Queue

Create MCP
Master

Give Status to the Client A Status Command
Control
Process goes away when client detachQueue &Queue
Control
Process
 DATA PUMP Process Components
Master

Table created within the Schema who initiated
Table Expdp/Impdp

Table Name = JOB_NAME (If specified)
OR

SYS_<operation>_<mode>_NN
where NN  2-digit integer starting at 01
Operation  EXPORT | IMPORT
Mode  FULL | TABLESPACE | SCHEMA
TABLE| TRANSPORTABLE

Heart of Datapump Containing Dumpfile Set, Job
parameters, Status of Exported/Imported database
objects, Location of Dumpfile etc
 DATA PUMP Process Components

One Background process for each Datapump operation
Master
Control
Ora_dmNN_<instanceName>
Process

Responsible for

Assigning Data/Metadata Loading/Unloading to
Workers Process

Maintain Job State

Establish and Maintain Dump file Set information

Maintain Restart and dump file set information in
Master Table

Logging operations

Update Status Queue
 DATA PUMP Process Components
Status Queue
Status Queue

Populated by MCP

Consumed by Clients Shadow Process.

Following information is available for clients

Status of the Job

Logging Information

Errors Encountered

Queue name like
KUPC$S_1_<JobQueTimestamp>
KUPC$S_1_20060221135223

Queue is owned by SYS

Can check the name in Alert.log file
 DATA PUMP Process Components
Command & Control Queue

Used by MCP to send command and control of worker
Command
process.
& Control
Queue
Used by Client shadow process for executing
Abort/Restart request of Datapump job to MCP

Bidirectional as MCP send and listen to the queue

Queue name like
KUPC$C_1_<JobQueTimestamp>
KUPC$C_1_20060221135223

Queue is owned by SYS

Can check the name in Alert.log file
 DATA PUMP Process Components
Worker 1:
Number of Process depend on the value of PARALLEL
Metadata parameter.

Responsible for Loading/Unloading of Data and
Worker 2: Metadata.
Direct Path
data
Background process of the form
Ora_DWnn_<instanceName>
Worker 3:
Describe what object it is working on like Tables or
External
Table Indexes or views and so on

Send information about the current status of Database
objects like Pending, Completed, Failed and so on
 DATA PUMP Process Components

Represented by DUMPFILE parameter.
Dump File
Contains the Data as well as metadata information
Set

Location depend on the DIRECTORY setting

SQL>CREATE DIRECTORY dr1 AS ‘/oradata/dpdir’;

SQL>GRANT READ ON DIRECTORY dr1 TO hr; Import
SQL>GRANT WRITE ON DIRECTORY dr1 TO hr; Export
DATA PUMP Fileset Location Preferences

PER- FILE Directory Specifications e.g

1 DUMPFILE=DR1:ab.dmp

Use DIRECTORY Parameter e.g

2 DIRECTORY=DR1

Use DATA_PUMP_DIR Env Setting e.g.

3 $ DATA_PUMP_DIR=DR3

Use Default DATA_PUMP_DIR Database

4 Setting e.g.
Default to $ORACLE_HOME/rdbms/log
 DATA PUMP different File Types
DUMPFILE -- Used for creating Dump Files
LOGFILE -- Used for Creating Log file
SQLFILE -- Used for DDL Script File
 DATA PUMP Export Interfaces
Command Line Interface
$ expdp Id/Pass DIRECTORY=abc DUMPFLE=indy.dmp PARALLEL=4

Parameter File Interface

$ expdp PARFILE=indy.par
 DATA PUMP – Different Modes
Database Level with FULL=Y
D
Tablespace Level with TABLESPACE=users A
T
Schema Level with SCHEMAS
Need EXP_FULL_DATABASE role = hr A T
S B A
Table Level with TABLES=emp
Need EXP_FULL_DATABASE role - New in 10g C L
T E B
Only TablesTransportable
Including from same Schema can be
Tablespace A H
Default Mode. S
specified B E
P A
EXP_FULL_DATABASE role allow to specify L M A S
list of Schemas E A C
E E
 9i Export Vs 10g Datapump Export
Export [ exp ] Data Pump Export [ expdp ]

FILE DUMPFILE
LOG LOGFILE
GRANTS EXCLUDE & INCLUDE
INDEXES EXCLUDE & INCLUDE
CONSTRAINTS EXCLUDE & INCLUDE
FEEDBACK STATUS
OWNER SCHEMAS
ROWS=N CONTENT=METADATA_ONLY
ROWS=Y CONTENT=ALL
CONSISTENT FLASHBACK_SCN
INCLUDE=GRANTS
 9i Export Vs 10g Datapump Export
Export [ exp ] Data Pump Export [ expdp ]

RECORDLENGTH NONE
RESUMABLE NONE
RESUMABLE_NAME NONE
USERID NONE
VOLSIZE NONE
COMPRESS NONE
STATISTICS NONE
BUFFER NONE
DIRECT NONE
NONE TABLESPACES
 DATA PUMP Export- New Parameter

COMPRESSION -- To Compress Metadata in Dumpfile

METADATA_ONLY | NONE
By Default it is METADATA_ONLY

CONTENT -- To take export of Data|Metadata|Both

DATA_ONLY | METADATA_ONLY| ALL

KEEP_MASTER -- To Keep Master Table in the Schema

KEEP_MASTER=Y
 DATA PUMP Export- New Parameter
NETWORK_LINK -- To specify name of Database Link Name
Enable Network Export
NOLOGFILE -- By default Logfile export.log is created
VERSION -- To specify the Version of database objects to be
exported
SAMPLE -- To specify the Percentage of data to be
sampled
You can specify the specific table
SAMPLE=100
SAMPLE=“SCOTT”.”DEPT”:40
 DATA PUMP – Example # 1
$ expdp hr/hr FULL=Y PARALLEL=4
DUMPFILE=DR1:fullhr1%U.dmp,
DR2:fullhr2%U.dmp,
DR3:fullhr3%U.dmp,
DR4:fullhr4%U.dmp,
This will Create
• Files as fullhr101.dmp, fullhr201.dmp and so
• Job name as SYS_EXPORT_FULL_01
• Master Table Name as SYS_EXPORT_FULL_01
• Use DR1,DR2,DR3,DR4 Directory defined in
Database
 DATA PUMP – Example # 2
$ expdp hr/hr PARFILE= indy.par
DIRECTORY=DR1
DUMPFILE=hr1%U.dmp
SCHEMAS=HR
INCLUDE=PROCEDURE
INCLUDE=PACKAGE
INCLUDE=VIEW:”like ‘VW_H%’ “

This will export

• All Procedure, Packages and views like VW_H
• No Schema Definition/other System grants are exported
• Master Table Name as SYS_EXPORT_FULL_01
• Use DR1 Directory defined in Database
 New Interactive Mode Monitor
Jobs Remotely

$ expdp hr/hr ATTACH = jobname

Or
$ expdp hr/hr ATTACH
… You will be attached to the Job running as HR User
Or
$ expdp hr/hr dumpfile=a.dmp logfile=a.log full=y
… when Export is started press Cntrl C [^C]
… You will get EXPORT> prompt

EXPORT>
 DATA PUMP – Interactive Mode Monitor
Jobs Remotely

ADD_FILE -- Add file or use %U

CONTINUE_CLIENT -- Back to Logging Mode
EXIT_CLIENT -- Exit Interactive Mode
Session broke but job continue
KILL_JOB -- Session broke and Job killed
Cannot be restarted
PARALLEL -- Increase or Decrease Workers
START_JOB -- Start the job to which attached
STATUS -- Display the Current Status of Job
STOP_JOB -- Stop the Current Job
Job can be restarted
 New Interactive Mode Monitor
Jobs Remotely

New Parameter in Release 2

FILESIZE --You can restrict the size of file added with

ADD_FILE

In Rel 1, this file size was UNLIMITED

 Interactive Mode Example Monitor
Jobs Remotely

$ expdp hr/hr attach = export1

….
Export> parallel = 5
Export> add_file = export5.dmp
Export> start_job
Export> continue_client
 9i Import Vs 10g Datapump Import
Import [ imp ] Data Pump Import [ impdp ]

FILE DUMPFILE
LOG LOGFILE
GRANTS EXCLUDE & INCLUDE
INDEXES EXCLUDE & INCLUDE
CONSTRAINTS EXCLUDE & INCLUDE
FEEDBACK STATUS
ROWS=N CONTENT=METADATA_ONLY
ROWS=Y CONTENT=ALL
DESTROY RESUSE_DATAFILES
FROMUSER REMAP_SCHEMA
TOUSER REMAP_SCHEMA
INDEXFILE SQLFILE
IGNORE TABLE_EXISTS_ACTION
 9i Import Vs 10g Datapump Import
Import [ imp ] Data Pump Import [ impdp ]

RECORDLENGTH NONE
RESUMABLE NONE
RESUMABLE_NAME NONE
USERID NONE
VOLSIZE NONE
STATISTICS NONE
BUFFER NONE
DIRECT NONE
COMMIT NONE
FILESIZE NONE
NONE TABLESPACES
NONE SKIP_UNUSABLE_INDEXES
NONE STREAMS_CONFIGURATION
 DATA PUMP Import New Parameter
JOB_NAME | ATTACH | DIRECTORY | NETWORK_LINK | PARALLEL

ENCRYPTION_PASSWORD -- Allow export/import of Encrypted

column data
data
$expdp hr/hr dumpfile=dr1:abc.dmp encryption_password=6789
$impdp hr/hr dumpfile=dr1:abc.dmp encryption_password=6789
You cannot use this parameter with NETWORK_LINK

FLASHBACK_SCN -- Valid with NETWORK_LINK to get

FLASHBACK_TIME Data consistent as of this SCN |Time
Both of these parameter are mutually exclusive
 DATA PUMP Import New Parameter
REMAP_DATAFILE -- Useful when moving data between platform
Change Source file to Target file in DDL’s
REMAP_SCHEMA -- Load all Source Schema objects to Target
Schema
-- Can create Target Schema if dump file set
contains CREATE USER
remap_schema(HR,SCOTT)

REMAP_TABLESPACE -- All Source Tablespace are mapped to

destination tablespaces
-- You can specify multiple such parameters
remap_tablespace(users,indy_data)
 DATA PUMP Import New Parameter

TABLE_EXISTS_ACTION -- What to do if table already exist

SKIP Leave the table and move to next object [Default]
APPEND Load rows leaving existing data unchanged
TRUNCATE Delete existing rows and load rows
REPLACE Drop existing table and create and load rows
 DATA PUMP Import New Parameter
TRANSFORM -- Allow to alter Object Creation DDL before Loading
TRANSFORM = <TrName>:Value[:object_type]
Where <TrName> can be
SEGMENT_ATTRIBUTES = Y | N --YDefault
-- Y means following are included with DDL
- Physical attributes
- Storage attributes
- Tablespaces
- logging
STORAGE = Y | N --YDefault
-- Y means Storage clauses are included
OID = Y | N --YDefault
-- Y means object ID of abstract types is reused
PCTSPACE = Integer value
-- Reduce the space required for tablespace by
performing shrink on tablespace storage
 DATA PUMP Import New Parameter
Impdp hr/hr dumpfile=indy.dmp transform=SEGMENT_ATTRIBUTES:n:index
---- Ignore Index Storage and Tablespaces Clause
 DBMS_DATAPUMP API
Use the following steps to create Datapump script
1. Create a handle using DBMS_DATAPUMP.OPEN function
Handle1 :=
DBMS_DATAPUMP.OPEN(‘<operation>’,’<modes>’,’<dblink>,’<jobname>’,’<version>’);

Operation can be EXPORT | IMPORT | SQL_FILE

Modes can be FULL | SCHEMA | TABLESPACE | TRANSPORTABLE | TABLE
Dblink is the name of database link
Jobname is the name of Datapump job
Version can be LATEST | COMPATIBLE
2. Get the Dumpfile Set information
DBMS_DATAPUMP.GET_DUMPFILE_INFO(‘indy.dmp’,’DATA_PUMP_DIR’,’finfo,ftype);

3. Add any Object filtering

DBMS_DATAPUMP.METADATA_FILTER(handle1,’SCHEMA_NAME_EXPR’,’IN(‘’SCOTT’’)’);

4. Start the Datapump Job

DBMS_DATAPUMP.START_JOB(handle1);
Enterprise Manager-DbControl
QUESTIONS
ANSWERS
Thank You