Oracle etl openworld

ADVANCED PL/SQL AND ORACLE 9I ETL
Doug Cosman, SageLogix, Inc.
This paper will describe some of the more advanced PL/SQL capabilities, which can be exploited to improve
application performance as well as add new functionality to the database. Topics covered will include advantages of
nested tables over index_by tables, using table functions to return rows from a function, bulk binding, native
compilation, returning cursors, and streaming data output using pipelined table functions. While these topics are
important in their own right to the sophisticated PL/SQL developer, they are needed as background information for
understanding Oracle’s strategy for implementing an ETL solution within the database. It will be shown how all
these concepts come together in the new Oracle 9i ETL functionality which uses PL/SQL and 9i external tables to
transform data without the use of third party ETL tools.
BULK BINDING
Interaction with Oracle in any host language, including PL/SQL, involves the binding of host variables in and out of
the SQL engine. An ‘in-bind’ is when we pass a value from a program to the SQL engine, often either to constrain on
a column or to specify a value for a DML statement. The following UPDATE statement uses in-binds for both
purposes.
DECLARE
v_quantity NUMBER := 0;
v_sales_id NUMBER := 2314;
BEGIN
UPDATE f_sales_detail
SET quantity = v_quantity
WHERE sales_id = v_sales_id;
END;
Commonly, in-binds are only of interest because they are essential for SQL statements to be sharable. When DBA’s
talk of the importance of applications using ‘bind variables’ it is in the context of in-binds since, in applications that
use dynamic SQL, using literals instead of bind variables causes each SQL statement to be parsed. While this is a
critical consideration for overall database performance, the relative cost of the bind in this statement is trivial because
only a single bind is required regardless of how many rows are affected by the statement.
An ‘out-bind’ occurs when values are passed from the SQL engine back to the host language. Oracle makes the
distinction between values that are passed back via a RETURNING clause in SQL as opposed to when values are
passed back by during a fetch operation but for the purpose of this paper I will refer to both of these operations as
out-binds.
When processing a cursor, application developers can choose to either fetch back values one-at-a-time or returned in
a batch operation which will bind back many rows to the host application in a single operation. Before the release of
Oracle 8i values being bound out into PL/SQL host variables had to be fetched one at a time. The following
CURSOR FOR-LOOP construct is a familiar one.
DECLARE
CURSOR cust_cur (p_customer_id NUMBER) IS
SELECT *
FROM f_sales_detail
WHERE customer_id = p_customer_id;
v_customer_id NUMBER := 1234;
www.sagelogix.com Oracle Open World 2003

Advanced PL/SQL and Oracle 9i ETL Cosman
BEGIN
FOR rec IN cust_cur (v_customer_id) LOOP
INSERT INTO sales_hist
(customer_id, detail_id, process_date)
VALUES
(v_customer_id, rec.sales_id, sysdate);
END LOOP;
END;
In a CURSOR FOR-LOOP, a record variable is implicitly declared that matches the column list of the cursor. On
each iteration of the loop, the execution context is switched from the PL/SQL engine to the SQL engine, performing
an out-bind of the column values into the record variable once for each loop iteration. Likewise, an in-bind for the
insert statement will occur once on each iteration. Although stored PL/SQL code has the advantage over other host
languages of keeping this interaction within the same process, the context switching between the SQL engine and the
PL/SQL engine is relatively expensive making the above code very inefficient. In addition, the cursor is defined as
SELECT * instead of just selecting from the columns to be utilized which is also inefficient. Whether the code
references a column or not, Oracle will have to fetch and bind over all of the columns in the select list, slowing down
code execution.
A better way to perform the above task would be to utilize bulk binding, introduced in Oracle 8i, for both the fetch
and the insert statements. We have two new PL/SQL operators to accomplish this. The BULK COLLECT
statement is used to specify bulk out-binds while the FORALL statement is used to provide bulk in-binds for DML
statements.
DECLARE
TYPE sales_t IS TABLE OF f_sales_detail.sales_id%TYPE
INDEX BY BINARY_INTEGER;
sales_ids sales_t;
v_customer_id NUMBER := 1234;
max_rows CONSTANT NUMBER := 10000;
CURSOR sales(p_customer_id NUMBER) IS
SELECT sales_id
FROM f_sales_detail
WHERE customer_id = p_customer_id;
BEGIN
OPEN sales(v_customer_id);
LOOP
EXIT WHEN sales%NOTFOUND;
FETCH sales BULK COLLECT INTO sales_ids LIMIT max_rows;
FORALL i IN 1..sales_ids.COUNT
INSERT INTO sales_hist
(customer_id, detail_id, process_date)
VALUES
(v_customer_id, sales_ids(i), sysdate);
END LOOP;
CLOSE sales;
END;

In this example, the fetch statement returns with the sales_ids array populated with all of the values fetched for the
current iteration, with the maximum number of rows fetched set to 10,000. Using this method, only a single context
switch is required for the SELECT statement to populate the sales_ids array and another switch to bind all of the
fetched values to the INSERT statements. Note also that the FORALL statement is not a looping construct – the
array of values is given over in batch to the SQL engine for binding and execution. This second implementation will
run at approximately 15 times the speed of the first, illustrating the importance of efficient binding in data driven
code.
One potential issue with the bulk binding technique is the use of memory by the PL/SQL array variables. When a
BULK COLLECT statement returns, all of the fetched values are stored in the target array. If the number of values
returned is very large, this type of operation could lead to memory issues on the database server. The memory
consumed by PL/SQL variables is private memory, allocated dynamically from the operating system. In dedicated
server mode it would be the server process created for the current session that allocates memory. In the case where
such allocation becomes extreme, either the host will become memory bound or the dedicated server process will
reach a size where it tries to allocate beyond its addressing limits, normally 2 GB on many platforms. In either case
the server processes call to malloc() will fail raising an ORA-04030 out of process memory error.
To prevent this possibility when loading anything larger than a small reference table, use the optional LIMIT ROWS
operator, first introduced in Oracle 8.1.6, to control the ‘batch size’ of each BULK COLLECT operation. In the code
example below the cursor will iterate though batches of 10,000 rows fetching in the values and inserting 10,000 rows.
On the final iteration, the cursor will fetch the remaining balance. Placement of the EXIT WHEN clause should be
before the FETCH statement or the last, incomplete batch will not be processed.
The above example is only for the purpose of demonstrating coding techniques. The same logic could be
accomplished totally in SQL using an INSERT AS SELECT statement. Doing this in pure SQL would be faster yet
since no host binds or procedural execution would be required at all. This brings up an important point – never do
anything procedurally that can be accomplished through well crafted SQL.
SQL T YPES VS. PL/SQL TYPES
Every programming language has what is called a type system – the set of data types that the language implements.
Among other things, a language’s type system determines how a value of a particular type is represented. Since
PL/SQL was developed by Oracle as the procedural extension to SQL, these languages have an overlapping type
system. For example PL/SQL NUMBER type is the same data type as the SQL NUMBER type. This is one of the
main advantages to using PL/SQL as opposed to another language like Java for data intensive operations – there is no
type conversion cost when binding values from a SQL query back to the PL/SQL. However, being a superset, not all
PL/SQL data types are part of the SQL type system. For example, the types BOOLEAN and BINARY_INTEGER
are only found in PL/SQL.
COLLECTIONS
There are three flavors of collection types, one, which is only available in PL/SQL, and two others that are shared
between both languages.
ASSOCIATIVE ARRAYS (PL/SQL TABLES)
Probably the most familiar collection type is the PL/SQL index-by table, now called associative arrays in Oracle 9i
Release 2. The code block below is a typical use of an associative array.
DECLARE
TYPE num_array IS TABLE OF NUMBER INDEX BY BINARY_INTEGER;
powers num_array;
BEGIN
FOR i IN 1..100 LOOP
powers(i) := power(2, i);
END LOOP;

END;
The element type of an associative array can be almost any PL/SQL type including a record type. The first thing to
note is that PL/SQL associative array types are not SQL types – which is what one would expect since they are
indexed by BINARY_INTEGER, a non-SQL type. This type of array is by far the most commonly used in PL/SQL
code since it has the advantage of being the simplest to use. Array sizing and allocation is totally dynamic – there is
not maximum array size other than the memory constraints imposed by host environment. As illustrated above, the
code does not need to extend the size of the array to add new elements.
Indexing By VARCHAR2
A welcome addition in 9i Release 2 is the ability to use the VARCHAR2 data type as an index key. Similar to associative
arrays in Perl and Awk, or Hashtables in Java, this data type enables the look-up of a value using a VARCHAR2. Many
existing PL/SQL routines can benefit from this capability. Now it is obvious why the table data type has been renamed to
associative array.
NESTED TABLES
Unlike associative arrays, the nested table data type is also a SQL data type. A nested table is similar to an associative
array in that there is no maximum size to the array however prior to assigning a new element to a nested table a
PL/SQL program needs to explicitly extend the size before adding new elements. A nested table is an object type and
therefore needs to first be initialized with a constructor before being used. For many PL/SQL programs, these two
added requirements make regular associative arrays a better choice for basic array functionality in code, however we
will see that with nested tables a whole new set of options will open up that would not be possible with associative
arrays.
Like all collection types, before a variable of a particular collection type can be defined, its type must first be declared.
Since nested tables are a shared type, there are two ways that this can be done; locally in PL/SQL code or globally in
the database. The first example shows a local PL/SQL declaration.
DECLARE
TYPE nest_tab_t IS TABLE OF NUMBER;
nt nest_tab_t := nest_tab_t();
BEGIN
FOR i IN 1..100 LOOP
nt.EXTEND;
nt(i) := i;
END LOOP;
END;
Note that the variable was initialized to an empty nested table using the constructor for its type. Also, the example
shows how the nested table EXTEND method is used to allocate a new element to the array so that it can be assigned
to in the next statement.
However, the most interesting use for nested tables is when you take advantage of sharing types with the SQL engine.
This next example, which defines an object to hold demographic information associated with an email, lays the
groundwork for many other possibilities. This is a SQL statement that would typically be run from a tool like
SQL*Plus.
CREATE OR REPLACE TYPE email_demo_obj_t AS OBJECT (
email_id NUMBER,
demo_code NUMBER,
value VARCHAR2(30));
/
CREATE OR REPLACE TYPE email_demo_nt_t AS TABLE OF email_demo_obj_t;

/
Note that in SQL*Plus, the syntax of the CREATE TYPE statement requires both a semi-colon and a forward slash
on the next line, similar to the required syntax of a PL/SQL anonymous block, although this is in fact a SQL
statement. Now that the required SQL types have been defined globally in the database they can be referenced in
PL/SQL. The cool thing is that local variables in code of this type can now be treated like a SQL object, which
means that the SQL engine can be used to manipulate local nested table variables as if they were true database tables.
Table Functions
To do this, the PL/SQL code executes a SQL statement passing the local nested table variable to the server. There
are two special functions necessary to achieve this functionality. The TABLE function tells the server to bind over
the values of the nested table, perform the requested SQL operation and return the results back as if the variable was
a SQL table in the database. The CAST function is an explicit directive to the server to map the variable to the SQL
type that was defined globally in the previous step.
With this capability, many new operations become possible.. For example, one can take a nested table of objects that
have been created in code and send them to the server for ordering or aggregation. Almost any SQL operation is
possible. For example a nested table can be joined with other SQL tables in the database. The next example shows a
simple ordering of an array by the second field.
DECLARE
eml_dmo_nt email_demo_nt_t := email_demo_nt_t();
BEGIN
-- Some logic that populates the nested table …
eml_dmo_nt.EXTEND(3);
eml_dmo_nt(1) := email_demo_obj_t(45, 3, '23');
eml_dmo_nt(2) := email_demo_obj_t(22, 3, '41');
eml_dmo_nt(3) := email_demo_obj_t(18, 7, 'over_100k');
-- Process the data in assending order of email id.
FOR r IN (SELECT * FROM TABLE(CAST(eml_dmo_nt AS email_demo_nt_t)) ORDER BY 1)
LOOP
dbms_output.put_line(r.email_id || ' ' || r.demo_id);
END LOOP;
END;
Another possibility is to exploit this technique to support direct path inserts into the database. For data warehouse
applications, direct path is a way to dramatically increase insert performance by allowing the session’s server process
to format and write data blocks directly to the table segment, bypassing the buffer cache. This can be accomplished
using the APPEND hint. Optionally, one can also use the NOLOGGING hint to suppress most of the redo log
generation for the statement, however this should normally only be done for inserts into staging tables that would not
require recovery in the event of media failure.
However, direct path inserts are supported only for INSERT AS SELECT statements but not INSERT BY
VALUES. Recall in the previous section on the advantages of bulk binding, it was noted that performing bulk out-bind
using a FORALL statement to implement an INSERT of associative array values was up to 15 times faster than
a conventional CURSOR FOR-LOOP. However, the FORALL statement is implemented using an INSERT BY
VALUES clause. Using the nested table approach, if one wanted to perform the insert using direct path, one could
do so using the following syntax, assuming the type EMAIL_DEMO_NT_T was defined as a nested table of objects
that matched the data types of the EMAIL_DEMOGRAPHIC table. This method, when both direct path and
NOLOGGING features are appropriate, is the fastest way one can bind over a large number of values from PL/SQL
to SQL tables for INSERTS.

INSERT /*+ append nologging */ INTO email_demographic
(SELECT * FROM TABLE(CAST(eml_dmo_nt AS email_demo_nt_t)));
VARRAYS
The last collection type to be discussed is the varray. Like nested tables, varrays can be both PL/SQL types and SQL
types and therefore can take advantage of the many of the features listed above. The main differences with varrays in
PL/SQL is that their maximum size must be specified when the type is declared. It should be noted that both varray
types as well as nested table types can define the column type of a SQL table. In the former case, if the size of the
varray type is 4000 bytes or less, it can be stored in-line in the data block along with other column values. In contrast,
the column data for a nested table is stored in a system managed child table making it very similar to a normal
parent/child table relationship. Because they have a shared type, PL/SQL nested table or varray variables can be used
to atomically insert values into tables that use them. Apart from this capability, varrays are of less interest than nested
tables to the PL/SQL developer because they have the restriction of an upper bound and most anything one can do
in code with a varray, one can do with a nested table.
MULTI-DIMENSIONAL ARRAYS
Another new feature that has been provided in Oracle 9i, Release 1 is the long awaited capability of multi-dimensional
arrays, which Oracle has implemented as collections of collections. Multi-dimensional arrays have been around in
most programming languages for a long time and are now available in PL/SQL. Technically, all collection types
support only a single dimension, however by now allowing a collection element to become a collection, one has the
effectively the same data structure. The following code shows the way to declare and reference a two-dimensional
array of numbers.
DECLARE
TYPE element IS TABLE OF NUMBER INDEX BY BINARY_INTEGER;
TYPE twoDimensional IS TABLE OF element INDEX BY BINARY_INTEGER;
twoD twoDimensional;
BEGIN
twoD(1)(1) := 123;
twoD(1)(2) := 456;
END;
At first one would think that, while an interesting capability, it has no potential impact on performance but it will be
shown later in this paper how the combination of this capability along with the use of packaged variables can open up
the door to dramatically speeding up PL/SQL code.
PACKAGE VARIABLES
PL/SQL packages offer a number of advantages over stand-alone code or anonymous blocks. While some of the
advantages come from the familiar ability to organize code into logical collections of related procedures and functions,
an often ignored aspect is the use of package-level variables and package initialization sections.
A package variable is essentially a variable that is declared globally, typically at the top of the package body outside of
any procedure definition. Once set, a package variable will maintain its state for the life of the session, as opposed to
variables local to procedure definitions that only exist for the duration of the procedure call. Every package body
implementation can optionally include a block of code at the end of the specification referred to as the initialization
section. This code will run only once – when the package is first referenced and is normally used to initialize any
package variables that are used.
The use of package variables is a powerful technique to speed up SQL statements that has been used for many years.
Consider a procedure that is called repeatedly in a busy OLTP database that inserts a row into a table. One of the
values passed to the procedure is a key that is used to first lookup another value, which in turn, is used in the insert.

Most PL/SQL code will first execute a select statement for the row of interest, binding the value to a local variable
that will then be used on the subsequent insert statement. A unique indexed table look-up is relatively quick but if an
application is being driven hard, the cost can be more than one would expect. Referencing a pre-initialized array
value is approximately 20 times faster than an index based table look-up, making this technique a real time saver for
intensive operations.
Using package variables, the lookup can be avoided entirely by first initializing an array with the desired values. For
example, in a marketing application a procedure is passed a zip code as an argument and must perform an insert into a
fact table that requires the city name as a denormalized value in the table. Assume also that there is a lookup table,
with a numeric zip code as the primary key as well as the city and state information that the zip code maps to. Code
to avoid all of the look-ups would look like this.
CREATE OR REPLACE PACKAGE BODY direct_mkt AS
TYPE zip_array IS TABLE OF VARCHAR2(30) INDEX BY BINARY_INTEGER;
zip_deref zip_array;
PROCEDURE do_insert(f_name VARCHAR2, l_name VARCHAR2, zip NUMBER) IS
BEGIN
INSERT INTO user_data (f_nm, l_nm, city_nm)
VALUES (f_name, l_name, zip_deref(zip));
COMMIT;
END;
-- Package initialization section.
BEGIN
FOR rec IN (SELECT zip_code, city FROM dma) LOOP
zip_deref(rec.zip_code) := rec.city;
END LOOP;
END;
Until Oracle 9i Release 2, this technique couldn’t be used if the look-up key was non-numeric or composite, but now
with the combination of VARCHAR2 associative arrays and multi-dimensional arrays, it can be extended to almost
any look-up table of a reasonable size. For example consider a table of individuals mapped to email addresses that has
as it’s primary key, a composite index of numeric user id and an email address that they are associated with. The
following code shows how to implement this.
CREATE OR REPLACE PACKAGE BODY email_push AS
TYPE email_array IS TABLE OF VARCHAR2(30) INDEX BY VARCHAR2(30);
TYPE usermail_array IS TABLE OF email_array INDEX BY BINARY_INTEGER;
user_emails usermail_array;
FUNCTION lookup(p_userid NUMBER, p_email VARCHAR2) RETURN VARCHAR2 IS
BEGIN
RETURN user_emails(p_userid)(p_email);
END;
BEGIN
FOR rec IN (SELECT user_id, email, l_name FROM user_emails) LOOP
user_emails(rec.user_id)(rec.email) := rec.l_name;
END LOOP;
END;

NATIVE COMPILATION
Like Java, PL/SQL has always been an interpreted language, allowing code to be portable between platforms. When a
PL/SQL library unit is compiled, the source code is compiled into generic byte code that closely resembles assembler
code. At run time, the PL/SQL run time engine executes the byte code. The disadvantage of any interpreted
language is execution speed. To address this, Oracle introduced the native compilation facility, which requires that the
host on which the database resides has a native C language compiler. When a PL/SQL library unit is compiled using
native compilation, the code is first compiled down to byte code as is normally the case. Then the byte code is
converted to C source code which, in turn, is used as input to the native C compiler on the host to produce a shared
object library, in UNIX, or a DLL under Windows.
Native compilation of PL/SQL does increase the speed of execution about 300% when no interaction with the
database is performed, however when tested in the kind of environments described here, where large amounts of data
are fetched or written to the database, the performance gains are fairly modest. This is because most of the time
spent in data intensive code is the interaction with the database, not the execution of business logic. Note that while
interpreted and compiled PL/SQL can be mixed and used to interoperate, Oracle does not recommend that practice
for production environments.
RETURNING RESULT SETS
Using PL/SQL to create an API for applications to interact with the database has the advantage of removing
dependencies of client applications on the physical implementation of the schema. With this kind of approach, the
DBA is free to change the physical model of the schema, perhaps for performance reasons, and not have to worry
about breaking production applications. For write activity, this is the straightforward problem of implementing the
necessary procedures to update and insert records in the database at some level of abstraction that the application
requires. However the problem of returning result sets from a PL/SQL procedure or function is a bit more difficult.
There are three general approaches for doing this; returning cursors, returning collections, or using table functions.
CURSORS VARIABLES
One of the best ways to isolate an application from SQL dependencies is to write a package of PL/SQL functions
that return the REF CURSOR type to the calling programs written in other host languages such as Java or C. Cursor
variables can be either weakly typed, which are more flexible, or strongly typed, which provide greater type safety. Of
course, the application must know the number and data types of the returned columns as well as their semantics in
order to use the data, but it can be totally isolated from the way the data is stored. The following function returns a
weakly typed cursor using the new 9i type SYS_REFCURSOR.
FUNCTION email_cur RETURN sys_refcursor IS
rc sys_refcursor;
BEGIN
OPEN rc FOR
SELECT * FROM email;
RETURN rc;
END;
An application can call the function and bind the returned open cursor to a local cursor variable. The application then
iterates through the result set as if it had been defined and opened locally.
RETURNING COLLECTIONS
Another approach to returning result sets is to write a function that explicitly returns a PL/SQL collection type. The
method has been around for years but it is really best suited for use by calling PL/SQL programs. Also, since there
are no predefined collection types in PL/SQL, the returned type must either be declared in a shared package header
or be a SQL type declared globally in the database. A function that returns a shared collection type is shown below.
The type, email_demo_nt_t, was defined earlier in this paper.

FUNCTION get_email_demo(p_email_id NUMBER) RETURN email_demo_nt_t IS
eml_dmo email_demo_nt_t;
BEGIN
SELECT email_demo_obj_t(email_id, demo_id, value)
BULK COLLECT INTO eml_dmo
FROM email_demographic
WHERE email_id = p_email_id;
/* Apply some business logic and return the result set. */
RETURN eml_dmo;
END;
Note that when the BULK COLLECT feature is used, it is not necessary to initialize or extend a nested table because
Oracle does it automatically. But it is necessary to call the constructor function for the object type in the query itself
to be able to fetch scalar column values into the nested table of objects. This would not be necessary if fetching from
a column of that object type.
TABLE FUNCTIONS (REVISITED)
Most client programs however, don’t really want to deal with trying to bind to a PL/SQL user defined type; instead,
they want a cursor. The TABLE function provides a way to take a function like the one above and return its results
to the caller directly as a cursor. Recall that the TABLE function takes a variable of a globally defined collection type
as an argument, therefore a function with a return type of the same collection type, like the one above, can be used as
an argument to the TABLE function as well. Without modifying the above procedure, a program can return its
output collection as a cursor using the following syntax. Views can be wrapped around this kind of SQL statement to
make life easier for a calling application.
SELECT * FROM TABLE( CAST( get_email_demo(45) AS email_demo_nt_t ));
Pipelined Table Functions
While that approach works, it is really only appropriate for smaller result sets of perhaps a few thousand rows. When
the function executes to populate the result set, the data is buffered in the local variable of the procedure. Only after
the procedure has finished executing, will the rows be returned to the calling application. Memory to store the
buffered data is dynamically allocated from the operating system by the server process executing the procedure If the
result set was very large, operating system memory could become depleted.
Pipelined table functions are an Oracle 9i facility that address this issue by providing a mechanism to stream the
values from the function back to the calling application while the function is executing. A small amount of data
remains buffered in the function’s address space so that result sets can be sent back in batches, which is faster than
row-by-row processing. This is a far more scalable design for this functionality since the operating system memory
footprint is independent of the size of the result set.
To utilize this feature, the function must be declared as PIPELINED and collection objects must be returned one at a
time via a new function called PIPE ROW. The function contains a RETURN statement without arguments that is
used to terminate the cursor. The function can now be rewritten to take advantage of pipelining.

FUNCTION get_email_demo RETURN email_demo_nt_t PIPELINED IS
CURSOR email_demo_cur IS
SELECT email_demo_obj_t(email_id, demo_id, value)
FROM email_demographic;
eml_dmo_nt email_demo_nt_t;
BEGIN
OPEN email_demo_cur;
LOOP
FETCH email_demo_cur BULK COLLECT INTO eml_dmo_nt LIMIT 1000;
EXIT WHEN email_demo_cur%NOTFOUND;
FOR i IN 1..eml_dmo_nt.COUNT LOOP
/* Apply some business logic on the object here, and return a row. */
PIPE ROW (eml_dmo_nt(i));
END LOOP;
END LOOP;
RETURN;
END;
Note that while the return type of the function is still the collection type, the local variable being assigned is the object
type. In this example, the fetch is performed using the BULK COLLECTfeature. The documents illustrate the much
slower row-by-row fetch. Since the signature of the procedure has not been changed, only the implementation, it can
be called the same way as the previous table function using the TABLE and CAST functions.
ORACLE 9I ETL
All of the above information has value in and of itself, but in the context of this paper it has served to provide
background information for understanding the architecture of Oracle’s ETL solution, now part of the Oracle 9i
database. The term ETL stands for Extract, Transform, and Load. In a nutshell it’s the process of reformatting and
loading a data source of one format into the data model of the target database. The extract phase is just about
accessing the data, normally by reading from a flat file. The transformation phase is about converting the input
records into the format of the target database perhaps by changing the record layout, and encoding or transforming
some column values. Loading is simply the process of inserting the transformed data into the database, either directly
into the appropriate database tables or by loading into a staging table for further database processing. Before
describing the Oracle solution, there is still one more piece that is required to understand Oracle’s ETL solution.
This can be illustrated with an example that processes a file produced by a Web marketing application. The file is a
denormalized layout where all of the information produced in a user submission is written out to a single line in the
file. The layout consists of user names and two demographic attributes that are being transformed and loaded into a
normalized table of user ids, attribute codes and values. Each row of input must produce two rows of output. In a
real-world example there would be more attributes in the file. If there were more attributes, each input record would
produce an additional output record. For simplicity, assume the users and attribute codes have already been defined
in the database. The source and target record layouts are as follows. A comma character delimits each field.
EMAIL AGE INCOME
John.Doe@excite.com 71 over_100k
Table 1 Input Record
EMAIL_ID DEMO_CODE VALUE
2345 3 71
2345 7 over_100k
Table 2 Output Records

EXTERNAL TABLES
Oracle introduced the concept of external tables in Oracle 9i Release 1. An external table is simply a flat file on the
local file system that Oracle can now read directly, returning rows to the application as if it was a database table. The
database server reads the file directly using the information from the table definition, which closely resembles a
SQL*Loader control file, including .log and .bad file definitions. Before the table can be defined, an Oracle
DIRECTORY object must be created, which enables the owner to read and write files to that directory. The
definition is as follows.
CREATE TABLE ext_tab
(email VARCHAR2(50),
age NUMBER,
income VARCHAR2(20))
ORGANIZATION EXTERNAL (
TYPE oracle_loader
DEFAULT DIRECTORY data_dir
ACCESS PARAMETERS
(RECORDS DELIMITED BY NEWLINE
LOGFILE data_dir: 'ext_tab.log'
BADFILE data_dir: 'ext_tab.bad'
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(email CHAR(50),
age INTEGER EXTERNAL(2),
income CHAR(20) ) )
LOCATION ('ext_tab.dat') )
REJECT LIMIT UNLIMITED;
The table creation will only create data dictionary information since, by definition, the data is external to the database.
After the table definition is created, it can be queried like any other table except that, since it cannot be indexed – all
access involves a full table scan. The table can also be queried using Oracle’s parallel execution facility which will
cause multiple server processes to partition the file equally and read from it in parallel.
THE ORACLE 9I ETL SOLUTION
At many customer sites, ETL is accomplished by first running a third party ETL tool or a home-grown application
which would read the input file and produce the desired output file, in this case turning each input record line into
two lines in the new file. Third party tools are often smart enough to perform the necessary encoding of file values
to database codes during this process, however many home-grown programs would omit this step until later. In the
latter case, the next step is to use SQL*Loader to perform a direct path load into a database staging table where their
values would be encoded before being inserted into the production tables. The main point here is that to load and
transform, the data are sometimes read and written two or more times.
By combining the above capabilities Oracle has provided the infrastructure to perform ETL in a single step. The
Extract phase is now handled by the external table facility, the Transform capability is provided by either by a SQL
statement or more flexibly, by pipelined table functions, and the Load phase is implemented by direct path inserts of
the transformed data. With this approach the data is read and written only once. For this example, a pipelined table
function will be used to extract the data. Note that if parallel processing is required, the table function must be
defined with the PARALLEL_ENABLE clause or the process will serialize.

PACKAGE BODY etl IS
TYPE hash_table_t IS TABLE OF NUMBER INDEX BY VARCHAR2(30);
email_map hash_table_t;
FUNCTION transform (new_data SYS_REFCURSOR) RETURN email_demo_nt_t
PIPELINED
PARALLEL_ENABLE(PARTITION new_data BY ANY) IS
TYPE ext_tab_array IS TABLE OF ext_tab%ROWTYPE INDEX BY BINARY_INTEGER;
indata ext_tab_array;
email_demo_obj email_demo_obj_t := email_demo_obj_t(null,null,null);
demo_map hash_table_t;
BEGIN
LOOP
EXIT WHEN new_data%NOTFOUND;
FETCH new_data BULK COLLECT INTO indata LIMIT 1000;
FOR i IN 1..indata.COUNT LOOP
email_demo_obj.email_id := email_map(indata(i).email);
email_demo_obj.demo_code := 3;
email_demo_obj.value := indata(i).age;
PIPE ROW (email_demo_obj);
email_demo_obj.demo_code := 7;
email_demo_obj.value := indata(i).income;
PIPE ROW (email_demo_obj);
END LOOP;
END LOOP;
RETURN;
END;
BEGIN
FOR email IN (SELECT email_id, email FROM email) LOOP
email_map(email.email) := email.email_id;
END LOOP;
END;
This function and its call method below constitute the basic templates for using Oracle 9i ETL. There are several
things to note from this example. First, the PIPE ROW function is called twice for each row creating the pivot
required for the transformation. Second, package variables are used to avoid the repeated look-up of the email id.
Third, an open cursor to the select statement that queries the external table is passed to the function. This CURSOR
expression is necessary to read the external table in parallel. Fourth, the parameter uses the new pre-defined
SYS_REFCURSOR type – a weakly defined ref cursor.
It should be noted that for those who wish to avoid hand coding PL/SQL like the above, Oracle’s Warehouse Builder
product can used to generate code such as this, based upon meta data that describes the source and target data.
The impressive thing about this capability is it provides an elegant mechanism for allowing Oracle’s proven parallel
execution engine to read data, stream it through a PL/SQL co-process, and simultaneously write out the data. The
actual transformation call is just the INSERT of the data produced by the function.
INSERT /*+ append nologging */ INTO email_demographic
(SELECT /*+ parallel( a, 4 ) */ *
FROM TABLE( CAST( etl.transform( CURSOR(SELECT * FROM ext_tab ))
AS email_demo_nt_t)) a);

EPILOGUE
Is there yet a better way? For this particular transformation there is. Its interesting to note that the above
transformation could also be accomplished with pure SQL using Oracle’s new extension to the INSERT statement,
the MULTI-TABLE INSERT statement. This new syntax allows for each row returned by a subquery to be inserted
into multiple tables, or the same table multiple times. Since the statement must encode the values as well as perform
the pivot, the external table must be joined to the reference tables. Remember that the external table can’t be indexed
so it should be the driving table in the subquery.
INSERT /*+ append nologging */ ALL
INTO email_demographic (email_id, demo_id, value)
VALUES (email_id, 3, age)
INTO email_demographic (email_id, demo_id, value)
VALUES (email_id, 7, income)
(SELECT /*+ ordered index( b ) */ b.email_id, a.income, a.age
FROM ext_tab a, email b
WHERE a.email = b.email);
Besides being more simple, this query will perform about 4 times faster than the above PL/SQL implementation since
it’s always faster if something can be performed in pure SQL as opposed to any host language. Speed is impressive –
using a small 800 MHz Intel box running Red Hat Advanced Server 2.1,and Oracle 9i Release 2, the above
transformation of 1 million input rows which produces 2 million output rows, executed in approximately 10 seconds
without the use of parallelism. The multi-table insert capability also has limited conditional execution capabilities.
When combined with the SQL CASE statement, one can perform many simultaneous transformations using this
approach. However many transformations are too complex to be performed using this method and must be
accomplished in hand-written code. In this case, the PL/SQL techniques shown here are the best approach.
PERFORMANCE ISSUES
It should also be noted that there appears to be a performance bottleneck returning data through table functions. The
current performance of table functions is approximately 4 times slower compared with the equivalent BULK
COLLECT and FORALL logic presented earlier. I confirmed this with the PL/SQL group at Oracle, and they have
said that they expect to have this issue addressed in 9i, Release 3. However it should be noted that, even with this
bottleneck, performance remains very respectable. When this issue is fixed, I suspect that the Oracle 9i ETL solution
will provide the fastest production ETL method available.
Until the table function issue is sorted out, coding the same logic using bulk binding and perhaps the nested table
insert technique described earlier is probably the best approach. However the table function opens the door to
parallel, concurrent PL/SQL processing of parallel query output and will therefore be the optimal solution when the
bottleneck is resolved.
When Java was first introduced into the database in Oracle 8i, many predicted the demise of PL/SQL. However that
has turned out to be anything but true. Since then, Oracle’s PL/SQL group has done some excellent work to improve
the language with all of the features discussed in this document making it far more powerful then it ever was before.
CONCLUSION
Oracle’s 9i ETL solution offers some great possibilities to transform and load data and will no doubt gain broad
acceptance in the future. It should also be noted that in terms of cost, it is a very attractive solution since this
functionality is already included in the cost of the 9i server.
ABOUT THE AUTHOR
Doug Cosman is a senior consultant for SageLogix, Inc, based in Denver, Colorado. Doug is an Oracle DBA and
developer specializing in Internet technologies, PL/SQL development, Oracle instance tuning, and developer support
and mentoring. He has built databases and applications for clients like excite.com, GM, SmartMoney.com, IBM,
MatchLogic, and Wyndham Hotels. Doug can be reached at: doug_cosman@yahoo.com

Oracle etl openworld

Recommended

More Related Content

What's hot (18)

Viewers also liked (18)

Similar to Oracle etl openworld (20)

Oracle etl openworld