Information Integration Solutions

Center of Excellence

Parallel Framework Standard Practices

Investigate, Design, Develop:

Data Flow Job Development

Prepared by IBM Information Integration Solutions Center of Excellence

July 17, 2006

CONFIDENTIAL, PROPRIETARY, AND TRADE SECRET NATURE OF ATTACHED DOCUMENTS

This document is Confidential, Proprietary and Trade Secret Information (“Confidential Information”) of IBM, Inc. and is provided solely for the purpose
of evaluating IBM products with the understanding that such Confidential Information will be disclosed only to those who have a “need to know.” The
attached documents constitute Confidential Information as they include information relating to the business and/or products of IBM (including, without
limitation, trade secrets, technical, business, and financial information) and are trade secret under the laws of the State of Massachusetts and the United
States.

Copyrights

© 2006 IBM Information Integration Solutions

All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language in
any form by any means without the written permission of IBM. While every precaution has been taken in the preparation of this document to reflect
current information, IBM assumes no responsibility for errors or omissions or for damages resulting from the use of information contained herein.
 Information Integration Solutions
Center of Excellence

Document Goals
Intended Use This document presents a set of standard practices, methodologies, and examples for IBM
WebSphere® DataStage Enterprise Edition™ (“DS/EE”) on UNIX, Windows, and USS.
Except where noted, this document is intended to supplement, not replace the
installation documentation.
Target Audience The primary audience for this document is DataStage developers who have been trained in
Enterprise Edition. Information in certain sections may also be relevant for Technical
Architects, System Administrators, and Developers
Product Version This document is intended for the following product releases:
- WebSphere DataStage Enterprise Edition 7.5.1 (UNIX, USS)
- WebSphere DataStage Enterprise Edition 7.5x2 (Windows)

Document Revision History

Date Rev. Description
April 16, 2004 1.0 Initial Services release
June 30, 2005 2.0 First version based on separation of EE BP into four separate documents, merged
new material on Remote DB2, configuring DS for multiple users.
December 9, 2005 3.0 Significant updates, additional material
January 31, 2006 3.1 Updates based on review feedback. Added patch install checklist item (7.10) and
Windows 7.5x2 patch list.
February 17, 2006 4.0 Significant updates, new material on ETL overview, data types, naming standards,
USS, design standards, database stage usage, database data type mappings, updated
styles and use of cross-references.
March 10, 2006 4.1 Corrected missing Figure 9.
March 31, 2006 4.2 Added new material on establishing job boundaries, balancing job resource
requirements / startup time with required data volume and processing windows, and
minimizing number of runtime processes. Moved Baselining Performance
discussion to Performance Tuniing BP. Expanded performance tuning section.
May 08, 2006 4.3 Removed Architecture Overview (now a separate document). Expanded file stage
recommendations.
July 17, 2006 5.0 Updated directory naming standards for consistency with DS/EE Automation
Standards and Toolkit. Segmented content into “Red Book” and “Standards”.
Clarified terminology (“Best Practices”). Incorporated additional field feedback.

Document Conventions
This document uses the following conventions:
Convention Usage
Bold In syntax, bold indicates commands, function names, keywords, and options that
must be input exactly as shown. In text, bold indicates keys to press, function
names, and menu selections.
Italic In syntax, italic indicates information that you supply. In text, italic also indicates
UNIX commands and options, file names, and pathnames.
Plain In text, plain indicates Windows NT commands and options, file names, and
pathnames.
Bold Italic Indicates: important information.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 2 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Lucida Console Lucida Console text indicates examples

of source code and system output.
Lucida Bold In examples, Lucida Console bold indicates characters that the user types
or keys the user presses (for example, <Return>).

Lucida Blue In examples, Lucida Blue will be used to illustrate operating system command
line prompt.
 A right arrow between menu commands indicates you should choose each
command in sequence. For example, “Choose File Exit” means you should
choose File from the menu bar, and then choose Exit from the File pull-down
menu.
This line The continuation character  is used in source code examples to indicate a line
 continues that is too long to fit on the page, but must be entered as a single line on screen.

The following are also used:

• Syntax definitions and examples are indented for ease in reading.
• All punctuation marks included in the syntax—for example, commas, parentheses, or quotation
marks—are required unless otherwise indicated.
• Syntax lines that do not fit on one line in this manual are continued on subsequent lines. The
continuation lines are indented. When entering syntax, type the entire syntax entry, including
the continuation lines, on the same input line.
• Text enclosed in parenthesis and underlined (like this) following the first use of proper terms
will be used instead of the proper term.

Interaction with our example system will usually include the system prompt (in blue) and the
command, most often on 2 or more lines.

If appropriate, the system prompt will include the user name and directory for context. For example:

%etl_node%:dsadm /usr/dsadm/Ascential/DataStage >

/bin/tar –cvf /dev/rmt0 /usr/dsadm/Ascential/DataStage/Projects

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 3 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Table of Contents
1 DATA INTEGRATION OVERVIEW...............................................................................................................................6
1.1 JOB SEQUENCES....................................................................................................................................................................7
1.2 JOB TYPES...........................................................................................................................................................................8
2 STANDARDS.....................................................................................................................................................................13
2.1 DIRECTORY STRUCTURES.....................................................................................................................................................13
2.2 NAMING CONVENTIONS.......................................................................................................................................................18
2.3 DOCUMENTATION AND ANNOTATION......................................................................................................................................29
2.4 WORKING WITH SOURCE CODE CONTROL SYSTEMS.................................................................................................................31
2.5 UNDERSTANDING A JOB’S ENVIRONMENT...............................................................................................................................35
3 DEVELOPMENT GUIDELINES....................................................................................................................................39
3.1 MODULAR DEVELOPMENT ................................................................................................39
3.2 ESTABLISHING JOB BOUNDARIES...........................................................................................................................................39
3.3 JOB DESIGN TEMPLATES......................................................................................................................................................40
3.4 DEFAULT JOB DESIGN.........................................................................................................................................................41
3.5 JOB PARAMETERS................................................................................................................................................................42
3.6 PARALLEL SHARED CONTAINERS...........................................................................................................................................43
3.7 ERROR AND REJECT RECORD HANDLING................................................................................................................................43
3.8 COMPONENT USAGE............................................................................................................................................................51
4 DATASTAGE DATA TYPES...........................................................................................................................................54
4.2 NULL HANDLING................................................................................................................................................................56
4.3 RUNTIME COLUMN PROPAGATION.........................................................................................................................................58
5 PARTITIONING AND COLLECTING..........................................................................................................................59
5.1 PARTITION TYPES...............................................................................................................................................................59
5.2 MONITORING PARTITIONS.....................................................................................................................................................67
5.3 PARTITION METHODOLOGY...................................................................................................................................................68
5.4 PARTITIONING EXAMPLES.....................................................................................................................................................70
5.5 COLLECTOR TYPES..............................................................................................................................................................72
5.6 COLLECTING METHODOLOGY................................................................................................................................................73
6 SORTING...........................................................................................................................................................................74
6.1 PARTITION AND SORT KEYS.................................................................................................................................................74
6.2 COMPLETE (TOTAL) SORT...................................................................................................................................................75
6.3 LINK SORT AND SORT STAGE...............................................................................................................................................76
6.4 STABLE SORT.....................................................................................................................................................................77
6.5 SUB-SORTS........................................................................................................................................................................77
6.6 AUTOMATICALLY-INSERTED SORTS........................................................................................................................................78
6.7 SORT METHODOLOGY..........................................................................................................................................................79
6.8 TUNING SORT.....................................................................................................................................................................79
7 FILE STAGE USAGE.......................................................................................................................................................81
7.1 WHICH FILE STAGE TO USE.................................................................................................................................................81
7.2 DATA SET USAGE..............................................................................................................................................................81
7.3 SEQUENTIAL FILE STAGES (IMPORT AND EXPORT)...................................................................................................................82
7.4 COMPLEX FLAT FILE STAGE.................................................................................................................................................85
8 TRANSFORMATION LANGUAGES.............................................................................................................................87

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 4 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

8.1 TRANSFORMER STAGE..........................................................................87

8.2 MODIFY STAGE...................................................................................91
9 COMBINING DATA.........................................................................................................................................................94
9.1 LOOKUP VS. JOIN VS. MERGE...............................................................................................................................................94
9.2 CAPTURING UNMATCHED RECORDS FROM A JOIN....................................................................................................................94
9.3 THE AGGREGATOR STAGE....................................................................................................................................................95
10 DATABASE STAGE GUIDELINES.............................................................................................................................96
10.1 DATABASE DEVELOPMENT OVERVIEW...................................................................................................................................96
10.2 DB2 GUIDELINES...........................................................................................................................................................103
10.3 INFORMIX DATABASE GUIDELINES.....................................................................................................................................113
10.4 ODBC ENTERPRISE GUIDELINES......................................................................................................................................114
10.5 ORACLE DATABASE GUIDELINES.......................................................................................................................................117
10.6 SYBASE ENTERPRISE GUIDELINES......................................................................................................................................119
10.7 TERADATA DATABASE GUIDELINES...................................................................................................................................120
11 TROUBLESHOOTING AND MONITORING..........................................................................................................124
11.1 WARNING ON SINGLE-NODE CONFIGURATION FILES............................................................................................................124
11.2 DEBUGGING ENVIRONMENT VARIABLES.............................................................................................................................124
11.3 HOW TO ISOLATE AND DEBUG A PARALLEL JOB..................................................................................................................125
11.4 VIEWING THE GENERATED OSH......................................................................................................................................126
11.5 INTERPRETING THE PARALLEL JOB SCORE...........................................................................................................................127
12 PERFORMANCE TUNING JOB DESIGNS..............................................................................................................129
12.1 HOW TO DESIGN A JOB FOR OPTIMAL PERFORMANCE..........................................................................................................129
12.2 UNDERSTANDING OPERATOR COMBINATION........................................................................................................................131
12.3 MINIMIZING RUNTIME PROCESSES AND RESOURCE REQUIREMENTS........................................................................................133
12.4 UNDERSTANDING BUFFERING............................................................................................................................................134
APPENDIX A: STANDARD PRACTICES SUMMARY...............................................................................................140
APPENDIX B: DATASTAGE NAMING REFERENCE...............................................................................................146
APPENDIX C: UNDERSTANDING THE PARALLEL JOB SCORE.........................................................................148
APPENDIX D: ESTIMATING THE SIZE OF A PARALLEL DATA SET................................................................153
APPENDIX E: ENVIRONMENT VARIABLE REFERENCE.....................................................................................154
APPENDIX F: SORTING AND HASHING ADVANCED EXAMPLE.......................................................................160

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 5 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1 Data Integration Overview

Work performed by Data Integration jobs fall into 4 general categories:
• Reading input data including sequential files, databases and DS/EE Data Sets;
• Performing row validation to support data quality;
• Performing transformation from data sources to data targets; and
• Provisioning data targets.
Here is the general flow diagram for Data Stage Enterprise Edition jobs:

Before Job
Subroutine

Halt on Error? Yes Exit Failure

No

Create Reject
Files (Limited) Read Input Data

Halt on Error? Yes Exit Failure

No

Create Error and Perform

Reject Files Validations

Errors and
Warnings

Halt on Error? Yes Exit Failure

No

Create Error and Perform

Reject Files Transformations

Halt on Error? Yes Exit Failure

No Over Job W arning

Yes Exit Failure
Threshold?
Perform Load
Create Reject and/or Create
Files (Limited) Intermediate
Datasets

No

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 6 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part ofJobthis publication may be reproduced, transmitted, transcribed, stored in a
After
Subroutine
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1.1 Job Sequences

As shown in the previous diagram, ETL development is intended to be modular, built from individual
Parallel jobs assembled in DataStage Enterprise Edition (“DS/EE”) controlled as modules from master
DataStage Sequence jobs, as illustrated in the example below:

These job Sequences control the interaction and error handling between individual DataStage jobs, and
together form a single end-to-end module within a DataStage application.

Job sequences also provide the recommended level of integration with external schedulers (such as
AutoSys, Cron, CA7, etc). This provides a level of granularity and control that is easy to manage and
maintain, and provides an appropriate leveraging of the respective technologies.

In most production deployments, Job Sequences require a level of integration with various production
automation technologies (scheduling, auditing/capture, error logging, etc). These topics are discussed
in Parallel Framework Standard Practices: Administration, Management, and Production Automation.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 7 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1.2 Job Types

Nearly all data integration jobs fall into three major types: Transformation, Hybrid, and Provisioning.

- Transformation jobs prepare data for provisioning jobs

- Provisioning jobs load transformed data, and
- Hybrid jobs do both.

The following table defines when each type should be used:

Type Data Requirements Example

Transformatio Data must NOT be changed by Reference tables upon which all subsequent
n any method unless jobs jobs and/or the current data target (usually a
transforming an entire subject database) will depend, or long running
area have successfully provisioning processes. This prevents partial
completed, or where the replacement of reference data in the event of
resource requirements for data transformation failure, and preserves the
transformation are very large. compute effort of long running
transformation jobs.
Hybrid Data can be changed Non-reference data or independent data are
regardless of success or candidates. The data target (usually a
failure. database) must allow subsequent processing
of error or reject rows and tolerate partial or
complete non-update of targets. Neither the
transformation nor provisioning requirements
are large.
Provisioning Data must NOT be changed by Any target where either all sources have been
any method unless jobs successfully transformed or where the
transforming an entire subject resources required to transform the data must
area have successfully be preserved in the event of a load failure or
completed, or where the where the provisioning will take so long that
resource requirements for data it increases the probability of job failure if the
provisioning are very large. job includes transformation and provisioning.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 8 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1.2.1 Transformation Jobs

In transformation jobs, data sources, some of which may be write-through cache Data Sets, are
processed to produce a load-ready Data Set that represents either the entire target table or new records
to be appended to the target table. If the entire target table is held in the load-ready Data Set, that Data
Set qualifies as write-through cache and may be used as source data instead of the target table.

The following example transformation job demonstrates the use of write-through cache DS/EE Data
Sets:

The target table is among

the inputs.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 9 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

The following example transformation job does NOT produce

write-through cache – its sources do NOT include the target
table.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 10 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1.2.2 Hybrid Jobs

The following example hybrid job demonstrates several interesting techniques that might be used in
more complex jobs. Some of the more interesting solutions in this job are circled, and described below
following the highlighted areas from Left to Right:

A column generator inserts the key column for a join and generates a single value guaranteed to never
appear in the other input(s) to the join. By specifying a full-outer join we produce a Cartesian product
dataset. In this case, we replicated the Oracle structure (lower input) for each country found in the
write-through cache country dataset (upper input).

The key column for a Referential Integrity check is validated by a Transformer stage. If the key
column is NULL, it is rejected by the transformer to a reject port and the validation is not performed
for those records. The non-validated records, the validated records, and the write-through cache
records from the last load of the target database are merged.

The merged records are grouped and ordered before being de-duplicated to remove obsolete records.

The de-duplicated records are re-grouped and ordered before calculation of the terminating keys,
producing an ordered and linked associative table.

This job also loads the target database table and creates write-through cache. In this case, if the load
fails, the cache is deleted, forcing other jobs that might depend on this data to access the existing (not
updated) target database table. This enforces a coherent view of the subject area from either cache
(current state if all jobs complete successfully) or target tables (previous state if any job fails).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 11 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1.2.3 Provisioning Jobs

This example provisioning job demonstrates the straightforward approach to simple provisioning tasks.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 12 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2 Standards
Establishing consistent development standards helps to improve developer productivity and reduce
ongoing maintenance costs. Development standards can also make it easier to integrate external
processes such as automated auditing and reporting, and to build technical and support documentation.

2.1 Directory Structures

2.1.1 Data, Install, and Project Directory Structures
The following diagrams depict the IBM WebSphere DataStage software directory structures and the
support directory structures. These directories are configured during product installation.

Install File System

Install FS Scratch File Systems

...
/Scratch0 /ScratchN
/Ascential

/Project_A /P
/patches

...
/DataStage /Project_Z /

/DSEngine

Data File Systems

/PXEngine
...
/Data0 /DataN

/Configurations
/Project_A /P
...

/Projects
Figure 1: Recommended DataStage Install, Scratch, and Data Directories
1 Gigabyte /Project_Z /

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 13 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No/Project_A
part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
File systems are
...

highlighted in blue
/Project_Z
 Information Integration Solutions
Center of Excellence

By default, the DataStage Administrator client creates its

projects (repositories) in the Projects directory of the
DataStage installation directory. In general, it is a bad practice to create DataStage projects in the
default directory, as disk space is typically limited in production install directories. For this reason,
projects should be installed in their own file system.

NOTE: On some operating systems, it is possible to create separate file systems at non-root
levels. This is illustrated in the above diagram, as a separate file system for the Projects sub
directory within the DataStage installation.

The DataStage installation creates the following two directories:

$DSHOME/../Scratch
$DSHOME/../Datasets

The DataStage Administrator should ensure that these default directories are never used by any parallel
configuration files. Scratch is used by the EE framework for temporary files such as buffer overflow,
sort memory overflow. It is a bad practice to share the DataStage project file system and conductor file
system with volatile files like scratch files and Parallel data set part files, because they increase the risk
of filling the DataStage project file systems.

To scale I/O performance within DataStage, the administrator should consider creating separate file
systems for each Scratch and Resource partition. As a standard practice,

In order to scale I/O for DataStage, consider creating separate file systems for each Scratch and Data
resource partition. Consider naming the file systems in accordance with partition numbers in your
DataStage EE Configuration file. This best practice advocates creating subdirectories for each project
for each scratch and disk partition.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 14 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

/Staging

/dev /si /qa /prod

/Project_A /Project_A /Project_A /Project_A

/archive /archive /archive /archive

...

...
...

...

/Project_Z
Figure 2: DataStage Staging Directories
/Project_Z /Project_Z /Project_Z

Within the separate Staging file system, data directories are implemented for each deployment phase of
a job (development, system integration, qa, and production) as appropriate. If the file system is not
/archive
shared across multiple servers,/archive /archive
not all of these development /archive
phases may be present on a local file
system. Within each deployment directory, files are separated by Project name as shown below.

/Staging Top-Level Directory

/dev development data tree, location of source data files, target data files, error and reject
files.

/Project_A subdirectory created for each project

/archive location of compressed archives created by archive process of previously processed files

/si System Integration (also known as “test”) data tree

/qa Quality Assurance data tree

/prod Production data tree

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 15 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2.1.2 Extending the DataStage Project for External

Entities
It is quite common for a DataStage application to be integrated with external entities, such as the
operating system, another application or middle ware. The integration can be as simple as a file system
for housing source files, or it could require scripts for example integrating with an Enterprise
Scheduler.

To completely integrate all aspects of a DataStage application the directory structure that is used for
integration with external entities should be defined in a way that provides a complete and separate
structure in the same spirit as a DataStage project. A directory structure should be created that
organizes external entities and is directly associated with 1 and only 1 DataStage project. This will
provide a convenient vehicle to group and manage resources used by a project. The directory structure
will be made transparent to the DataStage application, through the use of environment variables.
Environment variables are a critical portability tool, which will enable DataStage applications to move
through the life cycle without any code changes.

Project_Plus Project_Plus Directory Hierar

/dev /si /qa /prod

/Project_A /Project_A /Project_A /Project_A

/bin /bin /bin /bin

/src /src /src /s

/doc /doc /doc /d

/datasets /datasets /datasets /datase

/logs /logs /logs /logs

/params /params /params /param

/schemas /schemas /schemas /schema

/scripts /scripts /scripts /scripts

/sql Figure 3: Project_Plus/sql

Directory Structure /sql /sql
...

...

...

...

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 16 of 179
© 2006 IBM Information Integration Solutions. All rights reserved./Project_Z
/Project_Z No part of this publication may be reproduced, transmitted, transcribed, stored in a
/Project_Z /Project_Z
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Within the Project_Plus hierarchy, directories are created for

each deployment phase of a job (development, system integration, qa, and production) as appropriate.
If the file system is not shared across multiple servers, not all of these development phases may be
present on a local file system.

Project_Plus Top-Level of Directory Hierarchy

/dev development code tree
/Project_A subdirectory created for each project

/bin location of custom programs, DataStage routines, BuildOps, utilities, and shells
/doc location of documentation for programs found in /bin subdirectory
/src location of source code and makefiles for items found in /bin subdirectory (Note:
depending on change management policies, this directory may only be present in
the /dev development code tree)

/datasets location of DataSet header files (.ds file)

/logs location of custom job logs and reports

/params location of parameter files for automated program control, a copy of dsenv and copies of
DSParams.$ProjectName project files

/schemas location of Orchestrate schema files

/scripts location of operating system (shell) script files

/sql location of maintenance or template SQL

/si system integration (aka “test”) code tree

/qa quality assurance code tree

/prod production code tree

In support of a Project_Plus directory structure environment variable parameters should be configured,

for example the following diagram shows Project_Plus variables as defined in the DataStage
Administrator.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 17 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 4: Project_Plus Environment Variables

In some implementations, there may be external entities that are shared with other DataStage projects,
for example all jobs are invoked with the same Script. A similar directory structure to the Project_Plus
structure could be configured and referred to as DataStage_Plus.

2.2 Naming Conventions

As a graphical development environment, DataStage offers (within certain restrictions) flexibility to
developers when naming various objects and components used to build a data flow. By default, the
Designer tool assigns default names based on the object type, and the order the item is placed on the
design canvas. While the default names may create a functional data flow, they do not facilitate ease of
maintenance over time, nor do they adequately document the business rules or subject areas.

A consistent naming standard is essential to

• maximize the speed of development
• minimize the effort and cost of downstream maintenance
• enable consistency across multiple teams and projects
• facilitate concurrent development
• maximize the quality of the developed application
• increase the readability of the objects in the visual display medium
• increase the understanding of components when seen in external systems, for example in
WebSphere MetaStage, or an XML extract

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 18 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

This section presents a set of standards and guidelines to

apply to developing data integration applications using
DataStage Enterprise Edition.

Any set of standards needs to take on the culture of an organization, to be tuned according to needs, so
it is envisaged that these standards will develop and will adapt over time to suit both the organization
and the purpose.

There are a number of benefits from using a graphical development tool like DataStage, and many of
these benefits were used to establish this naming standard:
• With rapid development, more effort can be put into analysis and design, enabling a greater
understanding of the requirements and greater control over how they are delivered.
• There can be a much tighter link between design and development.
• Since much of the development work is done using a click, drag and drop paradigm there is less
typing involved hence the opportunity to use longer more meaningful, more readable names,
while maintaining quality.

Throughout this section, the term “Standard” refers to those principles that are required, while the term
“Guideline” refers to recommended, but not required, principles.

2.2.1 Key Attributes of the Naming Convention

This naming convention is based on a three-part convention:
Subject, Subject Modifier, and Class Word

In the context of DataStage, the class word is used to identify either a type of object or the function that
a particular type of object will perform. In some cases where appropriate, objects can be sub-typed (for
example, a Left Outer Join). In these cases the class word represents the subtype.

For example, in the case of a link object, the class word refers to the functions of Reading, Reference
(Lookup), Moving or Writing data (or within a Sequence Job, the moving of a message).

In the case of a data store the class word will refer to the type of data store, for example: Data Set,
Sequential File, Table, View, and so forth.

Where there is no sub classification required then the class word will simply refer to the object. As an
example, a transformer might be named: Data_Block_Split_Tfm

As a guideline, the Class Word is represented as a two, three or four letter abbreviation. Where it is a
three or four letter abbreviation then it should be word capitalized. Where it is a two letter abbreviation
both letters should be capitalized.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 19 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

A list of frequently-used Class Word abbreviations

is provided in 12.4.2 DataStage Naming Reference.

One benefit of using the Subject, Subject Modifier, Class Word approach, over using the Prefix
approach, is to enable two levels of sorting or grouping. In WebSphere MetaStage, the object type is
defined in a separate field- there is a field that denotes whether the object is a column, a derivation, a
link, a stage, a job design, and so forth. This is the same or similar information that would be carried in
a prefix approach. Carrying this information as a separate attributes enables the first word of the name
to be used as the subject matter, allowing sort either by subject matter or by object type. Secondly the
class word approach enables sub-classification by object type to provide additional information.

For the purposes of documentation, all word abbreviations should be referenced by the long form to get
used to saying the name in full even if reading the abbreviation. Like a logical name, however, when
creating the object, the abbreviated form is used. This will help re-enforce wider understanding of the
subjects.

The key issue is readability. Though DataStage imposes some limitations on the type of characters and
length of various object names, the standard, where possible, will be to separate words by an
Underscore which will allow clear identification of each work in a name. This should be enhanced by
also using Word Capitalization, for example, the first letter of each Word should be capitalized.

2.2.2 Designer Object Layout

The effective use of naming conventions means that objects need to be spaced appropriately on the
DataStage Designer canvas. For stages with multiple links, expanding the icon border can significantly
improve readability. This type of approach takes extra effort at first, so a pattern of work needs to be
identified and adopted to help development. The “Snap to Grid” feature of Designer can help improve
development speed.

When development is more or less complete, attention should be given to the layout to enhance
readability before it is handed over to versioning.

Where possible, consideration should be made to provide DataStage developers with higher resolution
screens as this provides them with more screen display real-estate. This can help make them more
productive and makes their work more easily read.

2.2.3 Documentation and Metadata Capture

One of the major problems with any development effort, whatever tool you use, is maintaining
documentation. Though best intentions are always apparent, documentation is often something that is
left until later, inadequately carried out.

DataStage provides the ability to document during development with the use of meaningful naming
standards (as outlined in this section). Establishing standards also eases use of external tools and

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 20 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

processes such as WebSphere MetaStage, which can provide

impact analysis, as well as documentation and auditing.

2.2.4 Naming Conventions by Object Type

2.2.4.1 Projects
Each DataStage Project is a standalone repository. It may or may not have a one to one relationship
with an organizations’ project of work. This factor often can cause terminology issues especially in
teamwork where both business and developers are involved. The suffix of a Project name should be
used to identify Development (“Dev”), Test (“Test”), and Production (“Prod”).

The name of a DataStage Project may only be 18 characters in length, it can contain alpha-numeric
characters and it can contain underscores. However with the limit of 18 characters the name is most
often composed of abbreviations.

Examples of Project naming where the project is single application focused are:
• “Accounting Engine NAB Development” would be named: Acct_Eng_NAB_Dev
• “Accounting Engine NAB Production” would be named: Acct_Eng_NAB_Prod

Examples of Project naming where the project is multi-application focused are:

• Accounting Engine Development or Acct_Engine_Dev
• Accounting Engine Production or Acct_Engine_Prod

2.2.4.2 Category Hierarchy

DataStage organizes objects in its repository by Categories, allowing related objects to be grouped
together. Category Names can be long, are Alpha Numeric and can also contain both Spaces and
Underscores. Therefore Directory names should be Word Capitalized and separated by either an
underscore or a space.

DataStage enforces the top level Directory Structure for different types of Objects (for example, Jobs,
Routines, Shared Containers, Table definitions…). Below this level, developers have the flexibility to
define their own Directory or Category hierarchy.

2.2.4.3 Job Category Naming

Within Designer, dialog box fields that specify a new category have only one input area for defining
the Category name. Multiple levels of Hierarchy are named by specifying the Hierarchy levels
separated by a backslash (“\”). For example, the structure “A Test\Lower\Lower Still” is shown below:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 21 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 5: Creating Category Hierarchies

The main reason for having Categories is to group related objects. Where possible, a Category level
should only contain objects that are directly related. For example, a job category might contain a Job
Sequence and all the jobs and only those jobs that are contained in that sequence.

Organizing related DataStage objects within categories also facilitates backup/export/import/change

control strategies for projects since Manager can import/export objects by category grouping.

Categorization by Functional Module

For a given application, all Jobs and Job Sequences will be grouped in a single parent Category, with
sub-levels for individual functional modules. Note that Job names must be unique within a DataStage
project, not within a category.
Within each functional module category, Jobs and Job Sequences are grouped together in the same
scope as the technical design documents. For example, jobs that read write-through cache for a ECRP
subset in the ECRDEV project that cleanse and load multi-family mortgage data and are driven by a
sequencer might have a hierarchy that looks like the following example:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 22 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 6: Categorization by Functional Module

Categorization by Developer
In development projects, categories will be created for each developer as their personal sandbox and
place they perform unit test activities on jobs they are developing. It is the responsibility of each
developer to delete unused or obsolete code, and the responsibility of the development manager
assigned the DataStage Manager role to ensure that projects are not obese with unused jobs, categories
and metadata.

Remembering that Job names must be unique within a given project, two developers cannot save a
copy of the same job with the same name within their individual “sandbox” categories – a unique Job
name must be given.

In the previous illustration, project manager, two developers have private categories for sandbox and
development activities, and there are 2 additional high-level categories, ECRP and Templates.

2.2.4.4 Table Definition Categories

Unlike other types of DataStage objects, Table Definitions are always categorized using two level
names. By default, DataStage assigns the level names based on the source of the metadata import (for
example, Orchestrate, PlugIn, Saved, etc...), but this can be overridden during import.

Although the default table definition categories are useful from a functional perspective, establishing a
Table Definition categorization that matches project development organization is recommended.

New Table Definition categories can be created within the repository by right-clicking within the Table
Definitions area of the DataStage project repository and choosing the “New Category” command.

When implementing a customized Table Definition categorization, care must be taken to override the
default choices for category names during Table
Definition import. On import, the first level Table
Definition category is identified as the “Data Source
Type” and the second level categorization is referred to as
the “Data Source Name” as shown in the example on the
below. The placement of these fields varies with the
method of metadata import.

Temporary TableDefs created by developers to assist with

job creation appear under the Saved category by default.
Once created, if these TableDefs are to be used by other
jobs, they must be moved to the appropriate category and
re-imported from that category in every job where they
are used. TableDefs that remain in the Saved category

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 23 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

should be deleted as soon as possible. In this example, the

TableDefs have been grouped into a master category of
Custom, with sub-categories intended to identify the type of the source, e.g.: Datasets. Each subject
area will have a master category, e.g: DWPH1 or ECRP.

The following is one of the TableDefs from this project showing how to correctly specify the category
and sub-category.

An alternative implementation is to set the

“Data source name” to that of the source
system or schema.

Figure 7: Table Definition Categories

2.2.4.5 Jobs and Job Sequences

Job names must begin with a letter and can contain letters, numbers, and underscores only. Because the
name of can be long, Job and Job Sequence names should be descriptive and should use word
capitalization to make them readable.

Jobs and Job Sequences are all held under the Category Directory Structure of which the top level is
the category “Jobs”.

A Job will be suffixed with the class word “Job” and a Job Sequence will be suffixed with the class
word “Seq”.

Examples of Job naming are:

• CodeBlockAggregationJob
• CodeBlockProcessingSeq

Jobs should be organized under Category Directories to provide grouping such that a Directory should
contain a Sequence Job and all the Jobs that are contained within that sequence. This will be discussed
further in Section 2.2.4.2 Category Hierarchy.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 24 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2.2.4.6 Shared Containers

Shared containers have the same naming constraints as jobs in that the name can be long but can not
contain underscores, so word capitalization should be used for readability. Shared containers have their
own Category Directory and consideration should be given to a meaningful Directory Hierarchy. When
a Shared Container is used, a character code is automatically added to that instance of its use
throughout the project. It is optional as to whether you decide to change this code to something
meaningful.

To differentiate between Parallel Shared Containers and Server Shared Containers, the following Class
Word naming is recommended:
• Psc = Parallel (Enterprise Edition) Shared Container
• Ssc = Server Edition Shared Container

IMPORTANT: Use of Server Edition Shared Containers is discouraged within a parallel job.

Examples of Shared Container naming are:

• AuditTrailPsc (this is the original as seen in the Category Directory)
• AuditTrailPscC1 (This is an instance of use of the above shared container)
• AuditTrailPscC2 (This is another instance of use of the same shared container)
In the above examples the characters “C1” and the “C2” are automatically applied to the Shared
Container Stage by DataStage Designer when dragged onto the design canvas.

2.2.4.7 Parameters
A Parameter can be a long name consisting of alphanumeric characters and underscores. Therefore the
parameter name must be made readable using Capitalized words separated by underscores. The class
word suffix is “Parm”.

Examples of Parameter naming are:

• Audit_Trail_Output_Path_Parm
• Note where this is used in a stage property, the parameter name is delimited by the # sign:
#Audit_Trail_Output_Path_Parm#

2.2.4.8 Links
Within a DataStage Job, links are objects that represent the flow of data from one stage to the next.
Within a Job Sequence, links represent the flow of a message from one activity / step to the next.

It is particularly important to establish a consistent naming convention for link names, instead of using
the default “DSLink#” (where “#” is an assigned number). Within the graphical Designer environment,
stage editors identify links by name; having a descriptive link name reduces the chance for errors (for
example, during Link Ordering). Furthermore, when sharing data with external applications (for

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 25 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

example, through Job reporting), establishing standardized

link names makes it easier to understand results and audit
counts.

The following rules can be used to establish a link name:

• Use the prefix “lnk_” before the subject name to differentiate with stage objects
• The link name should define the subject of the data that is being moved
• For non-stream links, the link name should include the link type (reference, reject) to reinforce
the visual cues of the Designer canvas:
o “Ref” for reference links (Lookup)
o “Rej” for reject links (Lookup, Merge, Transformer, Sequential File, Database, etc)
• The type of movement may optionally be part of the Class Word, for example:
o “In” for input
o “Out” for output
o “Upd” for updates
o “Ins” for inserts
o “Del” for deletes
o “Get” for shared container inputs
o “Put” for shared container output
• As data is enriched through stages, the same name may be appropriate for multiple links. In this
case, always specify a unique link name within a particular Job or Job Sequence by including a
number. (The DataStage Designer does not require link names on different stages to be unique.)

Examples Link names:

• Input Transactions: “lnk_Txn_In”
• Reference Account Numbers: “lnk_Account_Ref”
• Customer File Rejects: “lnk_Customer_Rej”
• Reception Succeeded Message or “lnk_Reception_Succeeded_Msg”

2.2.4.9 Stage Names

DataStage assigns default names to stages as they are dragged onto the Designer canvas. These names
are based on the type of stage (object) and a unique number, based on the order the object was added to
the flow. Within a Job or Job Sequence, stage names must be unique.

Instead of using the full object name, a 2, 3, or 4 character abbreviation should be used for the Class
Word suffix, after the subject name and subject modifier. A list of frequently-used stages and their
corresponding Class Word abbreviation may be found in 12.4.2 DataStage Naming Reference.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 26 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2.2.4.10 Data Stores

For the purposes of this section, a data store is a physical piece of disk storage where data is held for
some period of time. In DataStage terms, this can be either a table in a database structure or a file
contained within a disk directory or catalog structure. Data held in a database structure is referred to as
either a Table or a View. In data warehousing, two additional subclasses of table might be used:
Dimension and Fact. Data held in a file in a directory structure will be classified according to its type,
for example: Sequential File, Parallel Data Set, Lookup File Set, etc.

The concept of source and target can be applied in a couple of ways. Every job in a series of jobs could
consider the data it gets in to be a source and the data it writes out as being a target. However for the
sake of this naming convention a Source will only be data that is extracted from an original system and
Target will be the data structures that are produced or loaded as the final result of a particular series of
jobs. This is based on the purpose of the project – to move some data from a source to a target.

Data Stores used as temporary structures to land data between jobs, supporting restart and modularity,
should use the same names in the originating job and any downstream jobs reading the structure.

Examples of Data Store naming are:

• Transaction Header Sequential File or Txn_Header_SF
• Customer Dimension or Cust_Dim (This optionally could be further qualified as Cust_Dim_Tgt
if you wish to qualify it as a final target)
• Customer Table or Cust_Tab
• General Ledger Account Number View or GL_Acctno_View

2.2.4.11 Transformer Stage and Stage Variables

A Transformer Stage name can be long – over 50 characters and can contain underscores. Therefore the
name can be descriptive and readable through word capitalization and underscores. DataStage
Enterprise Edition supports two types of Transformers:
• “Tfm”: Parallel (Enterprise Edition) Transformer
• “BTfm”: BASIC (Server Edition) Transformer

IMPORTANT: For maximum performance and scalability, BASIC Transformers

should be avoided in Enterprise Edition data flows.

A Transformer Stage Variable can have a long name consisting of alphanumeric characters but not
underscores. Therefore the Stage Variable name must be made readable only by using Capitalized
words. The Class Word suffix is Stage Variable or “SV”. Stage Variables should be named according
to their purpose.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 27 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

When developing Transformer derivation expressions, it

is
important to remember Stage variable names are case sensitive.

2.2.4.12 DataStage Routines

DataStage BASIC routine names will indicate their function and they will be grouped in sub-categories
by function under a main category of Custom, for example.:
Routines/Custom/SetDSParamsFromFile.

A How-To document describing the appropriate use of the routine must be provided by the author of
the routine, and placed in a documentation repository.

DataStage Custom Transformer routine names will indicate their function and they will be grouped in
sub-categories by function under a main category of Custom, for example:
Routines/Custom/DetectTeradataUnicode.

Source code, a makefile, and the resulting object for each Custom Transformer routine must be placed
in the project phase source directory, e.g.: /home/dsadm/dev/bin/source.

2.2.4.13 File Names

Source file names should include the name of the source database or system and the source table name
or copybook name. The goal is to connect the name of the file with the name of the storage object on
the source system. Source flat files will have a unique serial number composed of the date, “_ETL_”
and time, for example:
Client_Relationship_File1_In_20060104_ETL_184325.psv.

Intermediate datasets are created between modules. Their names will include the name of the module
that created the dataset OR the contents of the dataset in that more than one module may use the dataset
after it is written, for example:
BUSN_RCR_CUST.ds

Target output files will include the name of the target database or system, the target table name or
copybook name. The goal is the same as with source files – to connect the name of the file with the
name of the file on the target system. Target flat files will have a unique serial number composed of
the date, “_ETL_” and time, for example:
Client_Relationship_File1_Out_20060104_ETL_184325.psv

Files and datasets will have suffixes that allow easy identification of the content and type. DataStage
proprietary format files have required suffixes and are identified in italics in the table below which
defines the types of files and their suffixes.

File Type Suffix

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 28 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Flat delimited and non-delimited files

.dat.
Flat pipe (|) delimited files .psv
Flat comma-and-quote delimited files .csv.
DataStage datasets .ds.
DataStage filesets .fs
DataStage hash files .hash.
Orchestrate schema files .schema.
Flat delimited or non-delimited REJECT files .rej.
DataStage REJECT datasets _rej.ds.
Flat delimited or non-delimited ERROR files .err.
DataStage ERROR datasets _err.ds.
Flat delimited or non-delimited LOG files .log.

2.3 Documentation and Annotation

DataStage Designer provides description fields for each object type. These fields allow the developer to
provide additional descriptions that can be captured and used by administrators and other developers.

The Short Description field is also displayed on summary lines within the Director and Manager
clients. At a minimum, description annotations must be provided in the Job Properties Short
Description field for each job and job sequence, as shown below:

Figure 8: Job Level Short Description

Within a job, the Annotation tool should be used to highlight steps in a given job flow. Note that by
changing the vertical alignment properties (for example, Bottom) the annotation can be drawn around
the referenced stage(s), as shown in the following example.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 29 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 9: Example Job Annotation

DataStage also allows descriptions to be attached to each stage within the General tab of the stage
properties.

Each stage should have a short description of its function specified within the stage properties. These
descriptions will appear in the job documentation automatically generated from jobs and sequencers
adhering to the standards in this document. More complex operators or operations should have
correspondingly longer and more complex explanations on this tab.

Examples of such annotations include:

Job “short” description:
This Job takes the data from GBL Oracle Table AD_TYP and does a truncate load into
Teradata Table AD_TYP.

ODBC Enterprise stage read:

Read the GLO.RcR_GLOBAL_BUSN_CAT_TYP table from jpORACLE_SERVER using the
ODBC driver. There are no selection criteria in the WHERE clause.

Oracle Enterprise stage read:

Read the GLOBAL.GLOBAL_REST_CHAR table from jpORACLE_SERVER using the Oracle
Enterprise operator. There are no selection criteria in the WHERE clause.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 30 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Remove Duplicates stage

This stage removes all but one record with duplicate BUSN_OWN_TYP_ID keys.

Lookup stage
This stage validates the input and writes rejects.
This stage validates the input and continues.
This stage identifies changes and drops records not matched (not updated).

Copy stage
This stage sends data to the TDMLoadPX stage for loading into Teradata, and to a dataset for
use as write-through cache.
This stage renames and/or drops columns and is NOT optimized out.
This stage is cosmetic and is optimized out.

Sequential file stage:

This is the source file for the LANG table.
This is the target file for business qualification process rejects.

Transformer stage:
This stage generates sequence numbers that have a less-than file scope.
This stage converts null dates.

Modify stage:
This stage performs data conversions not requiring a transformer.

Teradata MultiLoad stage:

Load the RcR_GLOBAL_LCAT_TYP table.

Data Set stage:

This stage writes the GLOBAL_Ad_Typ dataset which is used as write-through cache to avoid
the use of Teradata in subsequent jobs.
This stage reads the GLOBAL_Lcat dataset, which is used as write-through cache to avoid the
use of Teradata.

2.4 Working with Source Code Control Systems

DataStage’s built-in repository manages objects (jobs, sequences, table definitions, routines, custom
components) during job development. However, this repository is not capable of managing non-
DataStage components (for example, UNIX shell scripts, environment files, job scheduler
configurations, etc.) that may be part of a completed application.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 31 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Source code control systems (such as ClearCase, PVCS,

SCCS) are useful for managing the development lifecycle of
all components of an application, organized into specific releases for version control.

DataStage does not directly integrate with source code control systems, but it does offer the ability to
exchange information with these systems. It is the responsibility of the DataStage developer to
maintain DataStage objects within the source code system.

The Manager client is the primary interface to the DataStage object repository. Using Manager, you
can export objects (job designs, table definitions, custom stage types, user-defined routines, etc.) from
the repository as clear-text format files. These files can then be checked into the external source code
control system.

The export file format for DataStage objects can be either .DSX (DataStage eXport format) or .XML.
Both formats contain the same information, although the XML file is generally much larger. Unless
there is a need to parse information in the export file, .DSX is the recommended export format.

2.4.1 Source Code Control Standards

The first step to effective integration with source code control systems is to establish standards and
rules for managing this process:

a) Establish Category naming and organization standard

DataStage objects can be exported individually or by category (folder hierarchy). Grouping
related objects by folder can simplify the process of exchanging information with the external
source code control system. This object grouping also helps establish a manageable “middle
ground” between an entire project exports and individual object exports.

b) Define rules for exchange with source code control

As a graphical development environment, Designer facilitates iterative job design. It would be
cumbersome to require the developer to check-in every change to a DataStage object in the
external source code control system. Rather, rules should be defined for when this transfer
should take place. Typically, milestone points on the development lifecycle are a good point for
transferring objects to the source code control system - for example, when a set of objects has
completed initial development, unit test, and so on.

c) Don’t rely on the source code control system for backups

Because the rules defined for transfer to the source code control system will typically be only at
milestones in the development cycle, they would not be an effective backup strategy.
Furthermore, operating system backups of the project repository files only establish a “point in
time”, and cannot be used to restore individual objects.

For these reasons, it is important that an identified individual maintains backup copies of the
important job designs using .DSX file exports to a local or (preferably) shared file system.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 32 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

These backups can be done on a scheduled basis by an

Operations support group, or by the individual
DataStage developer. In either case, the developer should create a local backup prior to
implementing any extensive changes.

2.4.2 Using Object Categorization Standards

As discussed in Section 2.2.4.2 Category Hierarchy, establishing and following a consistent naming
and categorization standard is essential to the change management process. The DataStage Manager
can export at the Project, Category, and individual Object levels. Assigning related objects to the same
category provides a balanced level of granularity when exporting and importing objects with external
source code control systems.

2.4.3 Export to Source Code Control System

The process of exporting DataStage objects to a source code control system is a straightforward
process. It can be done interactively by the developer or project manager using the Manager client, as
explained in this section.

The DataStage client includes Windows command-line utilities for automating the export process.
These utilities (dsexport and dscmdexport) are documented in the DataStage Manager Guide.

All exports from the DataStage repository are performed on the Windows workstation. There is
no server-side project export facility.

• Select the object or category in the Manager browser.

Figure 10: Manager Category browser

• Choose “Export DataStage Components” from the “Export” menu.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 33 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

NOTE: Objects cannot be exported from DataStage if they are open in Designer.
Make sure all objects are saved and closed before exporting.

Figure 11: Manager Export Options

To export a group of objects to a single export file, the option “Selection: By category”
should be specified in the “Options” tab.

The filename for export is specified in the “Export to file:” field at the top of the Export
dialog.

If you wish to include compiled Transformer objects for a selected job, make sure the
“Job Executables” category is checked.

• Using your source code control utilities, check-in the exported .DSX file

2.4.4 Import from Source Code Control System

In a similar manner, the import of objects from an external source code control system is a
straightforward process. Import can be interactive through the Manager client (as described in this
section), or automated through command-line utilities.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 34 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Unlike the export process, command-line import utilities are

available for both Windows workstation and DataStage server
platforms. The Windows workstation utilities (dsimport and dscmdimport) are documented in the
DataStage Manager Guide.

For test and production environments, it is possible to import the job executables from the DataStage
server host using the dsjob command-line, as documented in the DataStage Development Kit chapter of
the Parallel Job Advanced Developer’s Guide. Note that using dsjob will only import job executables -
job designs can only be imported using the Manager client or the dsimport or dscmdimport client tools.

• Use the source code control system to check-out (or export) the .DSX file to your client
workstation.

• Import objects in the .DSX file using Manager. Choose “Import DataStage Components” from
the “Import” menu. Select the file you checked out of your source code control system by
clicking on the ellipsis (“…”) next to the filename field in the import dialog. After selecting
your file, click OK to import.

Figure 12: Manager Import options

• The import of the .DSX file will place the object in the same DataStage category it originated
from. This means that if necessary it will create the Job Category if it doesn't already exits.

• If the objects were not exported with the “Job Executables”, then compile the imported objects
from Designer, or using the Multi-Job Compile tool.

2.5 Understanding a Job’s Environment

DataStage Enterprise Edition provides a number of environment variables to enable / disable product
features and to fine-tune job performance.

Although operating system environment variables can be set in multiple places, there is a defined order
of precedence that is evaluated when a job’s actual environment is established at runtime:

1) The daemon for managing client connections to the DataStage server engine is called dsrpcd.
By default (in a root installation), dsrpcd is started when the server installed, and should start

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 35 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

whenever the machine is restarted. dsrpcd can also be

manually started and stopped using the $DSHOME/uv –
admin command. (For more information, see the DataStage Administrator Guide.)

By default, DataStage jobs inherit the dsrpcd environment, which, on UNIX platforms is set in
the etc/profile and $DSHOME/dsenv scripts. On Windows, the default DataStage environment is
defined in the registry. Note that client connections DO NOT pick up per-user environment
settings from their $HOME/.profile script.

On USS environments, the dsrpc environment is not inherited since DataStage jobs do not
execute on the conductor node.

2) Environment variable settings for particular projects can be set in the DataStage Administrator
client. Any project-level settings for a specific environment variable will override any settings
inherited from dsrpcd.

IMPORTANT: When migrating projects between machines or environments, it is important to

note that project-level environment variable settings are not exported when a project is
exported. These settings are stored in a file named DSPARAMS in the project directory.

Any project-level environment variables must be set for new projects using the Administrator
client, or by carefully editing the DSPARAMS file within the project. Refer to the DataStage
Administration, Management, and Production Automation Best Practice for additional details.

3) Within Designer, environment variables may be defined for a particular job using the Job
Properties dialog box. Any job-level settings for a specific environment variable will override
any settings inherited from dsrpcd or from project-level defaults.

To avoid hard-coding default values for job parameters, there are three special values that can be used
for environment variables within job parameters:

• $ENV causes the value of the named environment variable to be retrieved from the operating
system of the job environment. Typically this is used to pickup values set in the operating
system outside of DataStage.

NOTE: $ENV should not be used for specifying the default $APT_CONFIG_FILE value because,
during job development, the Designer parses the corresponding parallel configuration file to
obtain a list of node maps and constraints (advanced stage properties).

• $PROJDEF causes the project default value for the environment variable (as shown on the
Administrator client) to be picked up and used to set the environment variable and job
parameter for the job.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 36 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

• $UNSET causes the environment variable to be

removed completely from the runtime environment.
Several environment variables are evaluated only for their presence in the environment (for
example, APT_SORT_INSERTION_CHECK_ONLY).

2.5.1 Environment Variable Settings

An extensive list of environment variables is documented in the DataStage Parallel Job Advanced
Developer’s Guide. This section is intended to call attention to some specific environment variables,
and to document a few that are not part of the documentation.

2.5.1.1 Environment Variable Settings for All Jobs

IBM recommends the following environment variable settings for all DataStage Enterprise Edition
jobs. These settings can be made at the project level, or may be set on an individual basis within the
properties for each job. It may be helpful to create a Job Template and include these environment
variables in the parameter settings.

Environment Variable Setting Description

$APT_CONFIG_FILE filepath Specifies the full pathname to the EE configuration file.
This variable should be included in all job parameters
so that it can be easily changed at runtime.
$APT_DUMP_SCORE 1 Outputs EE score dump to the DataStage job log,
providing detailed information about actual job flow
including operators, processes, and Data Sets.
Extremely useful for understanding how a job actually
ran in the environment. (see 12.4.2 Understanding the
Parallel Job Score)
$OSH_ECHO 1 Includes a copy of the generated osh in the job’s
DataStage log
$APT_RECORD_COUNTS 0 Outputs record counts to the DataStage job log as each
operator completes processing. The count is per
operator per partition.

This setting should be disabled by default, but part of

every job design so that it can be easily enabled for
debugging purposes.
$OSH_PRINT_SCHEMA 0 Outputs actual runtime metadata (schema) to DataStage
job log.

This setting should be disabled by default, but part of

every job design so that it can be easily enabled for
debugging purposes.
$APT_PM_SHOW_PIDS 1 Places entries in DataStage job log showing UNIX
process ID (PID) for each process started by a job.
Does not report PIDs of DataStage “phantom”
processes started by Server shared containers.
$APT_BUFFER_MAXIMUM_TIMEOUT 1 Maximum buffer delay in seconds

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 37 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

On Solaris platforms only:

When working with very large parallel Data Sets (where
the individual data segment files are larger than 2GB), you must define the environment variable
$APT_IO_NOMAP

On Tru64 5.1A platforms only:

On Tru64 platforms, the environment variable $APT_PM_NO_SHARED_MEMORY should be set to 1 to
work around a performance issue with shared memory MMAP operations. This setting instructs
EE to use named pipes rather than shared memory for local data transport.

2.5.1.2 Additional Environment Variable Settings

Throughout this document, a number of environment variables will be mentioned for tuning the
performance of a particular job flow, assisting in debugging, or changing the default behavior of
specific Enterprise Edition stages. The environment variables mentioned in this document are
summarized in Appendix D: Environment Variable Reference. An extensive list of environment
variables is documented in the DataStage Parallel Job Advanced Developer’s Guide.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 38 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

3 Development Guidelines
3.1 Modular Development
Modular development techniques should be used to maximize re-use of DataStage jobs and
components:

• Job parameterization allows a single job design to process similar logic instead of creating
multiple copies of the same job. The Multiple-Instance job property allows multiple invocations
of the same job to run simultaneously.

• A set of standard job parameters should be used in DataStage jobs for source and target
database parameters (DSN, user, password, etc) and directories where files are stored. To ease
re-use, these standard parameters and settings should be made part of a Designer Job Template.

• Create a standard directory structure outside of the DataStage project directory for source and
target files, intermediate work files, and so forth.

• Where possible, create re-usable components such as parallel shared containers to encapsulate
frequently-used logic.

3.2 Establishing Job Boundaries

It is important to establish appropriate job boundaries when developing with DS/EE. In some cases,
functional requirements may dictate job boundaries. For example, it may be appropriate to update all
dimension values before inserting new entries in a data warehousing fact table. But functional
requirements may not be the only factor driving the size of a given DataStage job.

While it may be possible to construct a large, complex job that satisfies given functional requirements,
this may not be appropriate. Factors to consider when establishing job boundaries include:

 Establishing job boundaries through intermediate Data Sets creates “checkpoints” that can be
used in the event of a failure when processing must be restarted. Without these checkpoints,
processing must be restarted from the beginning of the job flow. It is for these reasons that
long-running tasks are often segmented into separate jobs in an overall sequence.
o For example, if the extract of source data takes a long time (such as an FTP transfer over
a wide area network) it would be good to land the extracted source data to a parallel data
set before processing.
o As another example, it is generally a good idea to land data to a parallel Data Set before
loading to a target database unless the data volume is small or the overall time to
process the data is minimal.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 39 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

 Larger, more complex jobs require more system

resources (CPU, memory, swap) than a series of
smaller jobs, sequenced together through intermediate Data Sets. Resource requirements are
further increased when running with a greater degree of parallelism specified by a given
configuration file. However, the sequence of smaller jobs generally requires more disk space to
hold intermediate data, and the speed of the I/O subsystem can impact overall end-to-end
throughput.

Section 12.3: Minimizing Runtime Processes and Resource Requirements provides some
recommendations for minimizing resource requirements of a given job design, especially when
the volume of data does not dictate parallel processing.

 Breaking large job flows into smaller jobs may further facilitate modular development and re-
use if business requirements for more than one process depend on intermediate data created by
an earlier job.

 The size of a job directly impacts the speed of development tasks such as opening, saving, and
compiling. These factors may be amplified when developing across a wide-area or high-latency
network connection. In extreme circumstances this can significantly impact developer
productivity and ongoing maintenance costs.

 The startup time of a given job is directly related to the number of stages and links in the job
flow. Larger more complex jobs require more time to startup before actual data processing can
begin. Job startup time is further impacted by the degree of parallelism specified by the parallel
configuration file.

 Remember that the number of stages in a parallel job includes the number of stages within each
shared container used in a particular job flow.

As a rule of thumb, keeping job designs to less than 50 stages may be a good starting point. But this is
not a hard-and-fast rule. The proper job boundaries are ultimately dictated by functional / restart /
performance requirements, expected throughput and data volumes, degree of parallelism, number of
simultaneous jobs and their corresponding complexity, and the capacity and capabilities of the target
hardware environment.

Combining or splitting jobs is relatively easy, so don't be afraid to experiment and see what
works best for your jobs in your environment.

3.3 Job Design Templates

DataStage Designer provides the developer with re-usable Job Templates, which can be created from
an existing Parallel Job or Job Sequence using the “New Template from Job” command.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 40 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Template jobs should be created with:

- standard parameters (for example, source and target
file paths, database login properties…)
- environment variables and their default settings
(as outlined in Section 2.5.1 Environment Variable Settings)
- annotation blocks

In addition, template jobs may contain any number of stages and pre-built logic, allowing multiple
templates to be created for different types of “standardized” processing.

By default, the Designer client stores all job templates in the local “Templates” directory within the
DataStage client install directory, for example, C:\Program Files\Ascential\DataStage751\Templates

To facilitate greater re-use of job templates, especially in a team-based development, the template
directory can be changed using the Windows Registry Editor.

This change must be made on each client workstation, by altering the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Ascential Software\DataStage Client\CurrentVersion\Intelligent Assistant\Templates

3.4 Default Job Design

Default job designs include all of the capabilities detailed Section 2: Standards. Template jobs should
contain all the default characteristics and parameters the project requires. These defaults provide at a
minimum:

1. Development phase neutral storage (e.g.: dev, si, qa and prod);

2. Support for Teradata, Oracle, DB2/UDB and SQL Server login requirements;
3. Enforced project standards;
4. Optional operational metadata (runtime statistics) suitable for loading into a database; and
5. Optional auditing capabilities.

The default job design specifically will support the creation of write-through cache in which data in
load-ready format is stored in DS/EE Data Sets for use in the load process or in the event the target
table becomes unavailable.

The default job design incorporates several features and components of DataStage that are used
together to support tactical and strategic job deployment. These features include:
1. Re-start-able job sequencers which manage one or more jobs, detect and report failure
conditions, provide monitoring and alert capabilities and support checkpoint restart
functionality.
2. Custom routines written in DataStage BASIC (DS Basic) that detect external events,
manage and manipulate external resources, provide enhanced notification and alert
capabilities and interface to the UNIX operating system.
3. DataStage Enterprise Edition (DS/EE) ETL jobs that exploit job parameterization, runtime
UNIX environment variables, and conditional execution.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 41 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Each subject area is broken into sub-areas and each sub-area

may be further subdivided. These sub-areas are populated by a DataStage job sequencer utilizing 2
types of DataStage jobs at a minimum:
1. A job that reads source data and
• Transforms it to load-ready format
• Optionally stores its results in a write-through cache DataStage Data Set or loads the
data to the target table.
2. A job that reads the DataStage dataset and loads it to the target table.

Other sections will discuss in detail each of the components and give examples of their use in a
working example job sequencer.

3.5 Job Parameters

Parameters are passed to a job as either DataStage job parameters or as environment variables. Job
parameters can be set from a file and are distinguished by the presence of a ‘jp’ prefix to the variable
name. This prefix is part of the DataStage development standard. The names of environment variables
have no prefix when they are set (UNIX_VAR=”some value”) and a prefix of “$” when used
(myval=$UNIX_VAR).

Job parameters are passed from a job sequencer to the jobs in its control as if a user were answering the
runtime dialog questions displayed in the DataStage Director job-run dialogue. Default environment
variables cannot be reset during this dialog unless explicitly specified in the job.

The scope of a parameter depends on their type. Essentially:

o The scope of a job parameter is specific to the job in which it is defined and used. Job
parameters are stored internally within DataStage for the duration of the job, and are not
accessible outside that job.
o The scope of a job parameter can be extended by the use of job sequencer, which can manage
and pass job parameters among jobs.
o The scope of an environment variable is wider, as it is defined at operating system level, though
conversely the use of environment variables is limited within this exercise.

Job parameters are required for the following DataStage programming elements:
1. File name entries in stages that use files or Data Sets must NEVER use a hard-coded operating
system pathname.
a. Staging area files must ALWAYS have pathnames as follows:
/jpSTAGING/jpENVIRON/jpSUBJECT_AREA[filename.suffix]

b. DataStage datasets ALWAYS have pathnames as follows:

/jpDSTAGE_ROOT/jpENVIRON/datasets/[filename.suffix]

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 42 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2. Database stages must ALWAYS use variables for the

server name, schema (if appropriate), userid and
password.

Use and management of job parameters, as well as standardized routines for use in Job Sequencers are
discussed further in Parallel Framework Standard Practices: Administration, Management, and
Production Automation.

3.6 Parallel Shared Containers

Parallel Shared Containers allow common logic to be shared across multiple jobs. For maximum
component re-use, enable RCP at the project level and for every stage within the parallel shared
container. This allows the container input and output links to contain only the columns relevant to the
container processing. Using RCP, any additional columns are passed through the container at runtime
without the need to separate and remerge.

Because Parallel Shared Containers are inserted when a job is compiled, all jobs that use a shared
container must be recompiled when the container is changed. The Usage Analysis and Multi-Job
Compile tools can be used to recompile jobs that use a shared container.

3.7 Error and Reject Record Handling

Reject rows are those rows that fail active or passive business rule driven validation as specified in the
job design document. Error rows are those rows caused by unforeseen data events such as values too
large for a column or text in an unsupported language.

The exact policy for each reject is specified in the job design document, and further, whether the job or
ETL processing is to continue is specified on a per-job and/or per-sequence and/or per-script basis
based on business requirements.

Reject files will include those records rejected from the ETL stream due to Referential Integrity
failures, data rule violations or other reasons that would disqualify a row from processing. The
presence of rejects may indicate that a job has failed and prevent further processing. Specification of
this action is the responsibility of the Business Analyst and will be published in the design document.
Error files will include those records from sources that fail quality tests. The presence of errors may not
prevent further processing. Specification of this action is the responsibility of the Business Analyst and
will be published in the design document.

Both rejects and errors will be archived and placed in a special directory for evaluation or other action
by support staff. The presence of rejects and errors will be detected and notification sent by email to
selected staff. These activities are the responsibility of job sequencers used to group jobs by some
reasonable grain or by a federated scheduler.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 43 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

ETL actions to be taken for each record type is specified for

each stage in the job design document. These actions include:
1. Ignore – some process or event downstream of the ETL process is responsible for handling the
error.
2. Reprocess – rows are reprocessed and re-enter the data stream.
3. Push back – rows are sent to a Data Steward for corrective action.

The default action is to push back reject and error rows to a Data Steward.

3.7.1 Reject Handling with the Sequential File Stage

The Sequential File stage can optionally include a reject link, which outputs rows that do not match the
given table definition and format specifications. By default, rows that cannot be read are dropped by
the Sequential File stage. A message is always written to the Director log which details the count of
rows successfully read and rows rejected.

The Sequential File stage offers the following reject options:

Option Description
Continue Drop read failures from input stream. Pass successful reads
to the output stream. (No reject link exists)
Fail Abort job on read format failure (No reject link exists)
Output Reject switch failures to the reject stream. Pass successful
reads to the output stream. (Reject link exists)

The reject option should be used in all cases where active management of the rejects is required.

If a file is created by this option, it must have a *.rej file extension. Alternatively, a shared container
error handler can be used.

Rejects are categorized in the ETL job design document using the following ranking:

Category Description Sequential File Stage Option

1 Rejects are expected and can be Use the Continue option.
ignored Only records that match the
given table definition and
format are output. Rejects are
tracked by count only.
2 Rejects should not exist but should Use the Output option. Send
not stop the job, and should be the reject stream to a *.rej
reviewed by the Data Steward. file.
3 Rejects should not exist and should Use the Fail option.
stop the job.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 44 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

3.7.2 Reject Handling with the Lookup Stage

The Lookup stage compares a single input stream to one or more reference streams using keys, and
rejects can occur if the key fields are not found in the reference data. This behavior makes the Lookup
stage very valuable for positive (reference is found) and negative (reference is NOT found) business
rule validation. DS/EE offers the following options within a Lookup stage:

Option Description
Continue Ignore lookup failures and pass lookup fields as nulls to the
output stream. Pass successful lookups to the output
stream.
Drop Drop lookup failures from the input stream. Pass successful
lookups to the output stream.
Fail Abort job on lookup failure
Reject Reject lookup failures to the reject stream. Pass successful
lookups to the output stream.

The reject option should be used in all cases where active management of the rejects is required.
Furthermore, to enforce error management ONLY ONE REFERENCE LINK is allowed on a Lookup
stage. If there are multiple validations to perform, each must be done in its own Lookup.

If a file is created by this option, it must have a *.rej or *.err file extension. The *.rej extension is used
when rejects require investigation after a job run, the *.err extension when rejects can be ignored but
need to be recorded. Alternatively, a local error handler based on a shared container can be used.

Rejects are categorized in the ETL job design document using the following ranking:

Category Description Lookup Stage Option

1 Rejects are expected and can be Drop if lookup fields are
ignored necessary down stream or
Continue if lookup fields are
optional
2 Rejects can exist in the data, Send the reject stream to an
however, they only need to be *.err file or tag and merge
recorded but not acted on. with the output stream.
3 Rejects should not exist but should Send the reject stream to an
not stop the job, and should be *.rej file or tag and merge
reviewed by the Data Steward. with the output stream.
4 Rejects should not exist and should Use the Fail option.
stop the job.

3.7.3 Reject Handling with the Transformer Stage

Rejects occur when a transformer stage is used and a row:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 45 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

1. Satisfies requirements for a reject conditional output

stream; OR
2. Cannot satisfy requirements of any conditional output stream and is rejected by the default
output stream.

If a file is created from the reject stream, it must have a *.rej or *.err file extension. The *.rej extension
is used when rejects require investigation after a job run, the *.err extension when rejects can be
ignored but need to be recorded. Alternatively, a shared container error handler can be used.

Rejects are categorized in the ETL job design document using the following ranking:

Category Description Transformer Stage Option

1 Rejects are expected and can be Funnel the reject stream back
ignored. to the output stream(s).
2 Rejects can exist in the data, Send the reject stream to an
however, they only need to be *.err file or tag and merge
recorded but not acted on. with the output stream.
3 Rejects should not exist but should Send the reject stream to an
not stop the job, and be reviewed *.rej file or tag and merge
by the Data Steward. with the output stream.
4 Rejects should not exist and should Send the reject stream to a
stop the job. reject file and halt the job.

3.7.4 Reject Handling with target database stages

Some database stages (such as DB2/UDB Enterprise, ODBC Enterprise, and Oracle Enterprise)
offer an optional reject link that can be used to capture rows that cannot be written to the target
database. To capture rejects from a target database, a reject link must exist on that stage. Otherwise,
reject rows will not be captured. A message is always written to the Director log which details the
count of rows successfully read and rows rejected.

Target database stages offer the following reject options:

Option Description
No reject link exists Do not capture rows that fail to be written.
Reject link exists Pass rows that fail to be written to the reject stream.

The reject option should be used in all cases where active management of the rejects is required.

If a file is created by this option, it must have a *.rej file extension. Alternatively, a shared container
error handler is used.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 46 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Rejects are categorized in the ETL job design document using

the following ranking:

Category Description Target Database Stage Option

1 Rejects are expected and can be No reject link exists. Only
ignored records that match the given
table definition and database
constraints are written.Rejects
are tracked by count only.
2 Rejects should not exist but should Reject link exists. Send the
not stop the job, and should be reject stream to a *.rej file.
reviewed by the Data Steward.

3.7.5 Error Processing Requirements

Jobs will produce flat files containing reject and errors and may alternatively process rows on reject
ports and merge these rows with the normal output stream. This section deals with both methods of
handling errors.

3.7.5.1 Processing Errors and Rejects to a Flat File

Each job will produce a flat file for errors and a flat file for rejects with a specific naming convention:
1. The project name (jpPROJECT_NAME) and a underscore “_”;
2. The job name (jpJOB_NAME) and a underscore “_”;
3. The project phase (jpENVIRON) and a underscore “_”;
4. The job serial number (jpJOBSERIALNO) and a period “.”; and
5. The appropriate file type, one of “rej” or “err”.

For example, job DECRP_N_XformClients in the ECR_FACTS project in the development

environment with a serial number of 20060201-ETL-091504 would have these reject and error file
names:
ECR_FACTS_DECRP_N_XformClients_dev_20060201-ETL-091504.rej
ECR_FACTS_DECRP_N_XformClients_dev_20060201-ETL-091504.err

Rows will be converted to the common file record format with 9 columns (below) using Column
Export and Transformer stages for each reject port, and gathered using a Funnel stage that feeds a
Sequential File stage. The Column Export and Transformer stages may be kept in a template Shared
Container the developer will make local in each job.

The standard columns for error and reject processing are:

Column Name Key? Data Source

HOST_NAME Yes DSHostName transformer macro in the error handler

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 47 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

PROJECT_NAME Yes DSProjectName

transformer macro
in the error handler
JOB_NAME Yes DSJobName transformer macro in the error handler
STAGE_NAME Yes The name of the stage from which the error came
DATA_OBJ_NAME Yes The source table or file data object name
JOB_SERIALNO Yes jpJOBSERIALNO
ETL_ROW_NUM Yes Data stream coming in to the error handler
ETL_BAT_ID Yes Data stream coming in to the error handler
ROW_DATA No The columns from the upstream stages reject port
exported to a single pipe-delimited “|” varchar(2000)
column using the Column Export stage in the error
handler

In this example, the following stages process the only errors produced by a job:

The Column Export stage maps

the unique columns to the single
standard column.

The Transformer stage adds the

required key columns.

Figure 13: Error Processing Components

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 48 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

The input to the Column Export stage explicitly converts the

data unique to the reject stream (in this case, Track*) to a
single output column, ROW_DATA:

Figure 14: Error Processing Column Export stage

And the downstream Transformer stage builds the standard output record by creating the required
keys:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 49 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 15: Error Processing Transformer stage

3.7.5.2 Processing Errors and Rejects and Merging with an Output Stream
There may be processing requirements that specify that rejected or error rows be tagged as having
failed a validation and merged back into the output stream. This is done by processing the rows from
the reject ports and setting the value of a specific column with a value specified by the design
document. The following table identifies the tagging method to be used for the previously cited
operators.

Stage Description Method

Lookup A failed lookup will reject an intact input row Connect the reject port to a Transformer stage
whose key fails to match the reference link where those columns selected for replacement are
key. One or more columns may have been set to specific values. Connect the output stream of
selected for replacement when a reference key the Transformer and Lookup stages to a Funnel
is found. stage to merge the two streams.
Switch A failed switch will reject an intact input row Connect the reject port to a Transformer stage
show key fails to resolve to one of the Switch where columns are set to specific values. Connect
output stream. the output stream of the Transformer stage and one
or more output streams of the Switch stage to a
Funnel stage to merge the two (or more) streams.
Transformer A Transformer will reject an intact input row Connect the reject port to a Transformer stage
that cannot pass conditions specified on the where columns are set to specific values. Connect
output streams, OR with columns contain the output stream of the corrective Transformer
illegal values for some operation performed on stage and one or more output streams of the original
said columns. In either case, attaching a non- Transformer stage to a Funnel stage to merge the
specific reject stream (referred to as the stealth two (or more) streams.
reject stream) will gather rows from either
condition to the reject stream.

In this example, rows rejected by the Lookup stage are processed by a corrective Transformer stage
where the failed references as set to a specific value and then merged with the output of the Lookup
stage:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 50 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 16: Error Processing Lookup example

3.8 Component Usage

DataStage Enterprise Edition offers a wealth of component types for building ETL flows. This section
provides guidelines appropriate use of various stages when building a parallel job flows.

3.8.1 Server Edition Components

Avoid the use of Server Edition components in parallel job flows. Server Edition components limit
overall performance of large-volume job flows since many components such as the BASIC
Transformer use interpreted psuedo-code. In clustered an MPP environments Server Edition
components only run on the primary (conductor) node, severely impacting scalability and network
resources.

The ability to use a Server Edition component within a parallel job is intended only as a migration
option for existing Server Edition applications that might benefit by leveraging some parallel
capabilities on SMP platforms.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 51 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Server Edition Components that should be avoided within

parallel job flows include:
- BASIC Transformers
- BASIC Routines
- Server shared containers

Note that BASIC Routines are still appropriate, and necessary, for the job control components of a
DataStage Job Sequence and Before/After Job Subroutines for parallel jobs. This is discussed in more
detail in Parallel Framework Standard Practices: Administration, Management, and Production
Automation.

3.8.2 Copy Stage

For complex data flows, it is best to develop a job iteratively using the Copy stage as a “placeholder”.
Since the Copy stage does not require an output link, it can be used at the end of a data flow

o For simple jobs with only two stages, the Copy stage should be used as a placeholder so
that new stages can be inserted easily should future requirements change.
o Unless the Force property is set to “True”, a Copy stage with a single input link and a
single output link will be optimized out of the final job flow at runtime.

3.8.3 Parallel Data Sets

When writing intermediate results between DS/EE parallel jobs, always write to parallel Data Sets.
Data Sets achieve end-to-end parallelism across job boundaries by writing data in partitioned form, in
sort order, and in Enterprise Edition native format. Used in this manner, parallel Data Sets effectively
establish restart points in the event that a job (or sequence) needs to be re-run.

Data Sets offer parallel I/O on read and write operations, without overhead for format or data type
conversions.

NOTE: Because parallel Data Sets are platform and configuration-specific, they should not be
used for long-term archive of source data.

3.8.4 Parallel Transformer stages

The DataStage Enterprise Edition parallel Transformer stage generates “C” code which is then
compiled into a parallel component. For this reason, it is important to minimize the number of
transformers, and to use other stages (such as Copy) when derivations are not needed.

• The Copy stage should be used instead of a Transformer for simple operations including:
- Job Design placeholder between stages (unless the Force option =true, Enterprise Edition
will optimize this out at runtime)
- Renaming Columns
- Dropping Columns
Parallel Framework Red Book: Data Flow Job Design July 17, 2006 52 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

- Default Type Conversions [see Section 4.1.2:

Default and Explicit Type Conversions]

Note that rename, drop (if Runtime Column Propagation is disabled), and default type
conversion can also be performed by the output mapping tab of any stage.

• NEVER use the “BASIC Transformer” stage in large-volume job flows. Instead, user-defined
functions and routines can expand parallel Transformer capabilities. The BASIC Transformer
is intended as a “stop-gap” migration choice for existing Server Edition jobs containing
complex routines. Even then its use should be restricted and the routines should be converted as
soon as possible.

• Consider, if possible, implementing complex derivation expressions using regular patterns by

Lookup tables instead of using a Transformer with nested derivations.

For example, the derivation expression:

If A=0,1,2,3 Then B=”X” If A=4,5,6,7 Then B=”C”

could also be implemented with a lookup table containing values for column A and
corresponding values of column B.

• Optimize the overall job flow design to combine derivations from multiple Transformers
into a single Transformer stage when possible.

• Because the parallel Transformer is compiled, it is faster than the interpreted Filter and
Switch stages. The only time that Filter or Switch should be used is when the selection clauses
need to be parameterized at runtime.

• The Modify stage can be used for non-default type conversions, null handling, and
character string trimming. See Section 8.2: Modify Stage.

3.8.5 BuildOp stages

BuildOps should only be used when:
- Complex reusable logic cannot be implemented using the Transformer or
- Existing Transformers do not meet performance requirements

As always, performance should this should be tested in isolation to identify specific cause of
bottlenecks.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 53 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

4 DataStage Data Types

The DataStage Designer and Manager represent column data types using SQL notation. Each SQL data
type maps to an underlying data type in the Enterprise Edition engine. The internal Enterprise Edition
data types are used in schema files and are displayed when viewing generated OSH or viewing the
output from $OSH_PRINT_SCHEMAS.

The following table summarizes the underlying data types of DataStage Enterprise Edition:
SQL Type Internal Type Size Description
Date date 4 bytes Date with month, day, and year
Decimal, decimal (roundup(p)+1)/2 Packed decimal, compatible with IBM packed
Numeric decimal format.
Float, sfloat 4 bytes IEEE single-precision (32-bit) floating point
Real value
Double dfloat 8 bytes IEEE double-precision (64-bit) floating point
value
TinyInt int8, uint8 1 byte Signed or unsigned integer of 8 bits
(Specify unsigned Extended option for unsigned)
SmallInt int16, uint16 2 bytes Signed or unsigned integer of 16 bits
(Specify unsigned Extended option for unsigned)
Integer int32, unit32 4 bytes Signed or unsigned integer of 32 bits
(Specify unsigned Extended option for unsigned)
BigInt1 int64, unit64 8 bytes Signed or unsigned integer of 64 bits
(Specify unsigned Extended option for unsigned)
Binary, raw 1 byte per character Untyped collection, consisting of a fixed or
Bit, variable number of contiguous bytes and an
LongVarBinary, optional alignment value

VarBinary
Unknown, string 1 byte per character ASCII character string of fixed or variable length
Char, (Unicode Extended option NOT selected)
LongVarChar,
VarChar
NChar, ustring multiple bytes per ASCII character string of fixed or variable length
NVarChar, character (Unicode Extended option NOT selected)
LongNVarChar
Char, ustring multiple bytes per ASCII character string of fixed or variable length
LongVarChar, character (Unicode Extended option IS selected)
VarChar
Time time 5 bytes Time of day, with resolution to seconds
Time time(microseconds) 5 bytes Time of day, with resolution of microseconds
(Specify microseconds Extended option)
Timestamp timestamp 9 bytes Single field containing both date and time value
with resolution to seconds.
Timestamp timestamp(microseconds) 9 bytes Single field containing both date and time value
with resolution to microseconds.
(Specify microseconds Extended option)
1
BigInt values map to long long integers on all supported platforms except Tru64 where they map
to longer integer values.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 54 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

4.1.1 Strings and Ustrings

If NLS is enabled on your DataStage server, parallel jobs support two types of underlying character
data types: strings and ustrings. String data represents unmapped bytes, ustring data represents full
Unicode (UTF-16) data.

The Char, VarChar, and LongVarChar SQL types relate to underlying string types where each
character is 8-bits and does not require mapping because it represents an ASCII character. You can,
however, specify that these data types are extended, in which case they are taken as ustrings and do
require mapping. (They are specified as such by selecting the Extended check box for the column in
the Edit Meta Data dialog box.) An Extended field appears in the columns grid, and extended Char,
VarChar, or LongVarChar columns have ‘Unicode’ in this field. The NChar, NVarChar, and
LongNVarChar types relate to underlying ustring types so do not need to be explicitly extended.

4.1.2 Default and Explicit Type Conversions

DataStage Enterprise Edition provides a number of default conversions and conversion functions when
mapping from a source to a target data type. Default type conversions take place across the stage output
mappings of any Enterprise Edition stage. The following table summarizes Data Type conversions:

Source Target Field

Field d = There is a default type conversion from source field type to destination field type.
e = You can use a Modify or a Transformer conversion function to explicitly convert from
the source field type to the destination field type.
A blank cell indicates that no conversion is provided.

date

time
int8

uint8

sfloat
dfloat
int16

uint16
int32

uint32

int64

uint64

string

ustring

timestamp
decimal

raw
Int8 d d d d d d d d de d d de e e e
e
uint8 d d d d d d d d d d d d
Int16 de d d d d d d d d d d de
e
uint16 d d d d d d d d d d d de
e
Int32 de d d d d d d d d d d de e e
e
uint32 d d d d d d d d d d d de e
e
Int64 de d d d d d d d d d d d
uint64 d d d d d d d d d d d d
sfloat de d d d d d d d d d d d
dfloat de d d d d d d d d d d de e e
e e
decimal de d d d de d de de d de d de
e
string de d de d d de d d d de d d e e e

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 55 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

e
ustring de d de d d de d d d de d d e e
e
raw e e
date e e e e e e e
time e e e e e de
timesta e e e e e e e
mp

The conversion of numeric data types may result in a loss of precision and cause incorrect results,
depending on the source and result data types. In these instances, Enterprise Edition displays a warning
message in the job log.

When converting from variable-length to fixed-length strings, Enterprise Edition pads the remaining
length with NULL (ASCII zero) characters by default.
• The environment variable APT_STRING_PADCHAR can be used to change the default pad
character from an ASCII NULL (0x0) to another character; for example, an ASCII space
(0x20) or a Unicode space (U+0020). When entering a space for the value of
APT_STRING_PADCHAR do note enclose the space character in quotes.

• As an alternate solution, the PadString Transformer function can be used to pad a variable-
length (Varchar) string to a specified length using a specified pad character. Note that
PadString does not work with fixed-length (CHAR) string types. You must first convert a Char
string type to a Varchar type before using PadString.

• Some stages (for example, Sequential File and DB2/UDB Enterprise targets) allow the pad
character to be specified in their stage or column definition properties. When used in these
stages, the specified pad character will override the default for that stage only.

4.2 Null Handling

DataStage Enterprise Edition represents nulls in two ways:
- It allocates a single bit to mark a field as null. This type of representation is called an out-of-band
null.
- It designates a specific field value to indicate a null, for example a numeric field’s most negative
possible value. This type of representation is called an in-band null. In-band null representation can
be disadvantageous because you must reserve a field value for nulls and this value cannot be treated
as valid data elsewhere.

The Transformer and Modify stages can change a null representation from an out-of-band null to an in-
band null and from an in-band null to an out-of-band null.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 56 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

IMPORTANT: When processing nullable columns in a

Transformer stage, care must be taken to avoid data rejects. See Section 8.1.1: Transformer
NULL Handling and Reject Link.

When reading from Data Set and database sources with nullable columns, Enterprise Edition uses the
internal, out-of-band null representation for NULL values.

When reading from or writing to Sequential Files or File Sets, the in-band (value) must be explicitly
defined in the extended column attributes for each Nullable column, as shown in Figure 17:

Figure 17: Extended Column Metadata (Nullable properties)

The Table Definition of a stage’s input or output data set can contain columns defined to support out-
of-band nulls (Nullable attribute is checked). The next table lists the rules for handling nullable fields
when a stage takes a Data Set as input or writes to a Data Set as output.

Source Field Destination Field Result

not Nullable not Nullable Source value propagates to destination.
Nullable Nullable Source value or null propagates.
not Nullable Nullable Source value propagates; destination value is
never null.
Nullable not Nullable If the source value is not null, the source value
propagates.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 57 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

If the source
value is null, a fatal error occurs.

4.3 Runtime Column Propagation

Runtime column propagation (“RCP”) allows job designs to accommodate additional columns beyond
those defined by the job developer. Before a DataStage developer can use RCP, it must be enabled at
the project level through the Administrator client.

Using RCP judiciously in a job design facilitates re-usable job designs based on input metadata, rather
than using a large number of jobs with hard-coded table definitions to perform the same tasks. Some
stages, for example the Sequential File stage, allow their runtime schema to be parameterized further
extending re-use through RCP.

Furthermore, RCP facilitates re-use through parallel shared containers. Using RCP, only the columns
explicitly referenced within the shared container logic need to be defined, the remaining columns pass
through at runtime, as long as each stage in the shared container has RCP enabled on their stage Output
properties.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 58 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

5 Partitioning and Collecting

Partition parallelism is a key to establishing scalable performance of DataStage Enterprise Edition.
Partitioners distribute rows of a single link into smaller segments that can be processed independently
in parallel. Partitioners exist before any stage that is running in parallel. If the prior stage was running
sequentially, a “fan-out” icon is drawn on the link within the Designer canvas, as shown in this
example:

Stage Stage
running running in
sequentially parallel

Figure 18: “fan-out” icon

Collectors combine parallel partitions of a single link for sequential processing. Collectors only exist
before stages running sequentially and when the previous stage is running in parallel, and are indicated
by a “fan-in” icon as shown in this example:

Stage Stage
running in running
parallel sequentially

Figure 19: Collector icon

This section provides an overview of partitioning and collecting methods, and provides guidelines for
appropriate use in job designs. It also provides tips for monitoring jobs running in parallel.

5.1 Partition Types

While partitioning allows data to be distributed across multiple processes running in parallel, it is
important that this distribution does not violate business requirements for accurate data processing. For
this reason, different types of partitioning are provided for the parallel job developer.

Partitioning methods are separated into keyless and keyed classes:

- Keyless partitioning distributes rows without regard to the actual data values. Different types of
keyless partitioning methods define the method of data distribution.

- Keyed partitioning examines the data values in one or more key columns, ensuring that records
with the same values in those key column(s) are assigned to the same partition. Keyed partitioning
is used when business rules (for example, Remove Duplicates) or stage requirements (for example,
Join) require processing on groups of related records.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 59 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

The default partitioning method for newly-drawn

links is Auto partitioning.

The partitioning method is specified in the Input

stage properties using the “Partitioning” option as
shown on the right:

Figure 20: Specifying Partition method

5.1.1 Auto Partitioning

The default partitioning method for newly-drawn links, Auto partitioning specifies that the Enterprise
Edition engine will attempt to select the appropriate partitioning method at runtime. Based on the
configuration file, Data Sets, and job design (stage requirements and properties), Auto partitioning will
select between keyless (Same, Round Robin, Entire) and keyed (Hash) partitioning methods to produce
functionally correct results and, in some cases, to improve performance.

Within the Designer canvas, links with Auto partitioning are drawn with the following link icon:

Figure 21: Auto partitioning icon

Auto partitioning is designed to allow the beginning DataStage developer to construct simple data
flows without having to understand the details of parallel design principles. However, the partitioning
method may not necessarily be the most efficient from an overall job perspective.

Furthermore, the ability for the Enterprise Edition engine to determine the appropriate partitioning
method depends on the information available to it. In general, Auto partitioning will ensure correct
results when using built-in stages. However, since the Enterprise Edition engine has no visibility into
user-specified logic (such as Transformer or BuildOp stages) it may be necessary to explicitly specify a
partitioning method for some stages. For example, if the logic defined in a Transformer stage is based
on a group of related records, then a keyed partitioning method must be specified to achieve correct
results.

The “Preserve Partitioning” flag is an internal “hint” that Auto partitioning uses to attempt to preserve
carefully ordered data (for example, on the output of a parallel Sort). This flag is set automatically by

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 60 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

some stages (Sort, for example), although it can be

explicitly set or cleared in the “Advanced” stage
properties of a given stage, as shown in on the right:

Figure 22: Preserve Partitioning option

The Preserve Partitioning flag is part of the Data Set structure, and its state is stored in persistent Data
Sets.

There are some cases when the input stage requirements prevent partitioning from being preserved. In
these instances, if the Preserve Partitioning flag was set, a warning will be placed in the Director log
indicating that Enterprise Edition was unable to preserve partitioning for a specified stage.

5.1.2 Keyless Partitioning

Keyless partitioning methods distribute rows without examining the contents of the data:

Keyless Partition Method Description

Same Retains existing partitioning from previous stage.
Round Robin Distributes rows evenly across partitions, in a round
robin partition assignment.
Random Distributes rows evenly across partitions in a random
partition assignment.
Entire Each partition receives the entire Data Set.

5.1.2.1 Same Partitioning

Same partitioning in fact performs no partitioning to the input Data Set. Row ID's 0 1 2
3 4 5
Instead, it retains the partitioning from the output of the upstream stage, 6 7 8
as illustrated on the right:

Same partitioning doesn’t move data between partitions (or, in the case
of a cluster or Grid, between servers), and is appropriate when trying to 0 1 2
preserve the grouping of a previous operation (for example, a parallel 3 4 5
6 7 8
Sort).

Within the Designer canvas, links that have been specified with Same partitioning are drawn with a
“horizontal line” partitioning icon:

Figure 23: Same partitioning icon

It is important to understand the impact of Same partitioning in a given data flow. Because Same
does not redistribute existing partitions, the degree of parallelism remains unchanged:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 61 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

If the upstream stage is running sequentially, Same

partitioning will effectively cause a downstream parallel stage to also run sequentially

If you read a parallel Data Set with Same partitioning, the downstream stage runs with the degree of
parallelism used to create the Data Set, regardless of the current $APT_CONFIG_FILE

5.1.2.2 Round Robin Partitioning

…8 7 6 5 4 3 2 1 0
Round Robin partitioning evenly distributes rows across
partitions in a round-robin assignment, similar to dealing cards:

Round robin partitioning has a fairly low overhead. Round Robin

Since optimal parallel processing occurs when all partitions have

the same workload, Round Robin partitioning is useful for
redistributing data that is highly skewed (there are an unequal 6
3
7
4
8
5
number of rows in each partition). 0 1 2

5.1.2.3 Random Partitioning

Like Round Robin, Random partitioning evenly distributes rows across partitions, but using a random
assignment. As a result, the order that rows are assigned to a particular partition will differ between job
runs.

Since the random partition number must be calculated, Random partitioning has a slightly higher
overhead than Round Robin partitioning.

While in theory Random partitioning is not subject to regular data patterns that might exist in the
source data, it is rarely used in real-world data flows.

5.1.2.4 Entire Partitioning

Entire partitioning distributes a complete copy of the entire Data Set …8 7 6 5 4 3 2 1 0
to each partition, as illustrated on right:

Entire partitioning is useful for distributing the reference data of a ENTIRE

Lookup task (this may or may not involve the Lookup stage).

On clustered and Grid implementations, Entire partitioning may

have a performance impact, as the complete Data Set must be . . .
distributed across the network to each node. .
3
.
3
.
3
2 2 2
1 1 1
0 0 0

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 62 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

5.1.3 Keyed Partitioning

Keyed partitioning examines the data values in one or more key columns, ensuring that records with the
same values in those key column(s) are assigned to the same partition. Keyed partitioning is used when
business rules (for example, Remove Duplicates) or stage requirements (for example, Join) require
processing on groups of related records.

Keyed Partition Method Description

Hash Assigns rows with the same values in one or more key
column(s) to the same partition using an internal hashing
algorithm.
Modulus Assigns rows with the same values in a single integer key
column to the same partition using a simple modulus
calculation.
Range Assigns rows with the same values in one or more key
column(s) to the same partition using a specified range
map generated by pre-reading the Data Set.
DB2 For DB2 Enterprise Server Edition with DPF
(DB2/UDB) only – matches the internal partitioning of
the specified source or target table.

5.1.3.1 Hash Partitioning

Hash partitioning assigns rows with the same values in one or more Values of key column

key column(s) to the same partition using an internal hashing …0 3 2 1 0 2 3 2 1 1

algorithm.

If the source data values are evenly distributed within these key HASH
column(s), and there are a large number of unique values, then the
resulting partitions will be of relatively equal size.

As an example of hashing, consider the following sample Data Set: 0 1 2

3 1 2
0 1 2
ID LName FName Address 3
1 Ford Henry 66 Edison Avenue
2 Ford Clara 66 Edison Avenue
3 Ford Edsel 7900 Jefferson
4 Ford Eleanor 7900 Jefferson
5 Dodge Horace 17840 Jefferson
6 Dodge John 75 Boston Boulevard
7 Ford Henry 4901 Evergreen
8 Ford Clara 4901 Evergreen
9 Ford Edsel 1100 Lakeshore
10 Ford Eleanor 1100 Lakeshore

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 63 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Hashing on key column LName would produce the following

results:

Partition 0: Partition 1:
ID LName FName Address ID LName FName Address
5 Dodge Horace 17840 Jefferson 1 Ford Henry 66 Edison Avenue
6 Dodge John 75 Boston Boulevard 2 Ford Clara 66 Edison Avenue
3 Ford Edsel 7900 Jefferson
4 Ford Eleanor 7900 Jefferson
7 Ford Henry 4901 Evergreen
8 Ford Clara 4901 Evergreen
9 Ford Edsel 1100 Lakeshore
10 Ford Eleanor 1100 Lakeshore

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 64 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

In this case, there are more instances of “Ford” than “Dodge”,

producing partition skew, which would impact performance.

Also note that in this example the number of unique values will limit the degree of parallelism,
regardless of the actual number of nodes in the parallel configuration file.

Using the same source Data Set, hash partitioning on the key columns LName and FName yields the
following distribution with a 4-node configuration file:

Partition 0: Partition 2:
I LName FName Address I LName FName Address
D D
2 Ford Clara 66 Edison Avenue 4 Ford Eleanor 7900 Jefferson
8 Ford Clara 4901 Evergreen 6 Dodge John 75 Boston Boulevard
10 Ford Eleanor 1100 Lakeshore
Partition 1:
I LName FName Address Partition 3:
D I LName FName Address
3 Ford Edsel 7900 Jefferson D
5 Dodge Horace 17840 Jefferson 1 Ford Henry 66 Edison Avenue
9 Ford Edsel 1100 Lakeshore 7 Ford Henry 4901 Evergreen

In this example, the key column combination of LName and FName yields improved data distribution
and a greater degree of parallelism.

Also note that only the unique combination of key column values appear in the same partition when
used for hash partitioning. When using hash partitioning on a composite key (more than one key
column), individual key column values have no significance for partition assignment.

5.1.3.2 Modulus Partitioning

Modulus partitioning uses a simplified algorithm for assigning related records based on a single integer
key column. It performs a modulus operation on the data value using the number of partitions as the
divisor. The remainder is used to assign the value to a given partition:

partition = MOD (key_value / number of partitions)

Like hash, the partition size of modulus partitioning will be equally distributed as long as the data
values in the key column are equally distributed.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 65 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Since modulus partitioning is simpler and faster than hash,

it should be used if you have a single integer key column.
Modulus partitioning cannot be used for composite keys, or for a non-integer key column.

5.1.3.3 Range Partitioning

As a keyed partitioning method, Range partitioning assigns rows with
Values of key
the same values in one or more key column(s) to the same partition. column
Given a sufficient number of unique values, Range partitioning 4 0 5 1 6 0 5 4 3

ensures balanced workload by assigning an approximately equal

number of rows to each partition, unlike Hash and Modulus
partitioning where partition skew is dependent on the actual data RANGE
distribution. Rang
e
Map fi
le

To achieve this balanced distribution, Range partitioning must read

0 4
the Data Set twice: once to create a Range Map file, and the second to 1 4
actually partition the data within a flow using the Range Map. A 0 3

Range Map file is specific to a given parallel configuration file.

The “read twice” penalty of Range partitioning limits its use to specific scenarios, typically where
the incoming data values and distribution are consistent over time. In these instances, the Range
Map file can be re-used.

It is important to note that if the data distribution changes without recreating the Range Map, partition
balance will be skewed, defeating the intention of Range partitioning. Also, if new data values are
processed outside of the range of a given Range Map, these rows will be assigned to either the first or
the last partition, depending on the value.

In another scenario to avoid, if the incoming Data Set is sequential and ordered on the key column(s),
Range partitioning will result in sequential processing.

5.1.3.4 DB2 Partitioning

The DB2/UDB Enterprise stage matches the internal database partitioning of the source or target DB2
Enterprise Server Edition with Data Partitioning Facility database (previously called “DB2/UDB
EEE”). Using the DB2/UDB Enterprise stage, data is read in parallel from each DB2 node. And, by
default, when writing data to a target DB2 database using the DB2/UDB Enterprise stage, data is
partitioned to match the internal partitioning of the target DB2 table using the DB2 partitioning
method.

DB2 partitioning can only be specified for target DB2/UDB Enterprise stages. To maintain partitioning
on data read from a DB2/UDB Enterprise stage, use Same partitioning on the input to downstream
stages.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 66 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

5.2 Monitoring Partitions

At runtime, DataStage Enterprise Edition determines the degree of parallelism for each stage using:
a) the parallel configuration file (APT_CONFIG_FILE)
b) the degree of parallelism of existing source and target Data Sets (and, in some cases, databases)
c) and, if specified, a stage’s node pool (Stage/Advanced properties)

This information is detailed in the parallel job score, which is output to the Director job log when the
environment variable APT_DUMP_SCORE is set to True. Specific details on interpreting the parallel
job score can be found in 12.4.2Understanding the Parallel Job Score.

Partitions are assigned numbers, starting at zero. The partition number is appended to the stage name
for messages written to the Director log, as shown in the example log below where the stage named
“Peek” is running with four degrees of parallelism (partition numbers zero through 3):

Figure 24: Partition numbers as shown in Director log

To display row counts per partition in the Director Job Monitor window, right-click anywhere in the
window, and select the “Show Instances” option, as illustrated below. This is very useful in
determining the distribution across parallel partitions (skew). In this instance, the stage named “Sort_3”
is running across four partitions (“x 4” next to the stage name), and each stage is processing an equal
number (12,500) of rows for an optimal balanced workload.

Figure 25: Director Job Monitor row counts by partition

Setting the environment variable APT_RECORD_COUNTS will output the row count per link per
partition to the Director log as each stage/node completes processing, as illustrated below:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 67 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 26: Output of APT_RECORD_COUNTS in Director log

Finally, the “Data Set Management” tool (available in the Tools menu of Designer, Director, or
Manager) can be used to identify the degree of parallelism and number of rows per partition for an
existing persistent Data Set, as shown below:

Figure 27: Data Set Management Tool

In a non-graphical way, the orchadmin command line utility on the DataStage server can also be used
to examine a given parallel Data Set.

5.3 Partition Methodology

Given the numerous options for keyless and keyed partitioning, the following objectives help to form a
methodology for assigning partitioning:

Objective 1: Choose a partitioning method that gives close to an equal number of rows in
each partition, while minimizing overhead.

This ensures that the processing workload is evenly balanced, minimizing overall run time.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 68 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Objective 2: The partition method must match the

business requirements and stage functional requirements, assigning related records to the
same partition if required

Any stage that processes groups of related records (generally using one or more key columns)
must be partitioned using a keyed partition method.

This includes, but is not limited to: Aggregator, Change Capture, Change Apply, Join, Merge,
Remove Duplicates, and Sort stages. It may also be necessary for Transformers and BuildOps
that process groups of related records.

Note that in satisfying the requirements of this second objective, it may not be possible to
choose a partitioning method that gives close to an equal number of rows in each partition.

Objective 3: Unless partition distribution is highly skewed, minimize repartitioning,

especially in cluster or Grid configurations
Repartitioning data in a cluster or Grid configuration incurs the overhead of network transport.

Objective 4: Partition method should not be overly complex

The simplest method that meets the above objectives will generally be the most efficient and
yield the best performance.

Using the above objectives as a guide, the following methodology can be applied:
a) Start with Auto partitioning (the default)
b) Specify Hash partitioning for stages that require groups of related records
o Specify only the key column(s) that are necessary for correct grouping as long as the
number of unique values is sufficient
o Use Modulus partitioning if the grouping is on a single integer key column
o Use Range partitioning if the data is highly skewed and the key column values and
distribution do not change significantly over time (Range Map can be reused)
c) If grouping is not required, use Round Robin partitioning to redistribute data equally across
all partitions
o Especially useful if the input Data Set is highly skewed or sequential
d) Use Same partitioning to optimize end-to-end partitioning and to minimize repartitioning
o Being mindful that Same partitioning retains the degree of parallelism of the
upstream stage
o Within a flow, examine up-stream partitioning and sort order and attempt to preserve
for down-stream processing. This may require re-examining key column usage
within stages and re-ordering stages within a flow (if business requirements permit).
o Across jobs, persistent Data Sets can be used to retain the partitioning and sort order.
This is particularly useful if downstream jobs are run with the same degree of
parallelism (configuration file) and require the same partition and sort order.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 69 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

5.4 Partitioning Examples

In this section, we’ll apply the partitioning methodology defined earlier to several example job flows.
Additional, more advanced partitioning and sorting examples are given in 12.4.2Sorting and Hashing
Advanced Example.

5.4.1 Partitioning Example 1 – Optimized Partitioning

The Aggregator stage only outputs key column and aggregate result columns. To add aggregate
columns to every detail row, a Copy stage is used to send the detail rows to an Inner Join and an
Aggregator. The output of the Aggregator is then sent to the second input of the Join. The standard
solution would be to Hash partition (and Sort) the inputs to the Join and Aggregator stages as shown
below:

Figure 28: “Standard” Partitioning assignment

However, on closer inspection, the partitioning and sorting of this scenario can be optimized. Because
the Join and Aggregator use the same partition keys and sort order, we can move the Hash partition and
Sort before the Copy stage, and apply Same partitioning to the downstream links, as shown below:

Figure 29: Optimized Partitioning assignment

This example will be revisited in the Sorting discussion because there is one final step necessary to
optimize the sorting in this example.

5.4.2 Partitioning Example 2 – Use of Entire Partitioning

In this example, a Transformer is used to extract data from a single header row of an input file. Within
the Transformer, a new output column is defined on the header and detail links using a single constant
value derivation. This column is used as the key for a subsequent Inner Join to attach the header values
to every detail row. Using a “standard” solution, both inputs to the Join would be Hash partitioned and
sorted on this single join column (either explicitly, or through Auto partitioning):

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 70 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 30: “Standard” Partitioning assignment for a Join stage

Although Hash partitioning guarantees correct results for stages that require groupings of related
records, it is not always the most efficient solution, depending on the business requirements. Although
functionally correct, the above solution has one serious limitation. Remembering that the degree of
parallel operation is limited by the number of distinct values, the single value join column will assign
all rows to a single partition, resulting in sequential processing.

To optimize partitioning, consider that the single header row is really a form of reference data. An
optimized solution would be to alter the partitioning for the input links to the Join stage:
- Use Round Robin partitioning on the detail input to evenly distribute rows across all partitions
- Use Entire partitioning on the header input to copy the single header row to all partitions

Figure 31: Optimized Partitioning assignment based on business requirements

Because we are joining on a single value, there is no need to pre-sort the input to the Join, so we will
revisit this in the Sorting discussion.

In order to process a large number of detail records, the link order of the Inner Join is significant.
The Join stage operates by reading a single row from the Left input and reading all rows from the Right
input that match the key value(s). For this reason, the link order in this example should be set so that
the single header row is assigned to the Right input, and the detail rows are assigned to the Left input as
shown in the following illustration:

Figure 32: Specifying Link Order in Join stage

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 71 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

If defined in reverse of this order, the Join will attempt to read

all detail rows from the right input (since they have the same
key column value) into memory.

For advanced users, there is one further detail in this example. Because the Join will wait until it
receives an End of Group (new key value) or End of Data (no more rows on the input Data Set) from
the Right input, the detail rows in the Left input will buffer to disk to prevent a deadlock. (See Section
12.3: Minimizing Runtime Processes and Resource Requirements). Changing the output derivation on
the header row to a series of numbers instead of a constant value will establish the End of Group and
prevent buffering to disk.

5.5 Collector Types

Collectors combine parallel partitions of an input Data Set (single link) into a single input stream to a
stage running sequentially. Like partitioning methods, the collector method is defined in the stage
Input/Partitioning properties for any stage running sequentially, when the previous stage is running in
parallel as shown on the right:

Figure 33: Specifying Collector method

5.5.1 Auto Collector

The Auto collector eagerly reads rows from partitions
in the input Data Set without blocking if a row is
unavailable on a particular partition. For this reason,
the order of rows in an Auto collector is undefined,
and may vary between job runs on the same Data Set.
Auto is the default collector method.

5.5.2 Round Robin Collector

The Round Robin collector patiently reads rows from partitions in the input Data Set by reading input
partitions in round robin order. The Round Robin collector is generally slower than an Auto collector
because it must wait for a row to appear in a particular partition.

However, there is a specialized example where the Round Robin collector may be appropriate.
Consider an example where data is read sequentially and passed to a Round Robin partitioner:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 72 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Round Robin Round Robin

partitioner collector

Stage
Sequential Sequential
running in
input output
parallel

Figure 34: RoundRobin Collector example

Assuming the data is not repartitioned within the job flow and that the number of rows is not reduced
(for example, through aggregation), then a Round Robin collector can be used before the final
Sequential output to reconstruct a sequential output stream in the same order as the input data stream.
This is because Round Robin collector reads from partitions using the same partition order that a
Round Robin partitioner assigns rows to parallel partitions.

5.5.3 Ordered Collector

An Ordered collector reads all rows from the first partition, then reads all rows from the next partition
until all rows in the Data Set have been collected.

Ordered collectors are generally only useful if the input Data Set has been Sorted and Range
partitioned on the same key column(s). In this scenario, an Ordered collector will generate a sequential
stream in sort order.

5.5.4 Sort Merge Collector

If the input Data Set is sorted in parallel, the Sort Merge collector will generate a sequential stream of
rows in globally sorted order. The Sort Merge collector requires one or more key column(s) to be
defined, and these should be the same columns, in the same order, as used to sort the input Data Set in
parallel. Row order is undefined for non-key columns.

5.6 Collecting Methodology

Given the options for collecting data into a sequential stream, the following guidelines form a
methodology for choosing the appropriate collector type:
a) When output order does not matter, use Auto partitioning (the default)
b) When the input Data Set has been sorted in parallel, use Sort Merge collector to produce a
single, globally sorted stream of rows
o When the input Data Set has been sorted in parallel and Range partitioned, the Ordered
collector may be more efficient
c) Use a Round Robin collector to reconstruct rows in input order for round-robin partitioned
input Data Sets, as long as the Data Set has not been repartitioned or reduced.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 73 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

6 Sorting
Traditionally, the process of sorting data uses one primary key column and, optionally, one or more
secondary key column(s) to generate a sequential, ordered result set. The order of key columns
determines the sequence and groupings in the result set. Each column is specified with an ascending or
descending sort order. This is the method the SQL databases use for an ORDER BY clause, as
illustrated in the following example, sorting on primary key LName (ascending), secondary key
FName (descending):

Input Data: After Sorting by LName, FName:

I LName FName Address I LName FName Address
D D
1 Ford Henry 66 Edison Avenue 6 Dodge John 75 Boston Boulevard
2 Ford Clara 66 Edison Avenue 5 Dodge Horace 17840 Jefferson
3 Ford Edsel 7900 Jefferson 1 Ford Henry 66 Edison Avenue
4 Ford Eleanor 7900 Jefferson 7 Ford Henry 4901 Evergreen
5 Dodge Horace 17840 Jefferson 4 Ford Eleanor 7900 Jefferson
6 Dodge John 75 Boston Boulevard 10 Ford Eleanor 1100 Lakeshore
7 Ford Henry 4901 Evergreen 3 Ford Edsel 7900 Jefferson
8 Ford Clara 4901 Evergreen 9 Ford Edsel 1100 Lakeshore
9 Ford Edsel 1100 Lakeshore 2 Ford Clara 66 Edison Avenue
10 Ford Eleanor 1100 Lakeshore 8 Ford Clara 4901 Evergreen

However, in most cases there is no need to globally sort data to produce a single sequence of rows.
Instead, sorting is most often needed to establish order within specified groups of data. This sort can be
done in parallel.

For example, the Remove Duplicates stage selects either the first or last row from each group of an
input Data Set sorted by one or more key columns. Other stages (for example, Sort Aggregator, Change
Capture, Change Apply, Join, Merge) require pre-sorted groups of related records.

6.1 Partition and Sort Keys

Using the parallel Sort within DataStage Enterprise Edition:
- Partitioning: is used to gather related records, assigning rows with the same key column values to
the same partition
- Sorting: is used to establish group order within each partition, based on one or more key column(s)

NOTE: By definition, when data is re-partitioned, sort order is not maintained. To restore row
order and groupings, a sort is required after repartitioning.

In the following example, the previous input Data Set is partitioned on LName and FName columns.
Given a 4-node configuration file, we would see the following results:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 74 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Partition 0 Partition 2
ID LName FName Address ID LName FName Address
2 Ford Clara 66 Edison Avenue 4 Ford Eleanor 7900 Jefferson
8 Ford Clara 4901 Evergreen 6 Dodge John 75 Boston Boulevard
10 Ford Eleanor 1100 Lakeshore
Partition 1
ID LName FName Address Partition 3
3 Ford Edsel 7900 Jefferson ID LName FName Address
5 Dodge Horace 17840 Jefferson 1 Ford Henry 66 Edison Avenue
9 Ford Edsel 1100 Lakeshore 7 Ford Henry 4901 Evergreen

Applying a parallel sort to this partitioned input Data Set, using the primary key column LName
(ascending) and secondary key column FName (descending) would generate the resulting Data Set:

Partition 0 Partition 2
ID LName FName Address ID LName FName Address
2 Ford Clara 66 Edison Avenue 6 Dodge John 75 Boston Boulevard
8 Ford Clara 4901 Evergreen 4 Ford Eleanor 7900 Jefferson
10 Ford Eleanor 1100 Lakeshore
Partition 1
ID LName FName Address Partition 3
5 Dodge Horace 17840 Jefferson ID LName FName Address
3 Ford Edsel 7900 Jefferson 1 Ford Henry 66 Edison Avenue
9 Ford Edsel 1100 Lakeshore 7 Ford Henry 4901 Evergreen

Note that the partition and sort keys do not always have to match.

For example, secondary sort keys can be used to establish order within a group for selection with the
Remove Duplicates stage (which can specify First or Last duplicate to retain). Let’s say that an input
Data Set consists of order history based on CustID and Order Date. Using Remove Duplicates, we want
to select the most recent order for a given customer.

To satisfy these requirements we could:

 Partition on CustID to group related records
 Sort on OrderDate in Descending order
 Remove Duplicates on CustID, with Duplicate To Retain=First

12.4.2 Sorting and Hashing Advanced Example provides a more detailed discussion and example of
partitioning and sorting.

6.2 Complete (Total) Sort

If a single, sequential ordered result is needed, in general it is best to use a two step process:
- partition and parallel Sort on key column(s)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 75 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

- use a Sort Merge collector on these same key column(s) to

generate a sequential, ordered result set
This is similar to the way parallel database engines perform their parallel sort operations.

6.3 Link Sort and Sort Stage

DataStage Enterprise Edition provides two methods for parallel sorts – the standalone sort stage (when
execution mode is set to Parallel) and sort on a link (when using a keyed input partitioning method). By
default, both methods use the same internal sort package (the tsort operator).

The Link sort offers fewer options, but is easier to maintain in a DataStage job, as there are fewer
stages on the design canvas. The stand-alone sort offers more options, but as a separate stage makes job
maintenance slightly more complicated.

In general, use the Link sort unless a specific option is needed on the stand-alone Stage. Most often,
the standalone Sort stage is used to specify the Sort Key mode for partial sorts.

6.3.1 Link Sort

Sorting on a link is specified on the Input/Partitioning stage options, when specifying a keyed
partitioning method. (Sorting on a link is not available with Auto partitioning, although the Enterprise
Edition engine may insert a sort if required). When specifying key column(s) for partitioning, the
“Perform Sort” option is checked. Within the Designer canvas, links that have sort defined will have a
Sort icon in addition to the partitioning icon, as shown below:

Figure 35: Link Sort icon

Additional properties can be specified by right-clicking on the key column as shown in the following
illustration:

Key column options let the

developer specify:
- key column usage:
sorting, partitioning, or
both
- sort direction: Ascending
or Descending
- case sensitivity (strings)
- Sorting character set:
ASCII (default) or
EBCDIC (strings)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 76 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

- Position of nulls in the result set (for nullable columns)

Figure 36: Specifying Link Sort options

6.3.2 Sort Stage

The standalone Sort stage offers more options than the sort on a link. Specifically, the following
properties are not available when sorting on a link:

- Sort Key Mode (a

particularly important
performance
optimization)
- Create Cluster Key
Change Column
- Create Key Change
Column
- Output Statistics
- Sort Utility (don’t
change this!)
- Restrict Memory Usage

Figure 37: Sort Stage options

Of the options only available in the standalone Sort stage, the Sort Key Mode is most frequently used.

NOTE: The Sort Utility option is an artifact of previous releases. Always specify “DataStage”
Sort Utility, which is significantly faster than a “UNIX” sort.

6.4 Stable Sort

Stable sorts preserve the order of non-key columns within each sort group. This requires some
additional overhead in the sort algorithm, and thus a stable sort is generally slower than a non-stable
sort for the same input Data Set and sort keys. For this reason, disable Stable sort unless needed.

It is important to note that by default the Stable sort option is disabled for sorts on a link and Enabled
with the standalone Sort stage.

6.5 Sub-Sorts
Within the standalone Sort stage, the key column property “Sort Key Mode” is a particularly powerful
feature and a significant performance optimization. It is used when resorting a sub-grouping of a
previously sorted input Data Set, instead of performing a complete Sort. This “subsort” uses
significantly less disk space and CPU resource, and can often be performed in memory (depending on
the size of the new subsort groups).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 77 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

To resort based on a sub-grouping, all key columns must still

be defined in the Sort stage. Re-used sort keys are specified with the “Don’t Sort (Previously Sorted)”
property, while new sort keys are specified with the “Sort” key mode property, as shown in the
following example:

Figure 38: Sort Key Mode property

To successfully perform a subsort, keys with “Don’t Sort (Previously Sorted)” property must be at the
top of the list, without gaps between them. And, the key column order for these keys must match the
key columns and order defined in the previously-sorted input Data Set.

If the input data does not match the key column definition for a subsort, the job will abort.

6.6 Automatically-Inserted Sorts

By default, DataStage Enterprise Edition inserts sort operators as necessary to ensure correct results.
The parallel job score (see 12.4.2 Understanding the Parallel Job Score) can be used to identify
automatically-inserted sorts, as shown in this score fragment:
op1[4p] {(parallel inserted tsort operator {key={value=LastName},
key={value=FirstName}}(0))
on nodes (
node1[op2,p0]
node2[op2,p1]
node3[op2,p2]
node4[op2,p3]
)}

Typically, Enterprise Edition inserts sorts before any stage that requires matched key values or ordered
groupings of (Join, Merge, Remove Duplicates, Sort Aggregator). Sorts are only inserted automatically
when the flow developer has not explicitly defined an input sort.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 78 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

While ensuring correct results, inserted sorts can be a

significant performance impact if they are not necessary.
There are two ways to prevent Enterprise Edition from inserting an un-necessary sort:
a) Insert an upstream Sort stage on each link, define all sort key columns with the Sort Mode
key property “Don’t Sort (Previously Sorted)”
b) Set the environment variable APT_SORT_INSERTION_CHECK_ONLY. This will verify sort
order but not actually perform a sort, aborting the job if data is not in the required sort order.

Revisiting the partitioning examples in Section 5.4: Partitioning Examples, the environment
variable $APT_SORT_INSERTION_CHECK_ONLY should be set to prevent Enterprise
Edition from inserting un-necessary sorts before the Join stage.

6.7 Sort Methodology

Using the rules and behavior outlined in the previous section, the following methodology should be
applied when sorting in a DataStage Enterprise Edition data flow:

a) Start with a link sort

b) Specify only necessary key column(s)
c) Don’t use Stable Sort unless needed
d) Use a stand-alone Sort stage instead of a Link sort for options that not available on a Link sort:
 Sort Key Mode, Create Cluster Key Change Column, Create Key Change Column, Output
Statistics
 Always specify “DataStage” Sort Utility for standalone Sort stages
 Use the “Sort Key Mode=Don’t Sort (Previously Sorted)” to resort a sub-grouping of a
previously-sorted input Data Set
e) Be aware of automatically-inserted sorts
 Set $APT_SORT_INSERTION_CHECK_ONLY to verify but not establish required sort
order
f) Minimize the use of sorts within a job flow
g) To generate a single, sequential ordered result set use a parallel Sort and a Sort Merge collector

6.8 Tuning Sort

Sort is a particularly expensive task within DataStage Enterprise Edition which requires CPU, memory,
and disk resources.

To perform a sort, rows in the input Data Set are read into a memory buffer on each partition. If the sort
operation can be performed in memory (as is often the case with a subsort) then no disk I/O is
performed.

By default, each sort uses 20MB of memory per partition for its memory buffer. This value can be
changed for each standalone Sort stage using the “Restrict Memory Usage” option (the minimum is
1MB/partition). On a global basis, the environment variable APT_TSORT_STRESS_BLOCKSIZE can be

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 79 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

use to specify the size of the memory buffer, in MB, for all
sort operators (link and standalone), overriding any per-sort
specifications.

If the input Data Set cannot fit into the sort memory buffer, then results are temporarily spooled to disk
in the following order:
- scratch disks defined in the current configuration file (APT_CONFIG_FILE) in the “sort” named disk
pool
- scratch disks defined in the current configuration file default disk pool
- the default directory specified by the environment variable TMPDIR
- the directory “/tmp” (on UNIX) or “C:/TMP” (on Windows) if available

The file system configuration and number of scratch disks defined in parallel configuration file can
greatly impact the I/O performance of a parallel sort. Having a greater number of scratch disks for each
node allows the sort to spread I/O across multiple file systems.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 80 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

7 File Stage Usage

7.1 Which File Stage to Use
DataStage/EE offers various stages for reading from and writing to files. Recommendations for when
to use a particular stage, and any limitations, are summarized below:

File Stage Recommended Usage Limitations

Read and write standard files in a Cannot write to a single file in
single format. parallel, performance penalty
of conversion, does not support
Sequential File hierarchical data files.
Need to read source data in complex Cannot write in parallel;
(hierarchical) format, such as performance penalty of format
mainframe sources with COBOL conversion.
Complex Flat File copybook file definitions.
Intermediate storage between Can only be read from and
DataStage parallel jobs. written to by DataStage parallel
Data Set jobs or orchadmin command.
Need to share information with Slightly higher overhead than
external applications, can write in Data Set.
parallel (generates multiple segment
File Set
files).
Need to share data with an external Requires Parallel SAS, can
Parallel SAS application. (Requires only be read from / written to
SAS connectivity license for by DS/EE or Parallel SAS.
SAS Parallel Data Set
DataStage.)
Rare instances where lookup Can only be written – contents
reference data is required by multiple cannot be read or verified. Can
jobs and is not updated frequently. only be used as reference link
Lookup File Set
on a Lookup stage.

No DS/EE file stage supports “update” of existing records. Some stages (parallel Data Set) support
“Append” to add new records to an existing file, but this is not recommended as it imposes risks for
failure recovery.

7.2 Data Set Usage

Parallel Data Sets are the persistent (on-disk) representation of the in-memory data structures of
DS/EE. As such, Data Sets store data in partitioned form, using the internal format of the parallel
engine. In general, Data Sets provide maximum performance for reading and writing data from disk, as
no overhead is needed to translate data to the internal DS/EE representation.

However, Data Sets can only be read from and written to using a DataStage parallel job. If data is

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 81 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

7.3 Sequential File Stages (Import and

Export)
The Sequential File stage can be used to read from or write to one or more flat files of the same
format. Unlike the Complex Flat File stage, the Sequential File stage can only read and write data that
is in flattened (row/column) format.

7.3.1 Reading from a Sequential File in Parallel

The ability to read Sequential File(s) in parallel within Enterprise Edition depends on the Read
Method and the options specified:

Sequential File – options to read sequentially:

Read Method: Specific Files, only one file specified
may be a file or named pipe
Read Method: File Pattern

Sequential File – options to read in parallel:

Read Method: Specific Files, only one file specified, Readers Per Node option greater than 1
useful for SMP configurations - file may be either fixed or variable-width
Read Method: Specific Files, more than one file specified
each file specified within a single Sequential File stage must be of the same format
Read Method: File Pattern, set environment variable
$APT_IMPORT_PATTERN_USES_FILESET
Read Method: Specific Files, Read From Multiple Nodes option is set to Yes
useful for cluster and Grid configurations - file may only be fixed-width

Note that when reading in parallel, input row order is not maintained across readers.

7.3.2 Writing to a Sequential File in Parallel

It is only possible to write in parallel from a Sequential File stage when more than one output file is
specified. In these instances, the degree of parallelism of the write will correspond to the number of file
names specified.

A better option for writing to a set of Sequential Files in parallel is to use the FileSet stage. This will
create a single header file (in text format) and corresponding data files, in parallel, using the format
options specified in the FileSet stage. The FileSet stage will write in parallel.

7.3.3 Separating I/O from Column Import

If the Sequential File input cannot be read in parallel, performance can still be improved by separating
the file I/O from the column parsing operation. As shown in the following Job fragment, define a single
large string column for the non-parallel Sequential File read, and then pass this to a Column Import
stage to parse the file in parallel. The formatting and column properties of the Column Import stage
match those of the Sequential File stage.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 82 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 39: Column Import example

Note that this method is also useful for External Source and FTP sequential source stages.

7.3.4 Partitioning Sequential File Reads

Care must be taken to choose the appropriate partitioning method from a Sequential File read:

• Don’t read from Sequential File using SAME partitioning in the downstream stage! Unless
more than one source file is specified, SAME will read the entire file into a single partition,
making the entire downstream flow run sequentially (unless it is later repartitioned).

• When multiple files are read by a single Sequential File stage (using multiple files, or by using
a File Pattern), each file’s data is read into a separate partition. It is important to use ROUND-
ROBIN partitioning (or other partitioning appropriate to downstream components) to evenly
distribute the data in the flow.

7.3.5 Sequential File (Export) Buffering

By default, the Sequential File (export operator) stage buffers its writes to optimize performance. When
a job completes successfully, the buffers are always flushed to disk. The environment variable
$APT_EXPORT_FLUSH_COUNT allows the job developer to specify how frequently (in number of
rows) that the Sequential File stage flushes its internal buffer on writes.

Setting this value to a low number (such as 1) is useful for realtime applications, but there is a small
performance penalty associated with increased I/O. It is also important to remember that this setting
will apply to all Sequential File stages in the data flow.

7.3.6 Parameterized Sequential File Format

The Sequential File stage supports a Schema File option to specify the column definitions and file
format of the source file. Using the Schema File option allows the format of the source file to be
specified at runtime, instead of statically through Table Definitions.

The format of the Schema File, including Sequential File import / export format properties is
documented in the Orchestrate Record Schema manual. Note that this document is required, since the
Import / Export properties used by the Sequential File and Column Import stages are not documented
in the DataStage Parallel Job Developer’s Guide.

7.3.7 Reading and Writing Nullable Columns

When reading from or writing to Sequential Files or File Sets, the in-band (value) must be explicitly
defined in the extended column attributes for each Nullable column, as shown below:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 83 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 40: Extended Column Metadata (Nullable properties)

7.3.8 Reading from and Writing to Fixed-Length Files

Particular attention must be taken when processing fixed-length fields using the Sequential File stage:
• If the incoming columns are variable-length data types (for example, Integer, Decimal,
Varchar), the field width column property must be set to match the fixed-width of the input
column. Double-click on the column number in the grid dialog to set this column property.

• If a field is nullable, you must define the null field value and length in the Nullable section of
the column property. Double-click on the column number in the grid dialog or right mouse click
on the column and select edit column to set these properties.

• When writing fixed-length files from variable-length fields (eg. Integer, Decimal, Varchar), the
field width and pad string column properties must be set to match the fixed-width of the output
column. Double-click on the column number in the grid dialog to set this column property.

• To display each field value, use the print_field import property. Use caution when specifying
this option as it can generate an enormous amount of detail in the job log. All import and
export properties are listed in the Import/Export Properties chapter of the Orchestrate
Operators Reference.

7.3.9 Reading Bounded-Length VARCHAR Columns

Care must be taken when reading delimited, bounded-length Varchar columns (Varchars with the
length option set). By default, if the source file has fields with values longer than the maximum

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 84 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Varchar length, these extra characters will be silently

truncated.

The environment variable $APT_IMPORT_REJECT_STRING_FIELD_OVERRUNS will direct

Enterprise Edition to reject records with strings longer than their declared maximum column length.

7.3.10 Tuning Sequential File Performance

On heavily-loaded file servers or some RAID/SAN array configurations, the environment variables
$APT_IMPORT_BUFFER_SIZE and $APT_EXPORT_BUFFER_SIZE can be used to improve
I/O performance. These settings specify the size of the read (import) and write (export) buffer size in
Kbytes, with a default of 128 (128K). Increasing this size may improve performance.

Finally, in some disk array configurations, setting the environment variable

$APT_CONSISTENT_BUFFERIO_SIZE to a value equal to the read/write size in bytes can
significantly improve performance of Sequential File operations.

7.4 Complex Flat File Stage

The Complex Flat File (CFF) stage can be used to read or write one or more files in the same
hierarchical format. When used as a source, the stage allows you to read data from one or more
complex flat files, including MVS datasets with QSAM and VSAM files. A complex flat file may
contain one or more GROUPs, REDEFINES, or OCCURS clauses. Complex Flat File source stages
execute in parallel mode when they are used to read multiple files, but you can configure the stage to
execute sequentially if it is only reading one file with a single reader.

NOTE: The Complex Flat File stage cannot read from sources with OCCURS DEPENDING
ON clauses. (This is an error in the DataStage documentation.)

When used as a target, the stage allows you to write data to one or more complex flat files. It does not
write to MVS datasets.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 85 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

7.4.1 CFF Stage Data Type Mapping

When you work with mainframe data using the CFF stage, the data types are mapped to internal
Enterprise Edition data types as follows:

COBOL Type Description Size Internal Type Internal Options

S9(1-4) binary, native binary 2 bytes int16
COMP/COMP-5
S9(5-9) binary, native binary 4 bytes int32
COMP/COMP-5
S9(10-18) binary, native binary 2 bytes int64
COMP/COMP-5
9(1-4) binary, native binary 2 bytes uint16
COMP/COMP-5
9(5-9) binary, native binary 4 bytes uint32
COMP/COMP-5
9(10-18) binary, native binary 8 bytes uint64
COMP/COMP-5
X(n) character n bytes string(n)
X(n) character for filler n bytes raw(n)
X(n) varchar n bytes string(max=n)
9(x)V9(y)COMP-3 decimal (x+y)/2+1 bytes decimal[x+y,y] packed
S9(x)V9(y)COMP-3 decimal (x+y)/2+1 bytes decimal[x+y,y] packed
9(x)V9(y) display_numeric x+y bytes decimal[x+y,y] or zoned
string[x+y]
S9(x)V9(y) display_numeric x+y bytes decimal[x+y,y] or zoned, trailing
string[x+y]
S9(x)V9(y) SIGN IS display_numeric x+y bytes decimal[x+y,y] zoned, trailing
TRAILING
S9(x)V9(y) SIGN IS display_numeric x+y bytes decimal[x+y,y] zoned, leading
LEADING
S9(x)V9(y) SIGN IS display_numeric x+y+1 bytes decimal[x+y,y] separate, trailing
TRAILING SEPARATE
S9(x)V9(y) SIGN IS display_numeric x+y+1 bytes decimal[x+y,y] separate, leading
LEADING SEPARATE
COMP-1 float 4 bytes sfloat
COMP-2 float 8 bytes dfloat
N(n) or G(n) DISPLAY-1 graphic_n, graphic_g n*2 bytes ustring[n]
N(n) or G(n) DISPLAY-1 vargraphic_g/n n*2 bytes ustring[max=n]
Group subrec

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 86 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

8 Transformation Languages
8.1 Transformer Stage
The DataStage Enterprise Edition parallel Transformer stage generates “C” code which is then
compiled into a parallel component. For this reason, it is important to minimize the number of
transformers, and to use other stages (such as Copy) when derivations are not needed.

See Section 3.8.4: Parallel Transformer stages for guidelines on Transformer stage usage.

8.1.1 Transformer NULL Handling and Reject Link

When evaluating expressions for output derivations or link constraints, the
Transformer will reject (through the reject link indicated by a dashed line)
any row that has a NULL value used in the expression. When rows are
rejected by a Transformer, entries are placed in the Director job log.

Always include reject links in a parallel Transformer. This makes it easy

to identify reject conditions (by row counts). To create a Transformer reject
link in Designer, right-click on an output link and choose “Convert to
Reject:
Figure 41: Transformer Reject link

The parallel Transformer rejects NULL derivation results (including output link constraints) because
the rules for arithmetic and string handling of NULL values are, by definition, undefined. Even if the
target column in an output derivation allows nullable results, the Transformer will reject the row
instead of sending it to the output link(s).

For this reason, if you intend to use a nullable column within a Transformer derivation or output link
constraint, it should be converted from its out-of-band (internal) null representation to an in-band
(specific value) null representation using stage variables or the Modify stage.

For example, the following stage variable expression would convert a null value to a specific empty
string:

If ISNULL(link.col) Then “” Else link.col

Note that if an incoming column is only used in an output column mapping, the Transformer will allow
this row to be sent to the output link without being rejected.

8.1.2 Parallel Transformer System Variables

The system variable @ROWNUM behaves differently in the Enterprise Edition Transformer stage than
in the Server Edition Transformer. Because the DS/EE Transformer runs in parallel, @ROWNUM is

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 87 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

assigned to incoming rows for each partition. When

generating a sequence of numbers in parallel, or performing
parallel derivations, the system variables @NUMPARTITIONS and @PARTITIONNUM should be
used.

8.1.3 Transformer Derivation Evaluation

Output derivations are evaluated before any type conversions on the assignment. For example, the
PadString function uses the length of the source type, not the target. Therefore, it is important to make
sure the type conversion is done before a row reaches the Transformer.

For example, TrimLeadingTrailing(string) works only if string is a VarChar field. Thus, the
incoming column must be type VarChar before it is evaluated in the Transformer.

8.1.4 Conditionally Aborting Jobs

The Transformer can be used to conditionally abort a job when incoming data matches a specific rule.
Create a new output link that will handle rows that match the abort rule. Within the link constraints
dialog box, apply the abort rule to this output link, and set the “Abort After Rows” count to the number
of rows allowed before the job should be aborted (for example, 1).

Since the Transformer will abort the entire job flow immediately, it is possible that valid rows will not
have been flushed from Sequential File (export) buffers, or committed to database tables. It is
important to set the database commit parameters or adjust the Sequential File buffer settings (see
Section 7.3.5: Sequential File (Export) Buffering).

8.1.5 Transformer Decimal Arithmetic

When decimal data is evaluated by the Transformer stage, there are times when internal decimal
variables need to be generated in order to perform the evaluation. By default, these internal decimal
variables will have a precision and scale of 38 and 10.

If more precision is required, the environment variables APT_DECIMAL_INTERM_PRECISION and

APT_DECIMAL_INTERM_SCALE can be set to the desired range, up to a maximum precision of 255 and
scale of 125. By default, internal decimal results are rounded to the nearest applicable value. The
environment variable APT_DECIMAL_INTERM_ROUND_MODE can be used to change the rounding behavior
using one of the following keywords:

ceil
Rounds towards positive infinity. Examples: 1.4 -> 2, -1.6 -> -1

floor
Rounds towards negative infinity. Examples: 1.6 ->1, -1.4 -> -2

round_inf
Rounds or truncates towards nearest representable value, breaking ties by rounding positive values toward positive infinity and
negative values toward negative infinity.
Examples: 1.4 -> 1, 1.5-> 2, -1.4 -> -1, -1.5 -> -2

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 88 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

trunc_zero
Discard any fractional digits to the right of the rightmost fractional digit supported regardless of sign. For example, if
$APT_DECIMAL_INTERM_SCALE is smaller than the results of the internal calculation, round or truncate to the scale size.
Examples: 1.56 -> 1.5, -1.56 ->-1.5.

8.1.6 Optimizing Transformer Expressions and Stage Variables

In order to write efficient Transformer stage derivations, it is useful to understand what items get
evaluated and when. The evaluation sequence is as follows:
Evaluate each stage variable initial value

For each input row to process:

Evaluate each stage variable derivation value,

unless the derivation is empty

For each output link:

Evaluate the link constraint; if true

Evaluate each column derivation value

Write the output record

Else skip the link

Next output link

Next input row

The stage variables and the columns within a link are evaluated in the order in which they are displayed
in the Transformer editor. Similarly, the output links are also evaluated in the order in which they are
displayed.

From this sequence, it can be seen that there are certain constructs that would be inefficient to include
in output column derivations, as they would be evaluated once for every output column that uses them.
Such constructs are:

• Where the same part of an expression is used in multiple column derivations

For example, suppose multiple columns in output links want to use the same substring of an
input column, then the following test may appear in a number of output columns derivations:
IF (DSLINK1.col[1,3] = “001”) THEN ...

In this case, the evaluation of the substring of DSLINK1.col[1,3] is evaluated for each column
that uses it.

This can be made more efficient by moving the substring calculation into a stage variable. By
doing this, the substring is evaluated just once for every input row. In this case, the stage
variable definition would be:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 89 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

DSLINK1.col1[1,3]

and each column derivation would start with:

IF (StageVar1 = “001” THEN ...

In fact, this example could be improved further by also moving the string comparison into the
stage variable. The stage variable would be:
IF (DSLink1.col[1,3] = “001” THEN 1 ELSE 0

and each column derivation would start with:

IF (StageVar1) THEN

This reduces both the number of substring functions evaluated and string comparisons made in
the Transformer.

• Where an expression includes calculated constant values

For example, a column definition may include a function call that returns a constant value, such
as:
Str(“ “,20)

This returns a string of 20 spaces. In this case, the function would be evaluated every time the
column derivation is evaluated. It would be more efficient to calculate the constant value just
once for the whole Transformer.

This can be achieved using stage variables. This function could be moved into a stage variable
derivation; but in this case, the function would still be evaluated once for every input row. The
solution here is to move the function evaluation into the initial value of a stage variable.

A stage variable can be assigned an initial value from the Stage Properties dialog/Variables tab
in the Transformer stage editor. In this case, the variable would have its initial value set to:
Str(“ “,20)

You would then leave the derivation of the stage variable on the main Transformer page empty.
Any expression that previously used this function would be changed to use the stage variable
instead.

The initial value of the stage variable is evaluated just once, before any input rows are
processed. Then, because the derivation expression of the stage variable is empty, it is not re-
evaluated for each input row. Therefore, its value for the whole Transformer processing is
unchanged from the initial value.

In addition to a function value returning a constant value, another example would be part of an
expression such as:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 90 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

"abc" : "def"

As with the function-call example, this concatenation is evaluated every time the column
derivation is evaluated. Since the subpart of the expression is actually constant, this constant
part of the expression could again be moved into a stage variable, using the initial value setting
to perform the concatenation just once.

• Where an expression requiring a type conversion is used as a constant, or it is used in multiple

places.
For example, an expression may include something like this:
DSLink1.col1+"1"

In this case, the "1" is a string constant, and so, in order to be able to add it to DSLink1.col1, it
must be converted from a string to an integer each time the expression is evaluated. The
solution in this case is just to change the constant from a string to an integer:
DSLink1.col1+1

In this example, if DSLINK1.col1 were a string field, then, again, a conversion would be
required every time the expression is evaluated. If this just appeared once in one output column
expression, this would be fine. However, if an input column is used in more than one
expression, where it requires the same type conversion in each expression, then it would be
more efficient to use a stage variable to perform the conversion once. In this case, you would
create, for example, an integer stage variable, specify its derivation to be DSLINK1.col1, and
then use the stage variable in place of DSLink1.col1, where that conversion would have been
required.

It should be noted that when using stage variables to evaluate parts of expressions, the data type
of the stage variable should be set correctly for that context. Otherwise, needless conversions
are required wherever that variable is used.

8.2 Modify Stage

The Modify stage is the most efficient “stage” available, since it uses low-level functionality that is part
of every DataStage Enterprise Edition component.

As noted in the previous section, the Output Mapping properties for any parallel stage will generate an
underlying modify for default data type conversions, dropping and renaming columns.

The standalone Modify stage can be used for non-default type conversions (nearly all date and time
conversions are non-default), null conversion, and string trim. The Modify stage uses the syntax of the
underlying modify operator, documented in the Parallel Job Developers Guide as well as the
Orchestrate Operators Reference.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 91 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

8.2.1 Modify and Null Handling

The Modify stage can be used to convert an out-of-band null value to an in-band null representation
and vice-versa.

NOTE: The DataStage Parallel Job Developers Guide gives incorrect syntax for converting an
out-of-band null to an in-band null (value) representation.

To convert from an out-of-band null to an in-band null (value) representation within Modify, the syntax
is: destField[:dataType] = make_null(sourceField,value)
where:
- destField is the destination field’s name.
- dataType is its optional data type; use it if you are also converting types.
- sourceField is the source field’s name.
- value is the value of the source field when it is null.

To convert from an in-band null to an out-of-band null, the syntax is:

destField[:dataType] = handle_null (sourceField,value)

where:
- destField is the destination field’s name.
- dataType is its optional data type; use it if you are also converting types.
- sourceField is the source field’s name
- value is the value you wish to represent a null in the output.

The destField is converted from an Orchestrate out-of-band null to a value of the field’s data type. For
a numeric field value can be a numeric value, for decimal, string, time, date, and timestamp fields,
value can be a string.

8.2.2 Modify and String Trim

The function string_trim has been added to Modify, with the following syntax:

stringField=string_trim[character, direction, justify] (string)

You can use this function to remove the characters used to pad variable-length strings when they are
converted to fixed-length strings of greater length. By default, these characters are retained when the
fixed-length string is then converted back to a variable-length string.

The character argument is the character to remove. By default, this is NULL. The value of the direction
and justify arguments can be either begin or end; direction defaults to end, and justify defaults to begin.
Justify has no affect when the target string has variable length.

The following example removes all leading ASCII NULL characters from the beginning of name and
places the remaining characters in an output variable-length string with the same name:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 92 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

name:string = string_trim[NULL, begin](name)

The following example removes all trailing Z characters from color, and left-justifies the resulting hue
fixed-length string:
hue:string[10] = string_trim[‘Z’, end, begin](color)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 93 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

9 Combining Data
9.1 Lookup vs. Join vs. Merge
The Lookup stage is most appropriate when the reference data for all lookup stages in a job is small
enough to fit into available physical memory. Each lookup reference requires a contiguous block of
shared memory. If the Data Sets are larger than available memory resources, the JOIN or MERGE
stage should be used.

Limit the use of database Sparse Lookups (available in the DB2 Enterprise, Oracle Enterprise, and
ODBC Enterprise stages) to scenarios where the number of input rows is significantly smaller (for
example, 1:100 or more) than the number of reference rows. (see Section 10.1.7: Database Sparse
Lookup vs. Join).

Sparse Lookups may also be appropriate for exception-based processing when the number of
exceptions is a small fraction of the main input data. It is best to test both the Sparse and Normal to see
which actually performs best, and to retest if the relative volumes of data change dramatically.

9.2 Capturing Unmatched Records from a Join

The Join stage does not provide reject handling for unmatched records (such as in an InnerJoin
scenario). If un-matched rows must be captured or logged, an OUTER join operation must be
performed. In an OUTER join scenario, all rows on an outer link (for example, Left Outer, Right
Outer, or both links in the case of Full Outer) are output regardless of match on key values.

During an Outer Join, when a match does not occur, the Join stage inserts values into the unmatched
non-key column(s) using the following rules:
a) If the non-key column is defined as nullable (on the Join input links) then Enterprise Edition
will insert NULL values in the unmatched columns
b) If the non-key column is defined as not-nullable, then Enterprise Edition inserts “default”
values based on the data type. For example, the default value for an Integer is zero, the default
value for a Varchar is an empty string (“”), and the default value for a Char is a string of
padchar characters equal to the length of the Char column.

For this reason, care must be taken to change the column properties to allow NULL values
before the Join. This is most easily done by inserting a Copy stage and mapping a column from
NON-NULLABLE to NULLABLE.

A Transformer stage can be used to test for NULL values in unmatched columns.

In most cases, it is best to use a Column Generator to add an ‘indicator’ column, with a constant
value, to each of the inner links and test that column for the constant after you have performed the join.
This isolates your match/no-match logic from any changes in the metadata. This is also handy with
Lookups that have multiple reference links.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 94 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

9.3 The Aggregator Stage

9.3.1 Aggregation Method
By default, the default Aggregation Method is set to Hash, which maintains the results of each key-
column value/aggregation pair in memory. Because each key value/aggregation requires approximately
2K of memory, the Hash Aggregator should only be used when the number of distinct key values is
small and finite.

The Sort Aggregation Method should be used when the number of key values is unknown or very
large. Unlike the Hash Aggregator, the Sort Aggregator requires presorted data, but only maintains the
calculations for the current group in memory.

9.3.2 Aggregation Data Type

By default, the output data type of a parallel Aggregator stage calculation or recalculation column is
floating point (Double). To aggregate in decimal precision, set the optional property
“Aggregations/Default to Decimal Output” within the Aggregator stage.

You can also specify that the result of an individual calculation or recalculation is decimal by using the
optional “Decimal Output” sub-property.

Note that performance is typically better if you let calculations occur in floating point (Double) data
type and convert the results to decimal downstream in the flow. An exception to this is financial
calculations which should be done in decimal to preserve appropriate precision.

9.3.3 Performing Total Aggregations

The Aggregator counts and calculates based on distinct key value groupings. To perform a total
aggregation, use the stages shown on the right to:

- generate a single constant-value key column

using the Column Generator or an upstream
Transformer

- aggregate in parallel on the generated column (partition Round Robin, aggregate on

generated key column)
there is no need to sort or hash-partition the input data with only one key column value

- aggregate Sequentially on the generated column

Note that in this example use two Aggregators are used to prevent the sequential aggregation from
disrupting upstream processing.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 95 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10 Database Stage Guidelines

10.1 Database development overview
This section is intended to provide guidelines appropriate to accessing any database within DataStage
Enterprise Edition. Subsequent sections provide database-specific tips and guidelines.

10.1.1 Database stage types

DataStage Enterprise Edition offers database connectivity through native parallel and plug-in stage
types. For some databases (DB2, Informix, Oracle, and Teradata), multiple stage types are available:
Teradata MultiLoad (TPump)
Native Parallel Database Stages
DB2/UDB Enterprise
Informix Enterprise
ODBC Enterprise
Oracle Enterprise
SQL Server Enterprise
Teradata Enterprise

NOTE: Not all database stages (for

example, Teradata API) are visible in
the default DataStage Designer palette.
You may need to customize the palette
to add hidden stages.

Plug-In Database Stages

Dynamic RDBMS
DB2/UDB API
DB2/UDB Load
Informix CLI
Informix Load
Informix XPS Load
Oracle OCI Load
RedBrick Load
Sybase IQ12 Load
Sybase OC
Teradata API
Teradata MultiLoad (MultiLoad)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 96 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.1.1.1 Native Parallel database stages

In general, for maximum parallel performance, scalability, and features it is best to use the
native parallel database stages in a job design if connectivity requirements can be satisfied.

Because there are exceptions to this rule (especially with Teradata), specific guidelines of when to
use various stage types are provided in the database-specific topics in this section.

Because of their tight integration with database technologies, the native parallel stages often have more
stringent connectivity requirements than plug-in stages. For example, the DB2/UDB Enterprise stage is
only compatible with DB2 Enterprise Server Edition with DPF on the same UNIX platform as the
DataStage server.

Native parallel stages always pre-query the database for actual runtime metadata (column names, types,
attributes). This allows Enterprise Edition to match return columns by name, not position in the stage
Table Definitions. However, care must be taken to assign the correct data types in the job design.

10.1.1.2 ODBC Enterprise stage

In general, native database components (such as the Oracle Enterprise stage) are preferable to ODBC
connectivity if both are supported on the database platform, operating system, and version. Unlike the
database-specific parallel stages, the ODBC Enterprise stage cannot read in parallel (although a patch
to allow parallel read may be available on some platforms through IBM IIS Support). Furthermore, the
ODBC Enterprise stage cannot interface with database-specific parallel load technologies.

The benefit of ODBC Enterprise stage comes from the large number of included and third party ODBC
drivers to enable connectivity to all major database platforms. ODBC also provides an increased level
of “data virtualization” which can be useful when sources and targets (or deployment platforms) can
change.

DataStage Enterprise Edition bundles OEM versions of ODBC drivers from DataDirect. On UNIX, the
DataDirect ODBC Driver Manager is also included. “Wire Protocol” ODBC Drivers generally do not
require database client software to be installed on the server platform.

10.1.1.3 Plug-In database stages

Plug-in stage types are intended to provide connectivity to database configurations not offered by the
native parallel stages. Because plug-in stage types cannot read in parallel, and cannot span multiple
servers in a clustered or Grid configuration, they should only be used when it is not possible to use a
native parallel stage.

From a design perspective, plug-in database stages match columns by order, not name, so Table
Definitions must match the order of columns in a query.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 97 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.1.2 Database Metadata

10.1.2.1 Runtime metadata

At runtime, the DS/EE native parallel database stages always “pre-query” the database source or target
to determine the actual metadata (column names, data types, nullability) and partitioning scheme (in
some cases) of the source or target table.

For each native parallel database stage:

- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- both DS/EE and relational databases support null values, and a null value in a database
column is stored as an out-of-band NULL value in the DS/EE column

The actual metadata used by a DS/EE native parallel database stage is always determined at runtime,
regardless of the table definitions assigned by the DataStage developer. This allows the database stages
to match return values by column name instead of position. However, care must be taken that the
column data types defined by the DataStage developer match the data types generated by the database
stage at runtime. Database-specific data type mapping tables are included in the following sections.

10.1.2.2 Metadata Import

When using the native parallel DB2 Enterprise, Informix Enterprise or Oracle Enterprise stages,
use orchdbutil to import metadata to avoid type conversion issues. This utility is available as a server
command line utility and within Designer and Manager using “Import Orchestrate Schema
Definitions”, and selecting “Import from Database Table” option in the wizard as illustrated below:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 98 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 42: orchdbutil metadata import

One disadvantage to the graphical orchdbutil metadata import is that the user interface requires each
table to be imported individually.

When importing a large number of tables, it will be easier to use the corresponding orchdbutil
command-line utility from the DataStage server machine. As a command, orchdbutil can be scripted to
automate the process of importing a large number of tables.

10.1.2.3 Defining Metadata for Database Functions

When using database functions within a SQL SELECT list in a Read or Lookup, it is important to use
SQL aliases to explicitly name the calculated columns so that they can be referenced within the
DataStage job. The alias name(s) should then be added to the Table Definition within DataStage.

For example, the following SQL assigns the alias Total to the calculated column:

SELECT store_name, SUM(sales) Total

FROM store_info
GROUP BY store_name

Note that in many cases it may be more appropriate to aggregate using the Enterprise Edition
Aggregator stage. However, there may be cases where user-defined functions or logic need to be
executed on the database server.

10.1.3 Optimizing Select Lists

For best performance and optimal memory usage, it is best to explicitly specify column names on all
source database stages, instead of using an unqualified “Table” or SQL “SELECT *” read. For “Table”
read method, always specify the “Select List” subproperty. For “Auto-Generated” SQL, the DataStage
Designer will automatically populate the select list based on the stage’s output column definition.

The only exception to this rule is when building dynamic database jobs that use runtime column
propagation to process all columns in a source table.

10.1.4 Testing Database Connectivity

The “View Data” button on the Output / Properties tab of source database stages lets you verify
database connectivity and settings without
having to create and run a job. Test the
connection using View Data button. If the
connection is successful, you will see a
window with the result columns and data,
similar to the illustration on the right:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 99 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 43: Sample View Data Output

If the connection fails, an error message may appear, and you will be prompted to view additional
detail. Clicking YES will display a detailed dialog box with the specific error messages generated by
the database stage that can be very useful in debugging a database connection failure.

Figure 44: View Additional Error Detail

10.1.5 Designing for Restart

To enable restart of high-volume jobs, it is important to separate the transformation process from the
database write (Load or Upsert) operation. After transformation, the results should be landed to a
parallel Data Set. Subsequent job(s) should read this Data Set and populate the target table using the
appropriate database stage and write method.

As a further optimization, a Lookup stage (or Join stage, depending on data volume) can be used to
identify existing rows before they are inserted into the target table.

10.1.6 Database OPEN and CLOSE Commands

The native parallel database stages provide options for specifying OPEN and CLOSE commands.
These options allow commands (including SQL) to be sent to the database before (OPEN) or after
(CLOSE) all rows are read/written/loaded to the database. OPEN and CLOSE are not offered by plug-
in database stages.

For example, the OPEN command could be used to create a temporary table, and the CLOSE command
could be used to select all rows from the temporary table and insert into a final target table.

As another example, the OPEN command can be used to create a target table, including database-
specific options (tablespace, logging, constraints, etc) not possible with the “Create” option. In general,
it is not a good idea to let DataStage generate target tables unless they are used for temporary storage.
There are limited capabilities to specify Create table options in the stage, and doing so may violate
data-management (DBA) policies.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 100 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

It is important to understand the implications of specifying a

user-defined OPEN and CLOSE command. For example, when reading from DB2, a default OPEN
statement places a shared lock on the source. When specifying a user-defined OPEN command, this
lock is not sent – and should be specified explicitly if appropriate.

Further details are outlined in the respective database sections of the Orchestrate Operators Reference
which is part of the Orchestrate OEM documentation.

10.1.7 Database Sparse Lookup vs. Join

Data read by any database stage can serve as the reference input to a Lookup operation. By default, this
reference data is loaded into memory like any other reference link (“Normal” Lookup).

When directly connected as the reference link to a Lookup stage, the DB2/UDB Enterprise, ODBC
Enterprise, and Oracle Enterprise stages allow the lookup type to be changed to “Sparse”, sending
individual SQL statements to the reference database for each incoming Lookup row. Sparse Lookup is
only available when the database stage is directly connected to the reference link, with no intermediate
stages.

IMPORTANT: The individual SQL statements required by a “Sparse” Lookup are an expensive
operation from a performance perspective. In most cases, it is faster to use a DataStage JOIN
stage between the input and DB2 reference data than it is to perform a “Sparse” Lookup.

For scenarios where the number of input rows is significantly smaller (for example, 1:100 or more)
than the number of reference rows in a DB2 or Oracle table, a Sparse Lookup may be appropriate.

10.1.8 Appropriate Use of SQL and DataStage

When using relational database sources, there is often a functional overlap between SQL and DataStage
functionality. Although it is possible to use either SQL or DataStage to solve a given business problem,
the optimal implementation involves leveraging the strengths of each technology to provide maximum
throughput and developer productivity.

While there are extreme scenarios when the appropriate technology choice is clearly understood, there
may be “gray areas” where the decision should be made based on factors such as developer
productivity, metadata capture and re-use, and ongoing application maintenance costs.

The following guidelines can assist with the appropriate use of SQL and DataStage technologies in a
given job flow:

• When possible, use a SQL filter (WHERE clause) to limit the number of rows sent to
the DataStage job. This minimizes impact on network and memory resources, and
leverages the database capabilities.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 101 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

• Use a SQL Join to combine data from tables

with a small number of rows in the same
database instance, especially when the join columns are indexed. A join that reduces the
result set significantly is also often appropriate to do in the database.

• When combining data from very large tables, or when the source includes a large
number of database tables, the efficiency of the Enterprise Edition Sort and Join stages
can be significantly faster than an equivalent SQL query. In this scenario, it can still be
beneficial to use database filters (WHERE clause) if appropriate.

• Avoid the use of database stored procedures (for example, Oracle PL/SQL) on a per-row
basis within a high-volume data flow. For maximum scalability and parallel
performance, it is best to implement business rules using native parallel DataStage
components.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 102 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.2 DB2 Guidelines

10.2.1 DB2 Stage Types
DataStage Enterprise Edition provides access to DB2 databases using one of 5 stages, summarized in
the following table:

DataStage Stage DB2 Requirement Supports Parallel Parallel Parallel SQL

Stage Name Type Partitioned Read? Write? Sparse Open /
DB2? Lookup Close
DB2/UDB Native DPF, same Yes /
Enterprise Parallel platform as ETL directly to Yes Yes Yes Yes
server 2 each DB2
node
DB2/UDB API Plug-In Any DB2 via Yes /
DB2 Client or through DB2 No Possible No No
DB2-Connect node 0 Limitations
DB2/UDB Load Plug-In Subject to DB2
Loader No No No No No
Limitations
ODBC Enterprise Native Any DB2 via Yes /
DB2 Client or through DB2 No3 No No No
DBE-Connect node 0
Dynamic RDBMS Plug-In Any DB2 via Yes /
DB2 Client or through DB2 No Possible No No
DB2-Connect node 0 Limitations

For specific details on the stage capabilities, consult the DataStage documentation (DataStage Parallel
Job Developers Guide, and DataStage Plug-In guides)

10.2.1.1 DB2/UDB Enterprise stage

Enterprise Edition provides native parallel read, lookup, upsert, and load capabilities to parallel DB2
databases on UNIX using the native parallel DB2/UDB Enterprise stage. The DB2/UDB Enterprise
stage requires DB2 Enterprise Server Edition on UNIX with Data Partitioning Facility (DPF) option.
(Before DB2 v8, this was also called “DB2 EEE”.) Furthermore, the DB2 hardware/UNIX/software
platform must match the hardware/software platform of the DataStage ETL server.

2
It is possible to connect the DB2 UDB stage to a remote database by simply cataloging the
remote database in the local instance and then using it as if it were a local database. This will
only work when the authentication mode of the database on the remote instance is set to “client
authentication”. If you use the stage in this way, you may experience data duplication when
working in partitioned instances since the node configuration of the local instance may not be the
same as the remote instance. For this reason, the “client authentication” configuration of a
remote instance is not recommended.
3
A patched version of the ODBC Enterprise stage allowing parallel read is available from IBM IIS
Support for some platforms. Check with IBM IIS Support for availability.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 103 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

As a native, parallel component the DB2/UDB Enterprise

stage is designed for maximum performance and scalability.
These goals are achieved through tight integration with the DB2 RDBMS, including direct
communication with each DB2 database node, and reading from - writing to DB2 in parallel (where
appropriate), using the same data partitioning as the referenced DB2 tables.

10.2.1.2 ODBC and DB2 Plug-In Stages

The ODBC Enterprise and plug-in stages are designed for lower-volume access to DB2 databases
without the DPF option installed (prior to v8, “DB2 EE”). These stages also provide connectivity to
non-UNIX DB2 databases, databases on UNIX platforms that differ from the platform of the DataStage
ETL server, or DB2 databases on Windows or Mainframe platforms (except for the “Load” stage
against a mainframe DB2 instance which is not supported).

By facilitating flexible connectivity to multiple types of remote DB2 database servers, the use of
DataStage plug-in stages will limit overall performance and scalability. Furthermore, when used as data
sources, plug-in stages cannot read from DB2 in parallel.

Using the DB2/UDB API stage or the Dynamic RDBMS stage, it may be possible to write to a DB2
target in parallel, since the DS/EE framework will instantiate multiple copies of these stages to handle
the data that has already been partitioned in the parallel framework. Because each plug-in invocation
will open a separate connection to the same target DB2 database table, the ability to write in parallel
may be limited by the table and index configuration set by the D2 database administrator.

The DB2/API (plug-in) stage should only be used to read from and write to DB2 databases on non-
UNIX platforms (such as mainframe editions through DB2-Connect). Sparse Lookup is not supported
through the DB2/API stage.

10.2.2 Connecting to DB2 with the DB2/UDB Enterprise Stage

Create a Parallel job and add a DB2/UDB Enterprise stage. Add the following properties:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 104 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 45: DB2/UDB Enterprise stage properties

For connection to a remote DB2/UDB instance, you will need
to set the following properties on the DB2/UDB Enterprise stage in your parallel job:
• Client Instance Name. Set this to the DB2 client instance name.
If you set this property, DataStage assumes you require remote connection.
• Server. Optionally set this to the instance name of the DB2 server. Otherwise use the DB2
environment variable, DB2INSTANCE, to identify the instance name of the DB2 server.
• Client Alias DB Name. Set this to the DB2 client’s alias database name for the remote DB2
server database. This is required only if the client’s alias is different from the actual name of the
remote server database.
• Database. Optionally set this to the remote server database name. Otherwise use the
environment variables $APT_DBNAME or $APT_DB2DBDFT to identify the database.
• User. Enter the user name for connecting to DB2, this is required for a remote connection in
order to retrieve the catalog information from the local instance of DB2 and thus must have
privileges for that local instance.
• Password. Enter the password for connecting to DB2, this is required for a remote connection
in order to retrieve the catalog information from the local instance of DB2 and thus must have
privileges for that local instance.

10.2.3 Configuring DB2 Multiple Instances in One DataStage Job

Although it is not officially supported, it is possible to connect to more than one DB2 instance within a
single job. Your job must meet one of the following configurations (note: the use of the word “stream”
refers to a contiguous flow of one stage to another within a single job):

1. Single stream - Two Instances Only

reading from one instance and writing to another instance with no other DB2 instances (not
sure how many stages of these 2 instances can be added to the canvas for this configuration for
lookups)

2. Two Stream – One Instance per Steam

reading from instance A and writing to instance A and reading from instance B and writing to
instance B (not sure how many stages of these 2 instances can be added to the canvas for this
configuration for lookups)

3. Multiple Stream with N DB2 sources with no DB2 targets

reading from 1 to n DB2 instances in separate source stages with no downstream other DB2
stages

In order to get this configuration to work correctly, you must adhere to all of the directions specified
for connecting to a remote instance AND the following:

• You must not set the APT_DB2INSTANCE_HOME environment variable. Once this variable
is set, it will try to use it for each of the connections in the job. Since a db2nodes.cfg file can

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 105 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

only contain information for one instance, this will

create problems.

• In order for DataStage to locate the db2nodes.cfg, you must build a user on the DataStage
server with the same name as the instance you are trying to connect to (the default logic for the
DB2/UDB Enterprise stage is to use the instance’s home directory as defined for the UNIX user
with the same name as the DB2 instance). In the users UNIX home directory, create a sqllib
subdirectory and place the remote instance’s db2nodes.cfg there. Since the
APT_DB2INSTANCE_HOME is not set, DS will default to this directory to find the
configuration file for the remote instance.

To connect to multiple DB2 instances, we recommend using separate jobs with their respective DB2
environment variable settings, landing intermediate results to a parallel Data Set. Depending on
platform configuration and I/O subsystem performance, separate jobs can communicate through
named pipes, although this incurs the overhead of Sequential File stage (corresponding export/import
operators) which does not run in parallel.

Or, if the data volumes are sufficiently small, DB2 plug-in stages (DB2 API, DB2 Load, Dynamic
RDBMS) may be used to access data in other instances.

10.2.4 DB2/UDB Enterprise stage Column Names

At runtime, the native parallel DB2/UDB Enterprise stage translates column names exactly except
when a component of a DB2 column name is not compatible with Enterprise Edition column naming
conventions, which place no limit on the length of a column name, but have the following restrictions:
- the name must start with a letter or underscore character
- the name can contain only alphanumeric and underscore characters
- the name is case insensitive

When there is an incompatibility, Enterprise Edition converts the DB2 column name as follows:
- if the DB2 column name does not begin with a letter or underscore, the string
“APT__column#” (two underscores) is added to beginning of the column name, where
column# is the number of the column. For example, if the third DB2 column is named 7dig,
the Enterprise Edition column will be named “APT_37dig”
- if the DB2 column name contains a character that is not alphanumeric or an underscore, the
character is replaced by two underscore characters

10.2.5 DB2/API stage Column Names

When using the DB2/API, DB2 Load, and Dynamic RDBMS plug-in stages, set the environment
variable $DS_ENABLE_RESERVED_CHAR_CONVERT if your DB2 database uses the reserved
characters # or $ in column names. This converts these special characters into an internal representation
that DataStage can understand.

Observe the following guidelines when $DS_ENABLE_RESERVED_CHAR_CONVERT is set:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 106 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

- Avoid using the strings __035__ and __036__ in

your DB2 column names (these are used as the
internal representations of # and $ respectively)
- Import meta data using the Plug-in Meta Data Import tool; avoid hand editing (this
minimizes the risk of mistakes or confusion).
- Once the table definition is loaded, the internal column names are displayed rather than the
original DB2 names both in table definitions and in the Data Browser. They are also used in
derivations and expressions.
- The original names are used in generated SQL statements, however, and you should use
them if entering SQL in the job yourself.

10.2.6 DB2/UDB Enterprise stage Data Type Mapping

The DB2 database schema to be accessed must NOT have any columns with User Defined Types
(UDTs). Use the “db2 describe table [table-name]” command on the DB2 client for each table to be
accessed to determine if UDTs are in use. Alternatively, examine the DDL for each schema to be
accessed.

Table Definitions should be imported into DataStage using orchdbutil to ensure accurate Table
Definitions. The DB2/UDB Enterprise stage converts DB2 data types to Enterprise Edition data types,
as shown in the following table.

DB2 Data Type Enterprise Edition Data Type

CHAR(n) string[n] or ustring[n]
CHARACTER VARYING(n,r) string[max=n] or ustring[max=n]
DATE date
DATETIME Time or timestamp with corresponding fractional precision for time:

If the DATETIME starts with a year component, the result is a timestamp field.

If the DATETIME starts with an hour, the result is a time field.

DECIMAL[p,s] decimal[p,s] where p is the precision and s is the scale
DOUBLE-PRECISION dfloat
FLOAT dfloat
INTEGER int32
MONEY decimal
NCHAR(n,r) string[n] or ustring[n]
NVARCHAR(n,r) string[max=n] or ustring[max=n]
REAL sfloat
SERIAL int32
SMALLFLOAT sfloat
SMALLINT int16
VARCHAR(n) string[max=n] or ustring[max=n]

IMPORTANT: DB2 data types that are not listed in the above table cannot be used in the
DB2/UDB Enterprise stage, and will generate an error at runtime

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 107 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.2.7 DB2/UDB Enterprise stage options

The DB2/UDB Enterprise (native parallel) stage should be used for reading from, performing lookups
against, and writing to a DB2 Enterprise Server Edition database with Database Partitioning Feature
(DPF)
• As a native, parallel component, the DB2/UDB Enterprise stage is designed for maximum
performance and scalability against very large partitioned DB2 UNIX databases.

• DB2/UDB Enterprise stage is tightly integrated with the DB2 RDBMS, communicating directly
with each database node, reading from and writing to DB2 in parallel (where appropriate), and
using the same data partitioning as the referenced DB2 tables.

When writing to a DB2 database in parallel, the DB2/UDB Enterprise stage offers the choice of SQL
(insert / update / upsert / delete) or fast DB2 loader methods. The choice between these methods
depends on required performance, database log usage, and recoverability.

a) The Write Method (and corresponding insert / update / upsert / delete) communicates
directly with the DB2 database nodes to execute instructions in parallel. All operations are
logged to the DB2 database log, and the target table(s) may be accessed by other users.
Time and row-based commit intervals determine the transaction size, and the availability of
new rows to other applications.

b) The DB2 Load method requires that the DataStage user running the job have DBADM
privilege on the target DB2 database. During the load operation, the DB2 Load method
places an exclusive lock on the entire DB2 tablespace into which it loads the data and no
other tables in that tablespace can be accessed by other applications until the load
completes. The DB2 load operator performs a non-recoverable load. That is, if the load
operation is terminated before it is completed, the contents of the table are unusable and the
tablespace is left in a load pending state. In this scenario, the DB2 Load DataStage job must
be re- run in Truncate mode to clear the load pending state.

10.2.8 Performance Notes

In some cases, when using user-defined SQL without partitioning against large volumes of DB2 data,
the overhead of routing information through a remote DB2 coordinator may be significant. In these
instances, it may be beneficial to have the DB2 DBA configure separate DB2 coordinator nodes (no
local data) on each ETL server (in clustered ETL configurations). In this configuration, DB2 Enterprise
stage should not include the Client Instance Name property, forcing the DB2 Enterprise stages on
each ETL server to communicate directly with their local DB2 coordinator.

10.2.9 DB2 in the DataStage USS environment

The manner in which DataStage / USS Edition interfaces with DB2 is slightly different than it is in the
non-z/OS environment. All activity in the z/OS environment always goes through the DB2 coordinator
node so parallelism differs slightly depending on how DB2 is accessed.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 108 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

When accessing a DB2 table using the Table read method,

functions within the db2read operator are used to read the
DB2 SYSTABLES table to retrieve the tablespace and database name for the table. These values are in
turn used to read the SYSTABLEPART table to retrieve the number of partitions, partitioning index
name(s), and the partition limit key value(s). Finally the SYSKEYS and SYSCOLUMNS tables are
read using the index name to get the associated column metadata (name and type). This information
determines the number of db2read operators that the conductor builds into the score and the queries
that they execute, as illustrated in Figure 46:

Figure 46: DB2 read on DataStage/USS

For example, Table T is in tablespace TS and TS is partitioned into 3 partitions on Col1 (limits: F, P, T)
and Col2 (10, 30, 40).

The WHERE clauses which are created to read this tables are:

Where Col1 < ‘F’ or (Col1 = ‘F’ and (Col2 < 10 or Col2 = 10))

Where (Col1 > ‘F’ and Col1 < ‘P’) or (Col1 = ‘F’ and Col2 > 10) or (Col1 = ‘P’ and (Col2 < 20 or Col2 = 20))

Where Col1 > ‘T’ or (Col1 = ‘T’ and Col2 > 40)

The method that DataStage/USS Edition uses to write to DB2 UDB on z/OS works differently than the
read process, and is controlled by the number of nodes in the configuration file. Since all write
operations need to go through the DB2 coordinator node on z/OS (this is different than on non-z/OS
platforms), the number of operators do not have to match to the number of partitions. This is illustrated
in Figure 47.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 109 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 47: DB2 write on DataStage/USS

On DataStage/USS Edition, Lookups work differently depending on whether the lookup is done
normally (in memory) or using a sparse technique where each lookup is effectively a query to the
database. An example of an in-memory Normal Lookup is shown in Figure 48.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 110 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 48: In-Memory Lookup on DataStage/USS

Here we see that the Normal Lookup actually consists of
reading the DB2 table into memory and then performing the lookup against the memory copy of the
table. When the conductor creates the score, it matches the number of db2read operators to the
partitioning scheme of the table (similar to the read) and the number of lookup operators to the number
of nodes in the configuration file.

Contrast the Normal Lookup with the way a Sparse Lookup is done as shown in Figure 49, where each
lookup operator is issuing an SQL to DB2 for every row it processes. Since each of these queries must
go through the DB2 coordinator node we can effectively ignore the level of parallelism specified for
the table.

Figure 49: DB2 Sparse Lookup on DataStage/USS

Finally, using the DB2 load utility in USS is different from non-z/OS environments. The DB2 LOAD
utility is designed to run from JCL only. In order to invoke it from a DataStage/USS job, we call a
DB2 stored procedure called DSNUTILS. The LOAD utility has a second limitation in that data
cannot be piped into it, nor can it be read in from a USS HFS file. This requires DataStage/USS to
create an MVS flat file to pass to the loader – note that this is the only non-HFS file that DS/USS can
write to. Since there is no sequential file stage associated with this MVS load file, we need to add a
special resource statement in our configuration file to specify the MVS dataset name to use. Figure 50
illustrates the DB2 LOAD process on USS and also shows the format of the special resource statement
used to define the MVS dataset used during the load operation.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 111 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 50: Calling DB2 Load Utility on DataStage/USS

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 112 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.3 Informix Database Guidelines

10.3.1 Informix Enterprise Stage Column Names
For each Informix Enterprise stage:
- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- both DS/EE and Informix support null values, and a null value in a database column is
stored as an out-of-band NULL value in the DS/EE column

10.3.2 Informix Enterprise stage Data Type Mapping

Table Definitions should be imported into DataStage using orchdbutil to ensure accurate Table
Definitions. The Informix Enterprise stage converts Informix data types to Enterprise Edition data
types, as shown in the following table.

Informix Data Type Enterprise Edition Data Type

CHAR(n) string[n]
CHARACTER VARYING(n,r) string[max=n]
DATE date
DATETIME date, time or timestamp with corresponding fractional precision for time:

If the DATETIME starts with a year component and ends with a month, the
result is a date field.

If the DATETIME starts with a year component, the result is a timestamp field.

If the DATETIME starts with an hour, the result is a time field.

DECIMAL[p,s] decimal[p,s] where p is the precision and s is the scale
The maximum precision is 32. A decimal with floating scale is converted to
dfloat
DOUBLE-PRECISION dfloat
FLOAT dfloat
INTEGER int32
MONEY decimal
NCHAR(n,r) string[n]
NVARCHAR(n,r) string[max=n]
REAL sfloat
SERIAL int32
SMALLFLOAT sfloat
SMALLINT int16
VARCHAR(n) string[max=n]

IMPORTANT: Informix data types that are not listed in the above table cannot be used in the
Informix Enterprise stage, and will generate an error at runtime

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 113 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.4 ODBC Enterprise Guidelines

10.4.1 ODBC Enterprise Stage Column Names
For each ODBC Enterprise stage:
- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- names are translated exactly except when the external data source column name contains a
character that DataStage does not support. In that case, two underscore characters replace
the unsupported character
- both DS/EE and ODBC support null values, and a null value in a database column is stored
as an out-of-band NULL value in the DS/EE column

10.4.2 ODBC Enterprise stage Data Type Mapping

ODBC data sources are not supported by the orcdbutil utility. It is important to verify the correct
ODBC to Enterprise Edition data mapping, as shown in the following table:

ODBC Data Type Enterprise Edition Data Type

SQL_BIGINT int64
SQL_BINARY raw(n)
SQL_CHAR string[n]
SQL_DECIMAL decimal[p,s] where p is the precision and s is the scale
SQL_DOUBLE decimal[p,s]
SQL_FLOAT decimal[p,s]
SQL_GUID string[36]
SQL_INTEGER int32
SQL_BIT int8 [0 or 1]
SQL_REAL decimal[p,s]
SQL_SMALLINT int16
SQL_TINYINT int8
SQL_TYPE_DATE date
SQL_TYPE_TIME time[p]
SQL_TYPE_TIMESTAMP timestamp[p]
SQL_VARBINARY raw[max=n]
SQL_VARCHAR string[max=n]
SQL_WCHAR ustring[n]
SQL_WVARCHAR ustring[max=n]

Note that the maximum size of a DataStage record is limited to 32K. If you attempt to read a record
larger than 32K, Enterprise Edition will return an error and abort your job.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 114 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

IMPORTANT: ODBC data types that are not listed in the

above table cannot be used in the ODBC Enterprise stage, and will generate an error at runtime

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 115 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.4.3 Reading ODBC Sources

Unlike other native parallel database stages, the ODBC Enterprise stage does not support parallel read4,
since this capability is not provided by the ODBC API. Depending on the target database, and the table
configuration (row or page level lock mode if available), it may be possible to write to a target database
in parallel using the ODBC Enterprise stage.

4
On some platforms, a patch may be available through IBM IIS Support to support parallel reads
through ODBC. Parallel reads through ODBC match the degree of parallelism in the
$APT_CONFIG_FILE.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 116 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.5 Oracle Database Guidelines

10.5.1 Oracle Enterprise Stage Column Names
For each Oracle Enterprise stage:
- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- names are translated exactly except when the Oracle source column name contains a
character that DataStage does not support. In that case, two underscore characters replace
the unsupported character
- both DS/EE and Oracle support null values, and a null value in a database column is stored
as an out-of-band NULL value in the DS/EE column

10.5.2 Oracle Enterprise stage Data Type Mapping

Oracle Table Definitions should be imported into DataStage using orchdbutil to ensure accurate
Table Definitions. This is particularly important for Oracle databases, which are not heavily typed.
Enterprise Edition maps Oracle data types based on the rules given in the following table:

Oracle Data Type Enterprise Edition Data Type

CHAR(n) string[n] or ustring[n]
a fixed-length string with length = n
DATE timestamp
NUMBER decimal[38,10]
NUMBER[p,s] int32 if precision(p) < 11 and scale (s) = 0
decimal[p,s] if precision (p) >=11 or scale > 0
RAW(n) not supported
VARCHAR(n) string[max=n] or ustring[max=n]
a variable-length string with maximum length = n

Note that the maximum size of a DataStage record is limited to 32K. If you attempt to read a record
larger than 32K, Enterprise Edition will return an error and abort your job.

IMPORTANT: Oracle data types that are not listed in the above table cannot be used in the
Oracle Enterprise stage, and will generate an error at runtime

10.5.3 Reading from Oracle in Parallel

By default, the Oracle Enterprise stage reads sequentially from its source table or query. Setting the
partition table option to the specified table will enable parallel extracts from an Oracle source. The
underlying Oracle table does not have to be partitioned for parallel read within Enterprise Edition.

It is important to note that certain types of queries cannot run in parallel. Examples include:
- queries containing a GROUP BY clause that are also hash partitioned on the same field

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 117 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

- queries performing a non-collocated join (a SQL

JOIN between two tables that are not stored in the
same partitions with the same partitioning strategy)

10.5.4 Oracle Load Options

When writing to an Oracle table (using Write Method = Load), Enterprise Edition uses the Parallel
Direct Path Load method. When using this method, the Oracle stage cannot write to a table that has
indexes (including indexes automatically generated by Primary Key constraints) on it unless you
specify the Index Mode option (maintenance, rebuild).
• Setting the environment variable $APT_ORACLE_LOAD_OPTIONS to “OPTIONS (DIRECT=TRUE,
PARALLEL=FALSE) also allows loading of indexed tables without index maintenance. In this
instance, the Oracle load will be done sequentially.

The Upsert Write Method can be used to insert rows into a target Oracle table without bypassing
indexes or constraints. In order to automatically generate the SQL required by the Upsert method, the
key column(s) must be identified using the check boxes in the column grid.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 118 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.6 Sybase Enterprise Guidelines

10.6.1 Sybase Enterprise Stage Column Names
For each Sybase Enterprise stage:
- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- names are translated exactly except when the Sybase source column name contains a
character that DataStage does not support. In that case, two underscore characters replace
the unsupported character
- both DS/EE and Sybase support null values, and a null value in a database column is stored
as an out-of-band NULL value in the DS/EE column

10.6.2 Sybase Enterprise stage Data Type Mapping

Sybase databases are not supported by the orcdbutil utility. It is important to verify the correct
Sybase to Enterprise Edition data mapping, as shown in the following table:

Sybase Data Type Enterprise Edition Data Type

BINARY(n) raw(n)
BIT int8
CHAR(n) string[n] a fixed-length string with length n
DATE date
DATETIME timestamp
DEC[p,s] or DECIMAL[p,s] decimal[p,s] where p is the precision and s is the scale
DOUBLE PRECISION or FLOAT dfloat
INT or INTEGER int32
MONEY decimal[15,4]
NCHAR(n) ustring[n] a fixed-length string with length n - only for ASE
NUMERIC[p,s] decimal[p,s] where p is the precision and s is the scale
NVARCHAR(n,r) ustring[max=n] a variable-length string with length n - only for ASE
REAL sfloat
SERIAL int32
SMALLDATETIME timestamp
SMALLFLOAT sfloat
SMALLINT int16
SMALLMONEY decimal[10,4]
TINYINT int8
TIME time
UNSIGNED INT unit32
VARBINARY(n) raw[max=n]
VARCHAR(n) string[max=n] a variable-length string with maximum length n

IMPORTANT: Sybase data types that are not listed in the above table cannot be used in the
Sybase Enterprise stage, and will generate an error at runtime

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 119 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.7 Teradata Database Guidelines

10.7.1 Choosing the Proper Teradata Stage
Within DataStage Enterprise Edition, the following stages can be used for reading from and writing to
Teradata databases in a parallel job flow:

Source Teradata Stages Target Teradata Stages

Teradata Enterprise Teradata Enterprise
Teradata API Teradata API
Teradata MultiLoad (MultiLoad option)
Teradata MultiLoad (TPump option)

For maximum performance of high-volume data flows, the native parallel Teradata Enterprise stage
should be used. Teradata Enterprise uses the programming interface of the Teradata utilities
FastExport (reads) and FastLoad (writes), and is subject to all these utilities’ restrictions.

NOTE: Unlike the FastLoad utility, the Teradata Enterprise stage supports Append mode,
inserting rows into an existing target table. This is done through a shadow “terasync” table.

Teradata has a system-wide limit to the number of concurrent database utilities. Each use of the
Teradata Enterprise stages counts toward this limit.

10.7.2 Source Teradata Stages

Teradata Stage Stage Usage Guidelines Parallel Teradata
Type Read Utility Limit
Teradata Enterprise Native - Reading a large number of rows in parallel Yes applies
Parallel - Supports OPEN and CLOSE commands
- Subject to the limits of Teradata FastExport
Teradata API Plug-In - Reading a small number of rows sequentially No none

10.7.3 Target Teradata Stages

Teradata Stage Stage Usage Guidelines Parallel Teradata
Type Write Utility Limit
Teradata Enterprise Native - Writing a large number of rows in parallel Yes applies
Parallel - Supports OPEN and CLOSE commands
- Limited to INSERT (new table) or APPEND
(existing table)
- Subject to the limits of Teradata FastLoad
(but also supports APPEND)
- Locks the target table in exclusive mode
Teradata MultiLoad Plug-In - Insert, Update, Delete, Upsert of moderate data No applies
(MultiLoad utility) volumes
- Locks the target table(s) in exclusive mode

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 120 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Teradata MultiLoad Plug-In - Insert, Update, Delete, Upsert of

(TPump utility) small volumes of data within a
large database
- Does not lock the target tables
- Should not be run in parallel, because each
node and use counts toward system-wide
Teradata utility limit
Teradata API Plug-In - Insert, Update, Delete, Upsert of small Yes none
volumes of data
- Allows concurrent writes (does not lock target)
- Slower than TPump for equivalent operations

10.7.4 Teradata Enterprise Stage Column Names

For each Teradata Enterprise stage:
- rows of the database result set correspond to records of a DS/EE Data Set
- columns of the database row correspond to columns of a DS/EE record
- the name and data type of each database column corresponds to a DS/EE Data Set name and
data type using a predefined mapping of database data types to Enterprise Edition data types
- both DS/EE and Teradata support null values, and a null value in a database column is
stored as an out-of-band NULL value in the DS/EE column
- DS/EE gives the same name to its columns as the Teradata column name. However, while
DS/EE column names can appear in either upper or lower case, Teradata column names
appear only in upper case.

10.7.5 Teradata Enterprise stage Data Type Mapping

Teradata databases are not supported by the orcdbutil utility. It is important to verify the correct
Teradata to Enterprise Edition data mapping, as shown in the following table:

Teradata Data Type Enterprise Edition Data Type

byte(n) raw[n]
byteint int8
char(n) string[n]
date date
decimal[p,s] decimal[p,s] where p is the precision and s is the scale
double precision dfloat
float dfloat
graphic(n) raw[max=n]
integer int32
long varchar string[max=n]
long vargraphic raw[max=n]
numeric(p,s) decimal[p,s]
real Dfloat
smallint int16
time time
timestamp timestamp
varbyte(n) raw[max=n]
varchar(n) string[max=n]

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 121 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

vargraphic(n) raw[max=n]

IMPORTANT: Teradata data types that are not listed in the above table cannot be used in the
Teradata Enterprise stage, and will generate an error at runtime.

Aggregates and most arithmetic operators are not allowed in the SELECT clause
of a Teradata Enterprise stage.

10.7.6 Specifying Teradata Passwords with Special Characters

Teradata permits passwords with special characters and symbols. To specify a Teradata password that
contains special characters, the password must be surrounded by an “escaped” single quote as shown,
where pa$$ is the example password: \’pa$$\’

10.7.7 Teradata Enterprise Settings

Within the Teradata Enterprise stage, the DB Options property specifies the connection string and
connection properties in the form:

user=username,password=password[,SessionsPerPlayer=nn][,RequestedSessions=nn]

where SesionsPerPlayer and RequestedSessions are optional connection parameters that are required
when accessing large Teradata databases.

By default, RequestedSessions equals the maximum number of available sessions on the Teradata
instance, but this can be set to a value between 1 and the database vprocs.

The SessionsPerPlayer option determines the number of connections each DataStage EE player opens to
Teradata. Indirectly, this determines the number of DataStage players, and hence the number of UNIX
processes and overall system resource requirements of the DataStage job. SessionsPerPlayer should be
set such that:

RequestedSessions = (sessions per player * the number of nodes * players per node)

The default value for the SessionsPerPlayer suboption is 2.

Setting the SessionsPerPlayer too low on a large system can result in so many players that the job fails
due to insufficient resources. In that case SessionsPerPlayer should be increased, and/or
RequestedSessions should be decreased.

10.7.8 Improving Teradata Enterprise Performance

Setting the environment variable $APT_TERA_64K_BUFFERS may significantly improve
performance of Teradata Enterprise connections depending on network configuration. By default,
Teradata Enterprise stage uses 32K buffers. (Note that 64K buffers must be enabled at the Teradata
server level).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 122 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

10.7.9 Teradata on USS

On the USS platform the Teradata Enterprise Stage uses CLIv2 for channel-attached systems (OS/390
and z/OS).

To connect to a Teradata server, you must supply the client with the Teradata Director Program (TDP)
identifier, also known as the tdpid. On a network-attached system, the tdpid is the host name of the
Teradata server. On MVS, the tdpid must be in the form TDPx, where x is 0-9, A-Z (case insensitive),
$, #, or @. The first three characters must be TDP. That leaves 39 possible TDP names and is
different than the convention used for non-channel attached systems.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 123 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

11 Troubleshooting and Monitoring

11.1 Warning on Single-Node Configuration Files
Because the DS/EE configuration file can be changed at runtime, it is important that all jobs be tested
with a configuration file that has more than one node in its default node pool. This will ensure that the
jobs have been designed with proper partitioning logic.

If the job results are correct with a single-node configuration file, and incorrect with a multi-node
configuration file, the job’s partitioning logic and parallel design concepts (especially within
Transformer stages) should be examined.

11.2 Debugging Environment Variables

The following environment variables can be set to assist in debugging a parallel job:

Environment Variable Setting Description

$OSH_PRINT_SCHEMAS 1 Outputs the actual schema definitions used by the
DataStage EE framework at runtime in the DataStage
log. This can be useful when determining if the actual
runtime schema matches the expected job design table
definitions.
$APT_PM_PLAYER_TIMING 1 Prints detailed information in the job log for each
operator, including CPU utilization and elapsed
processing time.
$APT_PM_PLAYER_MEMORY 1 Prints detailed information in the job log for each
operator when allocating additional heap memory.
$APT_BUFFERING_POLICY FORCE Forces an internal buffer operator to be placed between
every operator. Normally, the DataStage EE framework
inserts buffer operators into a job flow at runtime to
avoid deadlocks and improve performance.

Using $APT_BUFFERING_POLICY=FORCE in
combination with $APT_BUFFER_FREE_RUN
Setting effectively isolates each operator from slowing
$APT_BUFFERING_POLICY=FORCE is not upstream production. Using the job monitor
recommended for production job runs. performance statistics, this can identify which part of a
job flow is impacting overall performance.
$DS_PX_DEBUG 1 Set this environment variable to capture copies of the
job score, generated osh, and internal Enterprise
Edition log messages in a directory corresponding to
the job name. This directory will be created in the
“Debugging” sub-directory of the Project home
directory on the DataStage server.
$APT_PM_STARTUP_CONCURRENCY 5 This environment variable should not normally need to
be set. When trying to start very large jobs on heavily-
loaded servers, lowering this number will limit the
number of processes that are simultaneously created
when a job is started.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 124 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

$APT_PM_NODE_TIMEOUT [seconds] For

heavily
loaded MPP or clustered environments, this variable
determines the number of seconds the conductor node
will wait for a successful startup from each section
leader. The default is 30 seconds.

11.3 How to Isolate and Debug a Parallel Job

There are a number of tools available to debug DataStage Enterprise Edition jobs. The general process
for debugging a job is:

 Check the Director job log for warnings. These may indicate an underlying logic problem or
unexpected data type conversion. When a fatal error occurs, the log entry is sometimes
preceded by a warning condition.

All fatal and warning messages should be addressed before attempting to debug, tune, or
promote a job from development into test or production. In some instances, it may not be
possible to remove all warning messages generated by the EE engine. But all warnings should
be examined and understood.

 Enable the Job Monitoring Environment Variables detailed in Section 2.5.1: Environment
Variable Settings and the DataStage Parallel Job Advanced Developers Guide.

 Use the Data Set Management tool (available in the Tools menu of DataStage Designer or
DataStage Manager) to examine the schema, look at row counts, and to manage source or target
Parallel Data Sets.

 Set the environment variable $DS_PX_DEBUG to capture copies of the job score, generated
osh, and internal Enterprise Edition log messages in a directory corresponding to the job name.
This directory will be created in the “Debugging” sub-directory of the Project home directory
on the DataStage server.

 Use $OSH_PRINT_SCHEMAS to verify that the job’s runtime schemas matches what the job
developer expected in the design-time column definitions. This will place entries in the Director
log with the actual runtime schema for every link using Enterprise Edition internal data types.

NOTE: For large jobs, it is possible for $OSH_PRINT_SCHEMAS to generate a log entry
that is too large for DataStage Director to store or display. To capture the full schema output in
these cases, enable both $OSH_PRINT_SCHEMAS and $DS_PX_DEBUG .

 Examine the score dump (placed in the Director log when $APT_DUMP_SCORE is enabled).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 125 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

 For flat (sequential) sources and targets:

o To display the actual contents of any file in hexadecimal and ASCII (including embedded
control characters or ASCII NULLs), use the UNIX command

od –xc –Ax

o To display the number of lines and characters in a specified ASCII text file, use the UNIX
command

wc –lc [filename]

Dividing the total number of characters number of lines provides an audit to ensure all rows
are same length.

NOTE: The wc command counts UNIX line delimiters, so if the file has any binary columns,
this count may be incorrect. It is also not useful for files of non-delimited fixed-length
record format.

11.4 Viewing the Generated OSH

Within Designer, jobs are compiled into OSH (Orchestrate SHell) scripts that are used to execute the
given job design at runtime. It is useful to examine the generated OSH for debugging purposes, and to
understand internally what is running.

To enable viewing of generated OSH, it must be enabled for a given project within the Administrator
client:

Figure 51: Generated OSH Administrator option

Once this option has been enabled for a given project, the generated OSH tab will appear in the Job
Properties dialog box:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 126 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Figure 52: Generated OSH in Designer Job Properties

11.5 Interpreting the Parallel Job Score

When attempting to understand an Enterprise Edition flow, the first task is to examine the score dump
which is generated when you set APT_DUMP_SCORE=TRUE in your environment. A score dump
includes a variety of information about a flow, including how composite operators and shared
containers break down; where data is repartitioned and how it is repartitioned; which operators, if any,
have been inserted by EE; what degree of parallelism each operator runs with; and exactly which nodes
each operator runs on. Also available is some information about where data may be buffered.

The following score dump shows a flow with a single Data Set, which has a hash partitioner that
partitions on key field a. It shows three stages: Generator, Sort (tsort) and Peek. The Peek and Sort
stages are combined; that is, they have been optimized into the same process. All stages in this flow are
running on one node. The job runs 3 processes on 2 nodes.

##I TFSC 004000 14:51:50(000) <main_program>

This step has 1 dataset:
ds0: {op0[1p] (sequential generator)
eOther(APT_HashPartitioner { key={ value=a }
})->eCollectAny
op1[2p] (parallel APT_CombinedOperatorController:tsort)}
It has 2 operators:
op0[1p] {(sequential generator)
on nodes (

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 127 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

lemond.torrent.com[op0,p0]
)}
op1[2p] {(parallel APT_CombinedOperatorController:
(tsort)
(peek)
)on nodes (
lemond.torrent.com[op1,p0]
lemond.torrent.com[op1,p1]
)}

In a score dump, there are three areas to investigate:

• Are there sequential stages?
• Is needless repartitioning occurring?
• In a cluster or Grid, are the computation-intensive stages shared evenly across all nodes?

More details on interpreting the parallel job score can be found in 12.4.2 Understanding the Parallel
Job Score.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 128 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

12 Performance Tuning Job Designs

The ability to process large volumes of data in a short period of time depends on all aspects of the flow
and environment being optimized for maximum throughput and performance. Performance tuning and
optimization is an iterative process that begins at job design and unit tests, proceeds through integration
and volume testing, and continues throughout an application’s production lifecycle.

This section provides tips for designing a job for optimal performance, and for optimizing the
performance of a given data flow using various settings and features within DataStage Enterprise
Edition.

12.1 How to Design a Job for Optimal Performance

Overall job design can be the most significant factor in data flow performance. This section outlines
performance-related tips that can be followed when building a parallel data flow using DataStage
Enterprise Edition.

a) Use Parallel Data Sets to land intermediate result between parallel jobs.
• Parallel Data Sets retain data partitioning and sort order, in the DS/EE native internal
format, facilitating end-to-end parallelism across job boundaries.
• Data Sets can only be read by other DS/EE parallel jobs (or the orchadmin command
line utility). If you need to share information with external applications, File Sets
facilitate parallel I/O at the expense of exporting to a specified file format.
• Lookup File Sets can be used to store reference data used in subsequent jobs. They
maintain reference data in DS/EE internal format, pre-indexed. However, Lookup File
Sets can only be used on reference links to a Lookup stage. There are no utilities for
examining data within a Lookup File Set.

b) Remove unneeded columns as early as possible within the data flow.

Every unused column requires additional memory which can impact performance (it also makes
each transfer of a record from one stage to the next more expensive).
• When reading from database sources, use a select list to read needed columns instead of
the entire table (if possible)
• Be alert when using runtime column propagation (“RCP”) – it may be necessary to
disable RCP for a particular stage to ensure that columns are actually removed using
that stage’s Output Mapping.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 129 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

c) Always specify a maximum length for Varchar

columns.
Unbounded strings (Varchar’s without a maximum length) can have a significant negative
performance impact on a job flow.
• There are limited scenarios when the memory overhead of handling large Varchar
columns would dictate the use of unbounded strings. For example:
 Varchar columns of a large (for example, 32K) maximum length that are rarely
populated
 Varchar columns of a large maximum length with highly varying data sizes

d) Avoid type conversions, if possible.

o When working with database sources and targets, use orchdbutil to ensure that the design-
time metadata matches the actual runtime metadata (especially with Oracle databases).
 Enable $OSH_PRINT_SCHEMAS to verify runtime schema matches job design
column definitions
o Verify that the data type of defined Transformer stage variables matches the expected result
type

e) Minimize the number of Transformers.

For data type conversions, renaming and removing columns, other stages (for example, Copy,
Modify) may be more appropriate. Note, however, that unless dynamic (parameterized)
conditions are required, a Transformer is always faster than a Filter or Switch stage.

• Avoid using the BASIC Transformer, especially in large-volume data flows. External
user-defined functions can expand the capabilities of the parallel Transformer.

• Use BuildOps only when existing Transformers do not meet performance

requirements, or when complex reusable logic is required.
Because BuildOps are built in C++, there is greater control over the efficiency of code,
at the expense of ease of development (and more skilled developer requirements).

f) Minimize the number of partitioners in a job. It is usually possible to choose a smaller

partition-key set, and to simply re-sort on a differing set of secondary/tertiary/etc. keys.

• When possible, ensure data is as close to evenly distributed as possible. When

business rules dictate otherwise and the data volume is large and sufficiently skewed, re-
partition to a more balanced distribution as soon as possible to improve performance of
downstream stages.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 130 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

• Know your data. Choose hash key columns

that generate sufficient unique key
combinations (while satisfying business requirements).
• Use SAME partitioning carefully. Remember that SAME maintains the degree of
parallelism of the upstream operator.

g) Minimize and combine use of Sorts where possible

o It is frequently possible to arrange the order of business logic within a job flow to leverage
the same sort order, partitioning, and groupings.
 If data has already been partitioned and sorted on a set of key columns, specifying
the “don’t sort, previously sorted” option for those key columns in the Sort stage
will reduce the cost of sorting and take greater advantage of pipeline parallelism.
o When writing to parallel Data Sets, sort order and partitioning are preserved. When reading
from these Data Sets, try to maintain this sorting, if possible, by using “SAME”
partitioning.
o The stable sort option is much more expensive than non-stable sorts, and should only be
used if there is a need to maintain an implied (i.e. not explicitly stated in the sort keys) row
order.
o Performance of individual sorts can be improved by increasing the memory usage per
partition using the “Restrict Memory Usage (MB)” option of the standalone Sort stage. The
default setting is 20MB per partition. In addition, the environment variable
APT_TSORT_STRESS_BLOCKSIZE can be used to set (in units of MB) the size of the
RAM buffer for all sorts in a job, even those that have the “Restrict Memory Usage” option
set.
12.2 Understanding Operator Combination

At runtime, DataStage Enterprise Edition analyzes a given job design and uses the parallel
configuration file to build a job score which defines the processes and connection topology (Data Sets)
between them used to execute the job logic.

When composing the score, Enterprise Edition attempts to reduce the number of processes by
combining the logic from 2 or more stages (operators) into a single process (per partition). Combined
operators are generally adjacent to each other in a data flow.

The purpose behind operator combination is to reduce the overhead associated with an increased
process count. If two processes are interdependent (one processes the other’s output) and they are both
CPU-bound or I/O-bound, there is nothing to be gained from pipeline partitioning5.

5
One exception to this guideline is when operator combination generates too few processes to
keep the processors busy. In these configurations, disabling operator combination allows CPU
activity to be spread across multiple processors instead of being constrained to a single processor.
As with any example, your results should be tested in your environment.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 131 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

However, the assumptions used by the Enterprise Edition

optimizer to determine which stages can be combined may
not always be the most efficient. It is for this reason that combination can be enabled or disabled on a
per-stage basis, or globally.

When deciding which operators to include in a particular combined operator (a.k.a. Combined Operator
Controller), Enterprise Edition is ‘greedy’ - it will include all operators that meet the following rules:

o Must be contiguous
o Must be the same degree of parallelism
o Must be ‘Combinable’, here is a partial list of non-combinable operators:
 Join
 Aggregator
 Remove Duplicates
 Merge
 BufferOp
 Funnel
 DB2 Enterprise Stage
 Oracle Enterprise Stage
 ODBC Enterprise Stage
 BuildOps

In general, it is best to let DSEE decide what to combine and what to leave uncombined. However,
when other performance tuning measures have been applied and still greater performance is needed,
tuning combination might yield additional performance benefits.

There are 2 ways to affect operator combination:

o The environment variable APT_DISABLE_COMBINATION, disables ALL combination
in the entire data flow, this is only recommended on pre 7.0 versions of DS/EE.
o Within Designer, combination can be set on a per-stage basis (on the Stage/Advanced tab)

The job score identifies what components are combined. (For information on interpreting a job score
dump, see 12.4.2 Understanding the Parallel Job Score” in this document.) In addition, if the “%CPU”
column is displayed in a Job Monitor window in Director, combined stages are indicated by parenthesis
surrounding the % CPU, as shown in the following illustration:

Figure 53: CPU-bound combined process in Job Monitor

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 132 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Choosing which operators to disallow combination for is as

much art as science. However, in general, it is good to
separate I/O heavy operators (Sequential File, Full Sorts, etc.) from CPU-heavy operators
(Transformer, Change Capture, etc.). For example, if you have several transformers and database
operators combined with an output Sequential File, it might be a good idea to set the sequential file to
be non-combinable. This will prevent IO requests from waiting on CPU to become available and vice-
versa.

In fact, in the above job design, the I/O-intensive FileSet is combined with a CPU-intensive
Transformer. Disabling combination with the Transformer enables pipeline partitioning, and improves
performance, as shown in this subsequent Job Monitor for the same job:

Figure 54: Throughput in Job Monitor after disabling combination

12.3 Minimizing Runtime Processes and Resource Requirements

The architecture of Enterprise Edition is well suited for processing massive volumes of data in parallel
across available resources. Toward that end, DS/EE executes a given job across the resources defined
in a the specified configuration file.

There are times, however, when it is appropriate to minimize the resource requirements for a given
scenario, for example:
 Batch jobs that process a small volume of data
 Real-time jobs that process data in small message units
 Environments running a large number of jobs simultaneously on the same server(s)

In these instances, a single-node configuration file is often appropriate to minimize job startup time and
resource requirements without significantly impacting overall performance.

There are many factors that can reduce the number of processes generated at runtime:
 Use a single-node configuration file
 Remove ALL partitioners and collectors (especially when using a single-node configuration
file)
 Enable runtime column propagation on Copy stages with only one input and one output
 Minimize join structures (any stage with more than one input, such as Join, Lookup, Merge,
Funnel)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 133 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

 Minimize non-combinable stages (as outlined in the

previous section) such as Join, Aggregator, Remove
Duplicates, Merge, Funnel, DB2 Enterprise, Oracle Enterprise, ODBC Enterprise, BuildOps,
BufferOp
 Selectively (being careful to to avoid deadlocks) disable buffering. (Buffering is discussed in
more detail in the following section.)
12.4 Understanding Buffering

There are two types of buffering in Enterprise Edition: ‘inter-operator transport’ and ‘deadlock
prevention’.

12.4.1 Inter-Operator Transport Buffering

Though it may look like it from the performance statistics and documentation might discuss ‘record
streaming’, strictly speaking, records do not stream from one stage to another. They are actually
transferred in blocks (just like on old magnetic tapes) called “Transport Blocks”. Each pair of operators
that have a producer/consumer relationship will share at least 2 of these blocks.

The first block will be used by the upstream/producer stage to output data it is done with. The second
block will be used by the downstream/consumer stage to obtain data that is ready for the next
processing step. Once the upstream block is full and the downstream block is empty, the blocks are
swapped and the process begins again.

This type of buffering (or ‘Record Blocking’) is rarely tuned. It usually only comes into play when the
size of a single record exceeds the default size of the transport block, then setting
APT_DEFAULT_TRANSPORT_BLOCK_SIZE to a multiple of (or equal to) the record size will
resolve the problem. Remember, there are 2 of these transport blocks for each partition of each link, so
setting this value too high will result in a large amount of memory consumption.

The behavior of these transport blocks is determined by these environment variables:

 APT_DEFAULT_TRANSPORT_BLOCK_SIZE
o Specifies the default block size for transferring data between players. The default value
is 8192, with a valid value range for between 8192 and 1048576. If necessary, the value
provided by a user for this variable is rounded up to the operating system's nearest page
size.
 APT_MIN_TRANSPORT_BLOCK_SIZE
o Specifies the minimum allowable block size for transferring data between players.
Default is 8192. Cannot be less than 8192; cannot be greater than 1048576. This
variable is only meaningful when used in combination with
APT_LATENCY_COEFFICIENT, APT_AUTO_TRANSPORT_BLOCK_SIZE and
APT_MAX_TRANSPORT_BLOCK_SIZE
 APT_MAX_TRANSPORT_BLOCK_SIZE

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 134 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

o Specifies the maximum allowable block size

for transferring data between players. Default
is 1048576. Cannot be less than 8192; cannot be greater than 1048576. This variable is
only meaningful when used in combination with APT_LATENCY_COEFFICIENT,
APT_AUTO_TRANSPORT_BLOCK_SIZE and
APT_MMIN_TRANSPORT_BLOCK_SIZE
 APT_AUTO_TRANSPORT_BLOCK_SIZE
o If set, the framework calculates the block size for transferring data between players
according to this algorithm:
if (recordSize * APT_LATENCY_COEFFICIENT <
APT_MIN_TRANSPORT_BLOCK_SIZE)
then blockSize = APT_MIN_TRANSPORT_BLOCK_SIZE
else if (recordSize * APT_LATENCY_COEFFICIENT >
APT_MAX_TRANSPORT_BLOCK_SIZE)
then blockSize = APT_MAX_TRANSPORT_BLOCK_SIZE
else blockSize = recordSize * APT_LATENCY_COEFFICIENT
 APT_LATENCY_COEFFICIENT
o Specifies the number of records to be written to each transport block
 APT_SHARED_MEMORY_BUFFERS
o Specifies the number of Transport Blocks between a pair of operators, must be at least 2

NOTE: The environment variables APT_MIN/MAX_TRANSPORT_BLOCK_SIZE,

APT_LATENCY_COEFFICIENT, and APT_AUTO_TRANSPORT_BLOCK_SIZE are used
only with fixed-length records.

12.4.2 Deadlock Prevention Buffering

The other type of buffering, “Deadlock Prevention” comes into play anytime there is a Fork-Join
structure in a job. Here is an example job fragment:

Figure 55: Fork-Join example

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 135 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

In this example, the Transformer creates a fork with 2 parallel

Aggregators, which go into an Inner Join. Note however, that
“Fork-Join” is a graphical description, it doesn’t necessarily have to involve a Join stage.

To understand deadlock-prevention buffering, it is important to understand the operation of a parallel

pipeline. Imagine that the Transformer is waiting to write to Aggregator1; Aggregator2 is waiting to
read from the Transformer; Aggregator1 is waiting to write to the Join; and Join is waiting to read from
Aggregator2. Like this:

Aggregator 1
Waiting to Qu
eu
d Write to Join Wr ed
e ue ite
Qu ite
Wr
Transformer Join
Waiting to Waiting to
write to read from
Aggregator1 Re Aggregator2
Qu ad
ad eue
Re d d
eue
Qu Aggregator2
Waiting to
read from
Transformer

(Here the arrows represent dependency direction, instead of data flow.)

Without deadlock buffering, this scenario would create a circular dependency where Transformer is
waiting on Aggregator1, Aggregator1 is waiting on Join, Join is waiting on Aggregator2, and
Aggregator2 is waiting on Transformer. Without deadlock buffering, the job would deadlock - bringing
processing to a halt (though the job does not stop running, it would eventually time out).

To guarantee that this problem never happens in Enterprise Edition, there is a specialized operator
called BufferOp. BufferOp is always ready to read or write and will not allow a read/write request to be
queued. It is placed on all inputs to a join structure (again, not necessarily a Join stage) by Enterprise
Edition during job startup. So the above job structure would be altered by the DS/EE engine to look
like this:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 136 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

BufferOp1
Aggregator 1

Transformer Join

Aggregator2
BufferOp 2

(Here the arrows now represent data-flow, not dependency.)

Since BufferOp is always ready to read or write, Join cannot be ‘stuck’ waiting to read from either of
its inputs, thus breaking the circular dependency and guaranteeing no deadlock will occur.

BufferOps will also be placed on the input partitions to any sequential stage that is fed by a parallel
stage, as these same types of circular dependencies can result from partition-wise Fork-Joins.

By default, BufferOps will allocate 3MB of memory each (remember that this is per operator, per
partition). When that is full (because the upstream operator is still writing but the downstream operator
isn’t ready to accept that data yet) it will begin to flush data to the scratchdisk resources specified in the
configuration file (detailed in Chapter 11, “The Parallel Engine Configuration File” of the DataStage
Manager guide).

TIP: For very wide rows, it may be necessary to increase the default buffer size
(APT_BUFFER_MAXIMUM_MEMORY) to hold more rows in memory.

The behavior of deadlock-prevention BufferOps can be tuned through these environment variables:

 APT_BUFFER_DISK_WRITE_INCREMENT
o Controls the “blocksize” written to disk as the memory buffer fills. Default 1 MB. May
not exceed 2/3 of APT_BUFFER_MAXIMUM_MEMORY.

 APT_BUFFER_FREE_RUN
o Maximum capacity of the buffer operator before it starts to offer resistance to incoming
flow, as a nonnegative (proper or improper) fraction of
APT_BUFFER_MAXIMUM_MEMORY. Values greater than 1 indicate that the buffer

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 137 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

operator will free run (up to a point) even

when it has to write data to disk.

 APT_BUFFER_MAXIMUM_MEMORY
o Maximum memory consumed by each buffer operator for data storage. Default is 3 MB.

 APT_BUFFERING_POLICY
o Specifies the buffering policy for the entire job. When it is not defined or defined to be
the null string, the default buffering policy is AUTOMATIC_BUFFERING. Valid
settings are:
 AUTOMATIC_BUFFERING: buffer as necessary to prevent dataflow
deadlocks
 FORCE_BUFFERING: buffer all virtual Data Sets
 NO_BUFFERING: inhibit buffering on all virtual Data Sets

WARNING: Inappropriately specifying NO_BUFFERING can cause dataflow deadlock during job
execution; use of this setting is only recommend for advanced users!

FORCE_BUFFERING can be used to reveal bottlenecks in a job design during development

and performance tuning, but will almost certainly degrade performance and therefore shouldn’t
be used in production job runs.

Additionally, the buffer mode, buffer size, buffer free run, queue bound, and write increment can be set
on a per-stage basis from the Input/ Advanced tab of the stage properties, as shown in the illustration
below:

Aside from ensuring that no dead-lock occurs, BufferOps also have the effect of “smoothing out”
production/consumption spikes. This allows the job to run at the highest rate possible even when a

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 138 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

downstream stage is ready for data at different times than

when its upstream stage is ready to produce that data. When
attempting to address these mismatches in production/consumption, it is best to tune the buffers on a
per-stage basis, instead of globally through environment variable settings.

Choosing which stages to tune buffering for and which to leave alone is as much art as science,
and should be considered among the last resorts for performance tuning. Stages
upstream/downstream from high-latency stages (such as remote databases, NFS mount points for data
storage, etc.) are a good place to start. If that doesn’t yield enough of a performance boost (remember
to test iteratively, while changing only 1 thing at a time), then setting the buffering policy to
“FORCE_BUFFERING” will cause buffering to occur everywhere.

By using the performance statistics in conjunction with this buffering, you may be able identify points
in the data flow where a downstream stage is waiting on an upstream stage to produce data. Each place
may offer an opportunity for buffer tuning.

As implied above, when a buffer has consumed its RAM, it will ask the upstream stage to “slow down”
- this is called “pushback”. Because of this, if you do not have force buffering set and
APT_BUFFER_FREE_RUN set to at least ~1000, you cannot determine that any one stage is waiting
on any other stage, as some other stage far downstream could be responsible for cascading pushback all
the way upstream to the place you are seeing the bottleneck.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 139 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix A: Standard Practices

Summary
This Appendix summarizes Standard Practices recommendations outlined in this document, along with
cross-references for more detail.

1. Standards
It is important to establish and follow consistent standards in:
 Directory structures for install and application support directories. An example directory
naming structure is given in Section 2.1:Directory Structures.
 Naming conventions, especially for DataStage Project categories, stage names, and links.
An example DataStage naming structure is given in Section 2.2: Naming Conventions.

All DataStage jobs should be documented with Short Description field, as well as Annotation fields.
See Section 2.3: Documentation and Annotation.

It is the DataStage developer’s responsibility to make personal backups of their work on their local
workstation, using the Manager DSX export capability. This can also be used for integration with
source code control systems. See Section 2.4: Working with Source Code Control Systems.

2. Development Guidelines
Modular development techniques should be used to maximize re-use of DataStage jobs and
components, as outlined in Section 3:Development Guidelines:
 Job parameterization allows a single job design to process similar logic instead of creating
multiple copies of the same job. The Multiple-Instance job property allows multiple invocations
of the same job to run simultaneously.
 A set of standard job parameters should be used in DataStage jobs for source and target
database parameters (DSN, user, password, etc) and directories where files are stored. To ease
re-use, these standard parameters and settings should be made part of a Designer Job Template.
 Create a standard directory structure outside of the DataStage project directory for source and
target files, intermediate work files, and so forth.
 Where possible, create re-usable components such as parallel shared containers to encapsulate
frequently-used logic.

DataStage Template jobs should be created with:

 standard parameters (for example, source and target file paths, database login properties…)
 environment variables and their default settings (as outlined in Section 2.5.1 Environment
Variable Settings)
 annotation blocks

Job Parameters should always be used for file paths, file names, database login settings. The scope of
a parameter is discussed further in Section 3.5: Job Parameters.
Parallel Framework Red Book: Data Flow Job Design July 17, 2006 140 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Parallel Shared Containers should be used to encapsulate

frequently-used logic, using RCP to maximize re-use.

Standardized Error Handling routines should be followed to capture errors and rejects. Further details
are provided in Section 3.7:Error and Reject Record Handling.

3. Component Usage
As discussed in Section 3.8: Component Usage, the following guidelines should be followed when
constructing parallel jobs in DS/EE:
 Never use Server Edition components (BASIC Transformer, Server Shared Containers) within
a parallel job. BASIC Routines are appropriate only for job control sequences.
 Always use parallel Data Sets for intermediate storage between jobs.
 Use the Copy stage as a placeholder for iterative design, and to facilitate default type
conversions.
 Use the parallel Transformer stage (not the BASIC Transformer) instead of the Filter or Switch
stages.
 Use BuildOp stages only when logic cannot be implemented in the parallel Transformer.

4. DataStage Data Types

 Be aware of the mapping between DataStage (SQL) data types and the internal DS/EE data
types, as outlined in Section 4:DataStage Data Types.
 Leverage default type conversions using the Copy stage or across the Output mapping tab of
other stages.

5. Partitioning Data
Given the numerous options for keyless and keyed partitioning, the following objectives help to form a
methodology for assigning partitioning:

Objective 1: Choose a partitioning method that gives close to an equal number of rows in
each partition, while minimizing overhead.

This ensures that the processing workload is evenly balanced, minimizing overall run time.

Objective 2: The partition method must match the business requirements and stage
functional requirements, assigning related records to the same partition if required

Any stage that processes groups of related records (generally using one or more key columns)
must be partitioned using a keyed partition method.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 141 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

This includes, but is not limited to: Aggregator,

Change Capture, Change Apply, Join, Merge, Remove
Duplicates, and Sort stages. It may also be necessary for Transformers and BuildOps that
process groups of related records.

Note that in satisfying the requirements of this second objective, it may not be possible to
choose a partitioning method that gives close to an equal number of rows in each partition.

Objective 3: Unless partition distribution is highly skewed, minimize repartitioning,

especially in cluster or Grid configurations
Repartitioning data in a cluster or Grid configuration incurs the overhead of network transport.

Objective 4: Partition method should not be overly complex

The simplest method that meets the above objectives will generally be the most efficient and
yield the best performance.

Using the above objectives as a guide, the following methodology can be applied:
a) Start with Auto partitioning (the default)
b) Specify Hash partitioning for stages that require groups of related records
o Specify only the key column(s) that are necessary for correct grouping as long as the
number of unique values is sufficient
o Use Modulus partitioning if the grouping is on a single integer key column
o Use Range partitioning if the data is highly skewed and the key column values and
distribution do not change significantly over time (Range Map can be reused)
c) If grouping is not required, use Round Robin partitioning to redistribute data equally across
all partitions
o Especially useful if the input Data Set is highly skewed or sequential
d) Use Same partitioning to optimize end-to-end partitioning and to minimize repartitioning
o Being mindful that Same partitioning retains the degree of parallelism of the
upstream stage
o Within a flow, examine up-stream partitioning and sort order and attempt to preserve
for down-stream processing. This may require re-examining key column usage
within stages and re-ordering stages within a flow (if business requirements permit).
Across jobs, persistent Data Sets can be used to retain the partitioning and sort order. This is
particularly useful if downstream jobs are run with the same degree of parallelism (configuration file)
and require the same partition and sort order.

Further details on Partitioning methods can be found in Section 5: Partitioning and Collecting.

6. Collecting Data
Given the options for collecting data into a sequential stream, the following guidelines form a
methodology for choosing the appropriate collector type:
a) When output order does not matter, use Auto partitioning (the default)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 142 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

b) When the input Data Set has been sorted in

parallel, use Sort Merge collector to produce a
single, globally sorted stream of rows
o When the input Data Set has been sorted in parallel and Range partitioned, the Ordered
collector may be more efficient
c) Use a Round Robin collector to reconstruct rows in input order for round-robin partitioned
input Data Sets, as long as the Data Set has not been repartitioned or reduced.

Further details on Partitioning methods can be found in Section 5: Partitioning and Collecting.

7. Sorting
Using the rules and behavior outlined in Section 6: Sorting, the following methodology should be
applied when sorting in a DataStage Enterprise Edition data flow:

a) Start with a link sort

b) Specify only necessary key column(s)
c) Don’t use Stable Sort unless needed
d) Use a stand-alone Sort stage instead of a Link sort for options that not available on a Link
sort:
 Sort Key Mode, Create Cluster Key Change Column, Create Key Change Column, Output
Statistics
 Always specify “DataStage” Sort Utility for standalone Sort stages
 Use the “Sort Key Mode=Don’t Sort (Previously Sorted)” to resort a sub-grouping of a
previously-sorted input Data Set
e) Be aware of automatically-inserted sorts
 Set $APT_SORT_INSERTION_CHECK_ONLY to verify but not establish required sort
order
f) Minimize the use of sorts within a job flow
g) To generate a single, sequential ordered result set use a parallel Sort and a Sort Merge
collector

8. Stage-Specific Guidelines
 As discussed in Section 8.1.1: Transformer NULL Handling and Reject Link, precautions
must be taken when using expressions or derivations on nullable columns within the parallel
Transformer:
o Always convert nullable columns to in-band values before using them in an expression
or derivation.
o Always place a reject link on a parallel Transformer to capture / audit possible rejects.
 The Lookup stage is most appropriate when reference data is small enough to fit into
available memory. If the Data Sets are larger than available memory resources, use the Join
or Merge stage. See Section 9.1: Lookup vs. Join vs. Merge.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 143 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

 Limit the use of database Sparse Lookups to

scenarios where the number of input rows is
significantly smaller (for example 1:100 or more) than the number of reference rows, or
when exception processing.
 Be particularly careful to observe the nullability properties for input links to any form of
Outer Join. Even if the source data is not nullable, the non-key columns must be defined as
nullable in the Join stage input in order to identify unmatched records. (See Section
9.2Capturing Unmatched Records from a Join).
 Use Hash method Aggregators only when the number of distinct key column values is
small. A Sort method Aggregator should be used when the number of distinct key values is
large or unknown.

9. Database Stage Guidelines

 Where possible, use the native parallel database stages for maximum performance and
scalability, as discussed in Section 10.1.1: Database stage types:

Native Parallel Database Stages

DB2/UDB Enterprise
Informix Enterprise
ODBC Enterprise
Oracle Enterprise
SQL Server Enterprise
Teradata Enterprise

 The ODBC Enterprise stage should only be used when a native parallel stage is not available
for the given source or target database.
 When using Oracle, DB2, or Informix databases, use orchdbutil to properly import design
metadata.
 Care must be taken to observe the data type mappings documented in Section 10: Database
Stage Guidelines when designing a parallel job with DS/EE.
 If possible, use a SQL where clause to limit the number of rows sent to a DataStage job.
 Avoid the use of database stored procedures on a per-row basis within a high-volume data flow.
For maximum scalability and parallel performance, it is best to implement business rules
natively using DataStage parallel components.

10. Troubleshooting and Monitoring

 Always test DS/EE jobs with a parallel configuration file ($APT_CONFIG_FILE) that has
two or more nodes in its default pool.
 Check the Director log for warnings, which may indicate an underlying problem or data type
conversion issue. All warnings and failures should be addressed (and removed if possible)
before deploying a DS/EE job.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 144 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

 The environment variable $DS_PX_DEBUG can be

used to capture all generated OSH, error and warning
messages from a running DS/EE job.
 Set the environment variable $OSH_PRINT_SCHEMAS to capture actual runtime schema to
the Director log. Set $DS_PX_DEBUG if the schema record is too large to capture in a
Director log entry.
 Enable $APT_DUMP_SCORE by default, and examine the job score by following the
guidelines outlined in Section 12.4.2 Understanding the Parallel Job Score.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 145 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix B: DataStage Naming

Reference
Every name should be based on a three-part concept: Subject, Subject Modifier, Class Word where the
following frequently-used Class Words describe the object type, or the function the object performs:
Project Repository and Components Tail Tail
Development <proj>Dev
Test <proj>Test File Stages
Production <proj>Prod Sequential File SF
BuildOp BdOp<name> Complex Flat File CFF
Parallel External Function XFn<name> File Set FS
Wrapper Wrap<name> Parallel Data Set DS
Lookup File Set LFS
Job Names and Properties External Source XSrc
Extract Job Src<job> XTgt
External Target
Load Load<job> SASd
Parallel SAS Data Set
Sequence <job>_Seq
Parallel Shared Container <job>Psc Processing Stages
Server Shared Container <job>Ssc Aggregator Agg
Parameter <name>Parm Change Apply ChAp
Change Capture ChCp
Links (prefix with “lnk_”) Copy Cp
Reference (Lookup) Ref Filt
Filter
Reject (Lookup, File, DB) Rej Funl
Funnel
Message (Sequence) Msg InJn
Join (Inner)
Get (Shared Container) Get LOJn
Join (Left Outer)
Put (Shared Container) Put ROJn
Join (Right Outer)
Input In FOJn
Join (Full Outer)
Output Out Lkp
Lookup
Delete Del Mrg
Merge
Insert Ins Mod
Modify
Update Upd Pivt
Pivot
Data Store Remove Duplicates RmDp
Database DB SAS processing SASp
Stored Procedure SP
Table Tab Sort Srt
View View Surrogate Key Generator SKey
Dimension Dim
Fact Fact Switch Swch
Source Src
Target Tgt Transformer Stage
Transformer (native parallel) Tfm
Development / Debug Stages BASIC Transformer (Server) BTfm
Column Generator CGen Stage Variable SV
Head Head
Peek Peek Real Time Stages
Row Generator RGen RTI Input RTIi
Sample Smpl RTI Output RTIo
XML Input XMLi

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 146 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

XML Output XMLo

XML Transformer XMLt

Restructure Stages
Column Export CExp
Column Import CImp

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 147 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix C: Understanding the

Parallel Job Score
Jobs developed in DataStage Enterprise Edition are independent of the actual hardware and degree of
parallelism used to run the job. The parallel Configuration File provides a mapping at runtime
between the compiled job and the actual runtime infrastructure and resources by defining logical
processing nodes.

At runtime, Enterprise Edition uses the given job design and configuration file to compose a job score
which details the processes created, degree of parallelism and node (server) assignments, and
interconnects (Data Sets) between them. Similar to the way a parallel database optimizer builds a query
plan, the DS/EE job score:

• Identifies degree of parallelism and node assignment(s) for each operator

• Details mappings between functional (stage/operator) and actual operating system processes
• Includes operators automatically inserted at runtime:
o Buffer operators to prevent deadlocks and optimize data flow rates between stages
o Sorts and Partitioners that have been automatically inserted to ensure correct results
• Outlines connection topology (Data Sets) between adjacent operators and/or persistent Data
Sets
• Defines number of actual operating system processes

Where possible, multiple operators are combined within a single operating system process to improve
performance and optimize resource requirements.

1. Viewing the Job Score

When the environment variable APT_DUMP_SCORE is set, the job score is output to the DataStage
Director log. It is recommended that this setting be enabled by default at the project level, as the job
score offers invaluable data for debugging and performance tuning, and the overhead to capture the
score is negligible.

As shown in the illustration below, job score entries start with the phrase “main_program: This step has
n datasets…” Two separate scores are written to the log for each job run. The first score is from the
license operator, not the actual job, and can be ignored. The second score entry is the actual job score.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 148 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Actual job score

2. Parallel Job Score Components

The Enterprise Edition parallel job score is divided into two sections, as shown in the example on the
right:
• Data Sets: starts with the words “main_program: This step has n datasets:”
The first section details all Data Sets, including persistent (on disk) and virtual (in memory,
links between stages). Terminology in this section can be used to identify the type of
partitioning or collecting that was used between operators. In this example, there are two virtual
Data Sets.

• Operators: starts with the words “It has n operators:”

The second section details actual operators created to execute the job flow. This includes:
o Sequential or Parallel operation, and the degree of parallelism per operator
o Node assignment for each operator. The actual node names correspond to node names in
the parallel configuration file. (in this example: “node1”, “node2”, “node3”, “node4”).
In this example, there are 3 operators, one running sequentially, two running in parallel across 4
nodes, for a total of 9 operating system process.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 149 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Note that the number of virtual Data Sets and the degree of
parallelism determine the amount of memory used by the inter-operator transport buffers. The memory
used by deadlock-prevention BufferOps can be calculated based on the number of inserted BufferOps.

3. Job Score: Data Sets

The parallel pipeline architecture of DataStage Enterprise Edition passes data from upstream producers
to downstream consumers through in-memory virtual data sets.

Data Sets are identified in the first section of the

parallel job score, with each Data Set identified by its
number (starting at zero). In the above example, the
first Data Set is identified as “ds0”, and the next
“ds1”.

Producers and consumers may be either persistent (on

disk) Data Sets or parallel operators. Persistent Data
Sets are identified by their Data Set name, while
operators are identified by their operator number and
name, corresponding to the lower section of the job
score, as illustrated in the example on the right:

The degree of parallelism is identified in brackets after the operator name. For example, in the example
on the right, operator zero (op0) is running sequentially, with 1 degree of parallelism [1p]. Operator 1
(op1) is running in parallel with 4 degrees of parallelism [4p].

Within the Data Set definition, the upstream producer is identified first, followed by a notation to
indicate the type of partitioning or collecting (if any), followed by the downstream consumer.
Producer

Partitioner Collector
Consumer
The notation between producer and consumer is used to report the type of partitioning or collecting (if
any) that is applied. The partition type is associated with the first term, collector type with the second.
The symbol between the partition name and collector name indicates:

-> Sequential producer to Sequential consumer

<> Sequential producer to Parallel consumer
=> Parallel producer to Parallel consumer (SAME partitioning)
#> Parallel producer to Parallel consumer (repartitioned; not SAME)
>> Parallel producer to Sequential consumer
> No producer or no consumer (typically, for persistent Data Sets)

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 150 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Finally, if the Preserve Partitioning flag has been set for a

particular Data Set, the notation “[pp]” will appear in this section of the job score.

4. Job Score: Operators

The lower portion of the parallel job score details
op0[1p] {(sequential
the mapping between stages and actual processes APT_CombinedOperatorController:
generated at runtime. For each operator, this (Row_Generator_0)
(inserted tsort operator {key={value=LastName},
includes (as illustrated in the job score fragment): key={value=FirstName}})
• operator name (opn) numbered sequentially ) on nodes (
node1[op0,p0]
from zero (example “op0”) )}
• degree of parallelism within brackets op1[4p] {(parallel inserted tsort operator
{key={value=LastName},
(example “[4p]”) key={value=FirstName}}(0))
• “sequential” or “parallel” execution mode on nodes (
node1[op2,p0]
• Components of the operator node2[op2,p1]
node3[op2,p2]
o typically correspond to the user- node4[op2,p3]
specified stage name in the Designer )}
op2[4p] {(parallel buffer(0))
canvas on nodes (
o may also include combined node1[op2,p0]
node2[op2,p1]
operators node3[op2,p2]
(APT_CombinedOperatorController) node4[op2,p3]
which include logic from multiple stages in a single operator
o may also include framework-inserted operators such as Buffers, Sorts
o may include “composite” operators (for example, Lookup)

Some stages are composite operators – to the DataStage developer, a composite operator appears to be
a single stage on the design canvas. But internally, a composite operator includes more than one
function.

For example, Lookup is a composite operator. It is composted of the following internal operators:
- APT_LUTCreateImpl:
op2[1p] {(parallel APT_LUTCreateImpl in Lookup_3)
Reads the reference data into memory on nodes (
ecc3671[op2,p0]
- APT_LUTProcessImpl: )}
op3[4p] {(parallel buffer(0))
Performs actual lookup processing once reference on nodes (
ecc3671[op3,p0]
data has been loaded ecc3672[op3,p1]
ecc3673[op3,p2]
ecc3674[op3,p3]
)}
At runtime, each individual component of a composite op4[4p] {(parallel APT_CombinedOperatorController:
(APT_LUTProcessImpl in Lookup_3)
operator is represented as an individual operator in the (APT_TransformOperatorImplV0S7_cpLookupTest1_Tran
sformer_7 in Transformer_7)
job score, as shown in the following score fragment (PeekNull)
) on nodes (
shown on the right: ecc3671[op4,p0]
ecc3672[op4,p1]
ecc3673[op4,p2]

Using this information together with the output from the

$APT_PM_SHOW_PIDS environment variable, you can evaluate the memory used by a lookup. Since

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 151 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

the entire structure needs to be loaded before actual lookup

processing can begin, you can also determine the delay
associated with loading the lookup structure.

In a similar way, a persistent Data Set defined to “Overwrite” an existing Data Set of the same name
will have multiple entries in the job score to: main_program: This step has 2 datasets:
ds0: {op1[1p] (parallel delete data files in delete temp.ds)
- Delete Data Files ->eCollectAny
op2[1p] (sequential delete descriptor file in delete temp.ds)}
- Delete Descriptor File ds1: {op0[1p] (sequential Row_Generator_0)
->
temp.ds}
It has 3 operators:
op0[1p] {(sequential Row_Generator_0)
on nodes (
node1[op0,p0]
)}
op1[1p] {(parallel delete data files in delete temp.ds)
on nodes (
node1[op1,p0]
)}
op2[1p] {(sequential delete descriptor file in delete temp.ds)
on nodes (
node1[op2,p0]
)}

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 152 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix D: Estimating the Size of a

Parallel Data Set
For the advanced user, this Appendix provides a more accurate and detailed way to estimate the size of
a parallel Data Set based on the internal storage requirements for each data type:

Data Type Size

Integers 4 bytes
Small Integer 2 bytes
Tiny Integer 1 byte
Big Integer 8 bytes
Decimal (precision+1)/2, rounded up
Float 8 bytes
VarChar(n) n + 4 bytes for non-NLS data
2n + 4 bytes for NLS data (internally stored as UTF-16)
Char(n) n bytes for non-NLS data
2n bytes for NLS data
Time 4 bytes
8 bytes with microsecond resolution
Date 4 bytes
Timestamp 8 bytes
12 bytes with microsecond resolution

For the overall record width:

- add (# nullable fields)/8 for null indicators
- one byte per column for field alignment (worst case is 3.5 bytes per field)

Using the internal DataStage Enterprise Edition C++ libraries, the method
APT_Record::estimateFinalOutputSize() can give you an estimate for a given record schema. As can
APT_Transfer::getTransferBufferSize(), if you have a transfer that transfers all fields from input to
output.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 153 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix E: Environment Variable

Reference
This Appendix summarizes the environment variables mentioned throughout this document. These
variables can be used on an as-needed basis to tune the performance of a particular job flow, to assist in
debugging, or to change the default behavior of specific DataStage Enterprise Edition stages. An
extensive list of environment variables is documented in the DataStage Parallel Job Advanced
Developer’s Guide.

NOTE: The environment variable settings in this Appendix are only examples. Set values that are
optimal to your environment.

1. Job Design Environment Variables

Environment Variable Setting Description
$APT_STRING_PADCHAR [char] Overrides the default pad character of 0x0 (ASCII null)
used when EE extends, or pads, a variable-length string
field to a fixed length (or a fixed-length to a longer
fixed-length).
See section 4.1.2: Default and Explicit Type
Conversions

2. Sequential File Stage Environment Variables

Environment Variable Setting Description
$APT_EXPORT_FLUSH_COUNT [nrows] Specifies how frequently (in rows) that the
Sequential File stage (export operator)
flushes its internal buffer to disk. Setting this
value to a low number (such as 1) is useful
for realtime applications, but there is a small
performance penalty from increased I/O.
$APT_IMPORT_REJECT_STRING_FIELD_OVERRUN 1 Setting this environment variable directs
S DataStage to reject Sequential File records
with strings longer than their declared
(DataStage v7.01 and later) maximum column length. By default,
imported string fields that exceed their
maximum declared length are truncated.
$APT_IMPORT_BUFFER_SIZE [Kbytes] Defines size of I/O buffer for Sequential File
reads (imports) and writes (exports)
respectively. Default is 128 (128K), with a
$APT_EXPORT_BUFFER_SIZE minimum of 8. Increasing these values on
heavily-loaded file servers may improve
performance.
$APT_CONSISTENT_BUFFERIO_SIZE [bytes] In some disk array configurations, setting this
variable to a value equal to the read / write
size in bytes can improve performance of
Sequential File import/export operations.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 154 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

$APT_DELIMITED_READ_SIZE [bytes]
Specifies the number of bytes the Sequential
File (import) stage reads-ahead to get the
next delimiter. The default is 500 bytes, but
this can be set as low as 2 bytes.

This setting should be set to a lower value

when reading from streaming inputs (for
example, socket or FIFO) to avoid blocking.
$APT_MAX_DELIMITED_READ_SIZE [bytes] By default, Sequential File (import) will read
ahead 500 bytes to get the next delimiter. If it
is not found the importer looks ahead
4*500=2000 (1500 more) bytes, and so on
(4X) up to 100,000 bytes.

This variable controls the upper bound which

is by default 100,000 bytes. When more than
500 bytes read-ahead is desired, use this
variable instead of
APT_DELIMITED_READ_SIZE.
$APT_IMPORT_PATTERN_USES_FILESET [set] When this environment variable is set
(present in the environment) file pattern reads
are done in parallel by dynamically building
a File Set header based on the list of files that
match the given expression. For disk
configurations with multiple controllers and
disk, this will significantly improve file
pattern reads.

3. DB2 Environment Variables

Environment Variable Setting Description
$INSTHOME [path] Specifies the DB2 install directory. This variable is
usually set in a user’s environment from
.db2profile.
$APT_DB2INSTANCE_HOME [path] Used as a backup for specifying the DB2
installation directory (if $INSTHOME is undefined).
$APT_DBNAME [database] Specifies the name of the DB2 database for
DB2/UDB Enterprise stages if the “Use Database
Environment Variable” option is True. If
$APT_DBNAME is not defined, $DB2DBDFT is
used to find the database name.
$APT_RDBMS_COMMIT_ROWS [rows] Specifies the number of records to insert between
Can also be specified with the “Row Commit commits. The default value is 2000 per partition.
Interval” stage input property.
$DS_ENABLE_RESERVED_CHAR_CONVE 1 Allows DataStage plug-in stages to handle DB2
RT databases which use the special characters # and $
in column names.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 155 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

4. Informix
Environment Variables
Environment Variable Setting Description
$INFORMIXDIR [path] Specifies the Informix install directory.
$INFORMIXSQLHOSTS [filepath] Specifies the path to the Informix sqlhosts file.
$INFORMIXSERVER [name] Specifies the name of the Informix server
matching an entry in the sqlhosts file.
$APT_COMMIT_INTERVAL [rows] Specifies the commit interval in rows for
Informix HPL Loads. The default is 10000 per
partiton.

5. Oracle Environment Variables

Environment Variable Setting Description
$ORACLE_HOME [path] Specifies installation directory for current
Oracle instance. Normally set in a user’s
environment by Oracle scripts.
$ORACLE_SID [sid] Specifies the Oracle service name,
corresponding to a TNSNAMES entry.
$APT_ORAUPSERT_COMMIT_ROW_INTERVAL [num] These two environment variables work
[seconds] together to specify how often target rows are
$APT_ORAUPSERT_COMMIT_TIME_INTERVAL
committed for target Oracle stages with
Upsert method.

Commits are made whenever the time

interval period has passed or the row interval
is reached, whichever comes first. By default,
commits are made every 2 seconds or 5000
rows per partition.
$APT_ORACLE_LOAD_OPTIONS [SQL* Specifies Oracle SQL*Loader options used
Loader in a target Oracle stage with Load method.
options] By default, this is set to
OPTIONS(DIRECT=TRUE,
PARALLEL=TRUE)
$APT_ORACLE_LOAD_DELIMITED [char] Specifies a field delimiter for target Oracle
stages using the Load method. Setting this
(DataStage 7.01 and later) variable makes it possible to load fields with
trailing or leading blank characters.
$APT_ORA_IGNORE_CONFIG_FILE_PARALLELIS 1 When set, a target Oracle stage with Load
M method will limit the number of players to
the number of datafiles in the table’s
tablespace.
$APT_ORA_WRITE_FILES [filepath] Useful in debugging Oracle SQL*Loader
issues. When set, the output of a Target
Oracle stage with Load method is written to
files instead of invoking the Oracle
SQL*Loader. The filepath specified by this
environment variable specifies the file with
the SQL*Loader commands.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 156 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

$DS_ENABLE_RESERVED_CHAR_CONVERT 1
Allows DataStage plug-in stages to handle
Oracle databases which use the special
characters # and $ in column names.

6. Teradata Environment Variables

Environment Variable Setting Description
$APT_TERA_SYNC_DATABASE [name] Starting with v7, specifies the database used for
the terasync table. By default, EE uses the
$APT_TERA_SYNC_USER [user] Starting with v7, specifies the user that creates
and writes to the terasync table.
$APT_TER_SYNC_PASSWORD [password] Specifies the password for the user identified by
$APT_TERA_SYNC_USER.
$APT_TERA_64K_BUFFERS 1 Enables 64K buffer transfers (32K is the
default). May improve performance depending
on network configuration.
$APT_TERA_NO_ERR_CLEANUP 1 This environment variable is not recommended
for general use. When set, this environment
variable may assist in job debugging by
preventing the removal of error tables and
partially written target table.
$APT_TERA_NO_PERM_CHECKS 1 Disables permission checking on Teradata
system tables that must be readable during the
TeraData Enterprise load process. This can be
used to improve the startup time of the load.

7. Job Monitoring Environment Variables

Environment Variable Setting Description
$APT_MONITOR_TIME [seconds] In v7 and later, specifies the time interval (in seconds)
for generating job monitor information at runtime. To
enable size-based job monitoring, unset this
environment variable, and set $APT_MONITOR_SIZE
below.
$APT_MONITOR_SIZE [rows] Determines the minimum number of records the job
monitor reports. The default of 5000 records is usually
too small. To minimize the number of messages during
large job runs, set this to a higher value (for example,
1000000).
$APT_NO_JOBMON 1 Disables job monitoring completely. In rare instances,
this may improve performance. In general, this should
only be set on a per-job basis when attempting to
resolve performance bottlenecks.
$APT_RECORD_COUNTS 1 Prints record counts in the job log as each operator
completes processing. The count is per operator per
partition.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 157 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

8. Performance-
Tuning Environment Variables
Environment Variable Setting Description
$APT_BUFFER_MAXIMUM_MEMORY 41903040 Specifies the maximum amount of virtual memory, in
(example) bytes, used per buffer per partition. If not set, the
default is 3MB (3145728). Setting this value higher
will use more memory, depending on the job flow, but
may improve performance.
$APT_BUFFER_FREE_RUN 1000 Specifies how much of the available in-memory buffer
(example) to consume before the buffer offers resistance to any
new data being written to it. If not set, the default is 0.5
(50% of $APT_BUFFER_MAXIMUM_MEMORY).

If this value is greater than 1, the buffer operator will

read $APT_BUFFER_FREE_RUN *
$APT_BUFFER_MAXIMIMUM_MEMORY before
offering resistance to new data.
When this setting is greater than 1, buffer operators will
spool data to disk (by default scratch disk) after the
$APT_BUFFER_MAXIMUM_MEMORY threshold. The
maximum disk required will be
$APT_BUFFER_FREE_RUN * # of buffers *
$APT_BUFFER_MAXIMUM_MEMORY
$APT_PERFORMANCE_DATA directory Enables capture of detailed, per-process performance
data in an XML file in the specified directory. Unset
this environment variable to disable.
$TMPDIR [path] Defaults to /tmp. Used for miscellaneous internal
temporary data including FIFO queues and
Transformer temporary storage.
As a minor optimization, may be best set to a
filesystem outside of the DataStage install directory.

9. Job Flow Debugging Environment Variables

Environment Variable Setting Description
$OSH_PRINT_SCHEMAS 1 Outputs the actual schema definitions used by the
DataStage EE framework at runtime in the DataStage
log. This can be useful when determining if the actual
runtime schema matches the expected job design table
definitions.
$APT_DISABLE_COMBINATION 1 Disables operator combination for all stages in a job,
forcing each EE operator into a separate process. While
not normally needed in a job flow, this setting may help
when debugging a job flow or investigating
The Advanced Stage Properties editor in performance by isolating individual operators to
DataStage Designer v7.1 and later allows separate processes.
combination to be enabled and disabled
for on a per-stage basis. Note that disabling operator combination will generate
more UNIX processes, and hence require more system
resources (and memory). Disabling operator

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 158 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

combination also disables internal optimizations for job

efficiency and run-times.
$APT_PM_PLAYER_TIMING 1 Prints detailed information in the job log for each
operator, including CPU utilization and elapsed
processing time.
$APT_PM_PLAYER_MEMORY 1 Prints detailed information in the job log for each
operator when allocating additional heap memory.
$APT_BUFFERING_POLICY FORCE Forces an internal buffer operator to be placed between
every operator. Normally, the DataStage EE framework
inserts buffer operators into a job flow at runtime to
avoid deadlocks and improve performance.

Using $APT_BUFFERING_POLICY=FORCE in
combination with $APT_BUFFER_FREE_RUN
Setting effectively isolates each operator from slowing
$APT_BUFFERING_POLICY=FORCE is not upstream production. Using the job monitor
recommended for production job runs. performance statistics, this can identify which part of a
job flow is impacting overall performance.
$DS_PX_DEBUG 1 Set this environment variable to capture copies of the
job score, generated osh, and internal Enterprise
Edition log messages in a directory corresponding to
the job name. This directory will be created in the
“Debugging” sub-directory of the Project home
directory on the DataStage server.
$APT_PM_STARTUP_CONCURRENCY 5 This environment variable should not normally need to
be set. When trying to start very large jobs on heavily-
loaded servers, lowering this number will limit the
number of processes that are simultaneously created
when a job is started.
$APT_PM_NODE_TIMEOUT [seconds] For heavily loaded MPP or clustered environments, this
variable determines the number of seconds the
conductor node will wait for a successful startup from
each section leader. The default is 30 seconds.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 159 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Appendix F: Sorting and Hashing

Advanced Example
The standard recipe for using the ‘Inter-Record Relationship Suite’ (Sort, Join, Merge, and related
stages) is:

Hash and Sort/Join/Merge on exactly the same keys, in the same order.

This approach is guaranteed to work, but is frequently inefficient as records are ‘over-hashed’ and
‘over-partitioned’. There is also an “advanced” rule:

a) Hash on any sub-set of the keys

b) Sort (join/etc.) on any super-set of the keys, in order

This Appendix contains descriptions of what happens “behind the scenes”. It will be followed by a
detailed example that discusses these ideas in much greater depth. If you have a lot of experience with
hashing and sorting, this may be review for you. The second portion of this Appendix assumes you
have read and thoroughly understand these concepts.

a) Sort within partitions, not globally.

Sorting is rarely required by the business logic. In most cases, sorting is needed to satisfy an input
requirement of a downstream stage. RemoveDuplicates, Join, Merge, for example, all require sorted
inputs. The reason for this requirement lies in the “light-weight” nature of these stages.
RemoveDuplicates, for example, will notice that two rows have identical values in the user-defined key
column only if the two rows are contiguous. Join, for example, only needs to see two records at a time
—one from each input stream—to do its job.

b) Hash gathers
Hash gathers into the same partition, rows from all partitions that share the same value in key
columns. This creates partition-wise concurrency (a.k.a. partition-wise co-location), i.e. related rows
are in the same partition, at the same time, but other rows may separate them within that partition.

c) Sort clusters and orders

Sorting is often overkill, key-clustering is sufficient in many cases (a-only, below). Sort actually does
two things:
(i) Groups rows that share the same values in key columns (forces related rows to be
contiguous, a.k.a. record adjacency, key clustering, etc.)
(ii) Orders the clusters resulting from (i).
These operations take place in parallel, across all partitions.

Remove Duplicates requires only (i): when it completes processing all the rows in a key cluster, it
does not care about the key value of the next cluster with respect to the current key value—in part
because this stage takes only one input. An illustrative piece of information, but there is little you

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 160 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

can do to take advantage of it as there are no stages which

guarantee key-clustering but do not perform a sort (some
databases might be able to do key-clustering more cheaply than a sort, this is one instance where
this might be handy). As you will see in this Appendix, however, there are more advanced methods
to sort and partition that can leverage this capability and mitigate the cost of sorting vs. clustering.

Join and Merge, on the other hand, require both (i) and (ii). This due, in part, to the fact these stages
take multiple input links—these stages only see 2 records at a timei, one from each input, so row order
between the two inputs is obviously critical. When this stage completes processing all the rows in a
key cluster, they DO care about the key value of the next cluster. If the values on both inputs aren't
ordered (vs. grouped/clustered for remove duplicates), Join/Merge can't effectively choose which input
stream to advance for finding subsequent key matches.

d) Partitioners respect row order but split clusters.

On multi-partitioned (i.e., non-sequential) inputs, partitioners (except for SAME), reshuffle rows
among partitions.

Partitioners, (and most other stages) do not gratuitously alter existing intra-partition row order. In other
words, Enterprise Edition will not, as a rule, allow a row in a partition to jump ahead of another row in
the same partitionii. Nonetheless, any existing sort order is usually destroyed—see example below.

To restore row-adjacency, a sort operation is needed even on previously sorted columns following any
partitioner.

There is a component that will allow you to partition sorted data and achieve a sorted result:
parallelsortmerge (PSM). Enterprise Edition itself normally manages this use of this component, but it
can be invoked via the generic stage. Whenever you re-partition your sorted data, follow the partitioner
with a PSM, and your data will retain its previous sort order. See usage notes in footnotesiii.

1. Inside a partitioner
In Enterprise Edition, partitioners, like stages, work in parallel, for example:
Repartitioning:

P0 P1
2 103
1 102
Note that ‘1’ and ‘101’ have 3 101
switched partitions.

Note that neither partition has a Partitioner

sorted result despite P1 having a
sorted input P0 P1
(read row order from the bottom 2 1
up, ‘streaming-style’). 101 103

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 161 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

3 102

Example: 6 rows in 2 partitions. After Hash partitioning on Street/Tree:

Illustrated Above Before Partitioner Illustrated Above After Partitioner
Partition 0 Partition 0
3 Eve Smith Pine 3 Eve Smith3 Eve Smith Pine 3 Eve Smith
1 Orlando Jones Elm 2 Adam Smith10 Rose Jones Pine 1 Orlando Jones
2 Adam Smith Pine 10 Rose Jones1
1 2 Adam Smith Pine 10 Rose Jones
Partition 1 1
10 Rose Jones Pine 1 Orlando JonesPartition 1
1 10 Boris Smith Walnut 10 Boris Smith
10 Boris Smith Walnut 10 Boris Smith2 2
2 2 10 John Zorn Walnut 10 John Zorn
10 John Zorn Walnut 10 John Zorn3 3
3 3 1 Orlando Jones Elm 2 Adam Smith

There is more than one way to correctly hash-partition any Data Set6.
Here is another possible outcome:
Also: Consider the result of running the same job with the same data, but a different number of
partitions.
Partition 0

Partition 1
1 Orlando Jones Elm 2 Adam Smith
10 Rose Jones Pine 1 Orlando Jones
1
10 Boris Smith Walnut 10 Boris Smith
2 2
2 Adam Smith Pine 10 Rose Jones
1
10 John Zorn Walnut 10 John Zorn
3 3
3 Eve Smith Pine 3 Eve Smith

2. Minimizing Record Movement for

Maximizing Performance
Now we have covered the basic rules and mechanics for hash-partitioning and sorting, let’s look at how
we can capitalize on these behaviors for performance benefits.

6
There is an exception to this rule: If your hash key has only one value.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 162 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Scenario Description:
Our customer is a national retail business with several
hundred outlets nation-wide. They wish to determine the weighted average transaction amount per-
item nation-wide, as well as the average transaction amount per-item, per store for all stores in the
nation, and append these values to the original data. This would make it possible to determine how well
each store is doing in relation to the national averages and track these performance trends over time.
There are many common extensions on gathering these kinds of sales metrics that take the following
ideas and increase the scale of the problem at hand, thereby increasing the value of this exercise.

Here is our source data:

Data Set 1: 32 Rows
Store Item Transaction Transaction
Location ID Date Amt
1 1 2004-01-01 1
1 1 2004-01-02 2
1 1 2004-01-01 3
1 1 2004-01-03 5
1 1 2004-01-04 5
1 1 2004-01-02 54
1 1 2004-01-04 7
1 1 2004-01-03 8
1 2 2004-01-04 2
1 2 2004-01-03 3
1 2 2004-01-01 45
1 2 2004-01-04 65
1 2 2004-01-02 7
1 2 2004-01-02 85
1 2 2004-01-01 9
1 2 2004-01-03 98
2 1 2004-01-03 23
2 1 2004-01-01 3
2 1 2004-01-02 32
2 1 2004-01-04 45
2 1 2004-01-02 54
2 1 2004-01-03 56
2 1 2004-01-01 7
2 1 2004-01-04 8
2 2 2004-01-04 23
2 2 2004-01-01 45
2 2 2004-01-03 534
2 2 2004-01-02 6
2 2 2004-01-04 65
2 2 2004-01-02 7
2 2 2004-01-01 78
2 2 2004-01-03 87

The screen capture below shows how to implement the business logic in an efficient manner, taking
advantage of Enterprise Edition’s ability to analyze a jobflow and insert sorts and partitioners in

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 163 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

appropriate places notice, there is only one sort and one re-
partition in the diagram, both on the output link of
JoinSourceToAggregator_1):

NOTE: In this job, automatic sort insertion,

and automatic partition insertion must be turned on7.

The Aggregator stage NationalAverageItemTransactionAmt will aggregate the data on ‘Item ID’, and
‘Transaction Date’ calculating the average of the ‘TransactionAmt’ column and place the results in a
column named ‘National Average Item Transaction Amt’. This is the nation-wide transaction average
per item (weighted by transaction, not store). To do this, Enterprise Edition will hash-partition and sort
on ‘Item ID’, and ‘Transaction Date’.

The Aggregator StoreAverageItemTransactionAmt will aggregate the data on ‘Store ID’, ‘Item ID’, and
‘Transaction Date’, and calculate the average of the ‘Transaction Amt’ column and place the results in
a column named ‘Store Average Item Transaction Amt’. This is the per-store transaction average per
item. Here, Enterprise Edition will hash-partition and sort on ‘Store ID’, ‘Item ID’, and ‘Transaction
Date’.

Since the aggregator reduces row count (to the group count), we will need to join each aggregator’s
output back to the original data in order to get the original row count. This is done with
JoinSourceToAggregator_1 and JoinSourceToAggregator_2, to get the original data with the averages
appended.
7
To enable automatic partition insertion, ensure that the environment variable, APT_NO_PART_INSERTION
is NOT defined in your environment, with ANY value. If you allow this environment variable to exist
with any value, you will disable this facility.

To enable automatic sort insertion, ensure that the environment variable, APT_NO_SORT_INSERTION is NOT
defined in your environment, with ANY value. If you allow this environment variable to exist with any
value, you will disable this facility.

Here you want to let DS/EE choose where to insert sorts and partitioners for you, so you want to leave
them enabled (the default).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 164 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

NOTE: For this scenario, you will need to set the

Aggregator’s “Method” to Sort, not Hash. The sort method
requires the input data to be hashed and sorted; in return, it guarantees the output order to be sorted
since the result of aggregation can be released for downstream processing as soon as the key change is
detected. The hash method only requires the input data to be hashed, however, it does not guarantee
output order. It does this by keeping running totals in memory for the aggregation for each output
group. Therefore, it consumes an amount of RAM proportionate to the number of output rows and the
number of columns involved in the aggregation.
JoinSourceToAggregator_2 produces the final result: the original input Data Set with two columns
appended (‘Average Item Transaction Amt’, and ‘Average Item Transaction Amt By Store’)
The output Data Set should look something like this
(A 3-node configuration file was used in this implementation):
Data Set 2: 32 Rows
PeekFinalOutput, Partition 0: 16 Rows
Store Item Transaction Transaction National Store
Location ID Date Amt Average Average
Item Item
Transaction Transaction
Amt Amt
1 1 2004/01/02 2 35.5 28
1 1 2004/01/02 54 35.5 28
2 1 2004/01/02 54 35.5 43
2 1 2004/01/02 32 35.5 43
1 1 2004/01/03 5 23 6.5
1 1 2004/01/03 8 23 6.5
2 1 2004/01/03 56 23 39.5
2 1 2004/01/03 23 23 39.5
1 2 2004/01/02 85 26.25 46
1 2 2004/01/02 7 26.25 46
2 2 2004/01/02 6 26.25 6.5
2 2 2004/01/02 7 26.25 6.5
1 2 2004/01/03 3 180.5 50.5
1 2 2004/01/03 98 180.5 50.5
2 2 2004/01/03 87 180.5 310.5
2 2 2004/01/03 534 180.5 310.5

PeekFinalOutput, Partition 2: 16 Rows

Store Item Transaction Transaction National Store
Location ID Date Amt Average Average
Item Item
Transaction Transaction
Amt Amt
1 1 2004/01/01 1 3.5 2
1 1 2004/01/01 3 3.5 2
2 1 2004/01/01 3 3.5 5
2 1 2004/01/01 7 3.5 5
1 1 2004/01/04 7 16.25 6
1 1 2004/01/04 5 16.25 6

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 165 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2 1 2004/01/04 8
16.25
2 1 2004/01/04 45 16.25 26.5
1 2 2004/01/01 45 44.25 27
1 2 2004/01/01 9 44.25 27
2 2 2004/01/01 78 44.25 61.5
2 2 2004/01/01 45 44.25 61.5
1 2 2004/01/04 2 38.75 33.5
1 2 2004/01/04 65 38.75 33.5
2 2 2004/01/04 65 38.75 44
2 2 2004/01/04 23 38.75 44

Since both the Aggregator and Join expect the data to arrive hashed and sorted on the grouping key(s)
—both operations that consume large amounts of CPU—a couple of questions arise with respect to
efficiency:
What is the minimum number of hash-partitioners needed to implement this solution correctly?
What is the minimum number of sorts needed to implement this solution?
What is the minimum number of times that sort will need to buffer the entire Data Set to disk to
implement this solution?

Though running the job sequentially eliminates questions related to partitioners, even sequential job
execution does not alter the answer for the sort-related questions, as only partition concurrency is
affected by sequential execution, i.e.: record adjacency assumes partition concurrency8.

An examination of the job above would suggest: 6, 6, and 6. A deeper examination (of the score dump,
appended to the end of this document for masochistsiv) might suggest: 4, 3, and 3. This is certainly an
improvement on the previous answer.

A much better answer is: 1, 3, and 1. Here’s a screen shot of this, more efficientv, solution (score dump
also attached below vi):

8
Records cannot be adjacent if they are not in the same partition.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 166 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

NOTE: In this job, automatic sort insertion, and automatic

partition insertion
must be turned offvii.

In our initial copy stage (DistributeCopiesOfSourceData), we hash and sort on ItemID and
TransactionDate only. Hashing on these fields will gather all unique combinations of ItemID and
TransactionDate into the same partition. This combination of hash and sort adequately prepares the
data for NationalAverageItemTransactionAmt, just as in the previous example.

However, the data is not properly prepared for StoreAverageItemTransactionAmt. What is wrong with
the data? The sort order does not include the StoreLocation. This is a problem for
StoreAverageItemTransactionAmt, as it expects all of the records for a particular
StoreLocation/TransactionDate/ItemId combination to arrive on the same partition, in order.

You may be wondering why the partitioning wasn’t mentioned as part of the problem. This is because
the data is already partitioned in a compatible manner for the aggregator. The ‘advanced’ rule for hash
partitioning is: you may partition on any sub-set of the aggregation/join/sort/etc. keys (viii This footnote
contains key concepts that this document addresses, but it is a lengthy parenthetical statement that
would interrupt the flow of the scenario discussion).

However, we still need to fix the sort order. One would expect that we would need to sort on
StoreLocation, TransactionDate, and ItemId, but we know that the data is already sorted on
TransactionDate, and ItemId. Sort offers an efficiency-mode for pre-sorted data, but you must use the
sort stage to access it, as it isn’t available on the link sort.

The settings in the sort should look like this:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 167 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

As you can see, we have instructed the sort stage that the data is already sorted on ItemID and
TransactionDate (as always with sorting records, key order is very important), and we only want to
‘sub-sort’ the data on StoreLocation (this option is only viable for situations where you need to
maintain the sort order on the initial keys). This lets sort know that it only needs to gather all records
with a unique combination of ItemID and TransactionDate in order to sort a batch of records, instead of
buffering the entire Data Set. If the group size was only several hundred records, but the entire Data Set
was 100 million records, this would save a tremendous amount of very expensive disk I/O as sort can
hold a few hundred records in memory in most cases (disk I/O is typically several orders of magnitude
more costly than memory I/O, even for ‘fast’ disks).

Also worth noting here: because we already hashed the data on ItemID and TransactionDate, ALL
extant values of the remaining columns are already in the same partition, which is what makes this sort
possible w/o re-partitioning (which is also quite expensive, especially in MPP environments where
repartitioning implies network I/O).

The previous two paragraphs contain two key concepts in Enterprise Edition (pun fully intended,
however dreadful).

Getting back to the aggregators. Since the aggregator does not need to disturb row-order (for pre-sorted
data), the rows will come out in the same order they went in (different rows, granted, but the group
keys will force the proper order). This means that the output of the DistributeCopiesOfSourceData and
NationalAverageItemTransactionAmt are already hashed and sorted on the keys needed to perform
JoinSourceToAggregator_1. This accomplishes the first goal, to append a column representing the
national (weighted) average item transaction amount.

The output of StoreAverageItemTransactionAmt contains the other column we need to append to our
source rows. However since we sub-sorted the data before this aggregator (unlike
NationalAverageItemTransactionAmt), we will have to prep the output from the first join to account for

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 168 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

the new row ordering of StoreAverageItemTransactionAmt.

This sort will look exactly like the other sort stage:

Remember to disable the ‘Stable Sort’ option if you do not need it (it will try to maintain row order
except as needed to perform the sort, useful for preserving previous sort orderings), as it is much more
expensive than non-stable sorts, and it is enabled by default.
Output from above solution:
Data Set 3:
PeekFinalOutput, Partition 1: 16 Rows
Store Item Transaction Transaction National Store
Location ID Date Amt Average Average
Item Item
Transaction Transaction
Amt Amt
1 1 2004/01/01 3 3.5 2
1 1 2004/01/01 1 3.5 2
1 1 2004/01/04 5 16.25 6
1 1 2004/01/04 7 16.25 6
2 1 2004/01/01 3 3.5 5
2 1 2004/01/01 7 3.5 5
2 1 2004/01/04 45 16.25 26.5
2 1 2004/01/04 8 16.25 26.5
1 2 2004/01/01 9 44.25 27
1 2 2004/01/01 45 44.25 27
2 2 2004/01/01 45 44.25 61.5
2 2 2004/01/01 78 44.25 61.5
1 2 2004/01/04 2 38.75 33.5
1 2 2004/01/04 65 38.75 33.5
2 2 2004/01/04 23 38.75 44
2 2 2004/01/04 65 38.75 44

PeekFinalOutput, Partition 2: 16 Rows

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 169 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

Store Item Transaction Transaction

Location ID Date Amt National Store
Average Average
Item Item
Transaction Transaction
Amt
1 1 2004/01/02 2 35.5 28
1 1 2004/01/02 54 35.5 28
2 1 2004/01/02 54 35.5 43
2 1 2004/01/02 32 35.5 43
1 1 2004/01/03 5 23 6.5
1 1 2004/01/03 8 23 6.5
1 2 2004/01/02 7 26.25 46
1 2 2004/01/02 85 26.25 46
2 2 2004/01/02 7 26.25 6.5
2 2 2004/01/02 6 26.25 6.5
1 2 2004/01/03 3 180.5 50.5
1 2 2004/01/03 98 180.5 50.5
2 1 2004/01/03 23 23 39.5
2 1 2004/01/03 56 23 39.5
2 2 2004/01/03 534 180.5 310.5
2 2 2004/01/03 87 180.5 310.5

This solution produces the same result but is achieved with only one complete sort, a single partitioner,
and two sub-sorts—a much more efficient solution for large data volumes.

Imagine a job with 100 million records as the input. With the initial solution, we had to sort (on disk)
300, 000,000 records in addition to hashing 300,000,000 records. The second solution only sorts (on
disk) 100,000,000 records, and only hashes, 100,000,000 records; a savings of 400,000,000 additional
record movements—half of them involving disk I/O—for a 100 million record input volume. That is a
LOT of saved processing power.

There is an even more efficient solution. It looks very similar to the first solution, but there is a critical
difference.

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 170 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

NOTE: In this job, automatic sort insertion, and automatic

partition insertion
must be turned off vii.

Looks a lot like solution 1, except w/o the sort on the output of JoinSourceToAggregator.
The difference is on DistributeCopiesOfSourceData:

Here, we have chosen to use the StoreLocation column as a part of our sorting key, but NOT to use it
for hashing. This is functionally equivalent to doing a sub-sort right before the
StoreAverageItemTransactionAmt aggregator. However it will not create additional processes to
handle the records and re-order them. This is a potentially huge savings on large data volumes
(remember the previous example). Also, the need for the second sort on the output of the
JoinSourceToAggregator_1 is not needed, for the same reasons. Here is the output from this version of
the job.

Comparing the efficiency of this solution with that of number two, we saved a sub-sort on 100 million
records - a significant savings.
Data Set 4:
PeekFinalOutput, Partition 1: 16 Rows
Store Item Transaction Transaction National Store
Location ID Date Amt Average Average
Item Item
Transaction Transaction
Amt Amt
1 1 2004/01/01 1 3.5 2
1 1 2004/01/01 3 3.5 2
2 1 2004/01/01 7 3.5 5
2 1 2004/01/01 3 3.5 5
1 1 2004/01/04 5 16.25 6
1 1 2004/01/04 7 16.25 6
2 1 2004/01/04 8 16.25 26.5
2 1 2004/01/04 45 16.25 26.5
1 2 2004/01/01 45 44.25 27
1 2 2004/01/01 9 44.25 27

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 171 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

2 2 2004/01/01 78
44.25
2 2 2004/01/01 45 44.25 61.5
1 2 2004/01/04 2 38.75 33.5
1 2 2004/01/04 65 38.75 33.5
2 2 2004/01/04 23 38.75 44
2 2 2004/01/04 65 38.75 44

PeekFinalOutput, Partition 2: 16 Rows

Store Item Transaction Transaction National Store
Location ID Date Amt Average Average
Item Item
Transaction Transaction
Amt Amt
1 1 2004/01/02 2 35.5 28
1 1 2004/01/02 54 35.5 28
2 1 2004/01/02 54 35.5 43
2 1 2004/01/02 32 35.5 43
1 1 2004/01/03 5 23 6.5
1 1 2004/01/03 8 23 6.5
2 1 2004/01/03 23 23 39.5
2 1 2004/01/03 56 23 39.5
1 2 2004/01/02 7 26.25 46
1 2 2004/01/02 85 26.25 46
2 2 2004/01/02 7 26.25 6.5
2 2 2004/01/02 6 26.25 6.5
1 2 2004/01/03 3 180.5 50.5
1 2 2004/01/03 98 180.5 50.5
2 2 2004/01/03 534 180.5 310.5
2 2 2004/01/03 87 180.5 310.5

Finally, in addition to the heavy penalty paid in disk I/O for using a full sort, sort, by definition,
inhibits pipe-lining (by buffering large amounts of data to disk since it needs to see all data before it
can determine the resulting sorted sequence)ix.

Here is a screen shot of a sort running on 40 million records:

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 172 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
 Information Integration Solutions
Center of Excellence

As you can see, although ~5 million records have entered the sort, no rows have left yet. This is
because a standard sort requires all rows to be present in order to release the first row, requiring a large
amount of scratch diskx. This situation is analogous to all of the sorts in solution 1, the link sort in
solution 2, and the link sort in solution three.

Here is an example of a ‘sub-sort’:

Here, you can clearly see that a sub-sort does not inhibit pipe-lining--very nearly the same number of
rows have entered and left the sort stage (and NO buffering is required to perform the subsort). This
allows down-stream stages to be processing data during the sorting process; instead of waiting until all
40 million records have been sorted (in this instance, we are sub-sorting the data we sorted in the
previous diagram).

Parallel Framework Red Book: Data Flow Job Design July 17, 2006 173 of 179
© 2006 IBM Information Integration Solutions. All rights reserved. No part of this publication may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language in any form by any means without the written permission of IBM.
i
This is an over-simplification; it’s only true for cases where the key is unique. In other cases, Join
needs to see all of the rows in the current cluster on at least one of the input links; otherwise a
Cartesian product is impossible.

ii
A common problem: Suppose you have two (or more), presorted, datasets with differing partition counts
and you wish to join/merge them. At least one of these dataset must be re-hashed, which would result in
having to completely re-sort that dataset despite having a sorted version already. This ‘problem’ is
addressed by the parallelsortmerge component iii.

Another common problem: You need to Hash and Sort on columns A and C to implement the business logic in
one section of the flow, but in another section you need to hash and sort on columns A and B. You could
hash only on A, but suppose that A has too small a number of unique values (country codes, gender
codes, race/gender/ethnicity codes are typical). This would allow you to combine other columns into
your hash key to reduce data-skew, but not introduce superfluous sorts.

A third, less common, problem: you created a fileset with 8 nodes, but the job that reads it only has 4
nodes. Normally EE would re-partition the data into 4 nodes + destroy your sort order. However, you can
use the ParallelSortMerge stage to ensure that no matter the degree of parallelism of the writer +
reader, the sort order will be preserved.

There are other situations where this is valuable but they are much less common.
iii

ParallelSortMerge Operator Options:

-key -- specify a key field; 1 or more
name -- input field name
Sub-options for key:
-ci -- case-insensitive comparison; optional
-cs -- case-sensitive comparison; optional; default
-nulls -- where null values should sort; optional
position -- string; value one of first, last; default=first
-asc or ascending -- ascending sort order; optional; default
-desc or descending -- descending sort order; optional
-param -- extra parameters for key; optional
params -- property=value pair(s), without curly braces
(mutually exclusive: -ci, -cs)
(mutually exclusive: -asc, -desc)
-warnLevel -- queue length at which to issue a warning; optional
records -- integer; 0 or larger; 2147483647 or smaller; default=10000
-doStats -- report statistics at the end of the run; optional
This operator may have following inputs
-Sorted -- dataset to be resorted/merged; exactly one occurrence required
This operator may have following outputs
-reSorted -- resorted dataset; exactly one occurrence required

<add example here of how psm works>

iv
Dump score for solution 1
main_program: This step has 16 datasets:
ds0: {op0[3p] (parallel SourceData.DSLink2)
eAny=>eCollectAny
op1[3p] (parallel DistributeCopiesOfSourceDta)}
ds1: {op1[3p] (parallel DistributeCopiesOfSourceDta)
eOther(APT_HashPartitioner { key={ value=ItemID },
key={ value=TransactionDate }
})#>eCollectAny
op2[3p] (parallel APT_HashedGroup2Operator in NationalAverageItemTransactionAmt)}
ds2: {op1[3p] (parallel DistributeCopiesOfSourceDta)
eOther(APT_HashPartitioner { key={ value=ItemID },
key={ value=TransactionDate },
key={ value=StoreLocation }
})#>eCollectAny
op3[3p] (parallel APT_HashedGroup2Operator in StoreAverageItemTransactionAmt)}
ds3: {op1[3p] (parallel DistributeCopiesOfSourceDta)
eOther(APT_HashPartitioner { key={ value=ItemID },
key={ value=TransactionDate }
})#>eCollectAny
op4[3p] (parallel APT_TSortOperator(0))}
ds4: {op2[3p] (parallel APT_HashedGroup2Operator in NationalAverageItemTransactionAmt)
[pp] eSame=>eCollectAny
op5[3p] (parallel APT_TSortOperator(1))}
ds5: {op3[3p] (parallel APT_HashedGroup2Operator in StoreAverageItemTransactionAmt)
[pp] eSame=>eCollectAny
op8[3p] (parallel APT_TSortOperator(2))}
ds6: {op4[3p] (parallel APT_TSortOperator(0))
 [pp] eSame=>eCollectAny
op6[3p] (parallel buffer(0))}
ds7: {op5[3p] (parallel APT_TSortOperator(1))
[pp] eSame=>eCollectAny
op7[3p] (parallel buffer(1))}
ds8: {op6[3p] (parallel buffer(0))
[pp] eSame=>eCollectAny
op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)}
ds9: {op7[3p] (parallel buffer(1))
[pp] eSame=>eCollectAny
op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)}
ds10: {op8[3p] (parallel APT_TSortOperator(2))
[pp] eSame=>eCollectAny
op12[3p] (parallel buffer(3))}
ds11: {op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)
[pp] eOther(APT_HashPartitioner { key={ value=ItemID,
subArgs={ cs }
},
key={ value=TransactionDate },
key={ value=StoreLocation,
subArgs={ cs }
}
})#>eCollectAny
op10[3p] (parallel JoinSourceToAggregator_2.DSLink18_Sort)}
ds12: {op10[3p] (parallel JoinSourceToAggregator_2.DSLink18_Sort)
[pp] eSame=>eCollectAny
op11[3p] (parallel buffer(2))}
ds13: {op11[3p] (parallel buffer(2))
[pp] eSame=>eCollectAny
op13[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)}
ds14: {op12[3p] (parallel buffer(3))
[pp] eSame=>eCollectAny
op13[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)}
ds15: {op13[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)
[pp] eSame=>eCollectAny
op14[3p] (parallel PeekFinalOutput)}
It has 15 operators:
op0[3p] {(parallel SourceData.DSLink2)
on nodes (
node1[op0,p0]
node1[op0,p1]
node1[op0,p2]
)}
op1[3p] {(parallel DistributeCopiesOfSourceDta)
on nodes (
node1[op1,p0]
node1[op1,p1]
node1[op1,p2]
)}
op2[3p] {(parallel APT_HashedGroup2Operator in NationalAverageItemTransactionAmt)
on nodes (
node1[op2,p0]
node2[op2,p1]
node3[op2,p2]
)}
op3[3p] {(parallel APT_HashedGroup2Operator in StoreAverageItemTransactionAmt)
on nodes (
node1[op3,p0]
node2[op3,p1]
node3[op3,p2]
)}
op4[3p] {(parallel APT_TSortOperator(0))
on nodes (
node1[op4,p0]
node2[op4,p1]
node3[op4,p2]
)}
op5[3p] {(parallel APT_TSortOperator(1))
on nodes (
node1[op5,p0]
node2[op5,p1]
node3[op5,p2]
)}
op6[3p] {(parallel buffer(0))
 on nodes (
node1[op6,p0]
node2[op6,p1]
node3[op6,p2]
)}
op7[3p] {(parallel buffer(1))
on nodes (
node1[op7,p0]
node2[op7,p1]
node3[op7,p2]
)}
op8[3p] {(parallel APT_TSortOperator(2))
on nodes (
node1[op8,p0]
node2[op8,p1]
node3[op8,p2]
)}
op9[3p] {(parallel APT_JoinSubOperator in JoinSourceToAggregator_1)
on nodes (
node1[op9,p0]
node2[op9,p1]
node3[op9,p2]
)}
op10[3p] {(parallel JoinSourceToAggregator_2.DSLink18_Sort)
on nodes (
node1[op10,p0]
node2[op10,p1]
node3[op10,p2]
)}
op11[3p] {(parallel buffer(2))
on nodes (
node1[op11,p0]
node2[op11,p1]
node3[op11,p2]
)}
op12[3p] {(parallel buffer(3))
on nodes (
node1[op12,p0]
node2[op12,p1]
node3[op12,p2]
)}
op13[3p] {(parallel APT_JoinSubOperator in JoinSourceToAggregator_2)
on nodes (
node1[op13,p0]
node2[op13,p1]
node3[op13,p2]
)}
op14[3p] {(parallel PeekFinalOutput)
on nodes (
node1[op14,p0]
node2[op14,p1]
node3[op14,p2]
)}
It runs 45 processes on 3 nodes.

v
Throughout this document the general meaning of the phrase ‘more efficient’ is fewer record
movements--i.e. a record changes partition, or order, fewer times. Since moving records around takes
CPU time and extra system calls, if you move records unnecessarily, your run time will be adversely
affected.

vi
Dump Score for Solution 2
main_program: This step has 15 datasets:
ds0: {op0[3p] (parallel SourceData.DSLink2)
eOther(APT_HashPartitioner { key={ value=ItemID,
subArgs={ cs }
},
key={ value=TransactionDate }
})#>eCollectAny
op1[3p] (parallel DistributeCopiesOfSourceDta.DSLink2_Sort)}
ds1: {op1[3p] (parallel DistributeCopiesOfSourceDta.DSLink2_Sort)
[pp] eSame=>eCollectAny
op2[3p] (parallel DistributeCopiesOfSourceDta)}
ds2: {op2[3p] (parallel DistributeCopiesOfSourceDta)
 [pp] eSame=>eCollectAny
op3[3p] (parallel SubSortOnStoreLocation)}
ds3: {op2[3p] (parallel DistributeCopiesOfSourceDta)
[pp] eSame=>eCollectAny
op4[3p] (parallel APT_SortedGroup2Operator in NationalAverageItemTransactionAmt)}
ds4: {op2[3p] (parallel DistributeCopiesOfSourceDta)
[pp] eSame=>eCollectAny
op6[3p] (parallel buffer(0))}
ds5: {op3[3p] (parallel SubSortOnStoreLocation)
[pp] eSame=>eCollectAny
op5[3p] (parallel APT_SortedGroup2Operator in StoreAverageItemTransactionAmt)}
ds6: {op4[3p] (parallel APT_SortedGroup2Operator in NationalAverageItemTransactionAmt)
[pp] eSame=>eCollectAny
op7[3p] (parallel buffer(1))}
ds7: {op5[3p] (parallel APT_SortedGroup2Operator in StoreAverageItemTransactionAmt)
[pp] eSame=>eCollectAny
op8[3p] (parallel buffer(2))}
ds8: {op6[3p] (parallel buffer(0))
[pp] eSame=>eCollectAny
op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)}
ds9: {op7[3p] (parallel buffer(1))
[pp] eSame=>eCollectAny
op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)}
ds10: {op8[3p] (parallel buffer(2))
[pp] eSame=>eCollectAny
op12[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)}
ds11: {op9[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_1)
[pp] eSame=>eCollectAny
op10[3p] (parallel SubSortOnStoreLocation2)}
ds12: {op10[3p] (parallel SubSortOnStoreLocation2)
[pp] eSame=>eCollectAny
op11[3p] (parallel buffer(3))}
ds13: {op11[3p] (parallel buffer(3))
[pp] eSame=>eCollectAny
op12[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)}
ds14: {op12[3p] (parallel APT_JoinSubOperator in JoinSourceToAggregator_2)
[pp] eSame=>eCollectAny
op13[3p] (parallel PeekFinalOutput)}
It has 14 operators:
op0[3p] {(parallel SourceData.DSLink2)
on nodes (
node1[op0,p0]
node1[op0,p1]
node1[op0,p2]
)}
op1[3p] {(parallel DistributeCopiesOfSourceDta.DSLink2_Sort)
on nodes (
node1[op1,p0]
node2[op1,p1]
node3[op1,p2]
)}
op2[3p] {(parallel DistributeCopiesOfSourceDta)
on nodes (
node1[op2,p0]
node2[op2,p1]
node3[op2,p2]
)}
op3[3p] {(parallel SubSortOnStoreLocation)
on nodes (
node1[op3,p0]
node2[op3,p1]
node3[op3,p2]
)}
op4[3p] {(parallel APT_SortedGroup2Operator in NationalAverageItemTransactionAmt)
on nodes (
node1[op4,p0]
node2[op4,p1]
node3[op4,p2]
)}
op5[3p] {(parallel APT_SortedGroup2Operator in StoreAverageItemTransactionAmt)
on nodes (
node1[op5,p0]
node2[op5,p1]
node3[op5,p2]
 )}
op6[3p] {(parallel buffer(0))
on nodes (
node1[op6,p0]
node2[op6,p1]
node3[op6,p2]
)}
op7[3p] {(parallel buffer(1))
on nodes (
node1[op7,p0]
node2[op7,p1]
node3[op7,p2]
)}
op8[3p] {(parallel buffer(2))
on nodes (
node1[op8,p0]
node2[op8,p1]
node3[op8,p2]
)}
op9[3p] {(parallel APT_JoinSubOperator in JoinSourceToAggregator_1)
on nodes (
node1[op9,p0]
node2[op9,p1]
node3[op9,p2]
)}
op10[3p] {(parallel SubSortOnStoreLocation2)
on nodes (
node1[op10,p0]
node2[op10,p1]
node3[op10,p2]
)}
op11[3p] {(parallel buffer(3))
on nodes (
node1[op11,p0]
node2[op11,p1]
node3[op11,p2]
)}
op12[3p] {(parallel APT_JoinSubOperator in JoinSourceToAggregator_2)
on nodes (
node1[op12,p0]
node2[op12,p1]
node3[op12,p2]
)}
op13[3p] {(parallel PeekFinalOutput)
on nodes (
node1[op13,p0]
node2[op13,p1]
node3[op13,p2]
)}
It runs 42 processes on 3 nodes.

vii
In this instance, you want auto insertion turned off b/c EE will see that you are ‘missing’ a
sort/partitioner and insert one for you, thus introducing the inefficiencies we are trying to avoid.

viii
To understand why this is true, look at this example:

Here is my source data:

ColumnAColumnBColumnC111112113121122123211212213221222223231232233
Here are the possible outcomes if I
hash-partition on ColumnA, and ColumnB:
ColumnAColumnBColumnCGroup 1111112113Group 2121122123Group 3211212213Group 4221222223Group 5231232233
There must be exactly 5 groups identified by the hash algorithm b/c there are exactly 5 unique
combinations of ColumnA and ColumnB. This does not mean that these groups will be in unique partitions:
consider a job that only has 3 partitions.

Any combination of these groups can be in any partition, regardless of the number of partitions: if you
are running a job with 6 partitions, you could have ALL 5 groups sent to the same partition (this is
unlikely, and the likelihood decreases with larger numbers of groups, in fact, the distribution of
groups across partitions is nearly even for large numbers of groups).

Here are the possible outcomes if I

hash-partition on ColumnA only:
ColumnAColumnBColumnCGroup 1111112113121122123Group 2211212213221222223231232233
As you can see, there are only two groups by hashing on ColumnA only. So hashing on fewer columns
resulted in fewer, larger groups.

One effect is that if we wanted to aggregate on ColumnA and ColumnB, summing ColumnC, this grouping is
OK, b/c: if all unique values of ColumnA are together, then all unique combinations of ColumnA and
ColumnB are together, as well as all unique combinations of ColumnA, ColumnB, and ColumnC.

In the scenario that we are discussing in the main document, we want to reduce the number of times that
we hash (b/c partitioning costs CPU time). We can do this by identifying the intersection of keys
needed among all of the hash-partitioners and hashing only on those keys:
TransactionDate and ItemId

NOTE: if you take this to an extreme, you will get a very small number of groups, which will,
effectively, reduce the parallelism of the job. In the above example, we would have only two groups.
Therefore, even if we ran the job 12-ways, we wouldn’t see any improvements in performance over a 2-way
job.

You need to understand your data and make educated decisions about your hashing strategy.

ix
This means that down-stream process will be sitting idle until the sort is completed, consuming RAM
and process space and offering nothing in return.

x
This is a slight oversimplification. It is only true on a per-partition basis, not for the entire
dataset.