Webservices POC on Message Broker

File Extender Node

Using
Message Broker 5.0

Author: Surendrakumar Jebakani

Reviewer: Srikanthan Ramanathan

Page 1 of 34

Webservices POC on Message Broker

DOCUMENT REVISION CONTROL

DATE
June 18, 2008

AUTHOR
Surendrakumar Jebakani

VERSION
1.0

Page 2 of 34

CHANGE DESCRIPTION
Creation

Webservices POC on Message Broker

TABLE OF CONTENTS

INTRODUCTION..........................................................................................................................................4
OBJECTIVE...................................................................................................................................................4
OVERVIEW OF FILE EXTENDER NODES.............................................................................................4
INSTALLATION............................................................................................................................................7
CONFIGURATION......................................................................................................................................14
USING FILE EXTENDER NODES IN MESSAGE FLOWS..................................................................24
SAMPLE CODES.........................................................................................................................................33
THINGS NEED TO BE FOLLOWED WHILE USING MBFE IN MICHELIN..................................33
REFERENCE...............................................................................................................................................34

Page 3 of 34

Webservices POC on Message Broker

Introduction
This document describes how to install, configure and use WebSphere Message Broker File
Extender (WMBFE). It defines the characteristics and functionality of WebSphere Message
Broker File Extender, a series of components that can be used to enable file handling within
WebSphere Business Integration Brokers. How to develop message flows using MBFE in all the
three scenarios (F2F, F2M and M2F) and sample codes are provided in this document.

Objective
At the end of the lab, you should be able to:
Understand the role of various components that is required to develop POC flow using File
Extender nodes.
Understand the File Extender nodes implementation steps in Message broker.

Overview of File Extender Nodes

IBM WebSphere Business Integration brokers are leading message brokers in the integration
market, offering in-flight routing, transformation, logging, and enrichment of messages.
IBM broadens its leadership with WebSphere Business Integration brokers by extending the
applicability of the broker's unique set of value-added features and functions with the release of
WebSphere Message Broker File Extender, a set of nodes specifically designed to handle files
residing on the same file system as the Message Broker run time.
WebSphere Message Broker File Extender increases the value of previous investments by
supporting and transforming information between legacy file-based applications and new
applications based on different protocols, standards, and formats.

Page 4 of 34

Webservices POC on Message Broker

Why WebSphere Message Broker File Extender?

IBM WebSphere Message Broker File Extender V5.0 handles files residing on the same system
as the Message Broker run time
It is a set of plug-in nodes fully integrated with the Message Broker Toolkit palette, giving users
of WebSphere Message Broker immediate productivity.
Extends the capabilities of WebSphere Message Broker into the domain of files, enabling users to
natively exploit huge amounts of information from the broker that many enterprises store in flat
files
Allows businesses to see the same level of reuse and value from the data held in files as they
currently get from the data being sent round their business in messages
Enables users to apply brokering services such as transformation, enrichment, logging, and
routing to file content
Seamlessly convert messages into files, file records, and vice versa, by simply wiring and
configuring the relevant input/output nodes into the appropriate message flow
Increases the value of previous investments by supporting and transforming information between
legacy file-based applications and new applications based on different protocols, standards, and
formats.
The Program consists of a set of broker plug-in nodes (namely a file input node and a file output
node) that are fully integrated with the Message Broker Toolkit palette, giving new and existing
users of WebSphere Business Integration Message Broker immediate productivity with this new
component.
The Program is a very important addition to the broker portfolio of add-on plug-ins nodes as it
enables users to natively exploit from the broker the huge amount of information that in the vast
majority of enterprises is still stored in flat files.
Users can apply brokering services such as transformation, enrichment, logging, and routing to
file content, and seamlessly convert messages into files, file records, and vice versa, by simply
wiring the relevant input/output node into the appropriate message flow.
The Program administration model adheres to the overall WebSphere Business Integration
Message Broker administration paradigm. Configuring the Program is as simple as dragging a
node into a message flow and editing a few user specified node properties using the standard
broker toolkit. In order to support change management procedures in an effective way, some of
the node properties can be overridden at deploy time.

Functions and Capabilities.

The following list summarizes the functions and capabilities of the WebSphere Message Broker
File Extender:

Page 5 of 34

Webservices POC on Message Broker

Ability to read from and write to a file directly using input and output nodes in a message flow
The WebSphere Message Broker File Extender File Input and File Output nodes allow you to
write a message flow that takes input from files stored in a local file system source directory
(possibly using wildcards to select specific file subsets), and to produce output in the form of files
written to a local file system target directory. When reading from a file, you have the option of
propagating the file content to the message flow either record by record or as a single buffer of
information; in both cases the information read from files can be parsed exploiting the native
IBM WebSphere Business Integration Message Broker MRM definitions.
Ability to operate on file information with ESQL and built-in IBM WebSphere Business
Integration Message Broker nodes
The information read from a file or written to a file can be handled from within a message flow
using the standard facilities offered by IBM WebSphere Business Integration Message Broker,
such as Compute nodes with ESQL language, Mapping nodes, Filter nodes, and MRM parsing.
Support for file-to-file message flows
When a WebSphere Message Broker File Extender File Input node is used in conjunction with a
WebSphere Message Broker File Extender File Output node, the resulting message flow can be
used to perform pure file brokering functions such as: file format conversion, file content
augmentation, and filtering.
Support for File to Message and Message to File message flows
When a WebSphere Message Broker File Extender File Input node is used in conjunction with an
MQ Output node, or vice versa, when an MQ Input node is used in conjunction with a WebSphere
Message Broker File Extender File Output node, the resulting message flow can be used to
implement two-way interfaces between message-based applications and file-based applications,
possibly including value-added function such as on the fly format conversion (for example from a
COBOL copybook-like format to XML) and routing.

Page 6 of 34

Webservices POC on Message Broker

Installation
Preparing for installation
WebSphere Message Broker File Extender requires one of these broker products:
WebSphere Business Integration Message Broker 5.0 with Fix Pack 05.
WebSphere Message Broker 6.0 with fix for APAR IY77292.
Please follow the below steps to install MBFE in your machine.
Get the installable MBFE software from \\10.145.174.26\ machine (folder name is
FileExtenderNode) and copy it to your local machine.
Make sure you are logged in as an administrator
Run setupwin32.exe which is in the C8433IE_Windows folder.
Do accept the terms in the license agreement then click NEXT.

Set the Directory path where the Repository for MBFE Objects needs to be created
and then NEXT.

Page 7 of 34

Webservices POC on Message Broker

Set the variable data directory name and then NEXT.

Page 8 of 34

Webservices POC on Message Broker

Select custom as the installation type and then NEXT.

Page 9 of 34

Webservices POC on Message Broker

Do select all the features and then NEXT.

Page 10 of 34

Webservices POC on Message Broker

Once the installation of MBFE is done, click Finish.

Get the FIXPACK version from the same FileExtenderNode folder and do follow the
same steps to install setupwin32.exe (FixPack) in your machine.
Once the FIXPACK installation is done check the directory which you given for
MBFE object to be created (In our case it is D:\IBM\MBFE)
You could see, inside the MBFE folder, all the necessary subfolders have been
created.
Get into the bin folder; here you will have to run mbfecfgqmgr.bat batch file to
create MB objects and repository for MBFE.
Before running the batch file, just point your cursor to the bin folder which must be
there in D:\IBM\MBFE\bin.

Enter the below command:

<Full name of the batch file> < Your Queue Manager >
For example: D:\IBM\MBFE\bin\mbfecfgqmgr.bat <Your Queue Manager>

Page 11 of 34

Webservices POC on Message Broker

You will get error messages if MQ and MBFE related jars are not set properly in
your classpath. (So one of the things you need to check when getting no class
definition found is to check if the classpath is set properly)

Make sure there is no syntax error in creating all the MQ objects for MBFE.

Ensure that all the required folders are created under IBM\MBFE.

Page 12 of 34

Webservices POC on Message Broker

We have done the installation now.

Page 13 of 34

Webservices POC on Message Broker

Configuration
This section describes information on WebSphere Message Broker File Extender node
terminals, attributes and on the message tree structure created/managed by the nodes at
runtime.
File Input Node
File Input Node has no input terminals and four output terminals (Failure, Out, Catch
andSynch).
Properties of a FileInput Node
Basic Properties:
File Pattern Filter: This is mandatory property to specify your input file path name within
the specified directory where input files are read. It is configurable as well.

Page 14 of 34

Webservices POC on Message Broker

To read all the files in the specified directory give * in the end .

It would process all the files contained in the directory named In.
Similarly D:\FIP_Input\In\*.trg will process all the files with extension trg contained in the
directory named In.

If you specify File Pattern Filter as D:\FIP_Input\In \FIP_TestInput.trg; it will process only
one file named FIP_TestInput.trg.
Depends on your project requirement you could specify the File Pattern Filter as it is
configurable. By selecting the promote property you could specify the file path in the bar file
itself, we will see this feature later.
Input Propagation Policy: This is also mandatory property to specify how the FileInput
node will propagate the input file content to the message flow.
Valid values are:
File Descriptor and Content Whole File: This is the Default Value; in this case, the FileInput
node will propagate the file descriptor and the whole file content. It is not recommend using this
option for very big files.

Page 15 of 34

Webservices POC on Message Broker

File Descriptor and Content Record by Record: In this case, on each iteration, the
FileInput node will propagate the file descriptor and the content of a record read
sequentially from the input file.
File Descriptor only: In this case, the FileInput node will propagate just the file
descriptor. The file content is not propagated. This option can be used to implement rule
based file routing using WebSphere Message Broker.

Default Properties

The Default tab also provides CCSID and Encoding properties (since the input source is
not a WebSphere MQ message, there is no MQMD header, so these settings are required
to let you control the parsing of files that do not conform to the machine default CCSID
and Encoding values).

Page 16 of 34

Webservices POC on Message Broker

Format Properties

The Format properties tab contains further properties that inform the node about how to
interpret the structure of the data in the file. These parsing options are provided as a
native function of the node and are independent of similar techniques you may already be
familiar with when using the Message Repository Manager domain.
The Format properties let you receive files which contain multiple record structures
delimited by carriage return line feeds.

Page 17 of 34

Webservices POC on Message Broker

Transaction Mode specifies how the input file is to be handled by FileInput node.
Record Batch Size: Required only if Transaction Mode= Yes, Record BatchScope, it
specifies the number of records to be included in each batch.
Retry Count: Number of retries to be attempted if the FileInput Node detects a problem
during the message flow execution.
Retry Intervals: Specifies the interval in seconds between successive retries.
In Doubt Policy: Specifies how transaction related problems are to be handled.
File Output Node
The File Output node receives the input as a logical tree description (the standard
serialized format used between wired nodes of a message flow) of a file record or
WebSphere MQ message (or from any other WebSphere Message Broker supported input
source). Typically the File Output node writes an output file to a directory local to the
run-time broker. The nodes properties (viewed in the Toolkit by right-clicking the node
and selecting Properties) are split into groups and displayed using the following tabs:

Page 18 of 34

Webservices POC on Message Broker

Basic
Format
Advanced
Troubleshooting
Description

The node has no output terminal, and successful writing of a file results in the
termination of the current message flow thread. You cannot place other message flow
nodes downstream of the file output node in any given branch of a message flow
(although execution of secondary branches of a message flow after a FlowOrder node are
still respected).
The File Output nodes Basic properties tab specifies an output directory location as
either an absolute or relative path.
The Format properties tab contains further properties that inform the node about how to
write structured data within a single file. These writing options are provided as native
function of the node and are independent of similar techniques that you may be familiar
with using the Message Repository Manager domain.
The Format properties let you write files that contain multiple record structures delimited
by carriage return line feeds. The batch processing possibilities and the relevant node
properties are discussed below.

Page 19 of 34

Webservices POC on Message Broker

The Advanced properties tab controls the naming of the output file and options for how
it is generated. You can generate a new file for every message received by the node,
overwrite existing files, and append data to existing files. You can generate file names
with counters at the beginning, end, or just before the file extension.

File Proxy Node

The File Proxy node is always located in a message flow immediately downstream of an
MQInput node. Its purpose is to maintain a transactional context between the reading of
WebSphere MQ messages that initiate the message flow and the writing of files that
terminate the instance of the flow.
The File Proxy node is only used in flows that take MQ messages as input and write
Files as output.
The nodes Advanced properties tab includes a Transaction Mode setting that
determines whether or not the action of the File Output node later in the flow is to be
conducted within the same unit of work as the original MQGet that was performed by the
MQ Input node. If set to yes, then the File Output node downstream in the message flow
will append data to the output file within the same unit of work (under syncpoint), as was
started when the MQ Input node took its messages from the input queue. If set to no, then
the File Output node downstream in the message flow will append data to the output file
regardless of the current WebSphere MQ unit of work.

Page 20 of 34

Webservices POC on Message Broker

Message Tree Structures

File Descriptor
The file descriptor (MBFEProperties) is a set of attributes that are created by a
FileInput when reading the file in order to identify characteristics.
A reduced version of this descriptor can be created/modified using ESQL in a Compute
node before FileOutput in order to drive its behavior (for example, setting the file name
to a context dependent value, like a timestamp or the result of a database lookup).

Usage example:
The below example can be used in a M2F or F2F flow to programmatically set the name
of the output file from within a compute node:
SET OutputRoot = InputRoot;
SET OutputLocalEnvironment = InputLocalEnvironment;
SET OutputLocalEnvironment.Variables.MBFEProperties.FileName = 'SampleFile.dat';

File Destination List Descriptor

Page 21 of 34

Webservices POC on Message Broker

The File Output node can be configured to write data from the logical tree it receives to
more than one output file. This is done dynamically using a Destination List that a
message flow has configured using its ESQL (or Java or plug-in node). The Destination
List concept works like the MQ Destination List, which lets an MQ Output node, write
output data to multiple destination messages.
The File Output nodes Advanced properties sheet contains the attribute Destination
Mode. Changing this to Output Directory List and creating entries in the
LocalEnvironment will make the node write the same file to all of the directory locations.
The ESQL below will create a directory list with three separate file locations:
Usage Example:
SET OutputRoot = InputRoot;
SET OutputLocalEnvironment = InputLocalEnvironment;
SET OutputLocalEnvironment.Destination.MBFE.DestinationData[1].Directory
= 'file:D:\Surendra\OUT1 ';
SET OutputLocalEnvironment.Destination.MBFE.DestinationData[2].Directory
= 'file:D:\Surendra\OUT2';
SET OutputLocalEnvironment.Destination.MBFE.DestinationData[2].Directory
= 'file:D:\Surendra\OUT3';
SET OutputLocalEnvironment.Variables.MBFEProperties.FileName= 'SampleFile.dat';

Page 22 of 34

Webservices POC on Message Broker

File action:
The File Descriptor contains an attribute that controls when an output file that is being
appended to by a File Output node inside a message-to-file (MtoF) message flow closes
the file and releases it to the file system. A common situation when this function might be
used is when using the PROPAGATE statement to deal with large incoming messages. A
single input message can be split into multiple sections using ESQL statements in a
Compute node.
The propagate statement finalizes the compute nodes output message trees and
propagates to the downstream nodes with the message flow. It then clears the output
message tree and reclaims the memory for further use. This technique can help you avoid
parsing a large message all at once, which could have a large memory requirement.
A common requirement is to separate each part of the propagated message into a different
file, and you can do so using the File Action tag, which tells the FileOutput node to close
the output file on receiving the message tree that had been finalized as a result of the
Propagate. The ESQL below sets the File Action attribute of the File Descriptor:
SET OutputLocalEnvironment.Variables.MBFEProperties.action = 'close';

Another scenario in which this function is useful is a message-to-file message flow where
multiple input messages must be transformed and aggregated into a single output file
until a particular group context or content decision is met. For example, a zero-length

Page 23 of 34

Webservices POC on Message Broker

message on input might represent "end of business day" and require the file to be closed.
Alternatively, an input message arriving after "end of business day" might require the file
to be closed and further messages queued ready for processing the following day.

Using File Extender Nodes in message flows

In the next few sections we will go through three very simple WebSphere Message
Broker File Extender based message flows in detail.
There are three scenarios:
File to File (F2F)
File to Message (F2M)
Message to File (M2F)

4.1.

Sample 1 File to File message flow

Below is the picture of very simple File to File message flow:

Setting the input directory path:

There are two ways you could set the input directory in the file input node:
By right clicking the file input node and setting the input directory in the File Pattern
Filter property.
Every time the input directory gets changed you will have to change the input directory
on the file input node and you will have to re-deploy.

Page 24 of 34

Webservices POC on Message Broker

By right clicking on the file input node and selecting the promote Property you could set
the input directory in the Bar file itself.
Step 1: Right click on the file input node and select Promote Property.

Page 25 of 34

Webservices POC on Message Broker

Step 2: Drag and drop the File Pattern Filter (Under Basic property in left side) in to the
right side under the File_To_File.

Page 26 of 34

Webservices POC on Message Broker

`
Step 3: Select OK and save.

Page 27 of 34

Webservices POC on Message Broker

Step 4: Right click on the blank area then select Properties.

Page 28 of 34

Webservices POC on Message Broker

Step 5: Here, we should not leave File Pattern Filter as blank; it will show error in the
application development perspective. Enter any value in the File Pattern Filter as the
exact directory path will be set in the Bar file.

Step 6: Select OK and save.

Step 7: Add the F2F message flow to the broker archive.

Page 29 of 34

Webservices POC on Message Broker

Step 8: In the Configure mode, give the correct directory path.

This way, you do not need to set the input directory name every time in the message
flow, just set the directory name in the Bar file and re-deploy.
Sample ESQL
CREATE COMPUTE MODULE File_To_File_Node_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET OutputRoot
= InputRoot;
SET OutputRoot.Properties.MessageType = 'XML';
SET OutputLocalEnvironment = InputLocalEnvironment;
SET OutputLocalEnvironment.Destination.MBFE.DestinationData.Directory
= 'file:D:\Surendra\In';
SET OutputLocalEnvironment.Variables.MBFEProperties.FileName =
'Surendra_Output.trg';
RETURN TRUE;

END;
ENDMODULE;

Page 30 of 34

Webservices POC on Message Broker

Sample 2 File to Message message flow

Below is the very simple picture of F2M message flow.

An interesting feature demonstrated by the Compute node ESQL code is the dynamic
creation of the WebSphere MQ message descriptor (MQMD) that will not be present
in the message tree of nonWebSphere MQ sources of data such as a file.
Sample ESQL code:
CREATE COMPUTE MODULE File_To_MQ_Compute1
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
--CALL CopyEntireMessage();
SET OutputRoot.Properties = InputRoot.Properties;
SET OutputRoot.Properties.Encoding
= MQENC_INTEGER_NORMAL;
SET OutputRoot.Properties.CodedCharSetId= 1208;
SET OutputRoot.Properties.ReplyIdentifier = MQCI_NONE;
SET OutputLocalEnvironment = InputLocalEnvironment;
SET OutputRoot.MQMD.Format = 'MQSTR';
SET OutputRoot.MQMD.Version = MQMD_CURRENT_VERSION;
SET OutputRoot.MQMD.StrucID= MQMD_STRUC_ID;
SET OutputRoot.MQMD.Encoding= MQENC_INTEGER_NORMAL;
SET OutputRoot.MQMD.CodedCharSetId = 1208;
SET OutputRoot.MQMD.PutDate = CURRENT_DATE;

Page 31 of 34

Webservices POC on Message Broker

SET OutputRoot.MQMD.PutTime = CURRENT_TIME;
SET OutputRoot.XML = InputRoot.XML;
RETURN TRUE;
END;
END MODULE;

Sample 3 Message to File

Below is the very simple picture of F2M message flow.

Here we are using the new node called File proxy node. If your input is MQ messages
then you always need the FileProxy node to specify the number of file records before
closing the file, or the number of seconds without activity before closing the file.
A FileProxy node must always be used in M2F scenarios, even if you do not have any
transactionality requirements.
Sample ESQL Code is:
CREATE COMPUTE MODULE MBFE_MQ_TO_FILE
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CALL CopyEntireMessage();
SET OutputLocalEnvironment.Destination.MBFE.DestinationData.Directory =
'file:D:\Surendra\OUT1';

Page 32 of 34

Webservices POC on Message Broker

SET OutputLocalEnvironment.Variables.MBFEProperties.FileName
= 'SampleFile.dat';
RETURN TRUE;
END;
CREATE PROCEDURE CopyEntireMessage() BEGIN
SET OutputRoot = InputRoot;
END;
END MODULE;

Sample Codes
1) F2F POC
File_TO_File_POC.zip

2) F2M POC
File_TO_Message_P
OC.zip

3) M2F POC
MQ_TO_File_POC.zip

Things need to be followed while using MBFE in Michelin

The common exception (ParseExcpListToText) routine that is being used in Michelin MB flows
would not handle the MBFE exception if the input is coming from a file. To handle the MBFE
exceptions, developers should use the Common procedure MBFEParseExcpList that is in the
clearcase
path
common_wbimb_v1.1.0_intg1\eai\common_wbimb\3_programs\
WBI_Common_FileExtender. It will be applicable for F2F scenario and F2M scenario.
If the input is generated by MQ then the existing common routine could be used to handle the
MBFE exceptions. (MQ to File scenario)

Page 33 of 34

Webservices POC on Message Broker

Reference
1. Message Broker and File Extender Nodes PDF ( From IBM Information center of Message Broker)
2. ftp://ftp.software.ibm.com/software/mqseries/fixes/wbimbfev50/fixpack03/C3466662.pdf
3. ftp://ftp.software.ibm.com/software/mqseries/fixes/wbimbfev50/fixpack04/5.0.0.4-ws-mbfe-multifp04.txt

Page 34 of 34