Event Processing and Integration with IAS Data Processors

InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd.
Practical Guide to
Data Processors

Data Processors
Data processors can be configured to act as:
Data transformation nodes, using PASCAL-like script language,
Gateways and bridges (e.g. HTTP/JSON and Queues/XML),
SQL Database interfaces using the data mapping script extension.
You can configure and use the Data Processors as single threaded programs
but you can define many logic implementations and run them in parallel as
threads.
You can choose the transaction support from the three available modes:
auto-commit, single phase (independent) commits, distributed two phase
commit with XA when the supported coordination software is used.
And last but not least, one can find the Data Processors as a very helpful
command line admin's tool.

Concept
The Data Processor is designed to process the incoming messages calling
desired action accordingly.
1. Begin remote transactions (optional),
2. Receive message,
3. Parse content (if configured) and call prerequisit actions (if defined).
4. Invoke the configured logic.
5. Commit or rollback transaction (in case of an exception).
The program can interact with databases and send messages according to
the requirements.

First Usage
export IAS_LANG_XSD=${your_project}/data/customer.xsd
File operations:
ias_qs_processor i file:customer.xml o file:stdout l fwd
ias_qs_processor i file:cust.xml o file:stdout_fmt l fwd
ias_qs_processor i file:c1.xml o file:somedir/c2.xml l fwd
l fwd vs l fwd! Turns the parsing off.
ias_qs_processor i file:customer.xml
o file:stdout_fmt?format=JSON l fwd

Directories
ias_qs_processor i dir:data/xml o file:output l fwd
The above command will display subsequent messages on the terminal
output. In order to direct the messages into series of files use the dir
protocol in output specification.
ias_qs_processor i dir:data/xml
o 'dir:output/${ID}.${COUNT}.out' l fwd
The parameter is a directory name (output in this case) and the file name
pattern.

Timer
The next command will produce two messages with a one second
(1000ms) delay.
ias_qs_processor N 2 i timer:clock?timeout=1000
o file:stdout l fwd

Queue Operations
ias_qs_processor i shm://sample/Q.TEST
o file:stdout l fwd
By default, the messages (and files) are read in the shared browser mode.
ias_qs_processor i shm://sample/Q.TEST?mode=input
o file:stdout l fwd
Note that if the transport layer does not support browsing (e.g.
HTTP/FCGI) omitting the modifier yields in an error.
By default, if there are no messages available, the reading ends
immediately. You can specify the timeout parameter (a number of
milliseconds or -1 for an infinite wait).
ias_qs_processor
i shm://sample/Q.TEST?mode=input?timeout=10000
o file:stdout l fwd

Network Operations 1
When the receiver is going to be a server, one can start it with the
command:
ias_qs_processor i nets://localhost:10000/?mode=input
o file:stdout_fmt l fwd
and invoke a client with the matching address specified on its output:
ias_qs_processor i file:data/xml/customer.xml o
netc://127.0.0.1:10000/ l fwd

Network Operations 2
The data flow in the other direction is also possible.
Server:
ias_qs_processor o nets://localhost:10000/?mode=input
i file:data/xml/customer.xml l fwd
Client:
ias_qs_processor i netc://127.0.0.1:10000/?mode=input o
file:stdout_fmt l fwd

Modes
Typical operation patterns are:
● Producer,
● Consumer,
● Requester,
● Responder,
● Publisher,
● Subscriber,
● Browser,
● Shared Worker Browser (default!)

Publish-Subscribe
The application calls the send() and receive() procedures and is not aware
of such a configuration.
Not all transports support this pattern.

Implementing a Service
In this case API allows to perform send() and receive() calls on both i/o
descriptors.
One can specify the same or different names for the requester input and
output.

Browser
autoRecycle specifies if the processing should
resume after end of data condition occurs.
● never
Browser is not reset when no message is
available.
● next
Browser will be reset where no message is
available but the exception is thrown this time.
The next call will grab a new message.
● immediate
Browser will be reset where no message is
available and the program attempts to read a
message immediately.

Shared Browser
This time, the dedicated thread reads messages on behalf of all logic threads.
This pattern allows to take full advantage of parallel processing.

Built-in Actions
Typical operation patterns are:
● Forward.
● Execute.
● Input data type match execute.
● Attribute based selection execute.
● Job Daemon.
● Synchronous proxy.
● Asynchronous proxy.
● Message preview.
● Message extraction.
● Queue System statistics producer.
● Writing custom logic.
● Java Script execute.
● Simplistic Process Server.

Performance Notes
Options:
T <no. of threads>
C <commit count>
N <no. of messages to process>
O <no. of messages to skip>

Parsing
export IAS_LANG_XSD=${your_project}/data/customer.xsd
Note the exclamation mark right after the logic name:
l fwd vs l fwd! Turns the parsing off.

Data Format Change
ias_qs_processor i mqm://MQTEST/Q1?mode=input
o mqm://MQTEST/Q2?format=JSON l fwd

Working with
Meta Attributes
ias_qs_processor i mqm://MQTEST/Q1?mode=input
o 'mqm://MQTEST/Q2?*Folder.ATTR1=VALUE' l fwd
The '*' attributes provided in URL are considered as message meta attributes and will
be seen in RFH2 or message properties according to the application or IBM
WebSphere MQ version.

Example Queue Setup
DEF QL(GATEWAY.IN)
DEF QL(SERVICE.IN)
DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)

Echo Service
If you do not have a service for this example just start
the following command to setup the echo program.
ias_qs_processor
i 'mqm://MQTEST/SERVICE.IN?mode=responder&timeout=1'
o dummy:
l fwd!

Gateway
Synchronous Proxy
ias_qs_processor
  i 'srvhttp://localhost:50000/mode=input&responderName=input'
  o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000'
  l proxy! m server
Note the srvhttp service and the proxy logic.

Gateway
Asynchronous Proxy
ias_qs_processor
  i 'asrvhttp://localhost:50000/mode=input&responderName=input'
  o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=1'
  l aproxy!
Note the asrvhttp service and the aproxy logic.

Testing
curl X POST d "{ 'key': 'value1234' }"
H "ContentType: application/json"
http://localhost:50000/
You can use the HTTP client of your choice.

Modes
processor Each processor thread opens a session and processes
incoming messages until the end conditions are met. The threads are not
restarted and program terminates when all threads terminate. This is a
default mode.
server The session processing is as in the processor mode. However
when the thread encounters the end of data condition, the session is
recreated. This mode is useful for the synchronous network servers and
when one wants to automatically release resources after some period of
inactivity.

Transaction Control
NONTXN No transaction call are executed.
TXN After each iteration the commit (or rollback) API
call will be executed on success (error).
XA No transaction control calls are executed and it is
assumed that proper XA resource manger is configured.

Configuration Files
ias_qs_processor f <configuration_file>
Configuration (XML or JSON)
- input and output specification
(Queues, HTTP(S), Files)
- data sources (data bases)
- logic(s) configuration
You can split the configuration into separate definitions of inputs, outputs
or/and datasources.
ias_qs_processor i inputs.xml o outputs.xml d ds.xml

Writing
Data Processor Configuration
Follow the XSD files from the IAS-QsystemLib project.
You can write in XML or JSON.
File name Contents
qsystem.service.xsd QSystem service data schema.
qsystem.workers.xsd QSystem main processor schema.
qsystem.workers.logic.xsd QSystem processor logic schema.
qsystem.workers.ds.xsd QSystem data sources (SQL) schema.
qsystem.workers.io.xsd QSystem input/output (messaging) schema.
qsystem.workers.txm.xsd QSystem XA/TX schema.
qsystem.workers.spec.xsd QSystem combined specification schema.
qsystem.workers.cache.xsd QSystem cache module schema.

Input Configuration
<inputSpec>
<inputs xsi:type="io:ConsumerInput"
inputName="input"
responderName="output">
<connection alias="qs.esb"/>
<destination>CIF.INQ.IN</destination>
<txnMode>NONTXN</txnMode>
<timeout>-1</timeout>
</inputs>
</inputSpec>
The above configuration creates a server.
The responderName field creates corresponding output to send replies.
The connection details can be specified here or are referenced via an alias name from
the registry file (as shown above).

Output Configuration
The above configuration creates a client.
The inputDestination field specified where the underlying protocol expects the replies .
<outputSpec>
<outputs xsi:type="io:RequesterOutput"
outputName="output">
<connection alias="qs.esb"/>
<destination>WWW.INQ.OUT</destination>
<timeout>2000</timeout>
<attrUpdates name="FMT" value="XML" override="true"/>
<inputDestination>WWW.INQ.IN</inputDestination>
</outputs>
</outputSpec>

Exit Configuration
The above configuration specifies that the cb::ValidateCustomer program is to be called.
The exit program parameters are matched against the input data type.
Available error actions are: IGNORE_ALL, IGNORE_PROGRAM, IGNORE_NONE.
<inputs xsi:type="io:ConsumerInput"
inputName="input"
responderName="output" >
<connection>
<protocol>mqm</protocol>
<host>MQTSTBRK</host>
</connection>
<destination>DB.INQ.IN</destination>
<txnMode>TXN</txnMode>
<timeout>-1</timeout>
<attrUpdates name="DB" value="@timestamp@" override="true"/>
<onSend>
<program>cb::ValidateCustomer</program>
<errorAction>IGNORE_ALL</errorAction>
</onSend>
</inputs>

Data Source
<datasourceSpec>
<datasources name="ds.mcif">
<connection>
<protocol>db2</protocol>
<user>user</user>
<password>password</password>
<location>CIFDB</location>
</connection>
<txnMode>TXN</txnMode>
</datasources>
</datasourceSpec>
This example provides all configuration without referencing the registry.

Logic – Explicit Execute
In this scenario the processor will start two threads.
Note how “input” and “output”, “error” i/o destinations are specified.
<logicSpec>
<logics xsi:type="logic:ExplicitExecute"
error="brk.error" instances="2">
<load>esb::broker::inq::routeInput</load>
</logics>
<logics xsi:type="logic:ExplicitExecute"
input="CIF" output="output" error="brk.error">
<load>esb::broker::inq::routeOutput</load>
</logics>
</logicSpec>

Logic – Mach Execute
<logicSpec>
<logics xsi:type="logic:MatchExecute">
<load>mcif::svc::party::getParties</load>
<load>mcif::svc::role::getContextRoles</load>
<load>mcif::svc::role::getPartyProducts</load>
<load>mcif::svc::comm::getCommunicators</load>
</logics>
</logicSpec>
In this scenario the processor will find programs that match the input data.

Logic – Input and Output
Each logic has at least the following defaults:
- input
- output
- error (optional)
Above are used by the predefined logic implementations
and the script execution framework.
You can specify many input and output definitions in the
configuration file. Then you can either assign them as defaults for
the desired logic or use them explicitly via std::send() or std::receive()
Script calls.

Environment
Data model:
IAS_LANG_XSD=/home/ias/data/xsd/model.xsd
Script Sources:
IAS_LANG_SRC_DIRS=/home/ias/lang

First Script
vi /home/ias/lang/update_msgs.y
The Eclipse source highlighting plugin is available.
See the Script Language for more details
(for example on writing procedures accessing databases and files and more).

Processing Data
Testing, browsing and displaying result on the screen:
ias_qs_processor i mqm://TEST/Q1
o file:stdout_fmt l exe update
Processing, moving and saving changed content in the output queue:
ias_qs_processor i mqm://TEST/Q1?mode=input
o mqm://TEST/Q2 l exe update

Learn More
http://www.invenireaude.com/content/books/QSystemProc/index.html?show=chapters/ref/index.html

Event Processing and Integration with IAS Data Processors

Recommended

More Related Content

What's hot (19)

Similar to Event Processing and Integration with IAS Data Processors (20)

Recently uploaded (20)

Event Processing and Integration with IAS Data Processors