Event Processing and Integration with IAS Data Processors
1 like781 views
The document provides a comprehensive overview of data processors, detailing their configuration for tasks such as data transformation, network communication, and transaction management. It outlines various operation modes, including single-threaded and multi-threaded processing, as well as command-line usage examples. Key features include support for message parsing, transaction control, and integration with databases and messaging systems.
Event Processing and Integration with IAS Data Processors
- 1. InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd. Practical Guide to Data Processors
- 2. Data Processors Data processors can be configured to act as: Data transformation nodes, using PASCAL-like script language, Gateways and bridges (e.g. HTTP/JSON and Queues/XML), SQL Database interfaces using the data mapping script extension. You can configure and use the Data Processors as single threaded programs but you can define many logic implementations and run them in parallel as threads. You can choose the transaction support from the three available modes: auto-commit, single phase (independent) commits, distributed two phase commit with XA when the supported coordination software is used. And last but not least, one can find the Data Processors as a very helpful command line admin's tool.
- 3. Architecture
- 4. Concept The Data Processor is designed to process the incoming messages calling desired action accordingly. 1. Begin remote transactions (optional), 2. Receive message, 3. Parse content (if configured) and call prerequisit actions (if defined). 4. Invoke the configured logic. 5. Commit or rollback transaction (in case of an exception). The program can interact with databases and send messages according to the requirements.
- 5. First Usage export IAS_LANG_XSD=${your_project}/data/customer.xsd File operations: ias_qs_processor i file:customer.xml o file:stdout l fwd ias_qs_processor i file:cust.xml o file:stdout_fmt l fwd ias_qs_processor i file:c1.xml o file:somedir/c2.xml l fwd l fwd vs l fwd! Turns the parsing off. ias_qs_processor i file:customer.xml o file:stdout_fmt?format=JSON l fwd
- 6. Directories ias_qs_processor i dir:data/xml o file:output l fwd The above command will display subsequent messages on the terminal output. In order to direct the messages into series of files use the dir protocol in output specification. ias_qs_processor i dir:data/xml o 'dir:output/${ID}.${COUNT}.out' l fwd The parameter is a directory name (output in this case) and the file name pattern.
- 7. Timer The next command will produce two messages with a one second (1000ms) delay. ias_qs_processor N 2 i timer:clock?timeout=1000 o file:stdout l fwd
- 8. Queue Operations ias_qs_processor i shm://sample/Q.TEST o file:stdout l fwd By default, the messages (and files) are read in the shared browser mode. ias_qs_processor i shm://sample/Q.TEST?mode=input o file:stdout l fwd Note that if the transport layer does not support browsing (e.g. HTTP/FCGI) omitting the modifier yields in an error. By default, if there are no messages available, the reading ends immediately. You can specify the timeout parameter (a number of milliseconds or -1 for an infinite wait). ias_qs_processor i shm://sample/Q.TEST?mode=input?timeout=10000 o file:stdout l fwd
- 9. Network Operations 1 When the receiver is going to be a server, one can start it with the command: ias_qs_processor i nets://localhost:10000/?mode=input o file:stdout_fmt l fwd and invoke a client with the matching address specified on its output: ias_qs_processor i file:data/xml/customer.xml o netc://127.0.0.1:10000/ l fwd
- 11. Modes Typical operation patterns are: ● Producer, ● Consumer, ● Requester, ● Responder, ● Publisher, ● Subscriber, ● Browser, ● Shared Worker Browser (default!)
- 14. Publish-Subscribe The application calls the send() and receive() procedures and is not aware of such a configuration. Not all transports support this pattern.
- 15. Implementing a Service In this case API allows to perform send() and receive() calls on both i/o descriptors. One can specify the same or different names for the requester input and output.
- 16. Browser autoRecycle specifies if the processing should resume after end of data condition occurs. ● never Browser is not reset when no message is available. ● next Browser will be reset where no message is available but the exception is thrown this time. The next call will grab a new message. ● immediate Browser will be reset where no message is available and the program attempts to read a message immediately.
- 17. Shared Browser This time, the dedicated thread reads messages on behalf of all logic threads. This pattern allows to take full advantage of parallel processing.
- 18. Built-in Actions Typical operation patterns are: ● Forward. ● Execute. ● Input data type match execute. ● Attribute based selection execute. ● Job Daemon. ● Synchronous proxy. ● Asynchronous proxy. ● Message preview. ● Message extraction. ● Queue System statistics producer. ● Writing custom logic. ● Java Script execute. ● Simplistic Process Server.
- 19. Considerations
- 21. Parsing export IAS_LANG_XSD=${your_project}/data/customer.xsd Note the exclamation mark right after the logic name: l fwd vs l fwd! Turns the parsing off.
- 24. Working with Meta Attributes ias_qs_processor i mqm://MQTEST/Q1?mode=input o 'mqm://MQTEST/Q2?*Folder.ATTR1=VALUE' l fwd The '*' attributes provided in URL are considered as message meta attributes and will be seen in RFH2 or message properties according to the application or IBM WebSphere MQ version.
- 25. Protocol Gateway
- 26. Example Queue Setup DEF QL(GATEWAY.IN) DEF QL(SERVICE.IN) DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)
- 28. Gateway Synchronous Proxy ias_qs_processor i 'srvhttp://localhost:50000/mode=input&responderName=input' o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000' l proxy! m server Note the srvhttp service and the proxy logic.
- 29. Gateway Asynchronous Proxy ias_qs_processor i 'asrvhttp://localhost:50000/mode=input&responderName=input' o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=1' l aproxy! Note the asrvhttp service and the aproxy logic.
- 31. Modes processor Each processor thread opens a session and processes incoming messages until the end conditions are met. The threads are not restarted and program terminates when all threads terminate. This is a default mode. server The session processing is as in the processor mode. However when the thread encounters the end of data condition, the session is recreated. This mode is useful for the synchronous network servers and when one wants to automatically release resources after some period of inactivity.
- 32. Transaction Control NONTXN No transaction call are executed. TXN After each iteration the commit (or rollback) API call will be executed on success (error). XA No transaction control calls are executed and it is assumed that proper XA resource manger is configured.
- 33. Configuration Files ias_qs_processor f <configuration_file> Configuration (XML or JSON) - input and output specification (Queues, HTTP(S), Files) - data sources (data bases) - logic(s) configuration You can split the configuration into separate definitions of inputs, outputs or/and datasources. ias_qs_processor i inputs.xml o outputs.xml d ds.xml
- 35. Writing Data Processor Configuration Follow the XSD files from the IAS-QsystemLib project. You can write in XML or JSON. File name Contents qsystem.service.xsd QSystem service data schema. qsystem.workers.xsd QSystem main processor schema. qsystem.workers.logic.xsd QSystem processor logic schema. qsystem.workers.ds.xsd QSystem data sources (SQL) schema. qsystem.workers.io.xsd QSystem input/output (messaging) schema. qsystem.workers.txm.xsd QSystem XA/TX schema. qsystem.workers.spec.xsd QSystem combined specification schema. qsystem.workers.cache.xsd QSystem cache module schema.
- 36. Input Configuration <inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.esb"/> <destination>CIF.INQ.IN</destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec> The above configuration creates a server. The responderName field creates corresponding output to send replies. The connection details can be specified here or are referenced via an alias name from the registry file (as shown above).
- 37. Output Configuration The above configuration creates a client. The inputDestination field specified where the underlying protocol expects the replies . <outputSpec> <outputs xsi:type="io:RequesterOutput" outputName="output"> <connection alias="qs.esb"/> <destination>WWW.INQ.OUT</destination> <timeout>2000</timeout> <attrUpdates name="FMT" value="XML" override="true"/> <inputDestination>WWW.INQ.IN</inputDestination> </outputs> </outputSpec>
- 38. Exit Configuration The above configuration specifies that the cb::ValidateCustomer program is to be called. The exit program parameters are matched against the input data type. Available error actions are: IGNORE_ALL, IGNORE_PROGRAM, IGNORE_NONE. <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output" > <connection> <protocol>mqm</protocol> <host>MQTSTBRK</host> </connection> <destination>DB.INQ.IN</destination> <txnMode>TXN</txnMode> <timeout>-1</timeout> <attrUpdates name="DB" value="@timestamp@" override="true"/> <onSend> <program>cb::ValidateCustomer</program> <errorAction>IGNORE_ALL</errorAction> </onSend> </inputs>
- 39. Data Source <datasourceSpec> <datasources name="ds.mcif"> <connection> <protocol>db2</protocol> <user>user</user> <password>password</password> <location>CIFDB</location> </connection> <txnMode>TXN</txnMode> </datasources> </datasourceSpec> This example provides all configuration without referencing the registry.
- 40. Logic – Explicit Execute In this scenario the processor will start two threads. Note how “input” and “output”, “error” i/o destinations are specified. <logicSpec> <logics xsi:type="logic:ExplicitExecute" error="brk.error" instances="2"> <load>esb::broker::inq::routeInput</load> </logics> <logics xsi:type="logic:ExplicitExecute" input="CIF" output="output" error="brk.error"> <load>esb::broker::inq::routeOutput</load> </logics> </logicSpec>
- 41. Logic – Mach Execute <logicSpec> <logics xsi:type="logic:MatchExecute"> <load>mcif::svc::party::getParties</load> <load>mcif::svc::role::getContextRoles</load> <load>mcif::svc::role::getPartyProducts</load> <load>mcif::svc::comm::getCommunicators</load> </logics> </logicSpec> In this scenario the processor will find programs that match the input data.
- 42. Logic – Input and Output Each logic has at least the following defaults: - input - output - error (optional) Above are used by the predefined logic implementations and the script execution framework. You can specify many input and output definitions in the configuration file. Then you can either assign them as defaults for the desired logic or use them explicitly via std::send() or std::receive() Script calls.
- 45. First Script vi /home/ias/lang/update_msgs.y The Eclipse source highlighting plugin is available. See the Script Language for more details (for example on writing procedures accessing databases and files and more).
- 46. Processing Data Testing, browsing and displaying result on the screen: ias_qs_processor i mqm://TEST/Q1 o file:stdout_fmt l exe update Processing, moving and saving changed content in the output queue: ias_qs_processor i mqm://TEST/Q1?mode=input o mqm://TEST/Q2 l exe update