[Type text]

CONTENTS
INTRODUCTION MESSAGE QUEUING PROTOCOL DEVELOPMENT
MESSAGES

MQSERIES PRIMER MESSAGING AND QUEUING MESSAGE SEGMENTING AND GROUPING MESSAGE TYPES MESSAGE DESCRIPTOR QUEUE MANAGER QUEUE MANAGER OBJECTS PROCESS DEFINITIONS MANIPULATING QUEUE MANAGER OBJECTS CLIENTS AND SERVERS CONCLUSION

INTRODUCTION TO MESSAGE QUEUING:

Message queuing allows distributed applications to communicate asynchronously by sending messages between the applications. The messages from the sending application are stored in a queue and are retrieved by the receiving application. The applications send or receive messages through a queue by sending a request to the message queuing system. Sending and receiving applications can use the same message queuing system or different ones, allowing the message queuing system to handle the forwarding of the messages from the sender queue to the recipient queue. `Queued messages can be stored at intermediate nodes until the system is ready to forward them to the next node. At the destination node, the messages are stored in a queue until the receiving application retrieves them from the queue. Message delivery is guaranteed even if the network or application fails. This provides for a reliable communication channel between the applications. The complexity and details of the underlying model (to store and forward messages between different environments) are handled by the message queuing system. By maintaining this level of abstraction, distributed applications can be developed without the need to worry about the details of how the information is transported. Because the sending and receiving applications operate independently of one another, the sending application is less dependent on the availability of the remote application, less dependent on the network between them, and less dependent on the computer system on which the receiving application runs. This leads to a higher level of availability for the participating applications. Messages and message queue operations can be configured by the applications to operate in specific modes. For example, a sending application can specify that queued messages should survive system crashes. As another example, the receiving application can specify a maximum waiting period for a receiving operation from a queue (in case no messages are available yet on the receiving queue). ADVANCE MESSAGE QUEUING PROTOCOL: The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.[1] AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardize middlewarehave happened at the API level (e.g. JMS) and this did not create interoperability.[2] Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language. DEVELOPMENT: AMQP was originated in 2003 by John O'Hara at JPMorgan Chase & Co. in London, UK. From the beginning AMQP was conceived as a co-operative open effort. Initial

[Type text]

development was by JPMorgan Chase & Co. from mid-2004 to mid-2006 who contracted iMatix Corporation for a C broker and protocol documentation. In 2005 JPMorgan Chase & Co. approached other firms to form a working group that included Cisco Systems, IONA Technologies, iMatix, Red Hat, and TWIST. In the same year JPMorgan Chase & Co. partnered with Red Hat to create Apache Qpid, initially in Java and soon after C++. Independently, RabbitMQ was developed in Erlang and released, followed later by the Microsoft and StormMQ implementations. The working group grew to 23 companies including Bank of America, Barclays, Cisco Systems, Credit Suisse, Deutsche Brse Systems,Goldman Sachs, HCL Technologies Ltd, Progress Software, INETCO Systems, Informatica Corporation (incl. 29 West), JPMorgan Chase Bank Inc. N.A, Microsoft Corporation, my-Channels, Novell, Red Hat, Inc., Software AG, Solace Systems, StormMQ, Tervela Inc., TWIST Process Innovations ltd, VMware (incl. Rabbit Technologies) and WSO2.

MESSAGES: Messages are unnamed and are published to an exchange. A message consists of a header and a content body. While the body is opaque data the header contains a number of optional properties:

routing-key: this field is used in ways dependent on the type of the exchange. immediate: the message will be handled as unroutable if at least one of the queues which would receive the message has no subscription on it. delivery-mode: indicates that a message might need persistence. For such messages the broker makes a best effort to prevent the message loss before consumption. If there is an uncertainty on the broker's end about the successful delivery of the message (e.g. in case of errors) it might deliver the message more than once. Non-persistent delivery modes do not have this behavior. priority: an indicator (a number between 0 and 9) the precedence level of the message.

expiration: the duration in milliseconds before the broker may handle the message as unroutable MQSERIES PRIMER MQSeries is IBMs award winning middleware for commercial messaging and queuing. It is used by thousands of customers in every major industry in many countries around the world. MQSeries speeds implementation of distributed applications by simplifying application development and test. MQSeries runs on a variety of platforms. The MQSeries products enable programs to communicate with each other across a network of unlike components, such
[Type text]

as processors, subsystems, operating systems and communication protocols. MQSeries programs use a consistent application program interface (API) across all platforms.

MQSeries

Application Program

MQI MQ Objects

[Type text]

NETWORK Application Program MQ Objects MQSeries

MQI

[Type text]

MQSeries at Run Time shows the main parts of an MQSeries application at run time. Programs use MQSeries API calls, that is the Message Queue Interface (MQI), to communicate with a queue manager (MQM), the run-time program of MQSeries. For the queue manager to do its work, it refers to objects, such as queues and channels. The queue manager itself is an object as well. The following provides a brief overview of MQSeries, including clients and servers. WHAT IS MESSAGING AND QUEUING? Message queuing is a method of program-to-program communication. Programs within an application communicate by writing and retrieving application-specific data (messages) to/from queues, without having a private, dedicated, logical connection to link them.

Messaging means that programs communicate with each other by sending data in messages and not by calling each other directly. Queuing means that programs communicate through queues. Programs communicating through queues need not be executed concurrently. With asynchronous messaging, the sending program proceeds with its own processing without waiting for a reply to its message. In contrast, synchronous messaging waits for the reply before it resumes processing. For the user, the underlying protocol is transparent. The user is concerned with conversational or data-entry type applications. MQSeries is used in a client/server or distributed environment. Programs belonging to an application can run in one workstation or in different machines on different platforms. Applications can easily be moved from one system or platform to another. The programs can be written in various programming languages, including Java. The same queuing mechanism is valid for all platforms, and so are the currently 13 APIs. Since MQSeries communicates via queues it can be referred to as using indirect program-to-program communication. The programmer cannot specify the name of the target application to which a message is sent. However, he or she can specify a target queue name; and each queue is associated with a program. An application can have one or more input queues and may have several output queues containing information for other servers to be processed, or for responses for the client that initiated the transaction.

The programmer does not have to worry about the target program being busy or not available. He or she isnt even concerned about the server being down or having no connection to it. The programmer sends messages to a queue that is associated with an application; and the application may or may not be available at the time of the request. MQSeries takes care of the transport to the target application and even starts it, if necessary.

[Type text]

If the target program is not available, the messages stay in a queue and get processed later. The queue is either in the sending machine or in the target machine, depending whether the connection between the two systems can be established or not. Applications can be running all day long or they can be triggered, that is, automatically started when a message arrives or after a specified number of messages have arrived. Messages and Queues The squares between the queues and the programs represent the Message Queuing Interface (API) the program uses to communicate with MQSeries run-time program, the queue manager. As said before, the API is a simple multi platform API consisting of 13 calls. The API will be discussed later. About Messages A message consists of two parts: 1. Data that is sent from one program to another 2. The message descriptor or message header The message descriptor identifies the message (message ID) and contains control information, also called attributes, such as message type, expiry time, correlation ID, priority, and the name of the queue for the reply. A message can be up to 4 MB or 100 MB long, depending on the MQSeries version you use. MQSeries Version 5 (for distributed platforms) supports a maximum message length of 100 MB. Message Segmenting and Grouping In MQSeries Version 5, messages can be segmented or grouped. Message segmenting can be transparent to the application programmer. If permitted, the queue manager segments a large message when it does not fit in a queue. On the receiving end, the application has the option to either receive the entire message in one piece or each segment separately. This may depend on the buffer size available for the application. A second method of segmenting leaves the programmer in control so that he or she can split a message according to logical boundaries or buffer size available for the program. The programmer puts each segment as a separate physical message; thus several physical messages build one logical message. The queue manager ensures that the order of the segments is maintained. To reduce traffic over the network, you can also group several small messages together and build one larger physical message. This message is then sent to the destination and is there disassembled. Message grouping also guarantees that the order the messages are sent in is preserved.
[Type text]

Distribution Lists Using MQSeries Version 5, you can send a message to more than one destination queue with one MQPUT call. This is done with a dynamic distribution list. A distribution list can be a file that is read at the time an application starts. It can be modified any time. It contains a list of queue names and the queue managers that own them. A message sent to multiple queues belonging to the same queue manager is sent over the network only once and so reduces network traffic. The receiving queue manager replicates the messages and puts them into the destination queues. This function is called late fan-out.

Message Types MQSeries knows four types of messages: Datagram: A message containing information for which no response is expected. Request: A message for which a reply is requested. Reply: A reply to a request message. Report: A message that describes an event such as the occurrence of an error or a confirmation on arrival or delivery. Persistent and Non-Persistent Messages Application design determines whether a message must reach its destination under any circumstances, or if it can be discarded when it cannot get there in time. MQSeries differentiates between persistent and non-persistent messages. Delivery of persistent messages is assured; they are written to logs to survive system failures. In an AS/400 these logs are Journal Receivers. Non-persistent messages cannot be recovered after a system restart. The Message Descriptor The table below contains some interesting attributes of the message descriptor. We mention them here because they explain some of the functions the queue manager provides for you. The version of the message descriptor depends on the MQSeries version and platform you use. For the functions introduced with Version 5 additional fields were needed to keep information about segments and their order and distribution list information, to name a few. This enlarged structure carries the version number 2. Other queue managers who dont support these functions (Version 1 queue managers) treat the additional information as data. Message and/or correlation ID are used to identify a specific request or reply message. The programmer can move a value in one or both fields or have MQSeries create a unique ID for him or her. Before the programmer puts the request message in the queue he or she can save the ID(s) and use them in a subsequent get operation for the reply message. The program that receives the request message copies this information into the reply message. This allows the originating program (the one that gets the reply) to instruct MQSeries to look for a specific message in the queue instead of getting the first one in the queue.

About the Queue Manager

[Type text]

The heart of MQSeries is the message queue manager (MQM), MQSeries run-time program. Its job is to manage queues and messages for applications. It provides the Message Queuing Interface (MQI) for communication with applications. Application programs invoke functions of the queue manager by issuing API calls. For example, the MQPUT API call puts a message on a queue to be read by another program using the MQGET API call. This scenario is shown in

Application Program A

Messages

Application Program B GET Q1 from

PUT to Q1

Program-to-Program Communication - One System

A program may send messages to another program that runs in the same machine as the queue manager (shown above), or to a program that runs in a remote system, such as a server or a host. The remote system has its own queue manager with its own queues.

About Queue Manager Clusters With MQSeries for MVS/ESA and Version 5.1 for distributed platforms, you can join queue managers together in clusters. Queue managers that form a cluster can run in the same machine or in different machines on different platforms. Usually, two of those cluster queue managers maintain a repository that contains information about all queue managers and queues in the cluster. This is called a full repository. The other queue managers maintain only a repository of objects they are interested in, a partial repository. The repository allows any queue manager in the cluster to find out about any cluster queue and who owns it. The queue managers
[Type text]

use special cluster channels to exchange information. Clustering also permits multiple instances of a queue (with the same name) on different queue managers. This allows for workload distribution, that is, the queue manager can send messages to different instances of an application. In normal distributed processing, we send messages to a specific queue owned by a specific queue manager. All messages destined for that queue manager are placed in a transmission queue on the senders side. This transmission queue has the same name as the destination queue manager. The message channel agents move the messages across the network and place them into the destination queues. Figure 6 shows the relationship between a transmission (Xmit) queue and the target queue manager.

Targ et MQPUT Que ue

Clus ter Xmi t Que ue

MQPUT to a Cluster Queue The vision of an MQSeries cluster is as the place where multiple instances of a queue can exist. They come and go as an administrator requires in order to satisfy changing availability and throughput requirements. This has to be achieved completely dynamically and without placing the administrator under a great burden to configure and control. In addition, the programmer does not have to think about multiple queues; he or she just treats them as if writing to a single queue. This is not to say that there is no burden on the programmer or administrator. Enhanced levels of availability and exploitation of parallelism do require some planning. The administrator or system designer must ensure that there is enough redundancy in the configuration to meet their needs. The application designer must ensure that messages are capable of being processed in multiple places.

[Type text]

You create multiple instances of a queue by defining a queue with the same name on multiple queue managers that belong to the cluster. You must also name the cluster when you define the queue. Without this attribute the queue would only be known locally. When the application specifies only the queue name, where is the message sent?

MQPUT

A B

A B

3 Figur ACCESSING SUCCESS e 8. QUEUE

gives you an idea. MQSeries distributes the messages round-robin. You can, however, change this default action by writing your own workload balancing exit routine. shows messages put in one of the three cluster queues named A. Each of the three queue managers on the right owns a queue with this name. By default, the first message is placed in queue A on queue manager 1, the next in queue A on queue manager 2, the third goes to queue manager 3 and the fourth message to the queue on queue manager 1 again.

In another scenario involving queue B, we notice that the third queue manager is down and the third instance of queue B is not available. The sending queue manager becomes aware of this problem because it subscribed to information about all queue manager and queues it is
[Type text]

interested in, that is, where it sends messages. As soon as it finds out that there is a problem with the third instance of B, it distributes messages to the first two instances only. Special messages about changes of the status of cluster objects are instantly published to all queue managers that subscribed to that object. About Queue Manager Objects This section introduces you to queue manager objects, such as queues and channels. The queue manager itself is an object, too. Usually, an administrator creates one or more queue managers and their objects. A queue manager can use objects of the following types: 1. Queues 2. Process definitions 3. Channels The objects are common across different MQSeries platforms. There are other objects that apply to MVS systems only, such as the buffer pool, PSID, and storage class. AS/400 MQ objects are known to the OS/400 operating system as object type *USRSPC (user space) in the QMQMDATA library. Queues Message queues are used to store messages sent by programs. There are local queues that are owned by the local queue manager, and remote queues that belong to a different queue manager. Queues are described in more detail in the section Channels A channel is a logical communication link. In MQSeries, there are two different kinds of channels: 1. Message channels A message channel connects two queue managers via message channel agents (MCAs). Such a channel is unidirectional. It comprises two message channel agents, a sender and a receiver, and a communication protocol. An MCA is a program that transfers messages from a trans-mission queue to a communication link, and from a communication link into the target queue. For bidirectional communication you have to define two channel pairs consisting of a sender and a receiver. Message channel agents are also referred to as movers.

2. MQI channels A Message Queue Interface (MQI) channel connects an MQSeries client to a queue manager in its server machine. Clients dont have a queue manager of their own. An MQI channel is bidirectional. shows both channels types. You see four machines, two clients connected to their server machine via MQI channels, and the server connected to another server or a host via two unidirectional message channels. Some channels can be defined automatically by MQSeries. There are different types of message channels, depending on how the session between the queue

[Type text]

managers is initiated and for what purpose they are used. To transmit non-persistent messages, a message channel can run at two speeds: fast and normal. Fast channels improve performance, but (non-persistent) messages can be lost in case of a channel failure. Process Definitions A process definition object defines an application to a queue manager. For example, it contains the name of the program (and its path) to be triggered when a message arrives for it. About Message Queues Queues are defined as objects belonging to a queue manager. MQSeries knows a number of different queue types, each with a specific purpose. The queues you use are located either in your machine and belong to the queue manager to which you are connected, or in your server (if you are a client). Figure 10 lists different queue types and their purposes. More detailed information is below.

[Type text]

MQ Client MQI Channels MQ M

MQ Client Message Channels

Local queue Remote queue Transmission queue (xmitq) Initiation queue Dynamic queue Alias queue Dead-letter queue Reply-to-queue Model queue Repository queue Figure 10. Queue Types Local Queue

is a real queue structure describing a queue local queue with special purpose local queue with special purpose local queue created "on the fly" if you dont like the name one for each queue manager specified in request message model for local queues holds cluster information

[Type text]

A queue is local if it is owned by the queue manager to which the application program is connected. It is used to store messages for programs that use the same queue manager. For example, program A and program B each has a queue for incoming messages and another queue for outgoing messages. Since the queue manager serves both programs, all four queues are local. Note: Both programs do not have to run in the same workstation. Client workstations usually use a queue manager in a server machine. Cluster Queue A cluster queue is a local queue that is known throughout a cluster of queue managers, that is, any queue manager that belongs to the cluster can send messages to it without the need of a remote definition or defining channels to the queue manager that owns it. Remote Queue A queue is remote if it is owned by a different queue manager. A remote queue definition is the local definition of a remote queue. A remote queue is not a real queue. It is a structure that contains some of the characteristics of a queue hosted by a different queue manager. The application programmer can use the name of a remote queue just as he or she can use the name of a local queue. The MQSeries administrator defines where the queue actually is. Remote queues are associated with a transmission queue. Notes: - A program cannot read messages from a remote queue. - You dont need a remote queue definition for a cluster queue. Transmission Queue This is a local queue with a special purpose. A remote queue is associated with a transmission queue. Transmission queues are used as an intermediate step when sending messages to queues that are owned by a different queue manager. Typically, there is only one transmission queue for each remote queue manager (or machine). All messages written to queues owned by a remote queue manager are actually written to the transmission queue for this remote queue manager. The messages will then be read from the transmission queue and sent to the remote queue manager. Using MQSeries clusters, there is only one transmission queue for all messages sent to all other queue managers in the cluster. Transmission queues are transparent to the application. They are used internally by the
[Type text]

queue manager. When a program opens a remote queue, the attributes of the queue are obtained from the transmission queue. Therefore, the results of a program writing messages to a queue will be affected by the transmission queue characteristics. Dynamic Queue Such a queue is defined "on the fly" when the application needs it. Dynamic queues may be retained by the queue manager or automatically deleted when the application program ends. Dynamic queues are local queues. They are often used in conversational applications, to store intermediate results. Dynamic queues can be: Temporary queues that do not survive queue manager restarts Permanent queues that do survive queue manager restarts

Alias Queue Alias queues are not real queues but definitions. They are used to assign different names to the same physical queue. This allows multiple programs to work with the same queue, accessing it under different names and with different attributes. Model Queue A model queue is not a real queue. It is a collection of attributes that are used when a dynamic queue is created. Initiation Queue An initiation queue is a local queue to which the queue manager writes a trigger message when certain conditions are met on another local queue, for example, when a message is put into an empty message queue or in a transmission queue. Such a trigger message is transparent to the programmer. Two MQSeries applications monitor initiation queues and read trigger messages, the trigger monitor which starts applications and the channel initiator which starts the transmission between queue managers. Note: Applications do not need to be aware of initiation queues, but the triggering mechanism implemented through them is a powerful tool to design and write asynchronous applications.

Reply-to-Queue A request message must contain the name of the queue into which the responding program must put the reply message. This can be considered the return address. The name of this queue together with the name of the queue manager that owns it is stored in the message header. This is the responsibility of the application program. Dead-Letter Queue

[Type text]

A queue manager must be able to handle situations when it cannot deliver a message. Here are some examples: The destination queue is full. The destination queue does not exist. Message puts have been inhibited on the destination queue. The sender is not authorized to use the destination queue. The message is too large. The message contains a duplicate message sequence number.

When the above conditions are met, the messages are written to the dead-letter queue. Such a queue is defined when the queue manager is created. It will be used as a repository for all messages that cannot be delivered. Repository Queue

[Type text]

Repository queues have existed since Version 5.1 and Version 2.1 for OS/390. They are used in conjunction with clustering and hold either a full or a partial repository of queue managers and queue manager objects in a cluster (or group) of queue managers.Creating a Queue Manager You may create as many queue managers as you like and have them running at the same time. You create a queue manager with the command crtmqm; to make it the default, specify the parameter /q. The following command creates the default queue manager MYQMGR (in a Windows NT environment): crtmqm /q MYQMGR Note: Queue manager names are case-sensitive. There are default definitions for objects every queue manager needs, such as model queues. These objects are created automatically. Most certainly, you will have to create other objects that pertain to the applications you run. Usually, those application specific objects are kept in a script file, such as mydefs.in. You apply them to a newly created queue manager with the command: Manipulating Queue Manager Objects MQSeries for distributed platforms provides the utility RUNMQSC to create and delete queue manager objects and to manipulate them. The queue manager must be running when you use the utility. RUNMQSC works in two ways: You can type the commands. You can create a file containing a list of commands and use this file as input.

The commands in Figure 11 on page 17 start the default queue manager (which is already running, as the response indicates) and create the local queue QUEUE1 for it. Another command alters the queue manager properties to define a dead-letter queue. To start the utility in an interactive mode, type runmqsc. To end it, type end. Another way to create MQSeries objects is by using an input file instead of typing the commands; for example: runmqsc < mydefs.in > a.a

Clients and Servers MQSeries distinguishes clients and servers. Before you install MQSeries on a distributed

[Type text]

platform you have to decide if the workstation will be an MQSeries client, an MQSeries server, or both. With MQSeries for Windows a new term was introduced, the leaf node (described later). There are two kinds of clients: Slim client or MQSeries client Fat client

Fat clients have a local queue manager; slim clients don't. When a slim client cannot connect to its server it cannot work, because the queue manager and queues for a slim client reside in the server. Usually, an MQSeries client is a slim client. Several of these clients share MQSeries objects, and the queue manager is one of them, in the server to which they are attached. Note: The MQSeries Client for Java is a slim client. In some cases it may be advantageous to have queues in the end user's workstation, especially in a mobile environment. That allows you to run your application when a connection between CONCLUSION: There is more to message queues than this short tutorial can present. Be sure to look in the man pages to see what else you can do, especially in the area of msgctl(). Also, there are more options you can pass to other functions to control how msgsnd() and msgrcv() handle if the queue is full or empty, respectively.

[Type text]