S2GX 2012 - Introduction to Spring Integration and Spring Batch

Introduction to
Spring Integration and Spring Batch
Gunnar Hillert, Spring Integration Team
Gary Russell, Spring Integration Team
Twitter: @ghillert, @gprussell

© 2012 SpringOne 2GX. All rights reserved. Do not distribute without permission.

What we will cover...

• Spring Integration
• Spring Batch
• Using Spring Batch and Spring Integration together

2

Integration Styles

JVM JVM

Core Messaging
EAI

• Business to Business Integration (B2B)
• Inter Application Integration (EAI)
• Intra Application Integration

B2B

External Business
Partner

4

Integration Styles

• File Transfer
• Shared Database
• Remoting
• Messaging

5

Common Patterns

Retrieve Parse Transform Transmit

6

Enterprise Integration Patterns

• By Gregor Hohpe & Bobby Woolf
• Published 2003
• Collection of well-known patterns
• http://www.eaipatterns.com/eaipatterns.html
• Icon library provided

7

“ Spring Integration provides an extension
of the Spring programming model
to support the well-known enterprise
integration patterns.

8

History

• Nov 13, 2007 - First commit: Nov 13, 2007
• Jan 23, 2008 - Spring Integration 1.0.0.M1
• Nov 26, 2008 - Spring Integration 1.0.0
• Nov 22, 2010 - Spring Integration 2.0.0
• Jan 6, 2012 - Spring Integration 2.1.0
• Sep 21, 2012 - Spring Integration 2.2.0.RC1

• 2013 - Spring Integration 3.0

9

What is Spring Integration?

• Light-weight messaging framework
• Provides an adapter-based platform

• Pipes and Filters at the core of Spring Integration’s architecture
– Endpoint (Filter)
– Channel (Pipe)
– Message

10

Advantages

• Provides building blocks to implement systems that are:
– are loosely Coupled (Logically or Physically)
– are Event Driven (EDA)
– have a staged event-driven architecture (SEDA)
• Sophisticated support for synchronous / asynchronous messaging

11

Configuration

• XML Namespace Support
• Annotation Support (e.g. @Transformer, @Router, @ServiceActivator)

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:int="http://www.springframework.org/schema/integration"
xsi:schemaLocation="..">
...
<int:channel id="orders"/>
<int:splitter input-channel="orders" expression="payload.items"
output-channel="drinks"/>
<int:channel id="drinks"/>
<int:router input-channel="drinks"
expression="payload.iced ? 'coldDrinks' : 'hotDrinks'"/>
...
</beans>

12

Spring Integration DSLs

• Scala
val messageFlow =
filter {payload: String => payload == "World"} -->
transform { payload: String => "Hello " + payload} -->
handle { payload: String => println(payload) }

messageFlow.send("World")

• Groovy
def builder = new IntegrationBuilder()

def flow = builder.messageFlow {
transform {payload->payload.toUpperCase()}
filter {payload-> payload =="HELLO"}
handle {payload->payload}
}

assert flow.sendAndReceive("hello") == "HELLO"
assert flow.sendAndReceive("world") == null

13

Deployment Options

• Embedded as a library
• Stand-alone deployment
• War deployment

14

What is in a Message?

• Unit of information
• Encapsulates data
• Passed between endpoints
• Consists of headers
– contains data relevant to the messaging system
• and a payload
– actual data for the receiver
– depending on use-cases: POJO instances or serialized data

15

What is in a Message?

package org.springframework.integration;

public interface Message<T> {

MessageHeaders getHeaders();

T getPayload();

}

16

Message Headers

• Message ID (automatically generated UUID)
• Timestamp
• Correlation Id
• Reply Channel
• Error Channel
• Expiration Date
• Priority
• ...

• Add your own headers using a Header Enricher

17

Function of a Message

• Command Message
C

• Event Message
E

• Document Message
D

18

What is a Channel?

• Channels connect producers and consumers (decoupling)
• MessageChannel Interface:
– PollableChannel (Polling Consumer)
– SubscribableChannel (Event Driven)
• Implementations:
– DirectChannel
– PublishSubscribeChannel
– QueueChannel
<int:channel id="input">
– PriorityChannel <int:queue capacity="10"/>
</int:channel>
– RendezvousChannel
– ExecutorChannel
19

What is an Endpoint?

• Polling or event-driven
• Inbound or outbound
• Unidirectional (Channel Adapter) or bidirectional (Gateway)
• Internal or external (application context)

<inbound-channel-adapter/>
<outbound-channel-adapter/>
<inbound-gateway/>
<outbound-gateway/>
<gateway/>
<service-activator/>

20

Router

• Message Router
• Content-based router
• Recipient list router (dynamic)
• Payload type router
• Header value router
• Exception type router

21

Transformer

• Delegating via ref/method
• Spring Expression Language
• Groovy, JRuby, Jython, JavaScript
• Object-to-JSON / JSON-to-Object
• Payload serializing/deserializing
• File-to-bytes, File-to-String
• JAXB, JibX, Castor, XMLBeans, Xstream
• XPath, XSLT
• Object XML Marshalling/Unmarshalling (Spring OXM)
• ...

22

Spring Integration Components

• Claim Check (In/Out) • Service Activator
• Content Enricher • Scripting support (JSR 223)
– Header Enricher – Ruby/JRuby, Javascript ...
– Payload Enricher • Groovy
• Control Bus • Message History
• Delayer • Message Store
• JMX Support – JDBC, Redis, MongoDB,
• Message Handler Chain Gemfire
• Messaging Bridge • Wire Tap
• Resequencer • ...

23

Adapters

• AMQP/RabbitMQ • MongoDB • Stored Procedures
• AWS* • POP3/IMAP/SMTP • TCP/UDP
• File/Resource • Print* • Twitter
• FTP/FTPS/SFTP • Redis • Web Services
• GemFire • RMI (SOAP or POX)
• HTTP (REST) • RSS/Atom • XMPP
• JDBC • SMB* • XPath
• JMS • Splunk* • XQuery*
• JMX • Spring Application
• JPA Events • ...
* Spring Integration Extensions Project

24

Tooling - Spring Tool Suite (STS)

• Namespace Support
• Visualization
• 4 Spring Integration specific STS Templates
– Simple Template (Core Components only)
– File Polling Template (File Adapter)
– War Template (Uses Twitter Adapter)
– Adapter Template (Create your own components)

25

Tooling - IntelliJ IDEA

• Spring Integration support since IDEA 10.5
• Namespace Support
• Bean visualization
• IntelliJ IDEA 12 (Preview) supports Spring Integration 2.1 & 2.2

26

Tooling - C24 Integration Objects (C24iO) Studio

• Supports a variety of standards:
– SWIFT
– ISO20022
– FIX
• Namespace Support:

<int-c24:transformer/>
<int-c24:validating-header-enricher/>
<int-c24:marshalling-transformer/>
...

• http://www.c24.biz/io-spring.html

27

Samples

• https://github.com/SpringSource/spring-integration-samples

• Contains 50 Samples and Applications
• Several Categories:
– Basic
– Intermediate
– Advanced
– Applications

30

Books

• Just Spring Integration
• Pro Spring Integration
• Spring Integration in Action

31

Source Code

• https://github.com/SpringSource/spring-integration
• https://github.com/SpringSource/spring-integration-samples
• https://github.com/SpringSource/spring-integration-extensions
• https://github.com/SpringSource/spring-integration-templates
• https://github.com/SpringSource/spring-integration-dsl-groovy
• https://github.com/SpringSource/spring-integration-dsl-scala

32

Contribute

• Post Question and Answers the Forums
– http://forum.springsource.org/forumdisplay.php?42-Integration
• Create Jiras
– https://jira.springsource.org/browse/INT
• Submit Pull Requests - Contributor Guidelines:
– github.com/SpringSource/spring-integration/wiki/Contributor-Guidelines

• New Spring Integration Extensions Repository

33

Batch Jobs
Differ from online/real-time processing applications:

• Long-running
– Often outside office hours
• Non-interactive
– Often include logic for handling errors or restarts
• Process large volumes of data
– More than fits in memory or a single transaction

35

Batch and offline processing
• Close of business processing
– Order processing
– Business reporting
– Account reconciliation
• Import/export handling
– a.k.a. ETL jobs (Extract-Transform-Load)
– Instrument/position import
– Data warehouse synchronization
• Large-scale output jobs
– Loyalty scheme emails
– Bank statements
• 36

Chunk-Oriented Processing
• Input-output can be grouped together
• Input collects Items before outputting:Chunk-Oriented Processing
• Optional ItemProcessor

38

More Complex Use Cases
• It's very common to use an off-the-shelf reader
and writer
• More complex jobs often require custom readers
or writers
• ItemProcessor is often used if there's a need to
delegate to existing business logic
• Use a writer if it's more efficient to process a
complete chunk

41

Job and Step in Context

42

JobRepository and Batch Metadata

43

ExecutionContext
• We need to know where a failure occurred to restart a batch
process
• Job Repository metadata is used to determine the step at
which the failure occurred
• Application Code (in reader/writer) needs to maintain state
within a step (e.g. current chunk)
• Spring Batch can supply that data during restart to facilitate
repositioning

44

Common Batch Idioms
• Batch jobs typically process large amounts of homogeneous
input
• Makes iteration a common concern: Repeat
• Transient errors during processing may require a Retry of
an input item
• Some input may not be valid, may want to Skip it without
failing
• Some errors should fail the job execution, allowing one to fix
the problem and Restart the job instance where it left off

45

Spring Batch
• Spring Batch supports these common concerns

• Abstracts them in the framework
– Job business logic doesn't need to care about details

• Allows for simple configuration with pluggable strategies

46

Business Logic Delegation – Spring Application

47

Business Logic Delegation – Spring Integration

48

Spring Batch Admin
• Sub project of Spring Batch
• Provides Web UI and ReSTFul interface to manage batch
processes

• Manager, Resources, Sample WAR
– Deployed with batch job(s) as single app to be able to control &
monitor jobs
– Or monitors external jobs only via shared database

49

Scaling and Parallel Processing
• First Rule:
– Use the simplest technique to get the job done in the required
time
– Do not optimize/parallelize unnecessarily

• Options:
– Multi-threaded Step (single process)
– Parallel Steps (single process)
– Remote Chunking of Step (multi process)
– Partitioning a Step (single or multi process)

50

Using Spring Batch
and
Spring Integration
together
51

Launching batch jobs through messages

• Event-Driven execution of the JobLauncher
• Spring Integration retrieves the data (e.g. file system, FTP, ...)
• Easy to support separate input sources simultaneously

Inbound Channel Adapter Transformer

FTP

D
File C
JobLaunchRequest

JobLauncher

52

Providing feedback with informational messages

• Spring Batch provides support for listeners:
– StepListener
– ChunkListener
– JobExecutionListener

<batch:job id="importPayments">
...
<batch:listeners>
<batch:listener ref="notiﬁcationExecutionsListener"/>
</batch:listeners>
</batch:job>

<int:gateway id="notiﬁcationExecutionsListener"
service-interface="o.s.batch.core.JobExecutionListener"
default-request-channel="jobExecutions"/>

53

Externalizing batch process execution

• Use Spring Integration inside of Batch jobs, e.g.:
– ItemProcessor
– ItemWriter
• Offload complex processing
• Asynchronous processing support:
– AsyncItemProcessor
– AsyncItemWriter
• Externalize chunk processing using ChunkMessageChannelItemWriter

54

Learn More. Stay Connected.
At SpringOne 2GX:

• Building an Enterprise CRM with Grails and Spring Integration
• What's New in Spring Integration
• Spring Integration in the Wild
• Batch Processing and Integration on Cloud Foundry
• Building for Performance with Spring Integration & Spring Batch
• Spring Integration, Batch, and Data: Future Directions
• Managing and Monitoring Spring Integration Applications
• Java Batch JSR-352

55

Learn More. Stay Connected.
• Web: www.springintegration.org
• Twitter: @m_f_ @ghillert @z_oleg @gprussell

Questions?

Thank You!!

56

S2GX 2012 - Introduction to Spring Integration and Spring Batch

Recommended

More Related Content

What's hot (16)

Viewers also liked (20)

Similar to S2GX 2012 - Introduction to Spring Integration and Spring Batch (20)

More from Gunnar Hillert (13)

Recently uploaded (20)

S2GX 2012 - Introduction to Spring Integration and Spring Batch

Editor's Notes