Apache Camel Developer's Cookbook

Table of Contents

Apache Camel Developer's Cookbook

Credits

About the Authors

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Instant Updates on New Packt Books

Preface

What is Camel?

Camel Concepts

The Camel DSL

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Structuring Routes

Introduction

Using Camel in a Java application

Getting ready

How to do it...

How it works...

There's more...

See also

Embedding Camel in a Spring application

Getting ready

How to do it...

How it works...

There's more...

See also

Using Camel components

Getting ready

How to do it...

How it works...

There's more...

See also

Reusing routing logic by connecting routes

Getting ready

How to do it...

How it works...

There's more...

See also

Asynchronously connecting routes

Getting ready

How to do it...

How it works...

There's more...

See also

Spanning Camel contexts within a single Java process

Getting ready

How to do it...

How it works...

See also

Using external properties in Camel routes

Getting ready

How to do it...

How it works...

There's more...

See also

Reusing endpoints

Getting ready

How to do it...

Reusing routing logic through template routes

Getting ready

How to do it...

How it works...

See also

Controlling route startup and shutdown

Getting ready

How to do it...

How it works...

There's more...

See also

2. Message Routing

Introduction

Content Based Routing

Getting ready

How to do it...

How it works...

There's more...

See also

Filtering out unwanted messages

Getting ready

How to do it...

How it works...

There's more...

See also

Wire Tap – sending a copy of the message elsewhere

Getting ready

How to do it...

How it works...

Deep copying of the exchange

There's more...

See also

Multicast – routing the same message to many endpoints

Getting ready

How to do it...

How it works...

There's more...

See also

Recipient List – routing a message to a list of endpoints

Getting ready

How to do it...

How it works...

There's more...

See also

Throttler – restricting the number of messages flowing to an endpoint

Getting ready

How to do it...

How it works...

There's more...

See also

Request-response route sending a one-way message

Getting ready

How to do it...

How it works...

There's more...

See also

One-way route waiting on a request-response endpoint

Getting ready

How to do it...

How it works...

There's more...

See also

Dynamic Routing – making routing decisions at runtime

Getting ready

How to do it...

How it works...

There's more...

See also

Load balancing across a number of endpoints

Getting ready

How to do it...

How it works...

There's more...

See also

Routing Slip – routing a message to a fixed list of endpoints

Getting ready

How to do it...

How it works...

There's more...

See also

3. Routing to Your Code

Introduction

Routing messages directly to a Java method

Getting ready

How to do it...

How it works...

There's more...

See also

Sending messages directly to a Camel endpoint

Getting ready

How to do it...

How it works...

There's more...

See also

Using a Java method as a Predicate

Getting ready

How to do it...

How it works...

There's more...

See also

Writing a custom Camel Processor

Getting ready

How to do it...

How it works...

There's more...

See also

Mapping the message to method parameters

Getting ready

How to do it...

How it works...

There's more...

See also

Writing a custom data marshaller

Getting ready

How to do it...

How it works...

There's more...

See also

Writing a custom data type converter

Getting ready

How to do it...

How it works...

There's more...

See also

4. Transformation

Introduction

Transforming using a Simple Expression

Getting ready

How to do it...

How it works...

There's more...

See also

Transforming inline with XQuery

Getting ready

How to do it...

How it works...

There's more...

See also

Transforming with XSLT

Getting ready

How to do it...

How it works...

There's more...

See also

Transforming from Java to XML with JAXB

Getting ready

How to do it...

How it works...

There's more...

See also

Transforming from Java to JSON

Getting ready

How to do it...

How it works...

There's more...

See also

Transforming from XML to JSON

Getting ready

How to do it...

How it works...

There's more...

See also

Parsing comma-separated values (CSV)

Getting ready

How to do it...

How it works...

There's more...

See also

Enriching your content with some help from other endpoints

Getting ready

How to do it...

How it works...

There's more...

See also

Normalizing messages into a common XML format

Getting ready

How to do it...

How it works...

There's more...

See also

5. Splitting and Aggregating

Introduction

Splitting a message into fragments

Getting ready

How to do it...

How it works...

There's more...

See also

Splitting XML messages

Getting ready

How to do it...

How it works...

There's more...

See also

Processing split messages in parallel

Getting ready

How to do it...

How it works...

There's more...

See also

Aggregating related messages

Getting ready

How to do it...

How it works...

There's more...

See also

Aggregating with timeouts

Getting ready

How to do it...

How it works...

See also

Aggregating with intervals

Getting ready

How to do it...

How it works...

See also

Processing aggregated messages in parallel

Getting ready

How to do it...

How it works...

There's more...

See also

Splitting a message, and processing and gathering responses

Getting ready

How to do it...

How it works...

There's more...

See also

Splitting messages and re-aggregating them using different criteria

Getting ready

How to do it...

How it works...

There's more...

See also

6. Parallel Processing

Introduction

Increasing message consumption through multiple endpoint consumers

Getting ready

How to do it...

How it works...

There's more...

See also

Spreading the load within a route using a set of threads

Getting ready

How to do it...

How it works...

There's more...

See also

Routing a request asynchronously

Getting ready

How to do it...

How it works...

There's more...

See also

Using custom thread pools

Getting ready

How to do it...

How it works...

There's more...

See also

Using thread pool profiles

Getting ready

How to do it...

How it works...

There's more...

See also

Working with asynchronous APIs

Getting ready

How to do it...

How it works...

There's more...

See also

7. Error Handling and Compensation

Introduction

Logging errors

Getting ready

How to do it...

How it works...

There's more...

See also

Dead Letter Channel – handling errors later

Getting ready

How to do it...

How it works...

There's more...

See also

Retrying an operation

Getting ready

How to do it...

How it works...

There's more...

See also

Conditional retry

Getting ready

How to do it...

How it works...

There's more...

See also

Customizing each redelivery attempt

Getting ready

How to do it...

How it works...

There's more...

See also

Catching exceptions

Getting ready

How to do it...

How it works...

There's more...

See also

Marking exceptions as handled

Getting ready

How to do it...

How it works...

There's more...

See also

Fine-grained error handling using doTry…doCatch

Getting ready

How to do it...

How it works...

There's more...

See also

Defining completion actions

Getting ready

How to do it...

How it works...

There's more...

See also

Defining completion actions dynamically

Getting ready

How to do it...

How it works...

There's more...

See also

8. Transactions and Idempotency

Introduction

Preventing duplicate invocation of routing logic

Getting ready

How to do it...

How it works...

There's more...

See also

Transactional file consumption

Getting ready

How to do it...

How it works...

There's more...

See also

Using transactions with a database

Getting ready

How to do it...

How it works...

There's more...

See also

Limiting the scope of a transaction

Getting ready

How to do it...

How it works...

There's more...

Rolling back a transaction

Getting ready

How to do it...

How it works...

There's more...

Using transactions with messaging

Getting ready

How to do it...

How it works...

There's more...

See also

Idempotency inside transactions

Getting ready

How to do it...

How it works...

There's more...

See also

Setting up XA transactions over multiple transactional resources

Getting ready

How to do it...

How it works...

There's more...

See also

9. Testing

Introduction

Testing routes defined in Java

Getting ready

How to do it...

How it works...

There's more...

See also

Using mock endpoints to verify routing logic

Getting ready

How to do it...

How it works...

There's more...

See also

Replying from mock endpoints

Getting ready

How to do it...

How it works...

See also

Testing routes defined in Spring

Getting ready

How to do it...

How it works...

There's more...

See also

Testing routes defined in OSGi Blueprint

Getting ready

How to do it...

How it works...

There's more...

See also

Auto-mocking of endpoints

Getting ready

How to do it...

How it works...

There's more...

Validating route behavior under heavy load

Getting ready

How to do it...

How it works...

There's more...

See also

Unit testing processors and Bean Bindings

Getting ready

How to do it...

How it works...

There's more...

See also

Testing routes with fixed endpoints using AOP

Getting ready

How to do it...

How it works...

There's more...

See also

Testing routes with fixed endpoints using conditional events

Getting ready

How to do it...

How it works...

There's more...

See also

10. Monitoring and Debugging

Introduction

Logging meaningful steps within your route

Getting ready

How to do it...

How it works...

There's more...

See also

Debugging using logging

Getting ready

How to do it...

How it works...

There's more...

See also

Throughput logging

Getting ready

How to do it...

How it works...

There's more...

See also

Enabling step-by-step tracing in code

Getting ready

How to do it...

How it works...

There's more...

See also

Disabling JMX

Getting ready

How to do it...

How it works...

There's more...

See also

Configuring JMX

Getting ready

How to do it...

How it works...

There's more...

See also

Naming your routes to make it easier to monitor

Getting ready

How to do it...

How it works...

There's more...

See also

Adding JMX attributes and operations

Getting ready

How to do it...

How it works...

There's more...

See also

Monitoring other systems using the Camel JMX Component

Getting ready

How to do it...

How it works...

There's more...

See also

Setting breakpoints in your routes

Getting ready

How to do it...

How it works...

There's more...

See also

11. Security

Introduction

Encrypting configuration properties

Getting ready

How to do it...

How it works...

There's more...

See also

Digitally signing and verifying messages

Getting ready

How to do it...

How it works...

There's more...

See also

Encrypting and decrypting a message

Getting ready

How to do it...

How it works...

There's more...

See also

Encrypting all or parts of an XML message

Getting ready

How to do it...

How it works...

There's more...

See also

Authentication and authorization using Spring Security

Getting ready

How to do it...

How it works...

There's more...

See also

12. Web Services

Introduction

Generating the service stubs from a WSDL

Getting ready

How to do it...

How it works...

There's more...

See also

Invoking a remote web service from Camel

Getting ready

How to do it...

How it works...

There's more...

See also

Implementing a web service with a Camel route

Getting ready

How to do it...

How it works...

There's more...

See also

Providing multiple web service operations within a single route

Getting ready

How to do it...

How it works...

There's more...

See also

Handling web service faults

Getting ready

How to do it...

How it works...

There's more...

See also

Web service proxying

Getting ready

How to do it...

How it works...

There's more...

See also

Index

Apache Camel Developer's Cookbook

Apache Camel Developer's Cookbook

Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2013

Production Reference: 1181213

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78217-030-3

www.packtpub.com

Cover Image by VisitRenoTahoe.com and Reno-Sparks Convention and Visitors Authority (RSCVA)

Credits

Authors

Scott Cranton

Jakub Korab

Reviewers

Bilgin Ibryam

Claus Ibsen

Christian Posta

Phil Wilkins

Acquisition Editors

Saleem Ahmed

Sam Wood

Lead Technical Editor

Ankita Shashi

Technical Editors

Dipika Gaonkar

Pramod Kumavat

Tarunveer Shetty

Project Coordinator

Anugya Khurana

Proofreaders

Simran Bhogal

Maria Gould

Indexers

Hemangini Bari

Tejal R. Soni

Graphics

Ronak Dhruv

Abhinash Sahu

Production Coordinator

Arvindkumar Gupta

Cover Work

Arvindkumar Gupta

About the Authors

Scott Cranton is an open source software contributor and evangelist. He has been working with Apache Camel since the release of version 1.5 almost 5 years ago, and has over 20 years of commercial experience in middleware software as a developer, architect, and consultant. During his time at FuseSource, and now Red Hat, he has worked closely with many core committers for Apache Camel, ActiveMQ, ServiceMix, Karaf, and CXF. He has also helped many companies successfully create and deploy large and complex integration and messaging systems using Camel and other open source projects.

He divides his professional time between hacking code, delivering webinars on using Camel and open source, and helping companies to learn how to use Camel to solve their integration problems.

I want to thank my amazing wife, Mary Elizabeth, for putting up with me these many years, and always answering the phone when I'd call late at night from the office while I've got a compile going. This book would not have been possible without her always being there for me no matter what. To my three wonderful children, Gilbert, Eliza, and Lucy, who always make me smile especially during crazy weekend writing sessions when they'd want me to take a break, … but Dad, it's the weekend… I love you all!

Jakub Korab is a consulting software engineer specializing in integration and messaging. With a formal background in software engineering and distributed systems, in the 14 years that he has worked in software across the telecoms, financial services, and banking industries, he naturally gravitated from web development towards systems integration. When he discovered Apache Camel, it became apparent to him how much time and effort it could have saved him in the past compared to writing bespoke integration code, and he has not looked back since.

Over the years, working as a consultant, he has helped dozens of clients build scalable, fault-tolerant, and performant systems integrations. He currently runs his own specialist consultancy, Ameliant, which focuses on systems integration and messaging using a stack of integration products from the Apache Software Foundation, of which Camel is a corner stone.

When not gluing systems together, you will find him spending time with his young family, and far too infrequently kitesurfing or skiing—neither of which he gets much chance to do in his adopted home, London.

The writing of this book has taken place against the background of starting a new company, a huge amount of work travel, a quickly growing baby, house move, and hundreds of little distractions that get in the way of sitting down in what is left of the day to put pen to paper. It could never have happened without the love, support, and understanding of my wife, 
Anne-Marie. It has been a team effort. Thank you.

Also to my little girl, Alex, for helping me keep it all in perspective.

Acknowledgments

We would like to thank the Apache Camel community for being so awesome. Very welcoming and vibrant, they have always been willing to answer questions and accept code contributions. Big thanks to the many, many Camel community members.

We would like to also thank all the reviewers, editors, and unseen people in the background. Getting a book out is a substantial project of which the writing is only one part, and it could not have happened without you.

Thanks to Rob Davies, Claus Ibsen, and the whole Fuse team for showing us how much fun open source is by always being there to answer questions, geek out about some new project, and drinking beer on those rare occasions when we could get together. It's been a real pleasure working with you all, and we hope it continues for a long time.

Special thanks to James Strachan for being so James; your infectious enthusiasm and love of writing code makes you truly inspirational to all those who meet you. We hope to keep playing with you on any and all of your latest new projects.

About the Reviewers

Bilgin Ibryam is a software engineer with master's degree in Computer Science and he currently works for Red Hat as Middleware Consultant. His passions include distributed applications, message-oriented middleware, and application integration. He is the author of a book about Apache Camel and Enterprise Integration Patterns called Instant Apache Camel Message Routing. He is also heavily involved with open source and is a committer to Apache OFBiz, and other Apache Camel projects. In his spare time, he enjoys contributing to open source projects and blogging at http://ofbizian.com. He can be contacted via his Twitter handle @bibryam.

Claus Ibsen has worked as a software engineer, architect, and consultant for more than 15 years. For the past five years he has been working full time as an open source developer at FuseSource and Red Hat in the middleware team, on the Fuse products. He has worked extensively on the Apache Camel project for over six years, being the top contributor, and has given talks at conferences about integration using Apache Camel.

He is author of Camel in Action, Manning Publications, 2011.

I would like to congratulate Scott and Jakub for stepping up and writing this fantastic book. It is great to see the Camel community grow from strength to strength. This book will help new users to become more successful with riding Apache Camel, and experienced users can find valuable information from many of its recipes.

Christian Posta is based in Phoenix, AZ, and is a Principal Consultant and Architect. He specializes in messaging-based enterprise integrations with high scalability and throughput demands. He has been in development for over 10 years covering a wide range of domains; from embedded systems to UI and UX design and lots of integration in between. He's passionate about software development, loves solving tough technical problems, and enjoys learning new languages and programing paradigms. His favorite languages are Python and Scala, but he spends a lot of time writing Java too. He is a committer on Apache Camel, Apache ActiveMQ, and Apache Apollo projects as well as PMC on ActiveMQ. He blogs about Camel, ActiveMQ, and integration at http://www.christianposta.com/blog as well as tweets about interesting technology on his Twitter handle @christianposta.

Thanks to Scott and Jakub for asking me to review the book. I think this work is the ultimate complement to the other Camel book. The material cuts right to the chase and shows you how to accomplish otherwise challenging integrations using this wonderful library.

Phil Wilkins has spent nearly 25 years in the software industry working with both multinationals and software startups. He started out as a developer and has worked his way up through technical and development management roles. The last 12 years have been primarily in Java-based environments. He now works as an enterprise technical architect within the IT group for a global optical healthcare manufacturer and retailer.

Outside of his work commitments, he has contributed his technical capabilities to supporting others in a wide range of activities from the development of community websites to providing input and support to people authoring books, and developing software ideas and businesses.

When not immersed in work and technology, he spends his free time pursuing his passion for music and time with his wife and two boys.

I'd like to take this opportunity to thank my wife Catherine and our two sons Christopher and Aaron for their tolerance for the innumerable hours that I've spent in front of a computer contributing to both my employer and the many other IT related activities that I've supported over the years.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Instant Updates on New Packt Books

Get notified! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page.

Preface

Apache Camel is a Java framework for building system integrations.

Why, you may well ask, does anyone need such a framework? System integration is pretty much a solved problem. After all, we have been connecting various frontends to web services, message brokers, and databases for years! Surely this is a well-understood domain that requires no further abstractions.

Not quite.

Apache Camel, since its release in 2007, has disrupted the integration market much like the Spring Framework disrupted the Java EE market back in 2003. Camel enables a new way of doing, and thinking about, system integrations that results in much cleaner, easier to understand code, which in turn results in less work, less bugs, and easier maintenance. These are big claims, and to validate them you only need to look at the large and active Apache Camel community, the growing number of commercial integration products based on Camel, and the talks on Camel that appear at most middleware developer conferences to feel that there is a good buzz around Camel, and for very good reason.

This book is targeted at readers who already have some familiarity with Camel, and are looking for tips on how Camel may be able to better help them solve more complex integration challenges. This book is structured as a series of over 100 how-to recipes, including step-by-step instructions on using Camel to solve common integration tasks. Each recipe includes a brief explanation of what Camel is doing internally, and references on where to find more information for those who want to dig deeper.

This book may not be a good introduction/beginner book about Camel, though if you have familiarity with other integration technologies, and learn well by doing, you may find this book's recipe approach helpful. This book does not spend a lot of time explaining Camel concepts in great depth.

For readers looking for more conceptual coverage of Camel (with lots of code examples), we would recommend reading the excellent book Camel in Action by Claus Ibsen and Jonathan Anstey, published by Manning. For a more introductory guide, look at Instant Apache Camel Message Routing by Bilgin Ibryam, published by Packt Publishing. The Apache Camel website (http://camel.apache.org) is the authoritative site on Camel, with a long list of articles and documentation that will help you on your journey of using Camel.

What is Camel?

This section provides a quick overview of what Camel is, and why it was created. Its goal is to help remind the reader of the core concepts used within Camel, and to help the reader understand how the authors define those concepts. It is not intended as a comprehensive introduction to Camel. Hopefully, it will act as a quick reference for Camel concepts as you use the various recipes contained within this book.

Integrating systems is hard work. It is hard because the developers doing the integration work must understand and how the endpoint systems expose themselves to external systems, how to transform and route the data records (messages) from each of the systems. They must also have a working knowledge of the ever growing number of technologies used in transporting, routing, and manipulating those messages. What makes it more challenging is that the systems you are integrating with were probably written by different teams of developers, at different times, and are probably still changing even while you are trying to integrate them. This is equivalent to connecting two cars while they are driving down the highway.

Traditional system integrations, in the way that we have built them in the past decades, require a lot of code to be created that has absolutely nothing to do with the higher-level integration problem trying to be solved. The vast majority of this is boilerplate code dealing with common, repetitive tasks of setting up and tearing down libraries for the messaging transports and processing technologies such as filesystems, SOAP, JMS, JDBC, socket-level I/O, XSLT, templating libraries, among others. These mechanical concerns are repeated over and over again in every single integration project's code base.

In early 2000, there were many people researching and cataloging software patterns within many of these projects, and this resulted in the excellent book Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf, published by Addison Wesley. This catalog of integration patterns, EIPs for short, can be viewed at http://www.enterpriseintegrationpatterns.com. These patterns include classics such as the Content Based Router, Splitter, and Filter. The EIP book also introduces a model of how data moves from one system to another that is independent of the technology doing the work. These named concepts have become a common language for all integration architects and developers making it easier to express what an integration should do without getting lost in how to implement that integration.

Camel embraces these EIP concepts as core constructs within its framework, providing an executable version of those concepts that are independent of the mechanics of the underlying technology actually doing the work. Camel adds in abstractions such as Endpoint URIs (Uniform Resource Identifier) that work with Components (Endpoint factories), which allows developers to specify the desired technology to be used in connecting to an endpoint system without getting lost in the boilerplate code required to use that technology. Camel provides an integration, domain-specific language (DSL) for defining integration logic that is adapted to many programming languages (Java, Groovy, Scala, and so on) and frameworks (Spring, OSGi Blueprint, and so on), so that the developer can write code that is an English-like expression of the integration using EIP concepts. For example:

consume from some endpoint,

split the messages

  based on an expression, and

  send those split messages to some other endpoint

Let us look at a concrete example to show you how to use Camel.

Imagine that your boss comes to you, asking you to solve an integration problem for your project. You are asked to: poll a specific directory for new XML files every minute, split those XML files that contain many repeating elements into individual records/messages (think line items), and send each of those individual records to a JMS queue for processing by another system. Oh, and make sure that the code can retry if it hits any issues. Also, it is likely the systems will change shortly, so make sure the code is flexible enough to handle changes, but we do not know what those changes might look like. Sound familiar?

Before Camel, you would be looking at writing hundreds of lines of code, searching the Internet for code snippets of how best to do reliable directory polling, parsing XML, using XPath libraries to help you split those XML files, setting up a JMS connection, and so forth. Camel hides all of that routine complexity into well-tested components so you just need to specify your integration problem as per the following example using the Spring XML DSL:

  file://someDirectory?delay=60000/>

    /xpath/to/record

    jms:queue:myProcessingQueue/>

Wow! We still remember when we first saw some Camel code, and were taken aback by how such a small amount of code could be so incredibly expressive.

This Camel example shows a Route, a definition (recipe) of a graph of channels to message processors, that says: consume files from someDirectory every 60,000 milliseconds; split that data based on an XPath expression; and send the resulting messages to a JMS queue named myProcessingQueue. That is exactly the problem we were asked to solve, and the Camel code effectively says just that. This not only makes it easy to create integration code, it makes it easy