Changing Views on Integration (AUSOUG Webinar Series, May 2020)

Changing Views
on Integration
from Service Bus
to API Gateway,
Serverless & iPaaS
Changing Views on Integration - AUSOUG Webinar Series
AUSOUG Webinar – 6th May 2020
Lucas Jellema, CTO & Architect AMIS | Conclusion

Agenda
What is integration?
Integration challenges
“Traditional” integration approach
Recents insights and developments
Integration Patterns
Out of the Box
Conclusion

System
A
System
B
XX
Simple definition: enabling systems to talk to each other
9

System
A
System
B
Ensure that data is where it is needed when it is needed in the form in which it is needed
10

Integration Sample Scenarios
• Orders in Web Shop need to be distributed to and processed by Order Entry,
Order Picking (and Shipping) and Invoicing
• Measurements from Sensors need to be cleansed, filtered, aggregated
and routed to Monitoring application
• Social media team needs to have access in their content management system to
all posts across all platforms mentioning the company
• Our customers need to be able to track and trace their requests as they are
processed throughout our organization
• Changes in address details for patients need to be applied
in all systems that hold patient records
• The website needs to show to our customers the actual listing of products, reviews,
prices and stock information
• The loyalty points associated with an in-shop purchase have to be processed
instantaneously to allow them to be used by the customer for their next purchase
Data
Access

Integration Challenges
● Monolithic / aging / closed systems
● Disparate data models
● Architectural complexity
● Flexibility and productivity
● Costs substantial and Business Value not so clear
● Heterogeneous ecosystem
● Technical debt
● Bloated projects / overhead
● Visibility / error handling
● Production deployment
● Non-functional Quality of Service
● availability, performance,
scalability, security
13

14
Mobile API, ETL,
IoT, CQRS, SaaS
Enablement, …

“Traditional” integration approach
15

ESB as basic architecture element
16
global internal specific
External Interfaces – Proxy Services
Integration Logic
(Validate Enrich Tranform Route Operate)
Internal Interfaces – Business Interfaces
EnterpriseServiceBus

Application Server
Enterprise Service Bus
Backend
Database
Backend
Alternative
Datastore
Backend
custom App
Backend
Standard App
Some
External
Service
UI
(Web Portal, Mobile)
B2B PartnerService
SaaS
Load balance, En/De-cryption,
VETRO, Monitoring, Analytics,
Throttling, Caching, Authentication,
Authorization, Orchestration,
Circuit Breaking, Exception
Handling & Retries, Logging,
Technology & Application Adapters
& Connectors, Workflow
coordination, Job Scheduling & Mgt
DWH??

Limited and expensive Scalability
1
8
Loadbalancer
ESB 1 ESB 2 ESB 3

Oracle SOA Cloud
Service
Fixed Technologies Set
<XML />
{
„dialect“:“JSON“
}
Caught in a vendor and/or technology Trap
1
9

Limited agility
• Hard to implement Continuous Integration / Continuous Delivery
• Complex infrastructure
• Missing automated testing capabilities
• Single Point Of …
• Monolithic deployment approach
• Monolithic management (scalability, load balance etc.)
• Expert knowledge needed
• Implementation
• Deployment
• Error analysis and resolution
• TCO
http://geek-and-poke.com/

The world is getting more complex interesting!
22

New insights, new context, eroded dogmas
23
Increased scale – because of real -time and web scale demands
(introduced by IoT and a further growth of the use of Apps )
Need for speed - real time insights from fast data streams
No/less strict delineation between Enterprise (message based) Integration and ETL/Data Integration
Tighter security requirements (GDPR to name one)
Faster change cycles (Agile, DevOps)
24/7 Operations (no down time or batch | release | maintenance windows)
Scarcity of talent/developers
Is a canonical model/language such a good idea? What about ‘domain/context mapping’
and ‘anti corruption layer’ from domain driven design

New insights, new context, eroded dogmas (2)
24
We have learned to ask: How fresh should data be? How crucial is ACID and instant consistency?
Realization that reusability (service & data) is not the holy grail (one of the fallacies of SOA: SPoF)
Increased usage of (several) standard applications (primarily SaaS)
The advent of cloud (necessity to support hybrid scenarios - across on-premises and cloud
as well as across multi-vendor cloud)
New architecture patterns such as domain driven design, microservices, CQRS,
event sourcing, “dumb pipes, smart end points”
Advances in technology, containers, serverless, REST, API technology – JSON (vs XML), REST (vs SOAP),
containers, serverless, “embedded HTTP server”, industry standards like Apache Kafka
TCO of Middleware Platforms (run/operate & develop/evolve)
Strategic roadmap of middleware vendors: Cloud, Cloud Native & Microservices

Two dimensions of integration
26

● Synchronous (End User) ”integration”
● API Gateway as runtime component for
○ API policy enforcement
○ Request routing
● (Near) Real-time, synchronous interactions
● APIs born from business requirements
○ Who is using the API?
○ How is the API being used?
○ Not so much focus on reusable, generic interfaces
● Use of lightweight, standard protocols
Vertical integration
27
Customer API
CRM Database
Client
Application
VerticalIntegration
API
Gateway

Multi Purpose API
API Gateway
Backend
Database
Backend
Alternative
Datastore
Backend
custom App
Backend
Standard
App
UI B2B Partner
(Internal)
Microservice
Some
External
Service
Single Purpose API
Multi Purpose API
Load balance, Routing,
Monitoring, Analytics,
Throttling, Authentication,
Authorization, Circuit
breaking, controlled
release ( A/B Testing,
Canary Release )…

Evolution of ‘vertical integration flow’ implementation
30
Kubernetes
Implemen
tation
Implemen
tation
API Gateway
Implemen
tation
App DB
Implemen
tation
API Gateway
App DB
Application Server
Enterprise
Service Bus
App DB
Serverless
Functions
Implemen
tation
API Gateway
App DB
λ
VerticalIntegration

Real Life Case of Vertical Integration:
Loyalty Card Scan
Stores (PoS)
Siebel DB
OUR DB
OSB
Web
Logic
Camp
aigner
RETEK
PLQ
LINK
JMS
MFT
Siebel DB
Data
Warehouse
Loyalty X
Siebel
ODI
Golden
Gate
Retail Chain - on
premises infrastructure
Primary Region UK
(secondary) Disaster Failover
Region Germany
OSBOSB
100sK Card Scans per day;
card scan is transaction;
response time < 800 ms
OSB

Loyalty Card Scan
Stores (PoS)
Siebel DB
OUR DB
OSB
Web
Logic
Camp
aigner
RETEK
PLQ
LINK
JMS
MFT
Siebel DB
Data
Warehouse
Loyalty X
Siebel
ODI
Golden
Gate
Retail Chain - on
Primary Region UK
Region Germany
API Gateway
Card
Scan

Loyalty Card Scan
Stores (PoS)
Siebel DB
OUR DB
OSB
Web
Logic
Camp
aigner
RETEK
PLQ
LINK
JMS
MFT
Siebel DB
Data
Warehouse
Loyalty X
Siebel
ODI
Golden
Gate
Retail Chain - on
Primary Region UK
Region Germany
API Gateway
Kubernetes
Card
Scan
Car
d
Sca
n
Car
d
Sca
n
Car
d
Sca
n
Car
d
Sca
n

Horizontal integration
● Classic System2System integration
● Eventual
● Background, Asynchronous interaction patterns
● Batch processing, increasingly Straight Through Processing (STP)
● High volume, complex scheduling and orchestration
● Technical interfaces, heavy-duty, proprietary protocols
● Evolving Insights: relaxed freshness and consistency demands, data
replication
● Changing context: cloud & hybrid, “dumb pipes, smart endpoints”, Standard
Apps & SaaS evolution, agile (change, deploy, scale, pay)
System
A
System
B

Horizontal integration
3
5
Backend
Database
Backend
custom App
Backend
Standard App
SaaS
(periodic) data synchronization, replication
Fulfill synchronous request or command
IoT
Event driven, asynchronous notification
Execute workflow (across multiple systems & people?)
Technology & Application Adapters & Connectors,
Workflow coordination, Event Bus, Exception
Handling, Job Scheduling & Mgt, Transformation,
Routing, Monitoring, Streaming Analysis,
Authentication,
API
Gateway
µ

Challenges with Horizontal Integration
● Handle requirements for speed, consistency, volume & [global] distribution
● Detect Change
● Extract Data
● Publish Change
● Transport
○ deal with limited connectivity,
temporary unavailability
● Convert/Transform
● Apply
● Monitor
● Handle exceptions
● Enforce constraints & authorization
Source
System
Destination
Detect changes
Extract Data
Transport Data
Convert Data
Apply Data
Spot and Handle
Errors
Governance &
Evolution
Enforce Constraints &
Authorization
Decoupling Queue

(Cloud) Adapters & iPaaS platforms
To connect, detect & extract (from source) and connect & apply (to sink)
Online Meetup Introducing Apache Kafka - Part Two of Three
Debezium

Integration and Integration?
Evolved from Oracle Service
Bus and SOA Suite
Evolved from Oracle Data
Integrator (ODI) and GoldenGate

Data Pipeline is special type of Horizontal Integration
A.S. Watson - Integration Architecture Future
Raw/
Operational
Data Sources
Data
Warehouse
Data Lake
Ingestion
Data
Wrangling
“Data Mart”
single purpose
data set
3rd party data
sources
algorithms
Machine
Learning Model
Dashboard, Notification &
Alert Report, Visualization,
Gamification, Data Explorer
Business insight: what,
when, who and why
Machine
Learning
Deploy &
Leverage
Business
Intelligence
Master Data Management, Meta-Data Management, Data Governance
Data Access, Authorization, Privacy
“Data Mart”
single purpose
data set
“Data Mart”
single purpose
data set
“Data Mart”
single purpose
data set
Internal and
Public APIs
Data
Services
Hot Flow of Real Time Streaming Data

The Third Dimension
Both Horizontal and Vertical Integration
are pre-determined
• end-to-end chains from specified
source(s) to a designated target
• triggered by synchronous requests and
time events (batch)

Events aka Messages
Producers
Consumers
Event Hub
Independent intermediary
Robust, Scalable, Fast,
History Retention
Open
Online Meetup Introducing Apache Kafka - Part One of Three
Multi: multiple publishers to same topic,
multiple subscribers on same topic (or none);
Decoupled: publishers to not know about
consumers and vice versa; consumers can
receive messages from when they were off line

Event-Driven Integration
42
Instead of the API definition and
runtime endpoint, Event Consumers
need the Message definition, the
Topic name and the endpoint of the
Event Store (aka Event Hub)
Optional: forwarding endpoint of
consumer when Pub/Sub with Push

Out of the box
Ahead of Time Integration & CQRS
Event Sourcing
Collective Data Set

Utility Company looking for Performance, Scalability and
Availability in their core data sets
CRM
Meter Readings
batch
Invoicing
Billing
Marketing
Campaigns

Enterprise
Service
Bus
CRM
Meter Readings
batch
Invoicing
Billing
Marketing
Campaigns
Customer
Portal

Enterprise
Service
Bus
CRM
Meter Readings
batch
Invoicing
Billing
Marketing
Campaigns
Customer
Portal
Load on Core Systems &
effect on Performance
Availability Core Systems,
Sizing & Cost &
Complexity ESB

CRM
Meter Readings
Invoicing
Billing
Marketing
Campaigns
CRM
Cache DB
Meter
Readings
Cache DB
Customer
Portal

Practical Challenges
• Publish all [fine grained]
change events from
sources
• Event schema evolution
• Lag in Cache Databases
• “eventual consistency”
• Differences between sources
and cache databases
• Initial creation of Cache Databases
• General Scale and Incidental event storm
• Business and Mission Critical availability
• Data Authorization
• Logic enforced in source (applications)
• GDPR
CRM
Meter Readings
Invoicing
Billing
Marketing
Campaigns
CRM Cache
DB
Meter Readings
Cache DB
Customer
Portal

CQRS => Specialized Query Stores
Cater for different (read) Data Usage Scenarios
Different in:
● Why
● What – grain, aggregation, breadth, type of and variation in queries
● How – format, language, accuracy and freshness, consistency
● When – frequency, time of day | week | month, 24/7
● Where – latency, band width, off line
● Who – scale, different roles & user groups
5
2

CQRS => Optimized Data Set for Special Use Case
● Shape & Format
● Right Grained (aggregation, consolidation)
● Filtered
● Enriched (from multiple sources)
● Location
● Time (refreshed at the right time)
● Search Options
● BASE (eventually – when? – consistent)
● Authorization
● GDPR
5
3
C Q
BFF API
Frontend

CQRS: A small step for mankind but a giant leap for [IT] architects

Typical CQRS Architecture
Event Platform
Source systems
Poll
Consume
Detect &
Retrieve
Change
Change Reporters Distributed Change Event Hub Change Projectors Destination systems
Transform
& Apply
(re)Calculate
& Record
Convert,
Translate,
StoreEvent Schema
Governance
Monitoring & Error Handling

Utility Company: Rebuild Cache Database
CRM
Meter Readings
CRM
Cache DB
Meter
Readings
Cache DB

Utility Company: Rebuild Cache Database
From Event Store
CRM
Cache DB
Meter
Readings
Cache DB

Event Sourcing Driving CQRS
Events Event Store
Current State
accountId:
123
amount: 10
Owner: Jane Doe

Event Sourcing Driving CQRS
Events Event Store
Current State
Other State
Aggregate

Event Sourcing
• Event Store is immutable – append-only log of domain state transitions
• It is the truth about data – everything else is derived
• Replay events
• to (re)construct a representation of the current state (aggregate)
• up to a specific time to recreate moments in time
• in Test environment to investigate an issue
• on a remote location to create mirror & share state across boundaries
• produce a fine grained audit trail
• Challenges
• Time required to reconstruct state
• Grain of aggregates / definition of domain events

Data Sharing Ecosystems
• Multiple independent
organizations
• Mutually benefit from
having access to
the same data
at the same time
• Share data & Have access to data with minimum of overhead and with equality among the
ecosystem partners
• Partners are on different network domains, physical locations, technology stacks, …
• New Partners can join
• Partners can leave
• Consensus on what constitutes correct data
Pragmatic CQRS - Digital Xchange

Collective Dataset
Company
A
Company
C
Company
D
Company
E
Company
F
Collective
Dataset
Company
B

Collective Dataset
Company
A
Company
C
Company
D
Company
E
Company
F Central
Database
Enterprise
Service Bus
Canonical SOAP WS/
REST API for Sending
Inserts/Updates/Deletes
and for Requesting
current data
Just In Time Integration
Company
B

Collective Dataset = Collective Data Change Events
Company
A
Company
C
Company
D
Company
E
Company
F
Central
Event
Platform Interfaces for Publishing
and Consuming Events
Note: events can be
consumed
asynchronously
Ahead of Time Integration
Company
B

Collective Dataset => Collective Data Store
Company
A
Company
C
Company
D
Company
E
Company
F
Ahead of Time
Integration
Company
B

Key Takeaways
• Integration is as relevant as ever – probably more so
• API, microservice, Data Pipeline, Streaming Data (fast & furious), Hybrid
Cloud (& on prem), IoT (physical and virtual), B2B, Big Data, SaaS
• The one-size-fits-all approach based on Enterprise Service Bus is replaced
• Three main integration styles (Vertical, Horizontal, Event-Driven)
• With fitting Patterns and Tools (Dumb Pipes/Smart Endpoints, microservices,
API Gateway, Serverless, Message Queue, Adapters & Connectors, … )
• Leveraging cloud technology
• Out of the Box
• Ahead of Time Integration (CQRS), Collective Data Set, Event Sourcing
• Asynchronous is the preferred communication style
• Decoupling is crucial for flexibility, availability (not brittle), scalability,
distributability (“social distancing” for systems and services)

Thank you for your attention
I hope this was useful
lucas.jellema@amis.nl | technology.amis.nl | @lucasjellema | lucas-jellema

Changing Views on Integration (AUSOUG Webinar Series, May 2020)

Recommended

More Related Content

What's hot (20)

Similar to Changing Views on Integration (AUSOUG Webinar Series, May 2020) (20)

More from Lucas Jellema (20)

Recently uploaded (20)

Changing Views on Integration (AUSOUG Webinar Series, May 2020)