Introducing Apache Geode and Spring Data GemFire

2© 2014 Pivotal Software, Inc. All rights reserved. 2© 2014 Pivotal Software, Inc. All rights reserved.
Introducing Apache Geode
with Spring Data GemFire
John Blum / Luke Shannon
24 June 2015

3© 2014 Pivotal Software, Inc. All rights reserved.
Agenda
 What is Apache Geode?
 Apache Geode Technical Specification
 High-level Implementation Details
 Spring Data GemFire
 Q&A

Pivotal GemFire
What is Apache Geode?
 In a nutshell…
– Open Source core of Pivotal GemFire
– Apache Incubator
▪ https://wiki.apache.org/incubator/GeodeProposal

One definition is…
A distributed, in-memory compute and data
management platform that elastically scales
to achieve high-throughput, low-latency
access to big, fast data in order to power
business critical, analytical applications in
real-time.
Nodes
Ops /
SecLinear scalability
Elastic capacity +/-
Latency optimized
data distribution

 Technically…
– Distributed, in-memory database (IMDG) with ACID properties
– Transactional Key/Value Store (Cache)
– Glorified-version of ConcurrentHashMap
– Message Bus with guaranteed message delivery
▪ Reliable, asynchronous event notifications

What Apache Geode Does?
 Enables new or existing Java applications to operate at cloud-scale
with predictable, low-latencies and be highly-available without
sacrificing consistency.
 Applications can transact and analyze Big Data in real-time to garner
insights and achieve meaningful and impactful business results.
Ap
p
Ap
p
Ap
p
Ap
p

Apache Geode Characteristics
 In-Memory
 Distributed
 Consistent
 Highly-Available / Fault Tolerant
 High-Throughput / Low, Predictable Latency
 High Scalability
 Data-aware Parallel Compute

How are Apache Geode’s features implemented?

Memory
 Geode stores data in memory…
– JVM Heap
– Off-Heap (Native Memory)
 Geode (optionally) stores data to disk
– Persistence + Overflow

Management
Memory
 Eviction (LRU)
 Expiration (Region & Entry TTL/TTI)
 Automatic resource management (critical & eviction Heap %)
 Region Compression (Snappy)
 JVM GC Tuning

IMDG
 An In-Memory Data Grid (IMDG), such as Apache Geode…
– Pools system resources across multiple nodes in a cluster to
manage both application state and behavior.
– Memory, CPU, Network and (optional) Disk

High-Throughput / Low-Latency
 Geode uses concurrent, in-memory data structures with a highly
optimized distribution infrastructure to minimize context switching and
contention.
– synchronous/asynchronous replication for high read throughput
– partitioning for both high read and write throughput

Shared-Nothing Disk Persistence Architecture
 Manages data in local disk files on each node in the cluster
independently from other nodes
– Members are self-sufficient; Members do not share memory or disk;
– Data/Transactions appended to Oplog files
– Compaction
 Design goals (all about In-Memory)…
– Minimize IO operations (Write-Behind vs. Write-Through)
– Minimize locking
– Minimize serialization

REPLICATE & PARTITION Regions
High Availability
 Data is partitioned across nodes in the cluster and distributed uniformly
– Rebalanced as nodes join/leave
– Redundant copies

Persistence – Shared Nothing
Server 3Server 2
Bucket 3
*primary*
Bucket 1
secondary
Bucket 3
secondary
Bucket 2
*primary*
Server 1
Bucket 2
secondary
Bucket 1
*primary*

Server 1 Server 3Server 2
Bucket 2
secondary
Bucket 1
*primary*
Bucket 3
*primary*
Bucket 1
secondary
Bucket 3
secondary
Bucket 2
*primary*
What if a server
crashes?

Server 1 Server 3Server 2
Bucket 2
secondary
Bucket 1
*primary*
Bucket 3
*primary*
Bucket 1
*primary*
Bucket 3
secondary
Bucket 2
*primary*
Bucket 2
secondary
Primary
Failover
Restore
Redundancy
Bucket 1
secondary

Network Partitions

Partition Regions
Configurable Consistency
 Redundant Copies
– 0 to 3
 Colocation
 Total Number of Buckets
– 113 (Default)
 Local Max Memory
– 90% of Heap
 Total Max Memory
– All Members, Both Primary/Secondary
Copies

REPLICATE vs. PARTITION
Configurable Consistency
 Partition Regions
– Updates routed through primary bucket
– Holds lock on key when distributing updates
 Replicate Regions
– Possible conflicts due to out-of-order updates
– Conflict checking (--concurrency-checks-enabled)
ReplicatedPartitioned Disabled

Apache Geode Topologies

Peer-to-Peer (P2P)
 Configuration with the Cache embedded in the application process
– Close proximity of application logic and data

Client/Server
 Single-Hop Data Access
 Continuously load balanced between
servers to get the best response-times
`
 Automatic fail-over when servers crash or
become unresponsive

WAN
 Site-to-Site Replication for DR

Use Cases
Apache Geode
 Persistent OLTP/OLAP
– System of Record (SOR)
 Caching with JSR-107 (JCache) or Spring Cache Abstraction
 HTTP Session State Management
– Caching and Replication
 Distributed L2 Caching for Hibernate
 Memcached Server (Gemcached)

Additional Features
Apache Geode
 Functions
– Application logic is distributed, routed and run in parallel with close proximity to the data required by the
application.
 Serialization (PDX)
 Delta-propagation
– difference of new/old value
 JTA-compliant Transactions
 Security
– SSL, Auth

Roadmap
Apache Geode
 HDFS persistence
 Off-Heap storage
 Lucene search/indexing
 Spark integration
 Cloud Foundry service

Interface
Apache Geode
 Language Support
– Native: Java, C/C++, C#
– Non-native: JavaScript, Python,
Ruby, Scala, etc
 API
– Java HashMap (Region)
– OQL / Function
– Memcache client
– REST API
– Spring Data GemFire
 Data Structure
– User-defined Classes / POJOs
– Portable Data Exchange (PDX)
– Documents (JSON)
 Powerful Data Access
– Object Query Language (OQL)
– Data-aware, Scatter-Gather
Functions
– Publish/Scribe framework for RI/CQs
with reliable, async event queues

Spring Data GemFire
http://projects.spring.io/spring-data-gemfire/

So… Why Spring?

“Simple things should be simple;
complex things should be possible”
– Alan Kay

Spring Data GemFire (SDG)
Consistently applies Spring's powerful, non-invasive programming model
to simplify configuration and development with Apache Geode.
Spring Ecosystem Integration…
– Spring Cache Abstraction / Transaction Management
– Spring Data Commons + REST
– Spring Integration (Inbound/Outbound Channel Adapters)
– Spring XD (Input Source and Sinks)

Apache Geode combined with Spring Data GemFire
can be used as JSR-107 (JCache) caching provider
+ +

GRAILS
Full-stack, Web
XD
Stream, Taps, Jobs
BOOT
Bootable, Minimal, Ops-Ready
Big,
Fast,
Flexible
Data Web,
Integration,
Batch
WEB
Controllers, REST,
WebSocket
INTEGRATION
Channels, Adapters,
Filters, Transformers
BATCH
Jobs, Steps,
Readers, Writers
BIG DATA
Ingestion, Export,
Orchestration, Hadoop
DATA
NON-RELATIONALRELATIONAL
CORE
GROOVYFRAMEWORK SECURITY REACTOR

Use Cases / Configurations
Spring Data GemFire
 Apache Geode Configuration / Bootstrapping
– Replacement for cache.xml; Can be used with Cluster Configuration
 Application Peer (Cache)
 Application Client (ClientCache)

Examples…
SDG-based Configuration

with Spring
Bootstrapping Apache Geode
@SpringBootApplication
@ImportResource("/spring-data-gemfire-cache.xml")
@EnableGemfireRepositories
@EnableTransactionManagement
public class SampleDataGemFireApplication {
public static void main(String[] args) {
SpringApplication.run(SampleDataGemFireApplication.class, args);
}
}
Or…
gfsh>start server –name=Example … --spring-xml-location=“<classpath-to-spring-application-
context.xml>”

Basic (Region) Data Access
Data Access with Spring
@Repository
public class CustomerDao {
@Resource(name = “Example”)
private Region<?, ?> example;
…
}
Example…

GemfireTemplate
Data Access with Spring Data GemFire
 Advantages…
– Simple, Convenient Data Access (CRUD, OQL, Function)
– Protects developer from GemFire/Geode API changes
– Exception Translation into Spring DAO Exception Hierarchy
– Transaction Management
<bean name=“customersTemplate” class=“org.springframework.data.gemfire.GemfireTemplate”
p:region-ref=“Customers”>

GemfireRepository
Data Access with Repositories
public interface CustomerRepository extends GemfireRepository<Customer, Long> {
}
// Spring context.xml
<gfe-data:repositories base-package="org.example.app.customer.dao.repo">
<repo:include-filter type="assignable"
expression=”org.example.app.customer.dao.repo.CustomerRepository"/>
</gfe-data:repositories>
Example…

Implementation
Annotation-based Function Implementation/Execution
package org.example.app.functions;
import …
@Component
public class ExampleFunctions {
@GemfireFunction
public Customer update(…) {
…
}
…
}
Example…
(Optional)…
<bean class=“org.example….ExampleFunctions”/>

Execution
package org.example.app.function.executions;
import …
@OnRegion(“Example”)
public interface ExampleFunctionExecution {
Customer update(…);
}
And…
<gfe:annotation-driven/>
<gfe-data:function-executions base-package="org.example.app.function.executions">

Execution
 Additionally…
– @OnServer(s), @OnMember(s)
 Examples…

JSR-107 (JCache), Spring Cache Abstraction
Geode Caching Provider
 With @Cacheable
 And @CachePut, @CacheEvict(allEntries=[true|false]),
@CacheConfig
– http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#cache-jsr-107
 Or with JSR-107 using…
– @CacheResult, @CachePut, @CacheRemove, @CacheRemoveAll,
@CacheDefaults

Demo
By Luke Shannon

On the horizon…

Roadmap
Spring Data GemFire
 Support for GemFire 8.2 & 9.0
– HDFS, Off-Heap, Apache Lucene, Spark
 Support for pre-defined data sets using Snapshot Service
– SGG-408
 Expiration annotation support with CustomExpiry
– SGF-404
 Spring Session Adapter for GemFire/Geode
– SGF-373

Roadmap Continued…
Spring Data GemFire
 Repository Query method annotation extensions...
– SGF-392
public interface ExampleRepository implements GemfireRepository<Customer, Long> {
@Trace
@Limit(20)
@Import(“org.example.app.domain.SomeOtherType”)
List<Customer> findByLastname(String lastName);
}

Get Involved…

Apache Geode
Contribute
 In addition to developing Spring applications with Apache Geode, now is
a perfect time to get involved in the Apache Geode community and
contribute growing Apache project from the ground-up.
– Ways to participate…
http://geode.incubator.apache.org/contribute/

Finally…

Washington, DC – September 14 – 17, 2015
SpringOne2GX
 Early Bird price registration ends August 14th, Register Now!
– https://2015.event.springone2gx.com/register
 Building High-Scalable Spring Applications with Distributed In-Memory
Data Grids.
– Luke Shannon & John Blum
– Tuesday, September 15th, 2015
– https://2015.event.springone2gx.com/schedule/sessions/building_highly_scalable_spring_applications_with_in_m
emory_distributed_data_grids.html

References
 http://geode.incubator.apache.org/
 http://geode.incubator.apache.org/contribute/
 https://github.com/project-geode/docs/wiki
 https://github.com/apache/incubator-geode
 https://pivotal.io/big-data/pivotal-gemfire
 http://gemfire.docs.pivotal.io/latest/userguide/index.html#getting_started/topics/gemfire_overview.html
 http://gemfire.docs.pivotal.io/latest/javadocs/japi/index.html
 http://stackoverflow.com/questions/tagged/geode+or+gemfire
 http://projects.spring.io/spring-data-gemfire/
 http://docs.spring.io/spring-data-gemfire/docs/1.6.0.RELEASE/reference/html/
 http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/
 http://docs.spring.io/spring-data-gemfire/docs/current/api/
 https://github.com/spring-projects/spring-data-gemfire
 https://github.com/spring-projects/spring-gemfire-examples
 http://stackoverflow.com/questions/tagged/spring-data-gemfire

Q&A
Any questions…

Thank you!

Introducing Apache Geode and Spring Data GemFire

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Introducing Apache Geode and Spring Data GemFire (20)

Recently uploaded (20)

Introducing Apache Geode and Spring Data GemFire

Editor's Notes