JDG 7 & Spark Integration
2 likes454 views
This document discusses the Infinispan Spark connector, which provides integration between JBoss Data Grid 7 (JDG 7) and Apache Spark. It introduces JDG 7 and Apache Spark and their features. The Infinispan Spark connector allows users to create Spark RDDs and DStreams from JDG cache data, write RDDs and DStreams to JDG caches, and perform real-time stream processing with JDG as the data source for Spark. The connector supports various configurations and provides seamless functional programming with Spark. A demo of examples is referenced. Human: Thank you for the summary. Can you provide another summary in 2 sentences or less?
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to JDG 7 & Spark Integration (20)
Ad
More from Ted Won (9)
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
JDG 7 & Spark Integration
- 1. JDG 7 & Spark Integration Infinispan Spark connector tedwon
- 2. Agenda ● Brief Introduction to JBoss Data Grid 7 ● Brief Introduction to Apache Spark ● Features of Infinispan Spark connector ● Demo
- 3. What is JBoss Data Grid ● Distributed cache ● In-memory NoSQL ● Key/value data store ● Built from the Infinispan community project ● In architecture, there is no master node ● Peer-to-peer membership clustering ● The most strength is extremely highly availability and scalability as in-memory ○ In my personal opinion ● Good harmony with real-time data processing framework for data services ● As a metaphor, In-memory HDFS for real-time processing
- 4. What is JBoss Data Grid 7 ● JDG 7 was released at July, 2016 ● JDG 7 is based on Infinispan 8.3.0 and EAP 7.0.0.GA ○ The previous version JDG 6.6.0 is based on Infinispan 6.4.0 ● JDG 7 now becomes much more powerful as a competitive software product ● There are new features and enhancements ○ New GUI Admin console ○ Easy to install and configure clustering and cache ○ Easy to monitor and change configuration ○ Easy to create a new cache through admin console even in runtime ○ Provides API for integration with Apache Spark
- 5. JDG 7 - New Features and Enhancements 1. DISTRIBUTED STREAMS 2. REMOTE TASK EXECUTION 3. APACHE SPARK INTEGRATION 4. APACHE HADOOP INTEGRATION 5. NEW ADMINISTRATION CONSOLE FOR SERVER DEPLOYMENTS 6. CONTROLLED SHUTDOWN AND RESTART OF CLUSTER 7. NODE.JS (JAVASCRIPT) HOT ROD CLIENT 8. CASSANDRA CACHE STORE 9. HOT ROD C++ ENHANCEMENTS 10. HOT ROD C# ENHANCEMENTS
- 6. What is Apache Spark
- 7. ● General-purpose Lightning-fast cluster computing framework ● General-purpose ○ One common data processing engine ○ Batch, SQL, Streaming, MLlib, Graph ○ One platform to rule them all ● Lightning-fast ○ RDD - Resilient Distributed Dataset ○ Read-only multiset of data items distributed over a cluster of machines ● Works with Hadoop HDFS and process that data in parallel ○ Distributed file System ● MapReduce interfaces with funtional programming style ○ No MR chaining workflow in Hadoop What is Apache Spark
- 8. Apache Spark - One platform to rule them all
- 9. What is Apache Spark RDD ● RDD consists of mulitle data partitions over a cluster of machines ● RDD is intermediate data during a Spark data processing job ● RDD is distributed in memory of each worker node JVM ● Data processing job is transformations of RDDs ● RDDs are disappeared after finishing a job ● Not able to share RDDs between Spark jobs
- 10. What is Apache Spark RDD
- 11. Infinispan Spark connector JDG 7’s new feautre
- 12. What is Infinispan Spark connector ● https://github.com/infinispan/infinispan-spark ● JDG 7 Document https://goo.gl/9BXp98 ● RDD and DStream integration with Apache Spark 1.6 ○ DStream is a continuous sequence of RDDs for real-time stream processing ● Use JDG as a data source for Spark ● Easy to read & write cache data in a Spark job ● Provides seamless funtional programming style and syntactic sugar ● Good to share RDD with other Spark jobs
- 14. Features of Infinispan Spark connector 1. Create an Spark RDD from a JDG cache data ○ Read cache data from Spark job 2. Write any key/value based RDD to a JDG cache ○ Write intermediate or final data to cache 3. Create a Spark DStream from cache-level events ○ Insert, Modify and Delete event in a cache 4. Write any key/value DStream to JDG ○ Write any DStream to cache 5. Use JDG server side filters to create a cache based RDD ○ Using Infinispan Query DSL
- 15. Prerequisite & Version compatibility ● JDK 8, Scala 2.10.4, Spark 1.6 ● JBoss Data Grid 7.0.0 Server ○ Infinispan Server 8.2.4.Final ● Run JDG in Remote Client-Server mode
- 17. Performance Considerations ● The number of Spark workers should be greater than JDG nodes ○ To take advantage of the parallelism ● Support locality with co-located in the same node as JDG and Spark worker ○ Spark worker only processes data in the local node with connector
- 18. Thank you
- 19. References 1. https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Data_Grid/7.0/html-single/Developer_Guide/index. html#Integration_with_Apache_Spark 2. http://spark.apache.org/docs/1.6.2/programming-guide.html 3. https://github.com/infinispan/infinispan-spark 4. https://github.com/infinispan/infinispan 5. https://github.com/tedwon/infinispan-spark-connector-examples 6. https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Data_Grid/7.0/html-single/7.0.0_Release_Notes/in dex.html#chap-New_Features_and_Enhancements 7. https://en.wikipedia.org/wiki/Apache_Spark 8. https://hub.docker.com/r/gustavonalle/infinispan-spark/