Infinspan: In-memory data grid meets NoSQL
8 likes3,061 views
The document discusses Infinispan, an open source in-memory data grid. It describes Infinispan's features like its map-like API, persistence, querying support and transactional capabilities. It also discusses how Infinispan can be used to build elastic, scalable data services and compares data grids to NoSQL databases.
Infinspan: In-memory data grid meets NoSQL
- 1. [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 2. Infinispan In-memory data grid meets NoSQL Manik Surtani Founder and project lead, Infinispan Senior Principal Engineer Red Hat, Inc. [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 3. “Could data storage be the one thing that hampers true cloud scalability and elasticity?” [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 4. Who is Manik? • R&D Engineer, Red Hat Inc. • Founder and project lead, Infinispan • Project lead, JBoss Cache • Frequent speaker on cloud computing and cloud data storage http://twitter.com/maniksurtani http://blog.infinispan.org [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 5. Agenda • What is Data-as-a-Service? • Introducing Infinispan • Implementing Data-as-a-Service with data grids • Data grids vs. NoSQL [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 6. Traditional 3-tier App [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 7. Typical IaaS App [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 8. Typical PaaS App [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 9. Where’s your data stored?? [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 10. Clouds are ephemeral! [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 11. State [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 12. Virtualizing Data • Some public services do exist • Amazon RDS and SimpleDB • FathomDB • Cloundant • MongoHQ • etc. [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 13. What about private clouds? • Not all cloud deployments are public! • Private cloud is very important • How can you build DaaS yourself? [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 14. Characteristics of DaaS • Elastic data • Needs to scale with other tiers • Response times should be linear • Needs to be highly available • Nodes will die! The service shouldn’t. [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 15. “Traditional” RDBMSs • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 16. “Traditional” RDBMSs • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 17. Distributed Data Grid • Far better suited to elastic data • Distributed by nature • Highly available by nature • A good building block for your data service [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 18. API is king • Apps should use their native data storage APIs • E.g., JPA (Java EE), ActiveRecord (Ruby), etc. • Key/value too low level • Akin to direct JDBC calls! [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 19. Introducing [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 20. What is Infinispan? • Open source (LGPL) in-memory Data Grid • Some concepts from Amazon Dynamo 2 usage modes • Embedded • Client-server • memcached • Hot Rod • REST [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 21. Embedded Mode [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 22. P2P Embedded Architecture [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 23. API • Map-like key/value store • Upcoming JPA-like layer • Other high-level APIs being discussed in the community e.g., ActiveRecord [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 24. Consistent hash based distribution • Self healing • No single point of failure Highly concurrent • MVCC locking [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 25. Persistence • Not just in memory! • Write through and write behind • Pluggable “drivers” Eviction and expiry • Efficient, adaptive algorithms • Addresses shortcomings of LRU & FIFO [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 26. XA Transactions • 2-phase commit based • Deadlock detection algorithms • Coming soon: Atomic Broadcast [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 27. Map/Reduce • In a pre-release state right now • Please try out 5.0.0.ALPHA3 with these APIs! Querying • Using Lucene and Hibernate Search to index [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 28. Client/Server Mode [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 29. Client/Server Architecture Supported Protocols • REST • Memcached • Hot Rod [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 30. WTF is Hot Rod? • Wire protocol for client server communications • Open • Language independent • Built-in failover and load balancing • Smart routing [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 31. Server Endpoint Comparison Client Clustered Smart Load Balancing/ Protocol Libraries ? Routing Failover REST Text N/A Yes No Any HTTP load balancer Only with predefined Memcached Text Plenty Yes No server list Currently Hot Rod Binary Yes Yes Dynamic only Java [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 32. So is Infinispan a data grid? • In-memory • P2P, distributed • Low-latency, fast key/ value store [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 33. So is Infinispan a data grid? • In-memory • P2P, distributed ... or is it a NoSQL • Low-latency, fast key/ value store database? • Persistence • Map/Reduce • Client/Server mode [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 34. So is Infinispan a data grid? • In-memory • P2P, distributed ... or is it a NoSQL • Low-latency, fast key/ value store database? • Persistence ... or something • Map/Reduce • Client/Server mode else? • Querying support • Transactional [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 35. DaaS with Infinispan [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 36. Architecture Manage and Monitor [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 37. Data Grids vs. NoSQL [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 38. Data Grids vs. NoSQL • Data grids/distributed caches as proto-NoSQL? • Been around since early 2000’s • Used as database offload • Bottleneck removal • Primary K/V stores for certain types of data • Low latency data access [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 39. Their roots Data Grids in-memory, non-clustered -> add clustering -> add persistence -> add txns -> add M/R NoSQL persistence -> add clustering -> add M/R -> add memory cache [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 40. Evolving: Enterprise NoSQL • Taking NoSQL mainstream • Adding enterprise features • Transactions • Querying • Adding common APIs • Easy access, portability [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 41. NoSQL standards • APIs - basic interactions and Map/Reduce • JCP for Java + others • http://bit.ly/data_grid_jsr • Wire protocol • Remote communications. • Hot Rod? • Query language • Without relational presumptions [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 42. Summing things up • Elastic data is hard • Public data services not always suitable • Data grids make elastic storage easy • Infinispan server endpoints help build elastic data tiers • Discussed data grids versus NoSQL [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 43. Questions? http://www.infinispan.org http://blog.infinispan.org http://github.com/infinispan http://twitter.com/infinispan - #infinispan #infinispan on FreeNode (IRC) [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 44. Starting an Infinispan Server • Hot Rod or memcached server endpoint $ bin/startServer.sh -r hotrod -c infinispan.xml $ bin/startServer.sh -r memcached -c infinispan.xml [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 45. Starting an Infinispan Server • REST endpoint • Deploy infinispan-server-rest.war in your favorite servlet container. [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 46. Roadmap 4.0.0 Starobrno • Map-like API • Async API • Consistent Hash based distribution • Write-through, write-behind • Eviction, expiration • Management tooling • REST API • Hibernate 2nd Level Cache • Released Feb 2010 [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 47. Roadmap 4.1.0 Radegast • Deadlock detection • Client/Server protocols • Memcached • Hot Rod • Smart clients using Hot Rod • Lucene Directory implementation • LIRS: adaptive, recency-based eviction policies • Released August 2010 [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org
- 48. Roadmap 5.0.0 Pagoa • JPA-like API • Fine-grained replication • Distributed code execution • Map/reduce • Virtual nodes for more even distribution • In active development 5.1.0 and beyond • Dynamic provisioning based on SLAs • Complex event processing features [email protected] | http://twitter.com/maniksurtani | http://blog.infinispan.org