GemFire In-Memory Data Grid
Download as PPTX, PDF5 likes4,260 views
GemFire is an in-memory data grid that pools memory across processes to manage application objects and behavior like caching, querying, transactions, and event notifications. It uses a shared-nothing architecture to allow for linear scalability and high availability. Data can be stored in replicated or partitioned regions. GemFire supports operations like read-through, write-through, and write-behind caching to integrate with backend data sources. It also allows querying data and continuous querying of events, as well as running application functions on members and data sets. An example broker application is discussed that could use GemFire for high availability, parallel aggregation, and scalability.
GemFire In-Memory Data Grid
- 1. GemFire: In-Memory Data GridSeptember 8th, 2011
- 3. Is it easy to scale Data Base?New users means, more application servers and more load to database.Application TierClientsData Base3
- 4. Moore's law: The number of transistors doubles approximately every 24 monthsWhat about data? 90% of today’s data were created in the last 2 yearsWeb logs, financial transactions, medical records, etc4
- 5. “Hardware can give you a generic 20 percent improvement in performance, but there is only so far you can go with hardware.”Rob Wallos,Global Head of marketing data Citi5
- 6. What is latency?Latency – is the amount of time that it takes to get information from one designated point to another.6
- 7. Why worry about it?Amazon - every 100ms of latency cost them 1% in salesGoogle - an extra 0.5seconds in search page generation time dropped traffic by 20%Financial - If a broker's electronic trading platform is 5ms behind the competition it could loose them at least 1% of the flow - that's 4$ million in revenues per ms.7
- 8. How to make data access even fast?Distributed Architecture
- 9. Drop ACID
- 10. Atomicity
- 11. Consistency
- 12. Isolation
- 13. Durability
- 15. Drop Disk8
- 16. Data GridData Grid is the combination of computers what works together to manage information and reach a common goal in a distributed environment.9
- 17. Shared nothing architectureIs a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system.Popularized by BigTable and NoSQL
- 19. Massive scalability of processing10
- 20. In-Memory Data GridData are stored in memory, always available and consistent.Low Latency
- 22. No Single Point of failure
- 23. Associate arrays
- 24. Replicated
- 25. Partitioned11
- 26. GemFireThe GemFire is in-memory distributed data management platform that pools memory across multiple processes to manage application objects and behavior.Caching
- 27. Querying
- 28. Transactions
- 31. CAP TheoremOnly two of these three desirable properties in distributed system can be achieved:Consistent
- 32. Available
- 34. RegionsData region is a logical grouping within a cache for a single data set.A region lets you store data in many VMs in the system without regard to which peer the data is stored on. Work similar to Map interface.14
- 35. Region ExampleCache cache = new CacheFactory().set("cache-xml-file", "cache.xml”).create();CacheServercacheServer = cache.addCacheServer();cacheServer.start();Regionpeople = cache.getRegion(”people");people.put(“John”, john);<cache> <regionname="people"> </region> </cache>Create Cache Server
- 37. Place an John entry into the region15
- 38. Replicated RegionEach replicated region holds the complete data set for the regionHigh Read Performance
- 39. Limited by JVM heap size
- 40. Used for meta data16
- 41. Partitioned RegionGemFire partitions your data so that each peer only stores a part of the region contents.Data spread across nodes
- 42. Members have access to all data
- 43. Used for Large data set
- 45. What happens if one node fails?Recovering redundancy can be configured to take place immediately after one node fail.This gives High Availability for partition regions.18
- 46. Local RegionThe local region has no peer-to-peer distribution activity.Client regions automatically defined as local regions:Direct to distributed system
- 48. Peer DiscoveryTo connect to distributed system the peer should introduce themself:Multicast based discovery
- 49. Locator separate component that maintains a discovery20
- 50. P2P topologyThe cache is embedded within the application process and shares the heap space with the application.21
- 51. Client/Server topologyA central cache is managed in one distributed system tier by a number of server members. Clients maintain their own caches that automatically call upon the server side.22
- 52. Multi-Site CachingDistributed systems at different sites are loosely coupled through gateway system members.23
- 53. Read ThroughWhen an entry is requested that is unavailable in the region, a Cache Loader may be called upon to load it from data source.Operation always managed by the partition node. 24
- 54. Write ThroughTo provide write-through caching with your external data source use CacheWriter.Only one writer is invoked for any event.25
- 55. Write BehindIn the Write-Behind mode, updated cache entries are asynchronously written to the back-end data source. 26
- 56. Event ListenerThe cache event listeners allow you to receive after-event notification of changes to the region and its entries.Handle following entity events:Create
- 57. Update
- 58. Destroy
- 59. InvalidateExecuted in all replicated regionsExecuted only in one partition region27
- 60. Listener Example<regionname=“people” refid=“PARTITION”> <region-attributes> <cache-listener> <class-name>com.mirantis.PeopleCacheListener</class-name> </cache-listener> <cache-loader><class-name>com.mirantis.PeopleCacheLoader</class-name></cache-loader> </region-attributes></region>public class PeopleCacheListener<K,V> extends CacheListenerAdapter<K,V> implements Declarable { public void afterCreate(EntryEvent<K,V> e) {System.out.println(e.getKey() + “ connected”);} public void afterDestroy(EntryEvent<K,V> e) {System.out.println(e.getKey() + “ left”); } …}28
- 61. QueryingObject Query Language (OQL) is SQL like query language standard for object-oriented databases.Support normal query and continuous querying (CQ).SELECT DISTINCT * FROM /portfolios WHERE status = 'active' AND type = ‘XYZ’Queryquery = qryService.newQuery(queryString);SelectResults results = (SelectResults)query.execute();for (Iteratoriter = results.iterator(); iter.hasNext(); ) { Portfolio activeXYZPortfolio = (Portfolio) iter.next(); ...}You can also use indexing to optimize your query performance.29
- 62. Continuous QueryingContinuous Querying (CQ) gives your clients a way to run queries against events.public class TradeEventListener implements CqListener {publicvoidonEvent(CqEventcqEvent) { … }publicvoidonError(CqEventcqEvent) { // handle the error } public void close() { // close the output screen for the trades ... }}CqAttributesFactorycqf = new CqAttributesFactory();cqf.addCqListener(tradeEventListener);CqAttributescqa = cqf.create();CqQuerypriceTracker = queryService.newCq(“tracker“, queryStr, cqa);priceTracker.execute();30
- 63. Function ExecutionApplication functions can be executed on:Members
- 64. Data setSimilar to Map-Reduce31
- 65. You can move the state or behaviorData BaseClientsApplication TierIMDG32
- 66. Example Broker ApplicationHigh Available
- 68. Exchange Server could have only one connection
- 69. Orders are swapped to Data Base
- 70. Scale on Demand 33
- 71. Learn moreVMWareGemFirehttp://www.vmware.com/products/vfabric-gemfire/overview.htmlMonitoring ToolsGemFireCommunity http://community.gemstone.com/display/gemfireHibernate L2 Cache