![References:
[1] In-Memory Big Data Management and Processing:A Survey, IEEETRANSACTIONSON
KNOWLEDGEAND DATA ENGINEERING JULY 2015
[2] Using In-MemoryAnalytics to Quickly Crunch Big Data by Lee Garber
[3] https://www.sas.com/en_us/insights/articles/big-data/in-memory-analytics-questions.html
[4] DataAnalytics using In-MemoryComputing: https://www.gridgain.com/
[5] How Computers Work: Disks And Secondary Storage:
http://homepage.cs.uri.edu/faculty/wolfe/book/Readings/Reading05.htm
[6] http://www.mcobject.com/in_memory_database
[7] In-Memory DatabaseComputing – Smarter way of data analysis:
http://www.xoriant.com/blog/big-data-analytics/memory-database-computing-faster-smarter-
analysis-big-data-world.html
[8] How Computers Work:The CPU and Memory:
http://homepage.cs.uri.edu/book/cpu_memory/cpu_memory.htm
18/9/2017 12](https://image.slidesharecdn.com/seminar-180209005108/85/In-Memory-Big-Data-Analytics-12-320.jpg)
In-Memory Big Data Analytics
- 1. IN-MEMORY BIG DATA ANALYTICS Supreeth MP 1st sem BDA 18/9/2017 1
- 2. Table of Contents: 1. Data is growing 2. What is In-Memory analytics? 3. Why In-Memory Now? 4. The landscape of disk-based and in-memory data management systems 5. In-Memory vsTraditional (on-disk) database management system 6. Optimization Aspects on In-Memory Data Management and Processing 7. Some questions on in-memory analytics 8. References 18/9/2017 2
- 3. Data is growing: • Continuous flow of data • Real-time, 24/7 streaming updates • More than 2.5 quintillion bytes of data added daily • Data is always available • Democratization of data • Main source for business decisions • Shift to digital and STP • Affordable technology • Better and faster analytics • Business Intelligence • Cloud and subscription-based computing 18/9/2017 3
- 4. What is In-Memory analytics? An in-memory analytics system basically is a database management system that stores data entirely in main memory that is in the RAM.This contrasts to traditional (on-disk) database systems, which are designed for data storage on persistent media such as hardisk. Because working with data in memory is much faster than writing to and reading from a file system. In-memory is ideal when: • Your database is too slow for interactive analytics • You need to perform real-time data analytics • You need to be offline and can't connect to your data live 18/9/2017 4
- 5. Why In-Memory Now? • RAM is 200 times faster than disk storage and typically enables data access 50 to 100 times quicker • Memory storage capacity and bandwidth have been doubling roughly every three years, while its price has been dropping by a factor of 10 every five years. • Modern high-end servers now have multiple sockets, each of which can have tens or hundreds of gigabytes of DRAM • Growth of distributed systems • The increasing adoption of 64-bit computer technology has made RAM more suitable for use with large datasets. • Database systems have been evolving over the last few decades. 18/9/2017 5
- 6. The landscape of disk-based and in-memory data management systems: 18/9/2017 6
- 7. In-Memory vsTraditional (on-disk) database management system: 18/9/2017 7
- 8. In-Memory vsTraditional (on-disk) database management system: Aspects DBDMS IMDBS File I/O Carries File I/O burden No file I/O burden Storage Usage Assumes storage is abundant Uses storage more efficiently Algorithms Algorithm optimized for disk Algorithms optimized for memory CPU Cycles More CPU cycles Less CPU cycles Persistence Non-volatile Volatile Lock Fine Locks Coarse Locks 18/9/2017 8
- 9. In-Memory vsTraditional (on-disk) database management system: 18/9/2017 9
- 10. Optimization Aspects on In-Memory Data Management and Processing: Aspects Concerns Techniques Index Cache consciousness, time/space efficiency Hash-based, tree-based Data Layout Cache consciousness, space efficiency Columnar layout Parallelism Linear scaling, partitioning Data-level, shared-memory scale-up and shared-nothing scale out parallelism Concurrency Control Overhead, correctness Coarse-grained locks Query Processing Code locality, register temporal locality, time efficiency Coarse-grained stored procedures Fault tolerance Durability, correlated failures, availability Checkpoints andTransaction logging Data Overflow Locality, Paging strategy, hot/cold classification Anti-caching 18/9/2017 10
- 11. Some questions on in-memory analytics: • What do companies need to think about as they take on an in-memory analytics path? • What are some potential speed bumps in adopting in-memory analytics? • What role do skills play here? • If an in-memory database system boosts performance by holding all records in memory, can’t we get the same result by creating a RAM disk and deploying a traditional database there? • Won’t an in-memory database require huge amounts of memory because database systems are large? • Isn’t the database just lost if there’s a system crash? 18/9/2017 11
- 12. References: [1] In-Memory Big Data Management and Processing:A Survey, IEEETRANSACTIONSON KNOWLEDGEAND DATA ENGINEERING JULY 2015 [2] Using In-MemoryAnalytics to Quickly Crunch Big Data by Lee Garber [3] https://www.sas.com/en_us/insights/articles/big-data/in-memory-analytics-questions.html [4] DataAnalytics using In-MemoryComputing: https://www.gridgain.com/ [5] How Computers Work: Disks And Secondary Storage: http://homepage.cs.uri.edu/faculty/wolfe/book/Readings/Reading05.htm [6] http://www.mcobject.com/in_memory_database [7] In-Memory DatabaseComputing – Smarter way of data analysis: http://www.xoriant.com/blog/big-data-analytics/memory-database-computing-faster-smarter- analysis-big-data-world.html [8] How Computers Work:The CPU and Memory: http://homepage.cs.uri.edu/book/cpu_memory/cpu_memory.htm 18/9/2017 12
- 13. THANKYOU