In-Memory Big Data Analytics

IN-MEMORY BIG DATA
ANALYTICS
Supreeth MP
1st sem BDA
18/9/2017 1

Table of Contents:
1. Data is growing
2. What is In-Memory analytics?
3. Why In-Memory Now?
4. The landscape of disk-based and in-memory data management systems
5. In-Memory vsTraditional (on-disk) database management system
6. Optimization Aspects on In-Memory Data Management and Processing
7. Some questions on in-memory analytics
8. References
18/9/2017 2

Data is growing:
• Continuous flow of data
• Real-time, 24/7 streaming updates
• More than 2.5 quintillion bytes of data added daily
• Data is always available
• Democratization of data
• Main source for business decisions
• Shift to digital and STP
• Affordable technology
• Better and faster analytics
• Business Intelligence
• Cloud and subscription-based computing
18/9/2017 3

What is In-Memory analytics?
An in-memory analytics system basically is a database management system that
stores data entirely in main memory that is in the RAM.This contrasts to traditional
(on-disk) database systems, which are designed for data storage on persistent media
such as hardisk. Because working with data in memory is much faster than writing to
and reading from a file system.
In-memory is ideal when:
• Your database is too slow for interactive analytics
• You need to perform real-time data analytics
• You need to be offline and can't connect to your data live
18/9/2017 4

Why In-Memory Now?
• RAM is 200 times faster than disk storage and typically enables data access 50 to 100 times
quicker
• Memory storage capacity and bandwidth have been doubling roughly every three years,
while its price has been dropping by a factor of 10 every ﬁve years.
• Modern high-end servers now have multiple sockets, each of which can have tens or
hundreds of gigabytes of DRAM
• Growth of distributed systems
• The increasing adoption of 64-bit computer technology has made RAM more suitable for
use with large datasets.
• Database systems have been evolving over the last few decades.
18/9/2017 5

The landscape of disk-based and in-memory
data management systems:
18/9/2017 6

In-Memory vsTraditional (on-disk) database
management system:
18/9/2017 7

management system:
Aspects DBDMS IMDBS
File I/O Carries File I/O burden No file I/O burden
Storage Usage Assumes storage is abundant Uses storage more efficiently
Algorithms Algorithm optimized for disk Algorithms optimized for memory
CPU Cycles More CPU cycles Less CPU cycles
Persistence Non-volatile Volatile
Lock Fine Locks Coarse Locks
18/9/2017 8

management system:
18/9/2017 9

Optimization Aspects on In-Memory Data
Management and Processing:
Aspects Concerns Techniques
Index Cache consciousness, time/space
efficiency
Hash-based, tree-based
Data Layout Cache consciousness, space efficiency Columnar layout
Parallelism Linear scaling, partitioning Data-level, shared-memory scale-up and
shared-nothing scale out parallelism
Concurrency
Control
Overhead, correctness Coarse-grained locks
Query Processing Code locality, register temporal locality,
time efficiency
Coarse-grained stored procedures
Fault tolerance Durability, correlated failures, availability Checkpoints andTransaction logging
Data Overflow Locality, Paging strategy, hot/cold
classification
Anti-caching
18/9/2017 10

Some questions on in-memory analytics:
• What do companies need to think about as they take on an in-memory analytics path?
• What are some potential speed bumps in adopting in-memory analytics?
• What role do skills play here?
• If an in-memory database system boosts performance by holding all records in memory,
can’t we get the same result by creating a RAM disk and deploying a traditional database
there?
• Won’t an in-memory database require huge amounts of memory because database systems
are large?
• Isn’t the database just lost if there’s a system crash?
18/9/2017 11

References:
[1] In-Memory Big Data Management and Processing:A Survey, IEEETRANSACTIONSON
KNOWLEDGEAND DATA ENGINEERING JULY 2015
[2] Using In-MemoryAnalytics to Quickly Crunch Big Data by Lee Garber
[3] https://www.sas.com/en_us/insights/articles/big-data/in-memory-analytics-questions.html
[4] DataAnalytics using In-MemoryComputing: https://www.gridgain.com/
[5] How Computers Work: Disks And Secondary Storage:
http://homepage.cs.uri.edu/faculty/wolfe/book/Readings/Reading05.htm
[6] http://www.mcobject.com/in_memory_database
[7] In-Memory DatabaseComputing – Smarter way of data analysis:
http://www.xoriant.com/blog/big-data-analytics/memory-database-computing-faster-smarter-
analysis-big-data-world.html
[8] How Computers Work:The CPU and Memory:
http://homepage.cs.uri.edu/book/cpu_memory/cpu_memory.htm
18/9/2017 12

In-Memory Big Data Analytics

Recommended

More Related Content

What's hot (20)

Similar to In-Memory Big Data Analytics (20)

Recently uploaded (20)

In-Memory Big Data Analytics