Parallel processing in data warehousing and big data
Download as PPTX, PDF0 likes1,424 views
A database is optimized for transactional processing like writes, while a data warehouse is optimized for analytical queries on historical data from multiple sources. A data warehouse contains transformed and aggregated data structured for analysis. It partitions analytical and operational tasks to improve performance and limit locking during data updates. Parallel processing in a data warehouse distributes queries across multiple CPUs to enable faster and more flexible reporting and business intelligence.
Ad
More Related Content
What's hot (20)
Similar to Parallel processing in data warehousing and big data (20)
Ad
Recently uploaded (20)
Ad
Parallel processing in data warehousing and big data
- 2. Database vs Data Warehouse Database Data Warehouse Used for Online Transactional Processing (OLTP) Used for Online Analytical Processing (OLAP) Entity – Relationship modelling techniques are used for RDBS design Data Modelling techniques are used for Data Warehouse design Optimized for write operations Optimized for read operations Performance is low for analytical queries Performance is high for analytical queries Minimal to no downtime expected Can have scheduled downtime for data warehouse refresh
- 3. Data Warehouse A decision support database maintained separately from transactional or operational database Contains huge amounts of historic data Single and consistent store of data from multiple sources Contains transformed informational data Structured for complex queries and data analysis
- 4. Data Warehousing The process of transforming data into information and making it available to the users in a timely and efficient manner
- 5. Why Data Warehousing Structured in a way to increase performance of analytical tasks Partitions analytical and operational tasks, limiting risks of locking users during data updates Faster and more flexible reporting for business intelligence Understanding how well the business is performing Example: Sales information
- 6. Data Warehousing – Structure
- 7. On-Line Analytical Processing (OLAP) OLAP CUBE OLAP CUBE OLAP is a mechanism to extract and view data from different point of views. It allows users to analyse database information from multiple sources simultaneously. Stores data from multiple dimensions.
- 8. OLAP Cube A data structure that allows fast analysis of data. Consists of numeric facts called measures which are categorized by dimensions. OLAP Cube Operations: Drill down Roll up Slice Dice Pivot
- 9. Parallel Execution in Data Warehousing Why: For queries requiring large table scans or joins For creation of large indexes For creation of large tables (including materialized views) For bulk operations like bulk inserts, merges, or deletes
- 10. Parallel Execution in Data Warehousing When to implement parallel processing: Sufficient I/O bandwidth Underutilized or intermittently used CPUs Sufficient memory to support intensive tasks like sorts, hashing, etc. When not to implement parallel processing: Queries are generally very short and simple CPU, memory, or I/O is already heavily utilized
- 11. Degree of Parallelism (DOP) Defined as the number of parallel executions servers associated with a single operation. Managing DOP in Oracle Database Systems: Adaptive multi-user algorithm (default): Reduces DOP as system load increases. User Profile Management: Sets limit to each user within a security domain. Database Resource Manager: Allocates resources based on user groups.
- 12. How Parallelism Works Parallel Execution divides SQL statements into multiple small units called granules. Each granule is executed by a separate process. Oracle initializes the flag PARALLEL_MIN_SERVERS and PARALLEL_MAX_SERVERS on instance startup. If number of parallel execution servers increases, the flag PARALLEL_MAX_SERVERS is modified.
- 13. How Parallelism Works One parallel execution server is set as the query coordinator. The query coordinator distributes tasks amongst all other execution servers. The execution servers scan the tables, process their results and send it back to the coordinator.
- 14. Automatic Degree of Parallelism The query optimizer determines the fastest possible plan. Parses the query, calculates the execution cost, and calculates the DOP based on that. The cheapest plan could also be a serial execution, and hence the architecture keeps that as an option. Uses PARALLEL_MIN_TIME_THRESHOLD flag for this. Default value of this flag is 10 seconds.
- 15. Automatic Degree of Parallelism
- 16. Parallelism In Action Schema: SALES HISTORY (available on livesql.oracle.com) No. of records: SALES: 918,843 CUSTOMERS: 55,500 Execution Time: 18.83 seconds
- 17. Parallelism In Action Schema: SALES HISTORY (available on livesql.oracle.com) No. of records: SALES: 918,843 CUSTOMERS: 55,500 Execution Time: 5.60 seconds