Aleksejs Nemirovskis - Manage your data using oracle BDA
1 like301 views
Manage Your Data, Using Oracle Big Data Appliance - Tips & Tricksngest, process and manage the data, using Oracle Big Data Appliance (end-to-end BigData solution from Oracle): - Oracle BDA architecture and componets overview - Oracle platform, Cloudera CDH, Clodera Manager and specific Oracle components; - Advantages and additional value of an Oracle BDA; - Challenges, faced inside whole stack (BDA, Cloudera); - Challenges, which came from original Hadoop EcoSystem; - Customer case (anonymized): how to utilize a power of an Oracle BDA, including external Informatica Big Data Management tool.
Ad
More Related Content
What's hot (20)
Similar to Aleksejs Nemirovskis - Manage your data using oracle BDA (20)
![【旧版】Oracle Exadata Cloud Service：サービス概要のご紹介 [2020年8月版]](https://cdn.slidesharecdn.com/ss_thumbnails/exadatacloudserviceoverview20200824-200824060458-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Andrejs Vorobjovs (20)
Ad
Recently uploaded (20)
Aleksejs Nemirovskis - Manage your data using oracle BDA
- 1. Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle Big Data Appliance Tips & Tricks
- 2. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. About me Aleksejs Nemirovskis Technology Architect Over 23 years of experience in IT industry with main focus to - DB architecture, design, analysis, development and performance tuning - Data Integration / Data Warehousing / ETL / EL-T / BI / Data Analytics - Oracle Technologies stack: DB, Data Integrations, Infrastructure, Cloud - Big Data (Apache Hadoop, CDH, Cloudera Manager) Experienced in all stages of software development life cycle (SDLC) from business requirements and technical definitions to development, testing and production support as well as in Agile software development techniques (Scrum, XP)
- 3. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Agenda ➢ Oracle Big Data Appliance architecture and components overview ➢ Oracle BDA System Overview ➢ Oracle BDA Hardware Overview ➢ Oracle BDA Software Overview ➢ Cloudera Distributed Hadoop (CDH) ➢ Advantages and additional value of an Oracle BDA ➢ End-to-end system ➢ Oracle XQuery 4 Hadoop ➢ Oracle DataSource for Apache Hadoop (OD4H) ➢ Challenges ➢ Oracle Big Data Appliance ➢ Cloudera ➢ Hadoop Eco-System ➢ Architecture & Implementation ➢ Customer case ➢ Infrastructure Setup ➢ Infrastructure Setup - PROD ➢ Data Processing ➢ Q&A
- 4. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle BDA ➢Oracle BDA architecture and components overview ➢Advantages and additional value of an Oracle BDA ➢Challenges ➢Customer case ➢Q&A
- 5. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance ➢ Cloudera Manager ▪ Cloudera Navigator ➢ Cloudera Distributed Hadoop ➢ Oracle Big Data Connectors ➢ Oracle Linux ➢ Oracle Server (hardware) System Overview
- 6. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance Hardware OverviewStarter Rack - 6 x Compute / Storage Nodes X7-2 Per Node configuration ➢ 2 x 24-Core (2.1GHz) Intel ® Xeon ® 8160 ➢ 8 x 32GB DDR4-2666 MHz Memory, expandable to 1.5TB ➢ 12 x 10 TB 7,200 RPM High Capacity SAS Drives ➢ 2 x 150GB M.2 SATA SSD Drives ➢ 2 x QDR 40Gb/sec InfiniBand Ports ➢ 1 x Dual-port InfiniBand QDR CX3 (40 Gb/sec) PCIe HCA ➢ 1 x Built-in RJ45 1 Gigabit Ethernet port Full Rack - 18 x Compute / Storage Nodes
- 7. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance Software Overview Integrated Software ➢ Cloudera Enterprise 5 – Data Hub Edition with support for: ▪ Cloudera’s Distribution including Apache Hadoop (CDH) ▪ Cloudera Impala ▪ Cloudera Search ▪ Apache HBase ▪ Apache Spark ▪ Apache Kafka ▪ Cloudera Manager with support for: o Cloudera Navigator o Cloudera Back-up and Disaster Recovery (BDR) ➢ Oracle Perfect Balance ➢ Oracle Table Access for Hadoop Optional Software ➢ Oracle Big Data SQL ➢ Oracle Big Data Connectors: ▪ Oracle SQL Connector for Hadoop ▪ Oracle Loader for Hadoop ▪ Oracle XQuery for Hadoop ▪ Oracle R Advanced Analytics for Hadoop ▪ Oracle Data Integrator ➢ Oracle Audit Vault and Database Firewall for Hadoop Auditing ➢ Oracle Data Integrator ➢ Oracle GoldenGate ➢ Oracle NoSQL Database Enterprise Edition ➢ Oracle Big Data Spatial and Graph ➢ Oracle Big Data Discovery
- 8. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance Cloudera Distributed Hadoop
- 9. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle BDA ➢Oracle BDA architecture and components overview ➢Advantages and additional value of an Oracle BDA ➢Challenges ➢Customer case ➢Q&A
- 10. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance End-to-End System` Maintained&SupportedbyOracle
- 11. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance Oracle XQuery 4 Hadoop
- 12. Copyright 2018 Accenture. All rights reserved. Oracle Big Data Appliance Oracle DataSource 4 Hadoop
- 13. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle BDA ➢Oracle BDA architecture and components overview ➢Advantages and additional value of an Oracle BDA ➢Challenges ➢Customer case ➢Q&A
- 14. Copyright 2018 Accenture. All rights reserved. Challenges Oracle Big Data Appliance Cloudera release: gap up to 2 years Hadoop HFDS: dedicated Name Node vs. Name/Data Node Support: Oracle vs. Cloudera vs. Hadoop
- 15. Copyright 2018 Accenture. All rights reserved. Challenges Cloudera HIVE vs. Impala Limitations (e.g. DBs/Objects to replicate) HDFS <> Sentry synchronization
- 16. Copyright 2018 Accenture. All rights reserved. Challenges Hadoop Eco-System Parallel writing into the Parquet files Spark and Sentry – how to deal with security, using Spark
- 17. Copyright 2018 Accenture. All rights reserved. Challenges Architecture & Implementation Partitioning – partitioning key challenge Compression – always needed? Fast processing > small files on HDFS (e.g. Resilient Distributed Datasets – Spark RDD)
- 18. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle BDA ➢Oracle BDA architecture and components overview ➢Advantages and additional value of an Oracle BDA ➢Challenges ➢Customer case ➢Q&A
- 19. Copyright 2018 Accenture. All rights reserved. Infrastructure Setup Node01 Node02 Node03 Oracle BDA Informatica BDM Customer Case 7 Clusters: BDA X6-2 Starter Rack (6 nodes) ➢ 264 CPU cores ➢ 1.5 TB RAM (+ 1.5 TB RAM per cluster is upcoming) ➢ 576 TB Storage 4 Domains: Informatica Big Data Management ➢ 10.2.1 in a full push-down mode ➢ 2 Domains: 3 nodes per domain ➢ 2 Domains: 3 + 3 (DR) nodes per domain
- 20. Copyright 2018 Accenture. All rights reserved. Infrastructure Setup - PROD Node01 Node02 Node03 Main Cluster Main Nodes Customer Case BDA X6-2 Starter Rack cluster (6 nodes + 10 nodes are upcoming) ➢ 264 CPU cores (+ 440 CPU cores are upcoming) ➢ 1.5 TB RAM (+ 6.3 TB RAM is upcoming) ➢ 576 TB Storage (+ 960 TB is upcoming) Informatica Big Data Management ➢ 10.2.1 in a full push-down mode
- 21. Copyright 2018 Accenture. All rights reserved. Customer Case Data Processing Sources Ingestion/Processing Stores Usage Cloudera Data Science Workbench
- 22. Copyright 2018 Accenture. All rights reserved.Copyright 2018 Accenture. All rights reserved. Manage Your Data using Oracle BDA Q & A