Big Data Analytics: Finding diamonds in the rough with Azure
Download as PPTX, PDF0 likes78 views
The document focuses on big data analytics, particularly using Azure technologies, discussing its definition, challenges, and solutions for managing large datasets. It highlights tools like Azure Data Factory and Azure Databricks for efficient data processing and emphasizes the importance of metadata management. The presentation aims to help data teams effectively extract value from big data while addressing common issues in data handling.
![Big Data in Azure: Storage
Azure Blob Storage
wasb[s]://containername@accountname.blob.core.windo
ws.net/file.csv
Azure Data Lake
abfs[s]://filesystemname@accountname.dfs.core.window
s.net/file.csv
Azure Data Lake (Gen.2)
• Endpoint: object store access Blob API using wasb[s]://
• Endpoint: file system access ADLS Gen 2 API using
abfs[s]://](https://image.slidesharecdn.com/azurebootcamp-190512092401/85/Big-Data-Analytics-Finding-diamonds-in-the-rough-with-Azure-12-320.jpg)
Big Data Analytics: Finding diamonds in the rough with Azure
- 2. Big Data analytics: Finding diamonds in the rough with Azure Christos Charmatzis @T.A. Geoforce Athens Global Azure Bootcamp 2019
- 3. DATA TEAM
- 4. Agenda • Introduction • When we have a Big Data problem • Finding the best solution for our Big Data • Working inside the Data Team • Extract the true value of our data
- 5. Introduction What is Big Data? "Big Data" is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software. Source: Wikipedia The concept gained momentum in the early 2000s when industry analyst Doug Laney articulated the now-mainstream definition of big data as the three Vs: 1. Volume 2. Velocity 3. Variety
- 6. And because everything is relative What is today’s small (1TB), was yesterday’s big..... And what is today’s big(100TB) is tomorrow’s small…. (we use 100TB, because is the dataset size of Sort Benchmark competition http://sortbenchmark.org/ )
- 7. When we have a Big Data problem? Example 2 • 3TB Datasets • Machine (M32ls Instance , 32 VCPU, 256 GiB RAM, 1,024 GiB Storage, ~€2,122.3736/mont h ) • Enterprise Database (e.g. SQL Server) • Aggregation, Statistics, Summaries Example 3 • 10TB Dataset • Aggregation, Statistics, Summaries, Transformations etc Example 4 • 450GB Dataset • Machine (M32ls Instance , 32 VCPU, 256 GiB RAM, 1,024 GiB Storage, ~€2,122.3736/mont h ) • Enterprise Database (e.g. SQL Server) • Transformations Example 1 • 450GB Datasets • Machine (M32ls Instance , 32 VCPU, 256 GiB RAM, 1,024 GiB Storage, ~€2,122.3736/mont h ) • Enterprise Database (e.g. SQL Server) • Aggregation, Statistics, Summaries STAY WHERE YOU ARE UPGRADE STORAGE GO TO THE CLOUD GO TO THE CLOUD
- 8. Big Data Infrastructure comparison Spark Cluster • Initial release > 2014 • Supported programming languages: Java, Scala, Python, R, Julia • Performance keys: Partitioning DB in premise • Initial release < 2014 • Supported programming languages: almost every programming language • Performance keys: Indexes !==
- 9. Big Data Performance In Spark always: • use “df.explain(true)” • Or check the DAG! Every time a block is changing the data are repartitioning!!!
- 10. Finding the best solution for our Big Data problem • Hadoop on a cluster of Azure Virtual Machines • Azure HDInsights (Clusters as-a-service) • Azure Databricks • Azure Data Factory (New & Improved!!!!) • Azure Data Lake Analytics (Queries as-a-service)
- 11. Big Data in Azure: Storage Azure Blob Storage • Object Storage • General purpose (files & workloads) Azure Data Lake • Hierarchical file system • Optimized for analytics workloads Azure Data Lake (Gen.2) • Multi-modal storage • Optimized for analytics workloads
- 12. Big Data in Azure: Storage Azure Blob Storage wasb[s]://[email protected] ws.net/file.csv Azure Data Lake abfs[s]://[email protected] s.net/file.csv Azure Data Lake (Gen.2) • Endpoint: object store access Blob API using wasb[s]:// • Endpoint: file system access ADLS Gen 2 API using abfs[s]://
- 13. Azure Data Factory A managed could service for building & operating data pipelines.
- 14. Azure Data Factory Source: https://channel9.msdn.com
- 15. Azure Data Factory (ADF) DEMO
- 16. Why do we need tools like ADF? 85% of the working time is on data wrangling!!!
- 17. ADF Pricing https://azure.microsoft.com/en-us/pricing/details/data-factory/ Tip: Look out!!! The data reads and writes are the most expensive in Big Data Analytics
- 18. Working inside the Data Team Rembrandt (1662). The Sampling Officials (Dutch: De Staalmeesters)) We must compare the results with last year's... I will run the whole thing, again Don't we have somewhere that report? Yes, there're in a folder, inside a VM, inside John's PC... No, we have uploaded them in blob storage... I don't remember Somewhere, inside a meeting room….
- 19. Metadata area With Data comes problems…. With Big Data comes Bigger Problems!!! Like…. • Many datasets • Frequently updates • Many fields • Many users
- 20. Where do I keep the metadata? • Azure Data Catalogue • DataBricks Delta Lake • Create your own meta-portal Be aware, always use metadata standards (ISO, Dublin Core, MPEG-7 …) More info: https://en.wikipedia.org/wiki/Metadata_standard#Available_metadata _standards
- 21. Azure Data Factory Metadata This activity allows for collecting metadata about Azure Data Factory. Get Metadata activity supports: • itemName • itemType • Size • Created • lastModified • childItems • contentMD5 • Structure • columnCount • exists
- 22. Extract real value from the data Visualize data | Write good experiments | Share results
- 23. And we just scratched the surface of that…
- 24. Conclusions • For ETL projects from in premise to cloud use Azure Data Factory • The size isn’t always the problem in your case • Velocity isn’t only on the code side, you HAVE to know your data • Create METADATA