.NET for Azure Synapse (and viceversa)
Download as PPTX, PDF0 likes84 views
Jupyter Notebooks and Apache Spark are first class citizens of the Data Science space, a truly requirement for the "modern" data scientist. Now with Azure Synapse these two computing powers are available to the .NET Developer. And .NET is available for all data scientists. Let's look what .net can do for notebooks and spark inside Azure Synapse and what are Synapse, notebooks and spark.
![Marco Parenzan
marco [dot] parenzan [at] 1nn0va [dot] it
www.marcoparenzan.com](https://image.slidesharecdn.com/2020-201203071606/85/NET-for-Azure-Synapse-and-viceversa-1-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to .NET for Azure Synapse (and viceversa) (20)
Ad
More from Marco Parenzan (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
.NET for Azure Synapse (and viceversa)
- 1. Marco Parenzan marco [dot] parenzan [at] 1nn0va [dot] it www.marcoparenzan.com
- 2. Marco Parenzan Solution Sales Specialist in Insight for Digital Innovation Azure MVP Community Lead 1nn0va // Pordenone Linkedin: https://www.linkedin.com/in/marcoparenzan/
- 3. Road to be a newbie “Data *” for a seasoned C# dev in a Pythonic world
- 4. Data Processing in a Azure World... • Thousands of IoT sensors in a factory, producing petabytes of data • Started with Stream Analytics... • Like Azure Data Explorer... • Data Warehouse... • ...but there is a standard? Yes!
- 5. Apache Spark The data c ompute exper ienc e
- 6. Spark Unifies: Batch Processing Interactive SQL Real-time processing Machine Learning Deep Learning Graph Processing An unified, open source, parallel, data processing framework for Big Data Analytics Spark Core Engine Spark SQL Batch processing Spark Structured Streaming Stream processing Spark MLlib Machine Learning Yarn Spark MLlib Machine Learning Spark Streaming Stream processing GraphX Graph Computation http://spark.apache.org Apache Spark
- 7. Data Sources (HDFS, SQL, NoSQL, …) Cluster Manager Node Node Node Cache Cache Cache Driver Program SparkContext General Spark Cluster Architecture • ‘Driver’ runs the user’s ‘main’ function and executes the various parallel operations on the worker nodes. • The results of the operations are collected by the driver • The worker nodes read and write data from/to Data Sources including HDFS. • Worker node also cache transformed data in memory as RDDs (Resilient Data Sets). • Worker nodes and the Driver Node execute as VMs in public clouds (AWS, Google and Azure).
- 8. Elements of Spark User’s Program Spark Session Task1 Task 2 Task 3 Task 4 Driver Executor Cluster Manager
- 9. Read from HDFS Write to HDFS Read from HDFS Write to HDFS Read from HDFS What makes Spark fast
- 10. DataFrame as the core of Spark • Recipe: • Create Session • Create Dataframe • Define a user defined function • Manipulate and view Data CSV Data JSON Data RDBMS Data Parquet Data Binary Data DataFrame User programs against the DataFrame abstraction
- 11. What about .NET? • In a recent survey, more than 70% of .NET devs expressed interest in Apache Spark • Millions of lines of big data-usable business logic are written in .NET • But .NET devs are locked out from big data processing – lack of .NET support in OSS big data solutions • We want a first-class .net experience in Spark
- 12. .NET for Apache Spark • .NET bindings (C# e F#) to Spark • Written on the Spark interop layer, designed to provide high performance bindings to multiple languages • Re-use knowledge, skills, code you have as a .NET developer • Compliant with .NET Standard • You can use .NET for Apache Spark anywhere you write .NET code • Original project Moebius • https://github.com/microsoft/Mobius
- 13. .NET Spark support Spark DataFramews with SparkSQL • Spark 2.3.x, 2.4.x, 3.0 • ~300 SparkSQL function • DeltaLake .NET Standard 2.0 • C#/F# • .NET Framework 4.6.1+ • .NET Core 2.1+ Batch&Streaming • Structured Streaming Data Science • ML.NET • Notebooks
- 14. Using .NET for Spark • Get started with .NET for Apache Spark | Microsoft Docs • https://docs.microsoft.com/en- us/dotnet/spark/tutorials/get- started?tabs=windows • Install .NET • Install Java • Install Apache Spark • Install .NET for Apache Spark • Create your app • Install NuGet package
- 15. Batch vs. Notebooks • Work on slow data stored into a Datalake • Submit a complete app in one single deploy • Receive the entire output • Use «spart-submit» • Run cell by cell (but also all at once)
- 17. Evolution of REPL • At the beginning there was mono • Then Dynamic/DLR (C# 4) • C#/F# interactive • .NET Try • In a world of: • Python • Mathematica
- 18. Jupyter • Evolution and generalization of the seminal role of Mathematica (notebook) • +Python adoption (ipynb) • +Web (HTTP+Markdown) • +Kernel
- 19. Demo
- 21. Run Spark with .NET in a container • Not an official Microsoft image • https://hub.docker.com/r/3rdman/dotnet-spark • Install nothing on your machine other than docker • Launch and Debug your code from Visual Studio and Visual Studio Code • Very good for development
- 22. Demo
- 23. The Azure Synapse Analytics Experience
- 24. Engine for business-changing insights with seamless ecosystem integration Azure Synapse Analytics Data integration Data warehousing Big data processing Azure Data Lake Storage + Common Data Model
- 25. Azure Synapse Analytics Limitless analytics service with unmatched time to insight Synapse Analytics Platform Azure Data Lake Storage Common Data Model Enterprise Security Optimized for Analytics Data lake integrated and Common Data Model aware METASTORE SECURITY MANAGEMENT MONITORING Integrated platform services for, management, security, monitoring, and meta-store DATA INTEGRATION Analytics Runtimes Integrated analytics runtimes available provisioned and serverless on-demand Synapse SQL offering T-SQL for batch, streaming and interactive processing Apache Spark for big data processing with Python, Scala and .NET PROVISIONED ON-DEMAND Form Factors SQL Languages Python .NET Java Scala R Multiple languages suited to different analytics workloads Experience Synapse Analytics Studio SaaS developer experiences for code free and code first Artificial Intelligence / Machine Learning / Internet of Things Intelligent Apps / Business Intelligence Designed for analytics workloads at any scale METASTORE SECURITY MANAGEMENT MONITORING
- 26. Manage – Apache Spark pools • Overview • Provides ability to Pause, Scale, Assign Tags and upload packages from Studio.
- 28. Spark ML Algorithms Spark ML Algorithms
- 29. Synapse Service Job Service Frontend Spark API Controller … Job Service Backend Spark Plugin Gateway Resource Provider DB Synapse Studio AAD Auth Service Instance Creation Service DBDB Azure Spark Instance VM VM VM VM VM … VM Synapse Job Service
- 30. Develop Hub - Notebooks • Notebooks • Allows to write multiple languages in one notebook • %%<Name of language> • Offers use of temporary tables across languages • Language support for Syntax highlight, syntax error, syntax code completion, smart indent, code folding • Export results
- 31. Demo
- 32. Conclusion Add a footer 32
- 33. Add a footer 33 Conclusion • Spark is here to stay • Now it is a place for .NET skills too • Azure Synapse Analytics the best fusion between the old and the new world