Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Global Azure Norway 2021)

Apr 16, 20213 likes2,039 views

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Presented at Global Azure Norway on April 16th, 2021)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines & Data Flows:
Introduction to Data Integration
in Azure Synapse Analytics
Cathrine Wilhelmsen
Global Azure Norway · April 16th, 2021

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Session Abstract
Do you regularly need to get data for your projects?
Yep! 🙋‍♀️
Data is at the core of every Business Intelligence, Data Science, and Machine Learning project.
You need data to understand what has happened in the past, to predict what may happen in
the future, to discover patterns and anomalies, and to gain the insight necessary for making
faster and better decisions.
But before you can do any of those things, you need to ingest, store, transform, integrate, and
prepare your data. Guess what? You can do all of those things in Azure Synapse Analytics –
without having to write any code!
In this session, we will cover the fundamentals of data integration in Azure Synapse Analytics.
First, we will discuss when Azure Synapse Analytics is the right tool of choice. Then, we will go
through what Pipelines and Data Flows are, and when to use them. Finally, we will see how
easy it is to ingest and transform data both on-premises and in the cloud.

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Store
Transform
Integrate
Prepare

© 2020 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse Analytics
What?
Pipelines & Data Flows
How?
Data Integration
When?
…the next 45 minutes…

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
All-in-one platform for analytical projects:
• Data Lake (All Data)
• Data Warehouse (Relational Data)
• Data Analytics (SQL, Spark)
• Data Integration (Ingest, Transform)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Synapse Analytics?
Deeply integrated with other services:
• Azure Purview
• Azure Machine Learning
• Azure Cosmos DB
• Power BI

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Who can use Azure Synapse Analytics?
Built for collaboration between:
• Data Engineers
• Data Analysts
• Data Scientists
• Data Consumers

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Integration
Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Integration in Azure Synapse Analytics
Copy Data Transform Data

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Pipelines?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Activities?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Datasets?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Linked Services?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flows?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Triggers?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Code-First
Scripts, Notebooks
Designer-First
Pipelines, Data Flows

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure Synapse or Azure Data Factory?

© 2021 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
cathrinew.net/adf

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra

Building a modern data warehouseJames Serra

Embarking on building a modern data warehouse in the cloud can be an overwhelming experience due to the sheer number of products that can be used, especially when the use cases for many products overlap others. In this talk I will cover the use cases of many of the Microsoft products that you can use when building a modern data warehouse, broken down into four areas: ingest, store, prep, and model & serve. It’s a complicated story that I will try to simplify, giving blunt opinions of when to use what products and the pros/cons of each.

Azure Synapse Analytics Overview (r2)James Serra

Microsoft Fabric Intro D KoutsanastasisUni Systems S.M.S.A.

Azure data platform overviewJames Serra

This document provides an overview and summary of the author's background and expertise. It states that the author has over 30 years of experience in IT working on many BI and data warehouse projects. It also lists that the author has experience as a developer, DBA, architect, and consultant. It provides certifications held and publications authored as well as noting previous recognition as an SQL Server MVP.

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)Cathrine Wilhelmsen

Modernizing to a Cloud Data ArchitectureDatabricks

Organizations with on-premises Hadoop infrastructure are bogged down by system complexity, unscalable infrastructure, and the increasing burden on DevOps to manage legacy architectures. Costs and resource utilization continue to go up while innovation has flatlined. In this session, you will learn why, now more than ever, enterprises are looking for cloud alternatives to Hadoop and are migrating off of the architecture in large numbers. You will also learn how elastic compute models’ benefits help one customer scale their analytics and AI workloads and best practices from their experience on a successful migration of their data and workloads to the cloud.

Introduction to Azure Data LakeAntonios Chatzipavlis

This document provides an introduction and overview of Azure Data Lake. It describes Azure Data Lake as a single store of all data ranging from raw to processed that can be used for reporting, analytics and machine learning. It discusses key Azure Data Lake components like Data Lake Store, Data Lake Analytics, HDInsight and the U-SQL language. It compares Data Lakes to data warehouses and explains how Azure Data Lake Store, Analytics and U-SQL process and transform data at scale.

Introduction to Azure DatabricksJames Serra

Databricks is a Software-as-a-Service-like experience (or Spark-as-a-service) that is a tool for curating and processing massive amounts of data and developing, training and deploying models on that data, and managing the whole workflow process throughout the project. It is for those who are comfortable with Apache Spark as it is 100% based on Spark and is extensible with support for Scala, Java, R, and Python alongside Spark SQL, GraphX, Streaming and Machine Learning Library (Mllib). It has built-in integration with many data sources, has a workflow scheduler, allows for real-time workspace collaboration, and has performance improvements over traditional Apache Spark.

AWS Lake Formation Deep DiveCobus Bernard

This document provides an overview of AWS Lake Formation and related services for building a secure data lake. It discusses how Lake Formation provides a centralized management layer for data ingestion, cleaning, security and access. It also describes how Lake Formation integrates with services like AWS Glue, Amazon S3 and ML transforms to simplify and automate many data lake tasks. Finally, it provides an example workflow for using Lake Formation to deduplicate data from various sources and grant secure access for analysis.

DW Migration Webinar-March 2022.pptxDatabricks

The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.

ADF Mapping Data Flows Training Slides V1Mark Kromer

Mapping Data Flow is a new feature of Azure Data Factory that allows users to build data transformations in a visual interface without code. It provides a serverless, scale-out transformation engine to transform data at scale in the cloud in a resilient manner for big data scenarios involving unstructured data. Mapping Data Flows can be operationalized with Azure Data Factory's scheduling, control flow, and monitoring capabilities.

Azure Synapse AnalyticsWinWire Technologies Inc

ADF Demo_ppt.pptxvamsytaurus

Azure Data Factory (ADF) is a cloud-based data integration service that allows users to easily construct ETL and ELT processes through a code-free visual interface or custom code. ADF can connect to both cloud and on-premises data sources, support data transformation, and also run existing SSIS packages that have been migrated to the cloud. Key components of ADF include storage accounts, containers, linked services, datasets, data pipelines, triggers, and data flows which allow users to move, transform and process data.

Learn to Use Databricks for Data ScienceDatabricks

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Azure Data Factory V2; The Data FlowsThomas Sykes

The document discusses Azure Data Factory V2 data flows. It will provide an introduction to Azure Data Factory, discuss data flows, and have attendees build a simple data flow to demonstrate how they work. The speaker will introduce Azure Data Factory and data flows, explain concepts like pipelines, linked services, and data flows, and guide a hands-on demo where attendees build a data flow to join customer data to postal district data to add matching postal towns.

Azure DataBricks for Data Engineering by Eugene PolonichkoDimko Zhluktenko

This document provides an overview of Azure Databricks, a Apache Spark-based analytics platform optimized for Microsoft Azure cloud services. It discusses key components of Azure Databricks including clusters, workspaces, notebooks, visualizations, jobs, alerts, and the Databricks File System. It also outlines how data engineers can leverage Azure Databricks for scenarios like running ETL pipelines, streaming analytics, and connecting business intelligence tools to query data.

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...DataScienceConferenc1

Modern Data Warehousing with the Microsoft Analytics Platform SystemJames Serra

The Microsoft Analytics Platform System (APS) is a turnkey appliance that provides a modern data warehouse with the ability to handle both relational and non-relational data. It uses a massively parallel processing (MPP) architecture with multiple CPUs running queries in parallel. The APS includes an integrated Hadoop distribution called HDInsight that allows users to query Hadoop data using T-SQL with PolyBase. This provides a single query interface and allows users to leverage existing SQL skills. The APS appliance is pre-configured with software and hardware optimized to deliver high performance at scale for data warehousing workloads.

Databricks Platform.pptxAlex Ivy

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

Azure Data Factory Data FlowMark Kromer

Azure Data Factory Mapping Data Flow allows users to stage and transform data in Azure during a limited preview period beginning in February 2019. Data can be staged from Azure Data Lake Storage, Blob Storage, or SQL databases/data warehouses, then transformed using visual data flows before being landed to staging areas in Azure like ADLS, Blob Storage, or SQL databases. For information, contact [email protected] or visit http://aka.ms/dataflowpreview.

Azure data analytics platform - A reference architecture Rajesh Kumar

This document provides an overview of Azure data analytics architecture using the Lambda architecture pattern. It covers Azure data and services, including ingestion, storage, processing, analysis and interaction services. It provides a brief overview of the Lambda architecture including the batch layer for pre-computed views, speed layer for real-time views, and serving layer. It also discusses Azure data distribution, SQL Data Warehouse architecture and design best practices, and data modeling guidance.

Databricks FundamentalsDalibor Wijas

This document is a training presentation on Databricks fundamentals and the data lakehouse concept by Dalibor Wijas from November 2022. It introduces Wijas and his experience. It then discusses what Databricks is, why it is needed, what a data lakehouse is, how Databricks enables the data lakehouse concept using Apache Spark and Delta Lake. It also covers how Databricks supports data engineering, data warehousing, and offers tools for data ingestion, transformation, pipelines and more.

Building End-to-End Delta Pipelines on GCPDatabricks

Delta has been powering many production pipelines at scale in the Data and AI space since it has been introduced for the past few years. Built on open standards, Delta provides data reliability, enhances storage and query performance to support big data use cases (both batch and streaming), fast interactive queries for BI and enabling machine learning. Delta has matured over the past couple of years in both AWS and AZURE and has become the de-facto standard for organizations building their Data and AI pipelines. In today’s talk, we will explore building end-to-end pipelines on the Google Cloud Platform (GCP). Through presentation, code examples and notebooks, we will build the Delta Pipeline from ingest to consumption using our Delta Bronze-Silver-Gold architecture pattern and show examples of Consuming the delta files using the Big Query Connector.

Data Mesh for DinnerKent Graziano

Data mesh is a decentralized approach to managing and accessing analytical data at scale. It distributes responsibility for data pipelines and quality to domain experts. The key principles are domain-centric ownership, treating data as a product, and using a common self-service infrastructure platform. Snowflake is well-suited for implementing a data mesh with its capabilities for sharing data and functions securely across accounts and clouds, with built-in governance and a data marketplace for discovery. A data mesh implemented on Snowflake's data cloud can support truly global and multi-cloud data sharing and management according to data mesh principles.

Differentiate Big Data vs Data Warehouse use cases for a cloud solutionJames Serra

It can be quite challenging keeping up with the frequent updates to the Microsoft products and understanding all their use cases and how all the products fit together. In this session we will differentiate the use cases for each of the Microsoft services, explaining and demonstrating what is good and what isn't, in order for you to position, design and deliver the proper adoption use cases for each with your customers. We will cover a wide range of products such as Databricks, SQL Data Warehouse, HDInsight, Azure Data Lake Analytics, Azure Data Lake Store, Blob storage, and AAS as well as high-level concepts such as when to use a data lake. We will also review the most common reference architectures (“patterns”) witnessed in customer adoption.

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen

More Related Content

What's hot (20)

Modernizing to a Cloud Data ArchitectureDatabricks

Introduction to Azure Data LakeAntonios Chatzipavlis

Introduction to Azure DatabricksJames Serra

AWS Lake Formation Deep DiveCobus Bernard

DW Migration Webinar-March 2022.pptxDatabricks

ADF Mapping Data Flows Training Slides V1Mark Kromer

Azure Synapse AnalyticsWinWire Technologies Inc

ADF Demo_ppt.pptxvamsytaurus

Learn to Use Databricks for Data ScienceDatabricks

Azure Data Factory V2; The Data FlowsThomas Sykes

Azure DataBricks for Data Engineering by Eugene PolonichkoDimko Zhluktenko

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...DataScienceConferenc1

Modern Data Warehousing with the Microsoft Analytics Platform SystemJames Serra

Databricks Platform.pptxAlex Ivy

Azure Data Factory Data FlowMark Kromer

Azure data analytics platform - A reference architecture Rajesh Kumar

Databricks FundamentalsDalibor Wijas

Building End-to-End Delta Pipelines on GCPDatabricks

Data Mesh for DinnerKent Graziano

Differentiate Big Data vs Data Warehouse use cases for a cloud solutionJames Serra

Modernizing to a Cloud Data ArchitectureDatabricks

Introduction to Azure Data LakeAntonios Chatzipavlis

Introduction to Azure DatabricksJames Serra

AWS Lake Formation Deep DiveCobus Bernard

DW Migration Webinar-March 2022.pptxDatabricks

ADF Mapping Data Flows Training Slides V1Mark Kromer

Azure Synapse AnalyticsWinWire Technologies Inc

ADF Demo_ppt.pptxvamsytaurus

Learn to Use Databricks for Data ScienceDatabricks

Azure Data Factory V2; The Data FlowsThomas Sykes

Azure DataBricks for Data Engineering by Eugene PolonichkoDimko Zhluktenko

[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...DataScienceConferenc1

Modern Data Warehousing with the Microsoft Analytics Platform SystemJames Serra

Databricks Platform.pptxAlex Ivy

Azure Data Factory Data FlowMark Kromer

Azure data analytics platform - A reference architecture Rajesh Kumar

Databricks FundamentalsDalibor Wijas

Building End-to-End Delta Pipelines on GCPDatabricks

Data Mesh for DinnerKent Graziano

Differentiate Big Data vs Data Warehouse use cases for a cloud solutionJames Serra

Similar to Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Global Azure Norway 2021) (20)

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Cathrine Wilhelmsen

Analytics in a Day Virtual WorkshopCCG

Azure synapse by usama whaba khanUsama Wahab Khan Cloud, Data and AI

This document provides an overview of Azure Synapse Analytics and its key capabilities. Azure Synapse Analytics is a limitless analytics service that brings together enterprise data warehousing and big data analytics. It allows querying data on-demand or at scale using serverless or provisioned resources. The document outlines Synapse's integrated data platform capabilities for business intelligence, artificial intelligence and continuous intelligence. It also describes the different types of analytics workloads that Synapse supports and key architectural components like the dedicated SQL pool and massively parallel processing concepts.

Analytics in a Day Ft. Synapse Virtual WorkshopCCG

Customer Migration to Azure SQL Database_2024.pdfGeorge Walters

New ways to apply infrastructure data for better business outcomesaccenture

This document discusses how Accenture modernized its IT infrastructure data platform by migrating to Microsoft Azure Cloud Data Services. Some key points: - Accenture wanted to revolutionize how it managed and analyzed infrastructure data to gain better insights and make more informed decisions. - It chose Microsoft Azure for its scalability, security, and ability to offer a cloud-native solution with minimal impact. - Migrating to Azure has provided numerous benefits like reduced costs, lower carbon footprint, improved analytics capabilities, and increased consistency and control. - Over 15 applications were migrated, processing over 600TB of data monthly while achieving 99.99% uptime. Insights from infrastructure data are now helping Acc

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Cathrine Wilhelmsen

The document is a presentation on Azure Data Factory that discusses what it is, when and why it would be used, and how to work with it. It defines Azure Data Factory as a data integration service that can copy and transform data. It demonstrates how to use Azure Data Factory to copy data between cloud and on-premises data stores, transform data using mapping and wrangling data flows, and schedule data pipelines using triggers. Common data architectures that use Azure Data Factory are also presented.

Learning Azure Synapse Analytics (Third Early Release) Paul Andrewmoneamramia

Ingesting Click Data for AnalyticsClickMeter

Ingesting click events for analyticsData Driven Innovation

Francesco Furiani - Marketing is a serious business, moreover tracking and monetizing the campaign that allows your marketing to flourish is very important: our tool allows anyone to monitor, compare and optimize all those campaigns (delivered via links) in one place and to deliver insights about who's using those links. Making this infrastructure, making it works, deliver results in real-time (when necessary) and keep everyone happy from the customer to the CFO will be the point of this talk, from the design to the final result with an eye on the costs/risks/benefits of having everything in the cloud.

1 Introduction to Microsoft data platform analytics for releaseJen Stirrup

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Cathrine Wilhelmsen

This document summarizes a presentation on Azure Data Factory (ADF) given by Cathrine Wilhelmsen. The presentation provided an overview of ADF and how it compares to SQL Server Integration Services (SSIS). It demonstrated how to lift and shift existing SSIS packages to ADF and how to map data flows between different data stores using ADF data flows. The presentation concluded with lessons learned and encouraged attendees to ask any remaining questions.

Harnessing Microsoft Fabric and Azure Service Fabric Analytics as a Service a...Microsoft Dynamics

Smarter Analytics: Supporting the Enterprise with AutomationInside Analysis

The Briefing Room with Barry Devlin and WhereScape Live Webcast on June 10, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=5230c31ab287778c73b56002bc2c51a The data warehouse is intended to support analysis by making the right data available to the right people in a timely fashion. But conditions change all the time, and when data doesn’t keep up with the business, analysts quickly turn to workarounds. This leads to ungoverned and largely un-managed side projects, which trade short-term wins for long-term trouble. One way to keep everyone happy is by creating an integrated environment that pulls data from all sources, and is capable of automating both the model development and delivery of analyst-ready data. Register for this episode of The Briefing Room to hear data warehousing pioneer and Analyst Barry Devlin as he explains the critical components of a successful data warehouse environment, and how traditional approaches must be augmented to keep up with the times. He’ll be briefed by WhereScape CEO Michael Whitehead, who will showcase his company’s data warehousing automation solutions. He’ll discuss how a fast, well-managed and automated infrastructure is the key to empowering faster, smarter, repeatable decision making. Visit InsideAnlaysis.com for more information.

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Cathrine Wilhelmsen

This document discusses Azure Data Factory (ADF) and how it can be used to build and orchestrate data pipelines without code. It describes how ADF is a hybrid data integration service that improves on its previous version. It also explains how existing SSIS packages can be "lifted and shifted" to ADF to modernize solutions while retaining investments. The document demonstrates creating pipelines and data flows in ADF, handling schema drift, and best practices for development.

Deep architectural competency for deploying azure solutionsSynergetics Learning and Cloud Consulting

This document outlines an educational program on deploying Azure solutions that includes webinars and onsite programs. The objectives are to transform partners' delivery organizations to be competent in addressing Azure opportunities. The program covers various technical topics through different methodologies, including webinars on application modernization, big data, DevOps, and datacenter modernization. It provides details on speaker profiles, participant prerequisites, and intended outcomes of increasing Azure capabilities.

Cortana Intelligence SolutionsDarwin Schweitzer

This document introduces Cortana Intelligence Solutions, which provides intelligent, interactive dashboards and proven solution architectures to help organizations transform data into insights. It highlights the business potential of big data and analytics, then demonstrates a Twitter time series analysis using Azure Time Series Insights. The document provides information on Cortana Intelligence Solutions and links to learn more, try sample solutions, deploy solutions, customize deployments, and give feedback.

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen

Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Cathrine Wilhelmsen

Analytics in a Day Virtual WorkshopCCG

Azure synapse by usama whaba khanUsama Wahab Khan Cloud, Data and AI

Analytics in a Day Ft. Synapse Virtual WorkshopCCG

Customer Migration to Azure SQL Database_2024.pdfGeorge Walters

New ways to apply infrastructure data for better business outcomesaccenture

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Cathrine Wilhelmsen

Learning Azure Synapse Analytics (Third Early Release) Paul Andrewmoneamramia

Ingesting Click Data for AnalyticsClickMeter

Ingesting click events for analyticsData Driven Innovation

1 Introduction to Microsoft data platform analytics for releaseJen Stirrup

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Cathrine Wilhelmsen

Harnessing Microsoft Fabric and Azure Service Fabric Analytics as a Service a...Microsoft Dynamics

Smarter Analytics: Supporting the Enterprise with AutomationInside Analysis

Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Cathrine Wilhelmsen

Deep architectural competency for deploying azure solutionsSynergetics Learning and Cloud Consulting

Cortana Intelligence SolutionsDarwin Schweitzer

More from Cathrine Wilhelmsen (20)

Fra utvikler til arkitekt: Skap din egen karrierevei ved å utvikle din person...Cathrine Wilhelmsen

One Year in Fabric: Lessons Learned from Implementing Real-World Projects (PA...Cathrine Wilhelmsen

Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen

Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...Cathrine Wilhelmsen

Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)Cathrine Wilhelmsen

The document is about how the author Cathrine Wilhelmsen built her own website analytics dashboard using Microsoft Fabric and Power BI. She collects data from the Cloudflare API and stores it in Microsoft Fabric. This allows her to visualize and access the analytics data on her phone through a mobile app beyond the 30 days retention offered by Cloudflare. In her presentation, she demonstrates how she retrieves the website data, processes it with Microsoft Fabric pipelines, and visualizes it in Power BI for a self-hosted analytics solution.

Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...Cathrine Wilhelmsen

Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)Cathrine Wilhelmsen

Cathrine Wilhelmsen gave a presentation on using Microsoft Data Factory for data integration within Microsoft Fabric. Data Factory allows users to define data pipelines to ingest, transform and orchestrate data workflows. Pipelines contain activities that can copy or move data between different data stores. Connections specify how to connect to these data stores. Dataflows Gen2 provide enhanced orchestration capabilities, including defining activity dependencies and schedules. The presentation demonstrated how to use these capabilities in Data Factory for complex data integration scenarios.

The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)Cathrine Wilhelmsen

Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Cathrine Wilhelmsen

Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Cathrine Wilhelmsen

Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)Cathrine Wilhelmsen

The document is about how the author created a mobile-friendly dashboard for her website analytics using Microsoft Fabric and Power BI. She collects data from the Cloudflare API and stores it in Microsoft Fabric. Then she visualizes the data in Power BI which can be viewed on her phone. This allows her to track website traffic and see which pages are most popular over time. She demonstrates her dashboard and discusses future improvements like comparing statistics across different time periods.

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Cathrine Wilhelmsen

Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023) We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout. The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits. I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...Cathrine Wilhelmsen

"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023) Do you sometimes think the world is moving so fast that you're struggling to keep up? Does it make you feel a little uncomfortable? Awesome! That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it. It just might not be easy. All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning. Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Cathrine Wilhelmsen

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022) What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights? We had no idea, but we gave it a go! Our ambition was to think big, start small, and act fast – to deliver business value early and often. Did we succeed? Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.

6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)Cathrine Wilhelmsen

6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022) Do you feel nervous about getting on stage to deliver a presentation? That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed? There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice. You can do this!

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Cathrine Wilhelmsen

Azure Data Factory for the SSIS Developer (SentryOne Webinar)Cathrine Wilhelmsen

Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Cathrine Wilhelmsen

Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Cathrine Wilhelmsen