Introducing:

Self-service data prep

with dataflows
November 2018

Adi Regev , Principal GPM

Anton Fritz, Senior PM
Achieving the digital feedback loop

Empowered employees

Employees

Transformed
Intelligent Data products and
customer Customers + Products optimized
engagement Intelligence
operations
The modern business intelligence challenges

Fragmented, incomplete data Requires a team of specialists Data preparation – the most time-
consuming task in analytics BI.
Pulling together data from traditional and Creating E2E BI solutions requires
cloud data sources and figuring out how multiple BI tools. This requires specific
to enrich it is extremely difficult.
knowledge of each of the tools and
complex integration to build and
maintain an E2E BI solution. Complex system design and
architecture

Traditional BI solutions require planning

and designing in advance. Items like table
structure and refresh order are complex
to design and implement.

Business data has no structural or Complex system integration

semantic consistency
Traditional BI solutions span multiple
Different applications, departments, and applications and services. Sharing data
analysts define data in different ways, across systems requires each system to
which makes data exploration, and reuse understand the location, structure and
of data and apps extremely challenging. meaning of the data
Traditional BI platform

Data visualization

BI modeling

Gateways and connectors

BI platform is evolving

Data visualization

BI modeling

Data prep

Gateways and connectors

 Power BI introduces self-service data-prep
capabilities
Cloud and on-premises
Self-service low code/no code In-lake transformations
connectors

Standard schema
Dataflow
Data reuse Integral part of Power BI stack
(Common Data Model)

Incremental refresh Excel like -auto re-calc

Power BI introduces dataflows
Reports & dashboards
Visualizations

Datasets
BI models

Dataflows
Data prep

Storage
Data (Azure Data Lake)

Gateways and connectors

Self-service demo walkthrough
Getting data into centralized storage leveraging dataflows
Demo recap

Sales data
Dataflow

• Clean and transformed data is

ingested into the data lake

• Data is in Common Data Model

format

• Anyone with the right

permissions can leverage
the data
Self-service demo walkthrough
Reuse data and perform in-lake computation
Demo recap

1 2 3
Sales data Sales with service calls Power BI app
Dataflow Dataflow

• Create dataflows with dataflow • Reuse data from others in your Connect with Power BI Desktop to
editor in Power BI department or other departments create models, reports, and
dashboards using dataflow data.
• Perform transformations and data • Perform in-lake computations
cleansing using Power Query
Online • Keep data consistent with smart re-
calc logic
• Map data to the Common Data
Model

• Schedule a full or incremental

refresh
The modern BI challenge

Fragmented, incomplete data Requires a team of specialists

Pulling together data from traditional and Creating E2E BI solutions requires
cloud data sources, and figuring out how multiple BI tools. This requires specific
to enrich it is extremely difficult. knowledge of each of the tools and
complex integration to build and
maintain an E2E BI solution.

Business data has no structural or Complex system integration

semantic consistency
Traditional BI solutions span multiple
Different applications, departments, and applications and services. Sharing data
analysts define data in different ways, across systems requires each system to
which makes data exploration, and reuse understand the location, structure and
of data and apps extremely challenging. meaning of the data.
Recap of what we saw:

Power BI users can now easily create dataflows to

Reports & dashboards prepare data in a centralized storage, using a
standardized schema, ready for easy consumption,
reuse, and generation of business insights
Datasets
• Data prep capabilities in Power BI

• The dataflows that are created can be managed in PowerBI.com

Dataflows
• Create entities that conform to the Common Data Model or entities
with custom schema

• Data in the Data Lake is seamlessly integrated with Power BI

Storage
artifacts and is available for consumption in Power BI Desktop

• Available to Pro and Premium customers

But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in applications or services?

Datasets

Dataflows

Storage
Your data
Azure integration
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

1. Provide Power BI
with details of the
Data Lake Storage
account
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

2. Enable workspace
Admins to assign
workspaces to the
storage account
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

3. Workspace Admins
can assign
workspaces to the
storage account,
Dataflows and
Dataflow data will be
stored there.
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

Just provide Power BI

with details of the Data
Lake Storage account,
and your organization's
dataflows will be stored
there
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder CDM folder CDM folder

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Enrich data using Azure services

Enriched
Sales with service call center

• CDM folder created with

dataflow in Power BI can be
processed and enriched with
Azure services

• For example, perform

sentiment analysis on service
calls data
But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in other applications or services?

Or to provide an easy way to consume analytical data

Datasets created outside of Power BI, in Power BI?

Dataflows

Storage
Use case: Dynamics 365 Finance & Operations data in
Power BI

Self service
Dynamics 365 for
customizations
Finance & Operations
in Power BI

Azure Data Azure Azure SQL Azure ML

Dataflow Dynamics 365 data Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder
But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in other applications or services?

Or to provide an easy way to consume analytical data

Datasets created outside of Power BI, in Power BI?

Dataflows

Storage
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Custom LOB
+
Datasets Developer
resources

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder CDM folder CDM folder

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Programmability
1 2 3
Power BI APIs Model file SDK Documentation

• We’ve added new web APIs to • A variety of libraries and sample • Documentation for APIs, model
Power BI to help enable many code will be available to enable file creation and working with
scenarios involving dataflows. programming scenarios. dataflows will be available at
• You can now programmatically • You can validate, create, or read launch.
import Power BI dataflows or add CDM model files and data • Step-by-step guides for key
a reference to external dataflows programmatically. dataflow scenarios will be
in Azure storage. • You can easily read CDM Model available as well.
• You can connect datasets to files from Azure Databricks or write
dataflows, refresh dataflows, Databricks tables to dataflows.
update their refresh schedules,
and more.
 Questions?

© Copyright Microsoft Corporation. All rights reserved.