New capabilities for modern data integration in the cloud

Modern DW for BI
Modern DW for SaaS
Lift & Shift Existing SSIS Packages to Cloud

Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure storage
Polybase
Azure SQL Data Warehouse
Data factory
Data factory
Azure Databricks
(Spark)
Analytical dashboards
(PowerBI)
Model & ServePrep & TrainStoreIngest Intelligence
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING
Azure Analysis ServicesOn Prem, Cloud
Apps & Data

Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure storage
Polybase
Data factory
Data factory
Azure Databricks
(Spark)
Browser/Device
Model & ServePrep & TrainStoreIngest Intelligence
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING
App Storage
SaaS App

On-Premise data sources
SQL DB Managed Instance
SQL Server
VNET
Azure Data Factory
Cloud data sources
Cloud
On-premises

Flexible Pipeline Model
Rich pipeline orchestration
Triggers: on-demand, schedule, event
Data Movement as a Service
Cloud, Hybrid
70+ connectors provided
SSIS Package Execution
In a managed cloud environment
Use familiar tools, SSMS & SSDT
Author & Monitor
Programmability (Python, .NET, Powershell, etc.)
Visual Tools

140 MillionPassengers are stranded
annually due to IROPS
IRREGUL AR OPERATIONS (IROPS) COST US AIRLINES
8.3 BillionIn revenue
USD
250Rebooking per passenger
from a cancelled flight
USD
4,000For rebooking
crew per airline
USD
GLOBALLY Passengers most frustrated by a lack of communication
51%
USD 16.7 Billion
Estimates cost of passengers’ time
lost due to IROPS
Loss in demand due to IROPS
USD 3.9 Billion
In annual costs to Airlines and customers
USD60Billion
Airline Pain Points
Poor Communication | Lost Revenue | High Re-accommodational Costs |
Upset Customers | Brand Damage
Airline IROPS (IRREGULAR OPERATIONS)
The Billion Dollar Problem Airlines Must Solve

Activity 1 Activity 2
Activity 3
“On Error”
Activity 1
params
params
My Pipeline 1
…
My Pipeline 2
For Each…
Activity 4
params
Trigger
Event
Wall Clock
On Demand
params

Scalable
Per job elasticity
Up to 1 GB/s
Simple
Visually author or via code (Python, .Net, etc)
Serverless, no infrastructure to manage
Access all your data
70+ connectors provided and growing (cloud, on premises, SaaS)
Data Movement as a Service: 20+ points of presence world wide
Self-hostable Integration Runtime for hybrid movement
Data Movement

Azure Database File Storage NoSQL Services and Apps Generic
Azure Blob Storage Amazon Redshift SQL Server Amazon S3 Couchbase Dynamics 365 Salesforce HTTP
Azure Data Lake
Store
Oracle MySQL File System Cassandra Dynamics CRM Salesforce Service
Cloud
OData
Azure SQL DB Netezza PostgreSQL FTP MongoDB SAP C4C ServiceNow ODBC
Azure SQL DW SAP BW SAP HANA SFTP Oracle CRM Hubspot
Azure Cosmos DB Google BigQuery Informix HDFS Oracle Service
Cloud
Marketo
Azure DB for
MySQL
Sybase DB2 SAP ECC Oracle Responsys
Azure DB for
PostgreSQL
Greenplum MariaDB Zendesk Oracle Eloqua
Azure Search Microsoft Access Drill Zoho CRM
Salesforce
ExactTarget
Azure Table
Storage
Hive Phoenix Amazon
Marketplace
Atlassian Jira
Azure File Storage Hbase Presto Megento Concur
Impala Spark PayPal QuickBooks Online
Vertica Shopify Xero
GE Historian Square
Web table
* Supported file formats: CSV, AVRO, ORC, Parquet, JSON

Data Movement Performance Optimization
Copy Scenario Behind the Scenes
Loading data into Azure Cosmos DB
• 10+ times throughput boost comparing to previous solution
• Scalable to utilize 100% Request Units (Rus) during ingestion
• Improved reliability
Loading from Azure Blob or ADLS into
SQL DW
Polybase used whenever possible. BULKINSERT otherwise
Applicable when format is text, ORC, Parquet and meets these criteria.
Loading from data sources other than
Azure Blob/ADLS into SQL DW
Use staged copy via Azure Blob
Loading data from Amazon Redshift
Use UNLOAD to copy data from Amazon Redshift to Amazon S3

Data Movement Security
Command
Channel
Data
Channel
Data
Channel
Local

On-premises data sources SQL Server
 OS: Windows/Linux
 SCALABILITY: Scale-Out feature
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS to design/deploy/
manage/execute/monitor packages
 EXTENSIBILITY: ISVs can build
components/extensions on SSIS
 PRICING: Bundled w/ on-prem SQL Server

On-premises data sources
Cloud data sources Azure SQL DB/Managed InstanceAzure Data Factory
Cloud
On-premises
SQL Server
 LIFT & SHIFT: Use Azure SQL DB/Managed
Instance (MI) to host SSISDB
 SCALABILITY: Use ADF to provision a
managed cluster of Azure VMs dedicated to
run your packages – Azure-SSIS Integration
Runtime (IR)
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS + ADF app to
design/deploy/manage/execute/monitor
packages (activities)
 EXTENSIBILITY: ISVs can build
components/extensions + SaaS on SSIS in
ADF via custom setup + 3rd party licensing
 PRICING: Pay per hour + Azure Hybrid Benefit
(AHB) to Bring Your Own License (BYOL)

On-premises data sources
Azure SQL DB/Managed Instance
VNet
Azure Data FactoryCloud data sources
Cloud
On-premises
SQL Server
 HYBRID: Join Azure-SSIS IR to a VNet that is
connected to your on-prem network to
enable on-prem data access
 MODERNIZATION: Schedule first-class SSIS
activities in ADF pipelines via SSMS and
chain/group them w/ other activities via ADF
app
 COMPLEMENTARY: Splice/inject built-in/3rd
party SSIS tasks and transformations in ADF
pipelines
 READINESS: Public Preview w/ 24/7 live-site
support

Azure-SSIS IR node
Container
ISV Setup1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key

Area Details
Control Flow • Control Flow-Released!!
• Azure Data Bricks Integration-Released!!
• File arrival trigger-Released!!
• Schedule roll up
• 20 plus activities and growing
Data Transformation Visual author data transformation executed with Spark
Data Movement • More connectors (Hybrid data movement)
• More geographic regions
• Continue performance Improvements (10x increase for CosmosDB connector)-Released!!
SSIS • More Region and VM Choices
• Enterprise Edition-Released!!
• Azure Hybrid Benefit (BYOL)-Released!!
• Custom Setup + 3rd Party Licensing/Extensibility/Ecosystem-Released!!
• First-Class SSIS Activities in ADF Pipelines-Released!!
Tools Visual Control flow, data movement and monitoring-Released!!
Programmatic Interfaces (.NET, Python, Powershell, Rest APIs, ARM templates)-Released!!
Source Control Integration- VSTS GIT & GitHub
Monitoring Single pane of glass to monitor & manage your pipelines

Create ADF if you have not
done so already

Click on “Configure SSIS
Integration Runtime”

Name and describe
your Azure-SSIS IR

Select its location
 Select the right location for your Azure-SSIS IR to achieve high
performance in ETL workflows – East US/East US 2/Central US/West
US 2/North Europe/West Europe/UK South/Australia East are
available in Public Preview, more will be provided in the future
 It needs not be the same as the location of your ADF, but it should
be the same as the location of your own Azure SQL DB/MI server
where SSISDB will be hosted and or the location of VNet connected
to your on-prem network
 Avoid Azure-SSIS IR accessing SSISDB/data across different
locations

Select its node size
 A selection of node sizes specifying the number of cores (CPU) and
size of memory (RAM) per node is provided –
Standard_A4_v2/Standard_A8_v2/
Standard_D1_v2/Standard_D2_v2/Standard_D3_v2/Standard_D4_v2
are available in Public Preview, more will be provided in the future
 Select a large node size (scale up), if you want to run many
compute/ memory –intensive packages

Select its cluster size
 A range of 1-10 for the number of nodes per cluster is provided
 Select a large cluster with many nodes (scale out), if you want to
run many packages in parallel
 Like Azure VMs, you can manage the cost of running Azure-SSIS IR
by stopping/starting it, effectively releasing/reallocating its nodes,
as you see fit

Select its edition/license
 Standard/Enterprise edition/license is available in Public Preview
 Enterprise Edition lets you use advanced/premium features on your
Azure-SSIS IR

Select to use Azure Hybrid Benefit
(AHB)
 AHB lets you Bring Your Own License (BYOL) to benefit from
substantial cost savings – Essentially paying only for the
infrastructure and service management fees of Azure-SSIS IR

Bring your own Azure SQL DB/MI
server to host SSISDB
 A selection of your existing Azure SQL DB/MI servers in the same
location as your Azure-SSIS IR is provided – If there is none, create
a new one to ensure that your Azure-SSIS IR can easily access
SSISDB without incurring excessive traffics between different
locations
 The endpoint of your selected Azure SQL DB/MI server becomes
the connection info to enter on SSDT/SSMS

 If you select your existing Azure SQL DB w/ VNet service endpoints,
you must also join your Azure-SSIS IR to the same VNet, so we can
prepare and manage SSISDB on your behalf

 If you select your existing Azure SQL MI that was provisioned
inside a VNet, you must also join your Azure-SSIS IR to the same
VNet, but in a different subnet, so we can prepare and manage
SSISDB on your behalf

Select to use Azure Active
Directory (AAD) authentication
 AAD authentication w/ ADF Managed Service Identity (MSI) lets
you provision Azure-SSIS IR w/o Azure SQL DB/MI server admin
credentials – Need to add your ADF MSI into an AAD group that
has access permissions to your Azure SQL DB/MI server

Enter your Azure SQL DB/MI server
admin credentials
 Your Azure SQL DB/MI server admin username and password are
required for us to prepare and manage SSISDB on your behalf

Select the service tier and provide
access consent for your Azure SQL
DB server
 A selection of service tiers (e.g. Basic/S0/S1/etc.) for your Azure
SQL DB server is provided – Not applicable to Azure SQL MI
 Select a large service tier, if you want to achieve high database
performance and store many SSIS projects/packages/execution logs
 By default, you must allow Azure services to access your Azure SQL
DB server – Not applicable to Azure SQL MI/Azure SQL DB w/ VNet
service endpoints
 This is required for us to prepare and manage SSISDB on your behalf

Test connection to your Azure SQL
DB server
 Ensure that your Azure SQL DB server is accessible and has no
existing SSISDB for us to prepare and manage it on your behalf

Select the concurrency level of
your Azure-SSIS IR nodes
 A range of 1-8 for the number of parallel executions per node is
provided
 Select a low concurrency level, if you want to use more than one
cores to run a single large/compute-intensive package
 Select a high concurrency level, if you want to run one or more
small/light-weight packages in a single core

Add custom setup for your Azure-
SSIS IR
 Provide a Shared Access Signature (SAS) Uniform Resource
Identifier (URI) of your Azure Storage Blob container where your
setup script + its associated files are stored

Optionally join your Azure-SSIS IR
to a VNet and provide consent for
VNet configuration
 If you use Azure SQL MI that was provisioned inside a VNet/Azure SQL DB
w/ VNet service endpoints, you must also join your Azure-SSIS IR to the
same VNet, so we can prepare and manage SSISDB on your behalf
 If you have on-prem data sources/destinations in your SSIS packages, you
must join your Azure-SSIS IR to a VNet that is connected to your on-prem
network to enable on-prem data access
 By default, you must allow Azure services to configure the
permissions/settings of selected VNet for your Azure-SSIS IR to join

Select VNet and subnet for your
Azure-SSIS IR to join
 A selection of existing VNets and subnets in the location of your
Azure-SSIS IR is provided – ARM VNet is recommended, since
Classic VNet will be deprecated soon
 Ideally, your Azure-SSIS IR, Azure SQL DB/MI server, and VNet
connected to your on-prem network should all be in the same
location
 If that is not the case, you can first create your Azure-SSIS IR in the
location of Azure SQL DB/MI server and join it to a VNet in the same
location, then configure a VNet-to-VNet connection with the VNet
connected to your on-prem network

When you submit a
provisioning request, it
takes 20 – 30 minutes to
allocate/prepare the
nodes of your Azure-SSIS
IR and start billing

Once your Azure-SSIS IR
is started, you can deploy
SSIS packages to execute
on it and you can stop it
as you see fit

Before stopping your
Azure-SSIS IR, you need
to consider all of its
implications

When you stop your
Azure-SSIS IR, it takes 1 –
2 minutes to release the
allocated nodes and stop
billing

is stopped, you can
restart, edit, or delete it

When you edit your
Azure-SSIS IR, only
certain properties are
editable, and when you
finish, you can restart it

on premises
On SSMS, you can connect to SSIS PaaS
using its connection info and SQL/AAD
authentication credentials

on premises
On “Connection Properties” tab, enter
“SSISDB” as the target database to connect

on premises
in Azure
Once connected, you can deploy
projects/packages to SSIS PaaS from your
local file system/SSIS on premises

On Integration Services Deployment Wizard,
enter SSIS PaaS connection info and SQL
authentication credentials

Click on “Connections”
and ”Linked Services”

Click on “+ New” and
”Azure SQL Database”

Name and describe your
linked service

Select ADF Default Azure
IR to connect to your
Azure SQL DB that is
attached to Azure-SSIS IR

Select your Azure SQL DB
server and SSISDB to
connect

Enter your Azure SQL DB
server admin credentials

Test connection and save
your linked service

Click on “+” and
”Pipeline”

pipeline

Drag and drop “Stored
Procedure” activity into
your pipeline

sproc activity

Select your linked service
and test connection

Use system sproc
“sp_executesql” to
execute T-SQL script

Add a sproc parameter
“stmt” of type “string”
with your T-SQL script to
create/start SSIS package
execution using SSISDB
sprocs as its value

Click on “Test
Run”/“Trigger Now”

Drag and drop “Execute
SSIS Package” activity into
your pipeline

SSIS activity

Define the timeout/retry
counts/retry interval for
your SSIS activity

Select Azure-SSIS IR that
is attached to SSISDB
where your package is
deployed

Check if your package
requires 32-bit runtime to
execute

Select the
predefined/customized
scope of logging for your
package execution

Provide the path to
identify your deployed
package in SSISDB, e.g.
FolderName/ProjectName
/PackageName.dtsx

Provide the path to
identify your package
execution environment in
SSISDB, e.g.
FolderName/Environment
Name, assigning values to
SSIS project/package
parameters

Click on
“Validate”/“Trigger Now”

Click on “Trigger” and
”New/Edit”

Click on “Choose
trigger…” and ”+ New”

trigger

Select a trigger of type
“Schedule”

Select the start date,
recurrence, and end date
of your trigger

Finish and publish your
trigger

Click on “Triggers” to see
your published triggers

Right-click on your
package and select
“Schedule”

ADF
pipeline/activity/trigger
to schedule your package
executions will be
automatically generated

Assign your package
execution properties and
environment

Create your package
execution schedule just
like using SQL Server
Agent on premises

You can monitor/edit
your auto-generated ADF
pipeline/activity/trigger
on ADF app

Edit your auto-generated
ADF pipeline/activity

Edit your auto-generated
ADF trigger

is provisioned, you can
monitor it

You can click on the
status of your Azure-SSIS
IR

You can start/stop your
Azure-SSIS IR and copy its
resource ID

You can click on your
Azure SQL DB/MI server
endpoint

You can copy your Azure
SQL DB/MI server
endpoint and monitor
your SSISDB

You can monitor your
pipeline runs

activity runs and rerun
your pipeline

You can obtain SSISDB
operation/execution ID
from the output of your
activity run to check on
SSMS

trigger runs

Create Automation account if
you have not done so already

If the latest AzureRM.Profile and
AzureRM.DataFactoryV2
modules are not listed, click on
“Browse gallery”

Search for the required modules
and import them

Click on “+ Add a runbook”

Create PSH runbook to
start/stop your Azure-SSIS IR

Enter PSH script, then save and
publish your runbook

Enter runbook parameters to
start/stop your Azure-SSIS IR

Check “Output”/”All Logs”

Click on “Schedules” and ”+
Add a schedule”

Create a schedule for your
runbook

Click on “Webhooks” and ”+
Add Webhook”

Create webhooks for your
runbook and save your URLs

Create ADF pipeline with a web
activity

Enter the webhook URL to start
your Azure-SSIS IR and select
POST method

Add a sproc activity to execute
your package(s)

Add a web activity to stop your
Azure-SSIS IR on completion of
the sproc activity

Trigger/schedule your pipeline

Alternatively, if you use SSIS
activity for this, specify a
sufficient number of retry
attempts that are frequent
enough to wait until your
Azure-SSIS IR is started

Enterprise Features Descriptions
CDC components • CDC Source/Splitter/Control Task are preinstalled on your Azure-SSIS IR
Enterprise Edition
• To connect to Oracle, you will also need to install CDC Designer/Service on
another machine
Oracle connectors • Oracle Connection Manager/Source/Destination are preinstalled on your Azure-
SSIS IR Enterprise Edition
• You will also need to install Oracle Call Interface (OCI) driver, and if necessary
configure Oracle Transport Network Substrate (TNS), on your Azure-SSIS IR (via
Custom Setup Interface)
Teradata connectors • You will need to install Teradata Connection Manager/Source/Destination and
Teradata Parallel Transporter (TPT) API + Teradata ODBC driver on your Azure-
SSIS IR Enterprise Edition (via Custom Setup Interface)

SAP BW connectors • SAP BW Connection Manager/Source/Destination are preinstalled on your Azure-
SSIS IR Enterprise Edition
• You will also need to install SAP BW driver on your Azure-SSIS IR (via Custom
Setup Interface)
• These connectors support SAP BW 7.0 or earlier versions
• To connect to later versions or other SAP products, you can install SAP
connectors from 3rd party ISVs on your Azure-SSIS IR (via Custom Setup Interface)

SSAS/AAS components • Data Mining Model Training/Dimension Processing/Partition Processing Destinations and
Data Mining Query Transform are preinstalled on your Azure-SSIS IR Enterprise Edition
• All support SSAS, but only Partition Processing Destination supports AAS
• To connect to SSAS, you will also need to configure Windows Authentication credentials in
your SSISDB
• On top of these components, Analysis Services Execute DDL/Processing and Data Mining
Query Tasks are also preinstalled on your Azure-SSIS IR Standard/Enterprise Edition
Fuzzy Grouping/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data
Term Extraction/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data

Node Size Cores (CPU)
Memory
(RAM)
Temp. Storage
Preview Prices
Standard Enterprise
Standard_A4_v2 4 8.00 GiB 40 GiB $0.420/hour $0.956/hour
Standard_A8_v2 8 16.00 GiB 80 GiB $0.862/hour $1.935/hour
Standard_D1_v2 1 3.50 GiB 50 GiB $0.296/hour $0.832/hour

 If you want to use gacutil.exe to install assemblies in the Global Assembly
Cache (GAC), you need to provide it as part of your custom setup, or use
the copy provided in the Public Preview container
 If you need to join your Azure-SSIS IR w/ custom setup to a VNet, only
ARM VNet is supported

version 5.4 or later
Azure Storage Explorer app

 “Sample” folder that contains a
custom setup to install a
simple task that just sleeps for
a few seconds on each node of
your Azure-SSIS IR – It also
contains a ”gacutil” folder,
which contains gacutil.exe

 Double click
“UserScenarios” and you
can find
 “.NET FRAMEWORK 3.5” folder,
that contains a custom setup
to install an earlier version of
the .NET Framework on each
node of your Azure-SSIS IR
 “BCP” folder that contains a
custom setup to install SQL
Server command line utilities
(MsSqlCmdLnUtils.msi),
including bulk copy program
(bcp), on each node of your
Azure-SSIS IR
 “EXCEL” folder that contains a
custom setup to install Open
Source Excel assemblies on
each node of your Azure-SSIS
IR

Oracle’s site
Oracle’s site

PSH scripts to manipulate your Azure Storage account
Azure Blob Download Task
Execute Process Task that executes it
Teradata’s site

sawinark@microsoft.com
https://www.yammer.com/azureadvisors/#/threads/inGroup?type=in_group
&feedId=12090139&view=all
https://blogs.msdn.microsoft.com/ssis/

https://docs.microsoft.com/en-us/azure/data-factory/tutorial-create-azure-ssis-runtime-portal
https://docs.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-auth
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-connect-to-catalog-database
https://docs.microsoft.com/en-us/sql/integration-services/ssis-quickstart-deploy-ssms
https://docs.microsoft.com/en-us/sql/integration-services/ssis-quickstart-run-ssms
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-schedule-packages
https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-stored-procedure-activity
https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-ssis-activity
https://docs.microsoft.com/en-us/azure/data-factory/monitor-integration-runtime#azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/manage-azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition
https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup
https://docs.microsoft.com/en-us/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-components

New capabilities for modern data integration in the cloud

Recommended

More Related Content

What's hot (20)

Similar to New capabilities for modern data integration in the cloud (20)

More from Microsoft Tech Community (20)

Recently uploaded (20)

New capabilities for modern data integration in the cloud

Editor's Notes