06 SSIS Data Flow

Editor's Notes

• #3: For beginners, the data flow can look very similar to the control flow. Ensure that students appreciate the difference between the arrows in control flow (precedence constraints) and data flow (service paths).Extended scenarios can consist of, for example, non-ETL data transfers, data cleansing, or text mining.When describing the sample data flow on the slide, use the analogy of a conveyor belt in a factory. Raw material (source data) is placed on the conveyor belt and passes through various processes (transformations). Quality assurance components might reject some material, in which case it can be scrapped (logged) or fixed and blended back in with the quality material. Eventually, finished goods (clean, valid data) arrive at the end of the conveyor belt (data warehouse).
• #5: Developing Project Data Sources and Package ConnectionsBecause the main purpose of SSIS is to move data from sources to destinations, the nextmost important step is to add the pointers to these sources and destinations. These pointersare called data sources and connections. Data sources are stored at the project level and arefound in Solution Explorer under the logical folder named Data Sources. Connections, on theother hand, are defined within packages and are found in the Connection Managers pane atthe bottom of the Control Flow or Data Flow tab. Connections can be based on project datasources or can stand alone within packages. The next sections walk you through the uses andimplementation of project data sources and package connections.
• #6: To work with the Data Flow Task, you can either drag a Data Flow Task from the Control Flow toolbox onto the workspace and then double-click it,or you can click the Data Flow tab within the SSIS Designer. After clicking the Data Flow tab, you see the Data Flow Designer, where you can use the data flow to handle and transform datasets. Lesson 2, “Creating and Editing Control Flow Objects,” showed how to use control flow tasksand containers.Notice the difference between the ControlFlow toolbox items and the Data Flow toolbox items.When you are working in the data flow, the toolbox shows items related to data flow development,including data flow sources, data flow transformations, and data flow destinations.In this lesson, you will look at the details of the source and destination adapters as well asthe transformations.
• #7: If you run a package in SSIS Designer, you can view the data in a data flow by attaching data viewers to a path. A data viewer can be configured to display data in a grid, histogram, scatter plot, or column chart. A data viewer is a useful debugging tool. For more information, see Debugging Data Flow. For example, if a path connects an OLE DB source and a Sort transformation, the OLE DB source is the source of the path, and the Sort transformation is the destination of the path. The source is the component where the path starts, and the destination is the component where the path ends. The SSIS Designer provides the Data Flow Path Editor dialog box for setting path properties, viewing the metadata of the data columns that pass through the path, and configuring data viewers.
• #8: ADO.NET SourceProvides connections to tables or queries through an ADO.NETprovider.Excel SourceAllows extractions from an Excel worksheet defined in an Excelfile.Flat File Source Connects to a delimited or fixed-width file created with differentcode pages.OLE DB Source Connects to installed OLE DB providers, such as SQL Server,Access, SSAS, and Oracle.Raw File SourceStores native SSIS data in a binary file type useful for data staging.XML SourceAllows raw data to be extracted from an XML file; requires an XMLschema to define data associations.
• #9: Data flow source adapters use package connections, which point to the server instance or file location of the data source. (The only exception is the raw file adapter, which does not use a package connection.) A source adapter extracts data from sources and moves it into the data flow, where it will be modified and sent to a destination.The source for a data flow typically has one regular output. The regular output contains output columns, which are columns the source adds to the data flow.
• #10: Data flow destinations are similar to sources in that they use package connections. However, destinations are the endpoints in a package, defining the location to which the data shouldbe pushed.For example, if you are sending data to an Excel file from a database table, your destination will be an Excel Destination adapter.All the source adapters (except the Data Reader source) have matching destination adapters in the SSIS data flow. And there are other destination adapters that let you send data toeven more destinationsADO.NET Destination Allows insertion of data by using an ADO.NET managedprovider.Data Mining Model Training Lets you pass data from the data flow into a data miningmodel in SSAS.DataReader Destination Lets you put data in an ADO.NET record set that can beprogrammatically referenced.Dimension Processing Lets SSAS dimensions be processed directly from dataflowing through the data flow.Excel Destination Used for inserting data into Excel, including Excel 2007.Flat File DestinationAllows insertion of data to a flat file such as a comma delimitedor tab-delimited file.OLE DB Destination Uses the OLE DB provider to insert rows into a destinationsystem that allows an OLE DB connection.Partition Processing Allows SSAS partitions to be processed directly fromdata flowing through the data flow.Raw File Destination Stores native SSIS data in a binary fi le type useful fordata staging.Recordset Destination Takes the data flow data and creates a record set in a package variable of type object.SQL Server Compact DestinationLets you send data to a mobile device running SQL Mobile.SQL Server Destination Provides a high-speed destination specific to SQL Server2008 if the package is running on SQL Server.
• #11: Like the source, the destination adapter requires configuration,both in the connection and table that the rows should be inserted into as well as in mapping the data flow columns to thedestination table columns.
• #13: As with the source and destination adapters, you drag transformations from the DataFlow toolbox onto the Data Flow tab of the SSIS Designer, and edit them by right-clicking thetransformation you want to change and then clicking Edit. You connect sources, transformations,and destinations through data paths, which you create by dragging the output arrowonto another component in the data flow. The green data path arrows are for rows that aresuccessfully transformed, and the red output path arrows are for rows that failed the transformationbecause of an error, such as a truncation or conversion error. Figure 1-25 shows adata flow that connects a source to several transformations through data paths and onto adestination.
• #14: Notice that this OLE DB Destination uses the AdventureWorksDW2008 connection and is configured by default to use the Table Or View—Fast Load option of the Data Access Mode drop-down list. This means that records will be processed with bulk insert statements rather than one row at a time.Figure 1-24 shows the Mappings tab of the same OLE DB Destination Editor. This is where you map columns available from the data flow to the destination columns in the destinationadapter. All the destination adapters have a Mappings tab. FIGURE 1-24 Each destination adapter requires you to map data from the data flow input columns to thedestination columns.Notice that not all columns are mapped. However, if one of the unmapped destination columnsis marked as NOT NULL, the destination fails the package when it is run. In the sectiontitled “Using Transformations” later in this lesson, you see how to use the Slowly ChangingDimension Transformation to handle new records and updates.
• #15: Transformations perform a wide variety of operations on the underlying data, and the transformation you choose depends on your data processing requirements. Some transformations operate similarly to other transformations; therefore, we can categorize them into naturalgroupings of like components.
• #16: Character MapPerforms common text operations, such as Uppercase, and allowsadvanced linguistic bit conversion operations.Copy Column Duplicates column values in each row to new named columns.Data ConversionCreates new columns in each row based on new data typesconverted from other columns—for example, converting text tonumeric.Derived Column Uses the SSIS Expression language to perform in-place calculationson existing values; alternatively, allows the addition ofnew columns based on expressions and calculations from othercolumns and variables.Export Column Exports binary large object (BLOB) columns, one row at a time,to a file.Import Column Loads binary files such as images into the pipeline; intended for aBLOB data type destination.OLE DB Command Performs database operations such as updates and deletes,one row at a time, based on mapped parameters from inputrows.
• #17: Aggregate Associates records based on defined groupings and generates aggregationssuch as SUM, MAX, MIN, and COUNT.Percent Sampling Filters the input rows by allowing only a defined percent to bepassed to the output path.Pivot Takes multiple input rows and pivots the rows to generate an outputwith more columns based on the original row values.Row Sampling Outputs a fixed number of rows, sampling the data from the entireinput, no matter how much larger than the defined output theinput is.Sort Orders the input based on defined sort columns and sort directionand allows the removal of duplicates across the sort columns.UnpivotTakes a single row and outputs multiple rows, moving columnvalues to the new row based on defined columns.
• #18: Cache Transform Allows data that will be used in a Lookup Transformation to becached and available for multiple Lookup components.Conditional Split Routes or filters data based on Boolean expressions to one or moreoutputs, from which each row can be sent out only one outputpath.Multicast Generates one or more identical outputs, from which every row issent out every output.Union All Combines one or more similar inputs, stacking rows one on top ofanother, based on matching columnsMerge Join Joins the rows of two sorted inputs based on a defined joincolumn(s), adding columns from each source.Lookup Allows matching between pipeline column values to external databasetables; additional columns can be added to the data flow fromthe external table.Merge Combines the rows of two similar sorted inputs, one on top of theother, based on a defined sort key.
• #19: Audit Adds additional columns to each row based on system packagevariables such as ExecutionStartTime and PackageName.Row Count Tracks the number of rows that flow through the transformationand stores the number in a package variable after the final row.
• #20: Slowly Changing DimensionProcesses dimension changes, including tracking dimensionhistory and updating dimension values. The Slowly ChangingDimension Transformation handles these common dimensionchange types: Historical Attributes, Fixed Attributes, and Changing Attributes.Data Mining QueryApplies input rows against a data mining model for prediction.Fuzzy Grouping Associates column values with a set of rows based on similarity,for data cleansing.Fuzzy Lookup Joins a data flow input to a reference table based on column similarity. The Similarity Threshold setting specifies the closeness of allowed matches—a high setting means that matching values are closer in similarity.Script Component Provides VB.NET scripting capabilities against rows, columns, inputs, and outputs in the data flow pipeline.Term Extraction Analyzes text input columns for English nouns and noun phrases.Term Lookup Analyzes text input columns against a user-defined set of words for association.