ETL

ETL Process
4 major components:
Extracting
Gathering raw data from source systems and storing it in ETL staging environment

Cleaning and conforming

Processing data to improve its quality, format it, merge from multiple sources, enforce conformed dimensions

Delivering
Loading data into data warehouse tables

Managing
Management of ETL environment

ETL: Extracting
Data profiling Identifying data that changed since last load extraction

ETL: Cleaning and Conforming

Data cleansing Recording error events Audit dimensions Deduping Creating and maintaining conformed dimensions and facts

ETL: Delivering
Implementation of SCD logic Surrogate key generation Managing hierarchies in dimensions Managing special dimensions such as date and time, junk, mini, shrunken, small static, and usermaintained dimensions
Mini dimensions
used to track changes of dimension attribute when type 2 technique is infeasible. Similar to junk dimensions Typically is used for large dimensions Combinations can be built in advance or on the fly Built from dimension table input

ETL: Delivering (Cont)

Small static dimensions
Dimensions created by the ETL system without real source Lookup dimensions for translations of codes, etc.

User maintained dimensions

Master dimensions without real source system Descriptions, groupings, hierarchies created for reporting and analysis purposes.

ETL: Delivering (Cont)

Fact table loading Building and maintaining bridge dimension tables Handling late arriving data Management of conformed dimensions Administration of fact tables Building aggregations Building OLAP cubes Transferring DW data to other environment for specific purposes

ETL: Managing
Management of ETL environment
Goals
Reliability Availability Manageability

Job scheduler backup system Recovery and restart system Version control system

ETL: Managing (Cont.)

Version migration system Workflow monitor Sorting system Analyzing dependencies and lineage Problem escalation system Parallelization Security system Compliance manager Metadata repository manager

ETL Process
Planning
High level source to target data flow diagram Selection and implementation of ETL tool Development of default strategies for dimension management, error handling, and other processes Development data transformations diagrams by target table Development of job sequencing

ETL Process
Developing one-time historic load
Build and test the historic dimension and fact tables load

Developing incremental load process

Build and test dimension and fact tables incremental load processes Build and test aggregate table loads and/or OLAP processing Design, build, and test the ETL system automation

ETL Tools: Build vs Buy

Many off-the-shelf tools exist Benefits are not seen right away
Setup Learning curve

High-end tools may not justify value for smaller warehouses

Off-the-shelf ETL Tools

Tool Oracle Warehouse Builder (OWB) Data Integrator (BODI) IBM Information Server (Ascential) SAS Data Integration Studio PowerCenter Oracle Data Integrator (Sunopsis) Data Migrator Integration Services Talend Open Studio DataFlow Data Integrator Transformation Server Transformation Manager Data Manager DT/Studio ETL4ALL DB2 Warehouse Edition Jitterbit Pentaho Data Integration Vendor Oracle Business Objects IBM SAS Institute Informatica Oracle Information Builders Microsoft Talend Group 1 Software (Sagent) Pervasive DataMirror ETL Solutions Ltd. Cognos Embarcadero Technologies IKAN IBM Jitterbit Pentaho

ETL Specification Document

Can be as large as 100 pages per business process; In reality, the work starts after the high level design is documented in a few pages. Source-to-target mappings Data profiling reports Physical design decisions Default strategy for extracting from each major source system Archival strategy Data quality tracking and metadata Default strategy for managing changes to dimension attributes

ETL Specification Document (Cont)

System availability requirements and strategy Design of data auditing mechanism Location of staging areas Historic and incremental load strategies for each table
Detailed table design Historic data load parameters (# of months) and volumes (# of rows) Incremental data volumes

ETL Specification Document (Cont)

Handling of late arriving data Load frequency Handling of changes in each dimension attribute (types 1,2,3) Table partitioning Overview of data sources; discussion of sourcespecific characteristics Extract strategy for the source data Change data capture logic for each source table Dependencies Transformation logic (diagram or pseudo code)

ETL Specification Document (Cont)

Preconditions to avoid error conditions Recovery and restart assumptions for each major step of the ETL pipeline Archiving assumptions for each table Cleanup steps Estimated effort

Overall workflow Job sequencing Logical dependencies

Loading Pointers
One time historic load
Disable RI constraints (FKs) and re-enable them after the load is complete Drop indexes and re-create them after the load is complete Use bulk loading techniques Not always the case

Loading Pointers (Cont)

Incremental load

Loading Pointers (Cont)

Sometimes historic and incremental load logic is the same; many times- is similar. Updating aggregations, if necessary Error handling

10

Sample: Generation of Surrogate Keys on SQL Server

As simple as: DECLARE @i INTEGER SELECT @i = MAX(ID) + 1 FROM TableName But may not work with concurrent processes OR Create PROCEDURE pGetNextID (@SeedName VARCHAR(32), @SeedValue BIGINT OUTPUT) AS UPDATE Lookup_Seed SET @SeedValue = SeedValue = SeedValue + 1 WHERE SeedID = @SeedName Lookup_Seed table: SeedID varchar (32) SeedValue bigint

Introduction to ETL Using Microsoft Tools

11

About SSIS
Microsofts ETL tool Solutions created in packages Developed in Business Intelligence Development Studio (BIDS)
Variation of visual studio

How to create SSIS project within BIDS

Click on Start->All Programs->Microsoft SQL Server 2008> SQL server Business Intelligence Studio From the File menu select New->Project In the Project Type, choose Business Intelligence Projects->Integration Services Project Enter name of the project and select storage location Click OK button

12

How to execute SSIS projects

Within BIDS, right click on the project name in the solution explorer and choose Execute Package
Output window shows results

Outside of BIDS
dtexec.exe dtexecui.exe SQL Server Management Studio SQL Server Agent Scheduler Custom .Net application

Tasks We Will Cover

Bulk insert
Bulk insert data from text files into SQL server database

Data flow
transforms, cleans, and modifies data as it is moved from source to destination

Execute SQL
Executes SQL statements in specified databases

13

Connection Managers
Is a connection to a data source Must be configured for each data source and destination We will use OLE DB and flat file connection managers
Others connection managers are available

Configuring Connection Manager

Open a package Click Control Flow tab Right click Connection Manager area, then click New Connection or choose a new connection of a specific type

14

Flat File Connection Manager

Configure General, Columns, and Advanced sections Use Preview option to view how you configurations were applied

OLE DB Connection Manager

Using BIDS 2008:
Select Native OLE DB/SQL Server Native Client 10.0 Enter server name Enter login info Select database Test connection Click OK

15

Data Migration Best Practices

Data types from source must match data types from destination
Conversions may be necessary

Do conversions as early as possible Bring over only data fields that need to be loaded into DW

Bulk Insert Task

Fastest way to copy data to SQL Server No transformations can be performed when copying data Usually is used to bring raw data from sources into Staging databases in the DW environment Configure General, Connection, Option, and Expression sections

16

Execute SQL Task

Executes one or more SQL statements or stored procedures Executing batches is possible by placing GO command to separate batches SQL Statements can be entered directly into the task editor window or may reside in a file or a variable May return result that can be captured in a variable May specify parameters for queries
Use variables

Precedence Constraints
Specify order of task execution 3 types On success green On failure red On completion Blue Configured on the control flow tab
Drag a green arrow that is coming out of a task Connect it to another tasks Right click on it to specify type. Default is On Success

17

Data Flow Task

Transforms, cleans, and modifies data as it is moved from source to destination Is added to the package on the Control Flow tab Its elements are configured on the Data Flow tab Package may contain multiple data flow tasks Elements are grouped into Source, Transformation, and Destination

Transformations
Numerous transformations are available We will look at Lookup transformation only
Essential for loading fact tables Lookup data from dimension table to retrieve key values to be loaded into fact table Caching can be used (either full or partial)

18

Overall Flow
Wipe out staging table
Execute SQL task

Wipe out DW table if data is fully refreshed

Execute SQL task Do not do this if data should be appended

Load raw data into staging table

Bulk Insert task/Data flow task

Load data from staging tables into dimensions

Execute SQL statement task/Data flow task

Load data into fact table

Data flow task

Create aggregations and/or OLAP cubes

Execute SQL task and/or SSAS

Load Data Into Fact Table: A closer Look

Create OLEDB source
use an sql command to select data from staging table

Create lookup transformation for each key to dimension table that will be stored in the fact table Create OLEDB destination
specify fact table name Specify mapping

19

Useful SQL Scripts

Check if a table exists prior to creating it
if not exists ( select * from information_schema.tables where table_name=xyz)

Useful SQL Scripts

Generate Dates dimension at daily grain
DECLARE @gencalendar TABLE (cal_date DATETIME PRIMARY KEY) DECLARE @p_date SMALLDATETIME SET @p_date = '20010101' WHILE @date <= '20151231' BEGIN INSERT INTO @gencalendar(cal_date) VALUES(@p_date) SET @date = dateadd(d, 1, @p_date) END SELECT * FROM @gencalendar

20

Useful SQL Scripts

If Dates dimension has key in a YYYMMDD format, this script populates entire table
DECLARE @p_date SMALLDATETIME SET IDENTITY_INSERT dimdates ON SET @p_date = '19000101' WHILE @p_date <= '20191231' BEGIN INSERT INTO dimdates(Datekey,CalendarDate,Calendaryear,Calendarhalfyear,Calendarquarter,Calendarmonth,Cal endarday) select CAST(convert(varchar(8),@p_date,112) as int) , CAST(convert(varchar(10),@p_date,101) as date), DATEPART(YY,@p_date) , Case When DATENAME(QQ, @p_date) < 3 then 1 else 2 END, DATEPART (QQ, @p_date), DATEPART(MM,@p_date), DATEPART (DD, @p_date) SET @p_date = dateadd(d, 1, @p_date) END

Useful SQL Scripts

If Dates dimension has key in a YYYMMDD format and the date key has already been inserted, this script populates remaining columns in this dimension
Update dimDate set CalMonthName = CAST(datekey as varchar) where isEndOfYear = 9 Update dimDate Set CalDate = Cast(substring(CalMonthName,5,2)+'/'+substring(CalMonthName,7,2) +'/'+substring(CalMonthName,1,4) as DATE) where isEndOfYear = 9 Update dimDate Set CalYear = DATEPART(YY,CalDate) ,CalMonth = DATEPART(MM,CalDate) ,CalMonthName = DATENAME (MM, CalDate) + ' ''' + RIGHT(Cast(CalYear as CHAR(4)),2) ,CalQuarter = DATENAME (QQ, CalDate) ,CalQuarterName = CAST(CalYear as CHAR(4)) +' Q' + CAST(CalQuarter as CHAR(1)) ,CalHY = Case When DATENAME(QQ, CalDate) < 3 then CAST(CalYear as CHAR(4)) +' H1' else CAST(CalYear as CHAR(4)) +' H2' END , isEndOfYear = 8 where isEndOfYear = 9

21

Useful SQL Scripts

Calculate persons age
create function dbo.fCalculateAge(@DOB datetime,@Date datetime) returns int as begin return ( select case when month(@DOB)>month(@Date) then datediff(yyyy,@DOB,@Date)-1 when month(@DOB)<month(@Date) then datediff(yyyy,@DOB,@Date) when month(@DOB)=month(@Date) then case when day(@DOB)>day(@Date) then datediff(yyyy,@DOB,@Date)-1 else datediff(yyyy,@DOB,@Date) end end) end

Useful SQL Scripts

Sample query for loading data into dimension table
INSERT INTO DimXXX (Col1,Col2) select distinct stgcol1,stgcol2 From Staging.dbo.tbl Where ltrim(rtrim(stgcol1))+'|'+ltrim(rtrim(stgcol2)) not in(select distinct ltrim(rtrim(col1))+'|'+ltrim(rtrim(col2))from DimXXX)

22

Useful SQL Scripts

Parsing names in a Last Name, First Name, Middle Name,Suffix format separated by comma, with Middle Name and suffix being optional
create table tmpnames ( fullname varchar(255)) The script is presented on the next slide.

select fullname, CASE WHEN charindex(',', fullname) > 1 THEN substring(fullname, 1, charindex(',', fullname)-1) WHEN charindex(',', fullname) =0 THEN fullname END as lname, CASE WHEN charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))) > 1 THEN substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))), 1, charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-1) WHEN charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))<=1 and charindex(',', fullname) >0 THEN ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))) ELSE '' END as fname, CASE WHEN charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))))))) > 1 and substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),1,charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))1) not in ('Sr','Jr','II','III','IV') THEN substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),1,charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))1) WHEN charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))<=1 and charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))) >0 and ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))) not in ('Sr','Jr','II','III','IV') THEN ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))) ELSE '' END as mname, CASE WHEN charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))))))))))) > 1 and substring(ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))))),1,charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))))))-1) in ('Sr','Jr','II','III','IV') THEN substring(ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))))),1,charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))))))-1) WHEN charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))<=1 and charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))) >0 THEN ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))) WHEN charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))))))<=1 and charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))))))) >0 THEN ltrim(rtrim(substring(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))),charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))+1,len(ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))))))-charindex(',', ltrim(rtrim(substring(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))),charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))+1,len(ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname)))))-charindex(',', ltrim(rtrim(substring(fullname,charindex(',', fullname)+1,len(fullname)-charindex(',', fullname))))))))))))

23

OLAP
SSAS is used
Cubes KPIs Data mining engine Actions

Cubes
is a multidimensional structure that contains dimensions and measures Dimensions define the structure of the cube, and measures provide the numerical values of interest to the end user. a cube allows a client application to retrieve values as if cells in the cube defined every possible summarized value. Cell positions in the cube are defined by the intersection of dimension members and contain aggregated measures Cubes can be queried using MDX (Multidimensional Expressions), not SQL

24

KPI
Key performance indicators From business perspective, it is a quantifiable measurement for gauging business success a KPI is a collection of calculations that are associated with a measure group in a cube that are used to evaluate business success

Data Mining Engine

Uses mathematical analysis to extract patterns and trends from that exist in data Some scenarios:
Forecasting sales Targeting mailings toward specific customers Determining which products are likely to be sold together Finding sequences in the order that customers add products to a shopping cart

25

Actions
SSAS commands that are used by clients
Types:
Drillthrough actions, which return the set of rows that represents the underlying data of the selected cells of the cube where the action occurs. Reporting actions, which return a report from Reporting Services that is associated with the selected section of the cube where the action occurs. Standard actions, which return the action element (URL, HTML, DataSet, RowSet, and other elements) that is associated with the selected section of the cube where the action occurs.

Creating Cubes
Click on Start->All Programs->Microsoft SQL Server 2008> SQL server Business Intelligence Studio From the File menu select New->Project In the Project Type, choose Business Intelligence Projects->Analysis Services Project Enter name of the project and select storage location Click OK button

26

Creating Cubes
Create a new data source Right click on Data Sources and choose new data source Create a new data source view Right click on Data Source Views and choose new data source view Create Dimensions Right-click on Dimensions and choose new dimension Create cubes Right click on Cubes and choose new cube

Creating Cubes
Once development completed, it is deployed and processed
Deploy=creating cube structure on the server Process=data calculations, aggregations, etc. Right click on the cube name in the solution explorer and choose Process (prior to this step right click the project name in solution explorer, choose properties and specify deployment location)
Cube is ready to be consumed by users

27