Pentaho etl-tool
Download as PPT, PDF3 likes1,718 views
The document discusses Kettle, an open source ETL tool from Pentaho. It provides an introduction to the ETL process and describes Kettle's major components: Spoon for designing transformations and jobs, Pan for executing transformations, and Kitchen for executing jobs. Transformations in Kettle perform tasks like data filtering, field manipulation, lookups and more. Jobs are used to call and sequence multiple transformations. The document also covers recent Kettle releases and how it can help address challenges in data integration projects.
Ad
More Related Content
What's hot (20)
Viewers also liked (14)
Ad
Similar to Pentaho etl-tool (20)
Ad
Recently uploaded (20)
Pentaho etl-tool
- 1. Kettle – ETL Tool Sreenivas K
- 2. Agenda Introduction − ETL Process − Pentaho's Kettle Data Integration Challenges Prerequisites and Recent Releases Pentaho DI Components Spoon − Transformations − Jobs
- 3. Introduction – ETL Process Major Components − Extracting Gathering raw data from source systems and storing it in ETL staging environment Data Profiling Identifying data that changed since last load. − Transforming- Cleaning and Conforming Processing data to improve its quality, format it, merge from multiple sources, enforce conformed dimensions Data cleansing Recording error events Audit dimensions Creating and maintaining conformed dimensions and facts
- 4. Introduction – ETL Process − Loading Loading data into data warehouse tables Managing hierarchies in dimensions Managing special dimensions such as date and time, junk, mini, shrunken, small static, and user-maintained dimensions Fact table loading Building and maintaining bridge dimension tables Handling late arriving data Management of conformed dimensions Administration of fact tables Building aggregations Building OLAP cubes Transferring DW data to other environment for specific purposes
- 5. Data Transformation and Integration Examples Data filtering − Is not null, greater than, less than, includes Field manipulation − Trimming, padding, upper and lowercase conversion Data calculations − + - X / , average, absolute value, arctangent, natural logarithm Date manipulation − First day of month, Last day of month, add months, week of year, day of year Data type conversion − String to number, number to string, date to number Merging fields & splitting fields Looking up date − Look up in a database, in a text file, an excel sheet, …
- 6. Introduction – Pentaho Kettle Kettle – Kettle Extraction Transformation Transportation & Loading tool Its open source business intelligence suite for powerful data integration by Pentaho. Founded in 2004. Products of Pentaho − Mondrain – OLAP server written in Java − Kettle – ETL tool
- 7. Data Integration - Challenges Data is everywhere Data is inconsistent − Records are different in each system Performance issues − Running queries to summarize data for stipulated long period takes operating system for task Data is never all in Data Warehouse − Excel sheet, acquisition, new application
- 8. Prerequisites Recent Releases Java Runtime Environment 1.5 and above Compatible with almost any platform Compatible with wide range of Databases technologies. 4/25 Data Integration 3.0.3 GA 4/18 Data Integration 3.1 Milestone 2/8 Data Integration 3.0.2 GA 12/12 Data Integration 3.0.1 GA 11/15 Data Integration 3.0 GA 10/31 Data Integration 3.0 RC2 10/24 Data Integration 2.5.2 GA 10/08 Data Integration 3.0 RC1 08/24 Data Integration 2.5.1 GA
- 9. Pentaho Components Spoon − GUI that allows you to design transformations and jobs that can be run with the Kettle tools — Pan and Kitchen − Transformations and Jobs can describe themselves using an XML file or can be put in a Kettle database repository. − Spoon is available as executable script and batch file to make use of tool in heterogeneous environment. Pan − A program to execute transformations designed by Spoon in XML or database repository. − Transformations are scheduled in batch mode to be run automatically at regular intervals Kitchen − Execute jobs designed by Spoon in XML or database repository
- 10. Repository Connection establishment Auto login − By setting manually KETTLE_REPOSITORY, KETTLE_USER and KETTLE_PASSWORD environmental variables. Login − By default PDI provides login username and password ad admin.
- 14. Transformation − Value: Values are part of a row and can contain any type of data − Row: a row exists of 0 or more values − Output stream: an output stream is a stack of rows that leaves a step. − Input stream: an input stream is a stack of rows that enters a step. − Hop: A hop is a graphical representation of one or more data streams between 2 steps. − Note: A note is a piece of information that can be added to a transformation Engine capable of performing a multitude of functions such as reading, manipulating and writing data to and from various data sources.
- 15. Jobs − Job Entry: A job entry is one part of a job and performs a certain − Hop: A hop is a graphical representation of one or more data streams between 2 steps − Note: a note is a piece of information that can be added to a job A way of calling transformations and controlling the sequence of their execution. Usually jobs are scheduled in batch mode to be run automatically at regular intervals.
- 16. Input Steps Output Steps Lookup Steps Transformation Steps Join Steps DW Steps Mapping Steps Job Steps