ETL using Big Data Talend
16 likes8,029 views
Today, when data is mushrooming and coming in heterogeneous forms, there is a growing need for a flexible, adaptable, efficient and cost effective integration platform which will take minimum on-boarding time and interact and entertain n number of platforms. Talend fits just perfect in this space with a proven track record, so learning talend makes lot of sense for anybody associated with data world. If you understand how to manage, transform, store your organisation data (retail, banking, airlines, research, insurance, cards etc.) and effectively represent it which is the backbone behind any successful MIS system/reporting/dash board then you are a key person that organisation most sought after.
Ad
More Related Content
What's hot (20)
Similar to ETL using Big Data Talend (20)
Ad
More from Edureka! (20)
Ad
Recently uploaded (20)
ETL using Big Data Talend
- 1. www.edureka.co/talend-for-big-data ETL using Big Data Talend View Talend For Big Data course details at www.edureka.co/talend-for-big-data
- 2. Slide 2 www.edureka.co/talend-for-big-data ๏ฎ Understand how ETL is complementing Hadoop Ecosystem ๏ฎ Adapt to ETL-Big Data industry ๏ฎ Understand why Talend is used with Big Data ๏ฎ Learn Big Data not in months but in Minutes ๏ฎ Understand the Use Case โ Banking Industry ๏ฎ Implement a Talend job with Hadoop At the end of this session, you will be able to: Objectives
- 3. Slide 3 www.edureka.co/talend-for-big-data ๏ฎA Graphical Abstraction Layer on top of Hadoop Applications โ this makes life so much easy in the Big Data buzz world ๏ฎThe surprising stuff about the current buzz and questions heralding the end of ETL and even data warehousing is the lack of pushback and analysis of some of the outlandish comments made ETL with Big Data ยป What no one seems to question in response to these sorts of comments is the naive assumptions these statements are based on !! ยป Is it realistic for most companies to move all of their data into Hadoop? The typical assertion is that "Hadoop eliminates the need for ETLโโฆ. Seriously ?
- 4. Slide 4 www.edureka.co/talend-for-big-data ETL with Big Data Machine Data Transactional Data Business Apps Data ETL Workflow Big Data Extra and Load
- 5. Slide 5 www.edureka.co/talend-for-big-data Is writing ETL scripts in MapReduce code still ETL? Is ETL running faster (in few cases & slower in others) on Hadoop eliminating ETL? Is introduction of Hadoop changing when, where and how ETL happens? Yes No Yes The question isn't really that are we eliminating ETL, but where does ETL take place & how are we changing its definition ETL with Big Data (Contd.)
- 6. Slide 6 www.edureka.co/talend-for-big-data Defining ETL E โข represents the ability to consistently and reliably extract data with high performance and minimal impact to the source system T โข represents the ability to transform one or more data sets in batch or real-time into a consumable format L โข stands for loading data into a persistent or virtual data store
- 7. Slide 7 www.edureka.co/talend-for-big-data How learning ETL (along Big Data) is addressing major business problems ? Why ETL + Hadoop? BIG DATA DATA INTEGRATION DATA QUALITY MDM ESB BPM TALEND UNIFIED PLATFORM
- 8. Slide 8 www.edureka.co/talend-for-big-data One Stop Solution!! Improves efficiency of big data job design with graphic interface Abstract and generates code Run transforms inside Hadoop Native support for HDFS, Sqoop, HBase, Mahout, Pig, Hive & MapReduce code generate Apache License 2.0 Embedded in Hortonworks Data Platform Certified with Cloudera, MapR and Grenplum An open source ecosystem
- 9. Slide 9 www.edureka.co/talend-for-big-data Talend Q. Why Talend? Ans . Because the more connected the world becomes, the more quickly a business must adapt
- 10. Slide 10 www.edureka.co/talend-for-big-data ๏ฎTalend is the only Graphical User Interface tool which is capable enough to โtranslateโ an ETL job to a MapReduce job. Thus, Talend ETL job gets executed as a MapReduce job on Hadoop and get the big data work done in minutes ๏ฎThis is a key innovation which helps to reduce entry barriers in Big Data technology and allows ETL job developers (beginners and advanced) to carry out Data Warehouse offloading to greater extent ๏ฎWith its Eclipse-based graphical workspace, Talend Open Studio for Big Data enables the developer and data scientist to leverage Hadoop loading and processing technologies like HDFS, HBase, Hive, and Pig without having to write Hadoop application code ๏ฎHadoop Applications, Seamlessly gets Integrated within minutes using Talend Why Talend?
- 11. Slide 11 www.edureka.co/talend-for-big-data ๏ฎBy simply selecting graphical components from a palette, arranging and configuring them, you can create Hadoop jobs For example: 1. Load data into HDFS (Hadoop Distributed File System) 2. Use Hadoop Pig to transform data in HDFS 3. Load data into a Hadoop Hive based data warehouse 4. Perform ELT (extract, load, transform) aggregations in Hive 5. Leverage Sqoop to integrate relational databases and Hadoop Why Talend? (Contd.)
- 12. Slide 12 www.edureka.co/talend-for-big-data Talend Hadoop Integration
- 13. Slide 13 www.edureka.co/talend-for-big-data ๏ฎ For Hadoop applications to be truly accessible to your organization, they need to be smoothly integrated into your overall data flows ๏ฎ Talend Open Studio for Big Data is the ideal tool for integrating Hadoop applications into your broader data architecture ๏ฎ Talend provides more built-in connector components than any other data integration solution available, with more than 800+ connectors that make it easy to read from or write to any major file format, database, or packaged enterprise application For Example, in Talend Open Studio for Big Data, you can use drag 'n drop configurable components to create data integration flows that move data from delimited log files into Hadoop Hive, perform operations in Hive, and extract data from Hive into a MySQL database (or Oracle, Sybase, SQL Server, and so on) Talend Hadoop Integration (Contd.)
- 14. Slide 14 www.edureka.co/talend-for-big-data ๏ฎ More and more enterprise wanted to scale up in Hadoop/Big Data technologies with use of existing pool of talent and reduce overspending on map-reduce programmer (which is pretty new and expensive) ๏ฎ High rise of job trend in Data Scientist/Data Analysis (Talend also comes along with basic BI transformations which reduces your dependency on simple excel dash board/ BI tools) ๏ฎ Gartner is featuring Talend as the best technology in market for Data Integration and Big Data ๏ฎ 3 major players in Big Data industry, Hortonworks, Cloudera, MapR have already tied up with Talend for big data solutions ๏ฎ And mostly any level person in industry can quickly get started on this without much pre-requisites Myth : I donโt know Java programming , how would this course help me learn and excel in Big Data? The biggest advantage you get with Talend for Big Data is โthere is no prerequisiteโ to learn this concept. Whether you come with prior knowledge of Hadoop or not , this course has some or other best things to offer Talend Hadoop Integration (Contd.)
- 15. Slide 15 www.edureka.co/talend-for-big-data Learn Big Data not in months but in Minutes!! Sounds too good ? But true Big Data in 10 minutes HADOOP HORTONWORKSMAPR CLOUDERA Go from zero to big data in under 10 minutes Get big data without coding. The Talend Big Data Sandbox is a ready-to-run virtual environment that includes Talend Platform for Big Data, popular Hadoop distributions and data examples
- 16. Slide 16 www.edureka.co/talend-for-big-data Who can use โTalend for Big Dataโ!!
- 17. Slide 17 www.edureka.co/talend-for-big-data Let us all see quickly, what Talend can do in minutes, reducing the man-hours in doing MapReduce programming in Hadoop, shall we? We are just about to see the Bigger Picture
- 18. Slide 18 www.edureka.co/talend-for-big-data A Banking industry use case : โAddressing the challenges in growing the business with use of Big Dataโ . We will use customer filled web-log data (collected by bank) and with the help of Pig-ETL job will answer the question โwhere should bank hold marketing campaigns for new product launch to get more businessโ , in ETL-Big Data Analytics style In this section, you will be able to sense the true power of Talend+Big Data Real time Use Case : ETL + Big Data
- 19. Slide 19 www.edureka.co/talend-for-big-data Our use case setup is using the below : ยป Hortonworks Sandbox 1.3 ยป Talend Open Studio for Big Data 5.5 ยป Windows 7 (64 Bit OS) ยป Machine : 4GB RAM , i3 processor Environment Setup
- 20. Slide 20 www.edureka.co/talend-for-big-data Use-case demonstration has been divided into steps such as : ยป Step 1: Generate huge web-log data (we are generating our own source sample data to simulate real time data) ยป Step 2: Load the data from local file system to HDFS (Hadoop) in seconds ยป Step 3: Read from HDFS, Process via Pig Scripts and achieve results Use case Design
- 21. Slide 21 www.edureka.co/talend-for-big-data Use-case Snapshot
- 23. Slide 23 www.edureka.co/talend-for-big-data References ๏ฎ https://www.talend.com/resource/hadoop-applications.html ๏ฎ http://www.edureka.co/blog/big-data-and-etl-are-family/
- 24. Slide 24 www.edureka.co/talend-for-big-data Course Topics ๏ฎ Module 1 ยป Role of Open Source ETL Technologies in Big Data ๏ฎ Module 2 ยป Talend: A Revolution in Big Data ๏ฎ Module 3 ยป Talend: Read & Write Various Types of Source/Target Systems ๏ฎ Module 4 ยป Talend: How to Transform your Business: Basic ๏ฎ Module 5 ยป Talend: How to Transform your Business: Advanced 1 ๏ฎ Module 6 ยป Talend: How to Transform your Business: Advanced 2 ๏ฎ Module 7 ยป Big Data Concepts: Required for Talend for Big Data ๏ฎ Module 8 ยป Introduction to Talend for Big Data ๏ฎ Module 9 ยป Hive in Talend for Big Data ๏ฎ Module 10 ยป Pig in Talend for Big Data and Project
- 25. Slide 25 LIVE Online Class Class Recording in LMS 24/7 Post Class Support Module Wise Quiz Project Work Verifiable Certificate www.edureka.co/talend-for-big-data How it Works?
- 27. Slide 27 Course Url