10 basic terms so you can talk to data engineer
0 likes60 views
This document defines 16 basic terms related to data engineering: 1. Apache Airflow is an open-source workflow management platform that uses directed acyclic graphs to manage workflow orchestration. 2. Batch processing involves processing large amounts of data at once, such as in ETL steps or bulk operations on digital images. 3. Cold data storage stores old, hardly used data on low-power servers, making retrieval slower. 4. A cluster groups several computers together to perform a single task.
Ad
More Related Content
What's hot (19)
Similar to 10 basic terms so you can talk to data engineer (20)
Ad
More from Worapol Alex Pongpech, PhD (9)
Ad
Recently uploaded (20)
10 basic terms so you can talk to data engineer
- 1. 16 basic terms So you can talk to data engineers Alex Pongpech
- 2. 1. AIRFlow ● Apache Airflow is an open-source workflow management platform. ● It started at Airbnb in October 2014 as a solution to manage the company's increasing complex workflows. ● Airflow uses directed acyclic graphs (DAGs) to manage workflow orchestration. 2
- 3. 2. Batch Processing: ● processing large amounts of data at once. ● Usage ○ The extract, transform, load (ETL) step in populating data warehouses ○ Performing bulk operations on digital images such as resizing, conversion, watermarking, or otherwise editing a group of image files. ○ Converting computer files from one format to another. For example, a batch job may convert proprietary and legacy files to common standard formats for end-user queries and display. 3
- 4. 3. Cold data storage– ● storing old data that is hardly used on low-power servers. Retrieving the data will take longer 4
- 5. 4. Cluster: ● several computers (or virtual machines or node) grouped together to perform a single task. 5
- 6. 5. Data Lakes: ● A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. ● You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions. ● not just store data 6
- 7. 6. Data Ingests: ● is the process of obtaining and importing data for immediate use or storage in a database. To ingest something is to "take something in or absorb something." ● Data can be streamed in real time or ingested in batches. 7
- 8. 7. Data Pipeline ● is a broader term that encompasses ETL as a subset. ● It refers to a system for moving data from one system to another. The data may or may not be transformed, and it may be processed in real-time (or streaming) instead of batches. 8
- 9. 8. Data Streaming: ● processing data in small chunks, one at a time, rather than processing all data at once. ● Streaming is necessary for processing infinite event streams. ● It’s also useful for processing large amounts of data, because it prevents memory overflows during processing 9
- 10. 9. Distributed Data Processing: ● breaking up data into partitions so that large amounts of data can be processed by many machines simultaneously. 10
- 11. 10.Extract, Transform and Load (ETL) – a process in a database and data warehousing meaning extracting the data from various sources, transforming it to fit operational needs and loading it into the database 11
- 12. 11. Extract Load Transform ● In contrast to ETL, in ELT models the data is not transformed on entry to the data lake, but stored in its original raw format. 12
- 13. 12. Hadoop Distributed File System (HDFS) ● Hadoop Distributed File System (HDFS) is primary data storage layer used by Hadoop applications. ● It employs DataNode and NameNode architecture. 13
- 14. 13. Metadata. ● Metadata is "data that provides information about other data". ● Many distinct types of metadata exist, including descriptive metadata, structural metadata, administrative metadata, reference metadata and statistical metadata. 14
- 15. 14 Real-time processing: ● Real-time data processing is the execution of data in a short time period, providing near-instantaneous output. ● The processing is done as the data is inputted, so it needs a continuous stream of input data in order to provide a continuous output. 15
- 16. 15. Scalable Scalable hardware or software can expand to support increasing workloads. This capability allows computer equipment and software programs to grow over time, rather than needing to be replaced. 16
- 17. 16. Workflow Orchestration ● An orchestration workflow, which is based on Business Process Manager Business Process Definition, defines a logical flow of activities or tasks from a Start event to an End event to accomplish a specific service. 17