Building a Data Lake on AWS

Nov 9, 20210 likes317 views

Build a simple data lake on AWS using a combination of services, including AWS Glue Data Catalog, AWS Glue Crawlers, AWS Glue Jobs, AWS Glue Studio, Amazon Athena, Amazon Relational Database Service (Amazon RDS), and Amazon S3. Link to the blog post and video: https://garystafford.medium.com/building-a-simple-data-lake-on-aws-df21ca092e32

Data Lake Demonstration
Building a Simple Data Lake on AWS
Gary A. Stafford

Twitter/LinkedIn
GaryStafford
Blog
garystafford.medium.com

Agenda
What is a Data Lake?
Dataset
Source Code
Architecture
Demonstration

What is a Data Lake?
“A data lake is a central location that holds a large amount of data in its native, raw
format. Compared to a hierarchical data warehouse, which stores data in files or
folders, a data lake uses a flat architecture and object storage to store the data.” -
Databricks
“A centralized repository that allows you to store all your structured and
unstructured data at any scale. You can store your data as-is, without having to
first structure the data, and run different types of analytics—from dashboards and
visualizations to big data processing, real-time analytics, and machine learning to
guide better decisions.” - AWS

Dataset
TICKIT database
E-commerce platform
Bringing together buyers and sellers of tickets to entertainments events
Designed for demonstrating Amazon Redshift
Small database consists of seven tables: two fact tables and five dimensions
Tables: Categories, Events, Venues, Users, Listings, Sales, Dates
https://docs.aws.amazon.com/redshift/latest/dg/c_sampledb.html

Dataset
Table Simulated Data Source Demo Data Source
Category Software as a Service (SaaS) Amazon RDS for PostgreSQL
Event Software as a Service (SaaS) Amazon RDS for PostgreSQL
Venue Software as a Service (SaaS) Amazon RDS for PostgreSQL
Listing Ecommerce Platform Amazon RDS for MySQL
Sales Ecommerce Platform Amazon RDS for MySQL
Date Ecommerce Platform Amazon RDS for MySQL
Users Customer Relationship Management (CRM) Microsoft SQL Server

github.com/garystafford/tickit-data-lake-demo

Architecture: AWS Services Used
AWS Glue Studio (alt. AWS Glue DataBrew)
AWS Glue Data Catalog (alt. Apache Hive on EMR)
AWS Glue Crawlers (alt. CDC with Kafka Connect or DMS)
AWS Glue Jobs (alt. AWS Glue DataBrew, or Spark or Presto on EMR)
Amazon Athena (alt. Presto on EMR)
Amazon S3

Architecture: Out of Scope (but critically important)
Change Data Capture (CDC): handling changes to systems of record
Transactional Storage Layer: Apache Hudi, Apache Iceberg, Delta Lake
Streaming Data: Spark Structured Streaming, Kinesis, Flink
Fine-grained Authorization: database-, table-, column-, and row-level access
Data Lineage: Tracking data as it flows from sources to consumption

Architecture: Out of Scope (but critically important)
Data Inspection: Scanning incoming data for sensitive info such as PII
DevOps/DataOps: Automating testing, deployment, job execution
Data Warehouse / Lake House Architecture
Data Lake Storage Tiering

( ELK Stack Training - https://www.edureka.co/elk-stack-trai... ) This Edureka Elasticsearch Tutorial will help you in understanding the fundamentals of Elasticsearch along with its practical usage and help you in building a strong foundation in ELK Stack. This video helps you to learn following topics: 1. What Is Elasticsearch? 2. Why Elasticsearch? 3. Elasticsearch Advantages 4. Elasticsearch Installation 5. API Conventions 6. Elasticsearch Query DSL 7. Mapping 8. Analysis 9 Modules

Unified Big Data Processing with Apache Spark (QCON 2014)Databricks

This document discusses Apache Spark, a fast and general engine for big data processing. It describes how Spark generalizes the MapReduce model through its Resilient Distributed Datasets (RDDs) abstraction, which allows efficient sharing of data across parallel operations. This unified approach allows Spark to support multiple types of processing, like SQL queries, streaming, and machine learning, within a single framework. The document also outlines ongoing developments like Spark SQL and improved machine learning capabilities.

Owning Your Own (Data) Lake HouseData Con LA

Data Con LA 2020 Description In this session, I introduce the Amazon Redshift lake house architecture which enables you to query data across your data warehouse, data lake, and operational databases to gain faster and deeper insights. With a lake house architecture, you can store data in open file formats in your Amazon S3 data lake. Speaker Antje Barth, Amazon Web Services, Sr. Developer Advocate, AI and Machine Learning

Building Modern Data Platform with Microsoft AzureDmitry Anoshin

This document provides an overview of building a modern cloud analytics solution using Microsoft Azure. It discusses the role of analytics, a history of cloud computing, and a data warehouse modernization project. Key challenges covered include lack of notifications, logging, self-service BI, and integrating streaming data. The document proposes solutions to these challenges using Azure services like Data Factory, Kafka, Databricks, and SQL Data Warehouse. It also discusses alternative implementations using tools like Matillion ETL and Snowflake.

Time to Talk about Data MeshLibbySchulze

This document discusses data mesh, a distributed data management approach for microservices. It outlines the challenges of implementing microservice architecture including data decoupling, sharing data across domains, and data consistency. It then introduces data mesh as a solution, describing how to build the necessary infrastructure using technologies like Kubernetes and YAML to quickly deploy data pipelines and provision data across services and applications in a distributed manner. The document provides examples of how data mesh can be used to improve legacy system integration, batch processing efficiency, multi-source data aggregation, and cross-cloud/environment integration.

Delta Lake with Azure DatabricksDustin Vannoy

Dustin Vannoy presented on using Delta Lake with Azure Databricks. He began with an introduction to Spark and Databricks, demonstrating how to set up a workspace. He then discussed limitations of Spark including lack of ACID compliance and small file problems. Delta Lake addresses these issues with transaction logs for ACID transactions, schema enforcement, automatic file compaction, and performance optimizations like time travel. The presentation included demos of Delta Lake capabilities like schema validation, merging, and querying past versions of data.

Elasticsearch in NetflixDanny Yuan

Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra

Databricks Platform.pptxAlex Ivy

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

Big data architectures and the data lakeJames Serra

The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include: - Defining top-down and bottom-up approaches to data management - Explaining what a data lake is and how Hadoop can function as the data lake - Describing how a modern data warehouse combines features of a traditional data warehouse and data lake - Discussing how federated querying allows data to be accessed across multiple sources - Highlighting benefits of implementing big data solutions in the cloud - Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (

Vector databases and neural searchDmitry Kan

Dmitry Kan, Principal AI Scientist at Silo AI and host of the Vector Podcast [1], will give an overview of the landscape of vector search databases and their role in NLP, along with the latest news and his view on the future of vector search. Further, he will share how he and his team participated in the Billion-Scale Approximate Nearest Neighbor Challenge and improved recall by 12% over a baseline FAISS. Presented at https://www.meetup.com/open-nlp-meetup/events/282678520/ YouTube: https://www.youtube.com/watch?v=RM0uuMiqO8s&t=179s Follow Vector Podcast to stay up to date on this topic: https://www.youtube.com/@VectorPodcast

Best Practices in the Use of Columnar DatabasesDATAVERSITY

This document discusses best practices for using columnar databases and provides an overview of columnar database fundamentals. It describes how columnar databases store columns of data separately, rather than successive rows, which reduces storage needs and speeds up I/O-bound queries by accessing only the relevant columns. The document recommends using columnar databases to save on storage, speed up queries involving inputs/outputs and scans, scale beyond cubes, perform real-time analytics on big data, and efficiently execute column-selective queries. It also includes a case study and conclusion.

Elastic Stack IntroductionVikram Shinde

ElasticsearchDivij Sehgal

Introducing the Snowflake Computing Cloud Data WarehouseSnowflake Computing

Introducing Snowflake, an elastic data warehouse delivered as a service in the cloud. It aims to simplify data warehousing by removing the need for customers to manage infrastructure, scaling, and tuning. Snowflake uses a multi-cluster architecture to provide elastic scaling of storage, compute, and concurrency. It can bring together structured and semi-structured data for analysis without requiring data transformation. Customers have seen significant improvements in performance, cost savings, and the ability to add new workloads compared to traditional on-premises data warehousing solutions.

AWS Lake Formation Deep DiveCobus Bernard

This document provides an overview of AWS Lake Formation and related services for building a secure data lake. It discusses how Lake Formation provides a centralized management layer for data ingestion, cleaning, security and access. It also describes how Lake Formation integrates with services like AWS Glue, Amazon S3 and ML transforms to simplify and automate many data lake tasks. Finally, it provides an example workflow for using Lake Formation to deduplicate data from various sources and grant secure access for analysis.

Deploy and Serve Model from Azure Databricks onto Azure Machine LearningDatabricks

The document discusses deploying a model trained in Azure Databricks onto Azure Machine Learning. It covers model training in Databricks, packaging the model and storing it in Azure Blob Storage, registering the model with Azure ML, deploying it to an Azure Kubernetes Service cluster, and serving it as a web service. Demo sections show training a model for semantic type detection in Databricks and deploying it using Azure ML. The goal is to make model deployment and consumption seamless across Azure services.

Data Warehousing Trends, Best Practices, and Future OutlookJames Serra

Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn: - Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart - Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon - Step by step approach to building an effective data warehouse architecture - Common reasons for the failure of data warehouse implementations and how to avoid them

A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks

Build real-time streaming data pipelines to AWS with Confluentconfluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Modern Data ArchitectureAlexey Grishchenko

글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)Amazon Web Services Korea

Data Lake는 오늘날 데이터 기반에 의사 결정을 하기 위한 가장 일반적인 데이터 분석 아키텍처로 떠오르고 있습니다. 잘 설계된 Data Lake는 기업이 데이터 자산으로부터 가장 많은 비지니스 가치를 창출하도록 보장합니다. 본 세션을 통해 AWS 기반의 Data Lake 아키텍처를 소개하고, 다양한 사례를 통해 AWS 고객들은 데이터 분석 플랫폼을 어떤 방식으로 설계해서 활용하고 있는지 살펴봅니다. 다시보기 링크: https://youtu.be/mE8V9oNXdrs

Azure Cosmos DBMohamed Tawfik

Microsoft Azure Cosmos DB is a multi-model database that supports document, key-value, wide-column and graph data models. It provides high throughput, low latency and global distribution across multiple regions. Cosmos DB supports multiple APIs including SQL, MongoDB, Cassandra and Gremlin to allow developers to use their preferred API based on their application needs and skills. It also provides automatic scaling of throughput and storage across all data partitions.

Introdution to Dataops and AIOps (or MLOps)Adrien Blind

An Introduction to Elastic Search.Jurriaan Persyn

Talk given for the #phpbenelux user group, March 27th in Gent (BE), with the goal of convincing developers that are used to build php/mysql apps to broaden their horizon when adding search to their site. Be sure to also have a look at the notes for the slides; they explain some of the screenshots, etc. An accompanying blog post about this subject can be found at http://www.jurriaanpersyn.com/archives/2013/11/18/introduction-to-elasticsearch/

Data platform modernization with Databricks.pptxCalvinSim10

The document discusses modernizing a healthcare organization's data platform from version 1.0 to 2.0 using Azure Databricks. Version 1.0 used Azure HDInsight (HDI) which was challenging to scale and maintain. It presented performance issues and lacked integrations. Version 2.0 with Databricks will provide improved scalability, cost optimization, governance, and ease of use through features like Delta Lake, Unity Catalog, and collaborative notebooks. This will help address challenges faced by consumers, data engineers, and the client.

Achieving Lakehouse Models with Spark 3.0Databricks

It’s very easy to be distracted by the latest and greatest approaches with technology, but sometimes there’s a reason old approaches stand the test of time. Star Schemas & Kimball is one of those things that isn’t going anywhere, but as we move towards the “Data Lakehouse” paradigm – how appropriate is this modelling technique, and how can we harness the Delta Engine & Spark 3.0 to maximise it’s performance?

Anatomy of a data driven architecture - Tamir Dresher Tamir Dresher

Building Data Lakes with Apache AirflowGary Stafford

AWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...Sungmin Kim

How to build Business Intelligence System from scratch on AWS (Day1, Day2) ------------------------------------------------------------------------------------------ 2020-03-18(수)~19(목) 2일 동안 온라인으로 진행한 Online AWS Analytics Immersion Day 전체 발표 자료 입니다. BI(Business Intelligence) 시스템을 설계하는 과정에서 AWS Analytics 서비스들을 어떻게 활용할 수 있는지 설명 드리고자 만든 자료 입니다. Target Audience ------------------- Online Analytics Immersion Day는 다음과 같은 고객을 대상으로 진행됩니다. - AWS Analytics Services (ex. Kinesis, Athena, Redshift, EMR, etc)의 기본 개념을 알고 있지만, 이러한 서비스 활용 방법 및 데이터 분석 시스템 구축 과정이 궁금하신 분 - 데이터 분석 시스템을 구축한 경험은 있지만, 자신이 만든 시스템을 아키텍처 관점에서 어떻게 평가하고 확인할 수 있는지 궁금하신 분

More Related Content

What's hot (20)

Databricks Platform.pptxAlex Ivy

Big data architectures and the data lakeJames Serra

Vector databases and neural searchDmitry Kan

Best Practices in the Use of Columnar DatabasesDATAVERSITY

Elastic Stack IntroductionVikram Shinde

ElasticsearchDivij Sehgal

Introducing the Snowflake Computing Cloud Data WarehouseSnowflake Computing

AWS Lake Formation Deep DiveCobus Bernard

Deploy and Serve Model from Azure Databricks onto Azure Machine LearningDatabricks

Data Warehousing Trends, Best Practices, and Future OutlookJames Serra

A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks

Build real-time streaming data pipelines to AWS with Confluentconfluent

Modern Data ArchitectureAlexey Grishchenko

글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)Amazon Web Services Korea

Azure Cosmos DBMohamed Tawfik

Introdution to Dataops and AIOps (or MLOps)Adrien Blind

An Introduction to Elastic Search.Jurriaan Persyn

Data platform modernization with Databricks.pptxCalvinSim10

Achieving Lakehouse Models with Spark 3.0Databricks

Anatomy of a data driven architecture - Tamir Dresher Tamir Dresher

Databricks Platform.pptxAlex Ivy

Big data architectures and the data lakeJames Serra

Vector databases and neural searchDmitry Kan

Best Practices in the Use of Columnar DatabasesDATAVERSITY

Elastic Stack IntroductionVikram Shinde

ElasticsearchDivij Sehgal

Introducing the Snowflake Computing Cloud Data WarehouseSnowflake Computing

AWS Lake Formation Deep DiveCobus Bernard

Deploy and Serve Model from Azure Databricks onto Azure Machine LearningDatabricks

Data Warehousing Trends, Best Practices, and Future OutlookJames Serra

A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks

Build real-time streaming data pipelines to AWS with Confluentconfluent

Modern Data ArchitectureAlexey Grishchenko

글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)Amazon Web Services Korea

Azure Cosmos DBMohamed Tawfik

Introdution to Dataops and AIOps (or MLOps)Adrien Blind

An Introduction to Elastic Search.Jurriaan Persyn

Data platform modernization with Databricks.pptxCalvinSim10

Achieving Lakehouse Models with Spark 3.0Databricks

Anatomy of a data driven architecture - Tamir Dresher Tamir Dresher

Similar to Building a Data Lake on AWS (9)

Building Data Lakes with Apache AirflowGary Stafford

AWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...Sungmin Kim

AWS Innovate: Build a Data Lake on AWS- Johnathon MeichtryAmazon Web Services Korea

This document discusses building a data lake on AWS. It describes using Amazon S3 for storage of structured, semi-structured, and unstructured data at scale. Amazon Kinesis is used for streaming ingest of data. A metadata catalogue using Amazon DynamoDB and AWS Lambda allows for data discovery and governance. IAM policies control access and encryption using AWS KMS provides security. APIs built using Amazon API Gateway provide programmatic access to the data lake resources.

AWS Big Data LandscapeCrishantha Nanayakkara

This document provides an overview of big data concepts and architectures, as well as AWS big data services. It begins with introducing big data challenges around variety, volume, and velocity of data. It then covers the Hadoop ecosystem including HDFS, MapReduce, Hive, Pig and Spark. The document also discusses data lake architectures and how AWS services like S3, Glue, Athena, EMR, Redshift, QuickSight can be used to build them. Specific services covered in more detail include Kinesis, MSK, Glue, EMR and Redshift. Real-world examples of big data usage are also presented.

Your First Data Lake on AWS_Simon ElishaHelen Rogers

This document discusses building a data lake on AWS. It describes how AWS services like S3, EMR, Glue, Kinesis, Athena and Redshift can be used together to build a scalable data lake for ingesting, storing, processing, analyzing and serving data at any scale. It provides architectures for batch and real-time data ingestion and processing, and interactive analysis on the data lake. Best practices around data formats, compression and partitioning are also covered.

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS Amazon Web Services LATAM

Data lakes allow organizations to store all types of data in a centralized repository at scale. AWS Lake Formation makes it easy to build secure data lakes by automatically registering and cleaning data, enforcing access permissions, and enabling analytics. Data stored in data lakes can be analyzed using services like Amazon Athena, Redshift, and EMR depending on the type of analysis and latency required.

Building Serverless Data Infrastructure in the AWS CloudRyan Plant

DBP-010_Using Azure Data Services for Modern Data Applicationsdecode2016

This document discusses using Azure data services for modern data applications based on the Lambda architecture. It covers ingestion of streaming and batch data using services like Event Hubs, IoT Hubs, and Kafka. It describes processing streaming data in real-time using Stream Analytics, Storm, and Spark Streaming, and processing batch data using HDInsight, ADLA, and Spark. It also covers staging data in data lakes, SQL databases, NoSQL databases and data warehouses. Finally, it discusses serving and exploring data using Power BI and enriching data using Azure Data Factory and Machine Learning.

AWS Tech Talks - Data Lake AnalyticsAmazon Web Services LATAM

This document discusses a webinar on data lakes and analytics hosted by Karlos Correia and Claudio Chiba, AWS solutions architects for the public sector. The agenda covers what a data lake is, why organizations use data lakes, how data lakes expand traditional analytics approaches, and the benefits of data lakes such as centralized data storage and schema-on-read capabilities. Amazon S3 and AWS analytics services are positioned as enabling technologies for building data lakes.

Building Data Lakes with Apache AirflowGary Stafford

AWS Analytics Immersion Day - Build BI System from Scratch (Day1, Day2 Full V...Sungmin Kim

AWS Innovate: Build a Data Lake on AWS- Johnathon MeichtryAmazon Web Services Korea

AWS Big Data LandscapeCrishantha Nanayakkara

Your First Data Lake on AWS_Simon ElishaHelen Rogers

Innovation Track AWS Cloud Experience Argentina - Data Lakes & Analytics en AWS Amazon Web Services LATAM

Building Serverless Data Infrastructure in the AWS CloudRyan Plant

DBP-010_Using Azure Data Services for Modern Data Applicationsdecode2016

AWS Tech Talks - Data Lake AnalyticsAmazon Web Services LATAM

More from Gary Stafford (6)

Stream Processing with Apache Spark, Kafka, Avro, and Apicurio Registry on AW...Gary Stafford

We will read and write messages to and from Amazon MSK in Apache Avro format. We will store the Avro-format Kafka message’s key and value schemas in Apicurio Registry and retrieve the schemas instead of hard-coding the schemas in the PySpark scripts. We will also use the registry to store schemas for CSV-format data files. Link to the blog post and video: https://itnext.io/stream-processing-with-apache-spark-kafka-avro-and-apicurio-registry-on-amazon-emr-and-amazon-13080defa3be

Building Open Data Lakes on AWS with Debezium and Apache HudiGary Stafford

Build a simple open data lake on AWS using a combination of open-source software (OSS), including Red Hat’s Debezium, Apache Kafka, and Kafka Connect for change data capture (CDC), and Apache Hive, Apache Spark, Apache Hudi, and Hudi’s DeltaStreamer for managing our data lake. We will use fully-managed AWS services to host the open data lake components, including Amazon RDS, Amazon MKS, Amazon EKS, and EMR. Link to the blog post and video: https://garystafford.medium.com/building-open-data-lakes-with-debezium-and-apache-hudi-c3370d3f86fb

How Mature is Your Infrastructure?Gary Stafford

Version 2 of the IaC Maturity Model Presentation What helps leading technology companies like Facebook, Amazon, Netflix, and Etsy increase their speed to market while lowering overall IT costs and increasing customer satisfaction? Examine how to apply the principles from Humble and Farley’s Continuous Delivery Maturity Model to the concepts found in Morris’ Infrastructure as Code, using the new Infrastructure as Code Maturity Model. Link to v2.1 of the IaC Maturity Model: https://github.com/garystafford/cd-maturity-model/raw/requirejs/images/IaC_Maturity_Model%20v2_1.pdf

Infrastructure as Code Maturity Model v1Gary Stafford

Enterprise DevOps Adoption LinkedInGary Stafford

This document discusses patterns of organizational structure and adoption for enterprise DevOps teams. It describes common organizational structure patterns such as separate development and operations teams, renaming operations to DevOps, and forming combined DevOps teams. It also outlines adoption patterns like starting with small automation efforts, using the strangler pattern to transition applications to the cloud, conducting a DevOps maturity model gap analysis, and ultimately enabling DevOps self-service. The document provides examples and considerations for various DevOps organizational structures and adoption approaches in enterprises.

From Zurich to the Cosmos, by Artist Steve CarpenterGary Stafford

Stream Processing with Apache Spark, Kafka, Avro, and Apicurio Registry on AW...Gary Stafford

Building Open Data Lakes on AWS with Debezium and Apache HudiGary Stafford

How Mature is Your Infrastructure?Gary Stafford

Infrastructure as Code Maturity Model v1Gary Stafford

Enterprise DevOps Adoption LinkedInGary Stafford

From Zurich to the Cosmos, by Artist Steve CarpenterGary Stafford

Recently uploaded (20)

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

I was devastated when I realized that I had fallen victim to an online fraud, losing a significant amount of money in the process. After countless hours of searching for a solution, I came across iCode cybertech. From the moment I reached out to their team, I felt a sense of hope that I can recommend iCode Cybertech enough for anyone who has faced similar challenges. Their commitment to helping clients and their exceptional service truly set them apart. Thank you, iCode cybertech, for turning my situation around! [email protected]

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

Conic Sectionfaggavahabaayhahahahahs.pptxtaiwanesechetan

Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...Abodahab

Classification_in_Machinee_Learning.pptxwencyjorda88

Minions Want to eat presentacion muy lindaCarlaAndradesSoler1

Geometry maths presentation for begginerszrjacob283

Thingyan is now a global treasure! See how people around the world are search...Pixellion

chapter3 Central Tendency statistics.pptjustinebandajbn

Medical Dataset including visualizationsvishrut8750588758

C++_OOPs_DSA1_Presentation_Template.pptxaquibnoor22079

chapter 4 Variability statistical research .pptxjustinebandajbn

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

Data Analytics Overview and its applicationsJanmejayaMishra7

183409-christina-rossetti.pdfdsfsdasggsagfardin123rahman07

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

By James Francis, CEO of Paradigm Asset Management In the landscape of urban safety innovation, Mt. Vernon is emerging as a compelling case study for neighboring Westchester County cities. The municipality’s recently launched Public Safety Camera Program not only represents a significant advancement in community protection but also offers valuable insights for New Rochelle and White Plains as they consider their own safety infrastructure enhancements.

computer organization and assembly language.docxalisoftwareengineer1

DPR_Expert_Recruitment_notice_Revised.pdfinmishra17121973

04302025_CCC TUG_DataVista: The Design Storyccctableauusergroup

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

Conic Sectionfaggavahabaayhahahahahs.pptxtaiwanesechetan

Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...Abodahab

Classification_in_Machinee_Learning.pptxwencyjorda88

Minions Want to eat presentacion muy lindaCarlaAndradesSoler1

Geometry maths presentation for begginerszrjacob283

Thingyan is now a global treasure! See how people around the world are search...Pixellion

chapter3 Central Tendency statistics.pptjustinebandajbn

Medical Dataset including visualizationsvishrut8750588758

C++_OOPs_DSA1_Presentation_Template.pptxaquibnoor22079

chapter 4 Variability statistical research .pptxjustinebandajbn

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

Data Analytics Overview and its applicationsJanmejayaMishra7

183409-christina-rossetti.pdfdsfsdasggsagfardin123rahman07

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

computer organization and assembly language.docxalisoftwareengineer1

DPR_Expert_Recruitment_notice_Revised.pdfinmishra17121973

04302025_CCC TUG_DataVista: The Design Storyccctableauusergroup

Building a Data Lake on AWS

1. Data Lake Demonstration Building a Simple Data Lake on AWS Gary A. Stafford

2. Twitter/LinkedIn GaryStafford Blog garystafford.medium.com

3. What is a Data Lake?

4. Agenda What is a Data Lake? Dataset Source Code Architecture Demonstration

5. What is a Data Lake? “A data lake is a central location that holds a large amount of data in its native, raw format. Compared to a hierarchical data warehouse, which stores data in files or folders, a data lake uses a flat architecture and object storage to store the data.” - Databricks “A centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.” - AWS

6. What is a Data Lake?

7. Dataset

8. Dataset TICKIT database E-commerce platform Bringing together buyers and sellers of tickets to entertainments events Designed for demonstrating Amazon Redshift Small database consists of seven tables: two fact tables and five dimensions Tables: Categories, Events, Venues, Users, Listings, Sales, Dates https://docs.aws.amazon.com/redshift/latest/dg/c_sampledb.html

9. Dataset

10. Dataset Table Simulated Data Source Demo Data Source Category Software as a Service (SaaS) Amazon RDS for PostgreSQL Event Software as a Service (SaaS) Amazon RDS for PostgreSQL Venue Software as a Service (SaaS) Amazon RDS for PostgreSQL Listing Ecommerce Platform Amazon RDS for MySQL Sales Ecommerce Platform Amazon RDS for MySQL Date Ecommerce Platform Amazon RDS for MySQL Users Customer Relationship Management (CRM) Microsoft SQL Server

11. Dataset

12. Source Code

13. github.com/garystafford/tickit-data-lake-demo

14. Architecture

15. Architecture: AWS Services Used AWS Glue Studio (alt. AWS Glue DataBrew) AWS Glue Data Catalog (alt. Apache Hive on EMR) AWS Glue Crawlers (alt. CDC with Kafka Connect or DMS) AWS Glue Jobs (alt. AWS Glue DataBrew, or Spark or Presto on EMR) Amazon Athena (alt. Presto on EMR) Amazon S3

17. Architecture: Out of Scope (but critically important) Change Data Capture (CDC): handling changes to systems of record Transactional Storage Layer: Apache Hudi, Apache Iceberg, Delta Lake Streaming Data: Spark Structured Streaming, Kinesis, Flink Fine-grained Authorization: database-, table-, column-, and row-level access Data Lineage: Tracking data as it flows from sources to consumption

18. Architecture: Out of Scope (but critically important) Data Inspection: Scanning incoming data for sensitive info such as PII DevOps/DataOps: Automating testing, deployment, job execution Data Warehouse / Lake House Architecture Data Lake Storage Tiering

19. Demonstration

Building a Data Lake on AWS

Recommended

More Related Content

What's hot (20)

Similar to Building a Data Lake on AWS (9)

More from Gary Stafford (6)

Recently uploaded (20)

Building a Data Lake on AWS