Super Sizing Youtube with Python
51 likes15,339 views
This document summarizes Mike Solomon's experience scaling YouTube's architecture using Python. It discusses: 1) Starting simply with all services on the same boxes before gradually separating services like search and thumbnails onto their own machines. 2) Systematically removing bottlenecks revealed by user demand growth and replacing components to maintain scalability. 3) Using caching, database partitioning, and bulk data migration techniques to scale the MySQL database horizontally as user traffic increased. 4) Favoring simplicity, customizing open source software, and taking a "Pythonic" approach to build scalable, transparent APIs.
1 of 25
Downloaded 414 times
Ad
Recommended
Bigtable
Bigtablezafargilani Bigtable is a distributed storage system for managing large structured datasets. It uses a sparse distributed multidimensional sorted map as its data model. Bigtable has been widely adopted by Google for applications requiring scalability, high performance, and high availability, serving over 60 Google products. It utilizes various technologies like the Google File System for storage and Chubby for coordination.
Apache hive introduction
Apache hive introductionMahmood Reza Esmaili Zand Apache Hive is a data warehouse software built on top of Hadoop that allows users to query data stored in various databases and file systems using an SQL-like interface. It provides a way to summarize, query, and analyze large datasets stored in Hadoop distributed file system (HDFS). Hive gives SQL capabilities to analyze data without needing MapReduce programming. Users can build a data warehouse by creating Hive tables, loading data files into HDFS, and then querying and analyzing the data using HiveQL, which Hive then converts into MapReduce jobs.
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks Recently, a set of modern table formats such as Delta Lake, Hudi, Iceberg spring out. Along with Hive Metastore these table formats are trying to solve problems that stand in traditional data lake for a long time with their declared features like ACID, schema evolution, upsert, time travel, incremental consumption etc.
Spark shuffle introduction
Spark shuffle introductioncolorant This document discusses Spark shuffle, which is an expensive operation that involves data partitioning, serialization/deserialization, compression, and disk I/O. It provides an overview of how shuffle works in Spark and the history of optimizations like sort-based shuffle and an external shuffle service. Key concepts discussed include shuffle writers, readers, and the pluggable block transfer service that handles data transfer. The document also covers shuffle-related configuration options and potential future work.
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...
The Top Five Mistakes Made When Writing Streaming Applications with Mark Grov...Databricks So you know you want to write a streaming app, but any non-trivial streaming app developer would have to think about these questions:
– How do I manage offsets?
– How do I manage state?
– How do I make my Spark Streaming job resilient to failures? Can I avoid some failures?
– How do I gracefully shutdown my streaming job?
– How do I monitor and manage my streaming job (i.e. re-try logic)?
– How can I better manage the DAG in my streaming job?
– When do I use checkpointing, and for what? When should I not use checkpointing?
– Do I need a WAL when using a streaming data source? Why? When don’t I need one?
This session will share practices that no one talks about when you start writing your streaming app, but you’ll inevitably need to learn along the way.
Snowflake Overview
Snowflake OverviewSnowflake Computing Organizations are struggling to make sense of their data within antiquated data platforms. Snowflake, the data warehouse built for the cloud, can help.
Big data architectures and the data lake
Big data architectures and the data lakeJames Serra The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include:
- Defining top-down and bottom-up approaches to data management
- Explaining what a data lake is and how Hadoop can function as the data lake
- Describing how a modern data warehouse combines features of a traditional data warehouse and data lake
- Discussing how federated querying allows data to be accessed across multiple sources
- Highlighting benefits of implementing big data solutions in the cloud
- Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
Simplify CDC Pipeline with Spark Streaming SQL and Delta LakeDatabricks Change Data Capture CDC is a typical use case in Real-Time Data Warehousing. It tracks the data change log -binlog- of a relational database [OLTP], and replay these change log timely to an external storage to do Real-Time OLAP, such as delta/kudu. To implement a robust CDC streaming pipeline, lots of factors should be concerned, such as how to ensure data accuracy , how to process OLTP source schema changed, whether it is easy to build for variety databases with less code.
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergFlink Forward Flink Forward San Francisco 2022.
Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features.
by
Andreas Hailu
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsGuido Schmutz Independent of the source of data, the integration and analysis of event streams gets more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. In this session we compare two popular Streaming Analytics solutions: Spark Streaming and Kafka Streams.
Spark is fast and general engine for large-scale data processing and has been designed to provide a more efficient alternative to Hadoop MapReduce. Spark Streaming brings Spark's language-integrated API to stream processing, letting you write streaming applications the same way you write batch jobs. It supports both Java and Scala.
Kafka Streams is the stream processing solution which is part of Kafka. It is provided as a Java library and by that can be easily integrated with any Java application.
This presentation shows how you can implement stream processing solutions with each of the two frameworks, discusses how they compare and highlights the differences and similarities.
Instana - ClickHouse presentation
Instana - ClickHouse presentationMiel Donkers Presentation on how we at Instana use ClickHouse. The problems we came across and how we solved them.
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann This presentation introduces Apache Flink, a massively parallel data processing engine which currently undergoes the incubation process at the Apache Software Foundation. Flink's programming primitives are presented and it is shown how easily a distributed PageRank algorithm can be implemented with Flink. Intriguing features such as dedicated memory management, Hadoop compatibility, streaming and automatic optimisation make it an unique system in the world of Big Data processing.
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveDataWorks Summit This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
Common Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta LakehouseDatabricks The Delta Architecture pattern has made the lives of data engineers much simpler, but what about improving query performance for data analysts? What are some common places to look at for tuning query performance? In this session we will cover some common techniques to apply to our delta tables to make them perform better for data analysts queries. We will look at a few examples of how you can analyze a query, and determine what to focus on to deliver better performance results.
Apache Spark Architecture
Apache Spark ArchitectureAlexey Grishchenko This is the presentation I made on JavaDay Kiev 2015 regarding the architecture of Apache Spark. It covers the memory model, the shuffle implementations, data frames and some other high-level staff and can be used as an introduction to Apache Spark
Key-Value NoSQL Database
Key-Value NoSQL DatabaseHeman Hosainpana This document discusses different types of distributed databases. It covers data models like relational, aggregate-oriented, key-value, and document models. It also discusses different distribution models like sharding and replication. Consistency models for distributed databases are explained including eventual consistency and the CAP theorem. Key-value stores are described in more detail as a simple but widely used data model with features like consistency, scaling, and suitable use cases. Specific key-value databases like Redis, Riak, and DynamoDB are mentioned.
Big data-analytics-cpe8035
Big data-analytics-cpe8035Neelam Rawat This document provides an introduction to big data, including its key characteristics of volume, velocity, and variety. It describes different types of big data technologies like Hadoop, MapReduce, HDFS, Hive, and Pig. Hadoop is an open source software framework for distributed storage and processing of large datasets across clusters of computers. MapReduce is a programming model used for processing large datasets in a distributed computing environment. HDFS provides a distributed file system for storing large datasets across clusters. Hive and Pig provide data querying and analysis capabilities for data stored in Hadoop clusters using SQL-like and scripting languages respectively.
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's PerspectiveDatabricks This document describes the Delta architecture, which unifies batch and streaming data processing. Delta achieves this through a continuous data flow model using structured streaming. It allows data engineers to read consistent data while being written, incrementally read large tables at scale, rollback in case of errors, replay and process historical data along with new data, and handle late arriving data without delays. Delta uses transaction logging, optimistic concurrency, and Spark to scale metadata handling for large tables. This provides a simplified solution to common challenges data engineers face.
Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And PracticeAdam Kawa This document provides an overview of Apache Hadoop and its two main components - HDFS and MapReduce. It describes the fundamental ideas behind Hadoop such as storing data reliably across commodity hardware and moving computation to data. It then discusses HDFS in more detail, explaining how it stores very large files reliably through data replication and partitioning files into blocks. It also covers the roles of the NameNode and DataNodes and common HDFS commands. Finally, it discusses some challenges encountered when using HDFS in practice and potential solutions.
Azure Data Factory presentation with links
Azure Data Factory presentation with linksChris Testa-O'Neill This is a slide deck that went through the fundamentals of Azure Data Factory. There are links to further information.
Accelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks AutoloaderDatabricks Tracking which incoming files have been processed has always required thought and design when implementing an ETL framework. The Autoloader feature of Databricks looks to simplify this, taking away the pain of file watching and queue management. However, there can also be a lot of nuance and complexity in setting up Autoloader and managing the process of ingesting data using it. After implementing an automated data loading process in a major US CPMG, Simon has some lessons to share from the experience.
This session will run through the initial setup and configuration of Autoloader in a Microsoft Azure environment, looking at the components used and what is created behind the scenes. We’ll then look at some of the limitations of the feature, before walking through the process of overcoming these limitations. We will build out a practical example that tackles evolving schemas, applying transformations to your stream, extracting telemetry from the process and finally, how to merge the incoming data into a Delta table.
After this session you will be better equipped to use Autoloader in a data ingestion platform, simplifying your production workloads and accelerating the time to realise value in your data!
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit The document compares the query execution plans produced by Apache Hive and PostgreSQL. It shows that Hive's old-style execution plans are overly verbose and difficult to understand, providing many low-level details across multiple stages. In contrast, PostgreSQL's plans are more concise and readable, showing the logical query plan in a top-down manner with actual table names and fewer lines of text. The document advocates for Hive to adopt a simpler execution plan format similar to PostgreSQL's.
Cassandra Introduction & Features
Cassandra Introduction & FeaturesDataStax Academy This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performanceDataWorks Summit Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years.
Inspired by the increasingly complex SQL queries run by the Presto user community, engineers at Facebook and Starburst have recently focused on cost-based query optimization. In this talk we will present the initial design and implementation of the CBO, support for connector-provided statistics, estimating selectivity, and choosing efficient query plans. Then, our detailed experimental evaluation will illustrate the performance gains for several classes of queries achieved thanks to the optimizer. Finally, we will discuss our future work enhancing the initial CBO and present the general Presto roadmap for 2018 and beyond.
Speakers
Kamil Bajda-Pawlikowski, Starburst Data, CTO & Co-Founder
Martin Traverso
Hive join optimizations
Hive join optimizationsSzehon Ho February 2015 Hive User Group meetup at LinkedIn
http://www.meetup.com/Hive-User-Group-Meeting/events/219794523/
Presentation about physical join strategies employed used by Apache Hive and how they may be employed to optimize workflows.
HBaseCon 2012 | HBase Schema Design - Ian Varley, Salesforce
HBaseCon 2012 | HBase Schema Design - Ian Varley, SalesforceCloudera, Inc. Most developers are familiar with the topic of “database design”. In the relational world, normalization is the name of the game. How do things change when you’re working with a scalable, distributed, non-SQL database like HBase? This talk will cover the basics of HBase schema design at a high level and give several common patterns and examples of real-world schemas to solve interesting problems. The storage and data access architecture of HBase (row keys, column families, etc.) will be explained, along with the pros and cons of different schema decisions.
Introduction to azure cosmos db
Introduction to azure cosmos dbRatan Parai Azure Cosmos DB is Microsoft's globally distributed, multi-model database service that supports multiple APIs such as SQL, Cassandra, MongoDB, Gremlin and Azure Table. It allows storing entities with automatic partitioning and provides automatic online backups every 4 hours with the latest 2 backups stored. The Azure Cosmos DB change feed and Data Migration Tool allow importing and exporting data for backups. An emulator is also available for trying Cosmos DB locally without an Azure account.
Data Applications and Infrastructure at LinkedIn__HadoopSummit2010
Data Applications and Infrastructure at LinkedIn__HadoopSummit2010Yahoo Developer Network Hadoop Summit 2010 - application track
Data Applications and Infrastructure at LinkedIn
Jay Kreps, LinkedIn
The Evolution of a Scrappy Startup to a Successful Web Service
The Evolution of a Scrappy Startup to a Successful Web ServicePoornima Vijayashanker Mint.com started as a prototype created by the author using open source tools with no prior startup experience. The initial prototype focused on differentiating features like aggregating financial accounts and transactions. As users grew, performance issues arose due to increased load on servers and databases. To address these growing pains, the architecture was optimized by separating tiers, adding caching, database sharding, and more. Key lessons were to focus first on critical user problems in prototypes, continuously measure performance, and optimize based on demand to balance latency, throughput, and quality as the user base expanded.
Ad
More Related Content
What's hot (20)
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergFlink Forward Flink Forward San Francisco 2022.
Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features.
by
Andreas Hailu
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsGuido Schmutz Independent of the source of data, the integration and analysis of event streams gets more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. In this session we compare two popular Streaming Analytics solutions: Spark Streaming and Kafka Streams.
Spark is fast and general engine for large-scale data processing and has been designed to provide a more efficient alternative to Hadoop MapReduce. Spark Streaming brings Spark's language-integrated API to stream processing, letting you write streaming applications the same way you write batch jobs. It supports both Java and Scala.
Kafka Streams is the stream processing solution which is part of Kafka. It is provided as a Java library and by that can be easily integrated with any Java application.
This presentation shows how you can implement stream processing solutions with each of the two frameworks, discusses how they compare and highlights the differences and similarities.
Instana - ClickHouse presentation
Instana - ClickHouse presentationMiel Donkers Presentation on how we at Instana use ClickHouse. The problems we came across and how we solved them.
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann This presentation introduces Apache Flink, a massively parallel data processing engine which currently undergoes the incubation process at the Apache Software Foundation. Flink's programming primitives are presented and it is shown how easily a distributed PageRank algorithm can be implemented with Flink. Intriguing features such as dedicated memory management, Hadoop compatibility, streaming and automatic optimisation make it an unique system in the world of Big Data processing.
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveDataWorks Summit This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
Common Strategies for Improving Performance on Your Delta Lakehouse
Common Strategies for Improving Performance on Your Delta LakehouseDatabricks The Delta Architecture pattern has made the lives of data engineers much simpler, but what about improving query performance for data analysts? What are some common places to look at for tuning query performance? In this session we will cover some common techniques to apply to our delta tables to make them perform better for data analysts queries. We will look at a few examples of how you can analyze a query, and determine what to focus on to deliver better performance results.
Apache Spark Architecture
Apache Spark ArchitectureAlexey Grishchenko This is the presentation I made on JavaDay Kiev 2015 regarding the architecture of Apache Spark. It covers the memory model, the shuffle implementations, data frames and some other high-level staff and can be used as an introduction to Apache Spark
Key-Value NoSQL Database
Key-Value NoSQL DatabaseHeman Hosainpana This document discusses different types of distributed databases. It covers data models like relational, aggregate-oriented, key-value, and document models. It also discusses different distribution models like sharding and replication. Consistency models for distributed databases are explained including eventual consistency and the CAP theorem. Key-value stores are described in more detail as a simple but widely used data model with features like consistency, scaling, and suitable use cases. Specific key-value databases like Redis, Riak, and DynamoDB are mentioned.
Big data-analytics-cpe8035
Big data-analytics-cpe8035Neelam Rawat This document provides an introduction to big data, including its key characteristics of volume, velocity, and variety. It describes different types of big data technologies like Hadoop, MapReduce, HDFS, Hive, and Pig. Hadoop is an open source software framework for distributed storage and processing of large datasets across clusters of computers. MapReduce is a programming model used for processing large datasets in a distributed computing environment. HDFS provides a distributed file system for storing large datasets across clusters. Hive and Pig provide data querying and analysis capabilities for data stored in Hadoop clusters using SQL-like and scripting languages respectively.
Delta from a Data Engineer's Perspective
Delta from a Data Engineer's PerspectiveDatabricks This document describes the Delta architecture, which unifies batch and streaming data processing. Delta achieves this through a continuous data flow model using structured streaming. It allows data engineers to read consistent data while being written, incrementally read large tables at scale, rollback in case of errors, replay and process historical data along with new data, and handle late arriving data without delays. Delta uses transaction logging, optimistic concurrency, and Spark to scale metadata handling for large tables. This provides a simplified solution to common challenges data engineers face.
Apache Hadoop In Theory And Practice
Apache Hadoop In Theory And PracticeAdam Kawa This document provides an overview of Apache Hadoop and its two main components - HDFS and MapReduce. It describes the fundamental ideas behind Hadoop such as storing data reliably across commodity hardware and moving computation to data. It then discusses HDFS in more detail, explaining how it stores very large files reliably through data replication and partitioning files into blocks. It also covers the roles of the NameNode and DataNodes and common HDFS commands. Finally, it discusses some challenges encountered when using HDFS in practice and potential solutions.
Azure Data Factory presentation with links
Azure Data Factory presentation with linksChris Testa-O'Neill This is a slide deck that went through the fundamentals of Azure Data Factory. There are links to further information.
Accelerating Data Ingestion with Databricks Autoloader
Accelerating Data Ingestion with Databricks AutoloaderDatabricks Tracking which incoming files have been processed has always required thought and design when implementing an ETL framework. The Autoloader feature of Databricks looks to simplify this, taking away the pain of file watching and queue management. However, there can also be a lot of nuance and complexity in setting up Autoloader and managing the process of ingesting data using it. After implementing an automated data loading process in a major US CPMG, Simon has some lessons to share from the experience.
This session will run through the initial setup and configuration of Autoloader in a Microsoft Azure environment, looking at the components used and what is created behind the scenes. We’ll then look at some of the limitations of the feature, before walking through the process of overcoming these limitations. We will build out a practical example that tackles evolving schemas, applying transformations to your stream, extracting telemetry from the process and finally, how to merge the incoming data into a Delta table.
After this session you will be better equipped to use Autoloader in a data ingestion platform, simplifying your production workloads and accelerating the time to realise value in your data!
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit The document compares the query execution plans produced by Apache Hive and PostgreSQL. It shows that Hive's old-style execution plans are overly verbose and difficult to understand, providing many low-level details across multiple stages. In contrast, PostgreSQL's plans are more concise and readable, showing the logical query plan in a top-down manner with actual table names and fewer lines of text. The document advocates for Hive to adopt a simpler execution plan format similar to PostgreSQL's.
Cassandra Introduction & Features
Cassandra Introduction & FeaturesDataStax Academy This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performanceDataWorks Summit Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years.
Inspired by the increasingly complex SQL queries run by the Presto user community, engineers at Facebook and Starburst have recently focused on cost-based query optimization. In this talk we will present the initial design and implementation of the CBO, support for connector-provided statistics, estimating selectivity, and choosing efficient query plans. Then, our detailed experimental evaluation will illustrate the performance gains for several classes of queries achieved thanks to the optimizer. Finally, we will discuss our future work enhancing the initial CBO and present the general Presto roadmap for 2018 and beyond.
Speakers
Kamil Bajda-Pawlikowski, Starburst Data, CTO & Co-Founder
Martin Traverso
Hive join optimizations
Hive join optimizationsSzehon Ho February 2015 Hive User Group meetup at LinkedIn
http://www.meetup.com/Hive-User-Group-Meeting/events/219794523/
Presentation about physical join strategies employed used by Apache Hive and how they may be employed to optimize workflows.
HBaseCon 2012 | HBase Schema Design - Ian Varley, Salesforce
HBaseCon 2012 | HBase Schema Design - Ian Varley, SalesforceCloudera, Inc. Most developers are familiar with the topic of “database design”. In the relational world, normalization is the name of the game. How do things change when you’re working with a scalable, distributed, non-SQL database like HBase? This talk will cover the basics of HBase schema design at a high level and give several common patterns and examples of real-world schemas to solve interesting problems. The storage and data access architecture of HBase (row keys, column families, etc.) will be explained, along with the pros and cons of different schema decisions.
Introduction to azure cosmos db
Introduction to azure cosmos dbRatan Parai Azure Cosmos DB is Microsoft's globally distributed, multi-model database service that supports multiple APIs such as SQL, Cassandra, MongoDB, Gremlin and Azure Table. It allows storing entities with automatic partitioning and provides automatic online backups every 4 hours with the latest 2 backups stored. The Azure Cosmos DB change feed and Data Migration Tool allow importing and exporting data for backups. An emulator is also available for trying Cosmos DB locally without an Azure account.
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit
Similar to Super Sizing Youtube with Python (20)
Data Applications and Infrastructure at LinkedIn__HadoopSummit2010
Data Applications and Infrastructure at LinkedIn__HadoopSummit2010Yahoo Developer Network Hadoop Summit 2010 - application track
Data Applications and Infrastructure at LinkedIn
Jay Kreps, LinkedIn
The Evolution of a Scrappy Startup to a Successful Web Service
The Evolution of a Scrappy Startup to a Successful Web ServicePoornima Vijayashanker Mint.com started as a prototype created by the author using open source tools with no prior startup experience. The initial prototype focused on differentiating features like aggregating financial accounts and transactions. As users grew, performance issues arose due to increased load on servers and databases. To address these growing pains, the architecture was optimized by separating tiers, adding caching, database sharding, and more. Key lessons were to focus first on critical user problems in prototypes, continuously measure performance, and optimize based on demand to balance latency, throughput, and quality as the user base expanded.
PHP – Faster And Cheaper. Scale Vertically with IBM i
PHP – Faster And Cheaper. Scale Vertically with IBM iSam Hennessy The only way to scale your PHP application is horizontally? If you believe that then you could be missing a huge opportunity. This talk will layout why scaling vertically with the Power System platform can be a superior alternative to a traditional LAMP stack. With simplify development, reduce operation costs and a true enterprise quality database
UnConference for Georgia Southern Computer Science March 31, 2015
UnConference for Georgia Southern Computer Science March 31, 2015Christopher Curtin I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
Natural Laws of Software Performance
Natural Laws of Software PerformanceGibraltar Software Just like you can't defeat the laws of physics there are natural laws that ultimately decide software performance. Even the latest technology beta is still bound by Newton's laws, and you can't change the speed of light, even in the cloud!
Architecting a Large Software Project - Lessons Learned
Architecting a Large Software Project - Lessons LearnedJoão Pedro Martins In large projects, the Software Architect role includes both the client communication and requirements management, and the solution design itself, making sure technical quality is garanteed. This presentation describes the lessons learned with a highly successful project that took 3 years of development until its production phase, in technical, functional, and architeture aspects.
Presented at the 50th meeting of the Netponto Community in Lisboa, Portugal.
Intro to Cloud Architecture
Intro to Cloud Architecturewlscaudill An introduction to thinking about the structure of your software and why it's necessary to achieve redundancy and scale.
Handling Data in Mega Scale Systems
Handling Data in Mega Scale SystemsDirecti Group This document discusses strategies for handling large amounts of data in web applications. It begins by providing examples of how much data some large websites contain, ranging from terabytes to petabytes. It then covers various techniques for scaling data handling capabilities including vertical and horizontal scaling, replication, partitioning, consistency models, normalization, caching, and using different data engine types beyond relational databases. The key lessons are that data volumes continue growing rapidly, and a variety of techniques are needed to scale across servers, datacenters, and provide high performance and availability.
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...
Engineering Machine Learning Data Pipelines Series: Streaming New Data as It ...Precisely This document discusses engineering machine learning data pipelines and addresses five big challenges: 1) scattered and difficult to access data, 2) data cleansing at scale, 3) entity resolution, 4) tracking data lineage, and 5) ongoing real-time changed data capture and streaming. It presents DMX Change Data Capture as a solution to capture changes from various data sources and replicate them in real-time to targets like Kafka, HDFS, databases and data lakes to feed machine learning models. Case studies demonstrate how DMX-h has helped customers like a global hotel chain and insurance and healthcare companies build scalable data pipelines.
Modern Application Stacks
Modern Application Stackschartjes Slides for a presentation I gave on May 4, 2011 at the GTA PHP user's group (http://meetup.gtaphp.org) about components for modern PHP application stacks
SQL Server Integration Services and Analysis Services
SQL Server Integration Services and Analysis ServicesMohan Arumugam SQL Server Integration Services and Analysis Services
AWS case study: real estate portal
AWS case study: real estate portalAndreas Chatzakis Spitogatos.gr is a leading real estate portal in Greece.
This presentation describes high level architecture and the learnings of a recent migration from dedicated servers to a more scalable solution in the cloud of Amazon.
Presentation was given at a meetup of AWS Usergroup Greece (AWSUGGR).
Class 7: Introduction to web technology entrepreneurship
Class 7: Introduction to web technology entrepreneurshipallanchao This document summarizes a lecture about building web applications. It discusses that building a web application typically involves teams with roles like developers, designers, project managers. It covers topics like hosting options, deployment processes, project management tools and techniques, and hiring and managing developers. The document provides overviews and comparisons of hosting options like dedicated servers, shared hosting, virtual private servers, and cloud hosting. It also discusses deployment methods, environments, and tools. Project management topics covered include release planning, agile methodologies, budgeting, and tips for successful projects. The document ends with sections on hiring developers, managing developers, code reviews, and outsourcing.
Web Speed And Scalability
Web Speed And ScalabilityJason Ragsdale The document provides tips for building a scalable and high-performance website, including using caching, load balancing, and monitoring. It discusses horizontal and vertical scalability, and recommends planning, testing, and version control. Specific techniques mentioned include static content caching, Memcached, and the YSlow performance tool.
Db trends final
Db trends finalCraig Mullins Trends and issues impacting database management systems circa 2004 included increasing complexity, lack of resources, and rapid changes in technology. New database management system versions were being released frequently with new features enabled for the internet and real-time usage. Emerging technologies like Java, .NET, and XML were becoming more widely adopted and database systems were taking on additional functionality beyond traditional querying and storage. The internet was driving changes requiring database administrators to have new skills to support increasingly complex enterprise infrastructure and applications.
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror Twitter's operations team manages software performance, availability, capacity planning, and configuration management for Twitter. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, and optimizing databases to reduce replication delay and locks. The team also created several open source projects like CacheMoney for caching and Kestrel for asynchronous messaging.
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The operations team manages software performance, availability, capacity planning, and configuration management using metrics, logs, and data-driven analysis to find weak points and take corrective action. They use managed services for infrastructure to focus on computer science problems. The document outlines Twitter's rapid growth and challenges in maintaining performance as traffic increases. It provides recommendations around caching, databases, asynchronous processing, and other techniques Twitter uses to optimize performance under heavy load.
Fixing_Twitter
Fixing_Twitterliujianrong Twitter's operations team manages software performance, availability, capacity planning, and configuration management. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, optimizing databases, and instrumenting all systems. Their goal is to process requests asynchronously when possible and avoid overloading relational databases.
Fixing twitter
Fixing twitterRoger Xia Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The Twitter operations team focuses on software performance, availability, capacity planning, and configuration management using metrics, logs, and science. They use a dedicated managed services team and run their own servers instead of cloud services. The document outlines Twitter's rapid growth and challenges in maintaining performance. It discusses strategies for monitoring, analyzing metrics to find weak points, deploying changes, and improving processes through configuration management and peer reviews.
Gavin M
Gavin MOntico - myYearbook.com grew from 100 million pageviews in 2007 to 2.5 billion pageviews in 2008 and achieved 99.94% uptime, scaling their infrastructure significantly.
- They migrated to a service-oriented architecture in 2008 to improve scalability and adopted technologies like PHP, PostgreSQL, Memcached, and Apache ActiveMQ to partition data and distribute load across servers.
- Lessons learned include planning for growth, scaling PostgreSQL horizontally and vertically, using caching, load balancing, and message queues to improve performance under high loads.
Ad
More from didip (9)
High Performance Erlang
High Performance Erlangdidip The document discusses the past, present, and future of HiPE (High Performance Erlang), a just-in-time native code compiler for Erlang. It provides details on recent versions of HiPE that are fully integrated with Erlang/OTP and allow compilation to SPARC and x86 backends. It also outlines current work on optimizations, formalizing Core Erlang, and exploring alternative memory architectures and language extensions.
Why I Love Python
Why I Love Pythondidip The document discusses why the author loves Python programming language. Some key reasons include:
1. Python allows the author to focus on concepts rather than fighting with compiler bugs or syntax issues found in other languages.
2. Python code is more readable and compact due to consistent formatting and use of idioms, allowing for rapid understanding.
3. The author finds Python more productive than other languages due to reduced typing and not having to wade through as much code.
Turbogears Presentation
Turbogears Presentationdidip This document provides an introduction to TurboGears, a Python web framework. It describes TurboGears as a "megaframework" and Pythonic answer to Ruby on Rails that enables rapid web development. The document summarizes key TurboGears components like SQLObject (ORM), CherryPy (application server), Kid (templating), and MochiKit (JavaScript library). It provides examples of using these components and describes how TurboGears integrates them to quickly generate a basic application structure and get started developing. Testing with TestGears/Nose and a sample "20-minute wiki" app are also briefly outlined.
Ecma 262
Ecma 262didip This document summarizes the third edition of the ECMA-262 standard, which specifies the ECMAScript language. It provides a brief history of ECMAScript, which originated from JavaScript and JScript. The standard was adopted by ECMA International in December 1999 and defines features like regular expressions, string handling, control statements, exception handling, and error definitions for the ECMAScript language. Work continues on enhancing ECMAScript with capabilities like scripting across the internet and tighter coordination with other standards bodies. The document outlines the standard's scope, references, language overview, notational conventions, lexical conventions, types, type conversion, execution contexts, and expressions.
Socket Programming In Python
Socket Programming In Pythondidip This document provides an overview of sockets programming in Python. It discusses:
1) The basic Python sockets modules (Socket and SocketServer) which provide low-level networking and simplified server development.
2) Examples of creating stream and datagram sockets, binding addresses, listening for connections, and accepting clients.
3) How the SocketServer module simplifies creating TCP servers with a basic "Hello World" example server.
Google Research Paper
Google Research Paperdidip By, of course, Larry Page & Sergei Brin
For further scalability tales, go to: http://rapd.wordpress.com
Feed Burner Scalability
Feed Burner Scalabilitydidip FeedBurner is a feed management provider that has scaled to serve 270,000 feeds and 100 million hits per day. The document discusses FeedBurner's scaling history and the solutions they implemented to address various scalability problems, including using load balancers and health checks, caching with Memcached, batch processing, partitioning tables, and implementing a disaster recovery plan using a "Panic App" to download and synchronize feeds.
Theory Psyco
Theory Psycodidip This document introduces the technique of just-in-time specialization for improving the performance of dynamic languages. It discusses two key aspects:
1. Just-in-time specialization allows values to be "unlifted" from run-time to compile-time during specialization, using actual run-time values to guide specialization. This provides stronger feedback than traditional specialization approaches.
2. Representation theory generalizes the traditional compile-time/run-time dichotomy to provide a theory of data specialization. It allows specializing code based on representations of values within their domains, rather than just types.
Together these techniques allow specialization to be more finely interleaved with execution and less
Ad
Recently uploaded (20)
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru Introduction to LPIC-1 Exam - overview, exam details, price and job opportunities
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots.
📕 Here's what you can expect:
- Modeling: Build end-to-end processes using BPMN.
- Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes.
- Operating: Control process instances with rewind, replay, pause, and stop functions.
- Monitoring: Use dashboards and embedded analytics for real-time insights into process instances.
This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes.
👨🏫 Speaker:
Andrei Vintila, Principal Product Manager @UiPath
This session streamed live on April 29, 2025, 16:00 CET.
Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55 We’re bringing the TDX energy to our community with 2 power-packed sessions:
🛠️ Workshop: MuleSoft for Agentforce
Explore the new version of our hands-on workshop featuring the latest Topic Center and API Catalog updates.
📄 Talk: Power Up Document Processing
Dive into smart automation with MuleSoft IDP, NLP, and Einstein AI for intelligent document workflows.
AI and Data Privacy in 2025: Global Trends
AI and Data Privacy in 2025: Global TrendsInData Labs In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo From predictive maintenance to robotic automation, AI is driving the future of manufacturing. But without high-quality annotated data, even the smartest models fall short.
Discover how data annotation services are powering accuracy, safety, and efficiency in AI-driven manufacturing systems.
Precision in data labeling = Precision on the production floor.
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In France
Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3 The latest updates on Manifest's pre-seed stage progress.
Build Your Own Copilot & Agents For Devs
Build Your Own Copilot & Agents For DevsBrian McKeiver May 2nd, 2025 talk at StirTrek 2025 Conference.
Cybersecurity Identity and Access Solutions using Azure AD
Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ Cybersecurity Identity and Access Solutions using Azure AD
How analogue intelligence complements AI
How analogue intelligence complements AIPaul Rowe
Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results.
News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?
Generative Artificial Intelligence (GenAI) in Business
Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma My talk for the Indian School of Business (ISB) Emerging Leaders Program Cohort 9. In this talk, I discussed key issues around adoption of GenAI in business - benefits, opportunities and limitations. I also discussed how my research on Theory of Cognitive Chasms helps address some of these issues
Drupalcamp Finland – Measuring Front-end Energy Consumption
Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove How to measure web front-end energy consumption using Firefox Profiler. Presented in DrupalCamp Finland on April 25th, 2025.
Into The Box Conference Keynote Day 1 (ITB2025)
Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp This is the keynote of the Into the Box conference, highlighting the release of the BoxLang JVM language, its key enhancements, and its vision for the future.
Rusty Waters: Elevating Lakehouses Beyond Spark
Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1 Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark?
At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍
Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen
HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/
HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar.
Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten.
In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich
- Zugriff auf die Konsole
- Auffinden und Interpretieren von Protokolldateien
- Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS)
- Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien
- Nutzung der Client Clocking-Funktion
Quantum Computing Quick Research Guide by Arthur Morgan
Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan This is a Quick Research Guide (QRG).
QRGs include the following:
- A brief, high-level overview of the QRG topic.
- A milestone timeline for the QRG topic.
- Links to various free online resource materials to provide a deeper dive into the QRG topic.
- Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic.
QRGs planned for the series:
- Artificial Intelligence QRG
- Quantum Computing QRG
- Big Data Analytics QRG
- Spacecraft Guidance, Navigation & Control QRG (coming 2026)
- UK Home Computing & The Birth of ARM QRG (coming 2027)
Any questions or comments?
- Please contact Arthur Morgan at [email protected].
100% human made.
Heap, Types of Heap, Insertion and Deletion
Heap, Types of Heap, Insertion and DeletionJaydeep Kale This pdf will explain what is heap, its type, insertion and deletion in heap and Heap sort