hbaseconasia2019 HBase at Tencent

0 likes264 views

The document discusses Tencent's use and optimization of HBase, detailing its implementation since 2013 across multiple clusters and applications. Key topics include the architecture of HBase services, various optimizations for data management, and practices for data migration and performance enhancement. Additionally, it highlights contributions to the HBase community and recent features added to improve efficiency.

Internet

HBase At Tencent
Andrew Cheng | 程广旭
Tencent | HBase Committer

Content
01. HBase Service In Tencent
02. Applications
03. Practices & Optimization

HBase Story in Tencent
l Began using since 2013
l Used version
l 0.94.17 -> 0.98.6 -> 1.2.5 -> 2.2.0 (ing)
l Largest cluster more than 500 nodes
90+
Clusters
4000+
Nodes
10PB+
Data
3Tri+
RPD

Overview
HBase Users come from 6 groups , more than 100+ different applications

Architecture
Tencent HBase Zookeeper
OpenTSDB
S2Graph
Spark Tookit
HBase Api
TDBank
Lhotse
RestServer
ThriftServer
Kylin
Phoenix Tenpay
Doss
monitoring
TNM2
Deploy CenterWepay Game
Advertiseme
nt
…

Tencent Ads – Real-Time Logjoin System
Mixer Exposure
TDBank
Tencent HBase
Model learning Freshness Budget control Report
Association Table
Flow Table
Click …
LogJoin LogJoin LogJoin LogJoin
Data Source
Transport
Logical
Storage
Consumer

Tenpay - Transaction record
Data Source MySQL
Binlog Paser DBSync
Cache Hippo
Storage Tencent HBase
Thrift Server
Application
C++
Read
Read
Write
Application
JAVA
Read
Write TDSort

Practices–Data migration
add_peer
disable_peer
Set REPLICATION_SCOPE => '1'
snapshot clone_snapshot
Set REPLICATION_SCOPE => '0'
Check Dataenable_peer
Client switch to new cluster
Cluster A Cluster B
ExportSnapshot
delete_snapshot
Business-insensitive data migration

Practices–Table
l Create table per day
l Large amount of data
l TTL is short
l Benefit
l Reduce the amount of data in compaction
l Easy to delete expired data

Optimization - Bandwidth
② RS2 and RS3 Wal data
① Input Data
③ RS2 and RS3 Flush data
⑤ RS2 and RS3 Large compact
④ RS2 and RS3 Small compact
RS1 RS2 RS3
Input Data
Wal
Flush
①
Small compact
Large compact
②
③
④
⑤
Input Data Input Data

Optimization - Bandwidth
l Enable compressing of CellBlocks
l Wal compressor
l Increase the size of memstore
l Reduce the number of threads about compaction
l Turn off major compaction
l create tables by day

Optimization - Online filtering of dirty data
l A large amount of data which have the same Rowkey
l How to find filter rowkeys?
l ResponseTooSlow
l How to set filter rowkeys?
l hbase.hregion.filter.rowkeys
l How to refresh filter rowkeys?
l update_config
Input Data
Filter
Enable
Write
Filter
Yes
Yes
No
No

Optimization - Prefix Bloom Filter（HBASE-20636）
l ROWPREFIX_FIXED_LENGTH
l ROWPREFIX_DELIMITER
uin ts action
Bloom Filter
Prefix
Create Table:
File info:

$Optimization - Prefix Bloom Filter（HBASE-20636） Scan Not Filter StoreFile Same prefix? {StartKey,EndKey} Computer hash value Hit BloomFilter? Prefix length >= prefix_length Yes Yes No Filter StoreFile No No Get prefix key by prefix_length Yes Read Rowkey Get prefix key by prefix_length Computer hash value Set BloomFilter Last line? Input Data Write BloomFilter information to StoreFile metadata Yes No Write$

Optimization - RestServer
RestServer A
Cluster A Cluster B Cluster C
RestServer CRestServer B RestServer D
User
Nginx

Optimization - RestServer
RestServer A
Cluster A Cluster B Cluster C
RestServer CRestServer B
User
Nginx
Mysql

Optimization - RestServer
l Only maintain one configuration
l use effectively resources
l User-friendly access

HBase Community
l 1 Committer, 2 Contributor
l Total commits: 80+
l Feature
l HBASE-20636 Introduce two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED
l HBASE-19799 Add web UI to rsgroup
l HBASE-20243 [Shell] Add shell command to create a new table by cloning the existent table
l HBASE-19483 Add proper privilege check for rsgroup commands
l ………

The document discusses recent advancements in HBase at Pinterest, detailing the use of frameworks such as Apache Omid for transaction management, Sparrow for scalability, Argus for real-time notifications, and Ixia for near-real-time indexing. It highlights the architecture and improvements made in each framework, emphasizing their roles in enhancing database performance and capabilities. The document outlines the pros and cons of these systems and their implementation status and future work.

HBaseConAsia2018 Keynote1: Apache HBase Project StatusMichael Stack

- Apache HBase 2.0.0 is a major new release that was over 4 years in development and focused on compatibility, scale, and performance improvements. - Key changes include a new master region assignment system, off-heap read/write paths, and in-memory compaction. - The goals were to support larger clusters with better resource utilization while fixing issues with the previous master region assignment system.

HBaseConAsia2018 Track1-1: Use CCSMap to improve HBase YGC timeMichael Stack

CCSMap is a new data structure introduced by Alibaba to improve the performance of HBase. It aims to reduce the overhead of the default Java ConcurrentSkipListMap (CSLM) data structure and improve young garbage collection times. CCSMap chunks data into fixed size blocks for better memory management and uses direct pointers between nodes for faster access. It also provides various configuration options. Alibaba has achieved significant performance gains using CCSMap in HBase, including reduced young GC times, and it continues working to integrate CCSMap further and add new features.

HBaseConAsia2018 Track3-3: HBase at China Life InsuranceMichael Stack

This document summarizes an HBase practice presentation at China Life Insurance Co., Ltd. It discusses scenarios for HBase integration, processing, querying, and exporting data. It also covers optimizations to the HBase cluster configuration and for writing and reading. Problems addressed include table copy failures and compactions that never end. Future work may involve using Phoenix for real-time querying and integrating real-time data sources like Kafka.

HBaseConAsia2018: Track2-5: JanusGraph-Distributed graph database with HBaseMichael Stack

This document provides an introduction to JanusGraph, an open source distributed graph database that can be used with Apache HBase for storage. It begins with background on graph databases and their structures, such as vertices, edges, properties, and different storage models. It then discusses JanusGraph's architecture, support for the TinkerPop graph computing framework, and schema and data modeling capabilities. Details are given on partitioning graphs across servers and using different indexing approaches. The document concludes by explaining why HBase is a good storage backend for JanusGraph and providing examples of how the data model would be structured within HBase.

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack

This document summarizes a presentation on scaling a 30 TB data lake using Apache HBase and Scala. It introduces Apache HBase and Spark as technologies for building fast data platforms. It then describes a case study where they were used to architect a retail analytics platform capable of processing 4.6 billion events weekly from 30 TB of historical data. Key aspects included using HBase for data deduplication and as a master data management system, and connecting Spark to HBase using a Scala DSL for efficient querying and updates at scale. Performance was improved 5x by reengineering the data pipeline to be highly concurrent and asynchronous.

Hadoop Networking at Datasifthuguk

This document summarizes Gareth Llewellyn's experience redesigning the network architecture at DataSift to improve performance and scalability. The initial Cisco-based design suffered from issues like buffering, head of line blocking, and oversubscription of uplinks. Gareth considered moving to an Arista leaf-spine architecture with Arista 7050 core switches and 7048 top-of-rack switches, which would provide better redundancy, scalability, and throughput while reducing complexity compared to the mesh design. Questions are welcomed about the new design.

Роман Новиков "Best Practices for MySQL Performance & Troubleshooting with th...Fwdays

The document outlines best practices for troubleshooting MySQL performance issues, emphasizing the importance of identifying the correct factors affecting database speed. It introduces various monitoring tools and metrics, such as utilization and error rates, to help diagnose and optimize performance problems. The presentation suggests that performance issues may originate from sources outside the database itself, highlighting the need for a holistic approach to performance management.

Amazon RedShift - Ianni Vamvadelishuguk

The document provides an overview of Amazon Redshift, a fully managed, petabyte-scale data warehouse service that integrates with other AWS services. It details features such as optimized hardware, security measures, data loading best practices, and workload management. Additionally, it discusses Redshift's pricing and use cases, emphasizing its efficiency for scalable data analysis and reporting.

Amazon Elastic Map Reduce - Ian Meyershuguk

The document discusses Amazon Web Services (AWS) and its cloud computing offerings, focusing on the advantages of utility computing, scalability, and cost management. It details various services including Elastic MapReduce, Simple Storage Service, and databases like RDS and DynamoDB, along with their performance metrics and use cases. Additionally, the document highlights the benefits of using AWS for IT infrastructure, emphasizing agility, reduced capital expenditure, and global reach.

HBaseCon 2015: HBase Operations in a FlurryHBaseCon

The document discusses the use and migration strategies of HBase at Flurry Analytics, including challenges faced during data migration between different data centers and HBase versions. It outlines various attempts for data migration, detailing their pros and cons, as well as upgrades to better accommodate new HBase and Hadoop versions. The authors also share operational issues encountered and solutions implemented to ensure system efficiency during the transition.

HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and CloudMichael Stack

New Journey of HBase in Alibaba and Cloud discusses Alibaba's use of HBase over 8 years and improvements made. Key points discussed include: - Alibaba began using HBase in 2010 and has since contributed to the open source community while developing internal improvements. - Challenges addressed include JVM garbage collection pauses, separating computing and storage, and adding cold/hot data tiering. A diagnostic system was also created. - Alibaba uses HBase across many core scenarios and has integrated it with other databases in a multi-model approach to support different workloads. - Benefits of running HBase on cloud include flexibility, cost savings, and making it

Володимир Цап "Constraint driven infrastructure - scale or tune?"Fwdays

Volodymyr Tsap discusses how to save money on infrastructure through constraint driven design. He provides examples of hardware configurations on AWS, bare metal servers, and PaaS platforms to demonstrate how costs can be optimized. Tsap also outlines ways to reduce software costs through choices in operating system, virtualization, databases, and orchestration. Infrastructure support costs depend on the complexity of the environment, with basic setups costing $500-800 per month while more advanced architectures are $4,000-6,000 per month. The overall message is that money saved through optimization can be invested in people.

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...Fwdays

The document discusses the architecture and approaches of a fast data platform for real-time analytics, highlighting key technologies such as Kafka and Cassandra. It contrasts big data methods with fast data strategies, emphasizing the importance of batch and stream processing in systems like Lambda and Kappa architectures. Key lessons learned include the need for careful database design, understanding Cassandra's internals, and the challenges of managing eventual consistency.

Rolling Out Apache HBase for Mobile Offerings at Visa HBaseCon

The presentation discusses the implementation of Apache HBase for Visa's mobile offerings, highlighting the need for scalable and real-time transaction history services. The speakers share their experiences and lessons learned throughout the adoption process, addressing key considerations such as performance optimization, transactional support, and data management. Ultimately, they emphasize the importance of continuous learning and adapting strategies based on evolving requirements and challenges in production environments.

Argus Production Monitoring at SalesforceHBaseCon

The document introduces Argus, a monitoring system at Salesforce that leverages time series data, service protection, alerting, and customizable dashboards. It emphasizes the need for a monitoring solution that adapts to technology changes, ensuring performance, trust, and programmatic access. The architecture includes a series of components such as metrics storage, alert evaluation, and a schema service to enhance monitoring capabilities and system efficiency.

HBaseCon 2015: State of HBase Docs and How to ContributeHBaseCon

The document discusses the state of Apache HBase documentation, outlining improvements made in the past year under Misty Stanley-Jones' leadership at Cloudera. Key highlights include the introduction of a quickstart guide, enhancements to blockcache and replication documentation, and the establishment of developer contribution guidelines. The overall documentation efforts have significantly increased, closing 122 JIRA issues within a year, emphasizing the importance of good documentation for user and developer experiences.

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...Michael Stack

This document discusses the use of HBase in a vehicle monitoring system. It describes challenges including handling huge amounts of vehicle data from 100k vehicles generating 2TB of data daily. It outlines decisions around using Java, Kafka, HBase, and microservices. The system architecture is shown storing vehicle data in HBase with data backup. Challenges with HBase like query speed are discussed. Prospects include rewriting components in Go, splitting to microservices, and data analysis.

HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataMichael Stack

This document discusses Apache Kylin, an OLAP engine for big data. It provides 3 key points: 1. Apache Kylin is designed to provide fast, interactive queries on large datasets stored in Hadoop. It uses pre-calculated cube structures stored in HBase to enable sub-second query performance on trillion row datasets. 2. HBase was selected as the storage engine because it is natively integrated with Hadoop, supports high throughput and low-latency queries, and can store very large datasets. Kylin leverages HBase's capabilities for cube storage, metadata storage, online calculation pushing, and caching. 3. Apache Kylin is used by over 1000 global companies across

HBaseCon 2015: Optimizing HBase for the Cloud in Microsoft Azure HDInsightHBaseCon

The document discusses optimizing HBase for cloud deployment on Microsoft Azure HDInsight, highlighting cost savings by using cloud storage which can reduce cluster costs by up to 10x for certain workloads. It includes various use cases such as key-value stores and sensor data stores, demonstrating significant performance improvements and cost reductions. Additionally, it provides insights into performance tuning and the announcement of HBase on Azure Data Lake for big data workloads.

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程HBaseCon

Introduction to streaming and messaging flume,kafka,SQS,kinesis Omid Vahdaty

The document provides an overview of streaming and messaging technologies, comparing batch processing with streaming data processing, and discussing various tools such as Flume, Kafka, AWS SQS, and AWS Kinesis. It highlights the pros and cons of each tool, their use cases, and the challenges associated with streaming data, including scalability, fault tolerance, and consistency. Additionally, it explains how these technologies can be integrated to enhance application performance and data handling capabilities.

HBaseCon2017 Apache HBase at DidiHBaseCon

Didi is a leading mobile transportation platform that aims to redefine mobility by utilizing an HBase architecture for its big data solutions. The Didi HBase platform supports extensive applications, including online reading/writing and batch job storage, while also addressing challenges related to multi-tenancy and resource management. Future goals include further integration with big data frameworks and improving the stability and capabilities of the HBase environment.

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020Redis Labs

The document discusses rate limiting and metering using Redis. It begins by introducing rate limiting and metering and why Redis is well-suited for these tasks. It then covers different Redis data structures that can be used, such as lists, hashes, sorted sets and strings. Common Redis commands for counting, setting keys and checking time to live are also presented. Different rate limiting design patterns and anti-patterns are described, including fixed window, sliding window and token bucket approaches. Finally, resources for further information are provided.

Ameya Kanitkar: Using Hadoop and HBase to Personalize Web, Mobile and Email E...WebExpo

This document discusses using Hadoop and HBase to build content relevance and personalization systems for big data applications. It provides an overview of Hadoop and HBase, and how they can be used together. As a case study, it describes how Groupon uses Hadoop and HBase for their deal relevance and personalization systems, including storing user data in HBase and running recommendation algorithms using MapReduce.

Big dataarchitecturesandecosystem+nosqlKhanderao Kand

This document provides an overview of big data architecture, the Hadoop ecosystem, and NoSQL databases. It discusses common big data use cases, characteristics, and tools. It describes the typical 3-tier traditional architecture compared to the big data architecture using Hadoop. Key components of Hadoop like HDFS, MapReduce, Hive, Pig, Avro/Thrift, HBase are explained. The document also discusses stream processing tools like Storm, Spark and real-time query with Impala. It notes how NoSQL databases can integrate with Hadoop/MapReduce for both batch and real-time processing.

Stratebi Big DataStratebi

This document discusses big data technologies for enterprise analytics. It begins by defining big data and classifying big data technologies into three groups: Apache Hadoop, NoSQL databases, and extended RDBMS. It then provides examples of using different technologies for enterprise data warehouse extensions, website clickstream analysis, and real-time analytics. The document also discusses Hadoop distributions and Pentaho's support for big data and provides some big data success stories.

Hadoop and Pig at Twitter__HadoopSummit2010Yahoo Developer Network

1. Hadoop is used extensively at Twitter to handle large volumes of data from logs and other sources totaling 7TB per day. Tools like Scribe and Crane are used to input data and Elephant Bird and HBase for storage. 2. Pig is used for data analysis on these large datasets to perform tasks like counting, correlating, and researching trends in users and tweets. 3. The results of these analyses are used to power various internal and external Twitter products and keep the business agile through ad-hoc analyses.

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Michael Sun presented on CBS Interactive's use of Hadoop for web analytics processing. Some key points: - CBS Interactive processes over 1 billion web logs daily from hundreds of websites on a Hadoop cluster with over 1PB of storage. - They developed an ETL framework called Lumberjack in Python for extracting, transforming, and loading data from web logs into Hadoop and databases. - Lumberjack uses streaming, filters, and schemas to parse, clean, lookup dimensions, and sessionize web logs before loading into a data warehouse for reporting and analytics. - Migrating to Hadoop provided significant benefits including reduced processing time, fault tolerance, scalability, and cost effectiveness compared to their

Gaming SEC Filings Using Machine Learning to Detect Vectors and Sentiment in ...Safe Software

The document outlines a presentation by Steven Cyphers on utilizing machine learning for analyzing SEC filings in the gaming sector to detect sentiment and trends. It discusses data processing techniques using NLP and Python, including automating data extraction and employing sentiment analysis to improve reporting efficiency. Key applications include semantic search, identifying topics, and graphing sentiment metrics over time.

More Related Content

What's hot (16)

Amazon RedShift - Ianni Vamvadelishuguk

Amazon Elastic Map Reduce - Ian Meyershuguk

HBaseCon 2015: HBase Operations in a FlurryHBaseCon

HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and CloudMichael Stack

Володимир Цап "Constraint driven infrastructure - scale or tune?"Fwdays

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...Fwdays

Rolling Out Apache HBase for Mobile Offerings at Visa HBaseCon

Argus Production Monitoring at SalesforceHBaseCon

HBaseCon 2015: State of HBase Docs and How to ContributeHBaseCon

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...Michael Stack

HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataMichael Stack

HBaseCon 2015: Optimizing HBase for the Cloud in Microsoft Azure HDInsightHBaseCon

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程HBaseCon

Introduction to streaming and messaging flume,kafka,SQS,kinesis Omid Vahdaty

HBaseCon2017 Apache HBase at DidiHBaseCon

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020Redis Labs

Amazon RedShift - Ianni Vamvadelishuguk

Amazon Elastic Map Reduce - Ian Meyershuguk

HBaseCon 2015: HBase Operations in a FlurryHBaseCon

HBaseConAsia2018 Keynote 2: Recent Development of HBase in Alibaba and CloudMichael Stack

Володимир Цап "Constraint driven infrastructure - scale or tune?"Fwdays

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...Fwdays

Rolling Out Apache HBase for Mobile Offerings at Visa HBaseCon

Argus Production Monitoring at SalesforceHBaseCon

HBaseCon 2015: State of HBase Docs and How to ContributeHBaseCon

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...Michael Stack

HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big dataMichael Stack

HBaseCon 2015: Optimizing HBase for the Cloud in Microsoft Azure HDInsightHBaseCon

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程HBaseCon

Introduction to streaming and messaging flume,kafka,SQS,kinesis Omid Vahdaty

HBaseCon2017 Apache HBase at DidiHBaseCon

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020Redis Labs

Similar to hbaseconasia2019 HBase at Tencent (20)

Ameya Kanitkar: Using Hadoop and HBase to Personalize Web, Mobile and Email E...WebExpo

Big dataarchitecturesandecosystem+nosqlKhanderao Kand

Stratebi Big DataStratebi

Hadoop and Pig at Twitter__HadoopSummit2010Yahoo Developer Network

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Gaming SEC Filings Using Machine Learning to Detect Vectors and Sentiment in ...Safe Software

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...Impetus Technologies

The document presents an overview of Change Data Capture (CDC) architecture, detailing its definition, various implementation methods, and use cases in enterprise environments. It highlights the transition from legacy systems to big data integrations, outlines challenges, and describes the architectural considerations for efficient CDC. Additionally, it emphasizes the role of tools like StreamAnalytix in simplifying CDC workflows for real-time analytics and data processing.

Big Data Taiwan 2014 Track2-2: Informatica Big Data SolutionEtu Solution

The document discusses the challenges of data fragmentation in big data environments, emphasizing the importance of data integration and quality management for business success. It outlines various technologies, including Informatica's solutions for Hadoop and big data processing, as well as methodologies for real-time data handling and transformation. The text also highlights the significance of effective data profiling, cleansing, and analysis to support informed decision-making within enterprises.

Hive @ Hadoop day seattle_2010nzhang

Hive is used at Facebook for data warehousing and analytics tasks on a large Hadoop cluster. It allows SQL-like queries on structured data stored in HDFS files. Key features include schema definitions, data summarization and filtering, extensibility through custom scripts and functions. Hive provides scalability for Facebook's rapidly growing data needs through its ability to distribute queries across thousands of nodes.

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

The document discusses Vespa, an open-source platform for big data serving, focusing on low-latency computations, real-time actions, and automated data-driven decisions. It contrasts Vespa's capabilities with other technologies like Elasticsearch and traditional databases, emphasizing its efficiency in handling large data sets and machine learning integration. A case study on Zedge illustrates how Vespa simplifies and accelerates search and recommendation systems for mobile apps.

Admin Tech Clash: Discussing Best (and Worst) Administration Practices from ...Christoph Adler

The document presents a session on best and worst administration practices in collaboration infrastructure, focusing on how to utilize data effectively for optimizing performance and resource management. Key topics include user activity analysis, server-side logging, and client-side performance metrics, highlighting tools and methods available for data extraction and visualization. The speakers share insights on handling various logs, maximizing the utility of databases, and addressing performance issues, avoiding product promotion during the discussion.

Admin Tech Clash: Discussing Best (and Worst) Administration Practices from ...Christoph Adler

This document summarizes a presentation on analyzing administrative data from Domino servers and clients. It discusses sources of usage data like activity trends, server-side logs in log.nsf, and client-side logging. Examples are given of visualizing this data to understand usage patterns, database demand, and network traffic. Third-party tools are also mentioned for enhanced analysis across multiple databases and servers.

Enterprise Data LakesFarid Gurbanov

The document discusses the evolution and advantages of enterprise data lakes over traditional RDBMS, emphasizing horizontal scalability and cost efficiency. It presents several use cases, including log analytics, preventive accident monitoring for fleets, and recommendation engines, outlining their architectures and requirements. Additionally, it highlights technologies such as Apache NiFi, Kafka, and Spark for integrated data science opportunities.

History of Apache Pinot Kishore Gopalakrishna

The document details the evolution and technical advancements of Apache Pinot, a distributed analytics system, highlighting its journey from 2014 to the present. It discusses the challenges faced with existing analytics stacks and how Pinot addressed them with innovations like the star-tree index, achieving significant improvements in performance and efficiency. Future roadmap items are also mentioned, aiming to enhance features and usability for analytics on mutable data.

Web performance optimizationKaliop-slide

The document focuses on web performance optimization, highlighting key factors such as front-end and back-end performance, real-life concerns, and various metrics to address. It emphasizes the importance of DevOps culture in deploying and fine-tuning services, as well as monitoring actual user experiences with tools like Real User Monitoring (RUM). The discussion also covers specific performance issues, best practices, and considerations for optimizing website performance across different environments.

PayPal merchant ecosystem using Apache Spark, Hive, Druid, and HBase DataWorks Summit

The document outlines PayPal's merchant analytics ecosystem, detailing the use of technologies like Spark, HBase, Druid, and Elasticsearch to process and serve large datasets. It highlights the team's learnings on optimizing performance, managing data pipelines, and implementing a data quality tool to enhance operational efficiency. Additionally, it discusses the integration of Druid for enhanced business intelligence capabilities with Tableau and the implementation of behavioral driven development for better testing and development practices.

OSMC 2013 | openTSDB - metrics for a distributed worldNETWAYS

OpenTSDB is an open source time series database that allows for storing and querying of large amounts of metrics data. It is written in Java and uses HBase for distributed storage. Data is stored by metric ID and timestamp as the row key in HBase. Tags are stored at the end of the row key. The speaker discusses their experiences using OpenTSDB at Gutefrage.net to store hundreds of millions of data points with thousands of time series. They discuss challenges with the UI and time resolution and look forward to improvements in OpenTSDB 2.

Membase Meetup 2010Membase

This document provides an overview and technical discussion of Membase. It begins with introducing Membase and how it allows both applications and databases to scale horizontally. The rest of the document discusses Membase architecture, deployment options, use cases, and a demo. It also briefly explores developing with Membase and the future direction of NodeCode, which will allow extending Membase through custom modules.

Hadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTDataCloudera, Inc.

The document discusses NTT Data Corporation's involvement in Hadoop and its integration within enterprise systems in Japan. It highlights NTT Data's history, market position, and contributions to Hadoop, including partnerships and enhancements aimed at improving big data processing capabilities. The text also outlines the evolution and life-cycle of Hadoop clusters, their applications in enterprise environments, and specific enhancements made for PostgreSQL integration.

hbaseconasia2017: Apache HBase at NeteaseHBaseCon

This document summarizes Netease's use of Apache HBase for big data. It discusses Netease operating 7 HBase clusters with 200+ RegionServers and hundreds of terabytes of data across more than 40 applications. It outlines key practices for Linux system configuration, HBase schema design, garbage collection, and request queueing at the table level. Ongoing work includes region server grouping, inverted indexes, and improving high availability of HBase.

Ameya Kanitkar: Using Hadoop and HBase to Personalize Web, Mobile and Email E...WebExpo

Big dataarchitecturesandecosystem+nosqlKhanderao Kand

Stratebi Big DataStratebi

Hadoop and Pig at Twitter__HadoopSummit2010Yahoo Developer Network

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Gaming SEC Filings Using Machine Learning to Detect Vectors and Sentiment in ...Safe Software

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...Impetus Technologies

Big Data Taiwan 2014 Track2-2: Informatica Big Data SolutionEtu Solution

Hive @ Hadoop day seattle_2010nzhang

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

Admin Tech Clash: Discussing Best (and Worst) Administration Practices from ...Christoph Adler

Enterprise Data LakesFarid Gurbanov

History of Apache Pinot Kishore Gopalakrishna

Web performance optimizationKaliop-slide

PayPal merchant ecosystem using Apache Spark, Hive, Druid, and HBase DataWorks Summit

OSMC 2013 | openTSDB - metrics for a distributed worldNETWAYS

Membase Meetup 2010Membase

Hadoop World 2011: Hadoop’s Life in Enterprise Systems - Y Masatani, NTTDataCloudera, Inc.

hbaseconasia2017: Apache HBase at NeteaseHBaseCon

More from Michael Stack (20)

hbaseconasia2019 HBase Table Monitoring and Troubleshooting System on CloudMichael Stack

hbaseconasia2019 Phoenix Practice in China Life Insurance Co., LtdMichael Stack

The document outlines the architecture and optimizations for the Phoenix data processing system at China Life Insurance Company, detailing its design, implementation scenarios, and future challenges. It describes key components such as data ingestion, real-time queries, and various optimization strategies to enhance performance. Additionally, it addresses problems encountered in the system and proposes future work focused on enhancing read/write isolation and optimizing compaction control.

hbaseconasia2019 HBase at DidiMichael Stack

The document discusses the usage and applications of HBase and GeoMesa in distributed computing, highlighting improvements in ZooKeeper and HBase for enhanced scalability and performance. It outlines various features and pain points, including support for multiple versions and querying capabilities. Additionally, it provides insights into specific improvements and quota management within the HBase framework.

hbaseconasia2019 The Practice in trillion-level Video Storage and billion-lev...Michael Stack

hbaseconasia2019 Spatio temporal Data Management based on Ali-HBase Ganos and...Michael Stack

hbaseconasia2019 Bridging the Gap between Big Data System Software Stack and ...Michael Stack

The document discusses the development of a distributed storage service for semiconductor wafer fabrication, focusing on overcoming the challenge of increasing data volume in manufacturing. It outlines a design that integrates various heterogeneous storage systems, ensures compatibility with existing Hadoop interfaces, and improves performance through load balancing and transparency. Ultimately, the service aims to enhance data management efficiency, leading to a 1% yield rate improvement in semiconductor production.

hbaseconasia2019 Pharos as a Pluggable Secondary Index ComponentMichael Stack

The document introduces Pharos, a pluggable secondary index component developed by Lei Wang of China Everbright Bank, aimed at improving data retrieval performance while maintaining simplicity in architecture. It discusses the requirements for a non-invasive and high-performance indexing solution and outlines its features, including multi-index support, and the differences between global and partition indexes. Future development plans emphasize transaction consistency and integration with SQL engines.

hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at AlibabaMichael Stack

The document discusses advancements and practices for Apache HBase and Phoenix on Alibaba Cloud, covering topics such as performance improvements, ecosystem tools, data migration, and client architecture. It emphasizes benefits like high throughput, low latency query execution, and strategies for indexing and data management. Additionally, it outlines best practices, pitfalls to avoid, and future work directions in enhancing search index capabilities and query performance.

hbaseconasia2019 OpenTSDB at XiaomiMichael Stack

The document provides an overview of OpenTSDB and its deployment on Kubernetes, addressing its scalability, configuration details, and key performance metrics. It outlines the features of OpenTSDB, including its compatibility with HBase, stateless architecture, and configuration management processes when running in a Kubernetes environment. Additionally, it includes specific use cases for monitoring, demonstrating the system's capabilities and performance benchmarks.

hbaseconasia2019 BigData NoSQL System: ApsaraDB, HBase and SparkMichael Stack

The document discusses the ApsaraDB HBase X-Pack, detailing its architecture, features, and applications in big data processing, including real-time analytics, personalized recommendations, and risk control mechanisms. It highlights the integration of various components such as Spark, Kafka, and offline warehouses for efficient data management and processing. Additionally, it addresses challenges in online and offline analysis and emphasizes performance improvements and cost-saving measures.

hbaseconasia2019 Test-suite for Automating Data-consistency checks on HBaseMichael Stack

The document discusses a test-suite developed by Flipkart for automating data-consistency checks on their HBase cluster, essential for supporting their large e-commerce operations. It highlights the need for data-correctness guarantees and details various approaches to testing, including algorithm and code testing as well as testing on a running cluster. The suite aims to facilitate faster upgrade cycles and addresses potential failure scenarios and issues uncovered during testing.

hbaseconasia2019 Distributed Bitmap Index SolutionMichael Stack

The document discusses the limitations of HBase in handling ad-hoc queries for large datasets with multiple tags and introduces a distributed bitmap index solution to address these challenges. It outlines the architecture for a new SQL engine, Lemon SQL, that enables efficient high-performance querying and better memory management for high-dimensional data. The proposed framework includes coprocessor-based bitmap index building and querying, aimed at enhancing data access and scalability for applications requiring complex search functionalities.

hbaseconasia2019 HBase Bucket Cache on Persistent MemoryMichael Stack

The document discusses the performance of HBase bucket cache on persistent memory (DCPMM) compared to traditional DRAM, highlighting that DCPMM offers similar or improved performance at a potentially lower cost with greater capacity. It details various configurations and performance benchmarks, underscoring the importance of system configuration on performance outcomes. Additionally, it emphasizes the need for caution regarding security and performance results based on testing conducted as of June 2019.

hbaseconasia2019 The Procedure v2 Implementation of WAL Splitting and ACLMichael Stack

The document discusses the implementation of Procedure V2 for WAL splitting and ACL in HBase, detailing its structure, execution, and rollback mechanisms. It emphasizes the advantages of using Procedure V2 over traditional Zookeeper coordination, including improved performance and reduced dependency on Zookeeper. Additionally, it outlines the processes for managing ACLs, coordinating procedures across multiple servers, and ensuring consistency in permission caches.

hbaseconasia2019 BDS: A data synchronization platform for HBaseMichael Stack

The document discusses the Apache HBase BDS system's data synchronization and migration platform, addressing key requirements such as cross-version migration without downtime and support for various data replication methods. It outlines significant challenges, including table structure transformation and ensuring data consistency during migrations, and proposes technical solutions like real-time data replication and a decoupled architecture. Additionally, it emphasizes high availability and efficient data migration processes while considering the impact on business operations.

hbaseconasia2019 Further GC optimization for HBase 2.x: Reading HFileBlock in...Michael Stack

The document discusses optimizations for garbage collection in HBase 2.x by enabling off-heap reading of HDFS blocks directly into pooled bytebuffers, which aims to reduce young generation garbage collection latency. It details the implementation process, core ideas involving reference counting, and performance evaluations under different cache hit ratios, highlighting significant improvements in throughput and reduced garbage collection pressure. Additionally, it outlines best practices and configuration settings for optimal use of the pooled bytebuffer allocator.

hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...Michael Stack

The document discusses HBCK2, a tool for fixing issues in HBase 2. Some key points: 1. HBCK2 is simpler than HBCK1, with fewer fix commands and no diagnosis commands. It requires a deeper understanding of HBase internals. 2. HBCK2 commands are master-oriented and fix issues one at a time. Common issues include regions not online, stuck procedures, and tables in the wrong state. 3. Recipes are provided to fix specific issues like missing meta regions or regions in transition using HBCK2 commands like assigns and bypass. 4. HBCK2 is still a work in progress but contributions are welcome

HBaseConAsia2019 KeynoteMichael Stack

The Apache HBase community event held on July 20, 2019, in Beijing showcased the current status and developments of HBase, highlighting its open-source nature and the importance of community contributions. Presentations covered its technical evolution, cloud advantages, and future enhancements, including improvements in stability, cost efficiency, and global deployment capabilities. The event emphasized diversity and collaboration in the development of HBase to ensure its relevance and innovation moving forward.

HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latenciesMichael Stack

This document discusses how Bloomberg uses HBase to serve billions of queries with millisecond latency. It covers HBase principles like being an ordered key-value store and providing ACID transactions. It also discusses modeling data for HBase, including dealing with data and query skew. Implementation details covered include caching, block size tuning, column families, and compaction. The overall goal is to optimize HBase for Bloomberg's low-latency data storage and retrieval needs.

HBaseConAsia2018 Track1-3: HBase at XiaomiMichael Stack

This document summarizes Xiaomi's implementation and use of HBase for data storage. It discusses Xiaomi's HBase clusters across multiple public cloud providers and data centers. It also describes Xiaomi's approaches to multi-tenancy, quota and throttling, synchronous replication between clusters, and high availability in the case of node or cluster failures. Synchronous replication provides stronger consistency guarantees but with some performance overhead compared to asynchronous replication.