Couchbase Sydney meetup #1 Couchbase Architecture and Scalability
0 likes370 views
This document provides an overview of a Couchbase meetup presentation on Couchbase architecture and scalability. The presentation covers the history of Couchbase, its key capabilities as a data management solution, architecture including services for data, indexing and querying, common use cases, what makes Couchbase unique, always-on availability features, administration and management, performance and scalability including auto sharding and cross data center replication, and examples of using its query language N1QL.
![[db tech showcase Tokyo 2017] C24:Taking off to the clouds. How to use DMS in...](https://cdn.slidesharecdn.com/ss_thumbnails/howtousedmsinawsmigrations-170911061230-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
![Java 8 Puzzlers [as presented at OSCON 2016]](https://cdn.slidesharecdn.com/ss_thumbnails/java8puzzlersoscon2016-160522043417-thumbnail.jpg?width=560&fit=bounds)
Ad
Similar to Couchbase Sydney meetup #1 Couchbase Architecture and Scalability (20)
Ad
Recently uploaded (20)
Couchbase Sydney meetup #1 Couchbase Architecture and Scalability
- 1. Couchbase Sydney Meetup Couchbase Architecture and Scalability Karthik Babu Sekar Solutions Architect, WW Field Operations @ Couchbase - LinkedIn - https://www.linkedin.com/in/karthikbabusekar - Twitter- https://twitter.com/SKBhere - E-mail – [email protected]
- 2. ©2016 Couchbase Inc.©2015 Couchbase Inc. History of Couchbase 2
- 3. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase provides a complete Data Management solution 3 High availability cache Key-value store Document database Embedded database Sync management Multi-purpose capabilities support a broad range of apps and use cases Enterprises often start with cache, then broaden usage to other apps and use cases
- 4. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase meets today’s & tomorrow’s requirements 4 Flexible data model Consistent performance at scale High availability Easy, affordable scalability 24x365
- 5. ©2016 Couchbase Inc.©2015 Couchbase Inc. Common Use-Cases 5 360 Degree Customer View Profile Management Catalog Fraud Detection Content Management Internet of Things Digital Communication Real Time Big Data Mobile Applications Personalization
- 6. ©2016 Couchbase Inc.©2015 Couchbase Inc. What makes Couchbase unique? 6 Performance & scalability leader Sub millisecond latency with high throughput; memory-centric architecture Multi- purpose Enterprise class Web Console Easy to deploy & manage; integrated Admin Console, single- click cluster expansion & rebalance Cache, key value store, document database, and local/mobile database in single platform Always-on availability Data replication across nodes, clusters, and data centers Enterprises choose Couchbase for several key advantages 24x365
- 7. ©2016 Couchbase Inc.©2015 Couchbase Inc. § Consolidated cache and database § Tune memory required based on application requirements Multi-purpose database supports many uses 7 7 Tunable built-in cache Flexible schemas with JSON Couchbase Lite § Represent data with varying schemas using JSON on the server or on the device § Index and query data with Javascript views § Light weight embedded DB for always available apps § Sync Gateway syncs data seamlessly with Couchbase Server
- 8. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase delivers always-on availability 8 High Availability Disaster Recovery Backup & Restore § In-memory replication with manual or automatic fail over § Rack-zone awareness to minimize data unavailability § Memory-to-memory cross cluster replication across data centers or geos § Active-active topology with bi- directional setup § Full backup or Incremental backup with online restore § Delta node catch-ups for faster recovery after failures 24x365
- 9. ©2016 Couchbase Inc.©2015 Couchbase Inc. Simplified administration for exceptional ease of use Online upgrades and operations Built-in enterprise class admin console Restful APIs § Online software, hardware and DB upgrades § Indexing, compaction, rebalance, backup & restore § Perform all administrative tasks with the click of a button § Monitor status of the system visual at cluster level, database level, server level § All admin operations available via UI, REST APIs or CLI commands § Integrate third party monitoring tools easily using REST
- 10. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase leads in performance and scalability Auto Sharding Memory-memory XDCR Multi Dimensional Scaling § No manual sharding § Database manages data movement to scale out – not the user § Market’s only memory-to- memory database replication across clusters and geos § Provides disaster recover / data locality § Hugely simplifies management of clusters § Easy to scale clusters by adding any number of nodes
- 11. Service-Based Architecture Multi-Dimensional Scalability (MDS) – Optionally Scale each service independently: • Data • Index • Query MDS is the architecture that enables independent scaling of data, query, and indexing workloads while being managed as one cluster.
- 12. ©2015 Couchbase Inc. Full Cluster Architecture 12 STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service
- 13. ©2015 Couchbase Inc. Full Cluster Architecture 13 STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service
- 15. High level architecture - Single Node Couchbase Server nodes are identical Two core components ü Cluster Manager ü Data Manager Three independent services ü Data Service ü Index Service ü Query Service Couchbase Server Node Data Manager Cluster Manager Management REST API Web UI Erlang / OTP Node / Cluster Coordination Data Service Storage Managed Cache Get/Set Map Reduce Views Query Service Query Engine Query Plan Cache Storage Index Service Indexing Engine Managed Cache Storage
- 16. Cluster Manager Management and coordination Responsible for ü UI / REST interface ü Configuration / Administration ü Coordinating cluster rebalancing ü Process monitoring ü Statistics ü No data processing Erlang/OTP code base ü Developed by Ericsson ü Telco-switch grade performance Couchbase Server Node Cluster Manager Management REST API Web UI Erlang / OTP Node / Cluster Coordination Data Manager Data Service Storage Managed Cache Get/Set Map- Reduce Views Query Service Query Engine Query Plan Cache Storage Index Service Indexing Engine Managed Cache Storage :8091 Admin Console/API
- 17. Node Manager Copyright © 2015 Couchbase, Inc. Three supported workloads Data Service ü Get/Set of individual documents ü Distributed secondary indexes (Views) Index Service ü Global secondary indexes ü Index maintenance and provision Query Service ü N1QL (SQL superset for JSON) ü Execution plans, index coordination C / C++ and Go code bases ü Very fast and memory efficient even with large memory footprint Couchbase Server Node Cluster Manager Management REST API Web UI Erlang / OTP Node / Cluster Coordination Data Manager Data Service Storage Managed Cache Get/Set Map- Reduce Views Query Service Query Engine Query Plan Cache Storage Index Service Indexing Engine Managed Cache Storage
- 18. Single node type / code base ü All services available on every node ü Services enabled independently Allows cluster to be configured for ü Varying hardware capacities per node ü Application workload tuning Couchbase Server Node Cluster Manager Management REST API Web UI Erlang / OTP Node / Cluster Coordination Data Manager Query Service Data Service Index Service Index Service Indexing Engine Managed Cache Storage Query Service Query Engine Query Plan Cache Storage Data Service Storage Managed Cache Get/Set Map- Reduce Views Multi-Dimensional Scalability Copyright © 2015 Couchbase, Inc.
- 19. ©2015 Couchbase Inc. 19 Couchbase Data Access • Everything is built on top of Key Value • A Document store is a special case of Key-Value • Views provide aggregation and real-time analytics through incremental map-reduce • Global Secondary Indexes provide low latency/high throughput indexes • N1QL is a language that provides a powerful and expressive way of accessing documents
- 20. Key Differentiators - N1QL N1QL is a Next-generation NoSQL query language for JSON. 20 § JOINS § Powerful Extensions for JSON § ODBC/JDBC drivers available
- 21. Introduction to N1QL SQL query language for documents • SELECT * FROM WHERE/LIKE/JOIN/GROUP/etc, CREATE INDEX • Extended for JSON to support nested and hierarchical data structures • Support for map-reduce views and Global Secondary Indexes • Query (DQL), Manipulation (DML), Description (DDL) • ODBC/JDBC drivers
- 22. SQL 4 documents - examples 2222 Listing products in a category Finding the most popular products in a category
- 23. Global Secondary Indexes New Storage Options for Global Secondary Index 1. Memory-Optimized Global Secondary Indexes (MOI) • Optimized for Memory: small memory footprint, optimized for lowest latency queries • Faster Indexing: fresh indexes under heavy mutations with lock free index maintenance • Predictable low latency with Query and Key Based Operations, even in presence of large number of indexes 1. Standard Global Secondary Indexes • Optimized for Storage + Memory: efficient spill to disk with ForestDB as the storage engine • Advanced IO Performance: new circular writes that minimize compaction overhead of append only writes
- 29. ©2016 Couchbase Inc. INTRODUCING SEARCH Structured Query AND Search One platform. | Search Feeling lucky?
- 34. ©2016 Couchbase Inc. INTRODUCING REALTIME ANALYTICS You may also like these products Thank you for your order! Order #: 3423943 Paid Partner Content Operations & Analytics. One platform.
- 38. ©2016 Couchbase Inc. INTRODUCING REALTIME ANALYTICS You may also like these products Thank you for your order! Order #: 3423943 Paid Partner Content Operations & Analytics. One platform.
- 39. ©2016 Couchbase Inc. Data Warehouse Hadoop THE PLATFORM FOR YOUR DIGITAL APPLICATIONS
- 40. ©2016 Couchbase Inc. THE DATA PLATFORM FOR THE DIGITAL ECONOMY Unified Adminstration Unified Programming Interface Data Query Index SearchMobileReplication Analytics {N1QL }
- 41. XDCR: Cross Data Center Replication Application can access both clusters (master – master) Scales out linearly Different from intra-cluster replication (“CP” versus “AP”)
- 42. XDCR: Flexible topologies One-one, one-many, many-one Differently sized and resourced clusters supported
- 44. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase Mobile Overview
- 45. ©2016 Couchbase Inc.©2015 Couchbase Inc. Always Available and Fast 45
- 46. ©2016 Couchbase Inc.©2015 Couchbase Inc. Native Support 46
- 47. ©2016 Couchbase Inc.©2015 Couchbase Inc. Couchbase Mobile Solution 47
- 48. Key Take-aways Couchbase Mobile = Couchbase Lite + Couchbase Sync Gateway + Couchbase Server Couchbase Lite: Only NoSQL database for mobile devices (phones, tables, embedded systems) Couchbase Sync Gateway: Offline/Online synchronization 48
- 49. ©2016 Couchbase Inc.©2015 Couchbase Inc. Backup and Restore with Couchbase § Zer0-downtime backup and restore § Built-in utilities: cbbackup / cbrestore § Full, differential and cumulative backup available per-bucket. § Restore from any point, to any bucket or topology
- 50. Demo & Q&A