Modern Data Backup and Recovery from On-Premises to the Public Cloud
0 likes276 views
The document discusses modern data backup and recovery strategies for MongoDB deployments moving from on-premises to public cloud environments. It describes the evolution of backup architectures from complex on-premises solutions utilizing multiple agents and load balancers to take backups, to more simplified future architectures that leverage features like WiredTiger checkpoints to backup directly to object storage. It also outlines the different backup options available with MongoDB Atlas in the cloud, including continuous backups with point-in-time recovery, queryable backups for granular restores, and utilizing native cloud provider snapshots.
Ad
More Related Content
What's hot (18)
Similar to Modern Data Backup and Recovery from On-Premises to the Public Cloud (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Modern Data Backup and Recovery from On-Premises to the Public Cloud
- 2. Modern Data Backup and Recovery from On-Premises to the Public Cloud Tomer Yakir Senior Engineer, Cloud
- 3. Tomer Yakir Senior Engineer, Cloud MongoDB [email protected] https://github.com/tomeryakir
- 4. This presentation contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward- looking statements are subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events. Furthermore, such forward-looking statements speak only as of the date of this presentation. In particular, the development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements. Safe Harbor Statement
- 5. Quick level set — We’re not going to talk about disaster recovery
- 7. Application Driver Primary Secondary Secondary Replication But what happens when you have data corruption in your primary or someone accidentally does something they shouldn’t have... ?
- 10. Data Recovery - That’s what we’re going to focus on today
- 11. Private DBaaS: On-Prem Public DBaaS: Fully Managed Built on the Same Code Base, Same API, Same Management UI MongoDB Offerings Hybrid DBaaS
- 12. The Future of Backup for Self Managing or On- Premises
- 13. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06
- 14. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group
- 15. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup
- 16. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB
- 17. ... ... Blockstore06 Blockstore04 Blockstore02 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 18. ... ... ... ... Oplog04 Oplog02 Blockstore06 Blockstore04 Blockstore02 Oplog06 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Oplog01 Oplog03 Oplog05 Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 19. On-Premises Backup Architecture Today Ops Manager Initial Sync Apply Ops Snapshot Backup Daemon HeadDB Source Backup Agent Monitoring Agent Oplog App Server for Backup Blockstore01
- 20. Backup Strategy ● Reduce the footprint and architectural complexities needed for backup ● Reduce cost ● Improve performance
- 22. What is it? ● Enhanced WiredTiger to take checkpoints of itself ● Checkpoints are then moved to long term storage ● HeadDB’s are completely eliminated — no more backup initial syncs! ○ Reduced storage requirements ○ Reduces architectural complexities ○ Reduces infrastructure required ● Consolidated agents, from three agents to one
- 23. ... ... ... ... Oplog04 Oplog02 Blockstore06 Blockstore04 Blockstore02 Oplog06 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Oplog01 Oplog03 Oplog05 Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 24. Oplog02 Blockstore New Backup Architecture — Phase 1 Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group BackupLoadBalancer App Server/Backup Daemon App Server/Backup Daemon Oplog01 S3 Snapshot Store
- 25. S3 Oplog Blockstore New Backup Architecture — Phase 2 Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup Oplog01 S3 Snapshot Store
- 26. S3 Oplog New Backup Architecture — Future Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group S3 Snapshot Store
- 27. Future Improvements ● Agents read/write directly to OpLog and snapshots store ● Leave a checkpoint behind
- 28. What this means for you... ● Less people, time, and energy you have to spend to deliver a best-in- class data recovery solution for your on-premises and self-managed MongoDB deployments
- 29. Atlas
- 30. Private DBaaS: On-Prem Public DBaaS: Fully Managed Built on the Same Code Base, Same API, Same Management UI Atlas: Same Features as Ops Manager & Cloud Manager Hybrid DBaaS
- 31. The Latest MongoDB Features • MongoDB Atlas comes out-of-the-box with MongoDB 3.4, 3.6, 4.0 • Transactions (4.0) • Change Streams (3.6) • JSON Schema (3.6) • Expressive nested array updates (3.6) • Expressive joins: $lookup (3.6) • Graph queries (3.4) • Facets & expressive aggregations (3.4) • Minor updates and major upgrades without downtime ● Data Explorer ○ Inspect schema & index utilization ○ CRUD ● Real-Time Performance Panel ○ Live telemetry: in-flight operations & resource consumption ● Performance Advisor ○ Always-on index recommendations
- 32. MongoDB Atlas: Global Cloud Database Self-service & elastic Deploy, modify, and upgrade on demand with best-in-class operational automation Automated database maintenance Database and infrastructure resources as code Scale up, out, or down in a few clicks or API calls Global & cloud-agnostic Available in dozens of regions across AWS, Azure, GCP Global clusters for read/write anywhere deployments and multi-region fault tolerance Easy migrations with a consistent experience across cloud providers Enterprise-grade security & SLAs Network isolation, VPC peering, end-to-end encryption, and role-based access controls Encryption key management, LDAP integration, granular database auditing SOC 2 / Privacy Shield / HIPAA Guaranteed reliability with SLAs Comprehensive monitoring Deep visibility into 100+ KPIs with proactive alerting Real-time performance tracking and Performance Advisor APIs to integrate with monitoring dashboards Managed backup Point-in-time data recovery Queryable backup snapshots Consistent snapshots of sharded deployments Cloud data mobility Stitch: Serverless platform services Simple, serverless functions for backend logic, service integrations, and APIs Database access from your frontend secured by straightforward, field-level access rules Database and authentication triggers to react to changes in real time
- 33. Atlas Backups: Data Recovery Roll back the clock when you run into issues triggered by user or application errors that are replicated from the primary to the rest of your cluster.
- 34. Point-in-Time Data Recovery • Lets you select a restore time based on your PIT window • Restores the closest snapshot and rolls ahead • Reduces the possibility of data loss
- 35. What About Small Disasters? • The application is working fine • But there is data missing or has been altered • No time to do a full restore
- 36. Queryable Backups • Ability to query your snapshots and restore data at the document level in minutes. • Reduces the operational overhead associated with: • Identifying whether data of interest has been altered • Pinpointing the best point in time to restore a database
- 37. Sample Queryable Script db = source.locations db2 = destination.locations zips = db.zipcodes zips2 = db2.zipcodes def restore(): print "Finding Missing Data" query = {'state': 'CO'} try: cursor = zips.find(query) except Exception as e: print "Unexpected error:", type(e), e for doc in cursor: zips2.insert(doc) source = queryable snapshot
- 39. Pick the fully managed backup solution that’s right for you Within Atlas is a modern data recovery platform that enables you to choose how you want to backup your data, depending on your requirements. Continuous ● Point-in-time restore ○ All but eliminate the possibility of data loss ● Queryable snapshots ○ Rapidly restore granular data Cloud Provider Snapshots ● Localized backup ○ Satisfy data sovereignty requirements ● Bring your own keys ● Take snapshots on-demand ● The cost effective option
- 42. Cloud Provider Snapshots (CPS) • Utilizes each providers native snapshot capabilities • Granular backup region selection • Satisfy data sovereignty requirements • Supports replica sets and sharded clusters • Pricing is based on snapshot size, not datasize • Less expensive, starting at $0.14 per GB of snapshot size • Available on AWS and Azure • AWS snapshots are incremental 10 GB 15 GB (5 new) 20 GB (5 new) Incremental snapshots for Atlas customers deploying on AWS
- 43. Cloud Provider Snapshots (CPS) Updates — Continued • Bring your own keys • AWS Key Management Service • Azure Key Vault • Configure snapshot time and retention • Choose when you want to take a snapshot • Choose how many snapshots you want to retain • On-demand snapshots (quicksave) • Takes a snapshot immediately if there’s not already one in progress • API for pipeline integrations
- 45. Whats Next?
- 46. Cloud Provider Snapshots Future ● New Modernized Scheduler ● Launch on GCP ○ GCP KMS (Bring your own Keys) ● Point in Time Restores with customizable window
- 47. Thank You!