AWS to Bare Metal: Motivation, Pitfalls, and Results

Jun 8, 20152 likes2,797 views

Like many startups, Wish grew up on AWS. As our cluster grew and the price of SSDs fell, we started exploring bare metal. Fast-forward 2 years and we have hundreds of MongoDB instances on bare metal fully integrated with our AWS infrastructure. It wasn't all smooth sailing, but the performance & cost improvements were worth it! Hear the story of how we did it and gain a framework for thinking about how to make the leap from cloud-centric architecture to a hybrid model.

Wish saved 35% on MongoDB costs
Improved latency by 20%
And reduced latency variance

HI, I’M ADAM.
(I’m a software engineer; I also run production…)

I WORK AT WISH.
(we’re a mobile eCommerce platform)

I WORK AT WISH.
(we also grow really fast…)

AWSTO BARE METAL
• The Why
• The Scope
• The Servers
• The Network
• The Operations
• The Results

there was spinning disk EBS
In the beginning

DB slows to a crawl
Replica set detects failure
Election kills the app for 30s
App slows down
EBS LATENCY SPIKE

Provisioned IOPS EBS launches
Summer 2012

AWS to Bare Metal: Motivation, Pitfalls, and Results

GOAL
Find lowest cost per query
for your workload

THROUGHPUT & LATENCY
• Typically: more throughput → more latency
• Application dictates max latency (p95?)
• For each hardware conﬁg…
• Find highest throughput under max latency

OURTOOL
• Restore the snapshot
• Clear ﬁlesystem caches
• Replay ops at conﬁgured throughput
• Report on latency / MongoDB stats

LATEST SPECS
• 2x Ivy Bridge 3.3 GHz (32 hyperthreads)
• 256 GB RAM
• 3.2TB LSI WarpDrive PCI-e
YO
U
R
M
ILEAG
E
M
AY
VA
RY!

NETWORKS ARE WEIRD
• Network engineering is weird for software people
• Need to master a few, big pieces
• We wasted a lot of time improvising…

PLANTO FAIL
• Every component and connection fails
• Switch dies?
• NIC dies?
• Switch ⟷ switch connection dies?
• DirectConnect dies?

THE OPERATIONS
• Migration / Rollback
• Backups
• Processes
• Documentation

MIGRATION (PREP)
• Add new nodes to replica set
• hidden: true, priority: 0
• Wait for them to sync

MIGRATION (READ-ONLY)
• Unhide nodes:
• hidden: false, priority: 0

MIGRATION (READ-WRITE)
• Force primary into colo:
• hidden: false, priority: 2

MIGRATION (DONE)
• Hide old AWS nodes:
• hidden: true, priority: 0

ROLLBACK
• No big deal
• Adjust hidden/priority to move trafﬁc back

BACKUPS
• EBS snapshots rock!
• Hidden member in EC2 for backup
• Nice for DR too…

PROCESSES
• No RackServer() API
• Ensure consistency:
• Checklists
• Veriﬁcation tools

DOCUMENTATION
• No DescribeInstances either…
• Consider life without AWS Management Console
• Worse: consider it being occasionally wrong

DOCUMENTATION
• Wiremaps
• Network maps (IPs,VLANs, etc)
• Equipment specs
• Serial numbers

As machine learning moves from niche to mainstream tech stacks how do DevOps engineers prepare for a very different set of problems. A brief look at the new issues that arise from machine learning, an overview of cutting-edge "old school" solutions and how to drag data science (kicking and screaming) into a world of automation. Video: https://www.youtube.com/watch?v=KHxZCRajRiA Join DevOps Exchange London here: http://meetup.com/DevOps-Exchange-London/ Follow DOXLON on twitter http://www.twitter.com/doxlon

Reliable, Scalable Kubernetes on AWSApplatix

Meetup #3: Migrate a fast scale system to AWSAWS Vietnam Community

The document discusses migrating a fast scaling system from physical servers to AWS. It outlines the agenda which includes an overview of physical systems and virtualization, the Topica Edumall system design, applying DevOps practices, and a Q&A. The Topica Edumall hybrid system is described as using VPS, physical servers, and AWS for core video streaming. Benefits of migrating the auto-scaling system to Docker and Kubernetes on AWS include high availability, easy scaling, auto-healing, and saving 40% costs with full utilization of AWS services. DevOps practices like CI/CD workflows are also discussed.

Using Redis as Distributed Cache for ASP.NET apps - Peter Kellner, 73rd Stre...Redis Labs

I will build from scratch in this session a Microsoft ASP.NET website that caches WebAPI REST calls with both MSOpenTech’s Redis implementation for running while developing in Visual Studio as well as running on a Windows server running IIS. I will show you how to build a safe reusable caching library in c# that can be used in any .net project. I will also demonstrate how to use the Redis cache services that are available on Microsoft’s Azure cloud platform. Further, I’ll demonstrate a real world web site that uses Azure Redis cache and show statistics on how Redis improves performance consistently and reliably.

Flynn Bundy - 60 micro-services in 6 months WinOps Conf

In this talk, I want to take the audience on a journey of how we (Coolblue) migrated 60 .Net micro-services to the AWS Cloud. This talk covers the high’s, low’s and everything in between when working in a multi-disciplinary Developer / Operations Cloud team. This talk will cover the evolution of our processes and toolsets to align with Chaos Engineering best practices. Most importantly, I want to highlight how we changed the way we thought about services and servers in general. The key takeaways from this talk would be related to: Continous Inspection (TeamCity) Continous Deployment (Octopus Deploy) Infrastructure as Code (Cloudformation) Chaos Engineering (Chaos Monkey) Monitoring and Logging (Datadog and Splunk) .Net and .Net Core (on Windows Server 2016) Automation in AWS Cloud

Network Infrastructure as Code with Chef and CiscoMatt Ray

Chef is an open source framework for managing infrastructure as code. It uses a declarative language and recipes to describe and configure systems to match a desired state. Chef works by having clients pull policies from a server and apply them locally, then reporting the state back. Cisco and Chef are working together to bring official Chef support to Cisco's NX-OS and IOS-XR platforms through Omnibus packages and a Cisco cookbook with resources for managing Cisco devices. This will allow network infrastructures to be managed as code through testing, versioning, and continuous delivery practices.

Neil Peterson - Azure CLI Deep DiveWinOps Conf

The document provides an overview of the Azure CLI, including: - The Azure CLI allows creation and management of Azure resources from the command line across platforms. - It can be installed via several options and authenticated interactively or non-interactively. - Resources can be created, queries performed, and configurations scoped via the CLI. - Interactive mode aids command discovery and the Cloud Shell integrates it in the Azure portal. - The open source project is on GitHub for contribution and support.

Scalable and reliable kubernetes on awsApplatix

This document summarizes a presentation about running Kubernetes reliably and at scale on AWS. It discusses some common issues like out of memory kills affecting Etcd and other Kubernetes processes. It also covers AWS resource limits and volume issues that need to be addressed for cloud hygiene. The presentation recommends using auto scaling groups to scale infrastructure and introduces Applatix as a platform to build and run containerized apps on Kubernetes in the cloud with automation and devops workflows for monitoring, auditing and governance.

Rik Hepworth - ARM Yourself for Effective Azure ProvisioningWinOps Conf

Visualization of RDS metrics using AWS CLI and JQuery at AWS Usergroup Leipzigroot360 GmbH

Cloudsolutionday 2016: Compliance and cost controlling on AWSAWS Vietnam Community

This document discusses cost controlling and compliance at Misfit, a company that has been part of Fossil Group since 2011. It provides lessons learned around cost controlling including separating AWS accounts by environment, tagging resources, and using unit costs to simplify conversations. It also discusses implementing common compliance standards like ISO, PCI, and HIPAA by prioritizing shared requirements, conducting gap assessments, and maintaining documentation and training programs. The document outlines challenges of compliance work including time requirements and gaining awareness from other teams.

Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...anynines GmbH

Containers and CloudStackShapeBlue

Webinar - DreamObjects/Ceph Case StudyCeph Community

This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.

Cloudsolutionday 2016: DevOps workflow with Docker on AWSAWS Vietnam Community

The document discusses DevOps workflow with Docker on AWS. It describes using Docker to isolate application environments, increasing team productivity and decreasing development team size. Key elements include using Gitlab for source control and CI/CD, building Docker images via Gitlab runners, and deploying to Kubernetes clusters. The workflow allows writing applications once and running them anywhere and forever through continuous integration and delivery of Docker images to private container registries on AWS.

Day 2 General Session Presentations RedisConfRedis Labs

The document discusses new memory technologies like persistent memory and their implications. It provides latency and bandwidth numbers for different memory types and notes that heterogeneous memory systems using tiers of DRAM and NVM provide opportunities for better performance and cost. Examples are given of key-value stores and databases leveraging NVM to achieve high performance while reducing costs. The talk also discusses how new distributed data structures like CRDTs could be used across servers with shared memory.

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015Datadog

Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Trivadis

Heutzutage schreibt man nicht nur Applikationen mit Code. Dank der Cloud wird die Konfiguration von Infrastruktur wie virtuellen Maschinen oder Netzwerken in Code definiert und automatisiert ausgeliefert. Man spricht von Infrastructure as Code, kurz: IAC. Für Infrastructure as Code auf Azure gibt es viele tools wie Ansible, Puppet, Chef, etc. Zwei Lösungen stechen durch Ihren unterschiedlichen Ansatz heraus - Die Azure Resource Manager Templates (ARM) als Microsoft-native Lösung, immer auf dem neusten Stand, aber an Azure gebunden. Auf der anderen Seite Terraform von HashiCorp mit einer deskriptiven Sprache als Grundlage, dafür weniger Features im Security-Bereich. Für einen Grosskunden haben wir die beiden Technologien verglichen. Die Resultate zeigen wir in dieser Session mit Livedemos auf.

Alexander Naydenko - Nagios to Zabbix Migration | ZabConf2016Zabbix

Introducing Kubestr - A New Way to Explore Your Kubernetes Storage OptionsLibbySchulze

Kubestr is a tool to help users identify, validate and evaluate the various storage options in their Kubernetes cluster. It can identify the different storage options present, validate that they are configured correctly, and evaluate the performance of storage using benchmarking tools like FIO to understand if the right storage is being used for their workloads and applications. The goal is to make it easy for users to debug, validate and benchmark their Kubernetes storage.

Python & Cassandra - Best FriendsJon Haddad

Netflix Cloud Platform and Open Sourceaspyker

Andrew Spyker presented on Netflix's cloud platform and open source projects. Some key points included: - Netflix has migrated from monolithic architectures to microservices and continuous delivery enabled by their open source libraries and services. - Their platform focuses on elasticity, high availability through automation, and operational visibility. - Netflix uses technologies like Eureka, Ribbon, Hystrix, and Servo to enable scalability, resilience, and monitoring across their distributed systems. - They contribute over 50 open source projects to help others adopt their cloud-native approaches and are working on data and UI related projects.

Virtualization at Gilt - Rangarajan RadhakrishnanDatadog

Gilt uses virtualization including LXC containers and CloudStack to improve efficiency and reliability. They aim to automate provisioning, use immutable containers, and avoid repeated config enforcement. Currently they use LXC containers provisioned through CloudStack and their in-house Galactica system, which integrates with services like Puppet, LDAP, DNS and load balancers. They are testing provisioning LXC containers on different Linux distros and configurations to optimize performance.

What We Learned From Building a Modern Messaging and Streaming System for CloudStreamNative

Sijie Guo discusses lessons learned from building Apache Pulsar, a modern messaging and streaming system. Key points include: - Pulsar was designed for the cloud-native era, separating compute and storage for scalability unlike earlier systems designed for on-premise use. - It supports unified messaging and streaming with a single API and multi-protocol support like Kafka, AMQP, and MQTT. - Features like multi-tenancy, geo-replication, and infinite stream storage help support use cases like multi-cloud, hybrid cloud, and managing entire data lifecycles. - The talk reflects on how Pulsar's architecture was influenced by trends in cloud computing,

Oracle Exalytics - Tips and Experiences from the Field (Enkitec E4 Conference...Mark Rittman

SupportNet - Your Virtual Hyperion Help DeskPerficient, Inc.

Infrastructure choices - cloud vs colo vs bare metalServer Density

This document discusses the differences between cloud, colocation, and bare metal infrastructure options. It covers key considerations for performance including CPU, memory, disk, and network latency and bandwidth. Colocation provides hardware at a specific location while maintaining internal skills, but has costs for total spend, hardware specifications, and power usage. Cloud infrastructure offers elastic workloads and support for demand spikes and unknown requirements, but bare metal is preferable for managed hardware replacement and networking needs. Overall, the best option depends on an organization's specific workload characteristics and skills.

Exalytics, DR, EPM Multi-Instance Over Bare Metal, and Tying it All TogetherAlithya

Exalytics for MII sales instituteBrama Dhaneswara

This document discusses Oracle's business intelligence and enterprise performance management products, with a focus on Oracle Exalytics. It notes that business analytics was the top priority for CIOs in 2012 according to Gartner. It describes Oracle Exalytics as optimized hardware for business intelligence and in-memory analytics. Examples are given of customers who saw significantly faster performance and reduced cycle times using Oracle Exalytics compared to alternative solutions.

More Related Content

What's hot (17)

Scalable and reliable kubernetes on awsApplatix

Rik Hepworth - ARM Yourself for Effective Azure ProvisioningWinOps Conf

Visualization of RDS metrics using AWS CLI and JQuery at AWS Usergroup Leipzigroot360 GmbH

Cloudsolutionday 2016: Compliance and cost controlling on AWSAWS Vietnam Community

Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...anynines GmbH

Containers and CloudStackShapeBlue

Webinar - DreamObjects/Ceph Case StudyCeph Community

Cloudsolutionday 2016: DevOps workflow with Docker on AWSAWS Vietnam Community

Day 2 General Session Presentations RedisConfRedis Labs

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015Datadog

Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Trivadis

Alexander Naydenko - Nagios to Zabbix Migration | ZabConf2016Zabbix

Introducing Kubestr - A New Way to Explore Your Kubernetes Storage OptionsLibbySchulze

Python & Cassandra - Best FriendsJon Haddad

Netflix Cloud Platform and Open Sourceaspyker

Virtualization at Gilt - Rangarajan RadhakrishnanDatadog

What We Learned From Building a Modern Messaging and Streaming System for CloudStreamNative

Scalable and reliable kubernetes on awsApplatix

Rik Hepworth - ARM Yourself for Effective Azure ProvisioningWinOps Conf

Visualization of RDS metrics using AWS CLI and JQuery at AWS Usergroup Leipzigroot360 GmbH

Cloudsolutionday 2016: Compliance and cost controlling on AWSAWS Vietnam Community

Cloud Infrastructures Slide Set 8 - More Cloud Technologies - Mesos, Spark | ...anynines GmbH

Containers and CloudStackShapeBlue

Webinar - DreamObjects/Ceph Case StudyCeph Community

Cloudsolutionday 2016: DevOps workflow with Docker on AWSAWS Vietnam Community

Day 2 General Session Presentations RedisConfRedis Labs

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015Datadog

Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Trivadis

Alexander Naydenko - Nagios to Zabbix Migration | ZabConf2016Zabbix

Introducing Kubestr - A New Way to Explore Your Kubernetes Storage OptionsLibbySchulze

Python & Cassandra - Best FriendsJon Haddad

Netflix Cloud Platform and Open Sourceaspyker

Virtualization at Gilt - Rangarajan RadhakrishnanDatadog

What We Learned From Building a Modern Messaging and Streaming System for CloudStreamNative

Viewers also liked (14)

Oracle Exalytics - Tips and Experiences from the Field (Enkitec E4 Conference...Mark Rittman

SupportNet - Your Virtual Hyperion Help DeskPerficient, Inc.

Infrastructure choices - cloud vs colo vs bare metalServer Density

Exalytics, DR, EPM Multi-Instance Over Bare Metal, and Tying it All TogetherAlithya

Exalytics for MII sales instituteBrama Dhaneswara

Considering bare metal as a viable cloud optionInternap

Bare metal servers provide higher performance than virtual servers by avoiding hypervisor overhead and directly utilizing server hardware. This allows bare metal servers to achieve higher IOPS, throughput, network speeds, and CPU and disk performance. Bare metal servers also offer more consistent performance compared to nominal specifications for virtual servers. Customers can provision bare metal servers on demand via APIs or a web portal for flexibility and automation similar to a cloud environment.

App Sharding to Autosharding at SailthruMongoDB

Sailthru is a marketing automation platform that uses MongoDB as its primary datastore. It has evolved its use of MongoDB from a single node replica set to autosharding across multiple nodes to support growing data and traffic volumes. Sailthru selected a hashed _id as its shard key to distribute data evenly. It also uses a "key collection" to optimize lookups by fields like email across shards. Sailthru migrated its data from its existing replica set to the new autosharding cluster without downtime using Mongo Connector.

A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloudDr. Wilfred Lin (Ph.D.)

This document discusses Oracle IaaS and its strategy to run any type of enterprise workload in the cloud. It highlights Oracle's breadth of compute, storage, and networking offerings including bare metal servers, virtual machines, containers, and dedicated compute. It emphasizes Oracle's performance leadership and the ability to run demanding workloads due to its global infrastructure of regions, availability domains, and ultra low latency/high bandwidth. The document also discusses Oracle's engineered systems, Cloud at Customer offering, and tools for migration and workload portability including Ravello.

The Best of Both Worlds: Speeding Up Drug Research with MongoDB & Oracle (Gen...MongoDB

1) Genentech uses MongoDB to speed up drug research by allowing flexible data collection and analysis of genetic testing results. 2) MongoDB integrates easily with their existing RDBMS and allows loading of CSV files directly into flexible document structures for analysis. 3) This helps Genentech introduce new genetic tests and instruments more quickly, in weeks rather than months, potentially saving lives.

Cloud Migration: Moving to the CloudDr.-Ing. Michael Menzel

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

Igor Anishchenko Odessa Java TechTalks Lohika - May, 2012 Let's take a step back and compare data serialization formats, of which there are plenty. What are the key differences between Apache Thrift, Google Protocol Buffers and Apache Avro. Which is "The Best"? Truth of the matter is, they are all very good and each has its own strong points. Hence, the answer is as much of a personal choice, as well as understanding of the historical context for each, and correctly identifying your own, individual requirements.

2015 Future of Cloud Computing StudyNorth Bridge

2015 North Bridge Future of Cloud Computing Study, with Wikibon |Broadest exploration of cloud trends, cloud migration & evolution of the cloud computing sector. Survey participation was the largest to date and included responses from 38 countries. 50 collaborators supported the 5th Annual Future of Cloud Computing study, which reveals that cloud has become an accepted and integral technology. Furthermore, the study shows that despite deployment gaps among clouds, we should expect a future powered by hybrid cloud technologies. The question of whether companies are using the cloud has morphed to how deeply cloud adoption is integrated within the business. From the bottom to the top, all products and services will in some way be powered by the cloud making the promise of goods and services that have the potential to be better tomorrow than today. IT departments have reclaimed the reins on driving company technology strategy and cloud adoption as roles, skills and processes have shifted. Importantly, We’re also seeing the emergence of the cloud as the only way businesses can truly get more out of their data including analyzing and executing on it real-time. On the investment front, 2015 could tip the scale from private to public capital for SaaS companies.

Cloud Migration Cookbook: A Guide To Moving Your Apps To The CloudNew Relic

Migrating to Cloud - A Step by Step Imaginea

Oracle Exalytics - Tips and Experiences from the Field (Enkitec E4 Conference...Mark Rittman

SupportNet - Your Virtual Hyperion Help DeskPerficient, Inc.

Infrastructure choices - cloud vs colo vs bare metalServer Density

Exalytics, DR, EPM Multi-Instance Over Bare Metal, and Tying it All TogetherAlithya

Exalytics for MII sales instituteBrama Dhaneswara

Considering bare metal as a viable cloud optionInternap

App Sharding to Autosharding at SailthruMongoDB

A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloudDr. Wilfred Lin (Ph.D.)

The Best of Both Worlds: Speeding Up Drug Research with MongoDB & Oracle (Gen...MongoDB

Cloud Migration: Moving to the CloudDr.-Ing. Michael Menzel

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

2015 Future of Cloud Computing StudyNorth Bridge

Cloud Migration Cookbook: A Guide To Moving Your Apps To The CloudNew Relic

Migrating to Cloud - A Step by Step Imaginea

Similar to AWS to Bare Metal: Motivation, Pitfalls, and Results (20)

Five Years of EC2 DistilledGrig Gheorghiu

Running MongoDB in the CloudTony Tam

SQL Server: Now It's Everywhere You Want to BeEd Leighton-Dick

Have you heard the news? SQL Server is coming to Linux! What? That can't be right... or can it? With the release of SQL Server 2017, Microsoft really is bringing SQL Server to the Linux platform. They've also added support for running SQL Server in Docker containers. How does it work? And how well does it work? In this session, we'll answer these questions (and more!), and we'll talk about how this enables some interesting new possibilities for DevOps.

How Serverless Changes DevOpsRichard Donkin

What does Serverless mean for DevOps, in practical terms? While Serverless does reduce the need for server-centric DevOps, it poses new challenges in many areas including security, app deployment and cloud resource provisioning, partly due to an explosion of "nanoservices". Based on a current project using AWS, we cover relevant tools, techniques and tips to deliver a smooth serverless experience for development through to production. Delivered at Bristol DevOps meetup, 27 Jun 2018. To see detailed notes covering extra points not on slides, click the Notes link just below (or download the Powerpoint). Update: here's the correct link for Gojko Adzic talk on the Backendless slide - https://www.youtube.com/watch?v=w7X4gAQTk2E

John adams talk cloudyJohn Adams

Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...QAware GmbH

CloudNativeCon North America 2017, Austin (Texas, USA): Talk by Josef Adersberger (@adersberger, CTO at QAware) Abstract: Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like: - What technical constraints of Kubernetes can be obstacles for applications and how to tackle these? - How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security? - How to industrialize and govern the migration process? - How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?

The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...Josef Adersberger

Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like: - What technical constraints of Kubernetes can be obstacles for applications and how to tackle these? - How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security? - How to industrialize and govern the migration process? - How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?

Deploying microservices on AWSMichael Haberman

This document discusses deploying microservices on AWS. It begins by explaining what microservices are and then discusses hosting options on AWS including EC2, ECS, and Lambda. ECS is identified as the preferred option since it allows hosting containers with Docker. The document then covers deployment aspects like using source control with Git for multiple environments, building and testing code, deploying single services or entire clusters, live testing, and monitoring with alerts.

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror

Twitter's operations team manages software performance, availability, capacity planning, and configuration management for Twitter. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, and optimizing databases to reduce replication delay and locks. The team also created several open source projects like CacheMoney for caching and Kestrel for asynchronous messaging.

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight

Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The operations team manages software performance, availability, capacity planning, and configuration management using metrics, logs, and data-driven analysis to find weak points and take corrective action. They use managed services for infrastructure to focus on computer science problems. The document outlines Twitter's rapid growth and challenges in maintaining performance as traffic increases. It provides recommendations around caching, databases, asynchronous processing, and other techniques Twitter uses to optimize performance under heavy load.

Fixing_Twitterliujianrong

Twitter's operations team manages software performance, availability, capacity planning, and configuration management. They use metrics, logs, and analysis to find weak points and take corrective action. Some techniques include caching everything possible, moving operations to asynchronous daemons, optimizing databases, and instrumenting all systems. Their goal is to process requests asynchronously when possible and avoid overloading relational databases.

Fixing twitterRoger Xia

Fixing Twitter and Finding your own Fail Whale document discusses Twitter operations. The Twitter operations team focuses on software performance, availability, capacity planning, and configuration management using metrics, logs, and science. They use a dedicated managed services team and run their own servers instead of cloud services. The document outlines Twitter's rapid growth and challenges in maintaining performance. It discusses strategies for monitoring, analyzing metrics to find weak points, deploying changes, and improving processes through configuration management and peer reviews.

Chirp 2010: Scaling TwitterJohn Adams

This document summarizes a keynote speech given by John Adams, an early Twitter engineer, about scaling Twitter operations from 2008-2009. Some key points: 1) Twitter saw exponential growth rates from 2008-2009, processing over 55 million tweets per day and 600 million searches per day. 2) Operations focused on improving performance, reducing errors and outages, and using metrics to identify weaknesses and bottlenecks like network latency and database delays. 3) Technologies like Unicorn, memcached, Flock, Cassandra, and daemons were implemented to improve scalability beyond a traditional RDBMS and handle Twitter's data volumes and real-time needs. 4) Caching,

Stackato v2Jonas Brømsø

Cassandra Summit 2014: Deploying Cassandra for Call of DutyDataStax Academy

Presenters: Seán O Sullivan, Service Reliability Engineer & Tim Czerniak, Software Engineer at Demonware This presentation covers the eight-month evaluation process we underwent to migrate some of Call of Duty’s core services from MySQL to Cassandra. We will outline our requirements, the process we followed for the evaluation, decisions we made around our schema, configuration and hardware, and some issues we encountered.

From ddd to DDD : My journey from data-driven development to Domain-Driven De...Thibaud Desodt

This will be a review of my progress through architecture styles and patterns and going over the transitions from Db-first style to cleaner OOP practices and proper domain isolation. I'll go over concepts such as "Transaction Script", "CQS", "Anemic Model" and other buzzwords  Expect some code snippets in C# You can find the original of the slides here : https://github.com/tsimbalar/from-ddd-to-ddd

A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?DATAVERSITY

Wordnik migrated from a MySQL relational database to the non-relational MongoDB database for 5 key reasons: speed, stability, scaling, simplicity, and fitting their object model better. They tested MongoDB extensively, iteratively improving their data mapping and access patterns. The migration was done without downtime by switching between the databases. While inserts were much faster in MongoDB, updates could be slow due to disk I/O. Wordnik addressed this through optimizations like pre-fetching on updates and moving to local storage. Overall, MongoDB was a better fit for Wordnik's large and evolving datasets.

Stackato v3Jonas Brømsø

Windows Azure introductionMicrosoft Iceland

The document introduces Microsoft's Windows Azure cloud platform. It summarizes that Windows Azure provides an operating system for the cloud that abstracts away hardware and provides services for automated management, scalable computing and storage. It allows developers to build applications and services that can easily scale across large, connected data centers. The talk demonstrates how Windows Azure allows building complex service architectures from simple components like web and worker roles that interact through a durable storage system. It emphasizes that the platform aims to provide a familiar development experience while handling all the complexities of highly scalable cloud services.

Inside Wordnik's ArchitectureTony Tam

Wordnik's architecture is built around a large English word graph database and uses microservices and ephemeral Amazon EC2 storage. Key aspects include: 1) The system is built as independent microservices that communicate via REST APIs documented using Swagger specifications. 2) Databases for each microservice are kept small by design to facilitate operations like backups, replication, and index rebuilding. 3) Services are deployed across multiple Availability Zones and regions on ephemeral Amazon EC2 storage for high availability despite individual host failures.

Five Years of EC2 DistilledGrig Gheorghiu

Running MongoDB in the CloudTony Tam

SQL Server: Now It's Everywhere You Want to BeEd Leighton-Dick

How Serverless Changes DevOpsRichard Donkin

John adams talk cloudyJohn Adams

Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...QAware GmbH

The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...Josef Adersberger

Deploying microservices on AWSMichael Haberman

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight

Fixing_Twitterliujianrong

Fixing twitterRoger Xia

Chirp 2010: Scaling TwitterJohn Adams

Stackato v2Jonas Brømsø

Cassandra Summit 2014: Deploying Cassandra for Call of DutyDataStax Academy

From ddd to DDD : My journey from data-driven development to Domain-Driven De...Thibaud Desodt

A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?DATAVERSITY

Stackato v3Jonas Brømsø

Windows Azure introductionMicrosoft Iceland

Inside Wordnik's ArchitectureTony Tam

More from MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB

This presentation discusses migrating data from other data stores to MongoDB Atlas. It begins by explaining why MongoDB and Atlas are good choices for data management. Several preparation steps are covered, including sizing the target Atlas cluster, increasing the source oplog, and testing connectivity. Live migration, mongomirror, and dump/restore options are presented for migrating between replicasets or sharded clusters. Post-migration steps like monitoring and backups are also discussed. Finally, migrating from other data stores like AWS DocumentDB, Azure CosmosDB, DynamoDB, and relational databases are briefly covered.

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB

MongoDB Kubernetes operator and MongoDB Open Service Broker are ready for production operations. Learn about how MongoDB can be used with the most popular container orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications. A demo will show you how easy it is to enable MongoDB clusters as an External Service using the Open Service Broker API for MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB

Our clients have unique use cases and data patterns that mandate the choice of a particular strategy. To implement these strategies, it is mandatory that we unlearn a lot of relational concepts while designing and rapidly developing efficient applications on NoSQL. In this session, we will talk about some of our client use cases, the strategies we have adopted, and the features of MongoDB that assisted in implementing these strategies.

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB

Encryption is not a new concept to MongoDB. Encryption may occur in-transit (with TLS) and at-rest (with the encrypted storage engine). But MongoDB 4.2 introduces support for Client Side Encryption, ensuring the most sensitive data is encrypted before ever leaving the client application. Even full access to your MongoDB servers is not enough to decrypt this data. And better yet, Client Side Encryption can be enabled at the "flick of a switch". This session covers using Client Side Encryption in your applications. This includes the necessary setup, how to encrypt data without sacrificing queryability, and what trade-offs to expect.

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB

When you need to model data, is your first instinct to start breaking it down into rows and columns? Mine used to be too. When you want to develop apps in a modern, agile way, NoSQL databases can be the best option. Come to this talk to learn how to take advantage of all that NoSQL databases have to offer and discover the benefits of changing your mindset from the legacy, tabular way of modeling data. We’ll compare and contrast the terms and concepts in SQL databases and MongoDB, explain the benefits of using MongoDB compared to SQL databases, and walk through data modeling basics so you feel confident as you begin using MongoDB.

MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB

The document discusses guidelines for ordering fields in compound indexes to optimize query performance. It recommends the E-S-R approach: placing equality fields first, followed by sort fields, and range fields last. This allows indexes to leverage equality matches, provide non-blocking sorts, and minimize scanning. Examples show how indexes ordered by these guidelines can support queries more efficiently by narrowing the search bounds.

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB

The document describes a methodology for data modeling with MongoDB. It begins by recognizing the differences between document and tabular databases, then outlines a three step methodology: 1) describe the workload by listing queries, 2) identify and model relationships between entities, and 3) apply relevant patterns when modeling for MongoDB. The document uses examples around modeling a coffee shop franchise to illustrate modeling approaches and techniques.

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB

Virtual assistants are becoming the new norm when it comes to daily life, with Amazon’s Alexa being the leader in the space. As a developer, not only do you need to make web and mobile compliant applications, but you need to be able to support virtual assistants like Alexa. However, the process isn’t quite the same between the platforms. How do you handle requests? Where do you store your data and work with it to create meaningful responses with little delay? How much of your code needs to change between platforms? In this session we’ll see how to design and develop applications known as Skills for Amazon Alexa powered devices using the Go programming language and MongoDB.

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB

Il n’a jamais été aussi facile de commander en ligne et de se faire livrer en moins de 48h très souvent gratuitement. Cette simplicité d’usage cache un marché complexe de plus de 8000 milliards de $. La data est bien connu du monde de la Supply Chain (itinéraires, informations sur les marchandises, douanes,…), mais la valeur de ces données opérationnelles reste peu exploitée. En alliant expertise métier et Data Science, Upply redéfinit les fondamentaux de la Supply Chain en proposant à chacun des acteurs de surmonter la volatilité et l’inefficacité du marché.