DevOps Incident Handling - Making friends not enemies.

Oct 14, 20143 likes1,426 views

David Mytton CEO of Server Density presented this talk to the DevOps Meetup in London. It takes you through how to handle DevOps incidents, outages and downtime -- and more specifically how to make friends, not enemies in the process.

How to win friends when
handling outages and downtime
David Mytton
London DevOps - Oct 2014
blog.serverdensity.com

Server monitoring, cloud management,
dashboards and alerting
serverdensity.com

How much do you spend?
You will have downtime

DevOps Incident Handling - Making friends not enemies.

Preparation - On Call
● Primary?
● Secondary?

Preparation - On Call
● Primary?
● Secondary?
● Reachability - Tube, 3G/4G
(edge?!), Do Not Disturb mode, at
the gym, family emergency,
system updates

Preparation - On Call
● Off call
● Rotations

Preparation - On Call
● Off call
● Rotations
● Illness

Preparation - On Call
● Off call
● Rotations
● Illness
● Work the next day?

Preparation - Documentation
● Searchable

Preparation - Documentation
● Searchable
● Easy to edit

Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure

Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
● Up to date

Preparation - Key Info
● Team contacts
● Key vendor contacts

Preparation - Key Info
● Team contacts
● Key vendor contacts
● Credentials to key systems

Unexpected failures
● Communication systems

Unexpected failures
● Communication systems
● Network connectivity

Unexpected failures
● Communication systems
● Network connectivity
● Access to support

ALERT!
1. Load up incident response checklist

ALERT!
1. Load up incident response checklist
2. Log incident in JIRA

ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room

ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post

ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
4. Public status post
5. Initial investigation

Key response principles
● Log everything

Key response principles
● Log everything
● Frequent public status updates

Key response principles
● Log everything
● Frequent public status updates
● Gather the team

Key response principles
● Log everything
● Frequent public status updates
● Gather the team
● Escalate!

Postmortem
● Within a few days
● Tell the story

Postmortem
● Within a few days
● Tell the story
● Provide technical detail

Postmortem
● Within a few days
● Tell the story
● Provide technical detail
● Explain what failed and why

Summary
● Preparation
● Documentation
● Communication
● Checklists
● Postmortem

どもありがとうございます
＠davidmytton
david@serverdensity.com
blog.serverdensity.com
www.serverdensity.com

The document outlines a 10 step guide for validating a business idea, beginning with searching Google to research product competition, customers, and related topics. It recommends tools like Google Trends, Keywords Planner, YouTube, and Maps to analyze search volume and interest for keywords. Further steps include searching for competitors, using free datasets, Google Patents, and website builders. It emphasizes testing ideas quickly through online forms and Google Analytics to iterate the concept based on user feedback.

APIA 2018: How Can Google Help You - a 10 Steps Guide to the Online Validatio...European Innovation Academy

The document outlines a 10 step guide for validating a business idea, beginning with searching Google to research the market, competitors, and customer demand. It recommends using tools like Google Trends, Keywords Planner, Maps, and YouTube to analyze interest in ideas and discover competitors. Later steps involve directly engaging potential customers through website feedback forms or analytics to test reactions to concepts before fully pursuing development. The key message is that iteration and testing ideas leanly is crucial before committing significant resources to a new business.

Gestione del tempo: Il principio di ParetoManager.it

The document discusses different tools available including the 4 quadrants, Pareto's principle, managing everything, a report card, objective-magnet, and forest and trees perspective. It specifically focuses on Tool #2, the Pareto principle or 80/20 rule, and how this could apply to the Italian economy, clothes, donors, and time. The reader is prompted to read notes in Italian about each topic in relation to the 80/20 rule.

Telexfree ItaliaGaggetti Luigi

University of Mauritius - UoM Codewars 2017 RFC implementationCodarren Velvindron

Introduction To ICT Security Audit OWASP Day Malaysia 2011Linuxmalaysia Malaysia

This document outlines an agenda for an IT audit and assessment event. It discusses identifying risks like outdated patches, weak passwords, open ports, and phishing. It provides resources for monitoring vulnerabilities and cracking passwords. Labs are suggested to practice tasks like patching, password cracking, port scanning, and network monitoring. The objective is to harden systems against common threats by finding vulnerabilities and ensuring regular security reviews are performed. Participants are encouraged to get involved to improve the security of their organization.

incident analysis - procedure and approachDerek Chang

This document provides guidance on conducting post-mortem analyses of outages. It discusses establishing standard procedures and templates to minimize defensiveness and promote learning. Key recommendations include focusing analyses on process improvement rather than blame, having a third party lead investigations, and ensuring action is taken on findings through follow up. Templates are provided to structure data collection, including timelines, root cause analysis, and review of monitoring and logging. The overall goal is for teams to walk away from outages looking like heroes by continually improving based on lessons learned.

Regus / URM Business Continuity SurveyRegus

The survey found that while most organizations have business continuity plans (BCPs) in place, many of these plans may not be fully effective due to a lack of business impact analysis (BIA) to identify critical processes. Additionally, there is often insufficient internal communication about BCPs to employees and a lack of proactive engagement with suppliers about continuity arrangements. To develop more robust BCPs, organizations need to conduct BIAs to understand recovery priorities, better communicate BCP details and staff responsibilities internally, and take a more proactive approach to ensuring continuity in the supply chain.

HSSE Management - JMT M99 Incident AnalysisYang Ming

Critical incident analysis of knmSoe Lu Kyaw

There were two main problems described in the document: 1) issues with cash collection between the company, KMN, and its clients, and 2) poor communication between KMN's marketing and accounts departments. This led to poor cash collection performance and impacted the relationship with clients and KMN's company image. The author reflects that most involved handled the incident appropriately and identifies alternative actions like additional cash collection training and using a dedicated team and software to improve the process going forward.

You build it, you run itSkyscanner

This document discusses the "You Build It, You Run It" (YBYR) approach where developers are responsible for the full software release cycle from defining requirements to monitoring in production. The approach aims to speed up feature delivery through faster recovery from failures and more frequent releases. While it provides benefits like fewer production failures, adopting YBYR requires cultural change and depends on context, with no single solution. Frameworks and coaching are important to implement the approach successfully over time without making things worse initially.

Project post-mortem analysisJaiveer Singh

International Business Management, Disney Land CaseSoe Lu Kyaw

Disney was motivated to set up parks abroad to expand sales and profits by tapping into global markets. They made different levels of financial investment in different countries based on perceived risk. In Japan Disney took no financial risk by partnering with a local company, while investing $140 million in France and over $300 million in Hong Kong due to greater opportunities and government support. Disney has succeeded by adjusting to foreign cultures and environments, but initially failed in Paris by not doing so. Setting up a park in Shanghai could be successful if Disney makes adjustments for China's large population, regulations, traffic, and affordability.

Post mortem reportKuaci Pedas

This document provides a template for a project post mortem report to record lessons learned from completed projects. The template includes sections for report details, project parameters, performance including accomplishments and problems, lessons learned, and approval. It is intended to inform future project teams of obstacles, challenges, successes and ways to improve aspects like planning, resources, scope, scheduling and more.

IT - Enterprise Service Operation CenterSameer Paradia

Post Mortem TemplateDennis Stevenson

The document summarizes a project retrospective meeting held by Dennis Stevenson on April 23, 2008. The meeting involved exercises to identify key events that led to the project outcome, determine root causes, and develop a shared action plan to monitor risks and improve outcomes for future projects. Participants analyzed events from a neutral perspective, prioritized critical factors, and identified ways to detect risks so issues can be addressed promptly. The retrospective concluded by summarizing what was achieved.

NoC simulators presentationHossam Hassan

The document provides an overview of simulators used for modeling and evaluating Networks-on-Chip (NoCs). It discusses what a simulator is and the benefits of using simulators. It also describes different types of simulators, such as cycle-accurate versus event-driven simulators. Finally, it summarizes several popular NoC simulators, including BookSim, NNSE, Noxim, Wormsim, HNoCS, SICOSYS, TOPAZ, and NIRGAM. The document provides high-level information on the capabilities and features of these simulators in 1-3 sentences for each.

Best practices for building network operations centerSatish Chavan

The document discusses best practices for building a network operations center (NOC). Some key points: - A NOC monitors and controls network activity from one or more locations. Early versions date back to the 1960s when AT&T opened centers to monitor switches and routes. - Modern NOCs use network monitoring software and sophisticated systems to detect issues across multiple layers of the network before they impact the business. - Maintaining skilled staff, efficient processes, integrated tools, automation, and a focus on performance, security and being proactive are characteristics of an effective NOC.

ITIL and DevOps can be friendsJan-Joost Bouwman

The document discusses how ITIL and DevOps approaches can work together at a large bank, ING. It describes how ING transitioned to more agile development methods over time from 2011-2015. There was initially a divide between risk management wanting to use full ITIL and DevOps teams finding ITIL too administrative. However, the document proposes ways to make ITIL processes more lean and integrated with agile approaches. For example, reserving sprint capacity for incidents, minimizing technical debt, and automating configuration management. The conclusion is that ITIL and agile/DevOps are not in conflict and both provide value when the processes are made lean and both sides understand each other's expertise.

Collective Bargaining in NursingSujata Mohapatra

Collective bargaining is a process of negotiations between employers and employee representatives, usually unions, to determine working conditions and terms of employment. It aims to reach binding agreements on wages, hours, benefits, grievance procedures, and other aspects of work. The document discusses the history and objectives of collective bargaining, how it works, advantages and disadvantages, and its role in the nursing profession specifically. It provides definitions of key terms, the differences between professional associations and unions, and factors that influence successful negotiations.

Network Operations Center Muhannad Kalbouneh

NOC Service deskamerica.gss

The 24x7 network and customer support service from GSS America at 8x5 costs provides high quality service at low costs. It offers 24/7 monitoring of networks and servers as well as technical support and service desk support. This helps companies manage their off-peak customers in a cost-effective way while still providing excellent around-the-clock support. GSS America's services are backed by industry-leading processes, security certifications, and infrastructure to ensure quality service and support.

NETWORK OPERATION CENTERRolando Rodriguez

El documento describe un Network Operation Center (NOC), el cual es un sitio desde donde se controlan y monitorean las redes de computación y telecomunicaciones de una organización las 24 horas del día los 365 días del año. Un NOC permite visualizar la topología de red, detectar alarmas de diferentes niveles, y trabajar de forma remota sin necesidad de estar físicamente en la oficina. El NOC se encarga de supervisar la red, identificar problemas, y escalar incidentes de forma jerárquica para resolverlos de manera rápida.

Incident ManagementAbhishek Agnihotry

Network Operation Center Best PracticesAyehu Software Technologies Ltd.

Digital Portfolios Silvia Rosenthal Tolisano

Experiences from DevOps production: Deployment, performance, failure.Server Density

Continuous Infrastructure FirstKris Buytaert

More Related Content

Viewers also liked (20)

incident analysis - procedure and approachDerek Chang

Regus / URM Business Continuity SurveyRegus

HSSE Management - JMT M99 Incident AnalysisYang Ming

Critical incident analysis of knmSoe Lu Kyaw

You build it, you run itSkyscanner

Project post-mortem analysisJaiveer Singh

International Business Management, Disney Land CaseSoe Lu Kyaw

Post mortem reportKuaci Pedas

IT - Enterprise Service Operation CenterSameer Paradia

Post Mortem TemplateDennis Stevenson

NoC simulators presentationHossam Hassan

Best practices for building network operations centerSatish Chavan

ITIL and DevOps can be friendsJan-Joost Bouwman

Collective Bargaining in NursingSujata Mohapatra

Network Operations Center Muhannad Kalbouneh

NOC Service deskamerica.gss

NETWORK OPERATION CENTERRolando Rodriguez

Incident ManagementAbhishek Agnihotry

Network Operation Center Best PracticesAyehu Software Technologies Ltd.

Digital Portfolios Silvia Rosenthal Tolisano

incident analysis - procedure and approachDerek Chang

Regus / URM Business Continuity SurveyRegus

HSSE Management - JMT M99 Incident AnalysisYang Ming

Critical incident analysis of knmSoe Lu Kyaw

You build it, you run itSkyscanner

Project post-mortem analysisJaiveer Singh

International Business Management, Disney Land CaseSoe Lu Kyaw

Post mortem reportKuaci Pedas

IT - Enterprise Service Operation CenterSameer Paradia

Post Mortem TemplateDennis Stevenson

NoC simulators presentationHossam Hassan

Best practices for building network operations centerSatish Chavan

ITIL and DevOps can be friendsJan-Joost Bouwman

Collective Bargaining in NursingSujata Mohapatra

Network Operations Center Muhannad Kalbouneh

NOC Service deskamerica.gss

NETWORK OPERATION CENTERRolando Rodriguez

Incident ManagementAbhishek Agnihotry

Network Operation Center Best PracticesAyehu Software Technologies Ltd.

Digital Portfolios Silvia Rosenthal Tolisano

Similar to DevOps Incident Handling - Making friends not enemies. (20)

Experiences from DevOps production: Deployment, performance, failure.Server Density

Continuous Infrastructure FirstKris Buytaert

OSMC 2015 | Testing in Production by Devdas BhagatNETWAYS

The document discusses various topics related to software testing and development including testing environments vs reality, risk management approaches, monitoring software usage through event processing and generating alerts. It notes that testing environments often differ significantly from production with unstable conditions, humans and latency. Two approaches to risk management are described: narrowly scoping problems and extensive testing vs rapid iteration, testing changes in real world, and only keeping what works. Event processing and monitoring tools can provide real-time insights into software usage and changes.

OSMC 2015: Testing in Production by Devdas BhagatNETWAYS

For most ecommerce companies, software is not the final deliverable product. It is a research tool, to determine what customers will pay for. To be able to get good data from software, monitoring and analytics must be built into the system. Alerting must come from business requirements and be based on application generated data. In the traditional operations world, we monitor what is easy, and avoid monitoring that which is difficult. This talk is an attempt to show people that monitoring must be driven by metrics from the CxO office, and then potentially involve technical metrics if needed. This talk explains why functional and business level monitoring is crucial. We also cover the tradeoffs from a DTAP model to continuous deployment. There will be a brief introduction to a couple of useful monitoring tools for functional monitoring. No special technical skills are expected of the audience, but having a general overview of the monitoring world is a good thing. This talk is not limited to ecommerce companies, but is most applicable to that environment.

BSIT3CD_Continuation of Cyber incident response (1).pdfStevenJoeBiago

This document discusses cyber incident response. It covers incident classification, network event monitoring, network monitoring tools, and detecting network events. Incident classification involves categorizing incidents by method of attack. Network event monitoring collects and analyzes data from devices to understand network conditions. Common tools to monitor include port scanners, interface monitors, packet flow monitors, and SNMP. Detecting events involves watching traffic for policy violations, exploits, and unknown traffic using NIDS and NIPS.

Firebird recovery tools and techniques by IBSurgeonAlexey Kovyazin

Peter holditch devopsPeter Holditch

This document discusses the value of DevOps and monitoring tools in improving collaboration between development and operations teams and justifying investments in automation. It notes that traditionally, dev teams focused on features while ops focused on incidents, but both were measured by vague business metrics like revenue and uptime. New tools can help baseline current problems, measure progress over time, and demonstrate business impact to obtain support for further investments. The document advocates for monitoring the full customer experience rather than individual system components.

Undo tech overview_201410gregthelaw

Scaling humans - Ops teams and incident managementServer Density

The document discusses the costs of downtime for companies and best practices for incident management teams to prepare for, respond to, and review incidents to minimize downtime costs. It notes that downtime costs companies billions per quarter and recommends teams prepare documentation and contact information, have on-call rotation schedules, log all responses to incidents, provide frequent status updates, gather teams to escalate incidents as needed, and conduct post-mortem reviews within days of an incident.

Don't lose revenue. Go viral with no downtime.WP Engine

Startup survival lessonsCotap Engineering

Understanding and implementing website securityDrew Gorton

Knowing security best practices only gets a team so far. They have to implement them too. This session will cover the security risks that a web development team faces and the underlying reasons why risks can go unaddressed. Ultimately, there are no excuses for leaving your web projects exposed to known vulnerabilities. This session will cover common security concerns for Drupal and the root problems a team needs to solve in order to mitigate these risks. We will cover: Three layers of web security, from the perspective of Drupal: Platform-level (e.g. Linux), Application-level (e.g. Drupal), and Organizational-level (e.g. procedures) Familiarity with your hosting platform’s security-related practices. Overview of common vulnerabilities in web applications (XSS, CSRF, HTTP vs HTTPS, etc.) Understanding how security concerns are handled for core and contrib. Clarifying support responsibilities and procedures so that security fixes are applied quickly. Attendees who build and/or manage Drupal sites will gain the most from the session. Attendees will leave with a complete picture of website security and concrete recommendations for how to improve the security of the sites they manage. It will cover recommendations for Drupal 7 and Drupal 8. Many of the topics that will be covered are in my Understanding and Implementing Website Security blog post series at https://pantheon.io/blog/understanding-and-implementing-website-security-part-1-you-are-target

Practical Cyber DefensePaul Dutot IEng MIET MBCS CITP OSCP CSTM

APIdays Singapore 2019 - Building Applications in the Cloud: Best Practices F...apidays

Continuous Deployment Applied at MyHeritageRan Levy

11 tools for your PHP devops stackKris Buytaert

11 tools for your devops stack summarizes key tools for a devops workflow including: Jenkins for continuous integration; FPM for packaging; Logstash for centralized logging; Graphite for metrics and monitoring; Puppet and Vagrant for configuration management and environment provisioning; and tools like Mcollective and Kibana to help with automation, monitoring and visualization. Sharing code, environments and experiences through open source is also emphasized as important for a collaborative devops culture.

The Continuous Delivery processXPeppers

Continuous Infrastructure First Ignite EditionKris Buytaert

This document discusses different approaches to implementing continuous infrastructure and collaboration between development and operations teams. It argues that starting with operations automating their own workflows first allows them to understand the tools and processes before enforcing them on developers, prevents technical debt, and enables delivery and provisioning from the start. Having dedicated operations resources focus on automation leads to improved collaboration and quality over having "devops" teams dictate tools to operations.

DevOps Days Kyiv 2019 -- continuous Infrafirstructure First //Kris buytaertMykola Marzhan

This document discusses different approaches to implementing continuous infrastructure and collaboration between development and operations teams. It argues that starting with operations automating their own workflows first allows them to understand the tools and processes before enforcing them on developers, prevents technical debt, and enables delivery and provisioning from the start. Getting operations involved early through dedicating resources to infrastructure as code avoids prolonged periods where operations teams are playing catch up or acting as heroes.

Handling incidentsServer Density

The document discusses how to handle incidents, downtime, and outages. It notes that the cost of downtime for companies in Q1 2015 was $2.9 billion, $870 million, and $4.1 billion. It recommends preparing for incidents by having on-call staff and documentation, responding quickly by following an incident response checklist and notifying stakeholders, and performing a postmortem within days to analyze what failed and how to prevent future issues.

Experiences from DevOps production: Deployment, performance, failure.Server Density

Continuous Infrastructure FirstKris Buytaert

OSMC 2015 | Testing in Production by Devdas BhagatNETWAYS

OSMC 2015: Testing in Production by Devdas BhagatNETWAYS

BSIT3CD_Continuation of Cyber incident response (1).pdfStevenJoeBiago

Firebird recovery tools and techniques by IBSurgeonAlexey Kovyazin

Peter holditch devopsPeter Holditch

Undo tech overview_201410gregthelaw

Scaling humans - Ops teams and incident managementServer Density

Don't lose revenue. Go viral with no downtime.WP Engine

Startup survival lessonsCotap Engineering

Understanding and implementing website securityDrew Gorton

Practical Cyber DefensePaul Dutot IEng MIET MBCS CITP OSCP CSTM

APIdays Singapore 2019 - Building Applications in the Cloud: Best Practices F...apidays

Continuous Deployment Applied at MyHeritageRan Levy

11 tools for your PHP devops stackKris Buytaert

The Continuous Delivery processXPeppers

Continuous Infrastructure First Ignite EditionKris Buytaert

DevOps Days Kyiv 2019 -- continuous Infrafirstructure First //Kris buytaertMykola Marzhan

Handling incidentsServer Density

More from Server Density (19)

Content marketing @ Server DensityServer Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

This document discusses improving incident response procedures through practices such as checklists, documented procedures, realistic incident simulations and postmortems. It recommends extended use of checklists to guide responses while still allowing for experience and independent thought. Regular incident response simulations that test both general processes and specific failures can help refine procedures and build confidence. Postmortems should objectively review incidents, suggest improvements and run through scenarios again over time to prevent complacency.

How to Monitor MySQLServer Density

Briefing: ContainersServer Density

Containers seem to have suddenly become the hot new thing everyone is talking about, but what are they? Why are they important? How should you use them and what does it mean for cloud infrastructure? This talk will examine the history, technical details and strategy around containerisation from the perspective of developers and operations, consider internal container OSs like Rocket and Ubuntu Core as well as management layers like Docker and Apache Mesos and take a look at why cloud providers are launching their own services around them. Presented by David Mytton at Datacloud Monaco 2015-06-04

Why puppet? Why now?Server Density

Why Puppet? Why now? Can you get by without using any config management? You probably think don't have time, or that your project is too small. What can using Puppet really add? How can you justify investing time up front? Maybe you can just do it later? Getting started with config management can often seem like a big project, especially if you only manage a few systems or have a small team. This talk will examine why you should use Puppet from the beginning. It will examine what you can do with Puppet that couldn't do otherwise, how much time it will save and why it's especially important if you think your project has even the smallest chance of scaling in the future. Presented by David Mytton at Puppet Camp London 2015-04-13

Infrastructure choices - cloud vs colo vs bare metalServer Density

This document discusses the differences between cloud, colocation, and bare metal infrastructure options. It covers key considerations for performance including CPU, memory, disk, and network latency and bandwidth. Colocation provides hardware at a specific location while maintaining internal skills, but has costs for total spend, hardware specifications, and power usage. Cloud infrastructure offers elastic workloads and support for demand spikes and unknown requirements, but bare metal is preferable for managed hardware replacement and networking needs. Overall, the best option depends on an organization's specific workload characteristics and skills.

Navigating the customer lifecycleServer Density

How to monitor NGINXServer Density

How to monitor MongoDBServer Density

David Mytton is a MongoDB master and the founder of Server Density. In this presentation David delves deeper into what's discussed in our how to monitor MongoDB tutorial (https://blog.serverdensity.com/monitor-mongodb/), with the aim of taking you through: Key MongoDB metrics to monitor. Non-critical MongoDB metrics to monitor. Alerts to set for MongoDB on production. Tools for monitoring MongoDB.

High performance Infrastructure Oct 2013Server Density

The document discusses high performance infrastructure for Server Density which includes 150 servers that have been running since June 2009 and migrated from MySQL to MongoDB. It stores 25TB of data per month. Key aspects of performance discussed are using fast networks like 10 Gigabit Ethernet on AWS, ensuring high memory, using SSDs over spinning disks for performance, and factors like replication lag based on location. The document also compares options like using cloud, dedicated servers, or colocation and discusses monitoring, backups, dealing with outages, and other operational aspects.

Puppet at the centre of everythingServer Density

The document discusses Server Density's architecture which includes 100 Ubuntu servers with 50% being virtual, using Nginx, Python, and MongoDB. It handles 25TB of data per month. Puppet is used for configuration, failover, code deploys, and system updates. The document also considers colocating servers versus using a dedicated provider and factors like hardware specs, costs, skills required, and fun.

NoSQL Infrastructure - Late 2013Server Density

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment. This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Remote startup - building a company from everywhere in the worldServer Density

This document discusses how Server Density grew from 2 employees in 2009 to 12 employees in 2013 while being fully remote. It outlines the company's timeline and some advantages of being fully remote such as access to worldwide talent, lower costs, and no distractions of an office. However, it also notes disadvantages like collaboration being more difficult without in-person interactions. While the company added an office in 2012, the document emphasizes that working remotely is a mindset that is difficult to adopt after initially co-locating. Effective communication and availability are important for fully distributed teams.

NoSQL InfrastructureServer Density

This document discusses MongoDB infrastructure at Server Density. It notes that Server Density uses 27 MongoDB nodes to store 20TB of data per month from their MySQL database. Some key reasons for choosing MongoDB include replication, official drivers, easy deployment, and fast performance out of the box. The document then discusses various MongoDB performance and infrastructure considerations like network throughput, replication lag, failover processes, disk types, backups, and monitoring.

StartOps: Growing an ops team from 1 founderServer Density

Bootstrapped startups don't have the luxury of a full team of ops engineers available to respond to issues 24/7, so how can you survive on your own? This talk will tell the story of how to run your infrastructure as a single founder through to growing that into a team of on call engineers. It will include some interesting war stories as well as tips and suggestions for how to run ops at a startup. Presented at DevOpsDays London 2013 by David Mytton.

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

MongoDB is easy to download and run locally but requires some thought and further understanding when deploying to production. At scale, schema design, indexes and query patterns really matter. So does data structure on disk, sharding, replication and data centre awareness. This talk will examine these factors in the context of analytics, and more generally, to help you optimise MongoDB for any scale. Presented at MongoDB Days London 2013 by David Mytton.

Puppet Camp Ghent 2013Server Density

This document discusses adding Forge modules to Puppet Enterprise. It describes moving the HTTP load balancer from Pound to nginx, keeping existing manifests pulled from GitHub, and using the Puppet Forge to install the puppetlabs/nginx module or integrating it via Git submodules. It also covers parameterizing classes on the Puppet Enterprise console and merging site.pp files, as well as updating nginx configurations on the fly and considering alternative nginx modules on the Forge.

Puppet Camp 2012Server Density

Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density. Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed. Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances. www.serverdensity.com/puppetcamp/

Content marketing @ Server DensityServer Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

How to Monitor MySQLServer Density

Briefing: ContainersServer Density

Why puppet? Why now?Server Density

Infrastructure choices - cloud vs colo vs bare metalServer Density

Navigating the customer lifecycleServer Density

How to monitor NGINXServer Density

How to monitor MongoDBServer Density

High performance Infrastructure Oct 2013Server Density

Puppet at the centre of everythingServer Density

NoSQL Infrastructure - Late 2013Server Density

Remote startup - building a company from everywhere in the worldServer Density

NoSQL InfrastructureServer Density

StartOps: Growing an ops team from 1 founderServer Density

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Puppet Camp Ghent 2013Server Density

Puppet Camp 2012Server Density

Recently uploaded (20)

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

Cyber Awareness overview for 2025 month of securityriccardosl1

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

Cyber Awareness overview for 2025 month of securityriccardosl1

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

DevOps Incident Handling - Making friends not enemies.

1. How to win friends when handling outages and downtime David Mytton London DevOps - Oct 2014 blog.serverdensity.com

2. David Mytton

3. Server monitoring, cloud management, dashboards and alerting serverdensity.com

4. Slides: twitter.com/davidmytton

5. Let’s talk about downtime

6. 2013 Spend: ~$5bn

7. 2013 Spend: ~$6bn

8. 2013 Spend: ~$4bn

9. How much do you spend? You will have downtime

11. Preparation

12. Preparation - On Call ● Primary?

13. Preparation - On Call ● Primary? ● Secondary?

14. Preparation - On Call ● Primary? ● Secondary? ● Reachability - Tube, 3G/4G (edge?!), Do Not Disturb mode, at the gym, family emergency, system updates

15. Preparation - On Call ● Off call

16. Preparation - On Call ● Off call ● Rotations

17. Preparation - On Call ● Off call ● Rotations ● Illness

18. Preparation - On Call ● Off call ● Rotations ● Illness ● Work the next day?

19. Preparation - Documentation

20. Preparation - Documentation ● Searchable

21. Preparation - Documentation ● Searchable ● Easy to edit

22. Preparation - Documentation ● Searchable ● Easy to edit ● Independent of your infrastructure

23. Preparation - Documentation ● Searchable ● Easy to edit ● Independent of your infrastructure ● Up to date

25. Preparation - Key Info

26. Preparation - Key Info ● Team contacts

27. Preparation - Key Info ● Team contacts ● Key vendor contacts

28. Preparation - Key Info ● Team contacts ● Key vendor contacts ● Credentials to key systems

29. Unexpected failures

30. Unexpected failures ● Communication systems

31. Unexpected failures ● Communication systems ● Network connectivity

32. Unexpected failures ● Communication systems ● Network connectivity ● Access to support

33. ALERT!

34. ALERT! 1. Load up incident response checklist

35. ALERT! 1. Load up incident response checklist 2. Log incident in JIRA

36. ALERT! 1. Load up incident response checklist 2. Log incident in JIRA 3. Log into Ops War Room

37. ALERT! 1. Load up incident response checklist 2. Log incident in JIRA 3. Log into Ops War Room 4. Public status post

38. ALERT! 1. Load up incident response checklist 2. Log incident in JIRA 3. Log into Ops War Room 4. Public status post 5. Initial investigation

39. Key response principles

40. Key response principles ● Log everything

41. Key response principles ● Log everything ● Frequent public status updates

42. Key response principles ● Log everything ● Frequent public status updates ● Gather the team

43. Key response principles ● Log everything ● Frequent public status updates ● Gather the team ● Escalate!

44. Postmortem

45. Postmortem ● Within a few days

46. Postmortem ● Within a few days ● Tell the story

47. Postmortem ● Within a few days ● Tell the story ● Provide technical detail

48. Postmortem ● Within a few days ● Tell the story ● Provide technical detail ● Explain what failed and why

49. Postmortem ● How it’s going to be fixed

50. stspg.io/ZDC

51. Summary ● Preparation ● Documentation ● Communication ● Checklists ● Postmortem

52. どもありがとうございます＠davidmytton [email protected] blog.serverdensity.com www.serverdensity.com

DevOps Incident Handling - Making friends not enemies.

Recommended

More Related Content

Viewers also liked (20)

Similar to DevOps Incident Handling - Making friends not enemies. (20)

More from Server Density (19)

Recently uploaded (20)

DevOps Incident Handling - Making friends not enemies.