SRE in Apiary

May 22, 20182 likes2,448 views

The document discusses Site Reliability Engineering (SRE) at Apiary. It provides details on Apiary's SRE team size and growth over time, their focus on culture over tools, shared responsibility for the platform, monitoring everything, gradual changes through continuous delivery, and emphasis on automation, on-call processes, incident response, and learning from postmortems. The goal of SRE is to decrease errors, eliminate toil, and focus on creative engineering work that improves reliability, performance, and scalability.

SRE in Apiary
CZJUG 21.5.2018
Ladislav Prskavec
@abtris
1

"What happens when a software engineer is tasked with what used to
be called operations."
» Ben Treynor Sloss, Vice President, Google Engineering,
founder of Google SRE
3

Apiary in numbers
» Apiary users: 336,786 Apiary API projects: 440,178
» Apiary engineers: 19
» Apiary platform engineers: 10 + 4
» Apiary SREs: 4
» App deploys: 15 (per week)
» Parsing service invocations [1 day]: ~200k
» CI build: ~19 min (8 parallel workers)
5

2014 - 2 people
software developer and ops guy
7

2015-2017 - 3 people
software developers
8

No team separation
» bounder context, but ...
» Shared ownership of platform - shared responsibility
» Shared tooling (debug, deploy, monitor)
» Shared codebase
» Brainstorm
» Motivation for good design (monitoring, future debugging)
11

Things break
» They do - better be ready
» Knowing when there's problem (logs, metrics, alerting)
» Having someone there - being oncall
» Responding (mitigation, resolution)
» Learning from it (postmortems)
12

Measure everything
» No gut feeling when we have the data (app metrics, runtime
metrics)
» Both production and non-production systems (e.g. our CI test
time)
» Thresholds, automated alerting
» Visualize the data (oncall dashboard, happiness dashboard)
13

Gradual changes
» Delivery vs deploy
» Continuous Integration / Continuous Delivery (CI/CD)
Automated testing within CI
» Testing environments (similar to production)
» Short iterations, fast rollbacks
» No-downtime deploy & immutable
» Rolling delivery
16

Tooling & automation
» oncall logistics
» schedules
» escalations
» alerting
» conflicts
» documentation
» runbooks
» internal processes
» domain dictionary
17

Reason 1. Decreasing changes of errors
» Source and great post: http://www.devops.ch/2017/05/10/devops-explained/
18

Reason 2: Eliminating toil, work that is:
» Repetitive
» Automatable
» Doesn't provide enduring value
» Scales linearly with service
» Compounds significantly and surprisingly
19

Reason 3: Focusing on creative
engineering work that:
» Improves reliability
» Improves performance & stability of systems
» Ensures scalability
» Reduces toil
» Is fun: improves morale, speeds up progress, allows skill
development
20

Incidents
Types:
» Low-priority incident
» High-priority incident
» Security incident
Both production and non-production systems
21

Being oncall
» Shared among developers (roles, not individuals, increase bus
factor) Responsible for the platform
» Safety net - you know who to call
» Runbooks - you know what to do
» Early alerting - proactively investigate
22

Incident response
If critical: Incident commander role Separate roles, if necessary:
» outbound and inbound communication
» root cause analysis
» issue mitigation
Tracking time (incident ack expiration) and keeping track Tooling
(alerts, paging, postmortem reminders)
23

Postmortems
» Root cause
» Lessons learned
» Actionable items
» Prevent future issues
» Create runbooks
» Blameless
» Generated reminders
24

Incident reviews
» Weekly, team lead sync
» Reviewing past incidents - types, occurrence, actionability
» Discuss improvements
» Incident fatigue prevention
25

Summary
» Culture is more important than process!
» Start early and work on improvements!
» Product owner for SRE work is useful role!
27

References
» SRE vs. DevOps: competing standards or close friends?
» SRE Weekly
» Awesome Site Reliability Engineering
Books
» SRE book
» Seeking SRE
28

This document discusses how small teams can get ready for Site Reliability Engineering (SRE). It describes the challenges faced by a small engineering team at a company with around 100 employees and 10 engineers. To address issues with productivity, reliability, and deployment speed, the team implemented several initiatives including adopting SCRUM, adding automated testing, simplifying deployments, and creating easy-to-use development environments. While these changes helped, the team knows there is still work needed in areas like data center operations and establishing formal SLAs and incident management processes as the company and services grow. The presentation concludes by discussing why SRE is preferable to just DevOps and provides resources for further learning.

A Crash Course in Building Site ReliabilityAcquia

From <a href="https://en.wikipedia.org/wiki/Site_reliability_engineering" target="_blank">Wikipedia</a>: Site reliability engineering (SRE) is a discipline that incorporates aspects of software engineering and applies that to operations whose goals are to create ultra-scalable and highly reliable software systems. Over the past year Acquia has built their own SRE team to help their products and services scale with the demand of our growing number of customers. We wish to share our experience so that others are enabled to do the same and reap the rewards. This presentation will discuss how the SRE team came about at Acquia, what achievements we have made so far, and the lessons we have learned along the way. We will then show the steps on how to introduce SRE to your workplace so you can deliver more reliable and scalable services to your customers! We will specifically cover: <ul> <li>SRE's basic concepts and history from Google</li> <li>The management support you will need to get started</li> <li>Introducing the idea of service level objectives and error budgets</li> <li>Operational Responsibility Assessments as a tool to measure risk</li> <li>Creating a Launch Readiness Checklist to standardize and improve product launches</li> <li>Finding ideal candidates for your SRE team</li></ul> The intended audience are software engineers, system administrators, and managers that have a desire to improve how they do their work and how their products/services perform.

Auto sre with keptnLibbySchulze

DevOps teams struggle with complex automation pipelines that are difficult to maintain, duplicated pipeline code across services, and inability to scale SRE due to manual expert tasks required for resiliency testing and SLO analysis. Keptn addresses these problems by providing data-driven delivery and operations automation based on declarative configuration, separating concerns between the application plane defining processes and control plane executing tools, and event-driven automation that removes hard-coded dependencies.

DevOps in PracticeDerek Chen

This document discusses DevOps practices at Trend Micro. It begins with an introduction of the author and his background. It then discusses why DevOps is important for Trend Micro's consumer products team, describing increased deployment frequency, stability, and productivity. The document defines DevOps as involving a culture of collaboration between development and operations teams. It emphasizes automating infrastructure provisioning and implementing a measurement and feedback culture. Quality is discussed as a key focus, with emphasis on testing earlier and enabling continuous quality. The document concludes that DevOps at Trend Micro involves applying CAMS principles: Culture, Automation, Measurement, and Sharing.

SRE vs DevOpsLevon Avakyan

SRE (service reliability engineer) on big DevOps platform running on the clou...DevClub_lv

Drive Continuous Delivery With Continuous TestingCA Technologies

Silos. Lack of visibility. Some agile teams… some not. Manual handoffs. Bottlenecks. This summer, it’s time to get outside (your old processes) and take some time off (your application release cycle). Take back your weekends and spend more time by the pool. We’ll show you how to automate, orchestrate, and facilitate continuous everything – and that includes continuous testing – one of the biggest bottlenecks of all. You’ll learn how to: Automatically shift quality left: Orchestrate and automate testing in every phase of the SDLC with automated promotion and feedback loops Accelerate testing in the cloud: Test web and mobile apps in parallel – achieve up to 10X improvement in testing time. Use tools of choice while optimizing every aspect of your complex, interdependent multi-application pipelines. Get started in less than 1 hour…. and for free! Achieve truly automated, continuous delivery (including continuous testing!!!) in the cloud with CA and Sauce Labs. Try Continuous Delivery Director free: https://cddirector.io/#/home Try Sauce Labs free: https://saucelabs.com/

10 Reasons Why You Should Consider Google App Engine (GAE) for Your Next ProjectAbeer R

DevOps & SRE at Google ScaleKaushik Bhattacharya

Google has highly optimized engineering processes developed over decades of building software at massive scale. They use practices like continuous integration/delivery, automated testing of all code changes, containerization, and Site Reliability Engineering. Much of Google's internal tools like Kubernetes, Tensorflow, and Borg that manage these processes are now available publicly on Google Cloud Platform. Migrating to Google Cloud allows companies to leverage the same infrastructure Google uses to build software securely and reliably at large scale.

DevOps as-a-Service (DaaS) valueMarc Hornbeek

DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOpsGene Kim

This document outlines the top 10 ways to fail at DevOps. It discusses failing to include management buy-in, becoming too reliant on open source software, and failing to consider IT history. It also notes that DevOps should not mean "DevNo-Ops" and that the same automation used in development and testing environments should also be applied to production, with the production environment in mind from the beginning. Centralizing DevOps and thinking failures in production are acceptable are also outlined as ways to fail at DevOps. The document provides context and examples for each of the top 10 ways.

Data-Driven DevOps: Improve Velocity and Quality of Software Delivery with Me...Splunk

Much of the value of DevOps comes from a (renewed) focus on measurement, sharing, and continuous feedback loops. In increasingly complex DevOps workflows and environments, and especially in larger, regulated, or more crystallized organizations, these core concepts become even more critical. This session will show how, by focusing on 'metrics that matter,' you can provide objective, transparent, and meaningful feedback on DevOps processes to all stakeholders. Learn from real-life examples how to use the data generated throughout application delivery to continuously identify, measure, and improve deployment speed, code quality, process efficiency, outsourcing value, security coverage, audit success, customer satisfaction, and business alignment.

You got a couple Microservices, now what? - Adding SRE to DevOpsGonzalo Maldonado

Continuous testing webinar 041017 slideshareQualiQuali

Modern release management teams pride themselves on setting up a seamless workflow for continuous integration and delivery. However, continuous testing – which is one of the most critical components of the workflow is often taken for granted or marginalized without clear ownership leading to impediments in quality. With the advent of DevOps and the movement to break down silos between developers and operations, it becomes critically important that all members of an IT team - regardless of what tools they use, or role they play - understand the essentials of continuous testing.

What's an SRE at Criteo - Meetup SRE ParisClément Michaud

This document summarizes the role of a Site Reliability Engineer (SRE) at Criteo. It discusses how Criteo embraced a DevOps philosophy by breaking down barriers between development and operations teams. SREs at Criteo work in small, specialized teams to maintain infrastructure and platforms while also providing support, automation, and on-call responsibilities. Their goal is to enable development teams through ownership of the services they provide and collaboration across organizational boundaries.

SRE in Enterprise - Local Journey DevopsDays GalwayKevin Connaughton

Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...Mike Villiger

SRE 101 (Site Reliability Engineering)Hussain Mansoor

This document provides an introduction to Site Reliability Engineering (SRE). It discusses DevOps principles and how SRE relates to and implements DevOps. Key aspects of SRE covered include guiding principles like eliminating toil, embracing risk, and measuring services through SLIs, SLOs, and error budgets. Specific SRE practices mentioned are removing toil, defining system criticalities, designing for availability, observability, chaos engineering, restricting production access, and focusing on metrics like MTTR and MTBF.

DevOps Testing | Continuous Testing In DevOps | DevOps Tutorial | DevOps Trai...Edureka!

***** DevOps Masters Program : https://www.edureka.co/masters-progra... ***** This tutorial on DevOps testing will help you understand how Continuous Testing takes place in the DevOps lifecycle and which tools are used for the same. The following topics have been covered in this video: 1. What Is Continuous Testing? 2. Various Testing Types 3. Tools Used For Continuous Testing 4. Demo: Maven, Selenium, TestNG & Jenkins Integration

Roles and Responsibilities of a DevOps EngineerZaranTech LLC

DevOps Training & Certification provided Online from USA industry expert trainers with real time project experience. COURSE PAGE: https://www.zarantech.com/devops-certification-training/ REGISTER FOR FREE LIVE DEMO: http://promo.zarantech.com/free-webinar-devops CONTACT: +1 (515) 309-7846 (or) Email - [email protected] Get More Free Videos - Subscribe ➜ https://goo.gl/5ZqDML "DevOps tutorial" "free DevOps training" "online DevOps training" "Best DevOps training" "DevOps for Beginners" "Best DevOps Training" Reviews / Testimonials from past trainees are saying: https://goo.gl/ZVfnE4 Recommendations on Facebook - http://www.facebook.com/ZaranTechLLC Testimonials on our website - http://www.zarantech.com/testimonials Refer your friends to ZaranTech - http://www.zarantech.com/be-a-friend-tell-a-friend.

Bjorn Rabenstein. SRE, DevOps, Google, and youIT Arena

Bjorn Rabenstein, Production Engineer at SoundCloud SRE, DevOps, Google, and you Site Reliability Engineering (SRE) was originally conceived internally at Google. By now, it has become public knowledge via various channels like conferences or books. But how can you apply SRE principles in your organization, given that you are not Google and cannot just blindly do everything exactly as Google does? And how does SRE relate to DevOps, which you might or might not have indulged in already? The speaker has seen both sides, with many years working as an SRE at Google and later as a Production Engineer at SoundCloud, a much smaller startup running many service using a highly innovative tech stack and a radical DevOps approach. Let’s dive into questions of culture and scale and come up with some helpful pointers how you can learn from the giant without losing you own way. Björn Rabenstein is a Production Engineer at SoundCloud and a Prometheus developer. Previously, Björn was a Site Reliability Engineer at Google and a number cruncher for science.

The Art of Container MonitoringDerek Chen

According to service scale, there are hundreds or thousands of running containers in your service. Should we monitor each container by microscope or monitor each microservice by magnifier? This depends which granularity can help us find and solve the problems. In this sharing, I will introduce how to use cAdvisor, Icinga2, InfluxDB and Grafana to build a self-hosted monitoring system. In addition, I also discuss with how to embrace open source and share some practical experiences.

DevOps Evolution - The Next Generation ?Marc Hornbeek

2017 DevSecOps SurveySonatype

More organizations are adopting mature DevOps practices, with 26% having mature practices and 41% improving. Those with mature practices are more likely to automate security testing in their CI/CD pipelines. While container security is a top concern, many organizations may not have the necessary governance policies for managing open source components, which pose a growing risk of security breaches.

How to Build the Right AutomationJules Pierre-Louis

Many companies are investing heavily in automation. Good high quality automation is key as companies move towards a successful DevOps model. The problem is that automation scripts can be very brittle and tend to not cover or test the entire application. They are also very difficult and time consuming to keep up to date. This presentation will include a demonstration of how to design, create and update automation scripts as well as their associated test data and end points. On this webcast learn how to make automated testing a reality.

QA in DevOps: Transformation thru Automation via JenkinsTatyana Kravtsov

This document outlines the agenda for a Jenkins World Tour 2015 presentation in Washington D.C. on QA in DevOps through automation using Jenkins. The presentation discusses the definition of DevOps and provides a 10 step process to DevOps transformation focusing on continuous integration, automated testing, code quality metrics, environment testing, and automated reporting. The presenter is Tanya Kravtsov, founder of the DevOpsQA NJ Meetup group.

Continuous Delivery DistilledMatt Callanan

Matt Callanan takes the 15 chapters of the famous "Continuous Delivery" book by Jez Humble & Dave Farey and distills it down into 1 hour of convincing arguments, walking through the pieces involved to make it happen including cultural challenges, automated testing, automated deployment & deployment pipelines. Not sure how to get started with DevOps? Finding it hard to convince colleagues & managers that CD is the way forward? Matt has used this presentation to help facilitate enterprise-wide adoption of Continuous Delivery. Slides from a presentation given at DevOps Brisbane March 2014.

The Next Wave of Reliability EngineeringMichael Kehoe

In 2018, Site Reliability Engineering (SRE) will turn 15 years old. Since Google's inception of the term SRE, companies across the world have adopted a new operations mindset along with automation, deployment and monitoring principals. Most of what SRE does now is well established throughout the industry, so what is the next-wave of reliability principals and automation frameworks? This session will dive into what the future holds for reliability engineering as a field and what will be the next areas of investment and improvement for reliability teams.

Software engineering the genesisPawel Szulc

The document discusses the origins of software engineering as a discipline. It summarizes discussions from a conference in 1968 where the term "software engineering" was first used. Key points discussed included that testing is best done iteratively during design rather than after, that small groups tend to be more successful than large groups on software projects, and that an organizational structure is needed for communication and decision making in large groups. The document also discusses criticisms of the "waterfall" development model and advocates for an iterative approach.

Create Your Future with z Systems CloudCA Technologies

The document discusses how the mainframe remains relevant despite predictions that it would become obsolete. It provides examples of how the mainframe continues to process a large volume of transactions and data efficiently. It then summarizes IBM's DevOps solutions for modernizing mainframe development and discusses how technologies like blockchain, analytics and APIs can help expose mainframe capabilities and drive innovation.

More Related Content

What's hot (20)

DevOps & SRE at Google ScaleKaushik Bhattacharya

DevOps as-a-Service (DaaS) valueMarc Hornbeek

DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOpsGene Kim

Data-Driven DevOps: Improve Velocity and Quality of Software Delivery with Me...Splunk

You got a couple Microservices, now what? - Adding SRE to DevOpsGonzalo Maldonado

Continuous testing webinar 041017 slideshareQualiQuali

What's an SRE at Criteo - Meetup SRE ParisClément Michaud

SRE in Enterprise - Local Journey DevopsDays GalwayKevin Connaughton

Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...Mike Villiger

SRE 101 (Site Reliability Engineering)Hussain Mansoor

DevOps Testing | Continuous Testing In DevOps | DevOps Tutorial | DevOps Trai...Edureka!

Roles and Responsibilities of a DevOps EngineerZaranTech LLC

Bjorn Rabenstein. SRE, DevOps, Google, and youIT Arena

The Art of Container MonitoringDerek Chen

DevOps Evolution - The Next Generation ?Marc Hornbeek

2017 DevSecOps SurveySonatype

How to Build the Right AutomationJules Pierre-Louis

QA in DevOps: Transformation thru Automation via JenkinsTatyana Kravtsov

Continuous Delivery DistilledMatt Callanan

The Next Wave of Reliability EngineeringMichael Kehoe

DevOps & SRE at Google ScaleKaushik Bhattacharya

DevOps as-a-Service (DaaS) valueMarc Hornbeek

DOES SFO 2016 - Scott Willson - Top 10 Ways to Fail at DevOpsGene Kim

Data-Driven DevOps: Improve Velocity and Quality of Software Delivery with Me...Splunk

You got a couple Microservices, now what? - Adding SRE to DevOpsGonzalo Maldonado

Continuous testing webinar 041017 slideshareQualiQuali

What's an SRE at Criteo - Meetup SRE ParisClément Michaud

SRE in Enterprise - Local Journey DevopsDays GalwayKevin Connaughton

Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...Mike Villiger

SRE 101 (Site Reliability Engineering)Hussain Mansoor

DevOps Testing | Continuous Testing In DevOps | DevOps Tutorial | DevOps Trai...Edureka!

Roles and Responsibilities of a DevOps EngineerZaranTech LLC

Bjorn Rabenstein. SRE, DevOps, Google, and youIT Arena

The Art of Container MonitoringDerek Chen

DevOps Evolution - The Next Generation ?Marc Hornbeek

2017 DevSecOps SurveySonatype

How to Build the Right AutomationJules Pierre-Louis

QA in DevOps: Transformation thru Automation via JenkinsTatyana Kravtsov

Continuous Delivery DistilledMatt Callanan

The Next Wave of Reliability EngineeringMichael Kehoe

Similar to SRE in Apiary (20)

Software engineering the genesisPawel Szulc

Create Your Future with z Systems CloudCA Technologies

Keynote at-icpc-2020Ralf Laemmel

Developer workflow analysis and ownership management present comprehension challenges for software ecosystems and global software engineering. Dark matter exists because tools are not fully integrated, logging is not designed for analysis, and developer workflow is unstructured. Probabilistic models using machine learning and heuristics can help associate activities with work items to address this. Ownership management challenges include ownership decay, asset subclassing, team-level ownership, and providing explainable recommendations.

Software Analytics:Towards Software Mining that Matters (2014)Tao Xie

This document discusses software analytics and summarizes several related papers and projects. It introduces Software Analytics, which aims to enable software practitioners to perform data exploration and analysis to obtain useful insights. It then summarizes papers on techniques for performance debugging by mining stack traces, scalable code clone analysis, incident management for online services, and using games to teach programming.

Creating An Incremental Architecture For Your SystemGiovanni Asproni

Please, Please, PLEASE Defend Your Mobile Apps!Jerod Brennen

The document discusses how to strengthen the security of mobile apps. It recommends conducting source code reviews, security testing apps during QA, and analyzing deployed apps. It provides examples of security checks like reviewing for vulnerabilities and threats. The document also shares tools for analyzing iOS and Android apps, such as reverse engineering toolkits and decompiling APK files. Resources are listed for tasks like monitoring network traffic and examining app databases and files.

DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an A...Gene Kim

DevOps in the Midst of an Airline Merger Susanna Brown, Managing Director Operations Technology, American Airlines Ben Chan, Director Shared Services, American Airlines Description: DevOps as a cultural change agent to bring enterprise/federated, infrastructure/development, employees/vendors together, while merging two major airlines. DevOps as a cultural change agent to bring enterprise/federated, infrastructure/development, employees/vendors together, while merging two major airlines. DevOps Enterprise Summit San Francisco 2016

Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINALAlex Tarra

This document discusses the principles and practices of Extreme Programming (XP), an agile software development methodology. It describes XP's emphasis on communication, rapid feedback cycles, incremental changes, and automated testing. The document outlines specific XP practices related to planning, designing, testing, and coding software. It also addresses criticisms of XP and compares it to traditional waterfall development methods. Overall, the document provides an overview of XP and argues that its lightweight approach can enable better results for rapid development projects compared to lack of methodology or strict waterfall practices.

Illogical engineersPawel Szulc

The document discusses the history of software engineering and early ideas in the field. It notes that the term "software engineering" was coined in 1968 to imply a more rigorous, engineering-based approach was needed for software development. Early conferences discussed ideas like integrating testing into design instead of after, and that small teams tend to be more successful than large ones. The document also examines some of the challenges faced and lessons learned over decades of software development.

Illogical engineersPawel Szulc

Microservices Architecture - Cloud Native AppsAraf Karsh Hamid

This document provides an overview of microservices architecture, including concepts, characteristics, infrastructure patterns, and software design patterns relevant to microservices. It discusses when microservices should be used versus monolithic architectures, considerations for sizing microservices, and examples of pioneers in microservices implementation like Netflix and Spotify. The document also covers domain-driven design concepts like bounded context that are useful for decomposing monolithic applications into microservices.

Software Analytics - Achievements and ChallengesTao Xie

Measuring Technical Lag in Software Deployments (CHAOSScon 2020)Tom Mens

Agile and Agile methods: what is the most important to understand to succeedVaidas Adomauskas

This document discusses Agile methods and how to succeed using them. It begins with an overview of traditional "waterfall" software development practices versus Agile practices. It then defines some common Agile terms and methods like Scrum, Test-Driven Development, eXtreme Programming, and Kanban. The document emphasizes that the most important things for success are respecting people, using the right tools for the job, measuring the right metrics, getting external help when needed, having courage, and starting to use Agile practices now rather than later.

Right-sized Architecture: Integrity for Emerging DesignsTechWell

In agile projects, design ideally "emerges" over the course of development. However, if teams primarily focus on independent user stories, they risk losing sight of the product's vision and the integrity of well-thought-out architecture. Ken Kubo shares techniques he's used to improve the chances that a product's design will emerge into a cohesive and coherent architecture that serves its customers for many years. Join Ken to find out how you can incorporate contextual design principles and simple, visual techniques as part of his "A-Little-Before-Its-Time Design" framework. You can add these practices into your agile workflow to maintain a shared team understanding of your product's vision and the system's emerging design. Ken believes that you can only realize all the promises of agile development with a clearly and constantly communicated product vision and a set of architecture goals. Lack of these key principles leads to sub-optimizing system development-or much worse, failure.

Cytoscape CI Chapter 2bdemchak

.NET Fest 2019. Леонид Молотиевский. DotNet Core in productionNETFest

Во время доклада, я поделюсь с Вами опытом, который мы получили, используя микросервисы в прод K8S кластере. Также, обозначу основные проблемы, с которыми столкнулась наша команда на этапе их диагностики. И, самое главное - что мы сделали чтобы избежать их в будущем. Отвечу на вопросы: Почему мы мигрировали в облако? Почему dotNet Core 2.2 вызвал кучу проблем? Данный доклад сохранит сотни часов вашим разработчикам и DevOps команде, жизнь которой может напоминать кошмар.

Software Mining and Software DatasetsTao Xie

This document discusses software mining and datasets. It begins by introducing Tao Xie and his research group at the University of Illinois which focuses on software analytics. It then discusses different types of software services and data, how data has become more pervasive, and challenges in making repositories more actionable. Key topics in software analytics research are discussed including the goal of enabling insights for practitioners. Examples of mined information from different repository types like source code, bug reports, and mailing lists are provided.

Software Analytics: Data Analytics for Software Engineering and SecurityTao Xie

Frodo Baggins presents on software analytics for software engineering and security tasks. The presentation discusses how software and how it is built and used is changing, with data now being ubiquitous and software having continuous development and release. Software analytics aims to enable software practitioners to perform data exploration and analysis to obtain useful insights. Examples of software analytics techniques discussed include XIAO for scalable code clone analysis, and SAS for incident management of online services. The presentation then shifts to discussing software analytics techniques for mobile app security, including WHYPER for natural language processing on app descriptions to link permissions to functionality, and AppContext for machine learning to classify malware.

DevOps – what is it? Why? Is it real? How to do it?Sailaja Tennati

Software engineering the genesisPawel Szulc

Create Your Future with z Systems CloudCA Technologies

Keynote at-icpc-2020Ralf Laemmel

Software Analytics:Towards Software Mining that Matters (2014)Tao Xie

Creating An Incremental Architecture For Your SystemGiovanni Asproni

Please, Please, PLEASE Defend Your Mobile Apps!Jerod Brennen

DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an A...Gene Kim

Jun 08 - PMWT Featured Paper -Tarabykin - XP PAPER - FINALAlex Tarra

Illogical engineersPawel Szulc

Microservices Architecture - Cloud Native AppsAraf Karsh Hamid

Software Analytics - Achievements and ChallengesTao Xie

Measuring Technical Lag in Software Deployments (CHAOSScon 2020)Tom Mens

Agile and Agile methods: what is the most important to understand to succeedVaidas Adomauskas

Right-sized Architecture: Integrity for Emerging DesignsTechWell

Cytoscape CI Chapter 2bdemchak

.NET Fest 2019. Леонид Молотиевский. DotNet Core in productionNETFest

Software Mining and Software DatasetsTao Xie

Software Analytics: Data Analytics for Software Engineering and SecurityTao Xie

DevOps – what is it? Why? Is it real? How to do it?Sailaja Tennati

More from Ladislav Prskavec (20)

Modern Web Architecture based on JS, API and MarkupLadislav Prskavec

The document discusses modern web architecture using JavaScript, APIs and markup. It focuses on static site generators like Gatsby.js and how they can be used to build sites sourced from headless CMS APIs or large CMSs like WordPress and Drupal. The key benefits of a JAMstack architecture are also covered, such as performance, scalability, security and developer experience. Examples of sites built with Gatsby.js are shown and the speaker demonstrates building a simple site with Gatsby.js.

How you can kill Wordpress!Ladislav Prskavec

SRE in StartupLadislav Prskavec

SRE (Site Reliability Engineering) is responsible for availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of services. An SRE team uses an "error budget" approach where new features can be launched if the service is within its agreed SLA, but launches are frozen if the SLA is not being met until enough of the error budget is earned back. SRE teams hire only coders who can speak the same language as developers and rotate developers into operations work. The goal of SRE is to minimize impact and prevent recurrence of outages through practices like post-mortem analysis and constant improvement of processes.

CI and CDLadislav Prskavec

Continuous integration (CI) involves developers integrating code into a shared repository multiple times per day, with each check-in being verified by automated builds to detect problems early. Continuous delivery (CD) means software can be released to production at any time. Continuous deployment means every change automatically gets deployed to production, resulting in many releases per day. Apiary uses continuous integration engines (CIEs) that support job definitions in repositories, autoscaling, pipelines, caching, Docker, matrix builds, and distributing jobs across multiple nodes to achieve CI and CD.

Datascript: Serverless ArchitetectureLadislav Prskavec

This document discusses serverless architecture and AWS Lambda. It provides information on Backend as a Service (BaaS), Function as a Service (FaaS), Amazon Lambda, Lambda configuration, costs, frameworks like Chalice and Serverless, libraries, security, testing, reference architectures, and challenges. Key topics covered include AWS Lambda configuration for inputs, processing, and outputs using various languages, the AWS Serverless Application Model for defining applications, and Amazon States Language for orchestrating functions.

Serverless ArchitectureLadislav Prskavec

This document discusses serverless architecture and frameworks. It describes backend as a service (BaaS) and function as a service (FaaS), with examples like Amazon Lambda, Google Cloud Functions, and Azure Functions. Frameworks for serverless apps are presented, including Apex, Serverless, Zappa, and Sparta. The document also covers security, testing, example architectures, and resources for serverless development.

CI and CDLadislav Prskavec

CI and CD processes like Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (CD) help software teams integrate code changes frequently and release software quickly. CI involves integrating code into a shared repository multiple times daily and automatically verifying each change. Continuous Delivery sets up software to be released to production at any time. Continuous Deployment means every code change is automatically deployed to production, allowing for many releases per day. Key aspects of CI include job definitions in repositories, autoscaling infrastructure, caching dependencies, using Docker for builds, and matrix builds across different environments and versions.

PragueJS meetups 30th anniversaryLadislav Prskavec

How to easy deploy app into any cloudLadislav Prskavec

The document discusses how to easily deploy applications into any cloud using Docker containers. It provides an overview of Docker, including how it allows building applications once and running them anywhere without recompilation. Examples are given of using Docker for development, continuous integration, and deploying and managing applications across clouds with tools like Docker Compose, Docker Machine, and Docker Swarm.

Docker - modern platform for developement and operationsLadislav Prskavec

The document discusses Docker, a platform for development and operations. It provides an overview of Docker including what Docker is, how Docker containers work through an analogy to logistics, and the benefits of Docker such as being fast, scalable, and enabling building applications once to run anywhere. The document also covers popular Docker tools like Docker Compose and Docker Machine as well as how Docker is used on major cloud platforms like Amazon Web Services, Microsoft Azure, and Google Cloud.

GDGSCL - Docker a jeho provoz v Heroku a AWSLadislav Prskavec

AWS Elastic Container ServiceLadislav Prskavec

Ladislav Prskavec is presenting on AWS Elastic Container Service (ECS). ECS is a highly scalable container management service that supports Docker containers and allows running applications on a managed cluster of EC2 instances without needing to install and manage your own cluster infrastructure. The key components of ECS include clusters, container instances, task definitions, schedulers, services, and tasks. Ladislav provides examples of using the AWS CLI to create an ECS cluster, run EC2 instances as container instances, define a task, create a service, and list/describe services.

Comparison nodejs frameworks using Polls APILadislav Prskavec

Docker Elastic BeanstalkLadislav Prskavec

This document discusses Docker and AWS Elastic Beanstalk. It provides information about the speaker, Ladislav Prskavec, including that he is an SRE at Apiary and organizer of PragueJS. It also discusses AWS regions, the eb CLI, Elastic Beanstalk platforms including Docker versions, and the Dockerrun.aws.json file. An example is provided of using eb create to launch an APIary Docker development environment on Elastic Beanstalk.

Docker včera, dnes a zítraLadislav Prskavec

Ladislav Prskavec gave a presentation on Docker at the Devel Conference on April 11, 2015. He discussed his roles as a sysadmin, developer, and tech writer. He then explained what Docker is and showed trends in Google searches and interest at previous Devel conferences. He demonstrated using Docker Compose to run multi-container applications and introduced Sinopia for private NPM registry. He closed by taking any questions and providing information on open jobs at Apiary.

Tessel is a microcontroller that runs JavaScript.Ladislav Prskavec

Docker.ioLadislav Prskavec

AngularJSLadislav Prskavec

This document provides an overview of AngularJS including reasons to use it, tools for building AngularJS applications, libraries that integrate with AngularJS, testing frameworks, and additional resources. It lists generators, build tools, and libraries for mobile development. It also mentions code playgrounds, testing frameworks like Karma and Protractor, and topics covered in other AngularJS resources.

Firebase and AngularJSLadislav Prskavec

This document discusses using Firebase as a backend solution for AngularJS applications. Firebase is a realtime database that allows storing and syncing data between clients and servers. It offers features like offline support, flexible data storage using JSON, and authentication integration. The document provides links to Firebase documentation on its REST API, integrations with other services, security rules, and tutorials for using Firebase with AngularJS applications.

Modern Web Architecture based on JS, API and MarkupLadislav Prskavec

How you can kill Wordpress!Ladislav Prskavec

SRE in StartupLadislav Prskavec

CI and CDLadislav Prskavec

Datascript: Serverless ArchitetectureLadislav Prskavec

Serverless ArchitectureLadislav Prskavec

CI and CDLadislav Prskavec

PragueJS meetups 30th anniversaryLadislav Prskavec

How to easy deploy app into any cloudLadislav Prskavec

Docker - modern platform for developement and operationsLadislav Prskavec

GDGSCL - Docker a jeho provoz v Heroku a AWSLadislav Prskavec

AWS Elastic Container ServiceLadislav Prskavec

Comparison nodejs frameworks using Polls APILadislav Prskavec

Docker Elastic BeanstalkLadislav Prskavec

Docker včera, dnes a zítraLadislav Prskavec

Tessel is a microcontroller that runs JavaScript.Ladislav Prskavec

Docker.ioLadislav Prskavec

AngularJSLadislav Prskavec

Firebase and AngularJSLadislav Prskavec

Recently uploaded (20)

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Procurement Insights Cost To Value Guide.pptxJon Hansen

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Cyber Awareness overview for 2025 month of securityriccardosl1

tecnologias de las primeras civilizaciones.pdffjgm517

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

How analogue intelligence complements AIPaul Rowe

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Procurement Insights Cost To Value Guide.pptxJon Hansen

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Cyber Awareness overview for 2025 month of securityriccardosl1

tecnologias de las primeras civilizaciones.pdffjgm517

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

SRE in Apiary

1. SRE in Apiary CZJUG 21.5.2018 Ladislav Prskavec @abtris 1

2. What is SRE? 2

3. "What happens when a software engineer is tasked with what used to be called operations." » Ben Treynor Sloss, Vice President, Google Engineering, founder of Google SRE 3

4. SRE implement DevOps — Google Cloud 4

5. Apiary in numbers » Apiary users: 336,786 Apiary API projects: 440,178 » Apiary engineers: 19 » Apiary platform engineers: 10 + 4 » Apiary SREs: 4 » App deploys: 15 (per week) » Parsing service invocations [1 day]: ~200k » CI build: ~19 min (8 parallel workers) 5

6. How we started with SRE team 6

7. 2014 - 2 people software developer and ops guy 7

8. 2015-2017 - 3 people software developers 8

9. 2018 - 4 people 2 seniors, 2 juniors 9

10. Culture Process > Tools 10

11. No team separation » bounder context, but ... » Shared ownership of platform - shared responsibility » Shared tooling (debug, deploy, monitor) » Shared codebase » Brainstorm » Motivation for good design (monitoring, future debugging) 11

12. Things break » They do - better be ready » Knowing when there's problem (logs, metrics, alerting) » Having someone there - being oncall » Responding (mitigation, resolution) » Learning from it (postmortems) 12

13. Measure everything » No gut feeling when we have the data (app metrics, runtime metrics) » Both production and non-production systems (e.g. our CI test time) » Thresholds, automated alerting » Visualize the data (oncall dashboard, happiness dashboard) 13

14. 14

15. 15

16. Gradual changes » Delivery vs deploy » Continuous Integration / Continuous Delivery (CI/CD) Automated testing within CI » Testing environments (similar to production) » Short iterations, fast rollbacks » No-downtime deploy & immutable » Rolling delivery 16

17. Tooling & automation » oncall logistics » schedules » escalations » alerting » conflicts » documentation » runbooks » internal processes » domain dictionary 17

18. Reason 1. Decreasing changes of errors » Source and great post: http://www.devops.ch/2017/05/10/devops-explained/ 18

19. Reason 2: Eliminating toil, work that is: » Repetitive » Automatable » Doesn't provide enduring value » Scales linearly with service » Compounds significantly and surprisingly 19

20. Reason 3: Focusing on creative engineering work that: » Improves reliability » Improves performance & stability of systems » Ensures scalability » Reduces toil » Is fun: improves morale, speeds up progress, allows skill development 20

21. Incidents Types: » Low-priority incident » High-priority incident » Security incident Both production and non-production systems 21

22. Being oncall » Shared among developers (roles, not individuals, increase bus factor) Responsible for the platform » Safety net - you know who to call » Runbooks - you know what to do » Early alerting - proactively investigate 22

23. Incident response If critical: Incident commander role Separate roles, if necessary: » outbound and inbound communication » root cause analysis » issue mitigation Tracking time (incident ack expiration) and keeping track Tooling (alerts, paging, postmortem reminders) 23

24. Postmortems » Root cause » Lessons learned » Actionable items » Prevent future issues » Create runbooks » Blameless » Generated reminders 24

25. Incident reviews » Weekly, team lead sync » Reviewing past incidents - types, occurrence, actionability » Discuss improvements » Incident fatigue prevention 25

26. Summary 26

27. Summary » Culture is more important than process! » Start early and work on improvements! » Product owner for SRE work is useful role! 27

28. References » SRE vs. DevOps: competing standards or close friends? » SRE Weekly » Awesome Site Reliability Engineering Books » SRE book » Seeking SRE 28

SRE in Apiary

Recommended

More Related Content

What's hot (20)

Similar to SRE in Apiary (20)

More from Ladislav Prskavec (20)

Recently uploaded (20)

SRE in Apiary