Monitoring "unknown unknowns" - Guy Fighel - DevOpsDays Tel Aviv 2017

Nov 16, 2017Download as PPTX, PDF0 likes396 views

This document discusses monitoring systems for unknown unknowns using machine intelligence. It suggests that observability goes beyond just monitoring and instrumentation to make systems debuggable and understandable. It outlines several techniques machines could use like anomaly detection, predictions, external knowledge, recommendations, and auto-correlations. The goal is to develop an "observability quadrant" like the Johari window to better understand systems with imperfect outputs. It recommends choosing the right tools, applying common schemas to event data, using techniques like fuzzy logic and similarity checks, and focusing first on simple correlations and time series models before building complex classifiers. The best practices are to start using such techniques in production by getting feedback and continually re-evaluating relevance.

Monitoring “unknown unknowns”
With Machine Intelligence

@guyfig
On-Call Engineer by Nature
"If a tree falls in a forest and no one is
around to hear it, does it make a sound?"

Observability is a superset between
monitoring and instrumentation.
Making systems debuggable and
understandable
@mipsytipsy
Do you really know what to observe?
Instrumentation - mostly Developer driven
What is the output? Dashboard? Exploration tool?

one can determine the behavior of the
entire system from the system's outputs
Observability In Control Theory

-Static thresholds
-Defined Alerts
-Static Runbooks
-Anomaly Detection
-Predictions
-External Knowledge
-Knowledge
-Recommendations
-Auto Collaboration
-Inference
-Auto Correlations
-Semantic Analysis
-Decision making
The Observability Quadrant (Based on Johari window)

Humans Driven Detection
Set thresholds to find patterns
Simulate based on known
Use percentiles, basic stats

Monitoring "unknown unknowns" - Guy Fighel - DevOpsDays Tel Aviv 2017

Will That Help in a “Fire-Fighting” Mode?

Find The Problem
Thresholds? Baseline? Anomaly?
- Scale matters
- Stationary noise matters
- Use Autocorrelation

Preprocessing Data
sklearn.preprocessing

Independent component analysis (ICA)
separates a multivariate signal into
additive subcomponents that are
maximally independent.
from sklearn.decomposition
import FastICA, PCA

Find The Problem
CPU
90%
Time in Minutes EC2 Instance
changed from
t2.small to m3.xl
Events & context matters
Anomly?

What Can Machines Do?
Process different types of data, transform it fast and handle huge amounts in real-time
Automate and adapt Anomaly Detection
Apply Semantic text similarities to find patterns (Information Retrieval)
Apply auto correlation models
Evolve and adapt (overtime) based on human interaction

The Goal - Centralization
Observability for systems with imperfect outputs
Events enrichments, symptoms detection and inference
Automatic Outlier Detection
Automatic Correlation
Get closer to the Control Theory mathematical definition

Pick The Right Tools
https://github.com/turi-code/SFrame

- Define the model. Use a single schema (Apache Avro)
- Events are agnostic. Can represent logs, stack trace, metric, user action, HTTP event,
etc.
- Every event should have a set of common fields as well as optional key/value
attributes
Get a Common SchemaUse Common Schema

Deterministic models are better to start with (Fuzzy Logic, Rules)
Choose your logic and start run it across your data (schema)
Apply similarity checks to strings first (TF-IDF, BM25, Fuzzy, other classifiers)
Look into correlations, start with simple obvious ones, before building classifiers
(Unsupervised/Semi-supervised learning is much more relevant overall)
Build your prediction models on time series data first. (Statistics has solid models)
Time and context are dimensions you will be able to start addressing
Best Practices

Use It In Production
- Your team == your users
- Ask for feedback
- Re-calculate relevancy
- Apply Recommendations
based on your own team
knowledge

This is my presentation at Monitorama PDX in Portland on May 5, 2014 Simple math to get some signal out of your noisy sea of data You’ve instrumented your system and application to the hilt. You can now “measure all the things”. Your team has set up thousands of metrics collecting millions of data points a day. Now what? Most IT ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this mountain of data and extracting signal from the noise is not easy. The choice of what analytic method to use ranges from simple statistical analysis to sophisticated machine learning techniques. And one algorithm doesn’t fit all data.

Approach AI assuranceAviral Srivastava

The document discusses various techniques for testing machine learning systems to ensure higher accuracy, reduced bias, strong generalization, and security. It recommends identifying the intent of tasks and checking training data and algorithms to maximize accuracy while reducing resources. Feature selection and dimensionality reduction can improve performance. Testing for unwanted bias, generalization on unseen data, endpoint response times, and attacks on sensors, data, or peripherals are suggested to validate security and safety critical systems. As IoT connections increase, compatibility and end-to-end scenario testing will be important for AI assurance.

PromanVan Dolino

A simulation is a representation of a system that can be manipulated to study the system's behavior. A model is created that portrays the key aspects of the system. The model's operation provides insights into the actual system. Simulations can be continuous, discrete, or combined. They are used to predict outcomes, understand processes, identify issues, evaluate alternatives, and gain insights without experimenting on the real system, which may be impossible, too expensive, or impractical.

Not Your Mommas Unit Tests - Parameterized Unit TestsBill Shelton

capture-recapture Single DefectJames Orr

This document provides supplemental data to support a previous post about using capture-recapture concepts for software inspections with a single defect. It describes the motivation and background for the research, including a reference study. It then explains that the data is from a simulated study recreating inspections with 3 to 6 inspectors and a single defect. The document presents results from 3 simulations showing the probability of one or more inspectors finding the single defect and the probability of two or more inspectors finding it, grouped by inspection effectiveness.

Pharmacokinetic pharmacodynamic modelingMeghana Gowda

Pharmacokinetic-pharmacodynamic modeling involves creating mathematical models to represent biological systems. These models use experimentally derived data and can be classified as either models of data or models of systems. Models of data require few assumptions, while models of systems are based on physical principles. The model development process involves analyzing the problem, collecting data, formulating the model, fitting the model to data, validating the model, and communicating results. Model validation assesses how well a model serves its intended purpose, though models can never be fully proven and are disproven through validity testing.

JPJ1425 Security Evaluation of Pattern Classifiers under Attackchennaijp

We are good IEEE java projects development center in Chennai and Pondicherry. We guided advanced java technologies projects of cloud computing, data mining, Secure Computing, Networking, Parallel & Distributed Systems, Mobile Computing and Service Computing (Web Service). For More Details: http://jpinfotech.org/final-year-ieee-projects/2014-ieee-projects/java-projects/

IEEE 2014 JAVA DATA MINING PROJECTS Security evaluation of pattern classifier...IEEEFINALYEARSTUDENTPROJECTS

A culture of Automation - Joe Smith - DevOpsDays Tel Aviv 2017DevOpsDays Tel Aviv

The document discusses strategies for running production services, including careful planning, extensive documentation, and good communication. It emphasizes establishing runbooks that document operational tasks and procedures. Over time, as teams grow and work expands, runbooks should be automated through scripts and tools to reduce manual effort and potential errors. Automating runbooks involves initially writing scripts for small tasks and evolving them through testing, bug fixing, and incorporating additional workflow steps. Fully automated runbooks improve reliability and allow work to be completed through execution of instructions rather than human judgement.

Building High Scale Systems Using Serverless Architecture for a Burning Man E...DevOpsDays Tel Aviv

The document describes how Midburn, an annual Burning Man event in Israel, redesigned their ticket sales system to handle high volumes of traffic. The initial system crashed under the load of 5000 concurrent users. They rebuilt the system using Heroku and Redis, but it still crashed when users refreshed. For 2017, they implemented a serverless architecture using Amazon API Gateway and Kinesis to create a scalable queueing and sales system. During testing, the new system handled over 3000 users, 300 purchases, and 3 million requests in 15 minutes without any errors. This ensured technical issues would not prevent people from purchasing tickets.

Leading DevOps Teams - Ben Rockwood - DevOpsDays Tel Aviv 2017DevOpsDays Tel Aviv

This document discusses how to lead DevOps teams by focusing on five key areas: diversity, vision, strategy, management, and tools. It emphasizes that culture emerges from these factors. Diversity of thought and new ideas are important for innovation. A shared vision and aligned strategy allow individuals to rally around a cause and unlock passion and energy. Management involves directing individuals and assets to execute the strategy. Tools should continuously improve organizational processes. The employee lifecycle is also examined, noting the importance of attraction, recruitment, onboarding, development and retention at each stage.

Securing your code when you don't even know where it is - Liz Rice - DevOpsDa...DevOpsDays Tel Aviv

This document discusses securing containers and microservices in modern DevOps environments. It notes that with containers and infrastructure as code, applications are deployed as "cattle" through automated pipelines rather than as individually managed "pets". This distribution of code across many containers and hosts increases security risks. The document recommends reducing image sizes, segmenting microservice networks, adding runtime protections, and embedding security checks into continuous integration/deployment pipelines to address these risks. It concludes that containers require collaborative approaches between security and DevOps teams.

CI/CD on Windows-Based Environments - Noam Shochat, eToro - DevOpsDays Tel Av...DevOpsDays Tel Aviv

The document discusses eToro's use of CI/CD pipelines on Windows environments. It describes eToro's legacy technology stack including over 130 services running on 500 servers across two data centers. It then outlines eToro's use of Jenkins to orchestrate CI/CD flows across different environments and languages, using tools like Puppet, Mcollective, PowerShell and the Azure SDK for provisioning, deployment and orchestration. The document presents eToro's hybrid use of Microsoft technologies alongside open source tools like Redis, RabbitMQ and HAProxy to support their social investment network platform.

How to survive continuous innovation - Sebastien Goasguen - DevOpsDays Tel Av...DevOpsDays Tel Aviv

The document discusses how to continuously innovate with software given the rapid pace of new technologies being introduced and the large research and development budgets of corporations. It emphasizes the need to build solutions that address business needs rather than adopting every new tool, and to develop an adaptive culture within teams that can respond to changing technologies and industry trends. Examples are provided of how open source technologies like Docker have evolved and best practices for evaluating and using new software.

It's not just about the t-shirts - Sharone Zitzman - DevOpsDays Tel Aviv 2017DevOpsDays Tel Aviv

The document discusses the challenges faced by open source project maintainers, including having to respond to many user questions and issues while also balancing personal life. It notes how open source maintainers volunteer their own time yet are responsible for software relied on by many organizations. The document advocates redefining what is considered "cool" to recognize the important work of open source maintainers in enabling new technologies and markets.

Intrusion Detectionbutest

The document discusses different techniques for intrusion detection systems, including misuse detection, anomaly detection, pattern matching, and machine learning methods. It proposes two ideas for improving intrusion detection: 1) using association pattern detecting to match patterns in sequential data, and 2) discovering new patterns from existing rule sets using data mining or machine learning.

Intrusion DetectionGregory Hanis

This document discusses intrusion detection techniques. It describes misuse detection, which detects known attacks based on predefined rules, and anomaly detection, which detects deviations from normal behavior. Common misuse detection methods include rule-based, state transition analysis, and expert systems. Anomaly detection methods include statistical methods, machine learning, and data mining. The document also proposes ideas to improve intrusion detection, such as using association rule mining to detect patterns in audit data and discovering new patterns by analyzing existing rulesets.

I Dunderstnhajaanwar

This document discusses intrusion detection techniques. It defines intrusion detection as the process of identifying intrusions, which are activities that violate a system's security policy. It describes different types of intrusion detection systems including host-based, distributed, and network-based. The main intrusion detection techniques are described as misuse detection, which detects known attacks, and anomaly detection, which detects deviations from normal behavior. Ideas for improving intrusion detection include using association pattern detecting to match patterns in sequential data and discovering new patterns by combining existing rulesets from different intrusion detection systems.

Drools5 Community Training Module#1: Drools5 BLiP IntroductionMauricio (Salaboy) Salatino

Machine Can ThinkRahul Jaiman

This document provides an overview of machine learning. It begins with an introduction and definitions, explaining that machine learning allows computers to learn without being explicitly programmed by exploring algorithms that can learn from data. The document then discusses the different types of machine learning problems including supervised learning, unsupervised learning, and reinforcement learning. It provides examples and applications of each type. The document also covers popular machine learning techniques like decision trees, artificial neural networks, and frameworks/tools used for machine learning.

Intro 2 Machine LearningBrockhaus Consulting GmbH

This document provides an introduction to machine learning, including definitions, applications, and types of problems. It discusses how machine learning can be applied to data from sensors in industrial settings for tasks like risk prevention in real-time, searching historical data for patterns related to failures, estimating machine lifespan and product quality, and enabling more automated intelligent systems. Large amounts of data are generated every day from various sources, making it impossible for humans to analyze, but machine learning can find hidden patterns in this data.

machinecanthink-160226155704.pdfPranavPatil822557

This document provides an overview of machine learning. It begins with an introduction and discusses the basics, types (supervised, unsupervised, reinforcement learning), technologies, applications, and vision for the next few years. Key points covered include definitions of machine learning, examples of applications (search engines, spam filters, personalized recommendations), and descriptions of different problem types (classification, regression, clustering) and learning approaches (decision trees, neural networks, Bayesian methods).

5.11 expert systemMoshikur Rahman

An expert system is a type of artificial intelligence software that uses a knowledge base of human expert knowledge to advise users. The expert system infers answers to user questions using an inference engine that applies rules to the knowledge base. A knowledge engineer collects information from human experts to build the initial knowledge base and rules for the system to use in providing automated expert advice where a human expert may be unavailable or too expensive.

OpenML data@SheffieldJoaquin Vanschoren

OpenML aims to enable collaborative machine learning by allowing researchers to explore data and algorithms together online at web scale in real time. It seeks to remove friction from the machine learning process by making it easy to contribute and reuse datasets, algorithms, and experiments. This will allow research to progress faster and at larger scale through open sharing and automated collaboration across fields and laboratories.

Numenta ACM Data Min - PowerPoint Presentationbutest

This document provides an overview of Hierarchical Temporal Memory (HTM) and Numenta. It discusses how HTM is inspired by neuroscience and aims to build a common cortical algorithm. HTM forms a hierarchical network of nodes that learns spatial patterns and temporal sequences to build a model of input data. The document describes Numenta's timeline and demos applying HTM to tasks like object recognition, web analytics, and biomedical imaging. Potential applications of HTM include web analytics, video analysis, fraud detection and more.

Machine LearningJoshua Robinson

Presentation_Malware Analysis.pptxnishanth kurush

This document discusses using machine learning algorithms to detect malware files. It introduces different types of malware and provides an overview of machine learning methods for malware detection, including random forests, gradient boosting, and adaboost. The objectives are to use machine learning to detect legitimate and malware files and achieve high testing accuracy. The dataset includes over 130,000 files labeled as legitimate or malware. Several algorithms are applied including decision trees, random forests and gradient boosting. Random forests achieved the highest accuracy of 99.35% at distinguishing between legitimate and malware files.

San Francisco Hacker News - Machine Learning for HackersAdam Gibson

This document discusses machine learning and data science. It explains that machine learning can be used to analyze large datasets with many variables, unlike traditional statistics which is better for smaller datasets with fewer variables. The document then provides examples of different types of data and how machine learning can be applied to images, text, time series data and more. It also defines the roles of data scientists and data engineers. Finally, it provides recommendations for learning machine learning including starting with basic analytics, understanding key concepts and using tools like scikit-learn, Spark and Hadoop.

Testing 2 - Thinking Like A TesterArleneAndrews2

The document discusses various techniques for testing software such as black box testing, white box testing, coverage-based testing, model-based testing, property-based testing, and agile testing. It provides details on different types of coverage like code coverage, data coverage, and model-based coverage. It also describes different testing techniques like equivalence partitioning, input domain testing, and syntax generation that can be used with model-based testing. The document emphasizes applying critical thinking skills to testing and considering different perspectives.

Approaches to unraveling a complex test problemJohan Hoberg

When testing a complex system you are often faced with complex test problems. Cause and effect cannot be deduced in advance, only in retrospect. According to the Cynefin framework, the general approach to tackle complexity is probe-sense-respond. Try something, analyze the outcome, and based on that outcome, try something else. This is the basis of all my approaches to begin unraveling complex test problems. But how do I select my test scope for a specific complex test problem?

More Related Content

Viewers also liked (7)