SplunkLive! Zurich 2017 - Advanced Analytics / Machine Learning
Download as PPTX, PDF2 likes849 views
Splunk is a powerful platform for understanding your data. This session will provide an overview of machine learning capabilities available across Splunk’s portfolio. We'll dive deeply into Splunk's Machine Learning Toolkit App, which extends Splunk Enterprise with a rich suite of advanced analytics, machine learning algorithms, and rich visualizations. It also provides customers with a guided model-building and operationalization environment. The demonstration will include the guided model-building UI for tasks such as predictive analytics, outlier detection, event clustering, and anomaly detection. We’ll also review typical use cases and real-world customers who are using the Toolkit to drive business results.
Ad
More Related Content
What's hot (20)
Similar to SplunkLive! Zurich 2017 - Advanced Analytics / Machine Learning (20)
Ad
More from Splunk (20)
Ad
Recently uploaded (20)
SplunkLive! Zurich 2017 - Advanced Analytics / Machine Learning
- 1. © 2017 SPLUNK INC.© 2017 SPLUNK INC. MAY 9, 2017 | ZURICH
- 2. © 2017 SPLUNK INC. Advanced Analytics and Machine Learning in Splunk Philipp Drieger | Sr. Sales Engineer | SME BA IoT ML MAY 9, 2017 | ZURICH
- 3. © 2017 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved. Safe Harbor Statement
- 4. © 2017 SPLUNK INC. Why Machine Learning?
- 5. © 2017 SPLUNK INC. Humans are good at learning, but we get lost in volume and details…
- 6. © 2017 SPLUNK INC. ▶ Improve decision making ▶ Uncover hidden trends or relationships ▶ Alert on deviations ▶ Forecast or anticipate incidents All of this requires diverse data from across many silos. Lots of unstructured, real-time data. Why Do We Need Machine Learning?
- 7. © 2017 SPLUNK INC. Run the Business in Real Time Data From the Past Real-Time Data Statistical Forecast T – a few days T + a few days Security Operations Center IT Operations Center Business Operations Center Predictive (Models) Descriptive (BI Tools, Data Lakes) Grey space
- 8. © 2017 SPLUNK INC. What is Machine Learning?
- 9. © 2017 SPLUNK INC. ▶ Face detection: find faces in images ▶ Spam filtering: identify SPAM messages ▶ Shopping recommendations: predict what customers would like to buy ▶ Fraud detection: identify credit card transactions which may be fraudulent in nature ▶ Weather forecast: predict whether or not it will rain tomorrow; estimate daily max/min ML is All Around You!
- 10. © 2017 SPLUNK INC. ML 101: What is Machine Learning? What: “Field of study that gives computers the ability to learn without being explicitly programmed” – A. Samuel, 1959 How: Generalizing (learning) from examples (data) Simple ML workflow: – EXPLORE data – FIT models based on data – APPLY models in production – VALIDATE models – REPEAT
- 11. © 2017 SPLUNK INC. 1. Supervised Learning: generalizing from labeled data Major Types of Machine Learning 1. Supervised Learning: generalizing from labeled data
- 12. © 2017 SPLUNK INC. Major Types of Machine Learning 2. Unsupervised Learning: generalizing from unlabeled data 2. Unsupervised Learning: generalizing from unlabeled data ?
- 13. © 2017 SPLUNK INC. ▶ System is rewarded (or punished) based on the outcomes it generates ▶ Action leads to a change in the state of the world and generates an error score Major Types of Machine Learning 3. Reinforcement Learning
- 14. © 2017 SPLUNK INC. The ML Process Get and explore data Select and fit an algorithm, generating a model Apply and validate models Surface model to consumers to solve problems Problem: <Stuff in the world> causes big time & money expense. Value Hypothesis Solution: Build ML model to forecast <possible incidents>, act pre-emptively & learn Operationalize
- 15. © 2017 SPLUNK INC. Splunk’s Machine Learning Tour
- 16. © 2017 SPLUNK INC. Overview of ML at Splunk CORE PLATFORM SEARCH PACKAGED PREMIUM SOLUTIONS MACHINE LEARNING TOOLKIT Platform for Operational Intelligence
- 17. © 2017 SPLUNK INC. Search Includes Machine Learning Core Platform Search is a powerful and highly flexible interface built with ML
- 18. © 2017 SPLUNK INC. Splunk IT Service Intelligence One of several Premium Solutions with Packaged ML Get Data Define services, entities and KPIs Monitor and troubleshoot Analyze and detect Data-Defined, Data-Driven Service Insights Adaptive Thresholds and Anomaly Detection
- 19. © 2017 SPLUNK INC. ▶ Assistants: Guided model building, testing and deployment for common objectives ▶ Showcases: Interactive examples for typical IT, security, business and IoT use cases ▶ Algorithms: 25+ standard algorithms included with the toolkit ▶ ML Commands: New SPL commands to fit, test and operationalize models ▶ Python for Scientific Computing Library: Access to 300+ open source algorithms Splunk Machine Learning Toolkit Extends Splunk platform functions and provides a guided modeling environment Build custom analytics for any use case
- 20. © 2017 SPLUNK INC. Algorithms Supported (v2.0)
- 21. © 2017 SPLUNK INC. Custom Machine Learning – Success Formula Domain Expertise (IT, Security, …) Data Science Expertise Splunk Expertise Identify use cases Drive decisions Set business/ops priorities SPL Data prep Statistics/math background Algorithm selection Model building Splunk ML Toolkit facilitates and simplifies via examples & guidance Operational success
- 22. © 2017 SPLUNK INC. Splunk Architecture and Machine Learning
- 23. © 2017 SPLUNK INC. Continuous Data Ingest at Scale DevelopVisualize PredictAlertSearch Engineers Data Analysts Security Analysts Business Users Native Inputs TCP, UDP, Logs, Scripts, Wire, Mobile Industrial Data SCADA, AMI, Meter Reads Modular Inputs MQTT, AMQP, COAP, REST, JMS HTTP Event Collector Token Authenticated Events Technology Partnerships Kepware, AWS IoT, Cisco, Palo Alto Maintenance Info Asset Info Data Stores External Lookups/EnrichmentOT Industrial Assets IT Consumer and Mobile Devices Real Time
- 24. © 2017 SPLUNK INC. Search Third-Party Applications Smartphones and Devices Tickets Email Send an email File a ticket Send a text Flash lights Trigger process flow Sense and Respond Every Search Can Use Machine Learning OT Industrial Assets Consumer and Mobile Devices AlertReal Time
- 25. © 2017 SPLUNK INC. Real Time Splunk: Data Fabric IT Users Analysts Biz Users Ad hoc Search Custom Dashboards Monitor and Alert Reports/ Analyze Clickstreams HadoopDevices Networks GPS/ Cellular Online Shopping Carts Servers Applications Data Warehouses Structured Data Sources CRM ERP HR Billing Product Finance DB Connect Look-ups ODBC SDK API OT Industrial Assets IT Consumer and Mobile Devices Analysts Biz Users
- 26. © 2017 SPLUNK INC. Show me the ML!
- 27. © 2017 SPLUNK INC. ML Use Cases and Customer Stories
- 28. © 2017 SPLUNK INC. Machine Learning Customer Success Network Incident Detection Service Degradation Detection Security / Fraud Prevention Machine Learning Consulting Services Analytics App Built on ML Toolkit Optimizing operations and business results Predict Gaming Outages Fraud Prevention Entertainment Company Cell Tower Incident Detection Optimize Repair Operations Prioritize Website Issues and Predict Root Cause
- 29. © 2017 SPLUNK INC. ML Toolkit Customer Use Cases ▶ Speeding website problem resolution by automatically ranking actions for support engineers ▶ Reducing customer service disruption with early identification of difficult-to-detect network incidents ▶ Minimizing cell tower degradation and downtime with improved issue detection sensitivity ▶ Improving cell tower uptime and reducing repair truck roles with anomaly detection and root cause analysis ▶ Predicting and averting potential gaming outage conditions with finer-grained detection ▶ Preventing fraud by Identifying malicious accounts and suspicious activities ▶ Ensuring mobile device security by detecting anomalies in ID authentication Entertainment Company
- 30. © 2017 SPLUNK INC. Detect Network Outliers Reduced downtime + increased service availability = better customer satisfaction ML Use Case Technical Overview ▶ A customized solution deployed in production based on outlier detection ▶ Leverage previous month data and voting algorithms “The ability to model complex systems and alert on deviations is where IT and security operations are headed … Splunk Machine Learning has given us a head start...” ▶ Monitor noise rise for 20,000+ cell towers to increase service and device availability, reduce MTTR
- 31. © 2017 SPLUNK INC. Reliable Website Updates Proactive website monitoring leads to reduced downtime ▶ Custom outlier detection built using ML Toolkit Outlier assistant ▶ Built by Splunk Architect with no Data Science background “Splunk ML helps us rapidly improve end-user experience by ranking issue severity which helps us determine root causes faster thus reducing MTTR and improving SLA” ▶ Very frequent code and config updates (1000+ daily) can cause site issues ▶ Find errors in server pools, then prioritize actions and predict root cause ML Use Case Technical Overview
- 32. © 2017 SPLUNK INC. Wrap Up
- 33. © 2017 SPLUNK INC. ▶ Get the Machine Learning Toolkit from Splunkbase ▶ Go watch Machine Learning Videos on Splunk YouTube Channel http://tiny.cc/splunkmlvideos ▶ Go watch the Machine Learning talks from .conf2016: • Advanced Machine Learning in SPL with the Machine Learning Toolkit by Jacob Leverich • Extending SPL with Custom Search Commands and the Splunk SDK for Python by Jacob Leverich ▶ Early Adopter and Customer Advisory Program: [email protected] ▶ Reach out to your Account Team – they are happy to help! What Else?
- 34. © 2017 SPLUNK INC. SEPT 25-28, 2017 Walter E. Washington Convention Center Washington, D.C. .conf2017 The 8th Annual Splunk Conference conf.splunk.com You will receive an email after registration opens with a link to save over $450 on the full conference rate. You’ll have 30 days to take advantage of this special promotional rate! SAVE OVER $450
- 35. © 2017 SPLUNK INC. Take the Survey on Pony Poll ponypoll.com/zurich2017
- 36. © 2017 SPLUNK INC.© 2017 SPLUNK INC. Thank you!
Editor's Notes
- #8: What’s needed : A solution that can monitor conditions of interest and analyze behaviors of interest across all business processes, and deliver actionable insights to business decision-makers Splunk handles the full continuum: past, present & future. DATA IS STILL IN MOTION, still in a BUSINESS PROCESS. Enrich real-time MACHINE DATA with structured HISTORICAL DATA Make decisions IN REAL TIME using ALL THE DATA
- #10: Q: What is a statistical model? A: A model is a little copy of the world you can hold in your hands. Formal: A model is a parametrized relationship between variables. FITTING a model sets the parameters using feature variables & observed values APPLYING a model fills in predicted values using feature variables Image source: http://phdp.github.io/posts/2013-07-05-dtl.html
- #11: Q: What is a statistical model? A: A model is a little copy of the world you can hold in your hands. Formal: A model is a parametrized relationship between variables. FITTING a model sets the parameters using feature variables & observed values APPLYING a model fills in predicted values using feature variables Image source: http://phdp.github.io/posts/2013-07-05-dtl.html
- #13: Example:
- #15: The ML process is itself a generalization of the different use cases. ML spans domains! Get all the relevant data to the problem; Explore the data Select and Fit an algorithm on the data, generating a model Apply & Validate models until predictions solve the problem Surface the model to X Ops, who consume the model to solve the problem The arrow means OPERATIONALIZE. Feed back incident data & other high-level analysis back into the ML Process. Keep exploring that data & fitting better models to align with reality. Loop Step #5 (Act) back to Step #1 (Data).
- #19: So, let’s look at a simple visual to discuss how it works? In four simple steps, customers can achieve data driven service insights. They Get the data in. (all the data…) They quickly define services, entities, and KPIs They monitor and troubleshoot They analyze and detect Through these steps, the customers is able to realize the value of Data Defined, Data Driven Service Insights.
- #20: Machine learning is bringing data analysis into a new era, allowing companies to use predictive analytics that continually “learn” from historical data. These analytics can optimize IT, security and business operations—helping to detect incidents, reduce resolution times, and predict and prevent undesired outcomes. The Splunk platform makes it easy for you to harness the power of machine learning by offering a rich set of machine learning commands and a guided workbench to create custom models for any use case. Assistants: Assistants let you choose the algorithm and then guide you through model creation, testing and deployment for common objectives like forecasting values, predicting numeric or categorical fields, and detecting numeric or categorical outliers. Showcases: Walk through interactive examples of model creation organized by common use cases for IT, security, IoT and business analytics. Examples include predicting disk failures, finding outliers in response time, predicting VPN usage and forecasting internet traffic. SPL ML Commands: The Splunk platform offers over 20 machine learning commands that can be applied directly to your data for detection, alerting or analysis. Commands such as outlier, predict, cluster and correlate utilize fixed algorithms, while others such asanomalydetection allow you to choose between several algorithms to best fit your needs. Want more flexibility? With the Machine Learning Toolkit, you get access to additional commands and open source algorithms to create custom models for any use case. Python for Scientific Computing Library: Use machine learning SPL commands like fit, apply and allow to directly build, test and operationalize models using open source Python algorithms from the Splunk Python for Scientific Computing Add-on.
- #21: MS: This slide needs some work and structure around all the types of algos we’re supporting – pre-processing, feature extraction, classification, regression, clustering, time-series forecasting, outlier detection, text analytics, etc.
- #24: Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most machine data generated by all of the devices, control systems, sensors, SCADA, networks, applications and end users connected by industrial networks. There are many options: Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API and more. You can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis. The Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The data volumes supported by Splunk are ideal for IoT and industrial data. There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include: Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS. Rest API Modular Input: Poll local and remote REST APIs and index the responses. Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data. Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper. DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection. MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics. AMQP Modular Input: Index data from message queues provided by AMQP brokers. JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ. COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server. SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution. Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events. Splunk isn’t the only technology that can benefit from collecting machine data, so let Splunk help send the data to those systems that need it. For those systems that want a direct tap into the raw data, Splunk can forward all or a subset of data in real time via TCP as raw text or RFC-compliant syslog. This can be done on the forwarder or centrally via the indexer without incrementing your daily indexing volume. Separately, Splunk can schedule sophisticated correlation searches and configure them to open tickets or insert events into SIEMs or operation event consoles. This allows you to summarize, mash-up and transform the data with the full power of the search language and import data into these other systems in a controlled fashion, even if they don’t natively support all the data types Splunk does.
- #25: Alerts are triggered when certain conditions are met by the results of the search upon which it is based. Alerts can be based on both historical and real-time searches. When an alert is triggered, it performs an alert action. This action can be the sending of the alert information to a designated set of email addresses, or the posting of the alert information to an RSS feed. Alerts can also be set up to run a custom script when they are triggered. You can base these alerts on a wide range of threshold and trend-based scenarios. Custom Alert Actions provide the ability to use Splunk Alerts to trigger custom actions or pre-packaged integrations with 3rd party products such as work order management systems, trouble ticketing or support systems. Splunk and partners provide a growing set of integrations including, ServiceNow, xMatters, Webhooks and more. With custom alert actions you can: Send message to IM clients (HipChat, Slack) Send SMS Automate the creation of tickets (ServiceNow, Jira) Take action or send events to firewalls, devices, management consoles Trigger device-level actions (change lights, sounds an alarm, send action to device) Trigger any organization-specific action (restart application, integrate with homegrown service, and more) This way you can set alerts on data coming from ICS, SCADA, sensor etc. data and alert operators or trigger actions in third party applications, enabling you to sense anomalous condition in the data and respond to these conditions.
- #27: Time for ML demo! Get the ML App: http://tiny.cc/splunkmlapp Want more? Take Splunk’s Analytics & Data Science course! Course prework: http://bit.ly/splunkanalytics
- #29: Our Early Adopter customers have had much success creating and operationalizing ML models. Some examples include: Zillow makes hundreds of website updates daily, including content from several partners nationally. These updates can often cause issues in the site. Zillow built an ML model that predicts which of these changes is likely to result in an issue to allow the team to fix them proactively. Once a potential or actual issue has been identified, the model can also provide guidance on likely root cause and resolution. TELUS has thousands of mobile phone towers across Canada; when one of these goes offline it can cause significant disruption for their customers. TELUS built a model to predict which towers are likely to fail so that they can proactively fix issues before they occur.
- #33: Time for ML demo! Get the ML App: http://tiny.cc/splunkmlapp Want more? Take Splunk’s Analytics & Data Science course! Course prework: http://bit.ly/splunkanalytics
- #35: And of course, your biggest education opportunity this year is .conf2017 which will be held right back here in Washington, DC on September 25 – 28. I know you have heard a lot about .conf2017 today but don’t forget that by attending SplunkLive! today we are extending you a discount of over $450. You will be able to register with a unique link that will be sent in the post SplunkLive! emails to go out next week.
- #37: Time for ML demo! Get the ML App: http://tiny.cc/splunkmlapp Want more? Take Splunk’s Analytics & Data Science course! Course prework: http://bit.ly/splunkanalytics