Innovate Better Through Machine data Analytics

Editor's Notes

• #11: - You’ve heard the culture story before, let’s go over that briefly
• #14: POLL: who measures deployment cycles in: - MONTHS? - WEEKS? - DAYS? - HOURS?
• #17: I promised you FIVE. Check it out – this is SIX.
• #19: Splunk Enterprise is fully featured, platform for collecting, searching, monitoring and analyzing machine data and getting operational intelligence. You can monitor both real-time (as the data is streaming) and historical data. Splunk collects machine data securely and reliably from wherever it’s generated in any formant. It stores and indexes the data in real time in a centralized location and protects it with role-based access controls. You can troubleshoot your network problems and investigate security incidents in minutes (not hours or days). Monitor your end-to-end infrastructure to avoid service degradation or outages. Gain real-time visibility and critical insights into customer experience, transactions and behavior.
• #21: Splunk can provide insight across the entire application delivery lifecycle. Developers can search and visualize data from entire build pipeline and production environments without needing to access production machines.
• #22: When using Splunk to mine machine data our customers and prospects can 1) INCREASE APP DELIVERY VELOCITY 2) IMPROVE CODE QUALITY 3) INCREASE BUSINESS IMPACT OF APPLICATION DELIVERY
• #24: * PUPPET
• #25: - Test harness, build automation
• #26: Comparing errors in QA vs PROD
• #30: Pro-actively identify business issues Visualize the source of the issue Collect & Analyze diagnostics data automatically Take pro-active actions to mitigate the problem Engage the right people immediately Proactively Identify: Stuck deals are identified before the call is made to support Visualize: Business Activity Monitoring tracks deals; Yellow indicates where the deal is stuck. Red would indicate a catastrophic system failure.
• #33: Splunk Enterprise is fully featured, platform for collecting, searching, monitoring and analyzing machine data and getting operational intelligence. You can monitor both real-time (as the data is streaming) and historical data. Splunk collects machine data securely and reliably from wherever it’s generated in any formant. It stores and indexes the data in real time in a centralized location and protects it with role-based access controls. You can troubleshoot your network problems and investigate security incidents in minutes (not hours or days). Monitor your end-to-end infrastructure to avoid service degradation or outages. Gain real-time visibility and critical insights into customer experience, transactions and behavior.
• #36: The new app collects and analyzes performance data from Puppet Enterprise. Customers get visibility into critical services, such as PuppetDB, the Puppet Serve, and console services. The app also helps to reduce troubleshooting times and proactively fix health issues in the Puppet environment, and includes the following insights: Console services response times to benchmark and actively plan console resources The number of request errors by Puppet clients to help recognize potential code or infrastructure issues Role-based access control dashboards to monitor user activity, including authentication errors to help with potential security issues Requests from PuppetDB to identify commonly executed or failed queries—isolating potential infrastructure bottlenecks PuppetDB node deactivation activity for isolation of security or automation issues Commonly submitted PuppetDB commands from client IPs to assist in pinpointing potential security issues Puppet Server compilation metrics that help teams evaluate the health of their automation environment and appropriately assign resource
• #37: The Chef Analytics App for Splunk is available for free on Splunkbase Splunk app marketplace and provides Chef users with visibility into metrics such as success / failure rates, most active users and most active organizations. The Chef Analytics App for Splunk also helps you understand the frequency of the details of errors across infrastructure so that you can catch and troubleshoot high impact issues, like a major bug in a cookbook or an infrastructure issue like network connectivity, in real time More details: https://www.chef.io/blog/2015/04/17/integrating-chef-analytics-with-splunk/
• #38: Why Splunk for AWS? Security Intelligence (Cloudtrail, Config Cloudwatch, VPC) Operational Intelligence (Cloudwatch, Config, ELB, Cloudfront) DevOps Intelligence (Cloudwatch, Lambda) Big Data Insights (Kinesis, EMR, IoT, S3) Data sources in Splunk App for AWS AWS Cloudtrail Service that delivers logs of admin activity on AWS infrastructure Examples: Start/Stop/Create instance Change of User roles/rights Modification of Network Configuration Delivers log files to customers; no UI, display, analysis, search AWS Config Provides resource inventory Provides configuration history & change information Enables security & governance Amazon Cloudwatch Metrics IP traffic information to/from VPC network interfaces Data stored and accessible from AWS Cloudwatch Logs Amazon Cloudwatch VPC Flow Logs IP traffic information to/from VPC network interfaces Data stored and accessible from AWS Cloudwatch Logs AWS Access Logs Elastic Load Balancing (ELB) Cloudfront CDN S3 AWS Billing Current Month via Cloudwatch metrics Monthly Detailed Billing
• #40: Secure - Supports TLS / SSL Support for configurable data collection to simplify data classification and access control index sourcetype source Support for collection of container labels and env keys which can further help with data classification Simple Easy to setup e.g., no need to deploy Splunk Universal Forwarder (UF) and scale it Much easier to collect data in Splunk Cloud deployments Scalable built on top of HTTP Event Collector (HEC) Support encryption via SSL as long as Splunk has a TCP SSL endpoint opened. The difference I would say is that with syslog over SSL there is no auth model other than the cert itself and you can only have one SSL cert for all of Splunk. With HEC you have fine grained control of which servers can send data and which indexes they can access as that can be configured per token.  Simplified classification by source and labels, you can do that in syslog driver as well. See labels and env in https://docs.docker.com/engine/admin/logging/overview/ Additional configurations on Splunk logging driver to see “process” field, https://docs.docker.com/engine/admin/logging/log_tags/. Available by default in Syslog. Splunk Cloud: HEC is fully supported in Splunk Cloud. You cannot however arbitrarily open a TCP/UDP port in Splunk Cloud which would be required if you needed to forward Syslog data to a UF running in Splunk Cloud. Need to confirm how to scale HEC in Splunk Cloud. https://docs.docker.com/engine/admin/logging/splunk/
• #41: HTTP Event Collector is easy way to send data to Splunk Enterprise. Notably, the EC enables you to send data over HTTP/ HTTPS directly to Splunk Enterprise from your application. The EC was developed with application developers in mind, so that all it takes is a few lines of code added to an app for the app to send data. Also, the EC is token-based, so you never need to hard-code your Splunk Enterprise credentials in your app or supporting files. HTTP Event Collector provides a new way for developers to send application logging and metrics directly to Splunk Enterprise via HTTP in a highly efficient, scalable and secure manner
• #42: AWS Lambda is an Amazon Web Services compute service that runs your back-end code in response to events, and manages compute resources for you. In cooperation with Amazon, Splunk is pleased to provide a built-in AWS Lambda Node.js blueprint for HTTP Event Collector. The blueprint makes it easy to get started quickly, sending events from AWS Lambda to HTTP Event Collector running on Splunk Enterprise or Splunk Cloud. You can also write a Lambda function from scratch, either in JavaScript using Node.js or in Java. AWS Lambda can receive event data from Amazon Kinesis, Amazon DynamoDB, Amazon S3, and other Amazon services, and then send it on to HTTP Event Collector. You can collect the data using HTTP Event Collector in Splunk Cloud, which also runs on AWS, or in Splunk Enterprise on-premises.
• #43: Splunk App for Stream is a free App that enables you to capture, visualize and analyze data in much more granular way then ever before. You can see everything – ALL user and applications behavior ],response times from every layer, DNS information, storage traffic, network traffic, your websites content, connections. Once this data is in Splunk you can correlate it with other data for much more comprehensive visibility. First Splunk App for Stream is a way of get wire data into Splunk Enterprise. By adding this comprehensive source of machine data, it enables you to extend Operational Intelligence use cases across IT security and the business. It is a software only solution with the ability that can be installed on VM on any host, it enables real-time insights into multi-cloud environments. And as such, it is easy to install anywhere on most of standard machines, it is a passive very efficient way to capture data.
• #44: To address the needs of developers, operations and product management, you need Operational Intelligence for your mobile apps. This is what we call mobile intelligence. Mobile intelligence provides real-time insight on how your mobile apps are performing, and can correlate with and enhance Operational Intelligence. Splunk software enables organizations to search, monitor, analyze and visualize machine-generated data from websites, applications, servers, networks, sensors and mobile devices. Splunk MINT helps organizations monitor mobile app usage and performance, gain deep visibility into mobile app transactions and accelerate development Deliver better performing, more reliable apps When a user has a problem with a mobile app, the issue could be isolated or spread across all app versions, handsets and OS types. With Splunk MINT, you can see issues with app performance or availability in real time. Bugs can be addressed quickly, and app developers can gain a head start in creating and delivering valuable app updates. Achieve End-to-End visibility When mobile apps fail, there are many potential sources of failure. With Splunk MINT, you can analyze overall transaction performance. And using Splunk MINT, you can correlate this data with information from back-end apps to gain detailed insight on transaction problems. As a result, operations can reduce MTTR and better anticipate future mobile app back-end requirements. Deliver real-time analytics Mobile apps give enterprises new ways of conducting digital business. With mobile app information in Splunk Enterprise, you can correlate usage and performance information— some call this omni-channel analytics—to better understand how users are engaging all aspects of your organization.
• #48: Fast feedback obtained with analytics improves app delivery velocity. From code definition to production, having insight for all the teams leads to less bugs, faster testing, faster releases, less production issues and accelerated innovation. Deliver end-to-end visibility across every DevOps toolchain component Iterate faster with correlated insight across the application delivery lifecycle Improve DevOps team efficiency by measuring and benchmarking release contributions  
• #49: Pinpoint and resolve code issues before they impact customers .Find and fix production issues faster Use objective metrics to ensure code is operational and meets quality SLAs
• #52: When using Splunk to mine machine data our customers and prospects can 1) INCREASE APP DELIVERY VELOCITY 2) IMPROVE CODE QUALITY 3) INCREASE BUSINESS IMPACT OF APPLICATION DELIVERY
• #58: Order Flow, message queues, Garbage Collection, Java Heap identify errors by java class, thread alert actions - jira ticket, service now ticket, webhook
• #59: PM’s love to look at feature usage; are new features being used? How do we allocate developer time to create/enhance features
• #63: Need to check all of them.
• #64: The NBC Universal’s Web Operations team is using Splunk platform to track performance of their releases from pre-production, QA and in production. Splunk software frees up their DevOps teams and enables them to focus on innovation while pushing code to production daily. The way they test these deployments is to release to half of our servers first while they are out of production rotation but still accessible through other non-public DNS paths for testing.  Their “Gamma” environment is a subset of production servers that have been released to for testing, but aren’t taking public traffic.  Theyuse Splunk to monitor new errors on the servers after releasing changes to them, which is easy to do since just before the servers were released to they already had an established error volume baseline from being in production.  If any new errors appear or the volume of traffic for 400/500 errors suddenly increases after the release, They immediately see the spike in our monitoring. Alerting on thresholds for event log errors - roll-back problematic deployments or diagnose a build in a subset of the production environment
• #65: Splunk is Used Enterprise-wide at Apollo Group Provide full visibility into sanity and Load testing in QA environments before deploying to production Using the Splunk SDK for Java to gather key machine metrics (CPU, memory, disk utilization) and log file errors metrics are compared to benchmarks and SLA thresholds Helps troubleshooting if tests do not contain the expected results Product Lifecycle Monitoring Custom Application Monitoring Mobile App Monitoring/Analytics Customer Analytics Security and Forensics/Analytics Network Monitoring/Analytics AWS Billing Monitoring/Analytics Cloud and Physical Server Monitoring/Analytics Issue resolution from days to minutes in both QA/staging and production environments – significant productivity/quality impact ROI in the millions - Engineering/IT teams are significantly more productive & Fewer outages
• #66: Challenges: No single place to access and visualize machine data Manual diagnosing and searching through data generated by servers and applications To retrieve information, sysadmins have to ssh into production machines before sending off to developers to grep through the logs With Splunk: Quickly validate and troubleshoot code pushes to production Ensure that new code does not negatively impact performance or user experience Reduced one application’s error rate by 2 orders of magnitude in a matter of weeks