Splunk Enterprise for IT Troubleshooting Hands-On

Editor's Notes

• #3: There has been an explosion of growth of IT data center technologies, IoT, mobile, distributed apps, virtualization, containers. This has brought increased efficiency and utilization, but also escalating IT complexity. IT Operations teams face many challenges as the complexity of systems increases and budgets tighten. Their customers (internal or external) are angry about latency or outages in key services.
• #5: There has been an explosion of growth of IT data center technologies, IoT, mobile, distributed apps, virtualization, containers. This has brought increased efficiency and utilization, but also escalating IT complexity. IT Operations teams face many challenges as the complexity of systems increases and budgets tighten. Their customers (internal or external) are angry about latency or outages in key services.
• #6: You may have lots of disparate and complex and siloed based solutions. When you need to find a solution to a problem, you may need to get a war room ready, which leads to finger pointing and trying to debug in your production environment. You may spend hours trying to find a solution. Often, you end up using a brute force approach like restarting the system, leaving no evidence of what the problem actually was. All of which means that IT is no longer spending time on innovating but losing valuable time just keeping the the lights on or fighting fires.
• #7: Splunk Enterprise is fully featured, platform for collecting, searching, monitoring and analyzing machine data and getting operational intelligence. You can monitor both real-time (as the data is streaming) and historical data. Splunk collects machine data securely and reliably from wherever it’s generated in any formant. It stores and indexes the data in real time in a centralized location and protects it with role-based access controls. You can troubleshoot your network problems and investigate security incidents in minutes (not hours or days). Monitor your end-to-end infrastructure to avoid service degradation or outages. Gain real-time visibility and critical insights into customer experience, transactions and behavior.
• #8: We don’t require you to have a deep understanding of your data, or to have a predefined schema and requirements. You don’t need expensive custom connecters to get data into Splunk. We have our own map reduce based high speed data index and retrieval mechanism. We can index data from any part of your infrastructure. We scale from a single server to petabytes of data, and you can use commodity hardware. You can leverage our Splunk Cloud offerng if you don’t want to manage your own Splunk instance. You can start getting into the core of the problem, If you have a system that does not have proactive capabilities you can do that with Splunk Enterprise. And expand from there into security, capacity planning applications management – truly big gold mine of use cases from your data. And our customers once they start to gain that operational visibility they evolve to getting deeper insights from your data. No database in the backend as we apply schema on the fly. You need raw data to be able to re-use it. We are creating intelligence on top of the data therefore easy scaling.
• #9: We’ve found that most companies start using Splunk in one of these 5 areas, and typically as more teams use Splunk, their usage traverses each of these 5 areas. Both IT and business professionals can analyze machine data to get real-time visibility and operational intelligence. With our platform for machine data, organizations can meaningfully improve their performance in a wide range of areas e.g. meet service levels, reduce costs, mitigate security risks, maintain compliance and gain insights.   Today we are going to focus on some of the major use cases and values related to the IT Operations space.
• #10: In IT Operations, this maturity model is a great template/mainstay when it comes to how Splunk is utilized. Most teams have downloaded Splunk on a laptop and from there it gets scaled to a server and to multiple server, etc. The idea from an ITOps maturity model is very much the same— Search and investigation. Using Splunk, organizations identify and resolve issues up to 70% faster and reduce costly escalations by up to 90%. Splunk is one place to find and fix problems, and investigate incidents across all your IT systems and infrastructure. Proactive monitoring. Monitor IT systems in real time to identify issues, problems and attacks before they impact your customers, services and revenue. Splunk keeps watch of specific patterns, trends and thresholds in your machine data so you don't have to. Trigger notifications in real-time via email or RSS, execute a script to take remedial actions, send an SNMP trap to your system management console or generate a service desk ticket. Operational visibility. See the whole picture, track performance and make better decisions. Visualize usage trends to better plan for capacity; spot SLA infractions, track how you are being measured by the business. Do all of this using your existing machine data without spending millions of dollars instrumenting your IT infrastructure. Real-time business insight. Make better-informed business decisions by understanding trends, patterns and gaining Operational Intelligence from your machine data. See the success of new online services by channel or demographic, reconcile 3rd-party service provider fees against actual use, find your heaviest users and heaviest abusers, and more. Because machine data captures every behavior, the possibilities are game changing. You'll find the lead times to get to this intelligence dramatically less than other solutions - measured in minutes/hours instead of months.   Who is at Search and Investigate? Raise your Hands. Proactive Monitoring and Alerting? Raise your Hands. Operational Visibility? Raise your Hands. Real-time Business Insight? Raise your Hands. Who thinks it makes sense for all of us to have our business at Real-time Business Insight? Why? So how do we get there?
• #11: Over the last couple of years Splunk has evolved from an engine for machine data to a platform for machine data – you can see that by looking at the number and variety of apps in Splunkbase, which range from add-ons and templates to full fledged apps to help you collect, analyze and harness data from every layer of your technology stack. These apps are built, not just by Splunk, but by our customers, technology partners (such as Cisco or NetApp), Splunk employees, or anyone who has come up with an app they want to share. Apps and add-ons help make it easy to get data into Splunk and out of Splunk. We are complementing other solutions in the data center Two important things to remember: If you’re looking for a logo here and don’t see it, Splunk still doesn’t limit you – you can always index data from that technology – Splunk apps and add-ons help you accelerate the process. We provide a full featured REST API and a variety of SDKs that help you build your own custom apps for technologies and insights custom to your business. This is to help you create a specific interface to your data in special format and development languages your team is used to. Splunk apps and add-ons are not comparable to having point solutions in every silo, because your data from each silo is more valuable when you can see it in context with other data from other technologies. Splunk apps simply help you get to the point faster where you can see correlations and comparisons of machine data ACROSS silos.
• #12: This slide has instructions for setting up the app we’ll be using for this session.
• #13: This is the Splunk user interface. When you first log in, you’ll see a list of apps down the left side and some icons in the center that will take you to product tours, the app base, and documentation. At the top of the page, there are some menus for settings. We’re going to use the Search & Reporting app, so click on the green button that says “Search & Reporting”.
• #14: The menus across the top stay put, and now there’s a new menu for apps instead of the nav bar on the left. You can click on that to see the list of apps. There’s an application menu below that with options for Search, Pivot, Reports, Alerts and Download. By default, you’ll see a search bar that looks just like one you’d see on your favorite search engine. When you first start putting data into Splunk, usually you’re going to want to see what it looks like. Let’s start with the simplest search possible – just put an asterisk in the search box. Choose “Last 60 minutes” for the time range, then click the magnifying glass to run the search.
• #15: You can see just by scrolling through the results that the data has many different formats – we have some logs in JSON, some access logs, and many others. Splunk has indexed all of this data and put it in one place. Just under the search bar, you can see a timeline showing how many events occurred at each time. Go ahead and click on a bar to see just the events for that time frame.   On the left side, we have all the fields that Splunk has extracted from our data. We have NUMBER of hosts and NUMBER different sourcetypes. A sourcetype is usually a data format – think of the various kinds of web logs, error logs, application logs, or you can define your own custom sourcetypes.
• #16: Let’s take a look at how field extraction works. Search for “sourcetype=customLog”. At first glance, it looks like we’re extracting all the fields, but let’s just make sure. Click on Extract New Fields at the bottom of the field list.
• #17: Click on any event in the list. Now we can see that the last field is not being extracted – all the other fields are highlighted. Click Next to continue.
• #18: Choose “Regular Expression”, but don’t worry, we’re not going to be writing any regular expressions in this session! Click Next…
• #19: Now you can just highlight the field you want to extract, give it a name and click Add Extraction.
• #20: Check it out – the field is being highlighted now, and we have a column in the table showing the values for our new field.
• #21: Now we just need to save it. Type some name in the blank and click Finish.
• #22: Click on Explore the fields I just created in Search to see the new field in action. You can see it in the field list, and in any event.
• #24: But now it’s time to get to work – we have a problem on the website! We’ll start by looking at the access logs by searching for sourcetype=access*. There are still a lot of events here, let’s look at the status field to see if there are any obvious issues. Just clicking on the status field gives us a count, but just the count doesn’t really give us the context we need to see if there’s a problem. Let’s click on Top values by time.
• #25: Let’s change the graph from a line to a column – it’ll be easier to see what’s going on. Let’s also change the format to stacked. Now it’s easy to see that there have been times in the last hour when we returned more 503 status codes than 200 status codes. Click “503” in the legend to narrow your search to just 503 status codes.
• #26: Right now, we’re looking at data from all the webservers at once, so let’s add “| stats count by host” to the search and look at the count of errors by host, in case one of the servers is having an issue. Now we’re getting somewhere! It looks like most of the errors are coming from webserver-01. Let’s see what information we can get about that host by doing a new search for “host=webserver-01”. Click on “webserver-01” in your search results, then click “New Search”
• #27: First we’ll click on “sourcetype” to see what data is available. There are three sourcetypes that might help us –df for disk space, vmstat for memory, and cpu. Let’s start with disk space – first let’s narrow down to the disk space data by clicking on “df”.
• #28: Now we can look at our disk space over time with just a couple of clicks – click on the field PercentUsedSpace, then click on Maximum value over time.
• #29: Well, we’re fine on disk space – never much higher than 70 percent full. Let’s try checking our CPU by changing the search to “host=webserver-01 sourcetype=cpu”.
• #30: Now, like we did with disk space, scroll down to PercentUserTime and then click Maximum value over time. Ah-ha! Looks like the CPU has been pegged at 100%.
• #31: We found one issue, but there are 503 errors coming from all the servers, so maybe we should keep looking in case something else is going on. Let’s try just searching for all errors by searching for the word “error”. Wow, that’s a lot of errors! Let’s check the hosts like we did before by clicking on “hosts” in the field list. We’ll see that the errors mostly seem to be coming from a test machine. Let’s exclude that machine from the search by going to any event with the test machine, clicking on the host name, then choosing “Exclude from search”. The logs that are left indicate a database problem.
• #32: Since the errors were about not being able to write log files, let’s start by looking at disk space. Search for host=mysql-02 sourcetype=df, then click on PercentUsedSpace and Maximum value over time, like we did before.
• #34: There we have it – a full disk. But wouldn’t it be better if we were more proactive? Let’s start by creating an alert that will let us know when we’re having issues with disk space on this machine. Timechart is great for looking at data over time, but for this alert, we just want to know if we ran out of disk space in the last hour, so let’s change “timechart” to “stats” to get a single value. It’s also helpful sometimes to rename fields to make them easier to read, so let’s add “as maxused” to the end of our search, which renames the field with our maximum value from max(PercentUsedSpace) to maxused
• #35: Now we can save this search as an alert. We can give it any name and decide if we want to run it on a schedule or real-time. Most of the time we’d want to use a schedule. Next, we’ll set a custom trigger condition using our maxused field – we’ll set the alert to fire if maxused is greater than 85% And because we don’t want to get flooded with alerts, we’ll turn on throttling so alerts are suppressed if we get multiple alerts. And finally, we just have to choose how to be notified. Click on “Add Actions”. “Add to Triggered Alerts” will put notifications in Splunk’s triggered alerts page, which you can then monitor for issues. You could run a script, maybe to rotate logs if disk space is running low. You could send an email, or do an HTTP Post to a URL, which could be helpful if you wanted to automate opening a ticket or a call to an alerting service.
• #37: Alerts are great, but maybe we also want to provide the NOC with a report they can check when they receive those alerts. So let’s start with the same search that we used for the alert. . Click the “Visualization” tab, then click “Line” and choose “Radial Gauge” from the menu that comes up. There, that’s easy to see. We could change the color ranges to be a little more appropriate for our disk space usage by clicking “Format”, then “Color Ranges”.   Great, let’s save this as a report! Click “Save As”, then “Report”. Give it a title and save it.
• #38: So what’s the difference between a report and a dashboard? Well, a report consisted of a single search. But wouldn’t it be nice to see the results of more than one search at a time? Let’s build a quick dashboard that just shows the disk space used by some MySQL servers. We’ll start with the report that we just built. To add this report to a new dashboard, click “Add to Dashboard”, give the new dashboard a name and save it, then click “View Dashboard”. Well, so far so good, but it looks just like the report. In order to make it more useful, let’s add another gauge for one of the other MySQL servers. Click “Edit” then “Add Panel”. Under “Add Prebuilt Panel”, there’s a panel called “A Sample Panel”. Add that panel to your dashboard. It shows disk space for mysql-03. Rearrange the panels any way you want by dragging and dropping.
• #39: Now let’s look at some other kinds of dashboards that you could build that could help you with troubleshooting. Click on the “Dashboards” menu, then click “Website Health” on the list of dashboards.   This dashboard puts a lot of information in one place, but it’s not really that much different than the one we started building – it just more panels.   At the top, we have a quick look at the number of errors and the average request time in the last five minutes. The next row has some time charts showing the values of metrics over time. The bottom row has some pie charts. It looks like there’s some trouble – is this one of the issues we’ve already looked at? We can see in the pie chart labeled “Errors by Server” at the bottom right that webserver-01 has more errors than the others and in the “Average Request Time by Server over Time” time chart that webserver-01 has the longest response times. Let’s click on the pie chart to get more information.
• #40: This dashboard is even simpler than the Website Health one, with the average and max CPU, disk, and memory used over time for each server. And with the data organized into a single dashboard, it’s easy to see that same CPU issue that we identified earlier, but we got here a lot faster using dashboards!
• #41: s
• #42: Monthly deploys to 900 per day– family search Academy Sports and Outdoors increased uptime to 99.7% Other areas that people are seeing value in is with: Reduce/avoid downtime Gain control over costs, capacity, user experience User and usage analytics to support real-time business decision-making Real-time and historical data analysis for trending and pattern detection
• #43: Next steps! You can take the USB key with you and use the data in the application on it to work with the features we’ve already looked at, or any of the other features that Splunk offers! You can also try out Splunk Cloud with your own data. Splunk Cloud gives you all the benefits of Splunk in a SaaS product. You can use the Splunk installer from your USB key or download a free copy from splunk.com – load in your own data and see what you can find! You can also see Splunk’s training options by going to splunk.com/education – we’ve barely scratched the surface in this session!