Splunk ITSI Sandbox Guidebook
4 likes1,422 views
This document provides instructions for touring example Glass Tables in an ITSI sandbox. It describes Glass Tables as highly customizable dashboards for visualizing ITSI services, KPIs, and health scores. Two specific Glass Tables are highlighted: 1) The "Buttercup Games Business Process" table shows the high-level business process status for context. 2) The "On Line Transaction Service" table shows a detailed customer-facing service map including transaction flows, component relationships, and critical metrics to help support personnel understand dependencies. The goal of the tour is to illustrate how Glass Tables can be tailored for different users and use cases, from executive summaries to technical deep dives.
![[cb22] Hayabusa Threat Hunting and Fast Forensics in Windows environments fo...](https://cdn.slidesharecdn.com/ss_thumbnails/d2t2s052022-10-28-dfirandthreathuntingwithwindowseventlogszachmathisversion3-230103072306-eb57bdc0-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Splunk ITSI Sandbox Guidebook (20)
Ad
More from Splunk (20)
Recently uploaded (20)
Splunk ITSI Sandbox Guidebook
- 3. 3 1 - Fly Over the Product For the traveller who is in a hurry, who wants the 30,000-foot view, this is the section for you! It is also the best place to begin, for the student who is largely unfamiliar with IT Service Intelligence. Instructions 1. After logging into Splunk, click on "Product Tour" 2. Click through the slides to preview services, entities, KPIs, thresholding, Deep Dives, Multi-KPI Alerts, Notable Events and the Service Analyzer 3. These topics, and more, are covered in more detail in the following chapters
- 11. 11
- 12. 12 ITSI represents a new way of dealing with IT Service challenges: • Data-driven approach uses ALL IT Data - events, metrics, logs, structured, unstructured, from-the-device, from-the-wire, etc. • Service-awareness provides actionable insights into high-visibility services • Customized contextual visualizations can be tailored for any person or group: highly technical to business-oriented • Mitigate problems before they impact customers
- 14. 14 3. Select Buttercup Games Business Process This Glass Table shows the high-level business process status Buttercup Games. It could be used by service owners, executive management or others who need to quickly understand the "big picture". 4. Select On Line Transaction Service This Glass Table shows a detailed view of a customer-facing service, including transaction flow, component relationships and dependencies, and critical health scores and metrics of key service points along the way. It makes excellent use of a pre- existing drawing, with live ITSI "widgets" placed strategically on top. This Glass Table would helpful for NOC, Tier 1&2 and similar support personnel who need to understand the complex relationships of all the service components supporting an important business service. 5. Select Buttercup Games Online Store This Glass Table shows a streamlined view of Buttercup Games' customer-facing service-- the "online store" summarized in the "Buttercup Games Business Process" Glass Table. This view provides more detail of the underlying technical services, their dependencies, and the overall transaction flow. It uses native Glass Table drawing tools, as well as service and KPI widgets, which display health and metric values live (updating over time). These widgets have configurable drill-down capabilities, including the ability to navigate to other, even-more-detailed Glass Tables. For example, if you click on the widget next to Web Tier, you will navigate to ... 6. Web Tier This Glass Table represents a more detailed visualization of the KPIs, overall Web Tier health score, and the health score of its dependent service, Middleware. Such Glass Tables allow technical personnel to quickly troubleshoot problems by being able to drill down to the detailed technical metrics which matter. 7. Select Buttercup Games Online Store (again) Several drill-down options are available when a widget is clicked. Click on the widget next to Database; this will navigate to a Deep Dive.
- 16. 16 4 - Troubleshooting Tour with Glass Tables and Deep Dives This section describes a possible problem scenario, and how ITSI could be used to efficiently troubleshoot to find root cause. This would typically be driven by a NOC or Tier 1 or Tier 2 support person. We're going to "set up" the failure scenario and first see how Glass Tables can accelerate the troubleshooting process, then continue isolating root cause with Deep Dives. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Glass Tables (Ch. 3) About the event generator... In order to make the ITSI Sandbox more interesting to play in, an event generator is included which continuously generates a simulated stream of realistic machine events, including web access, database, Linux metrics (from the *nix Technology Add-on) and others. Included in this stream of events are two failure scenarios, showing a sequence of failures and resulting service degradations, each scenario repeating hourly. Typically, the initial failures for each scenario occur at the top of the hour, and reset back to "OK" around the top of the next hour. However, the event generator (eventgen) timing may not be precise. The failure scenarios may occur at slightly different times from hour to hour, and may vary from sandbox to sandbox. Thus, within the Sandbox, it is impossible to predict exactly how the health scores and KPIs will appear, during any specific hour. This makes it difficult to setup a "clean" failure simulation. Please pardon any eventgen inconsistencies. We decided to put most of our effort into developing ITSI-- not an event generator.
- 17. 17 Instructions 1. Navigate to the Glass Table called, Buttercup Games Online Store: a. Click on Glass Tables in the upper menu bar to navigate to the page, Saved Glass Tables b. Click on Buttercup Games Online Store to navigate to this Glass Table 2. Modify the view time by clicking on the time picker in the upper right corner. In the pop-up window, type in an explicit time from the past, such as XX:15.0 from the previous hour (or the hour before that, etc). Be sure to use the correct HH:MM:SS.sss format (example: "10:15:00.0")
- 19. 19 5. The scenario: Customer Care has informed us that customers are calling to complain when they try to purchase through the Online Store; they are seeing slow response and occasional errors. The problems seem to be affecting both web-based and mobile-based customers. 6. Based on just the reports that the customer-facing web-based service is having problems, most support persons would begin troubleshooting "from the top"-- the web and mobile tiers in this case. If no obvious problems were found, they would proceed down the service dependency tree-- to the middleware tier, etc. 7. But using a Glass Table such as "Buttercup Games Online Store" provides instant and context-relevant visibility into service health scores and important KPIs, all in one place. In the above example, which supporting tier seems to be in distress? (Database) By being able to visualize the relevant services and their health scores, we have the ability to immediately focus our troubleshooting on the areas that are degraded. This can save huge amounts of time and greatly reduce the time required to find root cause. 8. On your Sandbox Glass Table, click on the widget beneath Database to drill down into the Database tier to continue the troubleshooting exercise. (Select Leave This Page if prompted)
- 20. 20 (Now in DB Deep Dive) 9. Click on the > next to Focus to collapse the service tree navigator panel on the right side; we will explore this feature later. 10. Change the Primary Time Range to Last 2 Hours by clicking on the time picker in the lower left corner: a. In the Relative section, type in "2" and select Hours Ago b. Click Apply
- 21. 21 (The Deep Dive should now displays some "mostly green" and some "mostly red" – your screen may not look exactly like the below, but you should see a point where things go from green to red and red to green) 11. We are looking at the Aggregated Health Score (top swimlane) and KPIs for the DB Service, across a time range which shows the service moving from "healthy" to "not healthy". 12. Slowly mouse over the swim lanes to compare values at various points in time. 13. Click the checkbox in the upper left to select all swimlanes, and use the “Bulk Actions” menu to “Show State Thresholds” or "Hide Thresholds", toggling to compare the swim lanes with and without the threshold colors/states overlaid. 14. Note that the ServiceHealthScore in the top swimlane is an aggregation of the service's KPIs and dependent services, ranging from 100-0. When did the health score begin to deteriorate, and which KPI(s) may have been part of the root cause?
- 22. 22 15. Click on the name-box for Storage Free Space: % System, then drag it upwards to reposition this swimlane. 16. A few of the swimlanes are continuously green, indicating that they are not particularly helpful in our troubleshooting exercise ("CPU Utilization", "Memory Free", etc). Click on the checkbox in the upper left corner to unselect all swimlanes, then select the checkbox for CPU Utilization: % User and Memory Free. Select Bulk Actions -> Delete to (temporarily) remove this swimlane from our Deep Dive. 17. Click on the darker blue tile within the DB Service Errors swimlane to reveal "raw errors" from the underlying Splunk search. Click on Hide Events to dismiss.
- 23. 23 18. Mouse over the Storage Free Space: % System swimlane, in the place where it goes from green to red. Note the high & low metric values shown for the swimlane, and that this metric has gone to 0%, indicating that a filesystem is full. 19. Click anywhere within the Storage Free Space: % System swimlane to reveal an options popup. Select Add Overlay as Lane. (Three new swimlanes are added at the bottom, representing the separate KPI values for the individual entities (hosts) which comprise this KPI) 20. Which host/server is suffering from a filesystem-full condition? (mysql-02)
- 25. 25 5 - Dive in to Deep Dive Deep Dives allow KPI metrics and health scores to be compared in side-by-side swimlanes, which allows trends and correlations to be more easily and quickly discovered. This chapter explores Deep Dives and how they can be used. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Troubleshooting (Ch. 4) also goes into Deep Dives Instructions 1. Navigate to the Deep Dive called, DB Deep Dive: a. Click on Deep Dives in the upper menu bar to navigate to the page, Saved Deep Dives b. Click on DB Deep Dive to navigate to this Deep Dive 2. Click on the > next to Focus to collapse the service tree navigator panel on the right side; we will explore this feature later. 3. Select an arbitrary time range by clicking on the Primary Time Range menu option at the bottom right; it functions like a standard Splunk search bar time picker 4. Zoom in to a tighter time range in the current view by click-holding anywhere in the swimlanes, then dragging horizontally to select the range. 5. Toggle the threshold health score colors by clicking on the checkbox in the upper left corner to select all swimlanes, then Bulk Actions -> Show State Thresholds/Show Level Thresholds/Hide Thresholds. 6. Click on the > next to Focus to open the service tree navigator panel on the right side. a. Click on a service node to navigate up and down the dependency tree of services b. After clicking on a service node, note that those service's KPIs are listed below. c. Click on the + on a listed KPI to add it to the current swimlanes d. Click on the > next to Focus to collapse the service tree navigator panel on the right side 7. Mouse-over the name-box for any swimlane to reveal the "options wheel", then select it to view available options:
- 26. 26 8. The student is encouraged to explore these options, which are covered in more detail at http://docs.splunk.com/Documentation/ITSI/latest/User/DeepDives 9. Click-hold on the name-box for any swimlane, and then drag it vertically to reposition this swimlane. 10. Click on the darker blue tile within the DB Errors (or any "event"-style) swimlane to reveal "raw errors" from the underlying Splunk search. Click on Hide Events to dismiss. 11. To save a Deep Dive after modifying the layout and/or visualization options, click on the Edit menu option in the upper right corner, then select Save 12. To compare the current time range against a different time range, click on Compare to ... in the lower left corner, then select a comparison time range. This causes each KPI to display twin swimlanes: primary time range above comparison time range. Note that when mousing over the swimlanes, the time display at the top now shows both times. 13. To dismiss the "twin" lanes display, deselect the checkbox next to Compare to ... in the lower left corner
- 28. 28 7 - Tour Multi-KPI Alerts Multi-KPI Alerts are Correlation Searches which can combine any KPIs to create meaningful, actionable alerts, using multiple correlation factors such KPI threshold indications, length of time in this state, time-of-day, and others. Multi-KPI alerts can find not just "failures", but early "canary in the coal mine" indications that the service is becoming unstable; it is possible to find problems BEFORE they impact customer-facing services. When a Multi-KPI Alert fires, it creates a Notable Event; it could also execute a script and/or send email. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Troubleshoot with Glass Tables and Deep Dives (Ch. 4) Instructions 1. Navigate to the Deep Dive called, DB Deep Dive: a. Click on Deep Dives in the upper menu bar to navigate to the page, Saved Deep Dives b. Click on DB Deep Dive to navigate to this Deep Dive 2. Click on the > next to Focus to collapse the service tree navigator panel on the right side. 3. Change the Primary Time Range to Last 2 Hours by clicking on the time picker in the lower left corner: a. In the Relative section, type in "2" and select Hours Ago b. Click on Apply
- 30. 30 4. We are looking at the Aggregated Health Score (top swimlane) and KPIs for the DB Service, across a range of time which shows the service moving from "healthy" to "not healthy". 5. Click/drag across a narrower range of time when the service transitions from green to yellow/orange. 6. Click on the checkbox in the upper left to unselect all swimlanes, then select the checkboxes next to the KPI swimlanes which were involved in this outage (turned red) during this period, such as Storage Free Space, DB Service Queries & DB Service Response Time. 7. In the upper left, select Bulk Actions -> Create Multi KPI Alert
- 32. 32 ITSI provides a sophisticated array of options for setting up Multi-KPI Alerts, also known as Correlation Searches. The goal is to allow the creation of useful alerts based on correlations of several KPIs-- fewer "noise" alerts, more actionable alerts. Here are some of the features and capabilities: • Control the range of time to correlate the KPIs across (time-picker in the upper right corner) • Add KPIs from any service • Create a KPI based on the aggregate health score of the KPIs, or on Status over time (upper right corner) • Re-weight the KPIs using the Importance sliders (lower right corner)
- 34. 34 6 - Dive in to the Notable Events Review When a Multi-KPI Alert fires, it creates a Notable Event. The Notable Events Review is Splunk's next-generation event management console. Notable Events Review provides a quick way view, sift and organize events, allowing us to triage, manage and streamline workflow more effectively. It has the ability filter Notable Events and events from other event management sources, based on various criteria, such as Severity, Status, Service and others. It also allows events to be modified, to change Owner, Severity, Status, and/or add comments. Events can also have workflow actions associated with them, to allow an operator the ability to quickly hit troubleshooting options, execute mitigation scripts, or open a "real" Incident Management trouble-ticket. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Tour Multi-KPI Alerts (Ch. 7) Instructions 1. Navigate to the Notable Events Review by clicking on Notable Events Review in the upper menu bar 2. Click on Show Timeline to reveal the timeline 3. See details for an event: Click on any event to open the Details panel on the right. Details include which KPIs contributed, and which services might be affected, as well as the ability to examine these in more detail in a Deep Dive. Severity, Status and Assignment can also be changed directly. 4. Modify Severity for an event: Click on the Severity dropdown at upper left of the Details panel, choose a different Severity 5. Choose a workflow action: Click on </> icon in upper right corner of Details panel to reveal the workflow options Custom workflow actions can be created for each type of Notable Event, to streamline workflow actions. These can be additional troubleshooting or mitigation scripts, or something as basic as opening a 'real' incident ticket. 6. Dismiss Details: Click on the X in the upper right corner of Details panel to dismiss 7. Filter the Notable Events by Severity: Click on the "gear wheel" in the upper right corner, then choose Edit Filter Settings. Click Add Filter, and then Severity. Click in the Severity box to see and choose from a list of the available severity levels.
- 35. 35 8. Filter by Status, Owner, Service, Time Range, Name ("Title") or freeform search criteria by adding other filters to your filter settings. 9. Click Done to (re)apply search filter criteria 10. Change view options: Click on the "gear wheel" in the upper right corner, then choose Edit View Settings. Select Viewing Option -> Prominent and Deduplication -> On, then Done An Event Count column has now been added for deduplicated events, and Severity color is now more 'prominent' 11. Add, remove or re-order columns: Click on the "gear wheel" in the upper right corner, then choose Edit View Settings. In Columns Shown, click X to remove a column, click + Add Column to add a column, or click/drag a column to re-order how it is viewed. 12. To sort the event rows: Click on the V chevron next to Sort By: (left side, above rows), then select a column to sort by. Toggle the sort order (ascending/descending) by clicking on the vertical arrow next to Sort By: More details are available here: http://docs.splunk.com/Documentation/ITSI/latest/User/NotableEventsReview The Notable Events Review allows an operator to: • Quickly and effectively find, deduplicate and manage just the events they want • Tie workflow actions to events, to streamline operations • Manage ITSI Notable Events and events from other sources
- 36. 36 7 - Dive in to the Service Analyzer The Service Analyzer is a "Big Picture" view of all services, and the "most interesting" KPIs (i.e., KPIs with degraded health scores). It is "no frills", designed for NOCs, Tier 1 or 2 support, and others who need a high level view of all services/KPIs, or a subset. It also provides a launching point for exploring Services, KPIs and Entities in more detail. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) Instructions 1. Navigate to the Service Analyzer by clicking on Service Analyzer in the upper menu bar, then choosing Default Service Analyzer. 2. Click on Middleware Service to navigate to its service health page. Here you can see the service tree on the left, the KPIs in the center, and the entities associated with a selected KPI on the right. 3. Click on DB Service in the left service tree panel to navigate to this service 4. From Service Health, you can also navigate to a deep dive containing the KPIs for that service using the link at the top of the KPI table in the center of the page. a. Notice the deep dive has been built for you on the fly, containing all the KPI’s associated with that service 5. Click on Storage Free Space: % System and notice that you now have a table on the right that shows the entities associated with this KPI. 6. Click on mysql-02 in the entity list to navigate to its Entity Health page. a. This is an entity-centric view, showing information about a specific entity, including which services and KPIs it supports. b. Clicking on a service name will navigate to that service health page
- 37. 37 8 - Side Trip to OS Host Details 7. If you are using one or more ITSI modules, relevant module dashboards for this entity will show up in the left-side Modules panel. In this case, "OS Host Details" is listed. More details about modules are available here: http://docs.splunk.com/Documentation/ITSI/latest/IModules/AboutITSIModules 8. Click on OS Host Details to navigate to this page. a. The OS Host Details section offers several dashboards with detailed status, performance and event reports. b. OS Host Details can also be accessed in Deep Dive. c. More details about the Operating System (OS) Module are available here: http://docs.splunk.com/Documentation/ITSI/latest/IModules/AbouttheOperatingSystemModule 7b - And Back to Service Analyzer 9. Navigate back to Service Analyzer 10. Click in the Select service(s) to monitor box to select & show only certain services 11. Click on the "Option Wheel" next to Top ... Services to control how many services are shown 12. Click on the "Option Wheel" next to Top ... KPIs to control how many KPIs are shown, and to select which KPIs are shown 13. To create an ad-hoc Deep Dive: a. Mouse over one or more Service or KPI tiles, then select the checkbox in the upper right corner of the tile b. Click Drilldown to Deep Dive Service Analyzer provides a "Big Picture" view of all services and the "most interesting" (not green) KPIs. It is also a launching point for exploring Services, KPIs and Entities in more detail, as well as for creating ad-hoc Deep Dives with selected KPIs.