SplunkLive! Analytics with Splunk Enterprise - Part 2
9 likes2,020 views
This document discusses Splunk's data modeling capabilities and how they enable faster analytics over raw machine data. It introduces data models, which allow domain knowledge to be shared and reused. Data models map data onto hierarchical structures and enable non-technical users to build reports without using the Splunk search language. The document covers best practices for building data models and how pivot searches are generated from the underlying data model objects. It also discusses managing, securing, and accelerating analytics with data models.
![Hurdles
index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67
clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24
clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND
t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers
min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type =
if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart
count by type| streamstats sum(*) as *
•
Simple searches easy… Multi-stage munging/reporting is hard!
•
Need to understand data’s structure to construct search
•
Non-technical users may not have data source domain knowledge
•
Splunk admins do not have end-user search context](https://image.slidesharecdn.com/analyticspart2splive-131031171328-phpapp02/85/SplunkLive-Analytics-with-Splunk-Enterprise-Part-2-7-320.jpg)
![Using the Splunk Search Language
Pivot Search String
| pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …]
Example:
| pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status
AS "status" SORT 0 status
Behind the scenes:
sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
| stats count AS "Count of HTTP_Request" by "status"
| sort limit=0 "status" | fields - _span
| fields "status", "Count of HTTP_Request"
| fillnull "Count of HTTP_Request"
| fields "status" *](https://image.slidesharecdn.com/analyticspart2splive-131031171328-phpapp02/85/SplunkLive-Analytics-with-Splunk-Enterprise-Part-2-31-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to SplunkLive! Analytics with Splunk Enterprise - Part 2 (20)
![[WSO2Con USA 2018] Patterns for Building Streaming Apps](https://cdn.slidesharecdn.com/ss_thumbnails/wso2conusa2018presentation-patternsforbuildingstreamingapps-180717043111-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Splunk (20)
Recently uploaded (20)
SplunkLive! Analytics with Splunk Enterprise - Part 2
- 1. Copyright © 2013 Splunk Inc. Analytics with Splunk Enterprise – Part 2
- 2. Legal Notices During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. ©2013 Splunk Inc. All rights reserved.
- 3. Analytics Big Picture Pivot Enables non-technical users to build complex reports without the search language Data Model Provides more meaningful representation of underlying raw machine data Analytics Store Acceleration technology delivers up to 1000x faster analytics over Splunk Enterprise 5
- 4. Agenda • Context • Data Models 101 • Building Data Models • From Data Models to Reports • Management, Acceleration, and Beyond • Q&A
- 5. Context
- 6. Splunk Search Language search and filter | munge | report | clean-up sourcetype=access_combined source = "/home/ssorkin/banner_access.log.2013.6.gz" | eval unique=(uid + useragent) | stats dc(unique) by os_name | rename dc(unique) as "Unique Visitors" os_name as "Operating System"
- 7. Hurdles index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as * • Simple searches easy… Multi-stage munging/reporting is hard! • Need to understand data’s structure to construct search • Non-technical users may not have data source domain knowledge • Splunk admins do not have end-user search context
- 8. Data Model Goals • Make it easy to share/reuse domain knowledge • Admins/power users build data models • Non-technical users interact with data via pivot UI
- 10. What is a Data Model? A data model is a search-time mapping of data onto a hierarchical structure Encapsulate the knowledge needed to build a search Pivot reports are build on top of data models Data-independent Screenshot here
- 11. A Data Model is a Collection of Objects Screenshot here
- 12. Objects Have Constraints and Attributes Screenshot here
- 13. Child Objects Inherit Constraints and Attributes Screenshot here
- 14. Child Objects Inherit Constraints and Attributes
- 16. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes
- 17. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) – Requiressearchstringattributes • Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
- 18. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) Requiressearchstringattributes Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
- 19. Object Attributes Auto-extracted – default and predefined fields Eval expression – a new field based on an expression that you define Lookup – leverage an existing lookup table Regular expression – extract a new field based on regex Geo IP – add geolocation fields such as latitude, longitude, country, etc.
- 20. Object Attributes Set field types Configure various flags Note: Child object configuration can differ from parent
- 21. Best Practices Use event objects as often as possible – Benefit from data model acceleration Resist the urge to use search objects instead of event objects!! – Event based searches can be optimized better Minimize object hierarchy depth when possible – Constraint based filtering is less efficient deeper down the tree Event object with deepest tree (and most matching results) first – Model-wide acceleration only for first event object and its descendants
- 22. Warnings! Object constraints and attributes cannot contain pipes or subsearches A transaction object requires at least one event or search object in the data model Lookups used in attributes must be globally visible (or at least visible to the app using the data model) No versioning on data models (and objects)!
- 23. From Data Models to Reports
- 24. Using the UI Subhead Count of http_success events, split by useragent events fields
- 25. Under the Hood: Object Search String Generation Event Object Syntax: <constraints search> | <my attribute definitions> Example: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
- 26. Under the Hood: Object Search String Generation Search Object Syntax: <base search> | <my attribute definitions> Example: _time=* host=* source=* sourcetype=* uri=* status<600 clientip=* referer=* useragent=* (sourcetype=access_* OR source=*.log) | eval userid=clientip | stats first(_time) as earliest, last(_time) as latest, list(uri_path) as uri_list by userid | earliest=* latest=* uri_list=*
- 27. Under the Hood: Object Search String Generation Transaction Object Syntax: <objects to group search> | transaction <group by fields> <group by params> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* | transaction clientip useragent | eval landingpage=mvindex(uri_path,1) | eval exitpage=mvindex(uri_path,-1)
- 28. Under the Hood: Object Search String Generation Child Object Syntax: <parent object search> | search <my constraints> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* status=2* | <my attribute definitions>
- 29. Using the Splunk Search Language Object Search String | datamodel <modelname> <objectID> search Example: | datamodel WebIntelligence HTTP_Request search Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
- 30. Under the hood: Pivot Search String Generation Pivot search = object search + filters + reporting + formatting Example: (sourcetype=access_* OR sourcetype=iis*) status=2* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Sucess" by ”useragent" | sort limit=0 "useragent" | fields - _span | fields "useragent" "Count of HTTP_Success" | fillnull "Count of HTTP_Success" | fields "useragent" *
- 31. Using the Splunk Search Language Pivot Search String | pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …] Example: | pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status AS "status" SORT 0 status Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Request" by "status" | sort limit=0 "status" | fields - _span | fields "status", "Count of HTTP_Request" | fillnull "Count of HTTP_Request" | fields "status" *
- 32. Warnings • | datamodel and | pivot are generating commands – They must be at the beginning of the search string • Use objectIDs NOT user-visible object names
- 34. Data Model on Disk Each data model is a separate JSON file Lives in <myapp>/local/data/models (or <myapp>/default/data/models for pre-installed models) Has associated conf stanzas and metadata
- 35. Editing Data Model JSON At your own risk! Models edited via the UI are validated Manually edited data models: NOT SUPPORTED Exception: installing a new model by adding the file to <myapp>/<local OR default>/data/models is probably okay
- 36. Deleting a Data Model Use the UI for appropriate cleanup Potential for bad state if manually deleting model on disk
- 37. Interacting With a Data Model Use data model builder and pivot UI – safest option! Use REST API – for developers (see docs for details) Use | datamodel and | pivot Splunk search commands
- 38. Permissions Data models have permissions just like other Splunk objects Edit permissions through the UI
- 39. Data Model Acceleration Admin or power user Backend magic Acceleration Non-technical user Run search using on-disk acceleration Run a pivot report No acceleration Kick off ad-hoc acceleration and run search
- 40. Model-Wide Acceleration Only accelerates first eventbased object and descendants Does not accelerate search and transaction-based objects Pivot search: | tstats count AS "Count of HTTP_Success" from datamodel="WebIntelligence" where (nodename="HTTP_Request") (nodename="HTTP_Request.HTTP_Success") prestats=true | stats count AS "Count of HTTP_Success”
- 41. Ad-Hoc Object Acceleration Kick off acceleration on pivot page (re) load for non-accelerated models and search/transaction objects Amortize cost of ad-hoc acceleration over repeated pivoting on same object Pivot search: | tstats count AS "Count of HTTP_Success" from sid=1379116434.663 prestats=true | stats count AS "Count of HTTP_Success”
- 42. Acceleration Disclaimers Works with search-head pooling – we collect on indexers Cannot edit accelerated models
- 43. Thank You
Editor's Notes
- #4: Splunk 6 takes large-scalemachine data analytics to the next level by introducing three breakthrough innovations:Pivot – opens up the power of Splunk search to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and making the data more useful to broader base of non-technical usersAnalytics Store – patent pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical operations, up to 1000x faster than Splunk 5Let’s dig into each of these new features in more detail.
- #5: What is Data Model, and why do I care?Building a Data ModelManagement, Acceleration, and BeyondThe Future!Q&A
- #7: -The Splunk search language is very expressive. - Can perform a wide variety of tasks ranging from filtering to data munging and reporting- There are various search commands for complex transformations and statistics (e.g. correlation, prediction etc)
- #8: What does the search do?Basically, first it normalizes the individual accesses, which should be representable as a model object.Next it aggregates by guid to create an "instance" object, which should be representable in a DM.It calculates a field on that instance object, "type".Then it builds a timechart. of those, using a special "_time" value.Low overhead to start but learning curve quickly gets steepObtaining website usage metrics should not require understanding Apache vs IIS formatAdmins won’t know apriori what questions are being asked of the data…so they can’t provide canned dashboards for all scenariosBackup search for example: eventtype=pageview | eval stage_2=if(searchmatch("uri=/download*"), _time, null()) | eval stage_1=if(searchmatch("uri=/product*"), _time, null()) | eval stage_3=if(searchmatch("uri=*download_track*"), _time, null()) | stats min(stage_*) as stage_* by cookie | search stage_1=* | where isnull(stage_2) OR stage_2 >= stage_1 | where isnull(stage_3) OR stage_3 >= stage_2 | eval stage = case(isnull(stage_2), "stage_1", isnull(stage_3), "stage_2", 1==1, "stage_3") | stats count by stage | reverse | accum count as cumulative_count | reverse | streamstats current=f max(cumulative_count) as stage_1_count last(cumulative_count) as prev_count
- #12: What are the important “things” in your data?E.g. WebIntelligence might haveHTTPAccessHTTPSuccessUser SessionHow are they related?There’s more than one “right” way to define your objects
- #13: Constraints filter down to a set of a dataAttributes are the fields and knowledge associated with the objectBoth are inherited!
- #14: A child object is a type of its parent object: e.g. An HTTP_Success object is a type of HTTP_AccessAdding a child object is essentially a way of adding a filter on the parentsA parent-child relationship makes it easy to do queries like “What percentage of my HTTP_Access events are HTTP_Success events?”
- #17: Constraints are essentially the search broken down into a hierarchy, attributes are the associated fields and knowledge
- #18: Arbitrary searches that include transforming commands to define the dataset that they representFix example here? TODO
- #19: Enable the creation of objects that represent transactionsUse fields that have already been added to the model via event or search objects
- #20: This is how we capture knowledge
- #21: Required: Only events that contain this field will be returned in PivotOptional: The field doesn't have to appear in every event Hidden: The field will not be displayed to Pivot users when they select the object in PivotUse this for fields that are only being used to define another attribute, such as an eval expression Hidden & Required: Only events that contain this field will be returned, and the field will be hidden from use in Pivot
- #22: Be careful about lookup permissions – must be available in the context where you want to use them
- #40: Divanny will help make this slide MUCH prettier
- #41: This could be more slides,more details
- #43: This could be more slides,more details