Lesser known-search-commands

Lesser Known Search
Commands
Kyle Smith | Integration Developer / Aplura LLC
September 26, 2017 | Washington, DC

During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
Forward-Looking Statements

Me
• Integration Developer with Aplura, LLC
• Working with Splunk for ~8 years
• Written many Public Splunk Apps (on splunkbase.splunk.com)
• Current Member of the SplunkTrust
• Wrote the “Splunk Developer’s Guide” - introduction to Splunk App Development
• Active on #splunk on IRC, answers.splunk.com, and Slack
• Co-leader of Baltimore Usergroup
• My Handle is “alacercogitatus” or just “alacer”

You
• Splunk
• Admin
• User
• Architect
• Evangelist
• Sales Engineer
• Anybody
• Want to learn about new search commands
• Enjoy Piña Coladas, getting caught in the rain
(well maybe not)
• Intermediate experience with SPL (know how to
“stats”)
Goals
• Show/expose you to possibly new commands
• Won’t become “expert” on these commands
• Take actionable items back to your business to “try new
things”

Agenda
• Administrative/Generating
• Iterative
• Statistics
• SPLacks
NOTE: These commands have been verified for Splunk 7.0.
Any other version may or may not work correctly.

Administrative
(generating)
Commands
• rest, makeresults, gentimes, metasearch,
metadata, untable
• Commands that do not pull full (raw) events or
are used to interact with Splunk
• Provide information about data or the Splunk
server(s)
• These are all “generating” commands

rest
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rest
The rest command reads a Splunk REST API endpoint and returns the resource data
as a search result.1
• MUST be the first search command in a search block
• Is “time agnostic” - It only queries - so time is not a factor in execution
• Limits results to what the requesting user is allowed to access

rest
This endpoint pulls the currently logged in user’s authorization context.
With this information, you could restrict specific dashboards or searches by
role, even more granular than the pre-built permissions.

makeresults
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Makeresults
Generates the specified number of search results. If you do not specify any of the
optional arguments, this command runs on the local machine and generates one
result with only the _time field. 1
• New in 6.3
• Easy way to “spoof” data to experiment with evals, and other SPL commands
• Fast, lightweight
• Use it to restrict a search using it in a subsearch

gentimes
• Useful for generating time buckets not present due to lack of events within those time buckets
• Must be the first command of a search ( useful with map, or append)
• “Supporting Search” - no real use case for basic searching
Generates timestamp results starting with the exact time specified as start time. Each result describes
an adjacent, non-overlapping time range as indicated by the increment value. This terminates when
enough results are generated to pass the endtime value.1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Gentimes

metasearch
Retrieves event metadata from indexes based on terms in the <logical-expression>.
Metadata fields include source, sourcetype, host, _time, index, and splunk_server. 1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Metasearch
• Useful for determining what is located in the indexes, based on raw data
• Does NOT present raw data
• Can only search on raw data, no extracted fields
• Can be tabled based on the metadata present
• Respects the time picker and default searched indexes

metadata
The metadata command returns a list of source, sourcetypes, or hosts from a
specified index or distributed search peer.
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Metadata
• Useful for determining what is located in the indexes, based on metadata
• Does NOT present raw data
• Does respect the time picker, however snaps to the bucket times of the found event

union

• Two different time ranges on same or disparate datasets
• Can be transforming or non-transforming and will do an append or
multisearch depending on location of the datasets.
• Provides optimized inter-leaving of datasets based on output of the datasets
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Union
Merges the results from two or more datasets into one dataset. One of the datasets
can be a result set that is then piped into the union command and merged with a
second dataset.

Iterative
• map, foreach
• Iterative commands do looping with events or
with fields in the search pipeline.
• EXPENSIVE commands

map
• Uses “tokens” ($field$) to pass values into the search from the previous results
• Best with either: Very small input set And/Or very specific search.
• Can take a long amount of time.
• Map is a type of subsearch
• Is “time agnostic” - time is not necessarily linear, and can be based off of the passed
events, if they include time.
The map command is a looping operator that runs a search repeatedly for each
input event or result. You can run the map command on a saved search, a
current search, or a subsearch.1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Map

map
It takes each of the results from the rest
search and searches the metadata in each
index. The results are returned as a table,
such as:

map
This can quickly show you where
there may be a sourcetype
misconfigured. Why would it be in two
different indexes (unless permissions
play a role)? The same can be done
for sources and hosts.

foreach
• Rapidly perform evaluations and other commands on a series of fields
• Can help calculate Z scores (statistical inference comparison)
• Reduces the number of evals required
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Foreach
Runs a templated streaming subsearch for each field in a wildcarded field list.1
Equivalent to ...
Can also use wildcards
foobar = This, foobaz = That è bar = This, baz = That

Statistics
• contingency, tstats, xyseries, streamstats,
eventstats, autoregress, untable
• These are commands that build tables, evaluate
sums, counts or other statistical values

untable
• Allows you to “undo” a table
• Generate a table using a field name as row values
• Great for doing additional evals and calculations after a transforming command
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Untable
Converts results from a tabular format to a format similar to stats output. This
command is the inverse of xyseries.

contingency
In statistics, contingency tables are used to record and analyze the relationship
between two or more (usually categorical) variables. 1
A contingency table is a table showing the distribution (count) of one variable in
rows and another in columns, and is used to study the association between the
two variables.
Contingency is best used where there is a single value of a variable per event.
• Web Analytics - Browsers with Versions
• Demographics - Ages with Locations or Genders
• Security - Usernames with Proxy Categories
• Great to compare categorical fields
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Contingency

xyseries
Converts results into a format suitable for graphing. 1
• Email Flow [ xyseries email_domain email_direction count ]
• One to Many relationships [ example Weather Icons ]
• Any data that has values INDEPENDENT of the field name
• host=myhost domain=splunk.com metric=kbps metric_value=100
• xyseries domain metric metric_value
• Works great for Categorical Field comparison
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Xyseries
Xyseries can help you build a chart with multiple data series.

eventstats
Adds summary statistics to all search results. 1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Eventstats

streamstats
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Streamstats
Adds cumulative summary
statistics to all search results
in a streaming manner.
The streamstats command
calculates statistics for each
event at the time the event is
seen.1

tstats
• Can only be used on indexed fields. EXTRACTED FIELDS WILL NOT WORK
• Quick way to access metadata or accelerated data (from data models or saved searches)
• Respects the time picker and default searched indexes
Use the tstats command to perform statistical queries on indexed fields in tsidx
files. The indexed fields can be from normal index data, tscollect data, or
accelerated data models.1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Tstats

mstats
• New in 7.0
• Can only be used on metric indexes
• Respects the time picker
Use the mstats command to analyze metrics. This command performs statistics
on the measurement, metric_name, and dimension fields in metric indexes.1
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Mstats
First, find all the metric names in the store

autoregress
• Allows advanced statistical calculations based on previous values
• Moving Averages of numerical fields
• Network bandwidth trending - kbps, latency, duration of connections
• Web Analytics Trending - number of visits, duration of visits, average download
size
• Malicious Traffic Trending - excessive connection failures
A Moving Average is a succession of averages calculated from
successive events (typically of constant size and overlapping) of a series
of values.
[1] https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Autoregress
Prepares your events for calculating the autoregression, or the moving average, by
copying one or more of the previous values for field into each event. 1

SPL
Hacks
(SPLacks)
• Eval function with a stats/timechart command
• Indirect Referencing
• These are not actual commands, but are more
along the lines of a hack

Inline
Timeline
Eval
• You
can
use
an
eval statement
in
a
timechart command
• There is a difference in significant
digits.
• Must rename the field.

Dynamic
Eval
(aka
Indirect
Reference)
• Not
a
search
command
• NOTE:
It’s
a
hack,
so
it
might
not
work
in
the
future.
• Works
great
for
perfmon sourcetypes,
but
can
be
applied
to
any
search
The Raw Event
07/29/2016 06:07:01.973 -0800
collection=DNS
object=DNS
counter="TCP Message Memory"
instance=0
Value=39176
The New Event
07/29/2016 06:07:01.973 -0800
collection=DNS
object=DNS
counter="TCP Message Memory"
instance=0
Value=39176
cnt_TCP_Message_Memory = 39176

Dynamic
Eval
-‐ Subsearch
• Not
a
search
command
• NOTE:
It’s
a
Splunk
hack,
so
it
might
not
work
in
the
future.

CLI
Commands
• Commands that are run from the command line to
help extract data, pull configurations, etc.

CLI
Commands
• $SPLUNK_HOME/bin/splunk reload monitor
– reloads monitor configuration, starts consuming data
– "add/edit/enable/disable monitor" changes since last reload
or Splunk restart
• Why?
– Adding a new app/datasource
– Don’t have to restart UF/HF for monitor configs (syslog comes
to mind)

CLI
Commands
• $SPLUNK_HOME/bin/splunk cmd pcregextest
– Useful for testing regular expressions for extractions
splunk cmd pcregextest mregex="[[ip:src_]] [[ip:dst_]]" ip="(?<ip>d+[[dotnum]]{3})" dotnum=".d+"
test_str="1.1.1.1 2.2.2.2"
Original Pattern: '[[ip:src_]] [[ip:dst_]]'
Expanded Pattern: '(?<src_ip>d+(?:.d+){3}) (?<dst_ip>d+(?:.d+){3})'
Regex compiled successfully. Capture group count = 2. Named capturing groups = 2.
SUCCESS - match against: '1.1.1.1 2.2.2.2'
#### Capturing group data #####
Group | Name | Value
--------------------------------------
1 | src_ip | 1.1.1.1
2 | dst_ip | 2.2.2.2

CLI
Commands
• $SPLUNK_HOME/bin/splunk btool
• Btool allows you to inspect configurations and what is actually being applied to your
sourcetypes
• splunk cmd btool --debug props list wunderground | grep -v "system/default"
/opt/splunk/etc/apps/TA-wunderground/default/props.conf [wunderground]
/opt/splunk/etc/apps/TA-wunderground/default/props.conf KV_MODE = json
/opt/splunk/etc/apps/TA-wunderground/default/props.conf MAX_EVENTS = 100000
/opt/splunk/etc/apps/TA-wunderground/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 30
/opt/splunk/etc/apps/TA-wunderground/default/props.conf REPORT-extjson = wunder_ext_json
/opt/splunk/etc/apps/TA-wunderground/default/props.conf SHOULD_LINEMERGE = true
/opt/splunk/etc/apps/TA-wunderground/default/props.conf TIME_PREFIX = observation_epoch
/opt/splunk/etc/apps/TA-wunderground/default/props.conf TRUNCATE = 1000000

CLI
Commands
• $SPLUNK_HOME/bin/splunk cmd python
• Executes the Splunk python interpreter
• $SPLUNK_HOME/splunk cmd splunkd print-modinput-config <mi> <mi>://<stanza>
• Prints the modularinput configuration
• $SPLUNK_HOME/splunk cmd splunkd print-modinput-config ga ga://kyleasmith.info |
/opt/splunk/bin/splunk cmd python
/opt/splunk/etc/apps/GoogleAppsForSplunk/bin/ga.py
• Returns and prints to the screen the modular input output (results of API calls)

Where to next?
Splunk Search and Performance Improvements
– Alex James, Manan Brahmkshatriya, Tuesday, Sept 26, 3:30 PM, Ballroom A
Regex in your SPL
– Michael Simko, Wednesday Sep 27, 11 AM, Room 150AB
Splunk, Docs, and You
– Rich Mahlerwein, Christopher Gales, Wednesday, Sep 27, 2:15PM, Room 103 AB
Speed up your searches
– Satoshi Kawasaki, Thursday, Sep 28, 10:30 AM, Room 150AB

Resources
and
Questions
• IRC
#splunk on
efnet.org (look
for
alacer)
• docs.splunk.com
• answers.splunk.com (I’m
alacercogitatus)
• wiki.splunk.com
• Slack!
Join
a
User
Group!
(https://splunk-‐usergroups.signup.team)
• The
Splunk
Trust
-‐ We
are
here
to
help!
(find
us
by
our
fez!)

© 2017 SPLU N K IN C .
Don't forget to rate this session in the
.conf2017 mobile app
© 2017 SPLU N K IN C .
Don't forget to rate this session in the
.conf2017 mobile app
Thank You
And JOIN the Community!

Lesser known-search-commands

Recommended

More Related Content

What's hot (6)

Similar to Lesser known-search-commands (20)

Recently uploaded (20)

Lesser known-search-commands