Scribd
Upload
516 views

Splunk Notes For Testing

Splunk can automatically extract fields from event data using predefined source types or by identifying common patterns like name-value pairs separated by equals signs. Users can also configure field extraction by using the interactive field extractor to generate regular expressions for extracting fields or by manually specifying extraction rules. Fields provide contextual information that makes event data more meaningful and enables deeper analysis.

Uploaded by

Shane Lawson
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
516 views

Splunk Notes For Testing

Splunk can automatically extract fields from event data using predefined source types or by identifying common patterns like name-value pairs separated by equals signs. Users can also configure field extraction by using the interactive field extractor to generate regular expressions for extracting fields or by manually specifying extraction rules. Fields provide contextual information that makes event data more meaningful and enables deeper analysis.

Uploaded by

Shane Lawson
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 4

(or kv, for "key/value

Fields are searchable name/value pairs in event data.


Enriching Your Data
To make your data more useable, add knowledge to it. What do we mean
by that? When you tell Splunk how to extract fields from your data, you
can start reasoning about those fields and give Splunk the knowledge to
classify your data for deeper analysis.
This chapter covers three areas:
Using Splunk to Understand Data
When you first encounter a new source of machine data, it can look like a
mess of meaningless numbers and cryptic text. The more you know about
the system pumping out machine data, however, the more the data will
make sense to you. But even if you know a data set well, further exploration
can still bring new insights.
The first step in getting to know data is using Splunk to identify fields in
the data. You can think of this like looking at all the pieces in a puzzle,
first noticing their shapes. The next step is to categorize data as a preamble
to aggregation and reporting. This is like sorting the puzzle pieces into
border pieces and interior pieces. The more you are able to understand
the data and piece the puzzle together, the clearer the picture becomes.
At last, the picture is complete (displaying the data) and you can share it
with others.
Identifying Fields: Looking at the Pieces of the Puzzle
Splunk recognizes many common types of data, referred to as source
types. If you set the right source type, Splunk can use preconfigured settings
to try to identify fields. This is the case with most types of web server
logs, for example.
But there are often hidden attributes embedded in machine data. For example,
a product category may be part of a URL. By examining events
that have certain product categories in their URLs, you can determine
response times and error rates for different sections of the site or information
about which products are viewed the most.

Automatic Field Discovery


When you search, Splunk automatically extracts fields by identifying
common patterns in the data, such as the presence of an equal sign (=)
between a key and a value. For example, if an event contains id=11
lname=smith Splunk automatically creates id and lname fields that
have the example values. And, as mentioned in Chapter 2, some fields
(such as ,source, sourcetype, host, _time, and linecount) are always
identified at index time

Dont see what youre looking for? Start searching for it. Splunk displays only a
certain number of fields in the UI by default. Hundreds more may be extracted
perfectly. Searching for them brings them to the top.

The Field Discovery switch onit identifies the first 50 fields that match name/value
pairs that it finds in the event data. the Fields sidebar in the UI turns this behavior
on and off. You can see some selected fields (fields that Splunk
selected by default or that you have selected), followed by fields that
Splunk pulled out because they appeared in multiple events. If you click
Edit, Splunk lists more fields that you can add to the group of selected
fields. Clicking any field shows you the top values extracted from your
search results.
For more information on automatic field extraction, see http://splunk.com/goto/
book#auto_fields.

Configuring Field Extraction


Configuring field extraction can happen in two ways. You can let Splunk
automate the configuration for you by using the Interactive Field Extractor,
or you can manually specify the configuration yourself.
The Interactive Field Extractor
From any event in your search results, you can start the Interactive Field
Extractor (IFX) by selecting Extract Fields from the Event options menu,
which you reach by clicking the down arrow to the left of an event in the
events list (see Figure 5-1).

Figure 5-1. Choosing Extract Fields from the Event Options menu starts the
Interactive Field Extractor
The IFX appears in another tab or window in your browser. By entering
the kinds of values you seek (such as a client IP address in web logs),
Splunk generates a regular expression that extracts similar values (this is
especially helpful for the regular expression-challenged among us). You
can test the extraction (to make sure it finds the field youre looking for)
and save it with the name of the field.
Manually Configuring Field Extraction
From Manager Fields Field extractions, you can manually specify regular
expressions to extract fields, which is a more flexible but advanced
method for extracting fields. The (This 50 field limit is a default that can be modified
by editing the [kv] stanza in
limits.conf.)
Say you search on sourcetype, a default field that Splunk Enterprise
automatically extracts for every event at index time. If your search is
sourcetype=veeblefetzer
for the past 24 hours, Splunk Enterprise returns every event with a sourcetype of
veeblefetzer in that time range. From this set of events, Splunk Enterprise
automatically extracts the first 50 fields that it can identify on its own. And it
performs extractions of custom fields, based on configuration files. All of these
fields will appear in the fields sidebar when the search is complete.
Now, if a name/value combination like userlogin=fail appears for the first time
25,000 events into the search, and userlogin isn't among the set of custom fields

that you've preconfigured, it likely won't be among the first 50 fields that Splunk
Enterprise finds on its own. However, if you change your search to
sourcetype=veeblefetzer userlogin=*
Then Splunk Enterprise will be smart enough to find and return all events
including both the userlogin field and a sourcetype value of veeblefetzer, and it
will be available in the field sidebar along with the other fields that Splunk
Enterprise has extracted for this search.

Internal
fields
_raw, _time,
_indextime, _cd

These are fields that contain general


information about events in Splunk
Enterprise.
Default
fields
host, index,
linecount, punct,
source, sourcetype,
splunk_server,
timestamp

These are fields that contain information


about where an event originated, in which
index it's located, what type it is, how many
lines it contains, and when it occurred.
These fields are indexed and added to the
Fields menu by default.
Default
datetime
fields
date_hour,
date_mday,
date_minute,
date_month,
date_second,
date_wday,
date_year, date_zone

These are fields that provide additional


searchable granularity to event timestamps.
Note: Only events that have timestamp
information in them as generated by their
respective systems will have date_* fields. If
an event has a date_* field, it represents the
value of time/date directly from the event

itself. If you have specified any timezone


conversions or changed the value of the
time/date at indexing or input time (for
example, by setting the timestamp to be the
time at index or input time), these fields will
not represent that.
You might also want to change the name of a field, or group it with other similar
fields. This is easily done with tags or aliases for the fields and field values.
Fields that begin with an underscore are internal fields.
Note: We do not recommend that you override internal fields unless you are
absolutely sure you know what you are doing.
_raw
Example: Return sendmail events that contain an IP address that starts with
"10".
eventtype=sendmail | regex _raw=*10.\d\d\d\.\d\d\d\.\d\d\d\*

_time
Example: Search all sources of type "mail" for mail addressed to the user
"[email protected]", then sorts the search results by timestamp.
sourcetype=mail [email protected] | sort _time

Use interactive field extraction to create new fields


You can create custom fields dynamically using the interactive field extractor
(IFX) in Splunk Web. IFX enables you to quickly turn any search into a field
extracting regular expression. You use IFX on the local indexer. For more
information about using IFX, see "Extract fields interactively with IFX" in this
manual.
Note: IFX is especially useful if you are not familiar with regular expression
syntax and usage, because it will generate field extraction regexes for you (and
enable you to test them). To access IFX, run a search and then select Extract fields
from the dropdown
that appears beneath timestamps in the field results. IFX enables you to extract
only one field at a time (although you can edit the regex it generates later to
extract multiple fields). In Splunk Web, you navigate to the Field extractions page by
selecting Settings
> Fields > Field extractions.

You might also like