Taming the Tiger: Tips and Tricks for Using Telegraf

© 2020 InfluxData. All rights reserved.1 © 2020 InfluxData. All rights reserved.1
Mike Devy -- Solutions Architect
Sam Dillard -- Sales Engineer
November 2020

❖ Overview
❖ Setup (if not completed)
❖ Global configuration
❖ Enabling plugins
❖ Filtering data
❖ Starlark
Agenda

© 2020 InfluxData. All rights reserved.3
Workshop Logistics
• Cloud2.0 account free tier
• No CC
• Rate limits -- let’s avoid them
• Browser
• SSH client
• Have documentation open for exercise
• General config:
https://docs.influxdata.com/telegraf/latest/administration/configuration/

Telegraf
• Lightweight; written in Go
• Plug-in driven
• Optimized for writing to InfluxDB
• Formatting
• Retries
• Modifiable batch sizes and jitter
• Tag sorting
• Preprocessing
• Converting tags to fields, fields to tags
• Regex transformations
• Renaming measurements, tags
• Aggregations (mean, min, max, count, variance, stddev, etc.)

Popular Inputs (Integrations)
Out-of-the-box Custom
Kubernetes (kubelet) HTTP/socket listener
Kube_inventory (apiserver) HTTP (formatted endpoints)
Kafka (consumer) Prometheus (/metrics)
SNMP Exec
AMQP (mq metadata) StatsD
Redis
Nginx
HAproxy
Jolokia2

Telegraf
CPU
Mem
Disk
Docker
Kubernetes
/metrics
Kafka
MySQL
Process
-transform
-decorate
-filter
Aggregate
-mean
-min,max
-count
-variance
-stddev
File
InfluxDB
Kafka
CloudWatch
CloudWatch

Telegraf
InfluxDB
Telegraf
Telegraf
Telegraf
Telegraf
Telegraf
Telegraf
Telegraf
Telegraf
Telegraf
Message Queue Telegraf
Kafka
Rabbit
Active
NSQ
AWS Kinesis
Google PubSub
MQTT

Parsing
● JSON
● CSV
● Graphite
● CollectD
● Dropwizard
● Form URL-encoded
● Grok

Telegraf
• Latest version available at
https://portal.influxdata.com/downloads/#telegraf
• Create the configuration file
$ telegraf config > telegraf.conf
# ~7500 lines
OR
$ telegraf -sample-config -input-filter cpu:mem -output-filter influxdb > telegraf.conf
# 830 lines

Starlark
• Embedded Python dialect
• Not full std lib
• But most basic Python functionality supported
• Current processors and aggregators + more
[[processors.starlark]]
namepass = ["mymeasurement"]
source = '''
def apply(metric):
I = metric.fields['current']
V = metric.fields['voltage']
metric.fields['power'] = I * V
return metric
'''
source = '''
def apply(metric):
used = float(metric.fields['used'])
total = float(metric.fields['total'])
metric.fields['usage'] = (used / total) * 100
return metric
'''

Setting up InfluxDB Cloud
Sign up @ https://cloud2.influxdata.com/signup

Demo Data

Finding your Organization ID
1
2

Create a Bucket
1
2

3
4

Find your host
https://docs.google.com/spreadsheets/d/1O57
Z2crFt_fzxXcLeAB69ZnmMd3DZfUMUZEOpigx5aY/ed
it#gid=0
1.Find your name
2.Copy from column D
3.Paste into console

Login
Using an SSH client
$ ssh centos@[ip-address]
$ ssh centos@3.133.90.20
The authenticity of host '3.133.90.20 (3.133.90.20)' can't be established.
ECDSA key fingerprint is SHA256:ki5Hm1AQ4UyyOn29uvoexeNO/Mk/Z0qourjEn+EpHdk.
Are you sure you want to continue connecting (yes/no)? yes
Password:
InfluxDaysIsTheBestTrainingEver!!!!

Getting Setup
Install Telegraf
[centos@influxdays ~]$ sudo rpm -ivh /home/centos/telegraf-1.16.1-1.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing…
1:telegraf-1.16.1-1 ################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/telegraf.service
to /usr/lib/systemd/system/telegraf.service.
[centos@influxdays ~]$

Create Telegraf Configuration
1
2

Create Telegraf Configuration
1
2
3
4
5

Test your Configuration (optional)
1 2
3
Paste in console

Select All
Update configuration file
Edit the configuration file and paste
[centos@influxdays ~]$ sudo vi /etc/telegraf/telegraf.conf
1
2
3
4

VI Cheat Sheet
[centos@influxdays ~]$ sudo vi/ etc/telegraf/telegraf.conf
gg # Go to first line; this may not do anything
dG # Delete all lines
i # Insert mode
<paste> # Paste the full config from InfluxDB Cloud
Escape # End Insert mode
:64 # Jump to line 64 to update Token
:wq # Save and exit

Hardcode the Token
Be sure to retain the double quotes
1
2
3

Start the Service
[centos@influxdays ~]$ sudo systemctl start telegraf
Check the logs
[centos@influxdays ~]$ $ journalctl -u telegraf
-- Logs begin at Mon 2020-11-02 19:23:33 UTC, end at Mon 2020-11-02 19:55:58 UTC. --
Nov 02 19:55:58 influxdays systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB.
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Starting Telegraf 1.16.1
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded inputs: cpu disk diskio mem net processes swap system
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded aggregators:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded processors:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded outputs: influxdb_v2
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Tags enabled: host=influxdays
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"influxdays",
Flush Interval:10s

Alternate Deployment
Launch Telegraf using the provided commands in the Cloud Web UI
Open a new SSH console window to continue with the exercises.

Exercise 1
Starting Point

Telegraf Configuration
/home/centos/lab/exercise
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
[[inputs.mem]]
[[outputs.influxdb]]

Exercise 1 - Lay of the land
Inspect the output of a given configuration
$ telegraf --test --config exercise
Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
Gather metrics once, write them, and exit
$ telegraf --once --config exercise

Inspect the output of a given configuration
[centos@influxdays lab]$ date; telegraf --test --config exercise ; date
Fri Nov 6 22:29:46 UTC 2020
2020-11-06T22:29:46Z I! Starting Telegraf 1.16.1
> mem,host=influxdays
active=151506944i,available=732938240i,available_percent=72.28993540203045,buffered=21
17632i,cached=249655296i,commit_limit=506941440i,committed_as=399839232i,dirty=16384i,
free=636620800i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,
huge_pages_total=0i,inactive=140464128i,low_free=0i,low_total=0i,mapped=38670336i,page
_tables=4157440i,shared=13058048i,slab=46571520i,sreclaimable=23306240i,sunreclaim=232
65280i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=125493248i,use
d_percent=12.377439593424661,vmalloc_chunk=35184362651648i,vmalloc_total=3518437208780
8i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604701786000000000
Fri Nov 6 22:29:46 UTC 2020

Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
$ date; telegraf --test-wait 10 --config exercise; date
Fri Nov 6 22:30:10 UTC 2020
active=155013120i,available=728371200i,available_percent=71.83948677135389,
buffered=2117632i,cached=249663488i,commit_limit=506941440i,committed_as=40
3222528i,dirty=16384i,free=632045568i,high_free=0i,high_total=0i,huge_page_
size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=140447744i,lo
w_free=0i,low_total=0i,mapped=39026688i,page_tables=4534272i,shared=1305804
8i,slab=46686208i,sreclaimable=23306240i,sunreclaim=23379968i,swap_cached=0
i,swap_free=0i,swap_total=0i,total=1013886976i,used=130060288i,used_percent
=12.827888224101224,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087
808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i
1604701810000000000
Fri Nov 6 22:30:20 UTC 2020

Write once on start up
$ date; telegraf --once --config exercise; date
Fri Nov 6 22:31:51 UTC 2020
2020-11-06T22:31:51Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:31:51Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:31:51Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp
[::1]:8086: connect: connection refused
2020-11-06T22:31:51Z E! [agent] Error writing to outputs.influxdb: could not
write any address
2020-11-06T22:31:51Z E! [telegraf] Error running agent: output plugins unable to
send 1 metrics
Fri Nov 6 22:31:51 UTC 2020

Mem

Exercise 2
Setup for Debugging

Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
[[inputs.mem]]
2020-11-06T22:34:07Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-11-06T22:34:07Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:07Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-11-06T22:34:07Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics

$ telegraf --debug --once --config exercise
2020-11-06T22:34:42Z D! [agent] Initializing plugins
2020-11-06T22:34:42Z D! [agent] Connecting outputs
2020-11-06T22:34:42Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:34:42Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:42Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:34:42Z D! [agent] Starting service inputs
2020-11-06T22:34:42Z D! [agent] Stopping service inputs
2020-11-06T22:34:42Z D! [agent] Input channel closed
2020-11-06T22:34:42Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:42Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics
2020-11-06T22:34:42Z E! [agent] Error writing to outputs.influxdb: could not write any address
2020-11-06T22:34:42Z D! [agent] Stopped Successfully
2020-11-06T22:34:42Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics

telegraf --once --config exercise
2020-11-06T22:35:58Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection
refused
2020-11-06T22:35:58Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:35:58Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial
tcp [::1]:8086: connect: connection refused
2020-11-06T22:35:58Z E! [agent] Error writing to outputs.influxdb: could not
write any address
2020-11-06T22:35:58Z E! [telegraf] Error running agent: output plugins unable
to send 1 metrics
Updated Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]

Mem
Stdout

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.file]]
Output on next slide...

2020-11-06T22:37:05Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:37:05Z D! [agent] Successfully connected to outputs.file
mem,host=influxdays
sreclaimable=23232512i,page_tables=4255744i,shared=13058048i,swap_cached=0i,vmalloc_total=35184372087808
i,write_back_tmp=0i,dirty=20480i,low_free=0i,huge_pages_free=0i,committed_as=403091456i,free=630804480i,
sunreclaim=23289856i,swap_free=0i,cached=250294272i,huge_pages_total=0i,write_back=0i,total=1013886976i,
active=156626944i,slab=46522368i,vmalloc_chunk=35184362651648i,vmalloc_used=6332416i,used=130670592i,ina
ctive=140836864i,available_percent=71.78292819889226,buffered=2117632i,available=727797760i,used_percent
=12.88808270479253,high_total=0i,huge_page_size=2097152i,low_total=0i,mapped=40583168i,swap_total=0i,com
mit_limit=506941440i,high_free=0i 1604702225000000000
2020-11-06T22:37:05Z D! [outputs.file] Wrote batch of 1 metrics in 92.731µs
2020-11-06T22:37:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Exercise 3
Enabling a plugin

Mem
Net
Stdout

Start
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]

Validate using --test
> net,host=influxdays,interface=ens5
bytes_recv=1708843i,bytes_sent=2007423i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=7752i,packets_sent=5597i
1604702319000000000
> net,host=influxdays,interface=all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_in
errors=0i,icmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i
,icmp_intimestamps=0i,icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,i
cmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtim
estampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i
,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=7981i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inr
eceives=7986i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5516i,ip_reasmfails=0i,ip_reasmoks=0i,ip_
reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=4i,tcp_estabresets=4i,tcp_incsumerrors=5
i,tcp_inerrs=5i,tcp_insegs=7733i,tcp_maxconn=-
1i,tcp_outrsts=68i,tcp_outsegs=5563i,tcp_passiveopens=68i,tcp_retranssegs=43i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=2
00i,udp_incsumerrors=0i,udp_indatagrams=312i,udp_inerrors=0i,udp_noports=2i,udp_outdatagrams=315i,udp_rcvbuferrors=0i,udp_sndbufe
rrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rc
vbuferrors=0i,udplite_sndbuferrors=0i 1604702319000000000
active=159830016i,available=723709952i,available_percent=71.37974637520149,buffered=2117632i,cached=250327040i,commit_limit=50694
1440i,committed_as=407773184i,dirty=8192i,free=626688000i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,h
uge_pages_total=0i,inactive=140804096i,low_free=0i,low_total=0i,mapped=40611840i,page_tables=4898816i,shared=13058048i,slab=46534
656i,sreclaimable=23240704i,sunreclaim=23293952i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=134754304i,used
_percent=13.290860538680004,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,write
_back_tmp=0i 1604702319000000000

$ telegraf --config exercise
2020-11-06T22:39:26Z I! Loaded inputs: mem net
2020-11-06T22:39:26Z I! Loaded aggregators:
2020-11-06T22:39:26Z I! Loaded processors:
2020-11-06T22:39:26Z I! Loaded outputs: file
2020-11-06T22:39:26Z I! Tags enabled: host=influxdays
2020-11-06T22:39:26Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
mem,host=influxdays
free=629272576i,huge_pages_free=0i,low_total=0i,commit_limit=506941440i,high_free=0i,huge_pages_total=0i,vmalloc_chunk=35184362651648i,used=132153344i,high_tot
al=0i,shared=13062144i,sunreclaim=23162880i,vmalloc_total=35184372087808i,write_back_tmp=0i,inactive=140779520i,used_percent=13.034327013586177,swap_cached
=0i,vmalloc_used=6332416i,write_back=0i,buffered=2117632i,cached=250343424i,committed_as=403091456i,swap_total=0i,total=1013886976i,available=726310912i,availa
ble_percent=71.63627990029532,active=158572544i,dirty=4096i,page_tables=4214784i,slab=46395392i,huge_page_size=2097152i,low_free=0i,mapped=40656896i,sreclai
mable=23232512i,swap_free=0i 1604702368000000000
net,host=influxdays,interface=ens5 drop_in=0i,drop_out=0i,bytes_sent=2037575i,bytes_recv=1723821i,packets_sent=5696i,packets_recv=7891i,err_in=0i,err_out=0i
1604702368000000000
net,host=influxdays,interface=all
udplite_rcvbuferrors=0i,ip_outdiscards=0i,ip_inaddrerrors=0i,icmp_inechoreps=0i,icmp_outtimestampreps=0i,icmp_outerrors=0i,icmp_outechoreps=0i,icmp_outaddrmaskreps
=0i,icmpmsg_outtype3=2i,udplite_noports=0i,udplite_incsumerrors=0i,udp_noports=2i,icmp_incsumerrors=0i,icmp_outechos=0i,icmp_outaddrmasks=0i,icmp_inechos=0i,icm
p_outparmprobs=0i,ip_inhdrerrors=0i,ip_outnoroutes=16i,ip_defaultttl=64i,icmp_outdestunreachs=2i,udp_indatagrams=316i,tcp_currestab=1i,ip_fragfails=0i,ip_reasmtimeout
=0i,ip_reasmoks=0i,icmp_outredirects=0i,tcp_retranssegs=45i,tcp_passiveopens=69i,tcp_maxconn=-
1i,tcp_inerrs=5i,ip_inreceives=8124i,ip_forwarding=2i,icmp_inmsgs=2i,icmp_indestunreachs=2i,icmpmsg_intype3=2i,ip_indiscards=0i,icmp_intimeexcds=0i,udp_incsumerrors
=0i,tcp_outsegs=5655i,ip_forwdatagrams=0i,ip_fragcreates=0i,icmp_inredirects=0i,icmp_outsrcquenchs=0i,icmp_inaddrmasks=0i,ip_indelivers=8119i,icmp_inparmprobs=0i,tc
p_insegs=7867i,tcp_attemptfails=72i,udp_inerrors=0i,tcp_estabresets=4i,tcp_outrsts=68i,udp_rcvbuferrors=0i,ip_reasmfails=0i,ip_fragoks=0i,ip_reasmreqds=0i,icmp_outtime
stamps=0i,icmp_intimestampreps=0i,icmp_inerrors=0i,tcp_incsumerrors=5i,udplite_inerrors=0i,ip_outrequests=5610i,icmp_insrcquenchs=0i,icmp_outmsgs=2i,icmp_outtimee
xcds=0i,udp_sndbuferrors=0i,tcp_rtoalgorithm=1i,udp_outdatagrams=319i,udplite_outdatagrams=0i,tcp_rtomin=200i,icmp_inaddrmaskreps=0i,udplite_sndbuferrors=0i,ip_inun
knownprotos=0i,icmp_intimestamps=0i,tcp_rtomax=120000i,tcp_activeopens=142i,udplite_indatagrams=0i 1604702368000000000
Validate without --test

Filtering

What Is An InfluxDB Series?
First, some defintions:
• Measurement: namespace for categorically alike metrics (~Table)
• Tag: key-value pair used as metadata for measureable object
– host=prod001 or app=cart or customer_id=00012
• Field: key-value pair representing a metric that will append new values
over time
– cpu_usage=35.5 or errors=42 or temp=65.0
• TagSet: unique combination of Tag key-value pairs
• FieldSet: unique combination of Field key-value pairs

What Is An InfluxDB Series?
Line/Record: <Measurement>,<TagSet> <FieldSet> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35,usage_system=15 t0
Point: <Measurement>,<TagSet> <FieldKey> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35 t0
Series: <Measurement>,<TagSet> <FieldKey>
→ cpu,host=prod001,region=us-west usage_user

Series Visualized (tree)
Cardinality is total number of “leaf nodes”

Filtering metrics
• Happens at plugin level
• Specify which metrics and which parts of metrics the given plugin sees or doesn’t see
• Given above, filtering is often used for “routing”
• By Measurement:
• namepass: passes only records with specified Measurement name
• namedrop: drops only records with specified Measurement name
• By Field:
• fieldpass: pass specified Field
• fielddrop: drop specified Field
• By Tag (slightly different implementation):
• tagpass or tagdrop: pass/drop whole record containing matched Tag
• taginclude or tagexclude: keep/drop matching Tag (key) in record

How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0,cached=12.5,buffered=3.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA cached=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA buffered=85.0 <timestamp>
namepass = “mem”

Telegraf metric:
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0,
usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fieldpass = “*_system”

Telegraf metric:
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0,
usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fielddrop = “*_softirq”

Telegraf metric:
cpu,region=us-west-1,host=hostB,container=containerB
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagpass: host = [“hostA”]

Telegraf metric:
cpu,region=us-west-1,host=hostB,container=containerB
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagdrop: host = [“hostB”]

Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
cpu,region=us-west-1,container=containerB,host=hostB
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
taginclude:
host = [“host”]

Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
cpu,region=us-west-1,container=containerB,host=hostB
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
tagexclude:
host = [“host”]

Exercise 4
Filtering

Sample Data
mem,host=influxdays
available=735477760i,inactive=167567360i,page_tables=4345856i,sreclaimable=22962176i,committed_as=402309120i,slab=46772224i,vmalloc_total=35184
372087808i,cached=243228672i,high_free=0i,huge_page_size=2097152i,huge_pages_total=0i,write_back=0i,total=1013886976i,dirty=0i,free=645414912i,lo
w_total=0i,sunreclaim=23810048i,swap_cached=0i,huge_pages_free=0i,low_free=0i,shared=13058048i,swap_total=0i,used=123125760i,used_percent=12.1
43933487118785,buffered=2117632i,vmalloc_used=6332416i,available_percent=72.54040908007482,high_total=0i,vmalloc_chunk=35184362651648i,active
=113328128i,commit_limit=506941440i,mapped=40271872i,swap_free=0i,write_back_tmp=0i 1602630666000000000
net,host=influxdays,interface=ens5
packets_recv=11760i,err_in=0i,err_out=0i,drop_in=0i,drop_out=0i,bytes_sent=1730127i,bytes_recv=1082190i,packets_sent=10073i 1602630666000000000
net,host=influxdays,interface=all
icmp_outechos=0i,icmp_inechoreps=0i,tcp_rtoalgorithm=1i,tcp_insegs=9422i,ip_inunknownprotos=0i,ip_indiscards=0i,ip_reasmreqds=0i,icmp_outdestunreac
hs=2i,icmpmsg_outtype3=2i,icmp_outaddrmasks=0i,icmp_outredirects=0i,icmp_indestunreachs=3i,icmp_incsumerrors=0i,udp_incsumerrors=0i,ip_inhdrerrors
=0i,ip_reasmoks=0i,tcp_maxconn=-
1i,udp_sndbuferrors=0i,tcp_incsumerrors=0i,tcp_estabresets=21i,udplite_rcvbuferrors=0i,ip_reasmtimeout=0i,icmp_outerrors=0i,icmp_intimeexcds=0i,udplite_
sndbuferrors=0i,icmpmsg_intype3=3i,tcp_activeopens=188i,tcp_rtomax=120000i,udp_outdatagrams=991i,udplite_incsumerrors=0i,ip_fragcreates=0i,ip_frago
ks=0i,icmp_inerrors=0i,tcp_rtomin=200i,udp_rcvbuferrors=0i,udp_noports=2i,ip_inaddrerrors=0i,ip_reasmfails=0i,ip_forwarding=2i,icmp_outechoreps=0i,icmp
_intimestamps=0i,icmp_outmsgs=2i,tcp_outrsts=141i,udp_indatagrams=932i,ip_inreceives=10233i,ip_outdiscards=0i,icmp_intimestampreps=0i,icmp_outtime
stamps=0i,icmp_outsrcquenchs=0i,tcp_passiveopens=269i,tcp_currestab=1i,udplite_indatagrams=0i,ip_outnoroutes=16i,icmp_outtimestampreps=0i,icmp_out
parmprobs=0i,icmp_inaddrmasks=0i,icmp_outaddrmaskreps=0i,icmp_inparmprobs=0i,ip_defaultttl=64i,ip_outrequests=8367i,icmp_inredirects=0i,icmp_insrcq
uenchs=0i,icmp_outtimeexcds=0i,icmp_inmsgs=3i,icmp_inaddrmaskreps=0i,ip_forwdatagrams=0i,ip_fragfails=0i,icmp_inechos=0i,tcp_inerrs=0i,tcp_attemptfai
ls=175i,tcp_outsegs=7517i,udplite_inerrors=0i,udplite_noports=0i,ip_indelivers=10227i,tcp_retranssegs=118i,udp_inerrors=0i,udplite_outdatagrams=0i
1602630666000000000

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
Pass fields
ending in “percent”
exact named “total”
[[inputs.net]]
Pass fields Beginning with “bytes”
Pass metrics where interface begins
with “en”
[[outputs.file]]

Exercise 4
Filtering
Solution

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
...

Correct Placement
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
...

Before & After
bytes_recv=1739020i,bytes_sent=2088482i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=8042i,packets_sent=5843i 1604702475000000000
active=154382336i,available=730259456i,available_percent=72.02572607067398,buffered=2117632i,cached=250368000i,commit_limit=506941440i,committed_a
s=403091456i,dirty=20480i,free=633192448i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=14068
9408i,low_free=0i,low_total=0i,mapped=40558592i,page_tables=4190208i,shared=13058048i,slab=46452736i,sreclaimable=23232512i,sunreclaim=23220224i,s
wap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=128208896i,used_percent=12.645284833010814,vmalloc_chunk=35184362651648i,vmalloc_t
otal=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604702475000000000
> net,host=influxdays,interface=all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_inerrors=0i,icmp_in
msgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i,icmp_intimestamps=0i,icmp_outaddr
maskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,icmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=
0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outty
pe3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=826
8i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inreceives=8273i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5747i,ip_reasm
fails=0i,ip_reasmoks=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=1i,tcp_estabresets=4i,tcp_incs
umerrors=5i,tcp_inerrs=5i,tcp_insegs=8010i,tcp_maxconn=-
1i,tcp_outrsts=68i,tcp_outsegs=5792i,tcp_passiveopens=74i,tcp_retranssegs=47i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerr
ors=0i,udp_indatagrams=322i,udp_inerrors=0i,udp_noports=2i,udp_outdatagrams=325i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,u
dplite_indatagrams=0i,udplite_inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i
1604702475000000000
….
After filtering...
….
> mem,host=influxdays available_percent=71.50255927540388,total=1013886976i,used_percent=13.168451628280902 1604702498000000000
> net,host=influxdays,interface=ens5 bytes_recv=1745105i,bytes_sent=2111895i 1604702498000000000

Placement Matters,
Indentation Does Not
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
bytes_recv=2082848i,bytes_sent=3148789i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=21247i,
packets_sent=19323i 1604702498000000000

Exercise 5
Dual Write

Mem
Net
/tmp/net
/tmp/mem

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
Pass the “mem” data to file /tmp/mem
[[outputs.file]]
Pass the “net” data to file /tmp/net

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]

Exercise 6
Listener

Mem
Net
/tmp/net.txt
/tmp/mem.txt
:8086
Add

telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
Set the listener port to ":8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]

[global_tags]
[agent]
interval = "2s"
debug = true
service_address = ":8086"
read_timeout = "2s"
[[outputs.file]]

Launch Telegraf Listener as background job
$ telegraf --config telegraf-listener.conf &
[1] 2219
[centos@influxdays ~]$ 2020-11-06T22:44:22Z I! Starting Telegraf
1.16.1

Start
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
[[outputs.file]]
namepass = ["net"]
Finish
[global_tags]
[agent]
interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
[[outputs.file]]
namepass = ["net"]

Launch Telegraf
$ telegraf -config exercise
2020-11-06T22:45:09Z I! Loaded inputs: mem net
2020-11-06T22:45:09Z I! Loaded aggregators:
2020-11-06T22:45:09Z I! Loaded processors:
2020-11-06T22:45:09Z I! Loaded outputs: file (2x) influxdb
2020-11-06T22:45:09Z I! Tags enabled: host=influxdays
2020-11-06T22:45:09Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
2020-11-06T22:45:11Z D! [outputs.influxdb] Wrote batch of 2 metrics in 1.205233ms
mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000

Inspect the filesystem
$ cat /tmp/mem
mem,host=influxdays
used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
$ cat /tmp/net
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000

[global_tags]
[agent]
interval = "2s"
debug = true
service_address = ":8086"
read_timeout = "2s"
[[outputs.file]]

Exercise 7
Starlark

Finish
[global_tags]
[agent]
interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
interface = ["en*"]
source = '''
def apply(metric):
for k, v in metric.fields.items():
if type(v) == "float" or type(v) == "int":
metric.fields[k] = v * 1000
return metric
'''
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
debug = true
#[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
..
..
[[outputs.file]]

Launch Telegraf
Without Starlark processor
After multiplying by 1000

Exercise 8
JSON

Mem
Net
/tmp/net.txt
/tmp/mem.txt
HTTP

Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
debug = true
#[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]

Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
name_override = “apps”
json_query = “apps”
tag_keys = [“name”,”id”]
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
debug = true
#[[inputs.mem]]
[[inputs.net]]
interface = ["en*"]
[[outputs.file]]

Blogs
https://www.influxdata.com/blog/using-a-telegraf-gateway/
https://www.influxdata.com/blog/telegraf-go-collection-agent/
Docs
https://docs.influxdata.com/telegraf/latest
Prebuilt dashboards
https://docs.influxdata.com/platform/monitoring/influxdata-platform/monitoring-dashboards/
Telegraf Resources

Questions?

Thank you

Taming the Tiger: Tips and Tricks for Using Telegraf

More Related Content

What's hot (20)

Similar to Taming the Tiger: Tips and Tricks for Using Telegraf (20)

More from InfluxData (20)

Recently uploaded (20)

Taming the Tiger: Tips and Tricks for Using Telegraf