Monitoring Your ISP Using InfluxDB Cloud and Raspberry Pi

• Travel addict
• Photography enthusiast
• Tech interested
• Robotics/Industrial engineer

source cloud.google.com
source edition.cnn.com

Monitor internet speed day & night
 Collect data
 View data

Monitor internet speed day & night
 Collect data
 View data
… and what can I do if I’m not at home?

NEED
 Measure
 Store
 Visualize
 Communication
HOW-TO
 Raspberry Pi
 Spreadsheet
 Spreadsheet
 Not possible

PRO
 Easy to implement
 Low effort to present data
CONS
 Local
 Requires PC to open file
 Writing data (concurrency)
 SD card corruption
Need to find a better solution!

NEED
 Measure
 Store
 Visualize
 Communication
HOW-TO
 Raspberry Pi
 InfluxDB
 Grafana
 Sense-hat
 Telegram

PRO
 Nothing to store locally
 Maintenance not on me!
 Possibility to learn flux!
CONS
 Not much!!
Time to switch to influxdb2!

 Several options available
 Needed
▪ PublicAPI
▪ Scriptable
▪ Debian-like compatible
 Not needed
▪ 100% uptime
▪ Certification
speedtest.net
speedtest CLI

 Time Series DB
 Needed
▪ Free
▪ Cloud based
▪ Public API
 Not needed
▪ 100% uptime
▪ Large storage
influxdata
cloud free tier

 Dashboard
 Needed
▪ Free
▪ Cloud based
▪ Nice-looking
▪ App based (desiderata)
 Not needed
▪ 100% uptime
▪ Lots of dashboards
Grafana
cloud free

 Somehow give fast feedback
about my network
 Needed
▪ Free
▪ Request based
▪ Easy to implement
 Not needed
▪ 100% uptime
telegram
custom bot

Let’s look at the implementation

RaspberryPi #1
RaspberryPi #2
info: astro-pi.org

INTERACTIVE
$ speedtest
Speedtest by Ookla
Server: <Server Name> (id = <Server ID>)
ISP: <ISP>
Latency: 6.87 ms (1.95 ms jitter)
Download: 77.37 Mbps (data used: 62.7 MB)
Upload: 19.98 Mbps (data used: 9.0 MB)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/<UUID>
SCRIPTED
$ speedtest --format json
{ "type": "result",
"timestamp": "2021-04-07T18:00:00Z",
"ping":
{ "jitter": 0.696,
"latency": 6.645 },
"download":
{ "bandwidth": 7662563,
"bytes": 47087360,
"elapsed": 5914 },
"upload":
{ "bandwidth": 2197125,
"bytes": 7004736,
"elapsed": 3607 },
"packetLoss": 0,
"isp": "<ISP>",
"interface":
{ "internalIp": "<LAN ip>",
"name": "<LAN interface name>",
"macAddr": "<LAN interface MAC>",
"isVpn": false,
"externalIp": "<Public IP>" },
"server":
{ "id": <Server ID>,
"name": "<Server Name>",
"location": "<Server Location>",
"valcountry": "Italy",
"host": "<Server Host>",
"port": 8080,
"ip": "<Server IP>" },
"result":
{ "id": "<UUID>",
"url": https://www.speedtest.net/result/c/<UUID>}
}

[[outputs.influxdb_v2]]
urls = [ "${SPEEDTEST_SERVERURL}" ]
token = "${SPEEDTEST_TOKEN}"
organization = "${SPEEDTEST_ORGANIZATION}"
bucket = "${SPEEDTEST_BUCKET}"
[[processors.converter]]
[processors.converter.fields]
string = [
"server_id",
]
integer = [
"server_port",
]
float = [
"download_bandwidth",
"download_bytes",
"download_elapsed",
"upload_bandwidth",
"upload_bytes",
"upload_elapsed",
"packetLoss",
"ping_latency",
"ping_jitter",
]
[[inputs.exec]]
interval = "15m"
commands = [
"/usr/bin/speedtest --accept-license --accept-gdpr -f
json",
]
name_override="${SPEEDTEST_MEASUREMENT}"
timeout = "60s"
data_format = "json"
json_time_format = "2006-01-02T15:04:05Z"
json_time_key = "timestamp"
tag_keys = [
"interface_externalIp",
"interface_internalIp",
"isp",
"server_host"
]
json_string_fields = [
"server_location",
"server_name",
"server_testcountry",
"server_ip",
"result_id",
"result_url",
]

from(bucket: mybucket)
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) =>
r["_measurement"] == mymeasurement and (
r["_field"] == "download_bandwidth" or
r["_field"] == "upload_bandwidth" or
r["_field"] == "ping_latency"
)
)
|> keep(columns: ["_time", "_field", "_value"])
|> aggregateWindow(every: v.windowPeriod, fn: valmean, createEmpty: false)
|> map(fn: (r) => ({
r with _value: if (r._field == "download_bandwidth" or r._field == "upload_bandwidth") then
r._value * 8.0 else r._value
})
)

r["_field"] == "download_bandwidth" or r["_field"] == "upload_bandwidth" or r["_field"] == "ping_latency" or r["_field"] ==
"server_name" or r["_field"] == "packetLoss" or r["_field"] == "server_location")
)
|> pivot(rowKey:["_time"], columnKey:["_field"], valueColumn:"_value")
|> group(columns: ["server_host"])
|> keep(columns: ["download_bandwidth", "upload_bandwidth", "ping_latency", "packetLoss", "server_location", "server_name", "serve
r_host"])
|> reduce(identity: {
server_name: "", server_location: "", valcount: 0.0, download_bandwidth: 0.0, upload_bandwidth: 0.0, ping_latency: 0.0, packet
Loss: 0.0, },
fn: (r, accumulator) => ({
server_name: r.server_name,
server_location: r.server_location,
valcount: accumulator.valcount + 1.0,
download_bandwidth: (r.download_bandwidth + accumulator.download_bandwidth * accumulator.valcount) / (accumulator.valcount + 1
.0),
upload_bandwidth: (r.upload_bandwidth + accumulator.upload_bandwidth * accumulator.valcount) / (accumulator.valcount + 1.0),
ping_latency: (r.ping_latency + accumulator.ping_latency * accumulator.valcount) / (accumulator.valcount + 1.0),
packetLoss: (r.packetLoss + accumulator.packetLoss * accumulator.valcount) / (accumulator.valcount + 1.0),
})
)
|> map(fn: (r) => ({ r with download_bandwidth: r.download_bandwidth * 8.0, upload_bandwidth: r.upload_bandwidth * 8.0 }))
|> drop(columns: ["server_host"])

r["_field"] == "results_id" or
r["_field"] == "server_name" or
r["_field"] == "server_location"
)
)
|> sort(columns: ["_time"], desc: true)

import "experimental"
import "date"
option task = {name: "DailyMinMax", cron: "0 2 * * *"}
today = () => (date.truncate(t: now(), unit: 1d))
yesterday = (boundary="start") => {
timeValue = if boundary == "end" then experimental.subDuration(d: 1ns, from: today()) else experimental.subDuration(d: 24h, from: today())
return timeValue
}
|> range(start: yesterday(), stop: yesterday(boundary: "end"))
|> timeShift(duration: 1h, columns: ["_start", "_stop", "_time"])
|> filter(fn: (r) => (r["_measurement"] == mymeasurement and (r["_field"] ==
"download_bandwidth" or r["_field"] == "upload_bandwidth" or r["_field"] == "pingLatency")))
|> reduce(identity: { valcount: 0.0, valmin: 0.0, valmax: 0.0, valmean: 0.0, }, fn: (r, accumulator) =>
({
valcount: accumulator.valcount + 1.0,
valmin: if accumulator.valcount == 0.0 then r._value else if r._value < accumulator.valmin then r._value else accumulator.valmin,
valmax: if accumulator.valcount == 0.0 then r._value else if r._value > accumulator.valmax then r._value else accumulator.valmax,
valmean: (r._value + accumulator.valmean * accumulator.valcount) / (accumulator.valcount + 1.0),
}))
|> map(fn: (r) => ({r with _time: yesterday(boundary: "end"), _measurement: "daily", data: r._field}))
|> to( bucket: <mybucket>, org: <myorganization>, tagColumns: ["data"], fieldFn: (r) =>
({ "valcount": r.valcount, "valmean": r.valmean, "valmin": r.valmin, "valmax": r.valmax, }))

 Visual feedback
 Periodical & on-demand
Python + AstroPi(kind-of)
info: astro-pi.org
[...]
self.query = 'import "math"
from(bucket: "' + self.bucket + '")
|> range(start: -1d)
r["host"] == "%s" and
r["_measurement"] == "' + self.measurement + '" and (
r["_field"] == "download_bandwidth" or
r["_field"] == "upload_bandwidth" or
r["_field"] == "ping_latency"
)
)
|> sort(columns: ["_time"], desc: false)
|> last()
|> map(fn: (r) => ({
r with _value: if (r._field == "download_bandwidth" or r._field ==
"upload_bandwidth") then math.round(x: (r._value * 8.0 / 10000.0)) /
100.0 else r._value
})
)
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn:
"_value")'
db_client = InfluxDBClient(url=self.url, token=self.token,
org=self.org)
db_data = db_client.query_api().query_stream(query=(self.query_string
% hostsname), org=self.org)
[...]

PROS
 Something quick and easy
 On demand
 Custom telegram bot on
Raspberry Pi
CONS
 No slack app & workspace
 No live notifications (yet)
 No service in case of network
issues

 Do not use random test server
 Select optimal test server
 Use good hardware at home

 Integration with Smart Speakers (?)
 Automatic daily reporting
 Event notifications
 …

We look forward to bringing together our
community of developers to learn, interact
and share tips and use cases.
10-11 May 2021
Hands-On Flux Training
18-19 May 2021
Virtual Experience
www.influxdays.com/emea-2021-virtual-experience/

Monitoring Your ISP Using InfluxDB Cloud and Raspberry Pi

Recommended

More Related Content

What's hot (20)

Similar to Monitoring Your ISP Using InfluxDB Cloud and Raspberry Pi (20)

More from InfluxData (20)

Recently uploaded (20)

Monitoring Your ISP Using InfluxDB Cloud and Raspberry Pi