THE CLOUD NATIVE

API PLATFORM

Kong Workshops
Observability

Professional Services Team

Jan 2024

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 1
 Course Agenda
1. Kong Gateway Installation 6. Advanced Plugins Review
2. Upgrading Kong Gateway 7. Troubleshooting
3. Securing Kong 8. Monitoring / Observability
4. Securing Services on Kong 9. Administering Kong Gateway
using Deck
5. OIDC Plugin
10. GitOps

THE CLOUD NATIVE

Copyright 2023 © Kong Inc. 2
API PLATFORM
 Observability
Agenda 1. Metrics
2. Logs
3. Tracing
Observability 4. Audit

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 3
 Observability

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 4
 Observability
Metrics Logs Traces Audits

Operational metrics that Kong logs have request and Traces are records of events Audit logs are records of
indicate APIs performance, response data that provide that capture the flow of Kong Admin API activities
API consumption and the valuable information for execution of requests going that can be used for security
health of your Kong nodes. monitoring, security, and through Kong proxy. It monitoring, compliance,
incident investigation provides a detailed view of incident response,
the steps taken for accountability, and
diagnostic, troubleshooting. governance purposes.

Vitals Multiple Log Plugins: Zipkin, Open telemetry System Level Configuration
Vitals with Prometheus HTTP Log | File Log | StatD Loggly
Vitals with InfluxDB SysLog | UDP Log | TCP Log
DataDog and Kafka Log | Splunk
Prometheus Plugin

THE CLOUD NATIVE 5

API PLATFORM
Copyright 2023 © Kong Inc. 5
 Metrics

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 6
Overview
■ Rich metrics available through plugins
■ Enabling only on data-planes
■ ServiceMonitor support through helm-chart

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 7
Vitals Strategy
Vitals Vitals

CP CP

DP DP DP DP DP DP

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 8
Kong Vitals

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 9
Prometheus Plugin
Cluster

DP DP DP CP

Prometheus-Plugin Plugin Plugin Plugin Plugin

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 10
Grafana Dashboards
Status 7424

Kube state metrics 13332

Node exporter 1860

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 11
Request Logging w/ File Log Plugin

Kong pods logs

requests to
/dev/stdout and
captured by Datadog
Agent

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 12
 Kong metrics to Datadog w/ Datadog Plugin

Kong pods sends

metrics to Datadog
Agents

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 13
Kong metrics to Datadog w/ Datadog Plugin

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 14
 Alerting (metrics)

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 15
 Alerting

Prometheus Prometheus
Alertmanager

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 16
 Logging

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 17
Kong Gateway Logs
● Usually under <prefix folder>/kong/logs folder (depending on the installation/environment)

● Different types of logging files based on nginx log

○ Proxy Access log
○ Proxy Error log
○ Admin Access log
○ Admin Error log
○ Portal Access log
○ Portal Error log
○ Status Error log

● Access log can be customized with nginx custom format

Default location of common logs:

/usr/local/kong/logs/access.log
/usr/local/kong/logs/admin_access.log
/usr/local/kong/logs/error.log
/usr/local/kong/logs/admin_gui_access.log

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 18
Kong Application Access Logs
● DPs and CP
○ Exposed on /dev/stdout

172.18.0.1 - - [14/Jul/2021:04:39:44 +0000] "GET /echo HTTP/1.1" 404 23 "-" "HTTPie/1.0.3"

● Will be picked up by EKS cluster logging and forwarded to Cloudwatch, Datadog, etc

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 19
Kong Gateway Logs
● Request and response data in JSON format
● Proxy logs can be forwarded to a third party tool using
Kong Plugin, such as:
○ Datadog
○ Loggly
○ Kafka
○ Splunk
○ HTTP
● Can be customized with kong.log.serializer
● Log format:
https://docs.konghq.com/hub/kong-inc/file-log/#log-f
ormat

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 20
Kong logging plugins
● Request and response data in JSON format
● Proxy logs can be forwarded to a third party tool like ELK stack to be analyzed

{
"latencies": {
"request": 515,
"kong": 58,
"proxy": 457
},
"service": {
"host": "httpbin.org",
"created_at": 1614232642,
"connect_timeout": 60000,
"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260",
"protocol": "http",
"read_timeout": 60000,
"port": 80,
"path": "/anything",
------

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 21
Request Logging w/ File Log Plugin

Kong pods logs

requests to
/dev/stdout and
captured by Datadog
Agent

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 22
Request Logging w/ File Log Plugin

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 23
 Tracing

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 24
Overview
■ Open telemetry support
■ Request tracing
■ w3c - W3C trace context
■ b3, b3-single - Zipkin headers
■ jaeger - Jaeger headers
■ ot - OpenTracing headers

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 25
Open telemetry Plugin
Cluster

DP DP DP CP

Open telemetry Plugin Plugin Plugin Plugin Plugin

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 26
 Audit Log

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 27
 vagrant@ubuntu-xenial:/kong$ http :8001/status
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Admin Audit Logs Date: Tue, 13 Nov 2018 17:32:47 GMT
Server: kong/0.34-enterprise-edition
Transfer-Encoding: chunked
X-Kong-Admin-Request-ID: ZuUfPfnxNn7D2OTU6Xi4zCnQkavzMUNM
● Allow administrators to keep detailed track of
configuration changes made by using Admin API or {
"database": {
Kong Manager to the Kong Control Plane configuration "reachable": true
},
throughout its lifetime

● Generated audit log trails are Workspace and http :8001/audit/requests

RBAC-aware HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
● Kong will generate granular audit log entries for all Connection: keep-alive

insertions, updates, and deletions to the cluster Content-Type: application/json; charset=utf-8

Date: Tue, 13 Nov 2018 17:35:24 GMT
database.
Server: kong/0.34-enterprise-edition
Transfer-Encoding: chunked
● Audit logs can be extracted from Kong database using X-Kong-Admin-Request-ID: VXgMG1Y3rZKbjrzVYlSdLNPw8asVwhET
Kong’s Admin Audit REST API
{

● Audit logs are retained by default for 30 days (this can "data": [
{
be changed in the configuration)
"client_ip": "127.0.0.1",
"method": "GET",
"path": "/status",

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 28
Audit log response
● RBAC User info
● Workspace info
● Entity Type
● Client IP
● Request ID
● Timestamp

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 29
 Demo

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 30
 Q&A

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 31
Thank You

THE CLOUD NATIVE

API PLATFORM
Copyright 2023 © Kong Inc. 32