Do you know what your drupal is doing? Observe it!

Luca Lusso Developer
Java, PHP, Go, Drupal, QA, Testing, ...
Cloud architect
AWS, Docker, Terraform, Ansible, ...
Open Source
Devel, Monolog, ...
Social
@lussoluca, drupal.org/u/lussoluca

So ware development
Design/UI/UX
Web marketing
QA & Testing
SEO & ML
www.wellnet.it

Almost everyone is working with distributed systems.
There are microservices, containers, cloud, serverless,
and a lot of combinations of these technologies. All of
these increase the number of failures that systems will
have because there are too many parts interacting.
And because of the distributed system’s diversity, it’s
complex to understand present problems and predict
future ones

"observability is a measure of how well
internal states of a system can be
inferred from knowledge of its external
outputs"

We want to observe production environments and
generic metrics like CPU and memory usage are not
suﬀicient anymore

3 PILLARS OF OBSERVABILITY3 PILLARS OF OBSERVABILITY
1. Structured logs
2. Metrics
3. (Distributed) Traces

STRUCTURED LOGSSTRUCTURED LOGS

1. Logs are about storing specific events

In Drupal 8 (an in general in PHP world) we have a
battle tested library to do logging
Monolog

Monolog is a PSR-3 compliant library from Jordi
Boggiano.
There is a module for both Drupal 7 and Drupal 8

We have to download the Monolog module using
Composer, to have both the module and the library
"require": {
"drupal/monolog": "1.3",

Then we have to create a file in the sites/default folder
of our Drupal website, for example
monolog.services.yml

monolog.services.yml
parameters:
monolog.channel_handlers:
default:
handlers: ['rotating_file']
formatter: 'json'
monolog.processors: [
'message_placeholder',
'current_user',
'request_uri',
'ip',
'referer',
'filter_backtrace',
'introspection'
]

monolog.services.yml (cont)
services:
monolog.handler.rotating_file:
class: MonologHandlerRotatingFileHandler
arguments: [
'public://logs/debug.log',
10,
'%monolog.level.info%'
]

Then in settings.php we add monolog.services.yml to
the list of container yamls
settings.php
$settings['container_yamls'][] =
DRUPAL_ROOT . '/sites/default/monolog.services.yml';

{
"message": "Session opened for admin.",
"context": [],
"level": 250,
"level_name": "NOTICE",
"channel": "user",
"datetime": {
"date": "2019-10-06 19:50:34.641160",
"timezone_type": 3,
"timezone": "UTC"
},
"extra": {
"file": "/var/www/html/web/core/modules/user/user.module",
"line": 556,
"class": null,
"function": "user_login_finalize",
"referer": "http://web.d86.docker.localhost/user/login",
"ip": "172.20.0.2",
"request_uri": "http://web.d86.docker.localhost/user/login
"uid": "1",
"user": "admin"
}
}

Structured logs makes it simple to query them for any
sort of useful information
We can write custom Monolog processors to add
application specific data to our logs

2. Metrics are a measurement at a point in time for the
system

Examples of the sort of metrics you might have would
be:
the number of times you received HTTP requests
how much time was spent handling requests
how many requests are currently in progress
the number of errors occurred

To instrument our application and record real-time
metrics we will use Prometheus (prometheus.io)

Prometheus was the second project to join the Cloud
Native Computing Foundation a er Kubernetes

To gather information from our production
environment we need two things:
instrument our application
extract data from the system

We start writing a simple module to implement
Observability in Drupal 8:
https://www.drupal.org/sandbox/lussoluca/3054802

The O11y module uses the
jimdo/prometheus_client_php library to implement a
Prometheus client (so you have to install it using
Composer)
Using the APC storage will reset metrics if server is
restarted, use the Redis storage to overcome this
$registry =
new PrometheusCollectorRegistry(
new PrometheusStorageAPC()
);

Prometheus has three types of metric:
1. Counters (represents a single monotonically
increasing counter)
$registry
->getOrRegisterCounter(
$namespace,
$name,
$help,
$labels
);

2. Gauges (represents a single numerical value that
can arbitrarily go up and down)
$registry
->getOrRegisterGauge(
$namespace,
$name,
$help,
$labels
);

3. Histograms (samples observations and counts them
in buckets)
$registry
->getOrRegisterHistogram(
$namespace,
$name,
$help,
$labels
);

o11y.module
function o11y_entity_insert(EntityInterface $entity) {
/** @var Drupalo11yMetrics $metrics */
$metrics = Drupal::service('o11y.metrics');
$counter = $metrics
->getOrRegisterCounter(
'drupal',
'entity_insert',
'Insert a new entity',
['type', 'bundle']
);
$counter
->incBy(
1,
[
$entity->getEntityTypeId(),
$entity->bundle()
]
);
}

The module esposes an URL with metrics in
Prometheus format (/metrics)
# HELP drupal_entity_insert Insert a new entity
# TYPE drupal_entity_insert counter
drupal_entity_insert{type="comment",bundle="comment"} 2
drupal_entity_insert{type="node",bundle="article"} 1

2. Extract data from the system

Node exporter
Prometheus exporter for hardware and OS metrics
exposed by *NIX kernels, written in Go with pluggable
metric collectors

cpu
meminfo
filesystem
diskstats
netdev
...

Now we need to configure and run a Prometheus
server

prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'drupal'
scrape_interval: 5s
static_configs:
- targets: ['apache:80']
- job_name: 'nodeexporter'
scrape_interval: 5s
static_configs:
- targets: ['nodeexporter:9100']

docker-compose.yml
prometheus:
image: prom/prometheus:v2.13.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_librari
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention=200h'
- '--web.enable-lifecycle'
expose:
- 9090
ports:
- 9090:9090

How to query and visualize data?

Grafana
Grafana allows you to query, visualize, alert on and
understand your metrics (and logs)

docker-compose.yml
grafana:
image: grafana/grafana:6.4.3
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
expose:
- 3000
ports:
- 3000:3000

Grafana uses PromQL to let the user select and
aggregate time series data in real time

// PHP request per second
increase(php_request[5m])
// Entity created per second
increase(drupal_entity_insert{bundle="page"}[5m])
// PHP Memory peak
avg_over_time(php_memory_peak[5m])
// CPU usage
rate(node_cpu_seconds_total{mode="user"}[5m])

Now that we have Grafana up and running we can use
it also for viewing logs
Loki is a new project from Grafana Labs to scrape and
aggregate logs inspired by Prometheus

DISTRIBUTEDDISTRIBUTED
TRACESTRACES

2. Metrics are a measurement at a point in time for the
system
3. Distributed traces deals with information that is
request-scoped

https://opentracing.io
Vendor-neutral APIs and instrumentation for
distributed tracing
OpenTracing will be merged with OpenCensus to a
new project call OpenTelemetry

Per-process logging and metric monitoring have their
place, but neither can reconstruct the elaborate
journeys that transactions take as they propagate
across a distributed system. Distributed traces are
these journeys

We take for example a Drupal Commerce 2 website
with product prices that comes from a remote
microservice

src/Resolver/PriceResolver.php
public function resolve(
PurchasableEntityInterface $entity,
$quantity,
Context $context) {
$resp = $this->client->request(
'GET',
'http://localhost:10080?sku=' . $entity->get('sku')->value
);
$data = json_decode($resp->getBody()->getContents());
$price = new Price((string) $data->price->price, 'EUR');
return $price;
}

We want to trace the jouney of a request that arrives to
Drupal, traverses all Drupal layers, goes to the external
service and comes back

https://www.jaegertracing.io/
Distributed tracing system released as open source by
Uber Technologies
Jaeger clients are language specific implementations
of the OpenTracing API

There are diﬀerent Jaeger clients for PHP
The most complete at the moment seems
jukylin/jaeger-php (the code is a little bit messy )

The O11y module instrument the application to:
generate a TraceID
trace Events
trace Twig
propagate TraceID to Guzzle calls

src/StackMiddleware/OpentracingMiddleware.php
public function handle(
Request $request,
$type = self::MASTER_REQUEST
$catch = TRUE) {
$span = $this->tracing->startSpan(
'Request',
[
'http.method' => $request->getMethod(),
'http.url' => $request->getRequestUri()
]
);
$handle = $this->httpKernel->handle($request, $type, $catch);
$span->finish();
$this->tracing->flush();
return $handle;
}

src/Http/ClientFactory.php
class ClientFactory extends CoreClientFactory {
public function fromOptions(array $config = []) {
$headers = [];
$tracer = $this->tracing->getTracer();
$span = $this->tracing->startSpan('external_call');
$tracer->inject(
$span->getContext(),
FormatsTEXT_MAP,
$headers
);
return parent::fromOptions(['headers' => $headers]);
}
}

One last thing we need is to correlate traces with logs,
so when we found a problem with a request we can go
from the trace to the logs (and viceversa)

The O11y module provides a new processor for
Monolog that adds a trace_id argument to every log

parameters:
monolog.channel_handlers:
default:
handlers: ['rotating_file']
formatter: 'json'
monolog.processors: [
'message_placeholder',
'current_user',
'request_uri',
'ip',
'referer',
'filter_backtrace',
'introspection',
'opentracing'
]

{
"message": "Session opened for admin.",
"context": [],
"level": 250,
"level_name": "NOTICE",
"channel": "user",
"datetime": {
"date": "2019-10-06 19:50:34.641160",
"timezone_type": 3,
"timezone": "UTC"
},
"extra": {
"file": "/var/www/html/web/core/modules/user/user.module",
"line": 556,
"class": null,
"function": "user_login_finalize",
"referer": "http://web.d86.docker.localhost/user/login",
"ip": "172.20.0.2",
"request_uri": "http://web.d86.docker.localhost/user/login
"uid": "1",
"user": "admin",
"trace_id": "15d12733c6ed916115d12733c6eb1881"
}
}

Next steps
Finish the O11y module
Integrate O11y with Webprofiler
Write instructions to setup all the stack

References
https://peter.bourgon.org/blog/2017/02/21/metrics-
tracing-and-logging.html
https://medium.com/opentracing/towards-turnkey-
distributed-tracing-5f4297d1736
https://medium.com/jaegertracing/jaeger-and-
opentelemetry-1846f701d9f2

Do you know what your drupal is doing? Observe it!

Recommended

More Related Content

Similar to Do you know what your drupal is doing? Observe it! (20)

More from Luca Lusso (7)

Recently uploaded (20)

Do you know what your drupal is doing? Observe it!