Logging & Metrics with Docker
Download as PPTX, PDF7 likes2,274 views
The document discusses a comprehensive monitoring solution for logging and metrics in Docker environments, highlighting challenges such as managing logs from multiple containerized applications. It details various methods for emitting logs, including using syslog, Docker logging drivers, and frameworks, and emphasizes the need for centralized log management. Additionally, it introduces the Sumo Logic Docker source as a solution for effective log collection and monitoring.
![docker run -v /tmp/clogs:/tmp/clogs -d
--name="sumo-logic-collector"
sumologic/collector:latest-file [Access ID] [Access key]](https://image.slidesharecdn.com/kansascitydevopseventsv1-150714104046-lva1-app6891/85/Logging-Metrics-with-Docker-10-320.jpg)
![docker run -d -p 514:514 -p 514:514/udp
--name="sumo-logic-collector”
sumologic/collector:latest-syslog [Access ID] [Access key]](https://image.slidesharecdn.com/kansascitydevopseventsv1-150714104046-lva1-app6891/85/Logging-Metrics-with-Docker-13-320.jpg)
![What Docker provides
Captures stdout/stderr
Feeds it to logging drivers
docker logs command
Returns the entire log every time
Works with json-file driver only
Can tail logs
docker logs –tf –-tail 0 [ID]](https://image.slidesharecdn.com/kansascitydevopseventsv1-150714104046-lva1-app6891/85/Logging-Metrics-with-Docker-20-320.jpg)
Logging & Metrics with Docker
- 1. Logging & Metrics With Docker A Comprehensive Monitoring Solution Stefan Zier June 13th, 2015
- 2. whoami Infrastructure, Backend Dev/Architect Chief Architect, Sumo Logic, since 2010 Server & Infrastructure, ArcSight (HP), 2001-2010
- 3. Mandatory Slide Showing Shipping Containers
- 4. Docker – What’s making debugging hard? One more layer of abstraction Container per app = File system per process File systems short lived, transient Resource schedulers = no container affinity to host
- 5. What Our Customers Are Telling Us We have one process per container We like to log to stdout We have multiple processes per container We run the Sumo Logic collector on the Docker host We are looking into using Beanstalk with Docker We are using Amazon ECS Everyone here loves Docker We are logging straight from the application We are using /dev/log for Syslog We want immutable infrastructure
- 6. Goal Get logs from our containerized applications to a centralized logging platform.
- 7. How do apps emit logs Append to a file Use syslog() Use log4j, log4net, slf4, etc. printf() to stdout
- 8. Getting logs out of the container - Files Use VOLUME to mount a host directory Collect files from the host Collect files from another container sharing the VOLUME Need to manage disk space, i.e. rotate logs App (where supported) Host Yet another container with logrotate
- 10. docker run -v /tmp/clogs:/tmp/clogs -d --name="sumo-logic-collector" sumologic/collector:latest-file [Access ID] [Access key]
- 11. Getting logs out - Syslog VOLUME /dev/log from host and use host syslogd Run a syslogd inside the container Emit TCP/UDP Write to a file using VOLUME Emit syslog TCP/UDP directly from the app
- 13. docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector” sumologic/collector:latest-syslog [Access ID] [Access key]
- 14. Getting logs out – Logging frameworks Sumo Logic blog on official collector images http://www.sumologic.com/blog/company/an-official-docker-image-for- the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker Rainer Gerhards on Rsyslog’s file input module http://www.slideshare.net/rainergerhards1/using-wildcards-with- rsyslogs-file-monitor-imfile OWASP Log Injection https://www.owasp.org/index.php/Log_injection
- 15. Getting logs out – Logging frameworks Directly to network destinations HTTP/HTTPS Also support files, stdout, etc.
- 17. Getting logs out – Logging frameworks Various application stacks http://help.papertrailapp.com/ Log4J https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html Apache Web Server http://httpd.apache.org/docs/trunk/mod/mod_syslog.html https://raymii.org/s/snippets/Apache_access_and_error_log_to_syslog.html Nginx http://nginx.org/en/docs/syslog.html Postgres http://www.postgresql.org/docs/9.1/static/runtime-config-logging.html Sumo Logic blog on official syslog collector image http://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector https://github.com/SumoLogic/sumologic-collector-docker
- 18. Getting logs out – stdout Simply printf() Logging framework to console Symlink to /dev/stdout or /dev/stderr Configure paths to /dev/stdout or /dev/stderr RUN ln -sf /dev/stdout /var/log/nginx/access.log RUN ln -sf /dev/stderr /var/log/nginx/error.log
- 20. What Docker provides Captures stdout/stderr Feeds it to logging drivers docker logs command Returns the entire log every time Works with json-file driver only Can tail logs docker logs –tf –-tail 0 [ID]
- 21. Docker Logging Drivers Configured on docker run stdout and stderr dispatched to drivers json-file (default pre 1.6) syslog journald No stats, no events json-file driver Output unbounded, can fill up the host disk Requires logrotate on the Docker host https://github.com/docker/docker/issues/7333
- 22. Stats
- 23. Docker Stats Per-container cgroups metrics (like docker stats) Memory CPU Block I/O CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O collector 2.23% 232.6 MiB/2 GiB 11.36% 191.9 KiB/636.3 KiB
- 24. Requirements How would we want it to work?
- 25. What information do we want to collect? Timestamp Log message Docker host info Container ID Image ID Process ID
- 26. How should it work? Use docker logging infrastructure Minimal moving parts Containerized - don’t touch the host Complete – pick up all available data Automatically discover new containers
- 27. Docker API The solution maybe?
- 28. Docker API Docker daemon has a REST API TCP or unix socket Streaming APIs Docker Events (container lifecycle updates) Container Stats (CPU, memory used, …) App Logs (container stdout/stderr)
- 29. Collecting via Docker API Discover new containers via events Start streaming their logs and stats When they go away, stop Do all of this via the API Send all of it to centralized log management
- 30. Collecting via Docker API, continued Single component to do it Zero footprint on the host Follows Docker standard way of logging
- 31. One more thing…
- 32. Introducing: Sumo Logic Docker Source
- 33. Sumo Logic Docker Source Active development Early access expected later this year
- 34. Demo Time