Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with OpenTelemetry
Download as PPTX, PDF0 likes27 views
It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).
Ad
More Related Content
Similar to Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with OpenTelemetry (20)
More from Eric D. Schabell (20)
Ad
Recently uploaded (20)
Ad
Mastering Fluent Bit: Ultimate Guide to Integrating Telemetry Pipelines with OpenTelemetry
- 1. chronosphere.io Mastering Fluent Bit Ultimate Guide to Integrating Telemetry Pipelines with OpenTelemetry Eric D. Schabell Director Community & Developer @schabell.org | @ericschabell{@fosstodon.org } Tech Meetup Glasgow - 1 May 2025
- 2. chronosphere.io Fluent Bit vs OTel Collector
- 4. chronosphere.io ● Logs first… ● Written in C ● Scales in edge & containerized environments ● Shines in high throughput & low latency
- 5. chronosphere.io OTel Collector ● Traces first… ● Written in Go ● Registry of collector components (vendors + community) ● Internal OTLP-native (standards)
- 6. chronosphere.io ● Fluent Bit <-> OTel Envelope ● OTel Collector <-> Fluent Bit Forward
- 7. chronosphere.io Use cases: ○ Fluent Bit - collection at edge, resource constrained ○ OTel Collector - centralized aggregation / advanced processing ○ Can run together as signal specific pipelines
- 10. chronosphere.io Input phase The first phase is INPUT, which is where Fluent Bit uses Input Plugins to gather information from specific sources.
- 11. chronosphere.io Parser phase The second phase is PARSER, which is where unstructured input data is turned into structured data.
- 12. chronosphere.io Filter phase Filtering phase is when we modify, enrich, or delete any of the collected Events.
- 13. chronosphere.io Buffer phase Buffering is where the data is stored, using in-memory or the file system based options.
- 14. chronosphere.io Routing phase This phase is where Fluent Bit uses Tag and Match concepts to determine which output destinations to send data.
- 15. chronosphere.io Output Phase The final phase is where Fluent Bit uses Output Plugins to connect with specific destinations.
- 16. chronosphere.io Let’s look under the hood Pipeline demo
- 17. chronosphere.io Host Pipelines and Prometheus Observability Backend (Prometheus, Jaeger, etc.), Telemetry pipeline scrape metrics metrics endpoint
- 18. chronosphere.io Q u e r y i n g Host Pipelines and Prometheus Telemetry pipeline scrape metrics metrics endpoint Alert Manager Dashboards WebUI Perses PromLens Email PagerDuty Slack etc…
- 19. chronosphere.io Let’s look under the hood Pipeline with metrics demo
- 20. chronosphere.io Host Pipelines to OpenTelemetry (input to console) Telemetry pipeline console output
- 21. chronosphere.io Host Pipelines to OpenTelemetry (input to console + envelope) Telemetry pipeline console output OTel envelope
- 22. chronosphere.io Pipelines and OpenTelemetry Host Observability Backend (Prometheus, Jaeger, etc.), Telemetry pipeline OTel Collector OTLP OTLP OTel console envelope console
- 23. chronosphere.io Console logging File logging: /tmp/output.json Pipelines and OpenTelemetry Host Telemetry pipeline OTel Collector OTLP OTLP OTel console envelope console
- 24. chronosphere.io Let’s look under the hood Pipeline with OpenTelemetry demo
- 28. chronosphere.io Questions? Eric D. Schabell Director Community & Developer @schabell.org | @ericschabell{@fosstodon.org } Tech Meetup Glasgow - 1 May 2025
Editor's Notes
- #1: It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).
- #3: Fluent Bit and OpenTelemetry Collector are both powerful telemetry collectors within the CNCF ecosystem. Both aim to collect, process, and route telemetry data and support all telemetry types. But they emerged from different projects with different strengths. FB started with logs and OTel started with traces. The common narrative suggests you must choose one or the other. But these projects can and should coexist. Many teams are successfully using both – leveraging each for what it does best. There are also times that an organization may choose one over the other depending on their system.
- #4: Fluent Bit: Fluent Bit was initially purpose-built for logs, which remain the best supported telemetry signal. Written in C, Fluent Bit is blazingly efficient with resources providing exceptional performance. Its ideal for edge and containerized environments at scale. Fluent Bit is also a CNCF graduated project with wide adoption, celebrating its 10 year anniversary! With a robust plugin ecosystem, that enables integration with a variety of input sources and output destinations. Beyond the provided plugins, you can extend Fluent Bit with Lua, WASM (WebAssembly), Go, and now with v4 Zig. It shines when used in environments requiring high throughput and low latency.
- #5: OTel Collector: OpenTelemetry Collector started out supporting distributed traces, still the most mature signal followed by metrics and logs. The OTel Collector is written in Go, a familiar language within the CNCF ecosystem. And is also backed by a rich registry of collector components from vendors and community contributors alike. The OpenTelemetry Collector is the natural choice if you're fully invested in the OpenTelemetry ecosystem. Its internal representations are OTLP-native, meaning new protocol developments are adopted quickly. Resource detection is well-supported and Semantic Conventions are first-class citizens.
- #6: Both projects are committed to supporting open observability and that includes speaking each others’ protocols. In practice that means: Fluent Bit can transform existing logs into the OTel Log schema with the OTel Envelope Offers OTLP Endpoints to ingest data using OTel’s protocol as well as an Exporter to send telemetry via OTLP. The OpenTelemetry Collector also has a Receiver and Exporter that enable you to ingest telemetry via Fluent’s Forward protocol
- #7: And they work beautifully together when: Fluent Bit handles collection at the edge or resource-constrained environments The Collector provides centralized aggregation and advanced processing Or you can run both separately as signal-specific pipelines What's crucial to remember is that both projects are constantly evolving. Fluent Bit is expanding its tracing and metrics capabilities. The OpenTelemetry Collector is improving its performance and adding new components. The choice that makes sense today should be reevaluated regularly, as feature gaps close and new capabilities emerge. This dynamic evolution is exactly what makes open source so powerful – the constant drive to improve based on real user needs. OSS: The only way OSS can stay alive and well is that we all contribute. Like with so many great OSS projects you can use them straight from the community repos and with support from vendors. This is what open source is all about – freedom and choice. The ability to pick the right tool for the right job, or combine tools to create something greater than the sum of its parts. As vendors and contributors, our job isn't to build walls between these projects. It's to build bridges. To make them work better together. To give users options rather than limitations.
- #9: This is the telemetry pipeline overview, all of the phases data goes through that you will learn in our workshop: https://o11y-workshops.gitlab.io/workshop-fluentbit
- #10: The first phase is INPUT, which is where Fluent Bit uses Input Plugins to gather information from specific sources. When an input plugin is loaded it creates an instance which we can configure using the plugins properties.
- #11: The second phase is PARSER, which is where unstructured input data is turned into structured data. Fluent Bit does this using Parsers that we can configure to manipulate the unstructured data producing structured data for the next phases of our pipeline.
- #12: Filtering phase is when we modify, enrich, or delete any of the collected Events. Fluent Bit provides many out of the box plugins as Filters that can match, exclude, or enrich your structured data before it moves onwards in the pipeline. Filters can be configured using the provided properties.
- #13: Buffering in the pipeline is where the data is stored, using in-memory or the file system based options. Note that when data reaches the buffer phase it's in an immutable state (no more filtering) and that buffered data is not raw text, but in an internal binary representation for storage.
- #14: The next phase is ROUTING, which is where Fluent Bit uses the previously discussed Tag and Match concepts to determine which output destinations to send data. During the INPUT phase data is assigned a Tag, during the ROUTING phase data is compared to Match rules from output configurations, if it matches then the data is sent to that output destination.
- #15: The final phase is OUTPUT, which is where Fluent Bit uses Output Plugins to connect with specific destinations. These destinations can be databases, remote services, cloud services, and more.
- #16: Example 1: Input log -> Fluent Bit -> processing (enhance / reduce) -> Output source (Console)
- #17: Preview example 2a: Input log -> Fluent Bit -> metrics endpoint <- Prometheus
- #18: Preview example 2b: Input log -> Fluent Bit -> metrics endpoint <- Prometheus
- #19: Example 2: Input log -> Fluent Bit -> metrics endpoint <- Prometheus
- #20: Preview example 3a: Input log -> Fluent Bit -> console output
- #21: Preview example 3b: Input log -> Fluent Bit -> console output + OTel envelope
- #22: Preview example 3c: Input source (log, metric, trace) -> Fluent Bit -> OTel envelope -> OTel collector (console). This example is to showcase filtering any source into OTel collection architecture.
- #23: Example 3d: Input source (log, metric, trace) -> Fluent Bit -> OTel envelope -> OTel collector (console + file). This example is to showcase filtering any source into OTel collection architecture.
- #24: Example 3: Input log -> Fluent Bit -> OTel Envelope -> OTel Collector -> Console + File
- #26: Fluent Bit (pipelines) - This workshop will guide you through the open source project Fluent Bit, what it is, a basic installation, and setting up a first cloud native observability pipeline project. The workshop is self-paced and available online, so attendees can continue to explore after the event: https://o11y-workshops.gitlab.io/workshop-fluentbit
- #27: Explore this instrumenting of your applications in this workshop, where a Java example for developers has been created for them to experience what it’s like to instrument an application in lab 8, see entire workshop at https://bit.ly/prom-workshop.
- #28: It's time you stopped letting your telemetry data pressure your budgets and get in the way of solving issues with agility! No more I say! Take back control of your telemetry data as we guide you through the open source project Fluent Bit. Learn how to manage your telemetry data from source to destination using the pipeline phases covering collection, parsing, aggregation, transformation, and forwarding from any source to any destination. Buckle up for a fun ride as you learn by exploring how telemetry pipelines work, how to set up your first pipeline, and exploring several common use cases that Fluent Bit helps solve. All this backed by a self-paced, hands-on workshop that attendees can pursue at home after this session (https://o11y-workshops.gitlab.io/workshop-fluentbit).
- #29: Let's face it. Kubernetes clusters generate logs. Lots of 'em. Often, these logs contain valuable information when it comes to troubleshooting. Routing and analyzing logs should be straightforward and reliable. This is where Telemetry Pipelines shine and make your day easier! In this demo, a telemetry pipeline is created to route logs from a Kubernetes cluster to multiple destinations. Create your first telemetry pipeline today! Visit chronosphere.io to learn more.