Unified Logging with Fluentd
Phil Wilkins
  • MEAP began June 2020
  • Publication in Early 2021 (estimated)
  • ISBN 9781617298356
  • 275 pages (estimated)
  • printed in black & white
Fluentd is a powerful log management tool that seamlessly handles messy logging data, from operational errors, to application events, and security events. It decouples log data, such as SNMP or slow database queries, from backend systems and easily sends it where it needs to go—thanks to 500+ flexible plugins covering all major services. Unified Logging with Fluentd teaches you how to use this free and open source tool to make your log processing a real asset for your application. You’ll learn how to configure Fluentd to solve common log problems, and how proper log management can improve performance and make management of software solutions easier. Through useful examples like sending log driven events to Slack, you’ll get hands-on experience applying structure to your unstructured data.

About the Technology

Fluentd is an open source log management tool supported by the CNCF that unifies your data collection in a language- and platform-agnostic manner. It brings together data from your databases, system logs, and application events, filters out the noise, and then structures that data so it can be easily fed out to multiple destinations. Through its flexible plugin architecture, Fluentd works with hundreds of different services, from commercial products like Splunk to open source tools like ElasticSearch or MongoDB. Prized for microservices architecture, Fluentd is also an excellent choice for legacy and monolithic applications. Its reduced footprint sibling Fluent Bit is even applicable for the Internet of Things.

About the book

Unified Logging with Fluentd is a guide to streamlining your log processing, turning logged data into a tool that will boost your application’s performance. It’s packed with tips and tricks on how proper Fluentd log management can improve your application’s security, speed, and cost-effectiveness. You’ll go beyond the basics of choosing your Fluentd plugins, and take a peek behind the scenes at how different configurations will impact and improve the way your systems function. By the time you’re done, you’ll be able to reliably configure Fluentd so that it can easily scale to handle even the largest data loads.
Table of Contents detailed table of contents

Part 1: Overview

1 Introduction to Fluentd

1.1 Why do we produce logs?

1.1.1 Four Golden Signals

1.1.2 Three Pillars of Observability

1.2 Unifying logs vs log analytics

1.2 Unifying logs vs log analytics

1.4 Software stacks

1.5 Log Event Lifecycle

1.6 Log routing as a vehicle for security

1.7 Evolution of Fluentd

1.7.1 Treasure Data

1.7.2 CNCF

1.7.3 Relationship to major cloud vendors PaaS/IaaS

1.8 The relationship between Fluentd and Fluent-Bit

1.9 Where can it be used

1.9.1 Platform constraints

1.10 Fluentd configuration UI based editing

1.11 Plugins

1.12 Use Cases for Fluentd and Fluent Bit

1.12.1 Actionable Log Events

1.12.2 Making Logs More Meaningful

1.12.3 Polyglot Environments

1.12.4 Multiple Types of Source and Target

1.12.5 Log Data Costs

1.12.6 Logs to metrics

1.13 Summary

2 Concepts, architecture and deployment of Fluentd

2.1 Architecture and Core Concepts

2.1.1 The makeup of a Log Event

2.1.2 Handling time

2.1.3 Architecture of Fluentd

2.1.4 Fluent configuration execution order

2.1.5 Exercise 2.1

2.2 Deployment of Fluentd

2.2.1 Deployment considerations

2.2.2 Minimum footprint

2.2.3 Simple deployment of Fluentd

2.2.4 Deploying a log generator

2.2.5 Installing Postman

2.3 Bringing Fluentd to life with ‘Hello World’

2.3.1 Hello World scenario

2.3.2 Hello World configuration

2.3.3 Starting Fluentd

2.4 Hello World with Fluent Bit

2.4.1 Starting Fluent Bit

2.4.2 Alternate Fluent Bit startup options

2.4.3 Fluent Bit configuration file comparison

2.4.4 Fluent Bit configuration file in detail

2.4.5 Exercise 2.2

2.5 Fluentd deployment with Kubernetes and Containers

2.5.1 Fluentd Daemonset

2.5.2 Dockerized Fluentd

2.6 Using Fluentd UI

2.6.1 Installing Fluentd with UI

2.7 Exercise answers

2.7.1 Exercise 2.1

2.7.2 Exercise 2.2

2.8 Summary

Part 2: Using Fluentd in depth

3 Using Fluentd to capture log events

3.1 Setting up the configuration files and simulator

3.2 Dry running to check a configuration

3.2.1 Scenario 3.1 – Configuration checking

3.3 Reading Log files

3.3.1 Scenario 3.2 Adapt to Fluent Bit

3.3.2 Risk of missing log events

3.3.3 Configuration considerations for tracking data

3.3.4 Wild cards in the path attribute

3.3.5 Expressing time

3.3.6 Controlling the impact wild cards in filenames

3.3.7 Exercise 3.3 Wildcards

3.3.8 Handling log rotation

3.4 Self-Monitoring

3.4.1 HTTP interface check

3.5 Imposing Structure on log events

3.5.1 Standard Parsers

3.5.2 3rd Party Parsers

3.5.3 Applying a regex parser to a complex log

3.5.4 Exercise 3.4 Parsing

3.6 Scenario & Exercise Solutions

3.6.1 Scenario 3.1 – Configuration checking

3.6.2 Scenario 3.2 Adapt to Fluent Bit

3.6.3 Exercise 3.3 Wildcards

3.6.4 Exercise 3.4 Parsing

3.7 Summary

4 Using Fluentd to output log events

4.1 File Output Plugin

4.1.1 Basic file output

4.1.2 Exercise 4.1 Output & buffers

4.1.3 Output formatting options

4.1.4 Applying Formatters

4.1.5 Scenario 4.2 formatters

4.2 MongoDB for output

4.2.1 Installing MongoDB

4.2.2 Deploying MongoDB plugin

4.2.3 Configuring the Mongo output plugin for Fluentd

4.2.4 Exercise 4.3 MongoDB Connectivity

4.3 Actionable log events

4.3.1 Actionable through service invocation

4.3.2 Actionable through user interaction tools

4.4 Slack to demonstrate social output

4.5 The right tool for the right job

4.6 Exercise and Scenario Answers

4.6.1 Exercise 4.1 Output & buffers

4.6.2 Scenario 4.2 formatters

4.6.3 Exercise 4.3 MongoDB Connectivity

4.7 Summary

5 Routing log events

6 Filtering and formatting

7 Boosting performance with buffers and caching

Part 3: Beyond the basics

8 Using Fluentd in a distributed and scalable manner

9 Zero Downtime Deployments

Part 4: Beyond the Fluentd - Logging Practices and Frameworks

10 Logging best practises

11 Logging frameworks – overview and illustration


Appendix A: Processing Times & Dates and Regular Expressions

A.1 Expressing relative time

A.2 Expressing dates and times

A.3 Expressing sizes

A.4 Regular expressions

A.4.1 Escape Codes

A.4.2 Repetition / Selection

A.4.3 Anchors, Groups and Alternates

Appendix B: Useful resources

B.1 Helpful Fluentd Resources

B.2 Helpful Fluentd 3rd Party Tools

B.3 Helpful Logging Practices Resources

B.4 Common Log Formats & descriptions

B.5 Helpful Ruby Resources

B.6 Kubernetes

B.8 Redis

B.9 Other handy sources of information

What's inside

  • Deploy Fluentd and Fluent Bit into varied environments
  • Configure Fluentd and Fluent Bit to solve common log problems
  • Fluentd for microservices
  • Connect a custom log source or destination with Fluentd’s extensibility framework
  • Create a custom plugin for niche problems
  • Logging best practices and common pitfalls

About the reader

For developers and operations people familiar with application logging or infrastructure monitor logs.

About the author

Phil Wilkins has spent over 25 years in the software industry. He works as a Technology Evangelist for Capgemini, specializing in cloud integration, APIs, and related technologies. Phil is TOGAF certified, and recognized by Oracle as an Ace Director (independent technology advocate) for his contributions to the integration and PaaS community.

placing your order...

Don't refresh or navigate away from the page.
Manning Early Access Program (MEAP) Read chapters as they are written, get the finished eBook as soon as it’s ready, and receive the pBook long before it's in bookstores.
print book $24.99 $49.99 pBook + eBook + liveBook
Additional shipping charges may apply
Unified Logging with Fluentd (print book) added to cart
continue shopping
go to cart

eBook $19.99 $39.99 3 formats + liveBook
Unified Logging with Fluentd (eBook) added to cart
continue shopping
go to cart

Prices displayed in rupees will be charged in USD when you check out.

FREE domestic shipping on three or more pBooks