Software Telemetry
Jamie Riedesel
  • MEAP began June 2020
  • Publication in Summer 2021 (estimated)
  • ISBN 9781617298141
  • 475 pages (estimated)
  • printed in black & white

Do you want to learn more about software telemetry? Don't look any further, this book is the one you need.

Sander Zegveld
Software telemetry is the discipline of tracing, logging, and monitoring infrastructure by observing and analyzing the events generated by the system. In Software Telemetry, you’ll master the best practices for operating and updating telemetry systems. This practical guide is filled with techniques you can apply to any organization upgrading their telemetry systems, from lean startups to well-established companies. You’ll learn troubleshooting techniques to deal with every eventuality, such as building easily-auditable systems, preventing accidental data leaks, and ensuring compliance with standards like GDPR.

About the Technology

Complex systems can become black boxes. Telemetry provides feedback on what’s happening inside. Telemetry systems are built for gathering, transforming, and communicating data on the performance, functionality, processing speeds, errors, and security events of production systems. There are many forms of telemetry systems, from classic centralized logging to cutting-edge distributed tracing that follows data across microservices. But despite their difference in functionality, all telemetry systems share core operational similarities—and best practices for optimizing them to support your business needs.

About the book

Software Telemetry is a guide to operating the telemetry systems that monitor and maintain your applications. It takes a big picture view of telemetry, teaching you to manage your logging, metrics, and events as a complete end-to-end ecosystem. You’ll learn the base architecture that underpins any software telemetry system, allowing you to easily integrate new systems into your existing infrastructure, and how these systems work under the hood. Throughout, you’ll follow three very different companies to see how telemetry techniques impact a greenfield startup, a large legacy enterprise, and a non-technical organization without any in-house development. You’ll even cover how software telemetry is used by court processes—ensuring that when your first telemetry subpoena arrives, there’s no reason to panic!
Table of Contents detailed table of contents

1 Introduction

1.1 Defining the styles of telemetry

1.1.1 Defining centralized logging

1.1.2 Defining metrics

1.1.3 Defining observability

1.1.4 Defining distributed tracing

1.1.5 Defining Security Information Event Management

1.2 How telemetry is consumed by different teams

1.2.1 Telemetry usage by Operations, DevOps, and SRE teams

1.2.2 Telemetry usage by Security and Compliance teams

1.2.3 Telemetry usage by Software Engineering and SRE teams

1.2.4 Telemetry usage by Customer Support teams

1.2.5 Telemetry usage by Business Intelligence

1.3 Challenges facing telemetry systems

1.3.1 Chronic under-investment harms decision making

1.3.2 Diverse needs resist standardization

1.3.3 Information spills and cleaning them up

1.3.4 Court-orders break your assumptions

1.4 What you will learn

1.5 Summary

Part 1: Telemetry System Architecture

2 The emitting stage: creating and submitting telemetry

2.1 Emitting from production code

2.1.1 Emitting telemetry into a log-file

2.1.2 Emitting telemetry into the system-log

2.1.3 Emitting telemetry into standard output

2.1.4 Formatting telemetry for emissions

2.2 Emitting from hardware

2.2.1 Explaining the Simple Network Management Protocol

2.2.2 Ingesting telemetry from a Cisco ASA firewall

2.3 Emitting from as-a-Service systems

2.3.1 Emitting events from a Software-as-a-Service system

2.3.2 Emitting events from Infrastructure-as-a-Service systems

2.4 Summary

3 The shipping stage: moving and storing telemetry

3.1 Emitter/Shipper functions, telemetry from production code

3.1.1 Shipping directly into storage

3.1.2 Shipping through queues and streams

3.1.3 Shipping to Software-as-a-Service systems

3.2 Shipping between Software-as-a-Service systems

3.3 Tipping points in Shipping Stage architecture

3.4 Summary

4 The shipping stage: unifying diverse telemetry formats

4.1 Shipping locally emitted telemetry

4.1.1 Shipping telemetry from a log-file

4.1.2 Shipping telemetry from the system logger

4.1.3 Shipping telemetry from standard-out

4.2 Unifying diverse emitting formats

4.2.1 Picking a shipping format

4.2.2 Converting Syslog into JSON or other object-encoding formats

4.2.3 Designing with cardinality in mind

4.3 Summary

5 The presentation stage: displaying telemetry

5.1 Displaying telemetry in metrics systems

5.1.1 Making pretty pictures with telemetry

5.1.2 Feeding the graphs with aggregation functions

5.1.3 Using aggregations with pdf_pages

5.2 Displaying telemetry in centralized logging systems

5.2.1 Needed features in a display system for centralized logging

5.2.2 Demonstrating centralized logging display

5.3 Displaying telemetry in security systems

5.4 Displaying telemetry in observability and distributed tracing systems

5.5 Displaying telemetry in large organizations

5.6 Summary

6 Marking up and enriching telemetry

6.1 Markup in the Emitting Stage

6.2 Markup and enrichment in the Shipping Stage

6.2.2 Extracting and enriching telemetry in-flight

6.2.3 Converting field types during the Shipping Stage

6.3 Enrichment in the Presentation Stage

6.4 How telemetry style affects markup and enrichment

6.4.1 Markup and enrichment with centralized logging

6.4.2 Markup and enrichment with SIEM systems

6.4.3 Markup and enrichment with metrics

6.4.4 Markup and enrichment with observability and distributed tracing systems

6.5 Summary

7 Handling multi-tenancy

7.1 How multi-tenant architectures come about

7.1.1 Evolving multi-tenancy in an early stage startup

7.1.2 Evolving multi-tenancy in a culture of free sharing

7.1.3 Evolving multi-tenancy in a culture of strong separation

7.2 Designing for multi-tenancy

7.2.1 Multi-tenancy in the Shipping Stage

7.2.2 Multi-tenancy in the Presentation Stage

7.3 Summary

Part 2: Use-cases revisited, applying architecture concepts

8 Growing cloud-based startup

9 Non-startup business

10 Long-established business IT

Part 3: Techniques for handling telemetry

11 Optimizing for regular expressions at scale

12 Standardizing logging and event formats

13 Using non-file emitting techniques

14 Managing cardinality

15 Ensuring metadata integrity

16 Redacting and reprocessing telemetry

17 Building policies for metadata retention and aggregation

18 Surviving legal processes

19 Applying metadata principles

What's inside

  • Processes for legal compliance
  • Cleaning up after data spills and leaks
  • Multi-tenant transformation processes
  • Updating metrics aggregation and sampling traces to display accurate data for longer
  • Handling toxic telemetry of confidential records
  • Revising software telemetry emissions to be easier to parse
  • Justifying increased spend on telemetry software

About the reader

For software developers and infrastructure engineers of any background.

About the author

Jamie Riedesel is a lead engineer at Dropbox. She has over twenty years of experience in IT, working in both government, education, legacy companies, and startups. She has specialized in DevOps for the past decade, running distributed systems in public clouds and designing software telemetry architectures.

placing your order...

Don't refresh or navigate away from the page.
Manning Early Access Program (MEAP) Read chapters as they are written, get the finished eBook as soon as it’s ready, and receive the pBook long before it's in bookstores.
print book $35.99 $59.99 pBook + eBook + liveBook
Additional shipping charges may apply
Software Telemetry (print book) added to cart
continue shopping
go to cart

eBook $33.59 $47.99 3 formats + liveBook
Software Telemetry (eBook) added to cart
continue shopping
go to cart

Prices displayed in rupees will be charged in USD when you check out.

FREE domestic shipping on three or more pBooks