12 Uncovering system-level failures and service communication issues

This chapter presents a pragmatic guide to diagnosing and fixing failures that emerge not from a single Java service, but from the interactions among many. It frames multi-service troubleshooting as a system problem, showing how seemingly unrelated symptoms—timeouts, retries, or slowdowns—can echo across APIs, queues, and background workers. The emphasis is on cultivating system-level visibility and thinking so that you can trace a request’s journey, connect symptoms to root causes, and stop localized glitches from turning into outages.

The discussion begins with service communication patterns—REST, gRPC, and messaging—highlighting their trade-offs and the subtle failure modes they introduce, from unclear contracts and weak typing to ordering, duplication, and delivery guarantees. It explains the foundations of distributed tracing—trace IDs, spans, and propagation—and how tools built on OpenTelemetry, together with backends like Jaeger or Zipkin, help reconstruct end-to-end request timelines. The chapter complements tracing with error monitoring (e.g., capturing exceptions and context), then dives into serialization and versioning pitfalls across JSON, Protobuf, and Avro, including how schema changes break consumers. It offers practical safeguards such as selective payload logging, redaction, CI-based schema compatibility checks, and schema registries to prevent silent data loss and deserialization errors.

Finally, it examines systemic failure modes that magnify small issues: cascading failures that exhaust thread pools, retry storms that overwhelm struggling services, timeout mismatches that create false failures, and misbehaving circuit breakers. You learn how to spot these patterns using thread dumps, structured logs, service metrics, and distributed traces, with particular attention to percentiles (P95/P99) for uncovering tail latencies. The guidance centers on correlating signals across tools, aligning client/server timeouts, applying backoff and jitter to retries, tuning pools and limits, and treating observability as core infrastructure. The closing message is clear: effective troubleshooting in distributed Java systems blends the right instrumentation with disciplined investigation to turn chaos into clarity.

An overview of the system for our scenario. The checkout service calls the order service via REST while the order service asynchronously talks to the fulfillment service through a Kafka topic. The fulfillment service stores details in a database.

A trace ID identifies each message so you can follow it throughout its road between multiple services.

OpenTelemetry collects relevant data from the running service. Jaeger and Zipkin are tools that aggregate and display nicely this data to help you easily troubleshoot the system.

Sentry aggregates and shows events such as exceptions providing a large field of details for each of these.

A failure or slowdown in one service (e.g., due to high latency, blocked resources, or an overloaded dependency) can propagate through the system. Upstream services that rely on it may begin to stall, exhaust thread pools, or trigger retries—eventually causing multiple services to degrade or fail, even if they were functioning correctly on their own.

A retry storm. Due to a failing operation, an app uncontrollably retries the execution of an action, putting a larger part of the system in danger.

When a downstream service becomes unresponsive, upstream services may begin retrying requests excessively. This amplifies load across the system, consuming resources and potentially turning a localized failure into a widespread outage.

The Checkout Service aborts the request after 2 seconds due to its configured timeout, while the Order Service completes its task in 3 seconds. This mismatch results in a failed request, even though the downstream service performs correctly—highlighting the need for consistent timeout configuration across services.

Identifying Timeout Mismatches Using Percentiles. P95 and P99 latencies approach or exceed the 2000 ms timeout threshold, revealing that some requests are likely to fail—despite the service working as expected.

• Distributed failures often don’t stay local—troubleshooting them requires system-level visibility and tools.
• Examine trace IDs and spans to reconstruct request paths, detect missing steps, and isolate problematic services in multi-hop flows. Use distributed tracing tools (like Jaeger or Zipkin) to follow a request across services and identify where time is lost or failures originate.
• In cases of serialization issues, log decoded payloads carefully, validate schemas in CI, and use tools like protoc, grpcurl, and jq to inspect data outside of the application.
• Detect cascading failures by analyzing thread dumps for repeating stack traces stuck on remote calls—often a sign of thread pool exhaustion.
• Identify retry storms by looking for repeating requests in logs, short retry spans in tracing tools, and sudden spikes in internal traffic without increased user load.
• Investigate timeout mismatches by comparing client and server span durations—if clients time out while servers keep processing, your configuration is misaligned.
• Watch for duplicate request IDs as a signal that clients are retrying unnecessarily due to mismatched expectations or timeouts.
• AI assistants can assist with analyzing logs, interpreting stack traces, spotting anomalies in traces, or offering context on unfamiliar errors, helping you troubleshoot faster and with more confidence.

FAQ