5 Serializing data with protocols
This chapter explains how Apache Thrift’s protocol layer standardizes the on-the-wire representation of data so systems written in different languages and running on different platforms can interoperate. Protocols sit atop transports: a TProtocol instance reads and writes bytes through a TTransport, allowing the same serialization logic to target memory, files, or networks. Thrift offers multiple interchangeable protocols—Binary (TBinaryProtocol), Compact (TCompactProtocol), and JSON (TJSONProtocol)—all exposing the same interface so they can be selected at compile time or runtime without changing application code. While the IDL compiler usually generates all serialization logic, the chapter walks through hand-crafted examples to reveal how the pieces fit together.
Serialization is defined by the TProtocol interface, which provides read/write methods for base types (bool, byte, i16/i32/i64, double, string, binary), containers (list, set, map), structs, and RPC messages. Strings are exchanged as UTF-8; binary carries raw bytes unchanged. Containers record element counts and types; duck-typed languages can pass “compatible” iterables, though mismatches (for example, using a list where a set is expected) can have semantic and performance side effects. Structs are serialized with field metadata—type plus a 16-bit field id—rather than field names, enabling readers to skip unknown fields and tolerate reordering, which is central to interface evolution. RPC communication wraps args and results structs in messages (CALL, REPLY, EXCEPTION, ONEWAY), and protocols expose access to the underlying transport to flush buffered data when needed. Cross-language examples in C++, Java, and Python illustrate that the same data encodes identically per protocol.
The protocols differ mainly in size, speed, and interoperability. Binary is straightforward and widely supported, often slightly faster in CPU-bound scenarios but producing larger payloads. Compact uses variable-length encodings to trim unused integer bits, commonly shrinking data by 20–50% at a small compute cost; this reduction can yield higher throughput in I/O-bound workflows. JSON trades size and speed for human readability and easy integration with third-party tools and document stores. Benchmarks in the chapter show Compact outperforming Binary when writing to disk, Binary edging Compact in memory-only runs, and JSON trailing both in size and time; adding a buffered transport significantly improves file I/O by batching writes. Because all protocols share the same interface, teams can profile under realistic loads and swap protocols to fit their performance, readability, and ecosystem needs.
The Apache Thrift protocol layer.
Apache Thrift cross-language serialization with the binary protocol.
Transport/protocol dependency.
The protocol/transport stack.
Apache Thrift protocols convert language specific string encodings into UTF-8 for serialization.
Messages are the unit of communication in Apache Thrift RPC, packaging structs, containers, and values passed between clients and servers.
Apache Thrift RPC Clients send T_CALL messages to call methods on servers and servers send T_REPLY messages to clients with function call results.
Summary
Apache Thrift protocols serialize application data into a standard format readable by any Apache Thrift language library supporting that protocol. The combination of transports and protocols creates a plug-in style architecture making Apache Thrift an extensible platform for data serialization, supporting a choice of protocols and the possible addition of new serialization protocols over time.
- Apache Thrift protocols provide cross-language serialization.
- The TProtocol interface provides the abstract interface for all Apache Thrift serialization formats.
- Protocols depend on the transport layer TTransport interface to read and write serialized bytes.
- One serialization protocol can be substituted for another with little or no impact on upper layers of software.
- The TProtocol interface defines the Apache Thrift type system exposed through Apache Thrift IDL.
- Protocols support the serialization of:
- o RPC messages.
- o Structs.
- o Collections—List, set, and map.
- o Base types—Ints, doubles, strings, and so on.
- Apache Thrift supplies three main protocols:
- o Binary—The default protocol, supported by the most languages, is fast and efficient.
- o Compact—Trades CPU overhead for reduced serialization size.
- o JSON—A text-based, widely interoperable, human-readable protocol with higher CPU overhead and relatively large serialization size.
FAQ
What does the Apache Thrift protocol layer do compared to the transport layer?
The transport layer moves raw bytes to and from endpoints (memory, files, sockets). The protocol layer turns language- and platform-specific values into a shared, language-neutral serialized form (and back), handling issues like endianness and type representation. This lets, for example, a C++ object be serialized and reconstituted as a Ruby or Java object. Any protocol can sit atop any transport.Which Thrift protocols are available and how do they differ?
- Binary (TBinaryProtocol): Default and widest support. Straightforward binary layout; generally fast; medium payload size.- Compact (TCompactProtocol): Variable-length integer encoding for smaller payloads (often 20–50% smaller). Slight CPU overhead versus Binary; supported by most languages.
- JSON (TJSONProtocol): Text-based and human-readable; largest payloads, slowest; best for interoperability with tools and JSON-centric systems.
How do I use a protocol with a transport, and what is getTransport() for?
- Construct a protocol around a transport: e.g., proto = new TBinaryProtocol(trans). The protocol writes/reads via that transport for its lifetime.- getTransport() returns the underlying transport, commonly used to flush buffers when needed (for example, after writing an RPC message).
- Because protocols depend only on the abstract TTransport, you can serialize to memory, disk, or over the network with the same protocol code.
How are base types, strings, and binary data serialized?
- Base types: Each has read/write methods (writeBool, writeI32, writeDouble, writeString, writeBinary, etc.).- Strings: Protocols use UTF‑8 on the wire. Languages with UTF‑16 (e.g., Java, .NET) convert to/from UTF‑8 during serialization.
- Python tip: You can force UTF‑8 string handling in generated code with: thrift -gen py:utf8strings
- Binary: Treated as raw bytes; protocols don’t transform it. Consumers need out‑of‑band knowledge of the binary’s meaning.
How are lists, sets, and maps serialized, and are uniqueness checks enforced?
- Containers use begin/end calls (writeListBegin/End, writeSetBegin/End, writeMapBegin/End). Begin records element count (and element/key/value types).- Containers can hold base types, other containers, or structs.
- Uniqueness: Thrift doesn’t enforce set or map key uniqueness while writing; it avoids overhead. The target language’s container enforces semantics when reading (e.g., duplicates dropped when loaded into a set).
How do structs get serialized/deserialized, and why are field IDs so important?
- Write: writeStructBegin → for each field: writeFieldBegin(name, type, id) → write value → writeFieldEnd → writeFieldStop → writeStructEnd.- Read: readStructBegin, then loop reading fields until a STOP marker; switch on field ID to read the correct type; skip unknown fields.
- Field IDs (16‑bit) identify fields on the wire; names are not serialized by Binary protocol. Stable, unique IDs enable order‑independent reads and schema evolution.
How does Thrift support interface evolution (adding/removing/reordering fields)?
- Readers loop through fields and can ignore those they don’t recognize (via skip).- Field order doesn’t matter; matching field IDs and types does.
- You can add new optional fields without breaking older readers; they’ll skip unknown fields. Keep field IDs stable across versions. All parties must use the same protocol type to interoperate.
How are RPC messages encoded and what message types exist?
- Messages wrap the structs used for calls and results. The compiler generates args (call parameters) and result (return values) structs.- writeMessageBegin(name, type, seqid) … writeMessageEnd().
- Message types: T_CALL (client → server), T_REPLY (server → client), T_EXCEPTION (error reply), T_ONEWAY (no reply).
- Sequence IDs help correlate requests/replies (servers echo the client’s seqid).
What should I watch out for with duck typing in dynamic languages like Python?
- Duck typing lets you pass a list where a set is expected because both are iterable, so serialization succeeds.- On read, the server constructs a set and duplicates are dropped silently. You pay the cost of transmitting duplicates with unexpected logical results.
- Best practice: Match IDL container types at call sites in dynamic languages to avoid surprises and wasted bandwidth.
Which protocol should I choose, and how do buffering and performance interact?
- Choose based on needs:- Binary: Broadest support; good all‑around performance.
- Compact: Smaller payloads → better throughput for I/O‑bound workloads; slight extra CPU.
- JSON: Human‑readable and interoperable; largest/slowest.
- Buffering (e.g., TBufferedTransport) batches small writes; flush after each logical message. Greatly improves file/socket performance by reducing syscalls.
- Benchmarks in the chapter show: Compact usually faster than Binary for disk I/O (fewer bytes), while Binary can be marginally faster in memory. JSON is slowest/largest. Always test with your data and workloads.
Programmer’s Guide to Apache Thrift ebook for free
