Overview
4 Moving bytes with transports
Transports are the byte-moving foundation of Apache Thrift, providing a uniform interface to read and write data regardless of the underlying end point. They sit at the bottom of Thrift’s layered stack beneath protocols and services, allowing applications to swap where bytes go—memory, disk, or network—without changing higher-level logic. The chapter uses small, transport-only programs to illustrate how this decoupling works and to show why transports are central to both serialization and RPC, while reminding that cross-language interoperability ultimately depends on pairing transports with Thrift protocols.
The core, shared abstraction is TTransport, whose essential operations—open, read, write, flush, and close—behave consistently in concept across languages, even as specifics differ (buffering, blocking, exceptions). The chapter surveys end point transports: memory buffers for fast, in-process I/O; files for logging and storage; and sockets for networking. C++, Java, and Python examples highlight language idioms and practical differences such as object size and layout (padding, endianness), mandatory flushing in some implementations, and read semantics that may block, return 0, or throw exceptions. The takeaway is to code to the TTransport interface and let protocols normalize serialized data across languages.
On the server side, TServerTransport (commonly TServerSocket) listens, accepts connections, and manufactures client-specific end point transports, with options for timeouts and interrupting blocking accepts. The chapter then introduces layered transports, which wrap other transports to add features like buffering (TBufferedTransport) and message framing (TFramedTransport). Framing prefixes each message with its size, enabling efficient, deterministic reads and powering nonblocking servers. By stacking layers (and even creating custom ones, like a tee for simultaneous logging and RPC), Thrift applications can tailor performance and behavior while keeping application code portable and end point–agnostic.
Summary
Practical Apache Thrift applications rarely restrict their use of the framework to the transport layer. However, transports underlie every Apache Thrift application and are a foundational part of Apache Thrift programming. In this chapter, you wrote sample programs exploring all of the key aspects of the Apache Thrift transport layer.
Key points in this chapter:
- Transports are the lowest layer of the Apache Thrift software stack.
- All Apache Thrift features depend on transports.
- Transports implement the TTransport interface.
- The TTransport interface defines device independent byte-level read and write operations.
- End point transports implement TTransport and perform read/write operations against a device.
- End point transports are offered in most languages for memory, disk, and network devices.
- Server Transports use the factory pattern to manufacture network transports as client connections are accepted.
- Layered transports implement TTransport and provide additional features on top of an underlying transport.
- Layered transports enable separation of concerns and reuse in the transport stack.
- The TFramedTransport is a commonly used layered transport and is required to connect to non-blocking servers.
- A transport stack may include any number of layered transports but must always lead to end point transports at the bottom of the stack.
FAQ
What is a transport in Apache Thrift and why is it important?
Transports are the lowest layer of the Thrift stack that move raw bytes to and from an end point (memory, file, socket, etc.). They provide a uniform TTransport interface so higher layers (protocols and RPC) can read/write without caring about the physical device, enabling portability and easy swapping of end points.What are “end point” transports and which types are commonly available?
End point transports directly read/write a physical or logical device. Common categories:
- Memory transports (for buffering/caching), e.g., TMemoryBuffer
- File transports (for logging/storage), e.g., TSimpleFileTransport, TFileObjectTransport
- Network transports (for RPC), e.g., TSocket
Availability and names vary by language, but they all implement the same TTransport interface.What does the TTransport interface look like and which methods matter most?
The TTransport abstraction typically provides: open, close, isOpen, read, readAll, write, flush, peek, readEnd, writeEnd. In practice, most application code relies on:
- open(): enable I/O
- write(): enqueue bytes
- read(): get bytes (may be partial)
- flush(): push buffered bytes to the end point
- close(): cleanly release the end pointDo I need to call open(), flush(), and close()? What are the semantics?
- open(): Call before I/O. Some transports open in the constructor, but calling open() is safe and portable.
- flush(): Required whenever writes are buffered (e.g., Java TSocket, TBufferedTransport, TFramedTransport). It ensures data actually leaves process buffers.
- close(): Explicitly close when done. Garbage-collected languages may delay resource release if you rely on finalizers.How do memory, file, and network transports differ in behavior and use cases?
- Memory (TMemoryBuffer): Fast, in-process buffering; great for staging, testing, or caching. Some implementations auto-grow; some expose underlying buffer seek behavior (e.g., Python StringIO/BytesIO requires seek before reading what you wrote).
- File (TSimpleFileTransport/TFileObjectTransport): Durable logging, archival, and debug capture; some are read-only or write-only; simple variants may not support seeking.
- Network (TSocket): RPC and remote I/O; usually requires open() to connect, read() can block or throw on close; platform-dependent buffering and timeouts.Why can’t I exchange objects across languages with transports alone?
Raw bytes written by one language (e.g., a C++ struct) won’t match another language’s expectations due to padding, endianness, and language-specific serialization formats (e.g., Java Serializable, Python pickle). Cross-language interoperability requires a Thrift protocol (e.g., TBinaryProtocol) layered over transports to normalize the on-the-wire representation.What practical differences should I expect across C++, Java, and Python transport usage?
- C++ often uses raw pointers/byte buffers; struct size can change due to padding/alignment; network writes may go out immediately.
- Java uses byte arrays and streams; socket I/O is buffered (flush is mandatory before reads); exceptions are checked or unchecked.
- Python uses dynamic bytes objects and file-like buffers; TSocket.read() may throw on close while file transport may return empty bytes; in Python 2 vs 3, buffers are StringIO vs BytesIO, abstracted by Thrift.What are server transports and how do they relate to client transports?
Server transports (e.g., TServerTransport, typically TServerSocket) don’t do application I/O themselves; they listen for inbound connections and manufacture per-client end point transports (usually TSockets) via:
- listen(): start accepting
- accept(): block until a client connects; returns a TTransport bound to the client
- interrupt() [optional]: break out of a blocking accept
- close(): stop listening
Timeouts are language-specific (e.g., setAcceptTimeout in C++, setSoTimeout in Java, settimeout on Python socket).What are layered transports and when should I use TBufferedTransport or TFramedTransport?
Layered transports wrap another TTransport and add features without reimplementing the end point:
- TBufferedTransport: accumulates small writes and pushes them on flush() to reduce system calls.
- TFramedTransport: frames messages with a 4-byte length prefix; required by Thrift’s nonblocking servers and useful when receivers must know message boundaries up front. Layers can be stacked (e.g., TFramed over TBuffered over TSocket).What is message framing and what should I know about endianness and tooling?
Framing prefixes each message with a 4-byte length (big-endian). Receivers read the size, then read exactly that many bytes. Notes:
- Big-endian size header (use htonl/ntohl in C/C++ when crafting manually).
- Flush() typically defines a frame boundary in TFramedTransport.
- Tools like telnet won’t work with framed servers because they don’t send the size prefix; use a Thrift client or construct frames yourself.
Programmer’s Guide to Apache Thrift ebook for free
