This chapter introduces buffers and streams as essential tools for handling modern JavaScript workloads that involve large files, binary formats, and data that arrives over time. Instead of converting everything into high-level JavaScript values right away, buffers let programs work with raw bytes efficiently, which is crucial for tasks like checking HTTP body sizes, decoding text safely, or assembling binary structures. Streams build on that idea by letting data be processed in chunks, reducing memory use and allowing output or input to begin before an entire payload has finished transferring.
The discussion then explores how buffers relate to typed arrays and DataView. Typed arrays provide a low-level, fixed-type view into memory, making it easy to read and write byte-oriented data, but they can be awkward for mixed binary formats because of issues like endianness and the need to combine different data representations. DataView solves that by offering precise control over reading and writing multi-byte values at specific offsets, making it especially useful for constructing and interpreting file headers and other structured binary data in a platform-independent way.
The chapter then shifts to streams as a way to process data in motion. It explains how readable, writable, and transform streams can be connected with piping so that bytes or text flow from source to sink, optionally passing through transformations like decoding or custom splitting logic. Finally, it shows how consumers can influence stream behavior with pull-based streams and backpressure, using high watermarks to trade memory for throughput. The overall message is that buffers handle efficient memory-level data access, while streams make JavaScript responsive and scalable when data is large, continuous, or expensive to process all at once.
A single buffer with multiple content encodings
The first 16 bytes of the PNG image header constructed in Listing 11.3.
For the stream in Listing 11.5, values are read from the stream immediately after they’re enqueued.
The sentences in Listing 11.7 are turned into words by the transform. The transform and the sink each read values from the previous stream as soon as they’re enqueued.
Summary
Buffers represent a block of memory. Typed arrays and DataView allow you to read from and write to that memory as an array of numeric types.
Encoders convert known types into buffers, and decoders convert buffers into known types.
A stream represents a flow of data. Streams can be read directly with a for await…of loop, or they can be piped into other streams.
Sources are streams that can only be read from, sinks are streams that can only be written to, and transforms are streams that do both.
Streams can be piped into other streams. The stream sending data is known as the producer, and the stream receiving data is known as the consumer.
As a general rule, streams enqueue data until the size of the queue reaches the number of chunks specified by the consumer’s high watermark. However, some streams may ignore the high watermark.
FAQ
What are buffers in JavaScript, and why are they useful?Buffers are raw blocks of memory that let JavaScript handle binary data directly instead of converting it into higher-level types like strings. They’re useful for efficient memory use, accurate byte-level processing, and working with large data such as files, uploads, and network requests.Why is checking HTTP request size with string.length a bad idea?Because Content-Length measures bytes, while string.length measures characters. Also, JavaScript strings are typically UTF-16, while HTTP data is usually UTF-8, so character count and byte count may not match.How do buffers help when reading an HTTP request body?They let you collect incoming request chunks as raw binary data, combine them with Buffer.concat(), and only decode to text after you’ve confirmed the content size and are ready to interpret it.What is the difference between Buffer and ArrayBuffer?Buffer is Node.js’s binary data type, while ArrayBuffer is the standardized ECMAScript buffer type available in all JavaScript environments. In Node.js, a Buffer can expose its underlying ArrayBuffer through its buffer property.What is a typed array, and how is it related to buffers?A typed array is a view over a buffer that treats the bytes as values of a specific numeric type, such as Uint8Array. Typed arrays let you read and write buffer contents as numbers rather than as an unstructured block of memory.How is DataView different from typed arrays?DataView is more flexible for mixed binary formats because it can read and write different numeric types at arbitrary offsets in a buffer. It also lets you control endianness, which makes it better for file formats and network protocols.Why does endianness matter when working with multi-byte values?Endianness determines the byte order used to store multi-byte numbers. Different systems or formats may use little-endian or big-endian, so relying on typed arrays alone can produce inconsistent byte sequences across platforms.What is a stream in JavaScript?A stream is a queue of data that arrives over time, often in chunks. Streams let you process data incrementally instead of waiting for everything to load at once.What is the difference between push streams and pull streams?Push streams enqueue data whenever they produce it, whether the consumer is ready or not. Pull streams wait until the consumer requests data before enqueuing more, which helps control flow and can reduce memory pressure.How do pipeThrough() and pipeTo() work in the Streams API?pipeThrough() sends a readable stream through a transform stream, and pipeTo() sends stream output into a writable stream. Together, they let you build real-time processing pipelines such as decoding bytes into text or transforming chunks before writing them out.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Secrets of the JavaScript Ninja, Third Edition ebook for free
