Appendix C. Qwen3 LLM source code

This appendix explains that the book’s “from scratch” promise applies to the reasoning methods, not to building the language model itself, and it therefore presents the Qwen3 implementation mainly as a reference. The model code is organized as a Python package import and is described only at a high level, with the goal of showing how the Qwen3 architecture is assembled rather than teaching LLM implementation in full detail. Readers unfamiliar with modern LLM design are encouraged to start with GPT-2 first, since Qwen3 is structurally similar but includes newer architectural refinements.

The model design centers on several modern replacements for older GPT-2 components. Qwen3 uses RMSNorm instead of LayerNorm, which still stabilizes activations but is cheaper because it skips mean-centering and reduces reduction operations. Its feed-forward network is a gated linear unit variant, often called SwiGLU, which uses three linear layers and a SiLU activation to create multiplicative interactions that improve expressivity while remaining parameter-efficient. The appendix also introduces RoPE for positional information, showing how position is encoded by rotating query and key vectors rather than adding separate position embeddings.

Attention and generation support are built around grouped query attention, which shares key and value projections across multiple query heads to reduce memory use and improve inference efficiency. The attention code optionally applies QKNorm for extra stability, and the transformer block combines RMSNorm, RoPE, masked GQA, and the feed-forward module with residual connections. The full Qwen3Model then wraps these blocks with token embeddings, a final normalization layer, an output head, RoPE buffers, and KV-cache-aware masking for autoregressive generation. A small KVCache class stores per-layer key-value pairs during decoding, and the tokenizer reimplementation handles the model’s special tokens and chat formatting, including the reasoning-oriented “thinking” variant behavior.

Figure C.1 Architectural comparison between Qwen3 and GPT-2. Both models process text through embedding layers and stacked transformer blocks, but they differ in certain design choices.

Figure C.2 Comparison of LayerNorm (used in GPT-2) and RMSNorm (used in Qwen3). LayerNorm (left) normalizes activations so that their average value (mean) is exactly zero and their spread (variance) is exactly one. RMSNorm (right) instead scales activations based on their root mean square, which does not enforce zero mean or unit variance, but still keeps the mean and variance within a reasonable range for stable training.

Figure C.3 In GPT-2 (top), the feed forward module consists of two fully connected (linear) layers separated by a non-linear activation function. In Qwen3 (bottom), this module is a gated linear unit (GLU) variant, which adds a third linear layer (linear layer 3) and multiplies the output of this linear layer 3 elementwise with the activated output of linear layer 1.

Figure C.4 Different activation functions that can be used in a feed forward module (neural network). GELU and SiLU (Swish) offer smooth alternatives to ReLU, which has a sharp kink at input zero.

Figure C.5 A comparison between MHA and GQA. Here, the group size is 2, where a key and value pair is shared among 2 queries.

Figure C.6 The Structure of the transformer block in Qwen3. Each block includes RMSNorm, RoPE, masked grouped-query attention, and a feed-forward module, and is repeated 28 times in the 0.6B-parameter model.

Figure C.7 Architecture of the Qwen3 0.6B model. The model consists of a token embedding layer followed by 28 transformer blocks, each containing RMSNorm, RoPE, QKNorm, masked grouped-query attention with 16 heads, and a feed-forward module with an intermediate size of 3,072.

FAQ