1 Why rearchitecting LLMs matters

Large language models are powerful because they can draw on broad knowledge across many domains, but that same generality makes them a poor fit for many real-world applications. In practice, organizations often try to force one generic model to handle very different tasks with prompt engineering, retrieval, or fine-tuning, yet these approaches can be costly, hard to control, and only weakly differentiated from what competitors can do with the same foundation models. The chapter argues that this is especially problematic in settings like finance and healthcare, where cost, efficiency, control, and explainability all matter.

The proposed answer is model rearchitecting: a structural approach that changes the model itself rather than only its instructions or weights. Instead of relying solely on fine-tuning, the pipeline combines pruning to remove less useful components, knowledge distillation to recover lost capability from a larger teacher model, and LoRA-based specialization to adapt the resulting model to a specific domain or task. The key idea is that domain data should guide the whole process, from calibration and pruning decisions to recovery and final specialization, producing models that are smaller, faster, and better aligned with the intended use case.

The chapter closes by positioning this approach as both practical and educational. It explains the software and hardware environment used in the book, centered on PyTorch, Hugging Face, and accessible GPU setups, and introduces the book’s progression from fundamentals to implementation to research deep dives. The broader roadmap aims to turn readers from model users into model architects, capable not only of building efficient specialized models but also of understanding internal activations, improving explainability, and applying these ideas to both small and large models in future work.

The model tailoring pipeline consists of core phases (shown with solid arrows) and optional phases (shown with dashed arrows). In the first phase, we adapt the structure to the model's objectives through pruning. Next, we recover capabilities it may have lost through knowledge distillation. Finally, we can optionally specialize the model through fine-tuning. An optional initial phase calibrates the base model, via a brief fine-tuning pass on the target dataset.

Dataset integration in the rearchitecting pipeline. The domain-specific dataset guides calibration of the base model, informs structural optimization decisions, and enables final specialization through LoRA fine-tuning. A general dataset supports Knowledge Recovery, ensuring the pruned model retains broad capabilities before domain-specific specialization. This dual approach optimizes each phase for the project’s objectives.

• The use of oversized, generic LLMs can lead to high production costs, little differentiation from competitors, and no explainability of decisions.
• Models become more effective and efficient by adapting their architecture to a specific domain and task.
• The model-architecting process consists of three phases: structure optimization, knowledge recovery, and specialization.
• The domain-specific dataset is a key element and common thread throughout the process, ensuring each optimization and specialization phase aligns with the final objective.
• Knowledge distillation transfers capabilities from the original teacher model to the pruned student model, enabling the student to learn not only the correct answers but also the reasoning process that leads to them.
• Fine-tuning techniques such as LoRA allow domain specialization by training only a small number of parameters, drastically reducing cost and time.
• Modern architectures like LLaMA, Mistral, Gemma, and Qwen share structural traits that make them well suited to rearchitecting techniques.
• By mastering these techniques, developers can go from being model users to model architects.

FAQ