7 Generating Python Code
The text presents a practical exploration of open source small language models specialized for Python code generation, emphasizing their role as coding assistants rather than replacements for software engineers. It argues that while large general-purpose and proprietary coding models can help with programming tasks, they still require expert supervision, struggle with complex real-world software engineering workflows, and raise concerns around cost, privacy, transparency, intellectual property, and safety. Smaller domain-specific coding models are positioned as a strong alternative because they focus on relevant programming knowledge, can be cheaper and easier to deploy, and may perform competitively on coding benchmarks despite having far fewer parameters.
The main hands-on example uses Salesforce CodeGen models, especially the 350M Python-only variant, to demonstrate Python code generation, optimization, ONNX conversion, and quantization. The text shows that CodeGen can complete function bodies, respond to natural-language or comment-based prompts, generate examples using common Python libraries, simulate code execution, and produce unit tests, with larger versions supporting richer prompting. It then explains how to export the model to ONNX using Hugging Face Optimum, apply dynamic 8-bit quantization, and benchmark the resulting versions. The reported results show meaningful reductions in model size, lower latency, higher throughput, and more consistent inference times, while also discussing alternative ONNX export and quantization workflows for architectures not yet supported by Optimum.
The text also covers evaluation and more modern coding models, including CodeGen 2.5 and StarCoder2. For evaluation, it describes using HumanEval-style programming problems, validating generated Python syntax with abstract syntax trees, and running unit tests to assess correctness, while warning about the risks of executing generated code. StarCoder2 is highlighted as a stronger small coding model with efficient attention mechanisms and support for 8-bit or 4-bit quantization through BitsAndBytes. Finally, the text explains how to run a 4-bit GGUF-quantized StarCoder2 model locally on modest hardware using llama.cpp through Python bindings, enabling offline code generation and potential IDE integration through an OpenAI-compatible local server.
The SWE-bench leaderboard.
Comparison of the most popular code assistant LLMs’ size when CodeGen was released.
Average inference time on CPU for the vanilla CodeGen model and its ONNX converted version.
Comparison about latency and throughput metrics related to the CodeGen 350M mono vanilla model and its ONNX converted version benchmarks
Comparison between duration times for 100 inference runs with the CodeGen 350M mono vanilla model and its ONNX converted version
Average inference time on CPU for the ONNX CodeGen model and its quantized version.
Comparison about latency and throughput metrics related to the CodeGen 350M mono ONNX model and its quantized version benchmarks.
Summary
- Specialized open source language models for code generation require smaller parameter counts than generalist models while achieving comparable performance.
- CodeGen models support multiple programming languages or Python-only code generation through the mono family.
- ONNX conversion of code generation models can reduce average inference time by approximately 19% compared to vanilla PyTorch implementations.
- 8-bit quantization of ONNX models can provide additional 60% reduction in inference time while maintaining code quality.
- Models not supported by Optimum require mid-level conversion using the transformers.onnx package with manual configuration.
- Abstract Syntax Tree parsing validates generated Python code for syntactic correctness before execution.
- HumanEval dataset provides 164 Python programming problems for evaluating code generation model performance.
- Code generation models support various prompt formats including function signatures, natural language comments, and multiline docstrings.
- StarCoder2 uses Grouped Query Attention to achieve faster inference times with reduced computational overhead.
- llama-cpp-python provides Python bindings for running GGUF models locally without GPU requirements.
- Code infilling requires specific token formatting with mask and separator tokens for proper context understanding.
- 4-bit quantization reduces model size by approximately 75% while maintaining acceptable code generation quality.
- Local deployment of code generation models enables offline coding assistance without external API dependencies.
- Metal Performance Shaders backend enables optimized inference on Apple Silicon processors.
- Code generation evaluation requires both syntactic validation and functional correctness testing against unit tests.
- ReCode evaluation framework provides robustness metrics for code generation under different prompt perturbations.
FAQ
What is the main focus of chapter 7, “Generating Python Code”?
The chapter focuses on using domain-specific open source small language models for Python code generation and programming assistance. It explains how specialized coding LLMs differ from general-purpose LLMs, how to run Python code generation with models such as CodeGen and StarCoder2, and how to optimize, convert, and quantize these models for faster inference on CPUs, GPUs, and even laptops.Why does the chapter emphasize specialized open source LLMs for code generation instead of large generalist models?
Specialized coding LLMs can be smaller, cheaper to train, easier to deploy, and more focused on programming tasks than large generalist models. Because they are trained on a narrower and more relevant data pool, they can sometimes match or outperform larger models on coding benchmarks. The chapter also notes that closed-source coding assistants may raise concerns around cost, privacy, intellectual property, data leakage, safety, and lack of transparency.Will generative AI replace human programmers according to the chapter?
The chapter argues that generative AI is unlikely to replace software engineers entirely. Coding is only one part of software development; engineers also need to understand user requirements, design systems, translate ideas into buildable products, reason across files and components, and make judgment calls. The chapter’s position is that a good software engineer can become more effective with a coding assistant, but a person without software engineering knowledge will not become an expert simply by using one.What is Salesforce CodeGen, and which CodeGen family is used for Python code generation?
Salesforce CodeGen is an open source Transformer-based model family for conversational program synthesis and code generation. It includes two main families: multi, trained on multiple programming languages such as C, C++, Go, Java, JavaScript, and Python, and mono, trained specifically for Python. Since the chapter focuses on Python, it uses the CodeGen mono models, especiallySalesforce/codegen-350M-mono as a baseline example.How can CodeGen be used to generate Python code from a prompt?
CodeGen can generate Python code by tokenizing a prompt and calling the model’sgenerate method. For example, a prompt such as def hello_world(): can produce a function body like print("Hello World"). Larger CodeGen models can also respond to natural language prompts, single-line comments, multiline comments, function signatures with docstrings, code verification prompts, and prompts requesting unit tests.How does the chapter convert and quantize CodeGen with Hugging Face Optimum and ONNX Runtime?
The chapter usesORTModelForCausalLM from Hugging Face Optimum to export CodeGen to ONNX, because CodeGen is a causal language model. After saving the ONNX model, it applies dynamic 8-bit quantization with ORTQuantizer and AutoQuantizationConfig. In the example, the ONNX model size is about 1.33 GB, while the 8-bit quantized model is reduced to about 346 MB.What performance improvements are observed after converting and quantizing CodeGen?
On a CPU-only Colab environment, converting the CodeGen 350M mono model to ONNX reduces average inference time from about 1186 ms to about 958 ms. Quantizing the ONNX model provides further improvements, with around a 60% reduction in average inference time compared with the ONNX version. The chapter also reports lower latency, higher throughput, and more consistent run durations after optimization.What can be done if a model architecture is not supported by Hugging Face Optimum?
If Optimum does not yet support a model architecture, the chapter suggests a mid-level approach. First, export the model to ONNX using thetransformers.onnx script with the proper feature, such as causal-lm for CodeGen. Then apply 8-bit dynamic quantization directly through the ONNX Runtime quantization API using quantize_dynamic and QuantType.QInt8.How does the chapter evaluate the quality of generated Python code?
The chapter discusses evaluation using OpenAI’s HumanEval dataset and ideas derived from Amazon Science’s ReCode work. HumanEval contains 164 Python programming problems with prompts, canonical solutions, tests, and entry points. The generated code is first checked for valid Python syntax using Python’s built-inast module. Then correctness is assessed by executing the generated solution against the HumanEval unit tests. The same evaluation process can be applied to the vanilla, ONNX, and quantized versions of a model.How can StarCoder2 3B be run locally on a laptop after 4-bit quantization?
The chapter uses a GGUF 4-bit quantized version of StarCoder2 3B withllama.cpp through the llama-cpp-python bindings. On a MacBook Air with an Apple M1 chip and 8 GB of unified memory, the model can be downloaded from Hugging Face, loaded with the Llama class, and run locally with CPU-only inference. The example uses the starcoder2-3b-Q4_K_M.gguf file, sets parameters such as context size, CPU threads, and generation options, and achieves acceptable latency and throughput for local coding assistance.
Domain-Specific Small Language Models ebook for free
