7 Generating Python Code

This chapter introduces practical ways to build and run domain-specific small language models for Python code generation. It contrasts closed, commercial assistants with open-source options and shows why specialized, compact models can rival much larger generalist systems on coding tasks. Benchmarks such as SWE-bench highlight that even strong models still require expert oversight, and the text argues that programmers remain central: coding assistants amplify skilled engineers rather than replace them. Specialization trims irrelevant knowledge, reduces cost and risk, and makes on-device deployment feasible.

A hands-on workflow starts with Salesforce’s CodeGen (350M mono) as a baseline, demonstrating prompts from function headers to natural language and comments. The chapter then optimizes inference by exporting to ONNX with Optimum and applying dynamic quantization, yielding sizable latency reductions, higher throughput, and more consistent runtimes; when a model isn’t supported, it shows a mid-level path using the Transformers ONNX exporter and ONNX Runtime quantization. For quality, it adopts a robust evaluation pipeline inspired by ReCode on HumanEval: tokenize and generate, validate syntax via Python’s AST, execute provided unit tests for correctness, and aggregate pass rates—while emphasizing safe execution practices.

Next, the chapter explores stronger and still-efficient models: CodeGen 2.5 (7B) with infill and FlashAttention for faster sampling, and BigCode’s StarCoder2 (3B) with Grouped Query Attention and long context windows. It demonstrates lightweight deployment through BitsAndBytes (8/4-bit) and GGUF formats, and shows how to run a 4-bit StarCoder2 locally on a MacBook Air (M1) using llama.cpp/llama-cpp-python, achieving usable, sub–half-second CPU-only latencies. It also covers serving via an OpenAI-compatible endpoint for IDE integration and converting HF checkpoints to GGUF when needed. The overarching message: with careful optimization, quantization, and disciplined evaluation, capable Python coding assistants can run efficiently on commodity hardware.

The SWE-bench leaderboard.

Comparison of the most popular code assistant LLMs’ size when CodeGen was released.

Average inference time on CPU for the vanilla CodeGen model and its ONNX converted version.

Comparison about latency and throughput metrics related to the CodeGen 350M mono vanilla model and its ONNX converted version benchmarks

Comparison between duration times for 100 inference runs with the CodeGen 350M mono vanilla model and its ONNX converted version

Average inference time on CPU for the ONNX CodeGen model and its quantized version.

Comparison about latency and throughput metrics related to the CodeGen 350M mono ONNX model and its quantized version benchmarks.

• Specialized open source language models for code generation require smaller parameter counts than generalist models while achieving comparable performance.
• CodeGen models support multiple programming languages or Python-only code generation through the mono family.
• ONNX conversion of code generation models can reduce average inference time by approximately 19% compared to vanilla PyTorch implementations.
• 8-bit quantization of ONNX models can provide additional 60% reduction in inference time while maintaining code quality.
• Models not supported by Optimum require mid-level conversion using the transformers.onnx package with manual configuration.
• Abstract Syntax Tree parsing validates generated Python code for syntactic correctness before execution.
• HumanEval dataset provides 164 Python programming problems for evaluating code generation model performance.
• Code generation models support various prompt formats including function signatures, natural language comments, and multiline docstrings.
• StarCoder2 uses Grouped Query Attention to achieve faster inference times with reduced computational overhead.
• llama-cpp-python provides Python bindings for running GGUF models locally without GPU requirements.
• Code infilling requires specific token formatting with mask and separator tokens for proper context understanding.
• 4-bit quantization reduces model size by approximately 75% while maintaining acceptable code generation quality.
• Local deployment of code generation models enables offline coding assistance without external API dependencies.
• Metal Performance Shaders backend enables optimized inference on Apple Silicon processors.
• Code generation evaluation requires both syntactic validation and functional correctness testing against unit tests.
• ReCode evaluation framework provides robustness metrics for code generation under different prompt perturbations.

FAQ