11 Deeper Customization Using LoRA

This chapter introduces LoRA as a practical way to make diffusion models more customizable, especially when full checkpoint fine-tuning is too expensive. It explains that training diffusion models needs far more memory than inference, with Flux being especially demanding, which makes traditional fine-tuning impractical for most users. LoRAs address this by offering a much smaller, more efficient method for changing model behavior without retraining the entire system.

The core idea is that LoRAs modify only key parts of the model, especially the attention layers, yet can still produce dramatic changes in style and output. The chapter shows how to load a LoRA into a ComfyUI workflow, adjust its strength, and use it to transform a plain Flux generation into something with a much more distinct look. It also emphasizes that LoRAs are especially useful for style adaptation, giving Flux the ability to produce results that feel more realistic, more artistic, or more visually specific.

Beyond style transfer, the chapter explores trigger words, concept LoRAs, and combining multiple LoRAs together. Trigger words help activate the desired behavior of a LoRA, while concept LoRAs can represent a character, person, or object consistently across different scenes, making them useful for personalized images and longer creative projects. By chaining multiple LoRAs, users can blend effects, though order and strength can matter, showing that LoRAs offer a flexible and powerful path toward deeper image generation control.

Attention layers are an essential part of how large models summarize important information

A LoRA learns only to modify the attention layers, which still has a big impact!

Generating a scene from Before the Law using Flux

The Load LoRA node in the ComfyUI workflow.

Visualizing how to connect the LoRA to the rest of the workflow.

Using the Amateur Photography LoRA dramatically changes the style of our image.

changing the model strength determines how much the LoRA style impacts the result.

The Tarot card LoRA completely changes the style of our image.

Using the correct “trigger words” can dramatically change the outcome of the generation.

The author of this book as generated using a concept LoRA

Keats the dog promises not to sue over trademark or brand infringement

Keats is just curious if there might be some treats down that path

LoRAs can be combined simply by chaining them.

A Tarot card depicting Keats the dog in a Kafka story.

Slightly different variant of tarot card made by swapping the LoRA order.

• Fine-tuning checkpoints requires significantly more VRAM than generating images.
• This means it is very difficult to fine-tune models like Flux.
• LoRAs work by modifying only the attention layers of models used for diffusion.
• Because of how they are designed, LoRAs are small and can be trained on consumer hardware.
• When using a LoRA in ComfyUI, it is placed immediately after the u-net and clip encoder are loaded
• LoRA can be used to easily and dramatically change the style output from Flux (and other diffusion models).
• Many LoRA rely on “trigger words” to get the best performance, make sure to include these in your prompt.
• Concept LoRA allows you to use a character or subject in your image generation.
• Multiple LoRAs can be combined together, but the order will impact the results.

FAQ