6 Beyond Natural Language Processing

Transformers are general-purpose sequence learners whose core remains the same across domains; what changes are the input and output interfaces. By rethinking tokenization or its equivalents, large language models can operate beyond natural language on code, mathematics, and images, often with the help of external tools that supply domain-specific capabilities. The unifying idea is to convert diverse data into sequences of vectors, train with gradient descent as usual, and tailor pre/post-processing so the model can reason and generate effectively in each modality.

For software, standard text tokenization largely suffices because code is structured text enriched by descriptive identifiers and comments. Performance is boosted with targeted data (SFT from code corpora), preference learning (RLHF from forums and contests), and tight integration of tools: compilers for executability checks, parsers for token-level syntax validation during generation, and linters or canonicalization to normalize formatting and reduce spurious token variation. Mathematics, by contrast, hinges on tokenization: preserve symbolic characters, prefer compact formats like TeX, and mitigate the many-to-one mapping of equivalent expressions. Numbers demand consistent schemes (e.g., digit-level tokenization) or specialized methods such as xVal that represent any numeric value with a scaled embedding. External tools further raise reliability—computer algebra systems to carry out exact steps and proof assistants like Lean to verify correctness—augmented by multiple sampled runs when formal verification is impractical.

In computer vision, the tokenizer/unembedding pair is replaced by a patch extractor and patch combiner: images are split into fixed-size patches, mapped to vectors via a small network (CNN or linear), processed by unchanged transformer layers, then reassembled into images. This same flexibility enables multimodal systems—image captioning (image in, text out) and text-to-image generation (text in, image out)—that mix modalities while preserving the transformer core. Prior lessons still apply: models reflect their training data and objectives, so requests outside that distribution (such as negations in prompts) can misfire, and careful prompting influences style and quality. Overall, effective cross-domain use comes from stable transformer internals plus modality-aware interfaces, consistent representations, rich domain data, and judicious tool use.

If we break an LLM into three primary components: input (tokenization), transformation (transformers), and output generation (unembedding), we can use new data modalities by changing at least one of the input or output components. Meanwhile, the transformer does not require modification for most use cases because it is general-purpose.

Two similar samples of code written in the programming languages Python (left) and Java (right). These show how byte pair encoding can identify similar tokens across different languages. The boxes show individual tokens. Standard tokenization methods for human languages do a reasonable job on code since it has many similarities to natural language.

Code written in Java, including a comment describing what the code does. Because (good) code (hopefully) has a lot of comments, there is a natural mix of natural language and code for the LLM to use to obtain information. When variables have descriptive names, it becomes easier for the model to correlate information between the code and the intent described in comments and variable names.

Developing an LLM for code applies multiple rounds of fine-tuning. Standard training procedures, such as those described in 4, produce an initial base LLM. Using a large amount of code, SFT creates an LLM that works well with code. Including RLHF as a second fine-tuning step improves the LLM’s ability to produce code.

A Python code example where the current tokens if(A > B has been generated. If the next token produced by the LLM is a newline, a syntax error will occur because a if statement must end in a colon to be valid. Running a syntax checker on each new token allows us to catch this error and force the LLM to pick an alternative token that doesn’t cause a syntax error.

A Java code example on how multiple ways to format the same code will lead to different tokenizations, even though each is semantically identical. Linters are a common tool to force code to follow a specific formatting rule. Instead, a linter can be used to create an identical “base” form, thus avoiding representing unnecessary information (like spaces vs tabs).

A symbolic math problem that the Minerva LLM can solve correctly. While this example mixes natural language with mathematical content, the standard tokenization used by many LLMs would not allow this kind of mathematical output and can cause some surprising issues. Image Creative Commons licensed from https://blog.research.google/2022/06/minerva-solving-quantitative-reasoning.html.

A mathematical equation in the top-left demonstrates two different representation problems that occur with math. The nicely formatted math requires a typesetting language. TeX and MathML are two different typesetting languages that have vastly different text and, thus, tokenization. Separate from the typesetting language, there are many ways to represent the same mathematical statement.

LLMs can not learn to do basic arithmetic unless they tokenize digits consistently. In this figure, underlines denote different tokens. The tokenized digits might represent the tens, hundreds, or thousands place for any given number. GPT-3 (left) is inconsistent in how numbers get tokenized, making adding two numbers needlessly complex. GPT-4 is better (but not perfect) at tokenizing numbers in a consistent way.

xVal uses a trick to help reduce the number of tokens and make them less ambiguous. By modifying how the LLM converts numbers to vectors, a single vector represents each number, such as the number 1. By always using the 1-token and multiplying it by the number observed, we avoid many edge cases in number token representation, such as numbers that never appeared in the training data. This conversion method also makes fractional numbers like 3.14 easier to support.

Given some mathematical goal, getting an LLM to use Lean (right path) might not result in a verifiably correct proof because it may not be effective at using Lean as a tool. Having the LLM produce a normal proof (left path) may yield a correct proof, but not a way for us to verify it is (in)correct.

On the left, a simplified diagram shows how text input is tokenized and embedded before going to the transformer. An unembedding layer then converts the transformer output into the desired text representation. The input and output will be images when performing a computer vision task. The transformer stays the same, but then we modify the method for breaking the image into a sequence of vectors to perform “patch extraction” instead of tokenization. The LLM produces image output using a “patch combiner” analogous to the unembedding later for text LLMs.

Extracting patches is a straightforward process. The patch extractor breaks up an image into square tiles called patches. Images consist of pixel values that are already numbers, so we convert each patch into a vector of numbers. Then, we use a small neural network as a pre-processor before passing the vectors to the full transformer-based neural network.

Note that compared to 6.13 the arrows go in the opposite direction. The purpose is to emphasize that the patch combiner and extractor do the same thing but operate in different directions. The neural network is more important in this stage as a way to force the transformer’s output to have the same shape as the original patches because we can control the output size of any neural network.

Four combinations showing different types of model inputs and output. The example of the furthest right represents a normal text-based LLM we have already learned about. To the left, we show possibilities like an image-generating model that takes text as input (“draw me a picture of a stop sign in a flood zone”) or an image captioning model that generates text that describes an image input (“this picture shows an s stop sign surrounded by murky water”).

This was generated with an old version of Stable Diffusion, a popular image generation model. Despite asking the model to “please do not draw a cat”, the model was trained to generate content. The request is outside what the model was incentivized to learn, so it can not handle it. This is similar to the issues with LLMs regurgitating close-but-wrong output because the model saw similar data during training.

• LLMs benefit when they can use external tools. For example, a code LLM can use syntax checkers and compilers to detect erroneous code generation. When the LLM finds an error, the output is re-generated, minimizing the risk of giving the user unhelpful or broken code.
• Tokenizers must be modified to support math by keeping unusual symbols used for expressing formatted math and changing digit representations. We can improve Math LLMs further by giving them tools like Computer Algebra Systems to detect and avoid errors.
• Transformers can be applied to images by breaking an image up into patches, where each patch becomes a vector and makes a sequence of inputs for the transformer to process. Patches are conceptually similar to tokens for text LLMs.
• Transformers can use different data modalities for input and output, allowing the creation of multi-modal models like those used in image captioning and image generation.

FAQ