Overview
3 Bridging the semantic gap with learned metrics: BERTScore and COMET
Automatic evaluation for open-ended language tasks has long been constrained by lexical metrics that reward word overlap rather than meaning. The chapter shows why BLEU, ROUGE, and edit-distance methods break down when systems paraphrase, reorder information, or abstract intelligently: semantically equivalent outputs can be unfairly penalized, while fluent but wrong outputs can still look good on paper. This semantic gap motivates a shift from counting matching tokens to measuring whether texts actually convey the same idea.
BERTScore addresses this problem by using contextual embeddings from pretrained Transformer encoders to compare texts in semantic space. Instead of requiring exact n-gram matches, it computes token-level cosine similarities, matches each token to its best counterpart in the other text, and combines recall, precision, and F1-style scores. The method is strengthened by inverse document frequency weighting, which gives more importance to informative words, and by baseline rescaling, which makes scores easier to interpret. Its central advantage is that it can recognize valid paraphrases and reordered statements that lexical metrics miss.
COMET goes further by learning an evaluation function directly from human judgments. Rather than relying on a fixed similarity formula, it trains on annotated translation quality data such as direct assessments, post-editing effort, or detailed error labels, and it can use the source, candidate, and reference together to judge whether meaning was preserved. This makes it more aligned with human preferences than hand-designed metrics, but also more expensive, less interpretable, and dependent on training data quality. The broader lesson is that metric choice should match the task: lexical metrics for speed and strict form, BERTScore for semantic similarity, and COMET for the strongest learned proxy for human evaluation.
The families of automatic evaluation metrics for open-ended tasks. Lexical metrics count word overlaps. Semantic metrics measure meaning similarity using embeddings. LLM-as-a-judge uses LLM as humans to assess directly.
Both max pooling and mean pooling are methods to aggregate information across many vectors. Max pooling selects (pools) the maximum values from input embeddings; this weighs towards the most prominent feature. Mean pooling takes the mean (average) of input embeddings to represent features equally.
Encoders like BERT read bidirectionally. When processing “attorney”, they can see “filed” and “brief” to understand its legal context. Decoders like GPT read left-to-right for token generation, unable to use future context.
Lexical metrics count the number of words that are at the same coordinates in a certain order. For example, the candidate: “the weather is freezing” and the reference “it is cold today” have no lexical overlaps, so both the BLEU and ROUGE scores would be zero despite the two sentences being similar in meaning.
Semantic metrics measure the similarity between passages in embedding space. Even when sentences use completely different vocabulary, if they convey similar meaning, their embeddings will be close together. “The attorney filed the brief” and “the lawyer submitted the document” occupy nearby regions despite sharing no words.
Geometric representation of two semantic vectors. Note that while the vectors have different lengths (magnitudes), they point in roughly the same direction, indicating a shared semantic topic.
Computing BERTScore. Given the reference and candidate, we compute embeddings and pairwise cosine similarity. The heatmap highlights the greedy matching, where each reference token is matched to its best candidate token. Optional importance weighting emphasizes less common words over common ones. And we will scale the score to be more evenly distributed between 0 and 1.
Greedy matching for BERTScore recall. We look at the maximum similarity for each reference token in “the weather is code today” and average them. In this example, we have 0.713 as the max for “the,” 0.515 for “weather,” 0.858 for “is,” 0.796 for “cold,” and 0.913 for “today.” Thus, the unweighted BERTScore recall is 0.759.
Importance weighting with IDF. The word “the” has a max similarity of 0.713. It has an IDF of 1.27. Thus, its weighted max similarity is 0.906. Repeat this for all of the words in the reference, take the sum, and then divide by the sum of the IDFs to get an importance-weighted BERTScoreRecall of 0.753.
Designed metrics apply fixed formulas to compute scores. Learned metrics use machine learning models trained on human judgments to predict scores.
DA captures overall impression, HTER captures correction effort, and MQM captures specific errors.
Two-input metrics have a blind spot: they can't verify meaning preservation from the source. COMET's three-input design catches errors that reference-only comparison misses. This is especially important when the reference itself might be an imperfect translation.
Two flavors of COMET. Regression predicts exact scores; ranking learns relative quality by positioning better translations closer to anchors in embedding space.
Left. Estimator model architecture. The candidate, source, and reference are independently encoded using a pretrained cross-lingual encoder. The resulting embeddings are then passed through a pooling layer to create a sentence embedding for each segment. Finally, the resulting sentence embeddings are concatenated into a vector to be passed to a feed-forward regressor. Right. Ranking model architecture. It uses source, reference, a ‘better’ candidate, and a ‘worse’ candidate. They are encoded, pooled, and minimize triplet margin loss to minimize the distance between ‘better’ candidates and the ‘anchors’.
Summary
- Lexical metrics fail at semantic equivalence. BLEU and ROUGE cannot recognize that "consumers prefer imported cars" means the same as "people like foreign cars" because they count word matches rather than measuring meaning. This limitation becomes critical as AI systems increasingly paraphrase, abstract, and synthesize rather than copy.
- Contextual embeddings enable semantic comparison. Unlike static word vectors that give "bank" the same representation regardless of context, BERT-style encoders produce different embeddings based on surrounding words. This allows metrics to capture that meaning depending on context.
- BERTScore uses greedy matching for semantic similarity. Rather than requiring exact alignment, BERTScore finds the best semantic match for each token in the candidate among all tokens in the reference. IDF weighting emphasizes distinctive content words over common function words, and baseline rescaling produces interpretable scores.
- COMET learns evaluation from human judgments. Instead of designing formulas based on assumptions about quality, COMET trains a neural network on human quality assessments. This learned approach achieves higher correlation with human judgments than designed metrics.
- Three-input design catches source-reference misalignment. By incorporating the source text alongside the candidate and reference, COMET can verify that meaning was preserved from the original—catching errors that reference-only metrics miss entirely.
- No single metric is perfect. Lexical metrics offer speed and interpretability. Semantic metrics offer better human correlation. COMET offers source grounding. Production systems typically combine multiple metrics, using fast lexical metrics for iteration and semantic metrics for final selection.
- The evaluation paradigm continues evolving. The shift from designed metrics (BLEU) to semantic metrics (BERTScore) to learned metrics (COMET) sets the stage for LLM-as-a-Judge approaches, where evaluation becomes a dialogue with a capable model rather than a formula to compute.
LLM Evaluation and Alignment, The Foundational Ideas ebook for free
