WHY RELEVANCE ≈ 0.73

Four metrics, two different shapes

Every metric is a percentage, so it's tempting to hold them all to the same bar. But they measure different shapes of thing. Two compare an answer to a fixed target. One measures how spread out a set of guesses is. That difference is the whole story.

"Relevance" is a dispersion measure

We never compare the answer to a "correct" relevance. Instead the recipe (the RAGAS method) is:

1. Take the model's answer.
2. Ask an LLM: "what 3 questions would this answer respond to?"
3. Measure how close each reconstructed question is to the original question (cosine similarity, 0–1).
4. Average the three.

So the score isn't "how good is the answer" — it's "how tightly do three reconstructed questions cluster around the real one." Here's that happening for all five sampled questions. Watch the pattern: the first guess is almost always excellent — it's the 2nd and 3rd that pull the average down.

"Just keep the best guess?"

If we averaged only the single best reconstruction per question, relevance would leap past 0.80. But that throws away the robustness of sampling — and rewards an answer for being reconstructable once, even if it rambles. Toggle it:

Overall Answer Relevance: 73%

Even perfect paraphrases aren't 1.0

The second reason 0.80 is hard: embeddings don't score synonyms as identical. Two questions that mean the same thing but share few words land surprisingly low. Here's the measured cosine for genuine human paraphrases — note "Magnum Opus" vs "Great Work" at just 0.66:

When even hand-written paraphrases live between 0.66 and 0.95, the average of three machine-generated ones naturally settles in the 0.70s. There's no prompt that fixes geometry.

So what we ship

The honest design treats each metric by its real nature:

• Faithfulness, Answer Similarity, Recall — compared to a fixed target (a judge verdict, the gold answer, the source chunk). They saturate near 1.0 when correct, so gate them at 0.80.
• Answer Relevance — a dispersion measure with a structurally lower ceiling. Report it, with a 0.65 regression floor — high enough to catch a real collapse (an off-topic answer scores ~0.30) without pretending 0.80 is reachable.