A search result can look sensible without containing the material someone needs. Evaluating retrieval means comparing the returned items with a stated information need and a set of relevance judgments. Elastic's ranking-evaluation guide identifies three ingredients: a document collection, typical search requests and ratings for those requests. TREC likewise builds evaluations around defined tasks, corpora, queries and judgments [1][2]. A score becomes interpretable only when those ingredients are visible. The same number can mean different things on two different collections.

Start with a sentence such as: this test checks whether the first five retrieved documents include the instructions needed for this question. That sentence defines a boundary. It does not say whether a generated answer explains the instructions accurately, whether a page loads quickly or whether a user completes the task. Those may all deserve their own checks. This guide follows one completely invented collection, with every relevant item declared in advance, so the arithmetic is easy to audit. The purpose is to understand what the measures count before trusting a dashboard that presents them as evidence of overall quality.

Retrieval evaluation requires an operational target and a unit of comparison. Elastic describes a collection of documents, representative requests and per-request relevance ratings; TREC organizes shared tasks around specified corpora, queries, judgments and protocols [1][2]. An evaluation run is therefore more than a vector of scores. Its meaning depends on the candidate universe, ranked unit, information need and rules used to assign relevance.

Define the boundary before calculating a metric. A top-five document evaluation tests selection and ordering among documents. It does not directly measure a later language model's answer, latency, permission handling or user satisfaction. Those outputs require evidence about their own behavior. The fixture below deliberately makes all eight document judgments complete for one invented query, removing uncertainty about the denominator. It is not a benchmark, a recommendation for test-set size or a claim about any retrieval model. By keeping the task and labels explicit, it becomes possible to distinguish an arithmetic error from an incomplete judgment set or a metric that simply answers the wrong question.

Our fictional drawing app has eight help documents, called A through H. The request is: find instructions for exporting a project as SVG through any supported interface. A covers desktop export, C covers web export, F covers batch export and H covers command-line export. Those four are relevant by our stated rule. B explains importing SVG; D changes the colour theme; E lists shortcuts; G manages fonts. Those four are not relevant to this request. These document descriptions and labels are invented for the exercise.

Notice how a change in the request could change a label. Someone asking specifically for desktop steps might not accept a command-line procedure. Elastic says judgments must be made for each query according to the user's information need [2]. Decide that need before looking at the system's preferred results. Also state whether you rate whole documents or individual passages. The historical TREC English-judgment guidance treats a document as relevant when any useful part addresses its topic; TREC's broader current guidance says judgment scales and coverage vary by task [1][4]. A label policy belongs to its evaluation, not to every possible search.

Define the synthetic corpus U = {A, B, C, D, E, F, G, H} and relevant set R = {A, C, F, H}. The information need accepts SVG-export instructions through any supported interface of an invented drawing app. The relevant items describe desktop, web, batch and command-line export respectively. B concerns SVG import, while D, E and G concern themes, shortcuts and fonts. All eight labels are specified by construction; no annotator or model was asked to infer them.

This definition makes relevance conditional on the need. Narrowing the request to desktop export would require reconsidering the judgments rather than reusing R automatically. Elastic explicitly requires a rating set for each test query [2]. Declare the evaluated unit as well: whole-document relevance and passage relevance can have different criteria. TREC's historical English-judgment page permits relevance from a useful part of a document, whereas its general guidance notes variation in scales and coverage across tasks [1][4]. Preserve those scopes. A useful evaluation record contains the labeling instruction and rationale, so later readers can tell whether a disagreement concerns retrieval or the definition of success.

Suppose the invented search returns A, B, C, D and E as its first five results. Two of those five, A and C, are relevant. Precision at five is therefore 2/5, or 40%: how much of this five-result list is useful under our labels? Recall at five is 2/4, or 50%: how much of the collection's four relevant documents did the list find? These are the precision and recall definitions applied to our fixture [2]. Neither percentage is the probability that a generated answer will be correct.

Now look only at the first three results, A, B and C. Precision at three is 2/3, about 66.7%, while recall remains 2/4, or 50%. Adding D and E made the list longer without finding another relevant item. Precision fell; recall did not rise. That is why a cutoff belongs beside the metric name. A system showing P@3 and another showing P@5 have not necessarily been compared on the same task. Decide how many results a person or later component will actually receive, then evaluate that boundary.

For the hypothetical result list L5 = [A, B, C, D, E], the relevant intersection is {A, C}. With five returned distinct documents and complete judgments, P@5 = |L5 ∩ R|/5 = 2/5 = 0.40, while R@5 = |L5 ∩ R|/|R| = 2/4 = 0.50. Elastic's definitions use the returned cutoff for precision and the relevant set for recall [2]. The measures answer different questions about the same retrieved set.

At cutoff three, L3 = [A, B, C] still contains two relevant documents, producing P@3 = 2/3 and R@3 = 1/2. The extra two positions reduce precision without changing recall. This result follows from the specified labels, not an empirical property of a model. Record k explicitly, along with conventions for fewer than k results, duplicate IDs and unrated items. The present calculation avoids those edge cases by returning exactly k distinct, fully judged documents. Once a real run departs from those conditions, a metric name alone is insufficient to reconstruct its denominator or compare its score fairly with another run.

Reorder the same five documents as B, D, A, E and C. Precision at five remains 40% and recall at five remains 50%, because A and C are still the two relevant items. Yet a reader now passes two unsuitable results before reaching the first useful one. Elastic notes that precision and recall at a cutoff do not distinguish positions within that cutoff [2]. A list can therefore keep both scores unchanged while making its first useful result harder to reach.

Reciprocal rank asks a narrower question: where is the first relevant result? The original order has its first relevant item at position one, giving 1/1 = 1. The reordered list reaches it at position three, giving 1/3, about 0.333. Mean reciprocal rank averages such query-level values across an evaluation set [2]. It emphasizes the first success and does not reward finding additional relevant items after that success. If the task is to gather several different instructions, keep a coverage measure as well. Choosing a metric means choosing which differences the evaluation will notice.

Compare L5 with L5′ = [B, D, A, E, C]. Their membership is identical, so both have P@5 = 0.40 and R@5 = 0.50. Their first relevant ranks differ: one versus three. This demonstrates the set-based nature of cutoff precision and recall documented by Elastic [2]. A metric can be calculated correctly while remaining insensitive to a behavior the application cares about.

Reciprocal rank is 1/r for the first relevant rank r. For the two constructed orderings it is 1 and 1/3 respectively. A mean reciprocal rank aggregates this quantity over queries, under the selected cutoff and no-hit convention [2]. The statistic does not distinguish two lists that share the same first relevant position but differ afterward. Therefore, it aligns naturally with a first-success question, while a multi-item research task also needs evidence of coverage. Keep the evaluation objective explicit rather than treating every rise in reciprocal rank as an improvement in all aspects of retrieval. A useful report can retain several complementary measures without collapsing them into one unexplained score.

A ranking measure can also reward useful items throughout a list while giving earlier positions more weight. Scikit-learn describes normalized discounted cumulative gain, or nDCG, as discounted relevance divided by the best possible discounted result [3]. The definition of relevance and the discount rule still matter. We will use a simple binary example: useful items get one, others zero, and the weight at position r is one divided by log base two of r plus one. These are declared calculation rules, not a measured user-attention pattern.

For A, B, C, D, E, the relevant positions are one and three, contributing 1 and 0.5, for a total of 1.5. The ideal top five places our four relevant documents first, with weights about 1, 0.631, 0.5 and 0.431, totaling 2.5616. The normalized result is about 0.586. For B, D, A, E, C, useful items at positions three and five contribute about 0.887, giving about 0.346 after the same normalization. The equal precision and recall scores now lead to different ranking scores, because the order affects the declared discount.

Use the explicit binary-gain convention DCG@5 = sum over r = 1…5 of rel(r)/log2(r + 1), with rel(r) in {0, 1}. Define nDCG@5 as DCG@5 divided by the ideal DCG at the same cutoff. This follows the discounted-gain and ideal-normalization approach described by scikit-learn [3]. No score ties or negative relevance values occur in this fixture, and the ideal denominator is positive.

For L5, DCG is 1/log2(2) + 1/log2(4) = 1.5. Four relevant documents exist, so IDCG@5 = 1 + 1/log2(3) + 1/log2(4) + 1/log2(5) = 2.561606…. Thus nDCG@5 is approximately 0.58557. For L5′, DCG is 1/log2(4) + 1/log2(6) = 0.886853…, and nDCG is approximately 0.34621. These are original arithmetic results, not reported benchmark values. For graded judgments, state the gain transformation rather than assuming all implementations interpret a rating identically. Also specify tie treatment and the zero-ideal-gain convention; the documented scikit-learn interface makes cutoff and tie behavior explicit [3].

Our eight-document example is unusually tidy because every label is known. Real evaluations may judge only part of a much larger collection. TREC's historical English-judgment guidance explains pooling: selected results are gathered for assessment, with completeness relying on an assumption that most relevant documents have been found. It also warns that the document collection and judgment files must match [4]. That is different from proving that every document outside the judged set is useless.

Suppose a real result lacks a relevance label. Record it as unjudged before deciding how the chosen metric will handle it. If it is genuinely relevant, silently treating it as a known negative can change the conclusion about the run. Equally, labeling every unknown result as relevant would be unjustified. The practical response is to inspect judgment coverage and keep the scoring policy visible. TREC says coverage and rating scales vary across tasks [1]. A recall figure calculated against known relevant items describes those judgments; it cannot by itself prove that all relevant material in an evolving collection has been found.

Complete labels are an explicit property of this synthetic fixture, not an assumption to carry into a production corpus. The historical TREC English-judgment page describes pooled assessment and explains completeness as an assumption that most relevant items have been assembled. It requires alignment between the document collection and qrels [4]. Current TREC guidance more broadly notes that judgment coverage varies across tasks [1].

Preserve a distinction among judged relevant, judged nonrelevant and unjudged items in the evaluation data, even if a chosen scoring protocol maps unknowns to a particular treatment. The mapping is a convention, not a factual judgment about the missing item. Recall's denominator is especially important: a result relative to known positives does not establish completeness over all possible relevant documents. Record the corpus version, judgment version and coverage of the returned candidates. If new retrieval methods expose many previously unjudged items, investigate those items before attributing every score change to better or worse relevance. This is a limit on the available evidence, not a claim that every pooled test collection is invalid.

A test suite contains different questions, and they may not have equal numbers of relevant documents. Consider a separate invented two-query example. For query one, a system finds two of four relevant items, giving recall of 50%. For query two, it finds two of twenty, giving recall of 10%. Giving the two queries equal weight produces average recall of 30%. Pooling the counts instead produces four found out of twenty-four relevant items, about 16.7%. Both calculations follow their definitions; they give different influence to the two questions.

State which calculation your report uses and retain the per-query results. Elastic recommends evaluating representative requests so that improving one type of query does not hide regressions in another [2]. For a practical review, you might group exact identifiers, broad exploratory requests and ambiguous wording, then inspect each group. Those are proposed categories, not measured proportions of a real workload. A high overall average can coexist with poor performance on a group that matters greatly to your users. Decide whose experience the evaluation is intended to represent before selecting the weighting.

Aggregation changes the evaluation target. In a separate synthetic pair of queries, relevant-set sizes are four and twenty, with two relevant items retrieved for each. Query recalls are 0.50 and 0.10. Their unweighted macro mean is 0.30; pooling true-positive counts and relevant denominators gives 4/24 = 0.166667…. The latter effectively gives greater weight to queries with larger relevant sets. This is an arithmetic distinction, not a prescription that one aggregation is always preferable.

Retain per-query measures and the aggregation rule alongside the summary. Elastic emphasizes a representative suite because optimizing one query can harm other query types [2]. Proposed diagnostic slices can separate exact identifiers, exploratory needs, rare categories or ambiguous terms, provided membership is defined before interpreting the result. Also report how many queries contribute to each summary and how failures or empty outputs are counted. Dropping difficult or missing-result cases from the denominator can make the reported average describe an easier population. An honest comparison specifies the intended population and weighting, then examines whether a summary conceals a material regression within it.

An evaluation set becomes less convincing as a final check if you repeatedly adjust the system to make its scores rise. Scikit-learn's evaluation guidance explains how parameter choices can leak information from a test set and overstate generalization. TREC describes a development stage followed by separate evaluation data [1][5]. The same principle applies when choosing retrieval weights, candidate counts or other settings: the cases used to choose a configuration should not be the only evidence used to assess it.

Use development examples to understand failures and make changes. Then assess the selected configuration on a held-out set whose labels did not guide those changes. If you examine that final set, tune again and report the improved result, be clear that it has now influenced development. Also consider related questions: several small rewrites of the same request may provide less separation than their different text suggests. Scikit-learn discusses group-aware and time-aware splitting when observations have those structures [5]. Apply that idea thoughtfully to your workload instead of assuming a random division automatically creates an independent challenge.

Selection and final assessment should use distinct evidence. Scikit-learn explicitly identifies the risk of repeatedly tuning hyperparameters against a test set, and describes validation data with a final held-out test [5]. TREC's published process likewise separates system development from later evaluation queries and labels [1]. Applying this principle to retrieval means distinguishing configuration selection from the score subsequently used to assess the selected configuration.

Keep an experiment log of candidate settings and the data used to choose among them. If the final test informs another weight change, it has become part of the adaptive development process; merely retaining its test filename does not restore independence. Dataset structure matters as well. Scikit-learn discusses group-wise and time-aware evaluation when observations are dependent or time-ordered [5]. For retrieval, a proposed analogous check is whether near-duplicate queries, shared source documents or successive versions of one request cross the intended split. Define the split around the generalization question you want to answer, while acknowledging that a held-out sample still cannot guarantee performance on every future query.

A relevance score compares a returned list with labels. It does not read a generated answer for you. Imagine that our drawing-app search retrieves A, the correct desktop-export instructions, but a later answer mistakenly describes importing a file. The retrieval result can be successful under its labels while the answer fails the user's task. Conversely, an answer could repeat a familiar fact without retrieving the evidence the evaluation expected. These are logical examples of different outputs, not observations about a named system.

Keep separate checks for the questions you actually need to answer: did retrieval find the supporting material, did the response use it accurately, and did the user receive a usable result? TREC's task-specific protocols and Elastic's rated-result metrics illustrate why the evaluated object must be defined [1][2]. The same boundary applies to other qualities. A ranking metric does not directly measure response time, information freshness or access permissions. Record those requirements independently when they matter. A useful dashboard makes each result's scope clear rather than allowing one attractive retrieval percentage to stand for the entire application.

The evaluated object constrains the conclusion. Precision, recall and the ranking measures above consume retrieved identities and relevance judgments; a downstream generated answer is not an argument to those calculations [2][3]. Therefore, an improvement in one of these metrics does not logically establish that generated claims are accurate, complete or properly supported. A hypothetical system can retrieve A correctly and still produce an import instruction when the request concerns export.

Evaluate retrieval coverage, evidence use and final-answer behavior as distinct outputs with appropriate labels or review criteria. The distinction is consistent with TREC's task-specific evaluation protocols [1], but the proposed application-level separation is an original design recommendation, not a claim that every TREC track uses the same pipeline. Keep latency, freshness and access-control requirements explicit as well; none is determined by the relevance arithmetic in this article. A correct relevance judgment can concern a document that is unsuitable for a separate operational requirement. Reporting those dimensions separately makes a disagreement informative instead of forcing unlike outcomes into a single number with an unclear interpretation.

A small evaluation record should identify the collection version, the questions, the relevance policy, the retrieved unit, the cutoff and the scoring rules. Save the original result lists and per-query outcomes, not just the final average. Include the date, the configuration and any unjudged results that need attention. These entries connect the document, request and rating ingredients described by Elastic with the task and protocol information emphasized by TREC [1][2]. They let someone else understand what changed when a later score moves.

Finally, define how difficult cases are reported. If a query has no labeled relevant items, ordinary recall has a zero denominator; do not silently invent a percentage. If the system returns no results, retain the case and apply the declared scoring policy. Keep absent-answer tasks visible as their own requirement when that is part of the product. Our fully labeled toy collection avoids these uncertainties to teach the calculations. Real evaluation becomes useful when it exposes its limits just as clearly: what was tested, what counted as relevant, which failures mattered and what still needs another kind of evidence.

Preserve a compact evaluation contract with the run: corpus and judgment versions, query set, retrieved unit, relevance threshold or grades, cutoff, gain and discount definitions, tie handling, unjudged policy, empty-output policy and aggregation weights. Store document identities and per-query values so the summary can be reproduced. These details connect Elastic's collection/request/rating model with TREC's explicitly task-dependent protocols [1][2]. Record which data informed configuration selection and which data remained held out.

Handle undefined cases deliberately. If the relevant set is empty, ordinary recall's denominator is zero; report the selected convention or a separate task stratum rather than treating an arbitrary number as a measured success. A zero ideal gain likewise requires a declared nDCG policy. Missing result lists should remain accounted for under the evaluation protocol. The complete eight-document fixture establishes exact arithmetic under stated assumptions, while a real run needs additional evidence about labels, sampling and deployment conditions. A defensible conclusion names the measured behavior and its limits, allowing future changes to be evaluated without turning a historical score into a universal claim about search quality.