<!-- Canonical: https://lucioduran.com/projects/bilingual-retrieval-evals
Language: en
Content-SHA256: fa14626a1feac11c8addc972b5b722d299b9b6510677149c8be4d8680709277f
-->

[Projects](<https://lucioduran.com/projects>) / Bilingual Retrieval Evals

# Bilingual Retrieval Evals

EN/ES retrieval evaluation with BM25 and multilingual MiniLM. Separates languages, directions, accents and cultural expressions, with replayable rankings and evidence.

Author: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 · License: [MIT](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/LICENSE>)

Page published: 2026-10-22 · Page updated: 2026-10-22

[Repository](<https://github.com/lucioduran/bilingual-retrieval-evals>) · [Original evidence](<https://raw.githubusercontent.com/lucioduran/bilingual-retrieval-evals/8cc64e903e89ef399db5c4648b23cee52038b83b/results/eval/report.json>) · [Release v0.1.0](<https://github.com/lucioduran/bilingual-retrieval-evals/releases/tag/v0.1.0>)

## Problem and relevance to agents

An aggregate retrieval score can hide a search engine that works in English and fails across languages. Translating queries does not remove cultural ambiguity or lexical distractors.

The project exposes which documents each method retrieves for parallel English and Spanish queries. It makes ranking errors visible even when Recall@3 is perfect.

## Architecture

1. **Corpus and explicit judgments.** 32 fictional documents, 68 query variants and candidate labels with supporting spans. Development and evaluation families remain separate.
2. **Directional retrieval.** Each query searches 16 target-language documents. Methods are BM25, accent-folded BM25 and a revision-pinned multilingual MiniLM encoder.
3. **Breakdowns and replay.** Saved vectors, complete rankings, group metrics, paired contrasts and family intervals permit metric reconstruction without downloading the model.

- [Design and methodology](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/docs/methodology.md>)

## What this demonstrates

- Multilingual evaluation with explicit denominators and four separate retrieval directions.
- Cultural queries, distractors and paired accent contrasts with traceable relevance judgments.
- Ranking-error and uncertainty analysis that does not treat correlated variants as independent samples.

## Published results

Local measurement on Intel Xeon W-3223 with two threads. Headline tables use 32 original queries from eight evaluation families, excluding accent-omission variants to avoid overweighting Spanish.

Cutoff 3, BM25 k1=1.2 and b=0.75, 128-token embedding limit. The full report includes 46 evaluation variants. Intervals use 2,000 family bootstrap resamples and a different estimator from the original-query pooled average.

### Retrieval over 32 original queries

| Method | Recall@3 | NDCG@3 | MRR@3 |
| --- | --- | --- | --- |
| BM25 | 0.719 | 0.719 | 0.719 |
| Accent-folded BM25 | 0.719 | 0.719 | 0.719 |
| Multilingual MiniLM | 1.000 | 0.977 | 0.969 |

### NDCG@3 by direction

| Direction | Queries | BM25 | MiniLM |
| --- | --- | --- | --- |
| EN → EN | 8 | 1.000 | 1.000 |
| EN → ES | 8 | 0.625 | 0.954 |
| ES → EN | 8 | 0.375 | 0.954 |
| ES → ES | 8 | 0.875 | 1.000 |

MiniLM reached 1.000 Recall@3 but ranked a distractor above the relevant document for a cultural query and a parcel query. Finding a document in the top three does not imply correct ordering.

Family intervals, entity breakdowns and seven accent pairs per target language are published with their denominators. [Results summary](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/docs/results.md>).

## Reproduce

Python 3.11 or later. Validation and replay use the standard library. Neural measurement requires pinned dependencies and an explicit public encoder download.

Commands use a POSIX shell and fresh output directories. Run one measurement at a time. The lock targets the measured Intel macOS environment; other systems may need compatible PyTorch distributions.

### Replay saved evidence

```
git clone https://github.com/lucioduran/bilingual-retrieval-evals.git
cd bilingual-retrieval-evals
git checkout 8cc64e903e89ef399db5c4648b23cee52038b83b
python -m bilingual_retrieval validate
python -m bilingual_retrieval verify --report results/dev/report.json
python -m bilingual_retrieval verify --report results/eval/report.json
```

### Repeat the local measurement

```
python3.11 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-local.lock
python -m bilingual_retrieval download
OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false python -m bilingual_retrieval run --split dev --output build/dev
OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false python -m bilingual_retrieval run --split eval --output build/eval
python -m bilingual_retrieval verify --report build/dev/report.json
python -m bilingual_retrieval verify --report build/eval/report.json
```

## Scope and limitations

- Synthetic author-known corpus. Labels still lack independent bilingual human review and do not represent dialect coverage or real traffic.
- Eight families provide weak uncertainty estimates. Entity and language groups are not causal treatments.
- Timings are one batched pass, not latency percentiles. Replay checks internal consistency, not the authenticity of embeddings or measurements.

- [Vector search and multilingual evaluation](<https://lucioduran.com/blog/vector-databases-pgvector-qdrant-pinecone>)

## Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.

- Reviewed revision: [` 8cc64e903e89ef399db5c4648b23cee52038b83b `](<https://github.com/lucioduran/bilingual-retrieval-evals/commit/8cc64e903e89ef399db5c4648b23cee52038b83b>)
- Revision recorded by the experiment: [` d89585a8e53aa43a0fa8bc6fe50d43635172c5de `](<https://github.com/lucioduran/bilingual-retrieval-evals/commit/d89585a8e53aa43a0fa8bc6fe50d43635172c5de>)
- Report timestamp (UTC): 2026-10-21T22:59:22.243350+00:00
- Release published: [v0.1.0](<https://github.com/lucioduran/bilingual-retrieval-evals/releases/tag/v0.1.0>) · 2026-10-21T23:04:15Z

- [README and usage](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/README.md>)
- [Original evidence](<https://raw.githubusercontent.com/lucioduran/bilingual-retrieval-evals/8cc64e903e89ef399db5c4648b23cee52038b83b/results/eval/report.json>)
- [Results summary](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/docs/results.md>)
- [Design and methodology](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/docs/methodology.md>)
- [Data and provenance](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/data/README.md>)
- [Development evidence](<https://raw.githubusercontent.com/lucioduran/bilingual-retrieval-evals/8cc64e903e89ef399db5c4648b23cee52038b83b/results/dev/report.json>)
- [Error analysis](<https://raw.githubusercontent.com/lucioduran/bilingual-retrieval-evals/8cc64e903e89ef399db5c4648b23cee52038b83b/results/eval/errors.json>)
- [Models, revisions and licenses](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/docs/model-notices.md>)
- [Measured environment dependencies](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/requirements-local.lock>)
- [MIT license](<https://github.com/lucioduran/bilingual-retrieval-evals/blob/8cc64e903e89ef399db5c4648b23cee52038b83b/LICENSE>)