When the query and the document aren't in the same language
A support agent in Lisbon searches an internal knowledge base for “erro de autenticação após renovação do certificado”. The articles are in English. Lexical retrieval returns two documents, both because they happen to contain the word “certificado” in a quoted configuration string.
There is no shared vocabulary to score. This is the one case where the two signals are not two imperfect options — one of them is structurally incapable, and the other might work.
Query: "erro de autenticação após renovação do certificado"
LEXICAL (BM25) DENSE (multilingual model)
1. Sample config (certificado field) ✗ 1. Authentication fails after ✓
certificate renewal
2. Localisation glossary ✗ 2. Certificate rotation checklist ~
(nothing else scores) 3. Auth error codes reference ~
4. Renewing an expiring cert ~
5. TLS handshake failures ~Why lexical retrieval scores zero, and why “zero” is the wrong word
Term-based scoring compares tokens. Portuguese tokens and English tokens are different strings, so the overlap is empty and the score is zero for every document — except where a proper noun, a code identifier, or a cognate coincides.
That exception is the dangerous part. A cross-language lexical query does not return nothing; it returns whatever matched by accident, ranked confidently. Cognates and shared technical strings produce a short list of irrelevant documents, which looks like a working retriever having a bad day rather than a component that cannot do the job. In fusion, that list contributes rank information as if it were meaningful.
Worth knowing what will and won’t coincide: shared identifiers and code strings will, technical terms borrowed from English often will, cognates sometimes will with different inflections that stemming won’t reconcile, and everything else won’t.
What a multilingual embedding does and doesn’t give you
A multilingual embedding model is trained so that text with the same meaning in different languages maps to nearby points. When it works, cross-language retrieval needs no translation step at all — you embed the Portuguese query and it lands near the English document.
Four caveats, all of which are corpus- and model-specific and none of which you can assume away:
Language coverage is uneven. A model’s quality across its supported languages follows the volume of training data. Its behaviour on a widely represented language and on a low-resource one are not comparable, and a model advertising a long list of languages is not claiming equal quality across them.
Same-language matches often win. Given a mixed corpus, a query in one language frequently ranks documents in that language above better documents in another, because language is part of what the representation encodes. Retrieval quietly becomes language-filtered.
Domain vocabulary transfers worst. Alignment across languages is strongest for common language and weakest for specialised terminology — which is what technical corpora are made of.
Verification requires per-language testing. An aggregate quality number over a multilingual test set tells you about the mix in that set. Your users are not distributed like that set.
So the honest statement is: a multilingual dense retriever makes cross-language retrieval possible, at a quality you have to establish per language pair on your own content.
Translating instead
The alternative is to translate, and there are two directions with different economics.
Translate the query at query time. One short translation per query, then retrieve normally in the corpus’s language — including with lexical retrieval, which is the point: translation restores the lexical signal, and with it identifier matching, exact phrases and everything else term-based retrieval does well. Costs a call in front of retrieval, adds non-determinism, and mistranslates domain terms unless you supply a glossary.
Translate the corpus at indexing time. Index every document in every language you support. Query-time cost is zero, both signals work natively per language, and you pay in storage, in indexing cost, and in a translation debt that must be repaid on every document update. For a small, stable, high-value corpus this is often the right answer and it is rarely considered.
A middle option worth naming: translate the query and retrieve in both forms, then fuse. The original query catches documents in the user’s own language; the translation catches the rest. This is the arrangement that behaves best on genuinely mixed corpora, and its cost is an extra retrieval round plus the merge.
Detect the language, then decide
None of the above works as a global setting, because a corpus with documents in several languages needs different handling per query. So the pipeline gains a step: identify the query’s language, then choose.
Two failure modes in that detector, both common:
Short queries are hard to identify. A two-word query gives a detector very little, and identifiers and product names give it nothing. Confidence matters here; a low-confidence detection should widen the strategy rather than commit to it.
Code-mixed queries are normal, not exceptional. Users routinely write a sentence in one language containing English technical terms. A detector returns one label; the query has two. This is where running both forms and fusing beats routing, because there is no single correct route.
The failure that hides in a mixed corpus
The reason this class of problem survives review: aggregate quality looks acceptable while one language fails completely.
If most of your traffic is in the corpus’s language, and a minority arrives in others, your overall numbers are dominated by the majority. The minority’s queries return the accidental-cognate list, users in that group learn the search does not work and stop using it, and their traffic disappears from your logs — which improves the aggregate further. Measure per language or you will not see it.
There is a related indexing failure worth checking at the same time: if your lexical analyser is configured for one language, documents in others are analysed under the wrong stemmer and, for languages without whitespace word boundaries, tokenised into uselessness. That is a per-language index defect that looks exactly like a per-language retrieval defect, and the fix is entirely different.
Telling whether it’s your problem
Run a language detector over a month of query logs and over a sample of your corpus, and put the two distributions side by side. Three findings and three responses:
- Queries and corpus in one language. No problem to solve; don’t add multilingual machinery.
- Queries in several languages, corpus in one. You need a cross-language strategy, and translating the query is the cheapest one that restores both signals.
- Both mixed. The hardest case. Expect same-language bias in the dense ranking and per-language analyser defects on the lexical side, and evaluate per language pair rather than in aggregate.
Then, for each language pair that matters, build a small set of queries with known correct passages and record where the passage lands. It is the same measurement as any other retrieval question — the only difference is that here you must keep the results separated by language, because averaging them is exactly how this failure stays hidden.