Odoo RAG: a docs assistant I can measure

An assistant that answers only from the Odoo 19 developer docs and my migration notes, cites its sources, and says so when it does not know. The part I care about most is the evaluation: every retrieval change is measured instead of assumed.

Source code on GitHub →

How it works

Question
→
Translate if Spanish
→
Vector + keyword search
→
Rank fusion
→
llama3.2
→
Cited answer or NOT_IN_DOCS

It runs on PostgreSQL with pgvector and local Ollama models, over 341 chunks from 14 documents. There are no API keys and no per-call cost. The trade-off is speed: a median of about 20 seconds per answer on a laptop.

odoo-rag

$ uv run python -m odoo_rag.ask "¿Cómo ejecuto solo algunos tests desde la línea de comandos?"Para ejecutar solo algunos tests desde la línea de comandos, puedes utilizarel comando `odoo-bin --test-tags` con los tags correspondientes.$ odoo-bin --test-tags '/sale,-slow'[2] developer_reference_backend_testing.rst | Testing Python code > Test selection > Invocation[3] developer_reference_backend_testing.rst | Testing Python code > Test selection > Examples$ uv run python -m odoo_rag.ask "What is the capital of France?"NOT_IN_DOCS

Replay of a real run, shortened. The first question is in Spanish; the docs are in English. The second is out of scope, so the assistant refuses.

The numbers

I compared four retrieval modes on 45 hand-written questions, 35 in English and 10 in Spanish. Recall@5 is how often the right passage appears in the top five results.

Recall@5, all questions
Keyword	73%
Vector	67%
Hybrid	73%
Hybrid + translation	82%

Recall@5, Spanish questions
Keyword	40%
Vector	20%
Hybrid	50%
Hybrid + translation	90%

The surprise was where the gain came from. Hybrid search alone did not beat keyword search on Recall@5. Translating the question did: Spanish went from 50% to 90%, and English stayed at 80%.

On 20 in-scope and 8 out-of-scope questions, 90% of answers contained the expected terms, 80% had valid citations, none was wrongly refused, and 7 of 8 out-of-scope questions were refused.

What it still gets wrong

A wrong answer on a key question. Asked how to stop a data file from overwriting records users edited, it answers forcecreate. The right answer is noupdate.
One refusal it should have made. It answered a question about the 2022 World Cup.
The evaluation has limits too. I wrote the questions against this same corpus, so the numbers compare modes and do not predict performance elsewhere. The citation metric checks that a cited excerpt exists, not that it supports the claim.

Both failures are recorded in the results file, so any future change has to prove it fixes them.