Sanity check on my hybrid RAG settings

Community Calls
1

Hi all, I’d love a quick review of my RAG retrieval settings and some advice on the vector store.

Current stack

  • Backend: FastAPI (Python) with Docker workers for ingestion
  • Vector DB: ChromaDB (one collection per project)
  • Keyword search: PostgreSQL full-text search (tsvector, ts_rank_cd)
  • Cache: Redis
  • Embeddings: Mistral, OpenAI, Jina (via Ollama)
  • LLMs: OpenAI, Mistral, Gemini
  • Reranker: BAAI/bge-reranker-base
  • Chunking: LlamaIndex SentenceSplitter

Indexing

  • Chunks of 1,000–1,500 characters with 200 overlap. Tables are split by whole rows.
  • A title/breadcrumb header (up to 200 characters) is added before embedding.
  • Normalized embeddings with cosine distance.

Retrieval

  • sim=1−dsim=1−d, with threshold τ=0.3τ=0.3 (range 0.2–0.45). If nothing passes, retry at 0.5τ0.5τ.
  • Small collections (300 chunks or fewer): score=0.65⋅sim+0.35⋅kwscore=0.65⋅sim+0.35⋅kw
  • Large collections: RRF, score=∑w/(60+rank)score=∑w/(60+rank). Vector, keyword and exact-ID lists each have w=1w=1, and structured records have w=2w=2.
  • Results found only by keyword search are kept if sim≥0.75×sim≥0.75× the best vector similarity.

Reranking

  • Reranks the top 30 within an 800 ms budget, reading up to 256 tokens per chunk, with d=1−σ(s)d=1−σ(s).
  • Skipped when the top result has sim≥0.75sim≥0.75 and leads the second by at least 0.10.

Final context

  • top-k = 5, with near-duplicates removed and at most 3 chunks per page.
  • Confidence requires top-1 similarity of at least 0.16 and a top-3 average of at least 0.20. Otherwise it retries with the original query.

Questions

  1. Is a weighted mix for small collections and RRF for large ones sensible, or should I use one method everywhere?
  2. Are 256 tokens per chunk too few for bge-reranker-base when chunks are about 1,400 characters?
  3. Are the confidence floors (0.16 / 0.20) too low for production?
  4. Since keyword search already runs in Postgres, would you move vectors to Postgres + pgvector, or switch to OpenSearch / Elasticsearch for native hybrid search (BM25 plus vectors) at multi-tenant scale? Which works better in production?

Thanks!

2

honestly your stack is fine, the part I’d worry about is the glue - Chroma + Postgres FTS + Redis workers means 3 places that can drift when a doc gets re-ingested. pgvector is the easy win if you already run Postgres, keeps keyword + vector in one place. we hit the same thing and ended up keeping chunks, embeddings and indexes in one table that updates incrementally in Pixeltable.. we wrote up a side by side vs Celery+Postgres+pgvector here if useful: github.com/pixeltable/showcase-call-intelligence

3

Thanks a lot, this is really helpful! You’re right about the glue. Keeping Chroma, Postgres FTS and Redis in sync during re-ingestion is exactly the kind of drift I’ve been worried about. Consolidating on Postgres + pgvector makes a lot of sense since we already run Postgres. I’ll go through your Pixeltable side-by-side and definitely try it out.

Appreciate you sharing it!

If you have any thoughts on the reranker token limit or the confidence thresholds, I’d love to hear them too.